25 Log Files That Will Transform Your vSphere Troubleshooting

Everyone loves logs right? Right?? I can’t be the only one.

In all seriousness though, logs are one of those things that you have to stay on top of, so for this post, we’re going to take a deep dive into the world of VMware logging. Yes, I know you are all super excited. Let’s run down the 25 most important VMware logging files and the various tools that you can use to keep track of your VMware environment. You likely won’t need all of these vSphere log files every single day, but when you do need them, you’ll be glad you have a general idea of where to go. To that end, I’d recommend bookmarking this page for future reference.

We’ll break this post into 2 sections. First, we’ll cover the vSphere log files on the host and then we’ll move on to the vCenter logs.

As a side note, it is worth pointing out that you can also you PowerCLI to query the VMware logging resources with the Get-Log cmdlet.

ESXi Host Logs

Part of the VMware logging you can benefit from comes from ESXi recording host activity in vSphere log files, using a Syslog facility. Typically one of the easiest and fastest ways to check these log files out is using tools like a cat on the command line. You can also view various logs through the DCUI as well. All VMware hosts run a service for logging system information.

This service, vmsyslogd, logs messages from the VMkernel and other system components for auditing and diagnostic purposes. By default, the logs are directed to a local scratch location or ramdisk. The scratch space is created automatically during ESXi installation in the form of a 4 GB Fat16 local scratch partition. If storage space is unavailable, the host will store data on a ramdisk, which is not persistent across reboots.

That being the case, many admins choose to send these logs to a persistent datastore as vSphere datastore log files or remote logging server for retention. One of the most important things to know is that it’s important to realize where you’re installing ESXi. If you opt to install it to a USB stick or sd card, if the host crashes, that can make for a bad day! This means that you’ll want to pay extra attention and have the logs written to a central logging location. I’ll highlight some of the most popular logs below and indicate what they do. I highly recommend having a Syslog server of some sort in your environment.

Another important vSphere log file that I like to point out is the boot.gz file. It’s located in /var/log/boot.gz. This one is important to troubleshoot slow boots and can indicate why the host is hanging or freezing. I’ve found myself using it from time to time and it’s been helpful.

You can change the location of the Syslog location under each host at the location below. To specify a remote logging host, enter a value in the format of protocol://hostname:port. Example: tcp://hostname:514

vCenter Logs

VMware logging in the VMware vCenter Server Appliance is located in the /var/log/vmware/ folder. You can get to it either via the console interface or via ssh. You can also export a bundle of logs via the vSphere Web Client.

Some important ones to keep in mind:

vmware-vpx\vpxd.log

Use this to troubleshoot issues related to vCenter operations. Everything from DB connectivity problems to vCenter crashes can be found here. This log will have a LOT of information in it and is a good place to start on many issues.

invsvc\wrapper.log

vmware logging to troubleshoot why the inventory service will not start.

sso\vmware-sts-idmd.log

This is a good log to use as a “one-stop-shop” for SSO authentication issues. Authentication requests/failures, as well as problems with an identity source, will post here.

vsphere-client\logs\vsphere_client_virgo.log

A great place when troubleshooting errors within the Web Client. If you receive errors from simply clicking on objects, you begin chasing them down here!

vRealize Log Insight

While VMware logging certainly helps troubleshooting issues when they appear or perform Root Cause Analysis (RCA), it is only possible if they are actually available. In case of a disaster or just a bad server event, you may lose your VMware logging, especially if VMware system logging is not configured on a host or with default values.

It is strongly recommended to set up a Syslog server in your environment to gather vSphere log files as well as others. VMware proposes vRealize Log Insight, a Syslog solution that embeds plenty of dashboards, rules, and filters tailored to vSphere environments.

“vRealize Log Insight offers a tailored solution to consolidate third-party software and VMware logging”

Formerly known as VMware Log Intelligence, VMware vRealize Log Insight Cloud is a cloud service that offers a managed solution to get visibility across various public and private cloud through log forwarding. You will find the features of any respectable Syslog server such as log aggregation, analytics, dashboards, custom alerting…

You can also install content packs to process data collected from third-party sources such as AD, Sharepoint, DellEMC, and other software vendors.

To properly protect your VMware environment, use Altaro VM Backup to securely backup and replicate your virtual machines. We work hard perpetually to give our customers confidence in their VMware backup strategy.

To keep up to date with the latest VMware best practices, become a member of the VMware DOJO now (it’s free).

Wrap Up

To wrap up, even though VMware logging is highly resilient, keep in mind that you need to consider a remote Syslog collector for your ESXi hosts. It’s one of the most important things to do as you don’t want to lose your VMware logging in case of a serious outage. You don’t have to use the VMware Syslog Collector if you already have one in place, but consider using that for a smaller shop with no existing Syslog server in place. VMware also distributes VMware logging solutions in the form of vRealize Log Insight Cloud or on-premise.