Save to My DOJO
Everyone loves logs right? Right?? I can’t be the only one.
In all seriousness though, logs are one of those things that you have to stay on top of, so for this post, we’re going to take a deep dive into the world of VMware logging. Yes, I know you are all super excited. Let’s run down the 25 most important VMware logging files and the various tools that you can use to keep track of your VMware environment. You likely won’t need all of these vSphere log files every single day, but when you do need them, you’ll be glad you have a general idea of where to go. To that end, I’d recommend bookmarking this page for future reference.
We’ll break this post into 2 sections. First, we’ll cover the vSphere log files on the host and then we’ll move on to the vCenter logs.
As a side note, it is worth pointing out that you can also you PowerCLI to query the VMware logging resources with the Get-Log cmdlet.
Part of the VMware logging you can benefit from comes from ESXi recording host activity in vSphere log files, using a Syslog facility. Typically one of the easiest and fastest ways to check these log files out is using tools like a cat on the command line. You can also view various logs through the DCUI as well. All VMware hosts run a service for logging system information.
This service, vmsyslogd, logs messages from the VMkernel and other system components for auditing and diagnostic purposes. By default, the logs are directed to a local scratch location or ramdisk. The scratch space is created automatically during ESXi installation in the form of a 4 GB Fat16 local scratch partition. If storage space is unavailable, the host will store data on a ramdisk, which is not persistent across reboots.
That being the case, many admins choose to send these logs to a persistent datastore as vSphere datastore log files or remote logging server for retention. One of the most important things to know is that it’s important to realize where you’re installing ESXi. If you opt to install it to a USB stick or sd card, if the host crashes, that can make for a bad day! This means that you’ll want to pay extra attention and have the logs written to a central logging location. I’ll highlight some of the most popular logs below and indicate what they do. I highly recommend having a Syslog server of some sort in your environment.
|2.||VMkernel warnings||/var/log/vmkwarning.log||Records activities related to virtual machines|
|3.||VMkernel summary||/var/log/vmksummary.log||Used to determine uptime and availability statistics for ESXi (comma separated)|
|4.||ESXi host agent log||/var/log/hostd.log||Contains information about the agent that manages and configures the ESXi host and its virtual machines|
|5.||vCenter agent log||/var/log/vpxa.log||Contains information about the agent that communicates with vCenter Server (if the host is managed by vCenter Server)|
|6.||Shell log||/var/log/shell.log||ESXi Shell as well as shell events (for example, when the shell was enabled)|
|7.||Authentication||/var/log/auth.log||Contains all events related to authentication for the local system|
|8.||System messages||/var/log/syslog.log||Contains all general log messages and can be used for troubleshooting. This information was formerly located in the messages log file|
|9.||Virtual machines||The same directory as the affected virtual machine’s configuration files, named vmware.log and vmware*.log. For example, /vmfs/volumes/datastore/virtual machine/vwmare.log||Contains virtual machine power events, system failure information, tools status and activity, time sync, virtual hardware changes, vMotion migrations, machine clones, and so on|
Another important vSphere log file that I like to point out is the boot.gz file. It’s located in /var/log/boot.gz. This one is important to troubleshoot slow boots and can indicate why the host is hanging or freezing. I’ve found myself using it from time to time and it’s been helpful.
You can change the location of the Syslog location under each host at the location below. To specify a remote logging host, enter a value in the format of protocol://hostname:port. Example: tcp://hostname:514
VMware logging in the VMware vCenter Server Appliance is located in the /var/log/vmware/ folder. You can get to it either via the console interface or via ssh. You can also export a bundle of logs via the vSphere Web Client.
|vCenter Server Appliance Log Location||Purpose|
|10.||vpxd/vpxd.log||The main vCenter Serverlog|
|11.||vpxd/vpxd-profiler.log||Profile metrics for operations performed in vCenter Server|
|12.||vpxd/vpxd-alert.log||Non-fatal information logged about the vpxd process|
|13.||perfcharts/stats.log||VMware Performance Charts|
|14.||eam/eam.log||VMware ESX Agent Manager|
|15.||invsvc||VMware Inventory Service|
|16.||netdumper||VMware vSphere ESXi Dump Collector|
|17.||vapi||VMware vAPI Endpoint|
|18.||vmdird||VMware Directory Service daemon|
|19.||syslog||vSphere Syslog Collector|
|20.||vmware-sps/sps.log||VMware vSphere Profile-Driven Storage Service|
|21.||vpostgres||vFabric Postgres database service|
|22.||vsphere-client||VMware vSphere Web Client|
|23.||vws||VMware System and Hardware Health Manager|
|24.||workflow||VMware vCenter Workflow Manager|
|25.||SSO||VMware Single Sign-On|
Some important ones to keep in mind:
Use this to troubleshoot issues related to vCenter operations. Everything from DB connectivity problems to vCenter crashes can be found here. This log will have a LOT of information in it and is a good place to start on many issues.
vmware logging to troubleshoot why the inventory service will not start.
This is a good log to use as a “one-stop-shop” for SSO authentication issues. Authentication requests/failures, as well as problems with an identity source, will post here.
A great place when troubleshooting errors within the Web Client. If you receive errors from simply clicking on objects, you begin chasing them down here!
While VMware logging certainly helps troubleshooting issues when they appear or perform Root Cause Analysis (RCA), it is only possible if they are actually available. In case of a disaster or just a bad server event, you may lose your VMware logging, especially if VMware system logging is not configured on a host or with default values.
It is strongly recommended to set up a Syslog server in your environment to gather vSphere log files as well as others. VMware proposes vRealize Log Insight, a Syslog solution that embeds plenty of dashboards, rules, and filters tailored to vSphere environments.
“vRealize Log Insight offers a tailored solution to consolidate third-party software and VMware logging”
Formerly known as VMware Log Intelligence, VMware vRealize Log Insight Cloud is a cloud service that offers a managed solution to get visibility across various public and private cloud through log forwarding. You will find the features of any respectable Syslog server such as log aggregation, analytics, dashboards, custom alerting…
You can also install content packs to process data collected from third-party sources such as AD, Sharepoint, DellEMC, and other software vendors.
To properly protect your VMware environment, use Altaro VM Backup to securely backup and replicate your virtual machines. We work hard perpetually to give our customers confidence in their VMware backup strategy.
To keep up to date with the latest VMware best practices, become a member of the VMware DOJO now (it’s free).
To wrap up, even though VMware logging is highly resilient, keep in mind that you need to consider a remote Syslog collector for your ESXi hosts. It’s one of the most important things to do as you don’t want to lose your VMware logging in case of a serious outage. You don’t have to use the VMware Syslog Collector if you already have one in place, but consider using that for a smaller shop with no existing Syslog server in place. VMware also distributes VMware logging solutions in the form of vRealize Log Insight Cloud or on-premise.
Not a DOJO Member yet?
Join thousands of other IT pros and receive a weekly roundup email with the latest content & updates!