Configure the Splunk Add-on for VMware to collect data from vCenter logs

vCenter logs contain information about access to the vCenter environment, audit information (who assigned permissions, added/edited/removed VMs), and health information about vCenter's processes.

For vCSA servers, vCSA's native syslog forwarding is used to pass this information to your Splunk platform. Nothing is installed onto the vCSA servers to collect this data. Windows-based vCenter environments require a Splunk platform forwarder and Splunk Add-on for vCenter Logs(Splunk_TA_vcenter).

Prepare to collect data

Set up a vCenter Server user account

Obtain VMware vCenter Server account credentials for each vCenter Server system.

These credentials allow the Splunk Add-on for VMware read-only API access to the appropriate metrics on each vCenter Server system in the environment. the Splunk App for VMware uses the credentials when the DCN polls vCenter Server systems for performance, hierarchy, inventory, task, and event data. These credentials are required for DCN configuration. You can use existing vCenter Server account credentials, or create a new account for Splunk App for VMware to access the vCenter Server data.

If you encounter issues setting the correct permissions for vCenter Server accounts, see "Permissions in vSphere."

You must have a user account to authenticate with vCenter. Your role determines access privileges. If you use ActiveDirectory for authentication on your Windows OS (vCenter) machines, see Create users in ActiveDirectory in this topic.

If you add a new vCenter Server user as administrator, the user automatically gets an Administrator role in vSphere.

Create a local user on your Windows OS (vCenter) machine

Log into the Windows OS with an administrator account.
In the Start menu, click Control Panel.
In the User Accounts screen, click Add or remove user accounts.
In the Manage Accounts window, click Create a new account.
Enter a name for the account (example: splunksvc).
In vSphere, select Standard user.
Click Create Account.
In the Manage Accounts screen, click on the new user.
In the Change an Account screen, click Create a password and assign the user a password. The new user account displays as a Standard user and the account shows that it is Password protected.
Verify that you now have a local Windows user compatible with the vSphere permissions system.

Create users in Active Directory

For machines that participate in an Active Directory (AD) domain, create a service account in the given domain using the appropriate control panel in Windows Server. Most VMware environments use a single Active Directory domain for authentication. However, if you use multiple AD domains, then create a service account in each domain that your VMware environment uses.

How you create a service account within Active Directory depends on your environment. Contact your AD administrator to learn how to do this for your environment.

Create roles on each vCenter server in your environment

Open the vSphere client and connect to the vCenter server.
Log in with administrative privileges.
Click Home in the path bar.
Under Administration click Roles.
Click Add Role.
In the Add new Role dialog box, enter a name for the role (for example, splunkreader).
Select the appropriate permissions for the role.

For information about collecting data via the VMware API, see Configure the Splunk Add-on for VMware to collect data from vCenter Server systems using the VMware API.

Configure DCNs to honor TLS protocols

You may need to set your DCN's to honor TLS protocols when making requests to the vCenter APIs.

On your DCN, Navigate to $SPLUNK_HOME/etc/system/local, and open web.conf with a text editor. If there is no web.conf create the file.
Add the below stanzas to your web.conf file.

[settings]
sslVersions = tls1.2
cipherSuite = AES256-SHA256

Validate and patch vCenter Server systems, add WSDL files

If you use vCenter Server 5.0 and 5.0.1, apply a patch to manage a known issue with the servers. See known issues in the release notes for details on acquiring and applying the patch.

If you use vSphere 5.0 or 5.0 update 1, be sure to add two missing WSDL files that the app needs to make API calls to vCenter. Access the VMware Knowledge Base for detailed installation instructions. The missing files are:

reflect-message.xsd
reflect-types.xsd

vCenter Log Collection (Windows vCenter and vCSA)

Collect Windows VMware vCenter Server log data

Use the Splunk Add-on for VMware vCenter to collect vCenter Server log data. Use a Splunk Universal Forwarder to forward the log data from your Windows vCenter Server to the indexer.

Install a Splunk forwarder.

Download the Universal Forwarder.
Install the Universal Forwarder. See Install a Universal Forwarder on Windows.

Configure forwarding. Configure the forwarder on your vCenter Server systems to send data to your indexers. Configure the forwarder in the outputs.conf file for each forwarder installed on a vCenter Server system. See Configure forwarding with outputs.conf.
Change your Splunk password.

The default password for the Splunk Enterprise admin user is changeme. Change the password using Splunk Web. See "Change the admin default password" in the Admin Manual.

Install the Splunk Add-on for vCenter Logs (Splunk_TA_vcenter).

Get the file Splunk_TA_vcenter-<version>-<build_number>.zip from the download package and install it on your vCenter Server systems.
Unzip the file, "Splunk_TA_vcenter-<version>-<build_number>.zip", into the apps directory under %SPLUNK_HOME%\etc\apps. When installing on a universal forwarder the path is C:\Program Files\SplunkUniversalForwarder\etc\apps otherwise it is C:\Program Files\Splunk\etc\apps.

On the system where you've installed the Splunk Enterprise forwarder, install the Splunk Add-on for vCenter Logs (Splunk_TA_vcenter).
Copy the inputs.conf file from $SPLUNK_HOME/etc/Splunk_TA_vCenter/default.
Paste $SPLUNK_HOME/etc/Splunk_TA_vCenter/default into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder.
Open the inputs.conf file.
Change the log path to the location of the vCenter Server Appliance logs data, C:\ProgramData\VMware\vCenterServer\logs. Edit the following stanzas in the inputs.conf file:

Windows vCenter server 6.x

[monitor://$ALLUSERSPROFILE\VMware\vCenterServer\logs\vws]
disabled = 0
index = vmware-vclog

[monitor://$ALLUSERSPROFILE\VMware\vCenterServer\logs\vmware-vpx]
blacklist = (.*(gz)$)|(\\drmdump\\.*)
disabled = 0
index = vmware-vclog

[monitor://$ALLUSERSPROFILE\VMware\vCenterServer\logs\perfcharts]
disabled = 0
index = vmware-vclog

(Optional) If you configured Splunk Enterprise as a heavy or light forwarder, and you want to monitor the license file and and tomcat configuration files, edit the following stanzas in the props.conf file.

Copy the $SPLUNK_HOME/etc/Splunk_TA_vCenter/default/props.conf file.
Paste $SPLUNK_HOME/etc/Splunk_TA_vCenter/default/props.conf into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder.
Open the local props.conf file.
Change the log path to that in which the vCenter Server Appliance logs data. Adjust the following stanzas:

Windows vCenter server

[source::(?-i)...\\VMware\\vCenterServer\\logs\\cim-diag.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\sms.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\stats.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vim-tomcat-shared.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-\d+.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-alert-\d+.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-profiler-\d+.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vws.log(?:.\d+)?]
[source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd.cfg]

Change the licenses path to the vCenter Server Appliance licenses path:

[source::(?-i)...\\VMware\\vCenterServer\\licenses]

Change the tomcat conf path to the vCenter Server Appliance tomcat conf path:

[source::(?-i)...\\VMware\\Infrastructure\\tomcat\\conf]

Change the path to the vCenter Server Appliance path:

[source::...\\Application Data\\VMware\\…]
[source::...\\VMware\\Infrastructure\\…]

Restart Splunk Enterprise. See "Start and stop Splunk" in the Admin Manual.
In %SPLUNK_HOME%\bin run the command splunk restart. Alternatively, select Start > Administrative Tools > Services > Splunkd restart in Windows services.

The Splunk Add-on for VMware collects log data from your Windows vCenter Server systems and forwards the data from vCenter Server to your Splunk platform indexers or combined indexer search heads.

Collect VMware vCenter Server Appliance (vCSA) log data

Use the Splunk Add-on for VMware to collect logs from the VMware vCenter Server Appliance. the Splunk Add-on for VMware stores VMware vCenter Server Appliance logs in /var/log/vmware.

Export vCenter logs to another system on which you have installed Splunk Enterprise.
Install a Splunk Enterprise forwarder on the same machine to forward the VMware vCenter Linux appliance logs. See "Forward VMware vCenter Linux appliance logs to Splunk Enterprise".

Export vCenter logs to an external system

Install a Splunk forwarder.

Download the Universal Forwarder.
Install the Universal Forwarder. See Install Universal Forwarder on *nix in the Splunk Universal Forwarder Manual.

Enable the VMware vCenter Server Appliance to store log files on NFS storage on a system on which you have installed Splunk Enterprise as a heavy forwarder or as a light forwarder. See NFS Storage on the VMware vCenter Server Appliance in the VMware vSphere documentation.
On the system on which you have installed the Splunk Enterprise forwarder, install the Splunk Add-on for vCenter Logs (Splunk_TA_vcenter).
Copy the inputs.conf file from $SPLUNK_HOME/etc/Splunk_TA_vCenter/default then paste it into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder and open file.

Change the log path to the location that the vCenter Server Appliance logs data (/var/log/vmware/). Edit the following stanzas in the inputs.conf file: Linux server appliance 6.x, 7.0.

[monitor:///var/log/vmware/vws]
disabled = 0
index = vmware-vclog

[monitor:///var/log/vmware/vpxd]
blacklist = (.*(gz)$)|(\\drmdump\\.*)
disabled = 0
index = vmware-vclog

[monitor:///var/log/vmware/perfcharts]
disabled = 0
index = vmware-vclog

Linux server appliance 6.x, 7.0 (not supported from 3.4.5)

[monitor:///var/log/vmware/vpx]
blacklist = (.*(gz)$)|(\\drmdump\\.*)
disabled = 0
index = vmware-vclog

(Optional) If you configured Splunk Enterprise as a heavy/light forwarder and you want to monitor the license file and tomcat configuration files, edit the following stanzas in the props.conf file:

Copy the $SPLUNK_HOME/etc/Splunk_TA_vCenter/default/props.conf file, then past into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder. Open the local props.conf file.

Change the log path to that in which the vCenter Server Appliance logs data. Adjust the following stanzas:

Linux server appliance 6.x

[source::(?-i).../var/log/vmware/perfcharts/stats.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpxd/vpxd-\d+.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpxd/vpxd-alert-\d+.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpxd/vpxd-profiler-\d+.log(?:.\d+)?]

Linux server appliance 5.x (not supported from 3.4.5)

[source::(?-i).../var/log/vmware/vpx/stats.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpx/vpxd-\d+.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpx/vpxd-alert-\d+.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpx/vpxd-profiler-\d+.log(?:.\d+)?]
[source::(?-i).../var/log/vmware/vpx/vws.log(?:.\d+)?]

Start Splunk Enterprise.

Forward VMware vCenter Linux appliance logs to Splunk Enterprise

To forward VMware vCenter Linux appliance logs to your Splunk Enterprise indexers or search head, install a Splunk Enterprise forwarder on the VMware vCenter Linux appliance. Access to vCSA shell access must be enabled.

Install a Splunk forwarder on the VMware vCenter Server Appliance.
Download the Splunk Add-on for vCenter Logs (Splunk_TA_vCenter) from Splunkbase and extract its contents to the $SPLUNK_HOME/etc/apps directory.
Copy the inputs.conf file from $SPLUNK_HOME/etc/Splunk_TA_vCenter/default then paste it into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder and open file.
(Optional) If you configured Splunk Enterprise as a heavy forwarder and you want to monitor the license file and and tomcat configuration files, copy the contents of the $SPLUNK_HOME/etc/Splunk_TA_vCenter/default/props.conf file and paste it into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder.
Start the Splunk Universal Forwarder.

Collect vCenter Server Appliance logs via syslog

Syslog type	Supported vCSA version	Log types
syslog-ng	5.5	`vpxd`, `vpxd-profiler`, `vpxd-alert`
rsyslog	6.x, 7.0	`vpxd`, `vpxd-profiler`, `vpxd-alert`

Syslog-ng on vCenter 5.5

vCenter 5.5 is not supported from VMware 3.4.5

Enable syslog forwarding using syslog-ng for vCSA 5.5 logs.

Open your vCenter deployment, and navigate to /etc/syslog-ng/.
In /etc/syslog-ng/, open the syslog-ng.conf file.
In the syslog-ng.conf file, replace <IP/HOSTNAME> with the IP address of the hostname of the machine where you want to receive the vCSA logs.

Example:

# vpxd source log
source vclog {
    file("/var/log/vmware/vpx/vpxd.log" follow-freq(60) log-prefix("vpxd ") flags(no-parse));
    file("/var/log/vmware/vpx/vpxd-alert.log" follow-freq(60) log-prefix("vpxd-alert ") flags(no-parse));
    file("/var/log/vmware/vpx/vpxd-profiler.log" follow-freq(60) log-prefix("vpxd-profiler ") flags(no-parse));    
    file("/var/log/vmware/vpx/vws.log" follow-freq(60) log-prefix("vws ") flags(no-parse));
    file("/var/log/vmware/vpx/stats.log" follow-freq(60) log-prefix("stats ") flags(no-parse));
    file("/var/log/vmware/vpx/cim-diag.log" follow-freq(60) log-prefix("cim-diag ") flags(no-parse));
    file("/var/log/vmware/vpx/sms.log" follow-freq(60) log-prefix("sms ") flags(no-parse));
    file("/var/log/vmware/vpx/cim-diag.log" follow-freq(60) log-prefix("cim-diag ") flags(no-parse));
    file("/var/log/vmware/vpx/vmware-vpxd.log" follow-freq(60) log-prefix("vmware-vpxd ") flags(no-parse));
};

# Remote Syslog Host
destination remote_syslog {
    tcp("<IP/HOSTNAME>" port(1517) template("${MSG} \n") template-escape(no));
};

# Log vCenter Server vpxd log remotely
log {
    	source(vclog);
    	destination(remote_syslog);	 
};

After changing the conf file, restart the syslog service for the changes to take effect. service syslog restart
Navigate to Splunk/etc/apps/Splunk_TA_vcenter/ and create a local folder.
In Splunk/etc/apps/Splunk_TA_vcenter/local, create an inputs.conf file.
Navigate to Splunk/etc/apps/Splunk_TA_vcenter/default/inputs.conf and copy the below stanza.

#[tcp://1517]
#connection_host = dns
#index = vmware-vclog
#sourcetype = vclog
#disabled = 0

Navigate to Splunk/etc/apps/Splunk_TA_vcenter/local/inputs.conf, and paste the copied stanza into the local version of inputs.conf.
Enable the copied stanza in local/inputs.conf by uncommenting it.

Note: Since TCP port 1514 is used for receiving ESXi logs, the 1517 port is used, by default, for vclogs. Other open ports can be used.

File properties	Description
`follow-freq`	Used to set the polling interval in seconds.
`log-prefix`	Used to set the prefix in each event data. Set log-prefix so your Splunk platform deployment can recognize sourcetype of different logs.
`flags`	Used to forward the log without any parsing.

For more information on configuration details, see the syslog-ng Open Source Edition Administrator Guide

Rsyslog on vCenter 6.x, 7.0

Enable syslog forwarding using rsyslog for vCSA 6.x or 7.0 logs.

Open your vCenter deployment, and navigate to /etc/.
In /etc/, open the rsyslog.conf file.
In the rsyslog.conf file, replace <IP/HOSTNAME> with the IP address of the hostname of the machine where you want to receive the vCSA logs.

Example:

$template vclogtemplate,"%syslogtag% %rawmsg%"

$ModLoad imfile
$InputFileName /var/log/vmware/vpxd/vpxd.log
$InputFileTag vpxd
$InputFileStateFile state-vpxd
$InputFileSeverity all
$InputRunFileMonitor

$ModLoad imfile
$InputFileName /var/log/vmware/vpxd/vpxd-profiler.log
$InputFileTag vpxd-profiler
$InputFileStateFile state-vpxd-profiler
$InputFileSeverity all
$InputRunFileMonitor

$ModLoad imfile
$InputFileName /var/log/vmware/vpxd/vpxd-alert.log
$InputFileTag vpxd-alert
$InputFileStateFile state-vpxd-alert
$InputFileSeverity all
$InputRunFileMonitor

$ModLoad imfile
$InputFileName /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log
$InputFileTag vws
$InputFileStateFile state-vws
$InputFileSeverity all
$InputRunFileMonitor

$ModLoad imfile
$InputFileName /var/log/vmware/perfcharts/stats.log
$InputFileTag stats
$InputFileStateFile state-stats
$InputFileSeverity all
$InputRunFileMonitor

 *.* @@<IP/HOSTNAME>:1517;vclogtemplate

After changing the conf file, restart the syslog service for the changes to take effect. service syslog restart
Navigate to Splunk/etc/apps/Splunk_TA_vcenter/ and create a local folder.
In Splunk/etc/apps/Splunk_TA_vcenter/local, create an inputs.conf file.
Navigate to Splunk/etc/apps/Splunk_TA_vcenter/default/inputs.conf and copy the below stanza.

#[tcp://1517]
#connection_host = dns
#index = vmware-vclog
#sourcetype = vclog
#disabled = 0

Navigate to Splunk/etc/apps/Splunk_TA_vcenter/local/inputs.conf, and paste the copied stanza into the local version of inputs.conf.
Enable the copied stanza in local/inputs.conf by uncommenting it.

Note: Since TCP port 1514 is used for receiving ESXi logs, the 1517 port is used, by default, for vclogs. Other open ports can be used.

File properties	Description
`$InputFileName`	Used to monitor specific files.
`$InputFileTag`	Used to set the prefix in each event data. Set `$InputFileTag` so your Splunk platform deployment can recognize sourcetype of different logs.
`$InputFileStateFile`	Used to keep track of which parts of the monitored file are already processed. Must be unique.
`$InputFileSeverity`	Used to set the type of log the user wants.
`$InputRunFileMonitor`	Used to activate the monitoring.

For more information on configuration details, see the text file input module page.

Splunk Add-on for VMware

Related Answers

Configure the Splunk Add-on for VMware to collect data from vCenter logs

Prepare to collect data

Set up a vCenter Server user account

Configure DCNs to honor TLS protocols

Validate and patch vCenter Server systems, add WSDL files

vCenter Log Collection (Windows vCenter and vCSA)

Collect Windows VMware vCenter Server log data

Collect VMware vCenter Server Appliance (vCSA) log data

Export vCenter logs to an external system

Forward VMware vCenter Linux appliance logs to Splunk Enterprise

Collect vCenter Server Appliance logs via syslog

Syslog-ng on vCenter 5.5

Rsyslog on vCenter 6.x, 7.0

Comments

Configure the Splunk Add-on for VMware to collect data from vCenter logs