Splunk® Supported Add-ons

Splunk Add-on for VMware

Download manual as PDF

Download topic as PDF

Configure the Splunk Add-on for VMware to collect data

Configure the Splunk Add-on for VMWare to collect Data Collection Node and Virtual Center data. Identify the data types that you want to collect, such as performance, inventory, or hierarchy data, from the following list.

  • vCenter logs (Intermediate Forwarder/Syslog Forwarder)
  • ESXi logs (Intermediate Forwarder/Syslog Forwarder)
  • Performance Metrics, Inventory, Tasks (vCenter API collected by Data Collection Node)

Configure collection of Performance, configuration, and event data

Set up a vCenter Server user account

Obtain VMware vCenter Server account credentials for each vCenter Server system.

These credentials allow the Splunk Add-on for VMware read-only API access to the appropriate metrics on each vCenter Server system in the environment. the Splunk App for VMware uses the credentials when the DCN polls vCenter Server systems for performance, hierarchy, inventory, task, and event data. These credentials are required for DCN configuration. You can use existing vCenter Server account credentials, or create a new account for Splunk App for VMware to access the vCenter Server data.

If you encounter issues setting the correct permissions for vCenter Server accounts, see "Permissions in vSphere."

You must have a user account to authenticate with vCenter. Your role determines access privileges. If you use ActiveDirectory for authentication on your Windows OS (vCenter) machines, see Create users in ActiveDirectory in this topic.

If you add a new vCenter Server user as administrator, the user automatically gets an Administrator role in vSphere.

Create a local user on your Windows OS (vCenter) machine

  1. Log into the Windows OS with an administrator account.
  2. In the Start menu, click Control Panel.
  3. In the User Accounts screen, click Add or remove user accounts.
  4. In the Manage Accounts window, click Create a new account.
  5. Enter a name for the account (example: splunksvc).
  6. In vSphere, select Standard user.
  7. Click Create Account.
  8. In the Manage Accounts screen, click on the new user.
  9. In the Change an Account screen, click Create a password and assign the user a password. The new user account displays as a Standard user and the account shows that it is Password protected.
  10. Verify that you now have a local Windows user compatible with the vSphere permissions system.

Create users in Active Directory

For machines that participate in an Active Directory (AD) domain, create a service account in the given domain using the appropriate control panel in Windows Server. Most VMware environments use a single Active Directory domain for authentication. However, if you use multiple AD domains, then create a service account in each domain that your VMware environment uses.

How you create a service account within Active Directory depends on your environment. Contact your AD administrator to learn how to do this for your environment.

Create roles on each vCenter server in your environment

  1. Open the vSphere client and connect to the vCenter server.
  2. Log in with administrative privileges.
  3. Click Home in the path bar.
  4. Under Administration click Roles.
  5. Click Add Role.
  6. In the Add new Role dialog box, enter a name for the role (for example, splunkreader).
  7. Select the appropriate permissions for the role.

Next steps for API data colleciton is to set up connections using the Distributed Search Scheduler. Please see the following topic for detailed steps:

Configure Data Collection from the Scheduler UI

Configure DCN's to honor TLS protocols

You may need to set your DCN's to honor TLS protocols when making requests to the vCenter APIs.

  1. On your DCN, Navigate to $SPLUNK_HOME/etc/system/local, and open web.conf with a text editor. If there is no web.conf create the file.
  2. Add the below stanzas to your web.conf file.
     
    sslVersions = tls1.2
    cipherSuite = AES256-SHA256
    

Validate and patch vCenter Server systems, add WSDL files

If you use vCenter Server 5.0 and 5.0.1, apply a patch to manage a known issue with the servers. See known issues in the release notes for details on acquiring and applying the patch.

If you use vSphere 5.0 or 5.0 update 1, be sure to add two missing WSDL files that the app needs to make API calls to vCenter. Access the VMware Knowledge Base for detailed installation instructions. The missing files are:

  • reflect-message.xsd
  • reflect-types.xsd

Validate time synchronization

Verify time synchronization throughout your environment to improve visibility into application and operating system health. Check the time synchronization for the following components in your environment.

  • Hosts
  • Splunk Enterprise search head and indexers

Consider using NTP or VMware host/guest time synchronization.

vCenter Log Collection (Windows vCenter and vCSA)

Collect Windows VMware vCenter Server log data

Use the Splunk Add-on for VMware vCenter to collect vCenter Server log data. Use a Splunk Universal Forwarder to forward the log data from your Windows vCenter Server to the indexer.

  1. Install a Splunk forwarder.
  2. Configure forwarding. Configure the forwarder on your vCenter Server systems to send data to your indexers. Configure the forwarder in the outputs.conf file for each forwarder installed on a vCenter Server system. See Configure forwarding with outputs.conf.
  3. Change your Splunk password.
  4. Install Splunk_TA_vcenter.
    • Get the file Splunk_TA_vcenter-<version>-<build_number>.zip from the download package and install it on your vCenter Server systems.
    • Unzip the file, "Splunk_TA_vcenter-<version>-<build_number>.zip", into the apps directory under %SPLUNK_HOME%\etc\apps. When installing on a universal forwarder the path is C:\Program Files\SplunkUniversalForwarder\etc\apps otherwise it is C:\Program Files\Splunk\etc\apps.
  5. Restart Splunk Enterprise. See "Start and stop Splunk" in the Admin Manual.
  6. In %SPLUNK_HOME%\bin run the command splunk restart. Alternatively, select Start > Administrative Tools > Services > Splunkd restart in Windows services.

The Splunk Add-on for VMware collects log data from your Windows vCenter Server systems and forwards the data from vCenter Server to your Splunk platform indexers or combined indexer search heads.

Collect VMware vCenter Server Appliance (vCSA) log data

Use the Splunk Add-on for VMware to collect logs from the VMware vCenter Server Appliance. the Splunk Add-on for VMware stores VMware vCenter Server Appliance logs in /var/log/vmware.

Export vCenter logs to an external system

  1. Enable the VMware vCenter Server Appliance to store log files on NFS storage on a system on which you have installed Splunk Enterprise as a heavy forwarder or as a light forwarder. See NFS Storage on the VMware vCenter Server Appliance in the VMware vSphere documentation.
  2. On the system on which you have installed the Splunk Enterprise forwarder, install Splunk_TA_vCenter.
  3. Copy the inputs.conf file from $SPLUNK_HOME/etc/Splunk_TA_vCenter/default then paste it into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder and open file.
  4. Change the log path to the location that the vCenter Server Appliance logs data (/var/log/vmware/). Edit the following stanzas in the inputs.conf file:
    [monitor://$ALLUSERSPROFILE\Application Data\VMware\vCenterServer\logs]
    disabled = 0
    [monitor://$PROGRAMFILES\VMware\Infrastructure\tomcat\logs]
    
  5. (Optional) If you configured Splunk Enterprise as a heavy forwarder and you want to monitor the license file and and tomcat configuration files, edit the following stanzas in the props.conf file:
    a. Copy the $SPLUNK_HOME/etc/Splunk_TA_vCenter/default/props.conf file, then past into the $SPLUNK_HOME/etc/Splunk_TA_vCenter/local folder. Open the local props.conf file.
    b. Change the log path to that in which the vCenter Server Appliance logs data. Adjust the following stanzas:
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\cim-diag.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\sms.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\stats.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vim-tomcat-shared.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-alert-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-profiler-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vws.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd.cfg]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-profiler-\d+.log(?:.\d+)?]
    
    c. Change the licenses path to the vCenter Server Appliance licenses path:
    [source::(?-i)...\\VMware\\vCenterServer\\licenses]
    d. Change the tomcat conf path to the vCenter Server Appliance tomcat conf path:
    [source::(?-i)...\\VMware\\Infrastructure\\tomcat\\conf] </pre>
    e. Change the path to the vCenter Server Appliance path:
    [source::...\\Application Data\\VMware\\…]
    [source::...\\VMware\\Infrastructure\\…]
  6. Optional If you configured the Splunk Enterprise instance as a light forwarder and you want to monitor the license file and and tomcat configuration files, adjust the following stanzas in the props.conf on the Splunk Indexers that receive the log files.
    a. Change the log path to that in which the vCenter Server Appliance logs data. Adjust in the following stanzas in the props.conf file:
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\cim-diag.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\sms.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\stats.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vim-tomcat-shared.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-alert-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-profiler-\d+.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vws.log(?:.\d+)?]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd.cfg]
    [source::(?-i)...\\VMware\\vCenterServer\\logs\\vpxd-profiler-\d+.log(?:.\d+)?]
    
    b. Change the licenses path to the vCenter Server Appliance licenses path:
     [source::(?-i)...\\VMware\\vCenterServer\\licenses] 
    c. Change the tomcat conf path to the vCenter Server Appliance tomcat conf path:
    [source::(?-i)...\\VMware\\Infrastructure\\tomcat\\conf] 
    d. Change the path to the vCenter Server Appliance path:
    [source::...\\Application Data\\VMware\\…]
    [source::...\\VMware\\Infrastructure\\…]
  7. Start Splunk Enterprise.

Forward VMware vCenter Linux appliance logs to Splunk Enterprise

To forward VMware vCenter Linux appliance logs to your Splunk Enterprise indexers or search head, install a Splunk Enterprise forwarder on the VMware vCenter Linux appliance. Access to vCSA shell access must be enabled.

  1. Install a Splunk forwarder on the VMware vCenter Server Appliance.
  2. Install Splunk_TA_vCenter on the Splunk platform forwarder.
    1. Get the Splunk_TA_vcenter-<version>-<build_number>.zip file from the download package and place it on vCenter.
    2. Unzip the Splunk App for VMware package.
    cd /opt/splunkforwarder
    Splunk_TA_vcenter-<version>-<build_number>.zip"
    3. Verify that you successfully extracted the Splunk_TA_vcenter/… in the $SPLUNK_HOME/etc/apps directory.
  3. Follow steps 3 - 4 of Export vCenter logs to an external system.
  4. Start Splunk Universal Forwarder.

Collect vCenter Server Appliance logs via syslog

Syslog type Supported vCSA version Log types
syslog-ng 5.5, 6.x vpxd, vpxd-profiler, vpxd-alert
rsyslog 6.x vpxd, vpxd-profiler, vpxd-alert

Syslog-ng on vCenter 5.5

Enable syslog forwarding using syslog-ng for vCSA 5.5 logs.

  1. Open your vCenter deployment, and navigate to /etc/syslog-ng/.
  2. In /etc/syslog-ng/, open the syslog-ng.conf file.
  3. In the syslog-ng.conf file, replace <IP/HOSTNAME> with the IP address of the hostname of the machine where you want to receive the vCSA logs. Example:
    # vpxd source log
    source vclog {
        file("/var/log/vmware/vpx/vpxd.log" follow-freq(60) log-prefix("vpxd ") flags(no-parse));
        file("/var/log/vmware/vpx/vpxd-alert.log" follow-freq(60) log-prefix("vpxd-alert ") flags(no-parse));
        file("/var/log/vmware/vpx/vpxd-profiler.log" follow-freq(60) log-prefix("vpxd-profiler ") flags(no-parse));    
        file("/var/log/vmware/vpx/vws.log" follow-freq(60) log-prefix("vws ") flags(no-parse));
        file("/var/log/vmware/vpx/stats.log" follow-freq(60) log-prefix("stats ") flags(no-parse));
        file("/var/log/vmware/vpx/cim-diag.log" follow-freq(60) log-prefix("cim-diag ") flags(no-parse));
        file("/var/log/vmware/vpx/sms.log" follow-freq(60) log-prefix("sms ") flags(no-parse));
        file("/var/log/vmware/vpx/cim-diag.log" follow-freq(60) log-prefix("cim-diag ") flags(no-parse));
        file("/var/log/vmware/vpx/vmware-vpxd.log" follow-freq(60) log-prefix("vmware-vpxd ") flags(no-parse));
    };
    
    # Remote Syslog Host
    destination remote_syslog {
        tcp("<IP/HOSTNAME>" port(1517) template("${MSG} \n") template-escape(no));
    };
    
    # Log vCenter Server vpxd log remotely
    log {
        	source(vclog);
        	destination(remote_syslog);	 
    };
    
  4. After changing the conf file, restart the syslog service for the changes to take effect. service syslog restart
  5. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/ and create a local folder.
  6. In Splunk/etc/apps/Splunk_TA_vcenter/local, create an inputs.conf file.
  7. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/default/inputs.conf and copy the below stanza.
    [tcp://1517]
    connection_host = dns
    index = vmware-vclog
    sourcetype = vclog
    disabled = 1
    
  8. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/local/inputs.conf, and paste the copied stanza into the local version of inputs.conf.
  9. Enable the copied stanza in local/inputs.conf by setting disabled = 0. Note: Since TCP port 1514 is used for receiving ESXi logs, the 1517 port is used, by default, for vclogs. Other open ports can be used.
File properties Description
follow-freq Used to set the polling interval in seconds.
log-prefix Used to set the prefix in each event data. Set log-prefix so your Splunk platform deployment can recognize sourcetype of different logs.
flags Used to forward the log without any parsing.

For more information on configuration details, see the syslog-ng Open Source Edition Administrator Guide

Rsyslog on vCenter 6.x

Enable syslog forwarding using rsyslog for vCSA 6.x logs.

  1. Open your vCenter deployment, and navigate to /etc/.
  2. In /etc/, open the rsyslog.conf file.
  3. In the rsyslog.conf file, replace <IP/HOSTNAME> with the IP address of the hostname of the machine where you want to receive the vCSA logs. Example:
    $template vclogtemplate,"%syslogtag% %rawmsg%"
    
    $ModLoad imfile
    $InputFileName /var/log/vmware/vpxd/vpxd.log
    $InputFileTag vpxd
    $InputFileStateFile state-vpxd
    $InputFileSeverity all
    $InputRunFileMonitor
    
    $ModLoad imfile
    $InputFileName /var/log/vmware/vpxd/vpxd-profiler.log
    $InputFileTag vpxd-profiler
    $InputFileStateFile state-vpxd-profiler
    $InputFileSeverity all
    $InputRunFileMonitor
    
    $ModLoad imfile
    $InputFileName /var/log/vmware/vpxd/vpxd-alert.log
    $InputFileTag vpxd-alert
    $InputFileStateFile state-vpxd-alert
    $InputFileSeverity all
    $InputRunFileMonitor
    
    $ModLoad imfile
    $InputFileName /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log
    $InputFileTag vws
    $InputFileStateFile state-vws
    $InputFileSeverity all
    $InputRunFileMonitor
    
    $ModLoad imfile
    $InputFileName /var/log/vmware/perfcharts/stats.log
    $InputFileTag stats
    $InputFileStateFile state-stats
    $InputFileSeverity all
    $InputRunFileMonitor
    
     *.* @@<IP/HOSTNAME>:1517;vclogtemplate
    
  4. After changing the conf file, restart the syslog service for the changes to take effect. service syslog restart
  5. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/ and create a local folder.
  6. In Splunk/etc/apps/Splunk_TA_vcenter/local, create an inputs.conf file.
  7. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/default/inputs.conf and copy the below stanza.
    [tcp://1517]
    connection_host = dns
    index = vmware-vclog
    sourcetype = vclog
    disabled = 1
    
  8. Navigate to Splunk/etc/apps/Splunk_TA_vcenter/local/inputs.conf, and paste the copied stanza into the local version of inputs.conf.
  9. Enable the copied stanza in local/inputs.conf by setting disabled = 0. Note: Since TCP port 1514 is used for receiving ESXi logs, the 1517 port is used, by default, for vclogs. Other open ports can be used.
File properties Description
$InputFileName Used to monitor specific files.
$InputFileTag Used to set the prefix in each event data. Set $InputFileTag so your Splunk platform deployment can recognize sourcetype of different logs.
$InputFileStateFile Used to keep track of which parts of the monitored file are already processed. Must be unique.
$InputFileSeverity Used to set the type of log the user wants.
$InputRunFileMonitor Used to activate the monitoring.

For more information on configuration details, see the text file input module page.

Configure your deployment to collect log data from ESXi hosts

ESXi server logs let you troubleshoot events and host issues.

Splunk Add-on for VMware accepts ESXi log data using syslogs from the following sources.

  • A Splunk platform forwarder as the data collection point, which can be the Splunk OVA for VMware. When you use the forwarder to collect ESXi logs, Splunk platform is the default log repository.
  • A syslog server with a Splunk platform forwarder monitoring logs.

The VMware environment supports the following ports for syslog data collection.

  • TCP port 1514: Not supported on VMware vSphere 4.1.
  • UDP port 514: Requires Splunk Enterprise root privileges.

Configure the Splunk Add-on for VMware to receive ESXi syslog data

  • To configure ESXi log data collection, identify the machine to use as your data collection point. Verify that the ESXi hosts can forward data to that data collection point.
  • For the first installation, use an intermediate forwarder as your data collection point. Configure hosts to forward syslog data to the intermediate forwarder.

Step 1: Install a Splunk Universal Forwarder on your syslog server

  1. Download the Splunk Universal Forwarder from Download Splunk Universal Forwarder page. Select the forwarder version and the OS version that you need.
  2. See "Deployment overview" in Forwarding Data to install the universal forwarder.

Step 2: Create an inputs.conf file

Create an inputs.conf file in the system/local folder to monitor the ESXi hosts log files on the syslog server. Set the index and the source type before sending it to the intermediate forwarder.

  1. For each monitor stanza in the inputs.conf file, specify the following settings:
    • sourcetype: vmw-syslog
    • index: vmware-esxilog. See "Configure your inputs" in Getting Data In for more information.
    The entry in the monitor stanza of the inputs.conf file is:
    [monitor:///var/log/.../syslog.log]
    disabled = false
    index = vmware-esxilog
    sourcetype = vmw-syslog
  2. Configure forwarding on your syslog server in outputs.conf to send data to your indexer or intermediate forwarder, which is the Splunk Enterprise instance on which Splunk_TA_esxilogs is installed. For more information about setting up forwarding for your indexers, see Configure forwarders with outputs.conf in Forwarding Data.

Step 3: Install and configure Splunk_TA_esxilogs

Install and configure Splunk_TA_esxilogs on the machine that receives log data from your syslog server.

Install Splunk_TA_esxilogs under $SPLUNK_HOME/etc/apps. This technology add-on is included in Splunk App for VMware. It collects syslog data from the ESXi hosts and maps the data into the dashboards in Splunk App for VMware.

Step 4: Configure Splunk_TA_esxilogs

  1. Assign the host field (on the machine where Splunk_TA_esxilogs is installed). The Splunk Add-on for VMware can not determine the originating host for the data when you use a syslog server as your data store and you forward that data to the Splunk platform indexer.
  2. (Optional) Create an index time extraction that takes the actual host name from the event that passes through, so that the log files can be associated with the correct host. By default, the host name is that of the syslog server. This step is not required when you use an intermediate forwarder, as the Splunk App for VMware automatically assigns the host based on the original data source.
  3. Assign the host field. Create a local version of props.conf and transforms.conf in the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory and add the regular expressions to extract the host field. In this example regular expression extraction in props.conf calls the set_host stanza of transforms.conf where the regular expression extraction extracts the host. The source and sourcetype fields are extracted by the settings in the props.conf and transforms.conf files in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default. Do not override these fields in the local versions of these files. Example of the entry for props.conf:
    [vmw-syslog]
    ……
    TRANSFORMS-vmsysloghost = set_host
    

    Here's the example for transforms.conf

    [set_host]
    REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+)
    DEST_KEY = MetaData:Host
    FORMAT = host::$1
    
  4. If the sourcetype is not correct, check the regular expressions in the stanzas [set_syslog_sourcetype] and [set_syslog_sourcetype_4x] in Splunk_TA_esxilogs/default/transforms.conf. The following is an example of an entry in transforms.conf:
    [set_syslog_sourcetype]
    REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+[^ ]+\s+)?([A-Za-z\-]+)(?:[^:]*)
    DEST_KEY = MetaData:Sourcetype
    FORMAT = sourcetype::vmware:esxlog:$1
    

    Where:

    • ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+[^ ]+\s+)? is used to extract the datetime field and host field
    • ([A-Za-z\-]+) is used to extract the sourcetype
    • (?:[^:]*) defines the limit. sourcetype is followed by : or [

Troubleshoot Splunk_TA_esxilogs

  • If the time is not extracted from the events, for example, Mar 26 19:00:20 esx1.abc.com Hostd:…, you can modify $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml or you can use splunk datetime.xml and change the entry for DATETIME_CONFIG to /etc/datetime.xml in /local/props.conf.
  • If you use VMware vSphere ESX 4.x, remove the comment tags from the following stanzas in transforms.conf on the search head. This ensures that datetime extraction is the same in all regular expressions. These stanzas are only used during search time extraction.
[esx_hostd_fields_4x]
[esx_vmkernel_fields_4x]
[esx_generic_fields_4x]
  • If the correct fields do not display in the ESXi Log Browser, modify the regular expressions in the [esx_hostd_fields], [esx_vmkernel_fields], and [esx_generic_fields] stanzas.

The following example is from syslog_datetime.xml.

[esx_hostd_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>: \[([^\s]+) (\w+) '([^']+)'(?: opID=([^\]]+))?\] ?(.*)
FORMAT = Pri::$1 Application::$2 Offset::$3 Level::$4 Object::$5 opID::$6 Message::$7
[esx_vmkernel_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR DATE TIME AND HOST FIELD EXTRACTION>:(vmkernel|vmkwarning):\s+(?:([\d\:\.]+)\s+)?(cpu\d+):(?:(\d+)\))?(?:\[([\:\w]+)\]\s+)?(.*)
FORMAT = Pri::$1 Type::$2 HostUpTime::$3 Cpu::$4 WorldId::$5 SubComp::$6 Message::$7
[esx_generic_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>:?\s*(.*)$
FORMAT = Pri::$1 Application::$2 Message::$3

Use an intermediate forwarder to configure Splunk to receive syslog data

Step 1: Set up your forwarder

  1. Install Splunk Enterprise 6.0.x configured as a heavy forwarder or light forwarder on a machine identified as the intermediate forwarder. If Splunk Enterprise is installed as the heavy forwarder, index time extraction happens on this intermediate forwarder. This forwarder can be the data collection node OVA. We recommend a ratio of one intermediate forwarder to 100 ESXi hosts.
  2. Set up forwarding to the port on which the Splunk indexers are configured to receive data. See "Set up forwarding" in Distributed Deployment.
  3. Install the Splunk_forwarder_for_vmware package. Get the file splunk_forwarder_for_vmware-<version>-<build_number>.zip from the download package and add it to $SPLUNK_HOME.
  4. Unzip the file and make sure that Splunk_TA_esxilogs is in the SPLUNK_HOME/etc/apps/ directory. Use UDP port 514. As the Splunk user on the intermediate forwarder, you must have root privileges to configure data inputs. If you do not have the required privileges, use TCP port 1514.

Step 2: Enable the ports to receive syslog data

Enable ports in Splunk Web using Settings or by modifying the inputs.conf file. In this example using Splunk Web, the TCP port is 1514.

  1. Select Settings > Data Inputs.
  2. Add TCP port 1514.
  3. In the Setup screen enter the following information:
    • TCP port: 1514
    • Accept conditions from all hosts: yes
    • Set sourcetype: Manual
    • Source type: vmw-syslog
  4. Select More Settings and enter the following information:
    • Set host: DNS
    • Set the destination index for the source: vmware-esxilog. This setting is the destination of the syslog data. Set the destination index for the source after you have installed the Splunk App for VMware components.

If you do not have access to Splunk Web, create an inputs.conf file in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ and add the following:

[tcp://1514]
disabled = 0

Configure ESXi hosts to send data

Configure the ESXi hosts to forward log data to your syslog server or intermediate forwarders. Enable syslog data collection on the firewall on each host from which you want to collect syslog data.

Configure ESXi hosts using the vSphere Client

  1. Select a host on the Hierarchy selector.
  2. Click the Configuration tab.
  3. In the Software section, click Advanced Settings.
  4. In Advanced Settings, scroll down and select Syslog.
  5. Change the setting Syslog.global.loghost to the machine receiving the data. For example, enter tcp://yourmachine.yourdomain:1514. vSphere version 4.1 forwards only to tcp. In this case, do not specify tcp://. ESXi hosts forward to UDP port 514 or TCP port 1514 by default. To forward to UDP port 514, make sure that the receiving machine is set up to do so. To forward to a different port, create a new outbound firewall rule as another Security Profile on the sending host.
  6. Click OK.
  7. In Software, click Security Profile.
  8. In Firewall, click Properties.
  9. In Firewall Properties Remote Access, select Syslog.
  10. Click Firewall.
  11. Select Allow connections from any IP address or specify the connections.
  12. Click OK.

Set up a host profile

The VMware ESXi and vCenter Server documentation describes how to set up syslog from a host profile.

Configure all hosts remotely

Splunk App for VMware can configure hosts remotely when you use an intermediate forwarder to collect syslog data. See Configure data collection.

Collect data from vCenter Server systems using the VMware API

The Splunk Add-on for VMware uses the VMware API to collect data about your virtual environment. The Splunk Add-on for VMware communicates with vCenter Server using network ports and Splunk management ports.

Sender Receiver Port number Description
Scheduler (on the search head) vCenter server 443 The scheduler uses port 443 to connect to the vCenter Server to verify that the vCenter Server credentials are valid. It also uses this port to discover the number of managed ESXi hosts in the environment.
Splunk Add-on for VMware Data Collection Node 8089 The Splunk App for VMware connects to the Data Collection Node (DCN) on the default Splunk management port, TCP 8089.
Scheduler Data Collection Node 8008 When the DCN and Splunk App for VMware have established a connection, the scheduler, which typically runs on the search head, allocates data collection jobs to the DCN on the TCP port 8008. TCP port 8008 is the gateway port. In your environment, if another service uses port 8008, you can configure a different port for communication between the data collection node and the gateway. Data collection nodes do not have to communicate on the same port.
[default]
gateway_port = 8008

To change the ports for each data collection node individually, set the port in each stanza.

Data Collection Node (DCN) vCenter Server 443 The DCN communicates with vCenter Server API on port 443 to execute the data collection tasks allocated to it.
Data Collection Node Splunk indexer 9997 The Data Collection Node uses port 9997 to forward data it has retrieved from the vCenter Server using the API.

After the Splunk Add-on for VMware establishes a connection with a vCenter Server, the DCN uses port 443 to obtain the credentials for vCenter Server. The DCN uses port 443 to determine the kind of data to collect, such as performance, inventory, or hierarchy data. Splunk App for VMware sends information to the data collection nodes using port 8008 about the information they need to collect from a specific vCenter Server system. The DCN retrieves the data from vCenter Server and forwards the data to the Splunk indexer on port 9997.

Control certificate validation for your data collection nodes

Control certificate validation your data collection nodes with the ta_vmware_config_ssl.conf file. Use it to enable and disable certificate validation for your DCN. By default, certificate validation is disabled.

  1. On your scheduler, navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/default and copy the ta_vmware_config_ssl.conf file.
  2. Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_vmware and create a local folder.
  3. Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local and paste the ta_vmware_config_ssl.conf file.
  4. Open the $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local/ta_vmware_config_ssl.conf and set validate_ssl_certificate option to true.
    [general]
    validate_ssl_certificate = true
    
  5. Save your changes.
  6. Restart your Splunk platform instance.

For more information, see the About securing inter-Splunk communication section of the Securing Splunk Enterprise documentation.

Configure VMX Logs to Syslog

Configure your Splunk platform infrastructure to collect vmware.log files from your VM infrastructure. This configuration provides your Splunk platform deployment with a source of data that lets you audit, troubleshoot and rebuild your VMX configuration files.

  1. Navigate to your virtual machine vmx file.
  2. Add vmx.log.destination = "syslog-and-disk" to your virtual machine vmx file.
  3. Name your vm log entry. (Example:vmx.log.syslogID = vmx[splunkdata])
  4. Check the log entry in /var/log/syslog of your ESXi host to verify the syslog is being forwarded.
PREVIOUS
Install the Splunk Add-on for VMware
  NEXT
Use the Collection Configuration page to add configurations

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters