Configure Splunk for ESXi logs

To configure ESXi log data collection, identify the machine to use as your data collection point and check that the ESX/i hosts are set up to forward data to that data collection point.

• For a first time install, use an intermediate forwarder as your data collection point and manually configure hosts to forward syslog data to the intermediate forwarder. A data collection node can be an intermediate forwarder.
• In a production installation, use the host profile to set up forwarding to your intermediate forwarders or syslog server.

Configure Splunk to receive syslog data

The Splunk App for VMware can receive ESXi log data via syslog from:

• A Splunk intermediate forwarder (This can be the data collection node OVA)
• A syslog server with a Splunk forwarder monitoring logs

Note that VMware vSphere 4.1 only supports syslog data collection using a UDP port. Data collection using a TCP port is not supported.

The Splunk App for VMware supports the following ports for syslog data collection:

• TCP port 1514
• UDP port 514 (Note: To use this port, Splunk requires a root privilege)

When you use the intermediate forwarder to collect ESXi logs, Splunk Enterprise is your default log repository.

Use your syslog server

Use the ESX/i Log Browser view in the Splunk App for VMware to check that syslog data is collected and indexed, and that the source type (vmw-syslog) of the data is correctly set.

To use your syslog server to collect and store ESXi logs:

1. Install a Splunk Universal Forwarder on your syslog server to forward the data from your syslog server to the Splunk instance receiving the log data. (This can be an indexer or an intermediate forwarder.)
   1. Download the the Universal Forwarder from www.splunk.com and click Free Download.
   2. Select the Universal Forwarder link on the page. Select the forwarder version you need on the Download Splunk Universal Forwarder page.
   3. See "Deployment overview" in the Forwarding Data Manual to install the universal forwarder.
2. Create an inputs.conf file in the system/local folder to monitor the esxi hosts log files on the syslog server and set the index and the source type before sending it to the intermediate forwarder. For each monitor stanza in the inputs.conf file specify the sourcetype for the data (vmw-syslog) and the index (vmware-esxilog) for the data. See "Configure your inputs" in the Getting Data in manual for more information. The entry in the monitor stanza of the inputs.conf file is:

   [monitor:///var/log/.../syslog.log]

   disabled = false

   index = vmware-esxilog

   sourcetype = vmw-syslog

3. Configure forwarding on your syslog server, in the outputs.conf file, to send data to your indexer or intermediate forwarder (the Splunk instance on which Splunk_TA_esxilogs is installed). For more information about setting up forwarding for your indexers, see Configure forwarders with outputs.conf in the Forwarding Data Manual.
4. On the machine that receives log data from your syslog server:
   1. Install Splunk_TA_esxilogs under $SPLUNK_HOME/etc/apps. This technology add-on is included in the Splunk App for VMware download. It collects syslog data from the ESXi hosts and performs the correct mappings to get the data into the dashboards in the app.
   2. Assign the host field (on the machine where Splunk_TA_esxilogs is installed). The Splunk App for VMware can not determine the originating host for the data when you use a syslog server as your data store and you forward that data to the Splunk indexer. Create an index time extraction that takes the actual host name from the event that passes through, so that the log files can be associated with the correct host. By default, the host name is that of the syslog server. This step is not required when you use an intermediate forwarder, as the Splunk App for VMware automatically assigns the host based on the original data source. To assign the host field, create a local version of props.conf and transforms.conf in the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory and add the regular expressions to extract the host field. In the following example, the regular expression extraction in props.conf calls the set_host stanza of transforms.conf where the regular expression extraction extracts the host. The source and sourcetype fields are extracted by the settings in the props.conf and transforms.conf files in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default. Do not override these fields in the local versions of these files.

      Add the following entry to props.conf:

      [vmw-syslog]

      ……

      TRANSFORMS-vmsysloghost = set_host

      Entry in transforms.conf

      [set_host]

      REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+)

      DEST_KEY = MetaData:Host

      FORMAT = host::$1

   3. If the sourcetype is not set correctly, check the regular expressions in the stanzas [set_syslog_sourcetype] and [set_syslog_sourcetype_4x] in Splunk_TA_esxilogs/default/transforms.conf. The following is an example of an entry in transforms.conf:

      [set_syslog_sourcetype]

      REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+[^ ]+\s+)?([A-Za-z\-]+)(?:[^:]*)

      DEST_KEY = MetaData:Sourcetype

      FORMAT = sourcetype::vmware:esxlog:$1

      Where:

      ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+[^ ]+\s+)? is used to extract the datetime field and host field

      ([A-Za-z\-]+) is used to extract the sourcetype

      (?:[^:]*) defines the limit. sourcetype is followed by : or [

   4. If the time is not extracted from the events, for example, Mar 26 19:00:20 esx1.abc.com Hostd:…, you can modify $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml or you can use splunk datetime.xml and change the entry for DATETIME_CONFIG to /etc/datetime.xml in /local/props.conf.

Note: Execute the following steps on the search head. These stanzas are only used during search time extraction.

1. If you use VMware vSphere ESX hypervisor 4.x, uncomment the following stanzas in transforms.conf to ensure that datetime extraction is the same in all regular expressions:

   [esx_hostd_fields_4x]

   [esx_vmkernel_fields_4x]

   [esx_generic_fields_4x]

2. If the correct fields do not display in the ESX/i Log Browser, then modify the regular expressions in the [esx_hostd_fields], [esx_vmkernel_fields], and [esx_generic_fields] stanzas. The following is an example from syslog_datetime.xml:

   [esx_hostd_fields]

   REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>: \[([^\s]+) (\w+) '([^']+)'(?: opID=([^\]]+))?\] ?(.*)

   FORMAT = Pri::$1 Application::$2 Offset::$3 Level::$4 Object::$5 opID::$6 Message::$7

   [esx_vmkernel_fields]

   REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR DATE TIME AND HOST FIELD EXTRACTION>:(vmkernel|vmkwarning):\s+(?:([\d\:\.]+)\s+)?(cpu\d+):(?:(\d+)\))?(?:\[([\:\w]+)\]\s+)?(.*)

   FORMAT = Pri::$1 Type::$2 HostUpTime::$3 Cpu::$4 WorldId::$5 SubComp::$6 Message::$7

   [esx_generic_fields]

   REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>:?\s*(.*)$

   FORMAT = Pri::$1 Application::$2 Message::$3

Use an intermediate forwarder

To set up forwarding to an intermediate forwarder:

1. Install Splunk Enterprise 6.0.x configured as a heavy forwarder or light forwarder on a machine identified as the intermediate forwarder. Note that if Splunk Enterprise is installed as heavy forwarder, then index time extraction happens on this intermediate forwarder. (This can be the data collection node OVA.) We recommend a ratio of 1 intermediate forwarder to 100 ESXi hosts.
2. Set up forwarding to the port on which the Splunk indexer(s) is configured to receive data. See "Set up forwarding" in the Distributed Deployment manual.
3. Install the Splunk_forwarder_for_vmware package. Get the file splunk_forwarder_for_vmware-<version>-<build_number>.zip from the download package and put it in $SPLUNK_HOME.
4. Unzip the file and check that Splunk_TA_esxilogs is in the SPLUNK_HOME/etc/apps/ directory.
5. Use UDP port 514. As the Splunk user on the intermediate forwarder, you must have root privileges to configure data inputs. If you do not have the required privileges, use TCP port 1514.
6. Enable the ports to receive syslog data. You can do this in Splunk Web using Settings or by manually modifying your inputs.conf file. In this example that uses Splunk Web, we use TCP port 1514.
   1. In Settings:
      1. Go to Data Inputs and add a new TCP port 1514.
      2. In the Setup screen enter the following information:

         TCP port: 1514

         Accept conditions from all hosts: yes

         Set sourcetype: Manual

         Source type: vmw-syslog

      3. Select More Settings and enter the following information:

         Set host: DNS

         Set the destination index for the source: vmware-esxilog. This is where the syslog data is sent. Do this after you have installed your app components.

   2. If you do not have access to Splunk Web, create an inputs.conf file in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ and add the following:

[tcp://1514]

disabled = 0

Configure ESXi Hosts to send data

Configure the ESXi hosts to forward log data to your syslog server or intermediate forwarder(s). Enable syslog data collection on the firewall on each host from which you want to collect Syslog data.

To manually configure ESXi hosts in the vSphere client:

1. Select a host on the Hierarchy selector.
2. Click the Configuration tab on the panel.
3. In the Software section, click Advanced Settings.
4. In the Advanced Settings dialog, scroll down and select Syslog.
5. Change the setting Syslog.global.loghost to the machine receiving the data. For example, enter tcp://yourmachine.yourdomain:1514. Note that vSphere version 4.1 only forwards to tcp. In this case, do not specify tcp://. ESXi hosts only forward to UDP port 514 or TCP port 1514. To forward to UDP port 514 check that your receiving machine is set up to do so.
6. Click OK.
7. In the Software section, click Security Profile.
8. Click Properties... in the Firewall section.
9. In the Firewall Properties Remote Access dialog, check the box for Syslog.
10. Click Firewall... and select Allow connections from any IP address or specify the connections allowed, then click OK.

Set up a Host profile

The VMware ESXi and vCenter Server documentation describes how to set up syslog from the Host Profile.

• See Set Up Syslog from the Host Profiles Interface in the VMware ESXi and vCenter Server 5 Documentation.
• See Set Up Syslog from the Host Profiles Interface in the vSphere Client 5.1

Configure all hosts remotely

The Splunk App for VMware can configure hosts remotely when you use an intermediate forwarder to collect syslog data. See Configure data collection.