Splunk® Supported Add-ons

Splunk Add-on for VMware

Download manual as PDF

Download topic as PDF

Configure the Splunk Add-on for VMware to collect log data from ESXi hosts

ESXi server logs let you troubleshoot events and host issues.

Splunk Add-on for VMware accepts ESXi log data using syslogs from the following sources.

  • A Splunk platform forwarder as the data collection point, which can be the Splunk OVA for VMware. When you use the forwarder to collect ESXi logs, Splunk platform is the default log repository.
  • A syslog server with a Splunk platform forwarder monitoring logs.

The VMware environment supports the following ports for syslog data collection.

  • TCP port 1514: Not supported on VMware vSphere 4.1.
  • UDP port 514: Requires Splunk Enterprise root privileges.

Configure the Splunk Add-on for VMware to receive ESXi syslog data

  • To configure ESXi log data collection, identify the machine to use as your data collection point. Verify that the ESXi hosts can forward data to that data collection point.
  • For the first installation, use an intermediate forwarder as your data collection point. Configure hosts to forward syslog data to the intermediate forwarder.

Step 1: Install a Splunk Universal Forwarder on your syslog server

  1. Download the Splunk Universal Forwarder from Download Splunk Universal Forwarder page. Select the forwarder version and the OS version that you need.
  2. See "Deployment overview" in Forwarding Data to install the universal forwarder.

Step 2: Create an inputs.conf file

Create an inputs.conf file in the system/local folder to monitor the ESXi hosts log files on the syslog server. Set the index and the source type before sending it to the intermediate forwarder.

  1. For each monitor stanza in the inputs.conf file, specify the following settings: The entry in the monitor stanza of the inputs.conf file is:
    [monitor:///var/log/.../syslog.log]
    disabled = false
    index = vmware-esxilog
    sourcetype = vmw-syslog
  2. Configure forwarding on your syslog server in outputs.conf to send data to your indexer or intermediate forwarder, which is the Splunk Enterprise instance on which Splunk_TA_esxilogs is installed. For more information about setting up forwarding for your indexers, see Configure forwarders with outputs.conf in Forwarding Data.

Step 3: Install and configure Splunk_TA_esxilogs

Install and configure Splunk_TA_esxilogs on the machine that receives log data from your syslog server.

Install Splunk_TA_esxilogs under $SPLUNK_HOME/etc/apps. This technology add-on is included in Splunk App for VMware. It collects syslog data from the ESXi hosts and maps the data into the dashboards in Splunk App for VMware.

Step 4: Configure Splunk_TA_esxilogs

  1. Assign the host field (on the machine where Splunk_TA_esxilogs is installed). The Splunk Add-on for VMware can not determine the originating host for the data when you use a syslog server as your data store and you forward that data to the Splunk platform indexer.
  2. (Optional) Create an index time extraction that takes the actual host name from the event that passes through, so that the log files can be associated with the correct host. By default, the host name is that of the syslog server. This step is not required when you use an intermediate forwarder, as the Splunk App for VMware automatically assigns the host based on the original data source.
  3. Assign the host field. Create a local version of props.conf and transforms.conf in the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory and add the regular expressions to extract the host field. In this example regular expression extraction in props.conf calls the set_host stanza of transforms.conf where the regular expression extraction extracts the host. The source and sourcetype fields are extracted by the settings in the props.conf and transforms.conf files in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default. Do not override these fields in the local versions of these files. Example of the entry for props.conf:
    [vmw-syslog]
    ……
    TRANSFORMS-vmsysloghost = set_host
    

    Here's the example for transforms.conf

    [set_host]
    REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+)
    DEST_KEY = MetaData:Host
    FORMAT = host::$1
    
  4. If the sourcetype is not correct, check the regular expressions in the stanzas [set_syslog_sourcetype] and [set_syslog_sourcetype_4x] in Splunk_TA_esxilogs/default/transforms.conf.
    The following is an example of an entry in transforms.conf:
    [set_syslog_sourcetype]
    REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([A-Za-z\-]+)(?:[^:]*)[:\[]
    DEST_KEY = MetaData:Sourcetype
    FORMAT = sourcetype::vmware:esxlog:$1
    

    Where:

    • ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+ is used to extract the datetime field and host field
    • ([A-Za-z\-]+) is used to extract the sourcetype
    • (?:[^:]*)[:\[] defines the limit. sourcetype is followed by : or [

Troubleshoot Splunk_TA_esxilogs

  • If the time is not extracted from the events, for example, Mar 26 19:00:20 esx1.abc.com Hostd:…, you can modify $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml or you can use splunk datetime.xml and change the entry for DATETIME_CONFIG to /etc/datetime.xml in /local/props.conf.
  • If you use VMware vSphere ESX 4.x, remove the comment tags from the following stanzas in transforms.conf on the search head. This ensures that datetime extraction is the same in all regular expressions. These stanzas are only used during search time extraction.
[esx_hostd_fields_4x]
[esx_vmkernel_fields_4x]
[esx_generic_fields_4x]
  • If the correct fields do not display in the ESXi Log Browser, modify the regular expressions in the [esx_hostd_fields], [esx_vmkernel_fields], and [esx_generic_fields] stanzas.

The following example is from syslog_datetime.xml.

[esx_hostd_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>: \[([^\s]+) (\w+) '([^']+)'(?: opID=([^\]]+))?\] ?(.*)
FORMAT = Pri::$1 Application::$2 Offset::$3 Level::$4 Object::$5 opID::$6 Message::$7
[esx_vmkernel_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR DATE TIME AND HOST FIELD EXTRACTION>:(vmkernel|vmkwarning):\s+(?:([\d\:\.]+)\s+)?(cpu\d+):(?:(\d+)\))?(?:\[([\:\w]+)\]\s+)?(.*)
FORMAT = Pri::$1 Type::$2 HostUpTime::$3 Cpu::$4 WorldId::$5 SubComp::$6 Message::$7
[esx_generic_fields]
REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>:?\s*(.*)$
FORMAT = Pri::$1 Application::$2 Message::$3

Use an intermediate forwarder to configure Splunk to receive syslog data

Step 1: Set up your forwarder

  1. Install Splunk Enterprise 6.0.x configured as a heavy forwarder or light forwarder on a machine identified as the intermediate forwarder. If Splunk Enterprise is installed as the heavy forwarder, index time extraction happens on this intermediate forwarder. This forwarder can be the data collection node OVA. We recommend a ratio of one intermediate forwarder to 100 ESXi hosts.
  2. Set up forwarding to the port on which the Splunk indexers are configured to receive data. See "Set up forwarding" in Distributed Deployment.
  3. Install the Splunk_forwarder_for_vmware package. Get the file splunk_forwarder_for_vmware-<version>-<build_number>.zip from the download package and add it to $SPLUNK_HOME.
  4. Unzip the file and make sure that Splunk_TA_esxilogs is in the SPLUNK_HOME/etc/apps/ directory. Use UDP port 514. As the Splunk user on the intermediate forwarder, you must have root privileges to configure data inputs. If you do not have the required privileges, use TCP port 1514.

Step 2: Enable the ports to receive syslog data

Enable ports in Splunk Web using Settings or by modifying the inputs.conf file. In this example using Splunk Web, the TCP port is 1514.

  1. Select Settings > Data Inputs.
  2. Add TCP port 1514.
  3. In the Setup screen enter the following information:
    • TCP port: 1514
    • Accept conditions from all hosts: yes
    • Set sourcetype: Manual
    • Source type: vmw-syslog
  4. Select More Settings and enter the following information:
    • Set host: DNS
    • Set the destination index for the source: vmware-esxilog. This setting is the destination of the syslog data. Set the destination index for the source after you have installed the Splunk App for VMware components.

If you do not have access to Splunk Web, create an inputs.conf file in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ and copy the following stanza from $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/inputs.conf:

#[tcp://1514]
#index = vmware-esxilog
#sourcetype = vmw-syslog
#connection_host = dns
#disabled = 0

Uncomment the above stanza in the inputs.conf of local folder.

Note: Do the same for UDP stanza if you are sending data to UDP port(514).

Configure ESXi hosts to send data

Configure the ESXi hosts to forward log data to your syslog server or intermediate forwarders. Enable syslog data collection on the firewall on each host from which you want to collect syslog data.

Configure ESXi hosts using the vSphere Client

  1. Select a host on the Hierarchy selector.
  2. Click the Configuration tab.
  3. In the Software section, click Advanced Settings.
  4. In Advanced Settings, scroll down and select Syslog.
  5. Change the setting Syslog.global.loghost to the machine receiving the data. For example, enter tcp://yourmachine.yourdomain:1514. To forward the logs to multiple destinations, place , between the two machine specifications. For example, enter tcp://yourmachine1.yourdomain:1514, tcp://yourmachine2.yourdomain:1514. vSphere version 4.1 forwards only to tcp. In this case, do not specify tcp://. ESXi hosts forward to UDP port 514 or TCP port 1514 by default. To forward to UDP port 514, make sure that the receiving machine is set up to do so. To forward to a different port, create a new outbound firewall rule as another Security Profile on the sending host.
  6. Click OK.
  7. In Software, click Security Profile.
  8. In Firewall, click Properties.
  9. In Firewall Properties Remote Access, select Syslog.
  10. Click Firewall.
  11. Select Allow connections from any IP address or specify the connections.
  12. Click OK.

Set up a host profile

The VMware ESXi and vCenter Server documentation describes how to set up syslog from a host profile.

Configure all hosts remotely

Splunk App for VMware can configure hosts remotely when you use an intermediate forwarder to collect syslog data. See Configure data collection.

Last modified on 01 November, 2019
PREVIOUS
Configure the Splunk Add-on for VMware to collect data from vCenter logs
  NEXT
Configure the Splunk Add-on for VMware to collect log data from vCenter Server systems using the VMware API

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters