Splunk® Supported Add-ons

Splunk Add-on for VMware ESXi Logs

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Install and configure the Splunk Add-on for VMware ESXi Logs

ESXi server logs allow you to troubleshoot events and host issues. Splunk Add-on for VMware ESXi logs accepts ESXi log data using syslogs from these sources.

  • A Splunk platform forwarder as the data collection point, which can be the Splunk OVA for VMware. When you use the forwarder to collect ESXi logs, Splunk platform is the default log repository.
  • A syslog server with a Splunk platform forwarder monitoring logs.

The VMware environment supports the following ports for syslog data collection.

  • TCP port 1514: Not supported on VMware vSphere 4.1.
  • UDP port 514: Requires Splunk Enterprise root privileges.

Configure the Splunk Add-on for VMware to receive ESXi syslog data

  • To configure ESXi log data collection, identify the machine to use as your data collection point. Verify that the ESXi hosts can forward data to that data collection point.
  • For the first installation, use an intermediate forwarder as your data collection point. Configure hosts to forward syslog data to the intermediate forwarder.

Step 1: Install a Splunk Universal Forwarder on your syslog server

  1. Download the universal forwarder.
  2. Install the Splunk universal forwarder. Go to the Install the universal forwarder documentation for installation steps.

Step 2: Create an inputs.conf file

  1. Create an inputs.conf file.
  2. Save the file to the system/local directory to monitor the ESXi hosts log files on the syslog server.
  3. For each monitor stanza in the inputs.conf file, specify these settings:
    • index = vmware-esxilog
    • sourcetype = vmw-syslog

    The entry in the monitor stanza of the inputs.conf file looks like this:

    [monitor:///var/log/.../syslog.log]
    disabled = false
    index = vmware-esxilog
    sourcetype = vmw-syslog
  4. Configure forwarding on your syslog server in outputs.conf to send data to your indexer or intermediate forwarder, which is the Splunk Enterprise instance on which the Splunk_TA_esxilogs package is installed. For more information about setting up forwarding for your indexers, go to Configure forwarders with outputs.conf.

Step 3: Install the Splunk_TA_esxilogs package

Download Splunk Add-on for VMware ESXi Logs from Splunkbase and the build will have Splunk_TA_esxilogs package in it. Install the Splunk_TA_esxilogs package on the machine that receives log data from your syslog server in the $SPLUNK_HOME/etc/apps directory.

Step 4: Configure the Splunk_TA_esxilogs package

  1. Assign the host field on the machine where you installed the Splunk_TA_esxilogs package. The Splunk Add-on for VMware ESXi logs can't determine the originating host for the data when you use a syslog server as your data store and you forward that data to the Splunk platform indexer.
  2. (Optional) Create an index-time extraction that takes the actual hostname from the event that passes through, so that the log files can be associated with the correct host. By default, the host name is that of the syslog server. This step isn't required when you use an intermediate forwarder, as the Splunk App for VMware automatically assigns the host based on the original data source.
  3. Create a local version of props.conf and transforms.conf files.
  4. Save the files to the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory and add the regular expressions to extract the host field. In this example regular expression extraction in props.conf calls the set_host stanza of transforms.conf where the regular expression extracts the host. The source and sourcetype fields are extracted by the settings in the props.conf and transforms.conf files in $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default. Don't override these fields in the local versions of these files.

    This is an example of the entry for props.conf:
    [vmw-syslog]
    ……
    TRANSFORMS-vmsysloghost = set_host

    This is an example entry for transforms.conf:

    [set_host]
    REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+)
    DEST_KEY = MetaData:Host
    FORMAT = host::$1
    
  5. If the sourcetype isn't correct, check the regular expressions in the [set_syslog_sourcetype] and [set_syslog_sourcetype_4x] stanzas in Splunk_TA_esxilogs/default/transforms.conf.

    This is an example of an entry in transforms.conf where ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+ is used to extract the datetime field and the host field and ([A-Za-z\-]+) is used to extract the sourcetype.
    [set_syslog_sourcetype]
    REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([A-Za-z\-]+)(?:[^:]*)[:\[]
    DEST_KEY = MetaData:Sourcetype
    FORMAT = sourcetype::vmware:esxlog:$1

Troubleshoot the Splunk_TA_esxilogs package

  • If the time isn't extracted from the events, for example, Mar 26 19:00:20 esx1.abc.com Hostd:…, you can modify $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml or you can use Splunk datetime.xml and change the entry for DATETIME_CONFIG to /etc/datetime.xml in /local/props.conf.
  • If you use VMware vSphere ESX 4.x, remove the comment tags from the following stanzas in transforms.conf on the search head. This ensures that datetime extraction is the same in all regular expressions. These stanzas are only used during search-time extraction.
    [esx_hostd_fields_4x]
    [esx_vmkernel_fields_4x]
    [esx_generic_fields_4x]
  • If the correct fields don't display in the ESXi Log Browser, modify the regular expressions in the [esx_vmkernel_fields] and [esx_generic_fields] stanzas.
    This is an example of the transforms.conf.
    [esx_vmkernel_fields]
    REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR DATE TIME AND HOST FIELD EXTRACTION>:(vmkernel|vmkwarning):\s+(?:([\d\:\.]+)\s+)?(cpu\d+):(?:(\d+)\))?(?:\[([\:\w]+)\]\s+)?(.*)
    FORMAT = Pri::$1 Type::$2 HostUpTime::$3 Cpu::$4 WorldId::$5 SubComp::$6 Message::$7
    [esx_generic_fields]
    REGEX = (?:^<(\d+)>)?<REPLACE WITH REGEX FOR SOURCETYPE EXTRACTION>:?\s*(.*)$
    FORMAT = Pri::$1 Application::$2 Message::$3

Use an intermediate forwarder to configure Splunk to receive syslog data

Step 1: Set up your forwarder

  1. Install Splunk Enterprise configured as a heavy forwarder or light forwarder on a machine identified as the intermediate forwarder. If Splunk Enterprise is installed as the heavy forwarder, index-time extraction happens on this intermediate forwarder. This forwarder can be the data collection node OVA. We recommend a ratio of one intermediate forwarder to 100 ESXi hosts.
  2. Set up forwarding to the port on which the Splunk indexers are configured to receive data. See Set up forwarding in Distributed Deployment.
  3. Download the Splunk Add-on for VMware ESXi Logs package and extract its contents to SPLUNK_HOME/etc/apps/ directory.

Step 2: Enable the ports to receive syslog data

Enable ports in Splunk Web or by modifying the inputs.conf file.

Use UDP port 514. As the Splunk user on the intermediate forwarder, you have to have root privileges to configure data inputs. If you do not have the required privileges, use TCP port 1514.

Enable ports in Splunk Web

  1. Select Settings > Data Inputs.
  2. Select TCP > New Local TCP.
  3. Enter 1514 in Port.
  4. Select Next.
  5. On the Input Settings enter this info:
    • Source type: New
    • Source Type: vmw-syslog
    • App Context: Splunk Add-on for ESXi logs
    • Method: DNS
    • Index: vmware-esxihost
  6. Select Review and Submit.

*This is the destination for the syslog data. Set the destination index for the source after you have installed the Splunk App for VMware components.

Enable ports in the inputs.conf file

If you don't have access to Splunk Web you can enable ports in the inputs.conf file.

  1. Create an inputs.conf file in the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory.
  2. Copy this stanza from $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/default/inputs.conf:
    #[tcp://1514]
    #index = vmware-esxilog
    #sourcetype = vmw-syslog
    #connection_host = dns
    #disabled = 0
  3. Paste the stanza in your new inputs.conf file in the $SPLUNK_HOME/etc/apps/Splunk_TA_esxilogs/local/ directory.
  4. Uncomment the stanza in the inputs.conf of local directory.

Do the same for UDP stanza if you are sending data to UDP port(514).

Configure ESXi hosts to send data

Configure the ESXi hosts to forward log data to your syslog server or intermediate forwarders. Enable syslog data collection on the firewall on each host from which you want to collect syslog data.

Configure ESXi hosts using the vSphere client

  1. Select a host on the Hierarchy selector.
  2. Go to the Configuration tab.
  3. In the Software section, select Advanced Settings.
  4. In Advanced Settings, scroll down and select Syslog.
  5. Change the setting Syslog.global.loghost to the machine receiving the data. For example, enter tcp://yourmachine.yourdomain:1514.
    • To forward the logs to multiple destinations, place a comma between the two machine specifications. For example, enter tcp://yourmachine1.yourdomain:1514, tcp://yourmachine2.yourdomain:1514.
    • vSphere version 4.1 only forwards to TCP. In this case, don't specify tcp://.
    • ESXi hosts forward to UDP port 514 or TCP port 1514 by default.
    • To forward to UDP port 514, make sure that the receiving machine is set up to do so.
    • To forward to a different port, create a new outbound firewall rule as another Security Profile on the sending host.
  6. Select OK.
  7. In Software, select Security Profile.
  8. In Firewall, select Properties.
  9. In Firewall Properties Remote Access, select Syslog.
  10. Select Firewall.
  11. Select Allow connections from any IP address or specify the connections.
  12. Select OK.

Set up a host profile

The VMware ESXi and vCenter Server documentation describes how to set up syslog from a host profile, see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-24FDBC4B-6D18-42DD-B9DD-97D6E0C5114F.html.

Install Splunk Add-on for VMware ESXi logs to a cloud environment

The Splunk add-on for VMware ESXi Logs is required on the search head tier in cloud environments for search-time extraction. Follow these steps

  1. Log in to your search head.
  2. On the Splunk Web home page, select the gear icon next to Apps.
  3. Select Browse More Apps.
  4. Search for the "Splunk Add-on for VMware ESXi logs" and select Install.
  5. Enter your Splunk.com login credentials.
  6. Read and accept the terms and conditions, and select Login and Download.
  7. Go to Apps > Manage Apps to review the installed app on the Apps page.

The vmware-esxihost index, which is part of the SA-VMWIndex-inframon or SA-VMWIndex, the package is required. If you are using Splunk Add-on for VMware Metrics, download the Splunk Add-on for VMware Metrics Indexes to obtain the SA-VMWIndex-inframon, package. If you are using the Splunk add-on for VMware, download the Splunk Add-on for VMware Indexes to obtain the SA-VMWIndex package.

Last modified on 19 January, 2024
PREVIOUS
Set up your system for the Splunk Add-on for VMware ESXi Logs
  NEXT
Troubleshoot the Splunk Add-on for VMware ESXi Logs

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters