A simple engine.conf

Before diving into a discussion on how to manage a large scale environment with the Splunk for VMware, we will use a simple example to highlight the basic configuration changes you must make to manage your resources using the Solution. The amount of visibility into your VMware environment directly depends on the number of target machines managed by the Solutions and and the level of detail you collect from the data gathered from the resources. See "what can I get" for information about the data you can collect in a VMware environment.

In this example we walk through what it takes to configure the engine to collect data in a small scale environment. We define small scale to be a small VC ( one that is managing 10 or less ESX/i hosts in total) and a small number of ESX/i hosts (also less than 10).

Note: There are no inherent limits to the number of hosts that a given FA VM can monitor, but for this example we recommend adding at most 10 ESX/i hosts.

We use a single engine.conf file that will be used by a single instance of the engine. The engine.conf file is read by the engine and covers three key areas; what target machine(s) to query, what type(s) of data to query, and the frequency at which to execute data gathering actions. Individual stanzas in engine.conf correspond to VC machines or ESX/i hosts to query for data. Within a stanza, actions correspond to the type(s) of data to query, while intervals and other settings correspond to data gathering frequency.

Walkthrough a simple FA engine.conf file

Note: Always stop Splunk before changing the configuration of the engine.conf file. The engine periodically reads the engine.conf file and operates based on any new changes to the config.

Log into the FA VM as the splunkadmin user and go to the FA Add-on local directory:

cd $SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/local

This is the "local" directory of the FA Add-on that runs the engine. As per Splunk conventions, your engine.conf and any other custom config file(s) should go into this directory. If the directory does not exist yet, you can easily create it as the splunkadmin user with the command:

mkdir $SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/local

In a text editor create the engine.conf file. If you are using the default engine instance, you must name it engine.conf

Create a default stanza

Use a default stanza to assign certain settings once. Set the FA name to the value you used for the FA VM's OS hostname (during FA VM configuration steps). We also assume that The FA VM "OS hostname" is set to splunkfa1. This was set during the FA VM installation and setup process. If you did not set the your FA VM's OS hostname, return to that section and complete that step before proceeding here. Note: In a small environment, there is no need to reset expiration timers

[default]
fa = splunkfa1
perfTypeWhitelist = cpu|mem|disk|net|clSvcs
hostPerfCounterWhitelist=AvgAct_KB|AvgConsum_KB|AvgGrtd_KB|AvgOvrhd_KB|AvgQueLat_ms|AvgRd|AvgRvcd_KBps|AvgSwpIn_KB|AvgSwpOut_KB|AvgTotLat_ms|AvgUsg_KBps|AvgUsg_pct|AvgVmctl_KB|AvgWr|AvgXmit_KBps|SumBusResets|SumCmdsAbort|SumRd|SumRdy_ms|SumWr|AvgSwpUsd_KB
vmPerfCounterWhitelist=AvgAct_KB|AvgConsum_KB|AvgGrtd_KB|AvgOvrhd_KB|AvgQueLat_ms|AvgRd|AvgRvcd_KBps|AvgSwpIn_KB|AvgSwpOut_KB|AvgSwpd_KB|AvgTotLat_ms|AvgUsg_KBps|AvgUsg_pct|AvgVmctl_KB|AvgWr|AvgXmit_KBps|SumBusResets|SumCmdsAbort|SumRd|SumRdy_ms|SumWr|AvgUsg_mhz

Configure vCenter Server data inputs

In a simple engine.conf file, the first stanza is typically used to get data from vCenter Server (VC). In this example, we get many different types of data from one VC using a single stanza. We collect inventory, hierarchy, tasks, events, and some VC-only performance data. In a real environment, you may need multiple stanzas for a single VC (or multiple engine.conf files).

1. Create a stanza for VC #1 in your environment
2. Use a single stanza to get all 5 types of data from VC
3. It is important to assign the host setting for VC machines to match the "VC instance name"
4. This example assumes that the VC instance name for VC1 is “VMWARE-VC1

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, TaskDiscovery, EventDiscovery
perfManagedEntityWhitelist = ClusterComputeResource
interval = 1

Configure ESX/i host data inputs

Once the inputs for the VC are defined, you can add stanzas to collect data from the ESX/i hosts you want to monitor. The monitored hosts should be managed by the VC specified in previous stanza. In this example we collect the data we want (performance and log data) from two ESX/i hosts.

This is a continuation of the example above and assumes that these settings will be added after the VC stanza in the same engine.conf file. In a real environment, you may need multiple stanzas for a single ESX/i host (or multiple engine.conf files).

For more information about the details of the engine.conf file (the fields, the values and settings), see "engine.conf settings" in this manual.

1. Create a stanza for each ESX/i host managed by VC #1 that you also want to monitor
2. Use a single stanza to get both types of data from each ESX/i host
3. As this is a managed host, there is no need to assign the host setting
4. Create more stanzas, up to 10 ESX/i hosts managed by VC #1

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

Add more stanzas to the file to represent each ESX/i host that exists in VC1.

Create a stanza for VC #2 in your environment

• This example assumes that the VC instance name for VC2 is “VMWARE-VC2”

[vc2]
url = https://vc2.company.com/sdk/webService
host = VMWARE-VC2
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, TaskDiscovery, EventDiscovery
perfManagedEntityWhitelist = ClusterComputeResource
interval = 1

Create a stanza for each ESX/i host managed by VC #2 that you also want to monitor. We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza

[host1inVC2]
url = https://host1_in_vc2.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

[host2inVC2]
url = https://host2_in_vc2.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

Continue to create more stanzas to represent each ESX/i host in VC2 that is monitored by Splunk for VMware.

When you have included all of the VC instances and all of the managed ESX/i hosts that you want to monitor you can now create stanzas to cover unmanaged ESX/i hosts in your environment.

Create stanzas to cover unmanaged ESX/i hosts in your environment

In this example the ESX/i host's fully qualified hostname is host4unmanaged.company.com. Explicitly assign the host setting for an unmanaged ESX/i host and give the host setting the same value as the ESX/i host's "fully qualified hostname". Unmanaged ESX/i hosts never need a blank VC stanza.

[host4unmanaged]
url = https://host4unmanaged.company.com/sdk/webService
host = host4unmanaged.company.com
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, LogDiscovery
interval = 1

In this example the ESX/i host's fully qualified hostname is host5unmanaged.company.com.

[host5unmanaged]
url = https://host5unmanaged.company.com/sdk/webService
host = host5unmanaged.company.com
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, LogDiscovery
interval = 1

Create more stanzas, as necessary, for each unmanaged ESX/i host that you want to monitor using Splunk for VMware.

Note: There are no inherent limits to the number of hosts that a given FA VM can monitor, but we recommend adding at most 10 ESX/i hosts in this simple example. For more comprehensive examples and a better understanding of when a given FA VM will hit its data gathering limits, see the topics in this manual beginning with "Data collection in one engine.conf file".

Wlakthrough a simple inputs.conf file

Once your engine.conf file is ready, then create an inputs.conf file. This file is used to start up an instance of the engine (the main data collection module inside the FA Add-on) and run it. Engine instances are run by Splunk based on the stanzas found in the inputs.conf file. Specifically, you must create a "scripted input" that calls the engine and takes the absolute path of the engine.conf file as an argument. While multiple engines can run concurrently, the example here only runs 1 instance (the "default engine instance").

In this example, we use a simple inputs.conf file. This file is only appropriate for monitoring a small vCenter (one that is managing less than 10 ESX/i hosts total) and a small number of ESX/i hosts (also less than 10). To create an inputs.conf file that scales up to run multiple engine instances for your particular environment, see "Scaling your deployment" in this manual. You can use the default engine instance by turning it on using a simple inputs.conf file. In a small environment you only need to run a single instance of the engine.

In this case you only need to create a single engine.conf file in the local directory, and a simple inputs.conf. When the FA VM boots up, it automatically runs a Splunk heavy forwarder, but the default engine instance defined in $SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/default is initially disabled.

To turn on the default engine instance:

1. If you have not already done so, log into the FA VM as the splunkadmin user.
2. Stop Splunk if it is running (Splunk is already be in your path).

splunk stop

3. Go to the FA Add-on local directory:

cd $SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/local

4. Using a text editor, create the inputs.conf file.
5. Set the host setting for all FA-specific data that is sent to the indexer(s). This ensures that the FA’s logs will be assigned the correct host field when sent to the indexer(s). The value should be the same one you used for the FA VM's OS hostname (during FA VM configuration steps). It should also be the same value used for the “fa” setting in the engine.conf [default] stanza.This example assumes that the FA VM's OS hostname was set to "splunkfa1"

[default]
host = splunkfa1

6. Turn on the default engine instance (stanza found in the FA Add-on's defaults/inputs.conf directory)

[script://./bin/Engine.pm]
disabled = false

Note: Default scripted inputs (found in default/inputs.conf) are disabled by default.

Start Splunk to run the engine

When you have created engine.conf and inputs.conf</ code> for your environment, start Splunk:

splunk start

When Splunk starts, the engine instance is started by the simple inputs.conf file. The engine looks for the file engine.conf in the local directory. When it finds it, Splunk starts collecting data from your environment and maps it to the Splunk App for VMware.