Configure engine.conf

Once you have service account(s) created and the Splunk Forwarder Virtual Appliance for VMware (FA VM) is configured, you are now ready to create the configuration files that are responsible for collecting data from the target machines in your VMware environment.

The Splunk Forwarder Virtual Appliance for VMware (FA VM) sends data to your Splunk indexers after you configure the engine.conf file(s) and inputs.conf files.

How to create engine.conf and inputs.conf files

You can create configuration files automatically using the configuration builder tool (enginebuilder.py) or manually (following the examples provided in this manual). Note: We recommend that you use the configuration builder tool (enginebuilder.py), especially when evaluating the Solution or when configuring the Solution for a large environment. For instructions on how to do this, see Create Configuration file automatically.

The manual process is recommended for a Splunk administrator who wants to deep dive into how the engine configuration works so that they can fine tune data collection in their environment. Troubleshooting your environment is also best done manually. Having an understanding of the files and configurations set up for your environment enables you to deep dive and solve problems that may otherwise take longer to diagnose. To understand enging.conf in more detail see Create engine.conf and inputs.conf manually where we create a simple engine.conf file. We have more configuration file examples at the end of this manual. Each example increases in the level of complexity included in it, showing more powerful capabilities of the files.

The configuration builder tool is located in the FA VM in the directory: $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/bin. As the splunkadmin user, you already have the configuration builder in your path, so you can run it from any directory location. Note that the engine.conf files that the tool creates are output to the same directory from which the tool is invoked.

Stopping the engine before modifying the conf files

Note: Always stop Splunk before changing the engine.conf file. This is to avoid saving interim or mal-formed copies of engine.conf while editing it, which prevents the engine from generating errors based on an incorrect config. The engine periodically reads the engine.conf file and will see any changes made to the file. For more information about starting / stopping Splunk, see "Start and stop Splunk" in the Splunk Admin Manual.

Option 1: Automatically generate engine.conf and inputs.conf files

To automatically generate the configuration files, run the configuration builder tool.

Tool File Description

• enginebuilder.py: a tool that you run to create configuration files
• engine.template: a template file that you edit to include key information about your environment. This file is used by enginebuilder.py.
• logincreator.pl: a tool that you run to check service account credentials

How it works

1. Edit the engine.template file with key information about your environment.

   You can include multiple VCs in the file

2. Run enginebuilder.py, which uses the engine.template file.
3. enginebuilder.py generates all of your engine.conf and inputs.conf files
   • It creates engine.conf files by data type: engine<datatype>.conf
   • It creates inputs.conf to start engine instances using engine.conf files
4. enginebuilder.py optionally runs the service account tool (logincreator.pl) to verify service accounts
   • It checks permissions on vCenter server and all ESX/i hosts
   • It runs logincreator.pl to verify accounts (it does NOT create or modify them)
   • you can run logincreator.pl by itself to create or verify logins. For more information, see Create service accounts in this manual.

Gather information for the engine.template file

Before you run the tool, you must have the following information to include in the engine.template file:

Setting	perfInstanceDataPerfTypeBlacklist
vcuser	The username for the service account created to access vCenter.
vcpwd	The password for the vcuser account above.
vc	The IP or hostname of the vCenter Server.
hostuser	The username for the service account created to access ESX/i hosts.
hostpwd	The password for the hostuser account above.
host_csv	A comma separated list of values containing ESX/i host IPs or hostnames. You can use“*” to generate files that cover ALL of the ESX/i hosts managed by the given VC.
perfInstanceData	This option is set to OFF by default. When turned ON it provides fine-grained control over the amount and kinds of performance data that you want the engine to collect

Important: The configuration builder tool assumes that all ESX/i hosts in the host_csv field use the same service account username and password (hostuser, hostpwd). If your ESX/i hosts do not use the same service account credentials, you may need to run the tool multiple times or generate the FA VM configuration files manually.

Here is an example engine.template file that generates configurations using only the specified ESX/i hosts:

vcuser=splunkuservc
vcpwd=splunkuser123
vc=vc1.company.com
hostuser=splunksvc
hostpwd=splunkuser123
host_csv=esx1.company.com,esx2.company.com,esx3.company.com
perfInstanceData=OFF

Here is an example engine.template file that generates configs using all of the ESX/i hosts found in VC:

vcuser=splunkuservc
vcpwd=splunkuser123
vc=vc1.company.com
hostuser=splunksvc
hostpwd=splunkuser123
host_csv=*
perfInstanceData=OFF

Run the configuration builder

Once you have the information for the engine.template file, you can run the tool:

1. In the FA VM, go to the location where you want to generate the files: cd $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local
2. Make a copy of the engine.template file. (Don't forget the "." at the end of the command.)

cp ../default/engine.template .

1. Using a text editor (such as “vi”) modify engine.template so that it now includes the correct values for your environment.
   1. Should you need to, you can create a template file that covers multiple VCs by copying all of the entries in engine.template and pasting them (together), below the current group of values. Continue to do this untll you have covered all of the VCs you want to add to the configuration.
2. Run enginebuilder.py with the appropriate options:

enginebuilder.py

1. • Note: run enginebuilder.py using the "-c” option if you also want to check the service account credentials.
2. Please wait – the tool can take some time to run as it pulls information from the VC. When done, the configuration builder will generate many engine.conf files and an inputs.conf file.
3. Your FA VM is now ready to run! You can start splunk:

1. splunk start

You now have an FA VM that is configured for your environment and ready to work.! When Splunk starts, the engine instance is started by the simple inputs.conf file. The engine looks for the engine.conf files in the "Splunk_TA_vmware/local" directory and will start operating properly as soon as it finds it. With these steps completed, you are done configuring the FA VM and can proceed to the "Validate your setup" topic in this manual.

Option 2: Manually creating engine.conf and inputs.conf files

This example uses the most basic case to configure the engine to collect data in a small scale environment. We define a small scale environment to be small VC, one that is managing 10 or fewer ESX/i hosts in total, and a small number of ESX/i hosts (fewer than 10).

To configure engine.conf and inputs.conf on larger scales to meet the needs of your environment, see "Your environment" in this manual.

You will create an engine.conf file to collect your data and create an inputs.conf file to run an instance of the engine, the main data collection module inside the Splunk Forwarder Virtual Appliance for VMware (FA VM). Engine instances are run by Splunk based on the stanzas found in the inputs.conf file.

Create a single engine.conf file

To create a simple engine.conf file:

1. Log into the FA VM as the splunkadmin user.
2. Stop Splunk. On boot-up, the FA automatically starts Splunk (Splunk is already in your path)

splunk stop

3. Go into the Splunk Technology Add-on for VMware (TA-vmware) "local" directory:

cd $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local

4. In a text editor (such as vi), create the engine.conf file. Note that if you use the default engine instance, you must name the file engine.conf

Create a default stanza

Use a default stanza to assign certain settings once. Set the FA attribute to the value you used for the FA VM's OS hostname when you configured the FA. In this example it is splunkfa1. If you did not set the FA VM's OS hostname, do so before creating the engine.conf file. See Configure default properties for the FA VM in this manual to do this. Note: In a small environment you do not have to reset expiration timers.

[default]
fa = splunkfa1

Configure vCenter Server data inputs

In a simple engine.conf file, the first stanza is typically used to get data from vCenter Server (VC). In this example, we get many different types of data from one VC using a single stanza. We collect inventory, hierarchy, tasks, events, and some VC-only performance data. In a larger environment, you may need multiple stanzas for a single VC (or multiple engine.conf files).

1. Create a stanza for VC #1 in your environment
2. Use a single stanza to get all 5 types of data from VC
3. It is important to assign the host setting for VC machines to match the "VC instance name"
   • The VC instance name is the name of the root node in the vCenter's "Hosts and Clusters" view as seen in the vSphere Client.
   • This example assumes that the VC instance name for VC1 is “VMWARE-VC1"
   • Note: The VC instance name may or may not look like the VC's OS hostname, computer name, DNS alias, etc. Those other values are not relevant here - make sure to get your VC instance name as shown in the vSphere Client.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, TaskDiscovery, EventDiscovery
perfManagedEntityWhitelist = ClusterComputeResource|ResourcePool
interval = 1

Configure ESX/i host data inputs

Once the inputs for the VC are defined, you can add stanzas to collect data from the ESX/i hosts you want to monitor. The monitored hosts should be managed by the VC specified in the previous stanza. In this example we collect the data we want (performance, task, event, and log data) from two ESX/i hosts.

This is a continuation of the example above and assumes that these settings will be added after the VC stanza in the same engine.conf file. In a larger environment, you may need multiple stanzas for a single ESX/i host (or multiple engine.conf files).

For more information about the details of the engine.conf file (the fields, the values and settings), see "engine.conf settings" in this manual.

1. Create a stanza for each ESX/i host managed by VC #1 that you also want to monitor
2. Use a single stanza to get all types of data from each ESX/i host
3. As this is a managed host, there is no need to assign the host setting
4. Create more stanzas, up to 10 ESX/i hosts managed by VC #1

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery, TaskDiscovery, EventDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery, TaskDiscovery, EventDiscovery
interval = 1

Note: There are no inherent limits to the number of hosts that a given FA VM can monitor, but we recommend adding at most 10 ESX/i hosts in this simple example. For more comprehensive examples and a better understanding of when a given FA VM will hit its data gathering limits, please read the topics beginning with "Data collection in one enginer.conf file" in this manual.

Create a single inputs.conf file

Now that you have your engine.conf file configured, you can create your inputs.conf file. You must create a "scripted input" that calls the engine and takes the absolute path of the engine.conf file as an argument. While multiple engines can run concurrently, the example here only runs one instance (the "default engine instance").

Stop Splunk and create a simple inputs.conf file

This example is focused on a small environment (< 10 ESX/i hosts managed by a single VC), so you can take advantage of the default engine instance by turning it on with a simple inputs.conf file. In a small environment you will only need to run a single engine instance. In this case, you create a single “engine.conf” file in the "local" directory, along with a very simple inputs.conf, and everything will start working. When the FA VM boots up, it automatically runs a Splunk heavy forwarder, but the default engine instance defined in $SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/default is initially disabled.

To turn on the default engine instance:

1. If you have not already done so, log into the FA VM as the splunkadmin user.
2. Stop Splunk if it is running (Splunk is in your path).

splunk stop

3. Go into the Splunk Technology Add-on for VMware (TA-vmware) "local" directory:

cd $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local

4. In a text editor (such as vi), create the inputs.conf file.
5. Place the following lines in the file to:
1. Set the host attribute for all FA VM-specific data that is sent to the indexer(s)
   • This ensures that the FA’s logs will be assigned the right host field.
   • The value should be the same one you used for the FA VM's OS hostname (during FA VM configuration steps)
   • It should also be the same value used for the “fa” setting in the engine.conf [default] stanza.
   • This example assumes that the FA VM's OS hostname was set to "splunkfa1"
2. Turn on the default engine instance (stanza found in the TA-vmware's defaults/inputs.conf file)

[default]
host = splunkfa1

[script://./bin/Engine.pm]
disabled = false

Note: You do not have to disable the default scripted inputs (found in default/inputs.conf) as they are already disabled by default.

Start Splunk to run the engine

When you have created and configured engine.conf and inputs.conf for your environment, start Splunk:

splunk start

When Splunk starts, the engine instance is started by the simple inputs.conf file. The engine looks for the file "engine.conf" in the "local" directory and will start operating properly as soon as it finds it. With these steps completed, you are done configuring the FA VM and can proceed to the "Validate your setup" topic in this manual.