Before diving into a discussion on how to manage a large scale environment with the Splunk for VMware, we will use a simple example to highlight the basic configuration changes you must make to manage your resources using the Solution. The amount of visibility into your VMware environment directly depends on the number of target machines managed by the Solutions and and the level of detail you collect from the data gathered from the resources. See "what can I get" for information about the data you can collect in a VMware environment.
In this example we walk through what it takes to configure the engine to collect data in a small scale environment. We define small scale to be a small VC ( one that is managing 10 or less ESX/i hosts in total) and a small number of ESX/i hosts (also less than 10).
Note: There are no inherent limits to the number of hosts that a given FA VM can monitor, but for this example we recommend adding at most 10 ESX/i hosts.
We use a single
engine.conf file that will be used by a single instance of the engine. The
engine.conf file is read by the engine and covers three key areas; what target machine(s) to query, what type(s) of data to query, and the frequency at which to execute data gathering actions. Individual stanzas in
engine.conf correspond to VC machines or ESX/i hosts to query for data. Within a stanza, actions correspond to the type(s) of data to query, while intervals and other settings correspond to data gathering frequency.
Walkthrough a simple FA
Note: Always stop Splunk before changing the configuration of the
engine.conf file. The engine periodically reads the
engine.conf file and operates based on any new changes to the config.
Log into the FA VM as the splunkadmin user and go to the FA Add-on local directory:
This is the "local" directory of the FA Add-on that runs the engine. As per Splunk conventions, your
engine.conf and any other custom config file(s) should go into this directory. If the directory does not exist yet, you can easily create it as the splunkadmin user with the command:
engine.conffile. If you are using the default engine instance, you must name it
Create a default stanza
Use a default stanza to assign certain settings once. Set the FA name to the value you used for the FA VM's OS hostname (during FA VM configuration steps). We also assume that The FA VM "OS hostname" is set to splunkfa1. This was set during the FA VM installation and setup process. If you did not set the your FA VM's OS hostname, return to that section and complete that step before proceeding here. Note: In a small environment, there is no need to reset expiration timers
[default] fa = splunkfa1 perfTypeWhitelist = cpu|mem|disk|net|clSvcs hostPerfCounterWhitelist=AvgAct_KB|AvgConsum_KB|AvgGrtd_KB|AvgOvrhd_KB|AvgQueLat_ms|AvgRd|AvgRvcd_KBps|AvgSwpIn_KB|AvgSwpOut_KB|AvgTotLat_ms|AvgUsg_KBps|AvgUsg_pct|AvgVmctl_KB|AvgWr|AvgXmit_KBps|SumBusResets|SumCmdsAbort|SumRd|SumRdy_ms|SumWr|AvgSwpUsd_KB vmPerfCounterWhitelist=AvgAct_KB|AvgConsum_KB|AvgGrtd_KB|AvgOvrhd_KB|AvgQueLat_ms|AvgRd|AvgRvcd_KBps|AvgSwpIn_KB|AvgSwpOut_KB|AvgSwpd_KB|AvgTotLat_ms|AvgUsg_KBps|AvgUsg_pct|AvgVmctl_KB|AvgWr|AvgXmit_KBps|SumBusResets|SumCmdsAbort|SumRd|SumRdy_ms|SumWr|AvgUsg_mhz
Configure vCenter Server data inputs
In a simple
engine.conf file, the first stanza is typically used to get data from vCenter Server (VC). In this example, we get many different types of data from one VC using a single stanza. We collect inventory, hierarchy, tasks, events, and some VC-only performance data. In a real environment, you may need multiple stanzas for a single VC (or multiple
- Create a stanza for VC #1 in your environment
- Use a single stanza to get all 5 types of data from VC
- It is important to assign the host setting for VC machines to match the "VC instance name"
- This example assumes that the VC instance name for VC1 is “VMWARE-VC1
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, TaskDiscovery, EventDiscovery perfManagedEntityWhitelist = ClusterComputeResource interval = 1
Configure ESX/i host data inputs
Once the inputs for the VC are defined, you can add stanzas to collect data from the ESX/i hosts you want to monitor. The monitored hosts should be managed by the VC specified in previous stanza. In this example we collect the data we want (performance and log data) from two ESX/i hosts.
This is a continuation of the example above and assumes that these settings will be added after the VC stanza in the same
engine.conf file. In a real environment, you may need multiple stanzas for a single ESX/i host (or multiple
For more information about the details of the
engine.conf file (the fields, the values and settings), see "engine.conf settings" in this manual.
- Create a stanza for each ESX/i host managed by VC #1 that you also want to monitor
- Use a single stanza to get both types of data from each ESX/i host
- As this is a managed host, there is no need to assign the host setting
- Create more stanzas, up to 10 ESX/i hosts managed by VC #1
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1
Add more stanzas to the file to represent each ESX/i host that exists in VC1.
Create a stanza for VC #2 in your environment
- This example assumes that the VC instance name for VC2 is “VMWARE-VC2”
[vc2] url = https://vc2.company.com/sdk/webService host = VMWARE-VC2 username = <vc_splunk_username> password = <vc_splunkuser_password> action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, TaskDiscovery, EventDiscovery perfManagedEntityWhitelist = ClusterComputeResource interval = 1
Create a stanza for each ESX/i host managed by VC #2 that you also want to monitor. We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza
[host1inVC2] url = https://host1_in_vc2.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1 [host2inVC2] url = https://host2_in_vc2.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1
Continue to create more stanzas to represent each ESX/i host in VC2 that is monitored by Splunk for VMware.
When you have included all of the VC instances and all of the managed ESX/i hosts that you want to monitor you can now create stanzas to cover unmanaged ESX/i hosts in your environment.
Create stanzas to cover unmanaged ESX/i hosts in your environment
In this example the ESX/i host's fully qualified hostname is
Explicitly assign the host setting for an unmanaged ESX/i host and give the host setting the same value as the ESX/i host's "fully qualified hostname".
Unmanaged ESX/i hosts never need a blank VC stanza.
[host4unmanaged] url = https://host4unmanaged.company.com/sdk/webService host = host4unmanaged.company.com username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, LogDiscovery interval = 1
In this example the ESX/i host's fully qualified hostname is
[host5unmanaged] url = https://host5unmanaged.company.com/sdk/webService host = host5unmanaged.company.com username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery, InventoryDiscovery, PerfDiscovery, LogDiscovery interval = 1
Create more stanzas, as necessary, for each unmanaged ESX/i host that you want to monitor using Splunk for VMware.
Note: There are no inherent limits to the number of hosts that a given FA VM can monitor, but we recommend adding at most 10 ESX/i hosts in this simple example. For more comprehensive examples and a better understanding of when a given FA VM will hit its data gathering limits, see the topics in this manual beginning with "Data collection in one engine.conf file".
Wlakthrough a simple
engine.conf file is ready, then create an
inputs.conf file. This file is used to start up an instance of the engine (the main data collection module inside the FA Add-on) and run it. Engine instances are run by Splunk based on the stanzas found in the
inputs.conf file. Specifically, you must create a "scripted input" that calls the engine and takes the absolute path of the
engine.conf file as an argument. While multiple engines can run concurrently, the example here only runs 1 instance (the "default engine instance").
In this example, we use a simple
inputs.conf file. This file is only appropriate for monitoring a small vCenter (one that is managing less than 10 ESX/i hosts total) and a small number of ESX/i hosts (also less than 10). To create an
inputs.conf file that scales up to run multiple engine instances for your particular environment, see "Scaling your deployment" in this manual. You can use the default engine instance by turning it on using a simple
inputs.conf file. In a small environment you only need to run a single instance of the engine.
In this case you only need to create a single
engine.conf file in the
local<code> directory, and a simple <code>inputs.conf. When the FA VM boots up, it automatically runs a Splunk heavy forwarder, but the default engine instance defined in
$SPLUNK_HOME/etc/apps/splunk_for_vmware_appliance/default is initially disabled.
To turn on the default engine instance:
- If you have not already done so, log into the FA VM as the
- Stop Splunk if it is running (Splunk is already be in your path).
- Go to the FA Add-on local directory:
- Using a text editor, create the
- Set the host setting for all FA-specific data that is sent to the indexer(s). This ensures that the FA’s logs will be assigned the correct host field when sent to the indexer(s). The value should be the same one you used for the FA VM's OS hostname (during FA VM configuration steps). It should also be the same value used for the “fa” setting in the engine.conf [default] stanza.This example assumes that the FA VM's OS hostname was set to "splunkfa1"
- Turn on the default engine instance (stanza found in the FA Add-on's defaults/inputs.conf directory)
[default] host = splunkfa1
[script://./bin/Engine.pm] disabled = false
Note: Default scripted inputs (found in
default/inputs.conf) are disabled by default.
Start Splunk to run the engine
When you have created
inputs.conf</ code> for your environment, start Splunk:
When Splunk starts, the engine instance is started by the simple
inputs.conf file. The engine looks for the file
engine.conf in the
local directory. When it finds it, Splunk starts collecting data from your environment and maps it to the Splunk App for VMware.