Data collection split between two engine.conf files
This is an example of collecting data from a medium sized environment, that is a single VC managing 20 ESX/i hosts. We use two engine.conf
files that separates the collection of inventory and hierarchy data gathering into its own file. As the number of ESX/i hosts is on the lower side of a medium environment, we can collect data using a single FA VM.
We split data collection up as follows:
- One
engine.conf
file for performance, log, and tasks and events data collection. In this way the inventory and hierarchy data collection actions are not starved. - One
engine.conf
file to collect tasks and events data and performance and log data. Collecting tasks and events data is not typically work-intensive and run on an infrequent schedule.
Note: Although we did not use a separate FA VM, using another engine.conf
file can help prevent data gaps.
In this example we assume that an average of 10 VMs are running on each ESX/i host. If there were significantly more VMs per ESX/i host, then we would split the performance data and log data collection into its own FA VM (see example 4). This example is already close to the limit of the processing power available in a single FA VM using default CPU and memory settings. An alternative approach is to add additional processing power to the FA VM (for example, add 1 vCPU and double the memory settings).
As this is a medium environment and we are using two engine / engine.conf
file instances inside a single FA VM, we must build a custom inputs.conf
file. The inputs.conf
file will have two stanzas, one for each engine / engine.conf
file that it contains.
Environment specification
- 1 VC, called VC1.
- 20 ESX/i hosts managed by VC 1.
Types of data collected
- Hierarchy (from VC1 and each managed EsX/i host).
- Inventory.
- Task.
- Event.
- Logs.
- Performance.
Recommended solution
- Use 1 FA VM.
- FA VM 1: Called splunkfa1. It runs two engine instances. One instance collects all performance, log, and tasks and events data for VC1, the other collects all tasks and events data and performance and log data for VC1.
- Use 2
engine.conf
file instances:engine-inv-hierarchy.conf
on FA VM 1.engine-perf-log-task-event.conf
on FA VM 1.
- One custom
inputs.conf
file having two stanzas. (One stanza for each engine/engine.conf
file that it contains.)
FA VM 1
FA VM 1: engine-inv-hierarchy.conf
- Note that all hierarchy data gathering stanzas MUST be located in the same configuration file, as shown here.
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the FA VM's OS hostname was set to "splunkfa1".
- As the environment has 20 hosts, set the inventory and hierarchy expiration timers accordingly.
[default] fa = splunkfa1 hierarchyExpiration = 2400 inventoryExpiration = 2400
- Use a single stanza to get the inventory and hierarchy data from vCenter.
- It is important to assign the host setting for vCenter machines to match the "VC instance name".
- This example assumes that the VC instance name for VC1 is:
“VMWARE-VC1”. [vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = HierarchyDiscovery, InventoryDiscovery interval = 1
- Use a single stanza to get hierarchy data from each ESX/i host.
- We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza.
- Since the all ESX/i hosts are managed in this example, there is no need to assign the host setting.
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery interval = 1
Hosts 3 to 20 can be recreated as necessary by copying a stanza above and giving them their own stanza name and url.
FA VM 1:engine-perf-log-task-event.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps.).
- This example assumes that the FA VM's OS hostname was set to "splunkfa1".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa1.
- Use a single stanza to get performance data from VC.
- When getting performance data from ESX/i hosts, we *must* have a stanza for each VC that manages them.
- In this case, a single VC manages all of the ESX/i hosts, so we only need a single stanza.
- As we already have a VC stanza for data gathering, there is no need to add a blank one.
- It is important to assign the host setting for VC machines to match the "VC instance name".
- This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.
[vc1-performance] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = PerfDiscovery perfManagedEntityWhitelist = ClusterComputeResource interval = 1
- Use a single stanza to get tasks and events data from vCenter.
- Tasks and Events only need to be called roughly every 5 minutes so we put these actions in their own stanza
- It is important to assign the host setting for VC machines to match the "VC instance name"
[vc1-tasks-events] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = TaskDiscovery, EventDiscovery interval = 290
- Use a single stanza to get performance and log data from each ESX/i host
- We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza
- Since the all ESX/i hosts are managed in this example, there is no need to assign the host setting
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery, LogDiscovery interval = 1
Hosts 3 to 20 are omitted but can be replicated by copying the stanzas above and changing the stanza name and url.
Data collection in one engine.conf file | Data collection split between four engine.conf files |
This documentation applies to the following versions of Splunk® App for VMware (Legacy): 1.0, 1.0.1, 1.0.2, 1.0.3
Feedback submitted, thanks!