Data collection split between six engine.conf files
This example uses separate engine.conf
files to collect data from a large VMware environment. The size of the environment can cause contention for resources and result in performance issues if data collection is not distributed between the engine.conf
files and if we limit the number of FA VMs used. We need to increase the resources to support the size of the environment and the large amount of data being collected.
Environment specification
- 1 VC.
- 80 ESX/i hosts managed by the VC.
Types of data collected
- Hierarchy.
- Inventory.
- Performance data from VC.
- Performance data from ESX/i hosts.
- Logs.
- Tasks and events.
Recommended Solution configuration
To support this large environment, we recommend using:
- 8
engine.conf
files. - 4 FA VMs.
- 4 custom
inputs.conf
files (each having 2 stanzas: one for eachengine.conf
file that it contains)
Given the number of ESX/i hosts from which we collect performance data and the large amount of inventory data we collect from a single VC, We recommend using 4 FA VMs to collect the data:
FA VM name | Description |
---|---|
splunkfa1 |
|
splunkfa2 |
|
splunkfa3 |
|
splunkfa4 |
|
- We distribute the collection of performance and log data. We use 2 FA VMs to collect he data from the hosts to eliminate contention from other data gathering activities and to prevent the loss of data.
- Each FA VM collects data for 40 hosts as the recommended host range is 30-50 hosts when using a dedicated FA VM to collect performance and/or log data.
- One FA VM is dedicated to collecting hierarchy data. Collection of tasks and events was added into the FA VM collecting hierarchy data as task and event collection has a low data volume and is not very work intensive.
- One FA VM is dedicated to collecting inventory data. Collection of VC performance data that is whitelisted to only get ClusterComputeResource metrics was added into the FA VM collecting inventory data, as task and event collection has a low data volume and is not very work intensive.
FA VM 1
FA VM 1: engine-hierarchy.conf
- Note that all hierarchy data gathering stanzas MUST be located in the same configuration file, as shown here:
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during the FA VM configuration steps).
- This example assumes that the first FA VMs OS hostname was set to
splunkfa1
.. - As the environment has 80 hosts, set the hierarchy expiration timer accordingly.
- As this is dedicated to getting hierarchy data, and the environment is sizable we can lengthen the time slot for hierarchy to gather data to 10 seconds.
[default] fa = splunkfa1 hierarchyExpiration = 9600 hierarchyTimeSlot = 10
- Use a single stanza to get the hierarchy data from vCenter.
- It is important to assign the host setting for vCenter machines to match the "vCenter instance name".
- This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = HierarchyDiscovery interval = 1
- Use a single stanza to get hierarchy data from each ESX/i host. We already have a stanza for this ESX/i host's managing vCenter in this file, so we do not need a blank VC stanza.
- As all of the ESX/i hosts are managed in this example, there is no need to assign the host setting.
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = HierarchyDiscovery interval = 1
Add the other stanzas to the file to support the remaining 78 hosts. The format of the stanzas is the same. The unique entries in each stanza are specifying the stanza name and the url.
FA VM 1: engine-task-event.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the first FA VM's OS hostname was set to "splunkfa1".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa1
- Use a single stanza to get tasks and events data from VC.
- Tasks and Events only need to be called roughly every 5 minutes when in their own stanza.
- It is important to assign the host setting for VC machines to match the "VC instance name".
- This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = TaskDiscovery, EventDiscovery interval = 290
FA VM 2
FA VM 2: engine-inv.conf
- Use a default stanza to assign certain settings once
- Set the fa name to the value you used for the FA VM's OS hostname (during FA VM configuration steps)
- This example assumes that the FA VM *2's OS hostname was set to "splunkfa2"
- Since the environment has 80 hosts, set the inventory expiration timer accordingly
[default] fa = splunkfa2 inventoryExpiration = 9600
- Use a single stanza to get the inventory data from VC
- It is important to assign the host setting for VC machines to match the "VC instance name"
- This example assumes that the VC instance name for VC1 is “VMWARE-VC1”
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = InventoryDiscovery interval = 1
FA VM 2: engine-vc-perf.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the second FA VM's OS hostname was set to "splunkfa2".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa2
- Use a single stanza to get all performance data from VC.
- It is important to assign the host setting for VC machines to match the "VC instance name".
- This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.
- As this is dedicated to getting only vCenter performance data, we can increase the interval and try to get data less frequently
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> action = PerfDiscovery perfManagedEntityWhitelist = ClusterComputeResource interval = 6
FA VM 3
FA VM 3: engine-logs-hosts-1-40.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the third FA VM's OS hostname was set to "splunkfa3".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa3
- When getting any data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
- Note that we need a "blank VC stanza" here (no action setting defined):
- The engine needs to know how to reach vCenter to get the ESX/i hostname in vCenter. It does not need to gather any specific data from the vCenter.
- In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need a single stanza.
- Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
- It is important to assign the host setting for vCenter machines to match the "VC instance name".
- This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> interval = 1
- Use a single stanza to get log data from each ESX/i host.
- Note: a "blank VC stanza" was added above to help with getting the ESX/i hostname in vCenter.
- As all ESX/i hosts are managed in this example, there is no need to assign the host setting.
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = LogDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = LogDiscovery interval = 1
Hosts 3 to 40 are omitted from the example. They can be reproduced by copying one of the above stanzas and changing the stanza name and the url.
FA VM 3: engine-perf-hosts-1-40.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the third FA VM OS hostname was set to "splunkfa3".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers
[default] fa = splunkfa3
- When getting performance data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
- Note that we need a "blank VC stanza" here (no action setting defined):
- The engine just needs to know how to reach VC to do "moid mapping" on performance data. It does not need to actually gather any specific kind of data from the vCenter.
- In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need a single stanza.
- Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
- It is important to assign the host setting for VC machines to match the "VC instance name".
- This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> interval = 1
- Use a single stanza to get performance data from each ESX/i host.
- Note: a "blank VC stanza" was added above to help with performance data gathering.
- As all ESX/i hosts are managed in this example, there is no need to assign the host setting
[host1inVC1] url = https://host1_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery interval = 1 [host2inVC1] url = https://host2_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery interval = 1
Hosts 3 to 40 are omitted from the example. They can be created by duplicating one of the above stanzas and changing the stanza name and the url.
FA VM 4
FA VM 4: engine-logs-hosts-41-80.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the fourth FA VM's OS hostname was set to "splunkfa4".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa4
- When collecting data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
- Note that we need a "blank VC stanza" here (no action setting defined):
- The engine needs to know how to reach the vCenter to get the ESX/i hostname in the vCenter. It does NOT need to collect specific data from the vCenter.
- In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need 1 stanza.
- Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
- It is important to assign the host setting for vCenter machines to match the "VC instance name".
- This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> interval = 1
- Use a single stanza to get log data from each ESX/i host.
- Note: a "blank VC stanza" was added above to help with getting the ESX/i hostname in vCenter.
- As all ESX/i hosts are managed in this example, there is no need to assign the host setting.
[host41inVC1] url = https://host41_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = LogDiscovery interval = 1 [host42inVC1] url = https://host42_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = LogDiscovery interval = 1
Hosts 43 to 80 are omitted from this example. They can be recreated by copying one of the stanzas above and changing the stanza name and the url.
FA VM 4: engine-perf-hosts-41-80.conf
- Use a default stanza to assign certain settings once.
- Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
- This example assumes that the fouth FA VM's OS hostname was set to "splunkfa4".
- We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.
[default] fa = splunkfa4
- When getting performance data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
- Note that we need a "blank VC stanza" here (no action setting defined):
- * The engine needs to know how to reach vCenter to do "moid mapping" on performance data. It does NOT need to collect data from the vCenter.
- In this case, only 1 VC manages all of the ESX/i hosts, so we only need a single stanza.
- Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
- It is important to assign the host setting for vCenter machines to match the "VC instance name".
- This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.
[vc1] url = https://vc1.company.com/sdk/webService host = VMWARE-VC1 username = <vc_splunk_username> password = <vc_splunkuser_password> interval = 1
- Use a single stanza to get performance data from each ESX/i host.
- Note: a "blank VC stanza" was added above to help with performance data gathering.
- As all the ESX/i hosts are managed in this example, there is no need to assign the host setting.
[host41inVC1] url = https://host41_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery interval = 1 [host42inVC1] url = https://host42_in_vc1.company.com/sdk/webService username = <esx_splunk_username> password = <esx_splunk_password> action = PerfDiscovery interval = 1
Hosts 43 to 840 are omitted from the example but can be re-created by copying the stanzas above and changing the stanza name and the url.
Data collection split between four engine.conf files | About configuration files |
This documentation applies to the following versions of Splunk® App for VMware (Legacy): 1.0, 1.0.1, 1.0.2, 1.0.3
Feedback submitted, thanks!