Data collection split between six engine.conf files

This example uses separate engine.conf files to collect data from a large VMware environment. The size of the environment can cause contention for resources and result in performance issues if data collection is not distributed between the engine.conf files and if we limit the number of FA VMs used. We need to increase the resources to support the size of the environment and the large amount of data being collected.

Environment specification

• 1 VC.
• 80 ESX/i hosts managed by the VC.

Types of data collected

1. Hierarchy.
2. Inventory.
3. Performance data from VC.
4. Performance data from ESX/i hosts.
5. Logs.
6. Tasks and events.

Recommended Solution configuration

To support this large environment, we recommend using:

• 8 engine.conf files.
• 4 FA VMs.
• 4 custom inputs.conf files (each having 2 stanzas: one for each engine.conf file that it contains)

Given the number of ESX/i hosts from which we collect performance data and the large amount of inventory data we collect from a single VC, We recommend using 4 FA VMs to collect the data:

FA VM name	Description
splunkfa1	Run 1 engine to collect all hierarchy data from VC 1 and the ESX/i hosts (1 - 80). Run 1 engine for tasks and events from VC1.
splunkfa2	Run 1 engine to collect all inventory data from VC1. Run 1 engine to collect all VC performance data from VC1.
splunkfa3	Run 1 engine to collect all ESX/i host logs. Run 1 engine to collect all ESX/i host performance data (for ESX/i hosts 1 - 40).
splunkfa4	Run 1 engine to collect all ESX/i host logs. Run 1 engine to collect all ESX/i host performance data (for ESX/i hosts 41 - 80).

• We distribute the collection of performance and log data. We use 2 FA VMs to collect he data from the hosts to eliminate contention from other data gathering activities and to prevent the loss of data.
• Each FA VM collects data for 40 hosts as the recommended host range is 30-50 hosts when using a dedicated FA VM to collect performance and/or log data.
• One FA VM is dedicated to collecting hierarchy data. Collection of tasks and events was added into the FA VM collecting hierarchy data as task and event collection has a low data volume and is not very work intensive.
• One FA VM is dedicated to collecting inventory data. Collection of VC performance data that is whitelisted to only get ClusterComputeResource metrics was added into the FA VM collecting inventory data, as task and event collection has a low data volume and is not very work intensive.

FA VM 1

FA VM 1: engine-hierarchy.conf

• Note that all hierarchy data gathering stanzas MUST be located in the same configuration file, as shown here:

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during the FA VM configuration steps).
• This example assumes that the first FA VMs OS hostname was set to splunkfa1..
• As the environment has 80 hosts, set the hierarchy expiration timer accordingly.
• As this is dedicated to getting hierarchy data, and the environment is sizable we can lengthen the time slot for hierarchy to gather data to 10 seconds.

[default]
fa = splunkfa1
hierarchyExpiration = 9600
hierarchyTimeSlot = 10

• Use a single stanza to get the hierarchy data from vCenter.
• It is important to assign the host setting for vCenter machines to match the "vCenter instance name".
• This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery
interval = 1

• Use a single stanza to get hierarchy data from each ESX/i host. We already have a stanza for this ESX/i host's managing vCenter in this file, so we do not need a blank VC stanza.
• As all of the ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

Add the other stanzas to the file to support the remaining 78 hosts. The format of the stanzas is the same. The unique entries in each stanza are specifying the stanza name and the url.

FA VM 1: engine-task-event.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the first FA VM's OS hostname was set to "splunkfa1".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa1

• Use a single stanza to get tasks and events data from VC.
• Tasks and Events only need to be called roughly every 5 minutes when in their own stanza.
• It is important to assign the host setting for VC machines to match the "VC instance name".
• This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = TaskDiscovery, EventDiscovery
interval = 290

FA VM 2

FA VM 2: engine-inv.conf

• Use a default stanza to assign certain settings once
• Set the fa name to the value you used for the FA VM's OS hostname (during FA VM configuration steps)
• This example assumes that the FA VM *2's OS hostname was set to "splunkfa2"
• Since the environment has 80 hosts, set the inventory expiration timer accordingly

[default]
fa = splunkfa2
inventoryExpiration = 9600

• Use a single stanza to get the inventory data from VC
• It is important to assign the host setting for VC machines to match the "VC instance name"
• This example assumes that the VC instance name for VC1 is “VMWARE-VC1”

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = InventoryDiscovery
interval = 1

FA VM 2: engine-vc-perf.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the second FA VM's OS hostname was set to "splunkfa2".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa2

• Use a single stanza to get all performance data from VC.
• It is important to assign the host setting for VC machines to match the "VC instance name".
• This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.
• As this is dedicated to getting only vCenter performance data, we can increase the interval and try to get data less frequently

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = PerfDiscovery
perfManagedEntityWhitelist = ClusterComputeResource
interval = 6

FA VM 3

FA VM 3: engine-logs-hosts-1-40.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the third FA VM's OS hostname was set to "splunkfa3".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa3

• When getting any data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
• Note that we need a "blank VC stanza" here (no action setting defined):

  • The engine needs to know how to reach vCenter to get the ESX/i hostname in vCenter. It does not need to gather any specific data from the vCenter.

• In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need a single stanza.
• Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
• It is important to assign the host setting for vCenter machines to match the "VC instance name".
• This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
interval = 1

• Use a single stanza to get log data from each ESX/i host.
• Note: a "blank VC stanza" was added above to help with getting the ESX/i hostname in vCenter.
• As all ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

Hosts 3 to 40 are omitted from the example. They can be reproduced by copying one of the above stanzas and changing the stanza name and the url.

FA VM 3: engine-perf-hosts-1-40.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the third FA VM OS hostname was set to "splunkfa3".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers

[default]
fa = splunkfa3

• When getting performance data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
• Note that we need a "blank VC stanza" here (no action setting defined):
• The engine just needs to know how to reach VC to do "moid mapping" on performance data. It does not need to actually gather any specific kind of data from the vCenter.
• In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need a single stanza.
• Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
• It is important to assign the host setting for VC machines to match the "VC instance name".
• This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
interval = 1

• Use a single stanza to get performance data from each ESX/i host.
• Note: a "blank VC stanza" was added above to help with performance data gathering.
• As all ESX/i hosts are managed in this example, there is no need to assign the host setting

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

Hosts 3 to 40 are omitted from the example. They can be created by duplicating one of the above stanzas and changing the stanza name and the url.

FA VM 4

FA VM 4: engine-logs-hosts-41-80.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the fourth FA VM's OS hostname was set to "splunkfa4".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa4

• When collecting data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
• Note that we need a "blank VC stanza" here (no action setting defined):

  • The engine needs to know how to reach the vCenter to get the ESX/i hostname in the vCenter. It does NOT need to collect specific data from the vCenter.

• In this case, only 1 vCenter manages all of the ESX/i hosts, so we only need 1 stanza.
• Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
• It is important to assign the host setting for vCenter machines to match the "VC instance name".
• This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
interval = 1

• Use a single stanza to get log data from each ESX/i host.
• Note: a "blank VC stanza" was added above to help with getting the ESX/i hostname in vCenter.
• As all ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host41inVC1]
url = https://host41_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

[host42inVC1]
url = https://host42_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

Hosts 43 to 80 are omitted from this example. They can be recreated by copying one of the stanzas above and changing the stanza name and the url.

FA VM 4: engine-perf-hosts-41-80.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the fouth FA VM's OS hostname was set to "splunkfa4".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa4

• When getting performance data from managed ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
• Note that we need a "blank VC stanza" here (no action setting defined):
• * The engine needs to know how to reach vCenter to do "moid mapping" on performance data. It does NOT need to collect data from the vCenter.
• In this case, only 1 VC manages all of the ESX/i hosts, so we only need a single stanza.
• Set the interval to 1 to avoid unnecessary delays in data gathering. (The default is 3 seconds.)
• It is important to assign the host setting for vCenter machines to match the "VC instance name".
• This example assumes that the vCenter instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
interval = 1

• Use a single stanza to get performance data from each ESX/i host.
• Note: a "blank VC stanza" was added above to help with performance data gathering.
• As all the ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host41inVC1]
url = https://host41_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

[host42inVC1]
url = https://host42_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

Hosts 43 to 840 are omitted from the example but can be re-created by copying the stanzas above and changing the stanza name and the url.