Data collection split between two engine.conf files

This is an example of collecting data from a medium sized environment, that is a single VC managing 20 ESX/i hosts. We use two engine.conf files that separates the collection of inventory and hierarchy data gathering into its own file. As the number of ESX/i hosts is on the lower side of a medium environment, we can collect data using a single FA VM.

We split data collection up as follows:

• One engine.conf file for performance, log, and tasks and events data collection. In this way the inventory and hierarchy data collection actions are not starved.
• One engine.conf file to collect tasks and events data and performance and log data. Collecting tasks and events data is not typically work-intensive and run on an infrequent schedule.

Note: Although we did not use a separate FA VM, using another engine.conf file can help prevent data gaps.

In this example we assume that an average of 10 VMs are running on each ESX/i host. If there were significantly more VMs per ESX/i host, then we would split the performance data and log data collection into its own FA VM (see example 4). This example is already close to the limit of the processing power available in a single FA VM using default CPU and memory settings. An alternative approach is to add additional processing power to the FA VM (for example, add 1 vCPU and double the memory settings).

As this is a medium environment and we are using two engine / engine.conf file instances inside a single FA VM, we must build a custom inputs.conf file. The inputs.conf file will have two stanzas, one for each engine / engine.conf file that it contains.

Environment specification

• 1 VC, called VC1.
• 20 ESX/i hosts managed by VC 1.

Types of data collected

• Hierarchy (from VC1 and each managed EsX/i host).
• Inventory.
• Task.
• Event.
• Logs.
• Performance.

Recommended solution

• Use 1 FA VM.
  • FA VM 1: Called splunkfa1. It runs two engine instances. One instance collects all performance, log, and tasks and events data for VC1, the other collects all tasks and events data and performance and log data for VC1.
• Use 2 engine.conf file instances:
  • engine-inv-hierarchy.conf on FA VM 1.
  • engine-perf-log-task-event.conf on FA VM 1.
• One custom inputs.conf file having two stanzas. (One stanza for each engine/engine.conf file that it contains.)

FA VM 1

FA VM 1: engine-inv-hierarchy.conf

• Note that all hierarchy data gathering stanzas MUST be located in the same configuration file, as shown here.
• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps).
• This example assumes that the FA VM's OS hostname was set to "splunkfa1".
• As the environment has 20 hosts, set the inventory and hierarchy expiration timers accordingly.

[default]
fa = splunkfa1
hierarchyExpiration = 2400
inventoryExpiration = 2400

• Use a single stanza to get the inventory and hierarchy data from vCenter.
• It is important to assign the host setting for vCenter machines to match the "VC instance name".
• This example assumes that the VC instance name for VC1 is:

“VMWARE-VC1”.
[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery, InventoryDiscovery
interval = 1

• Use a single stanza to get hierarchy data from each ESX/i host.
• We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza.
• Since the all ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

Hosts 3 to 20 can be recreated as necessary by copying a stanza above and giving them their own stanza name and url.

FA VM 1:engine-perf-log-task-event.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname (during FA VM configuration steps.).
• This example assumes that the FA VM's OS hostname was set to "splunkfa1".
• We are not gathering inventory or hierarchy data, so there is no need to reset expiration timers.

[default]
fa = splunkfa1.

• Use a single stanza to get performance data from VC.
• When getting performance data from ESX/i hosts, we *must* have a stanza for each VC that manages them.
• In this case, a single VC manages all of the ESX/i hosts, so we only need a single stanza.
• As we already have a VC stanza for data gathering, there is no need to add a blank one.
• It is important to assign the host setting for VC machines to match the "VC instance name".
• This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.

[vc1-performance]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = PerfDiscovery
perfManagedEntityWhitelist = ClusterComputeResource
interval = 1

• Use a single stanza to get tasks and events data from vCenter.
• Tasks and Events only need to be called roughly every 5 minutes so we put these actions in their own stanza
• It is important to assign the host setting for VC machines to match the "VC instance name"

[vc1-tasks-events]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = TaskDiscovery, EventDiscovery
interval = 290

• Use a single stanza to get performance and log data from each ESX/i host
• We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza
• Since the all ESX/i hosts are managed in this example, there is no need to assign the host setting

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery, LogDiscovery
interval = 1

Hosts 3 to 20 are omitted but can be replicated by copying the stanzas above and changing the stanza name and url.