Data collection split between four engine.conf files

This example shows a recommended solution for collecting data from a medium VMware environment, that is one with 30 to 50 ESX/i hosts. In this example we use the FA VM to collect data from 40 hosts, which lies in the middle of the recommended 30 to 50 hosts range when using a dedicated FA VM to gather performance / log data. This constitutes a full workload.

Splitting data collection between four engine.conf file instances can prevent data loss as we have eliminated the possibility of contention of resources from other data gathering activities.

We split data collection up as follows:

• We collect performance and log data using two different FA VMs due to the large quantity of data being collected.
• We used a dedicated FA VM to gather hierarchy, inventory, and tasks and events data. Tasks and events data is collected using the inventory engine.conf as data collection for them is not very work-intensive and it is scheduled infrequently.

Environment specification

• 1 VC, called VC1.
• 40 ESX/i hosts managed by VC 1.

Types of data collected

• Hierarchy (from VC1 and each managed EAX/i host).
• Inventory.
• Task.
• Event.
• Logs.
• Performance.

Recommended solution

• Use 2 FA VMs.
  • FA VM 1: Called splunkfa1. It runs two engine instances. One instancene collects all hierarchy data for VC1, the other collects all inventory and tasks and events for VC1.
  • FA VM 2: Called splunkfa2. It runs two engine instances. One to collect all ESX/i host logs (for ESX/i hosts 1 to 40), the other to collect all performance data from VC1 and the 40 ESX/i hosts.
• Use 4 engine.conf file instances:
  • engine-hierarchy.conf on FA VM 1.
  • engine-inv-task-event.conf on FA VM 1.
  • engine-logs.conf on FA VM 2.
  • engine-perf.conf on FA VM 2.
• Two custom inputs.conf files each having two stanzas. ( one for each engine/engine.conf file that it contains.)

Note: Given the number of ESX/i hosts from which we are collecting performance data and the high inventory data size, we use two FA VMs to collect the data. Inventory and performance data gathering is separated to support a larger number of hosts.

FA VM 1

FA VM 1: engine-hierarchy.conf

• Note that all hierarchy data gathering stanzas MUST be located in the same configuration file, as shown here.
• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname when configuring the FA VM. This is splunkfa1.
• As the environment has 40 hosts, set the hierarchy expiration timer accordingly.
• As this is dedicated to getting hierarchy data, and the environment is sizable, we can extend the time slot for collecting hierarchy data to 10 seconds.

[default]
fa = splunkfa1
hierarchyExpiration = 4800
hierarchyTimeSlot = 10

• Use a single stanza to collect the hierarchy data from VC1.
• Assign the host setting for VC1 to match the "VC instance name". The VC instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = HierarchyDiscovery
interval = 1

• Use a single stanza to get hierarchy data from each ESX/i host.
• We already have a stanza in this file for this ESX/i host's managing VC, VC1, so we do not need to add a blank VC stanza
• All ESX/i hosts are managed by VC1 in this example, therefore we do not need to assign the host setting.

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = HierarchyDiscovery
interval = 1

Continue adding stanzas to the file for each of the hosts, 3 to 40. The stanzas follow the same format as those already included for hosts 1 and 2. Add a unique stanza name and URL to each of the stanzas to represent the specific host.

FA VM 1: engine-inv-task-event.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname when configuring the FA VM. This is "splunkfa1".
• With an environment of 40 hosts, we set the inventory expiration timer accordingly.

[default]
fa = splunkfa1
inventoryExpiration = 4800

• Use a single stanza to get the inventory data from VC1.
• Assign the host setting for VC1 to match the "VC instance name" for VC1, “VMWARE-VC1”.

[vc1-inventory]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = InventoryDiscovery
interval = 1

• Use a single stanza to get tasks and events data from VC1.
• Tasks and Events are called approximately every 5 minutes, therefore create a stanza specifically for these actions. We do not need to collect the data more frequently than that.
• Assign the host setting for VC1 to match the "VC instance name".

[vc1-tasks-events]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = TaskDiscovery, EventDiscovery
interval = 290

FA VM 2

FA VM 2: engine-logs.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname when configuring the FA VM. This is "splunkfa2".
• We are not gathering inventory or hierarchy data, so we don't reset expiration timers.

[default]
fa = splunkfa2

• Include a stanza for VC1 ( or each vCenter that manages the hosts when collecting any of the types of data from the managed ESX/i hosts).
• Add a "blank VC stanza" (no action setting defined).
• The engine must connect to VC1 to get the ESX/i hostname in VC1. It does not need to collect data from vCenter.
• In this example, a single VC manages all of the ESX/i hosts, therefore we need a single stanza.
• Set the interval to 1 to avoid unnecessary delays in collecting data. (The default is 3 seconds.)
• Assign the host setting for VC1 to match the "VC instance name". In this example we assume that the VC instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
interval = 1

• Use a single stanza to get log data from each ESX/i host.
• Note:To get the ESX/i hostname in VC1, a "blank VC stanza" was added above
• As all the ESX/i hosts are managed in this example, you do not have to assign the host setting

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = LogDiscovery
interval = 1

Hosts 3 to 40 can be recreated as necessary by copying one of the stanzas above. Specify a different stanza name and url.

FA VM 2: engine-perf.conf

• Use a default stanza to assign certain settings once.
• Set the FA VM name to the value you used for the FA VM's OS hostname when configuring the FA VM. This is "splunkfa2".
• We are not gathering inventory or hierarchy data, so we don't reset expiration timers.

[default]
fa = splunkfa2

• Use a single stanza to get all performance data from vCenter.
• When getting performance data from ESX/i hosts, we *must* have a stanza for each vCenter that manages them.
• In this case, only 1 VC manages all of the ESX/i hosts, so we only need a single stanza.
• As we already have a VC stanza for data gathering, there is no need to add a blank one.
• It is important to assign the host setting for vCenter machines to match the "VC instance name".
• This example assumes that the VC instance name for VC1 is “VMWARE-VC1”.

[vc1]
url = https://vc1.company.com/sdk/webService
host = VMWARE-VC1
username = <vc_splunk_username>
password = <vc_splunkuser_password>
action = PerfDiscovery
perfManagedEntityWhitelist = ClusterComputeResource
interval = 1

• Use a single stanza to get performance data from each ESX/i host.
• We already have a stanza for this ESX/i host's managing VC in this file, so there is no need to add a blank VC stanza.
• As all ESX/i hosts are managed in this example, there is no need to assign the host setting.

[host1inVC1]
url = https://host1_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

[host2inVC1]
url = https://host2_in_vc1.company.com/sdk/webService
username = <esx_splunk_username>
password = <esx_splunk_password>
action = PerfDiscovery
interval = 1

Hosts 3 to 40 are omitted from the example, but they can be re-created by copying one of the above stanzas and providing a unique stanza name and url.