Is data coming in?

Use the App Install Health dashboard to check that data collection is working and that you are collecting the correct data types. If you are, then your dashboards will populating correctly.

The App Install Health Dashboard

The App Install Health dashboard is the first place of referenced to see that you have set up and configured your environment correctly. On your indexer/search head select Settings > App Install health on the app menu to display this dashboard.

We recommend that you wait for about 1 hour for views to populate before you troubleshoot your app.

Panel	Description
Data Collection Nodes	Shows the ip address of the data collection node, the user, and when it was last updated.
ESXi host log events over last 24 hours	Shows the logs that come in to Splunk from the ESXi hosts. If data does not come in, you may have a firewall issue. It will list the hosts for which ESXi logs have been indexed by Splunk in the last 24 hours. The second column lists whether ESXi logs have been indexed for those hosts in the last 15 minutes. This immediately highlights if there is a data issue with getting ESXi logs.
vCenter Forwarders count over the last 4 hours	Shows the that vCenter logs are coming in and the Sourcetypes collected.
VMware Sourcetypes Last Received Status	Look here to see that all your sourcetypes (such as esxilog, vclog, events, inv) are coming in to Splunk. You can see where the data came form, what time it was sent and when it was indexed. You can also see when it was last indexed.
Lookups in SA-VMW-Hierarchy Addon that must be populated	These lookup tables must be populated for the app to work correctly. FullHierarchy: This is the main lookup table. It defines the structure of your environment and contains all the hierarchy information about what vCenter is connected to what hosts and what virtual machines are on what hosts. Your app will not work if the FullHierarchy lookup is not populated. Time DatastoreSummary: This lookup has a list of virtual machines on host and the memory information. TimeVirtualMachinesOnDatastore: This lookup has a list of datastores' capacity and free space.
Hierarchy collection by time	Shows the hierarchy data collected over time. The data points show hierarchy data collection based on sessions, if any two sessions are separated by a time period of more than 2 hours there can be a problem in hierarchy data collection.
Vmware Performance Data TSIDX Namespaces by Count	All performance data is stored under TSIDX name spaces and no longer in summary indexes. If there are no namespaces, then performance data does not exist in the index. tsidx_namespaces for performance data are shown separately on this chart. vmw_perf_mem_virtualmachine vmw_perf_disk_virtualmachine vmw_perf_cpu_virtualmachine vmw_perf_datastore_virtualmachine vmw_perf_rescpu_virtualmachine vmw_perf_power_virtualmachine vmw_perf_sys_virtualmachine vmw_perf_net_virtualmachine vmw_perf_hbr_virtualmachine vmw_perf_mem_virtualmachine
VMware Performance Data TSIDX Namespaces by Filesize	Displays the file size for the hierarchy data collected.
VMware Performance Data TSIDX Namespaces	Displays the hierarchy data collected show in a table.
Current Entity Data	This table is populated with information from your most recent hierarchy and shows that it is up to date. It provides a high level visual confirmation of current entity data in your environment. It tells you the number of machines from which you collect data and what they are. It also displays information about your current hierarchy. The entity data types displayed are ClusterComputeResource, Datacenter, Folder, HostSystem, RootFolder, and VirtualMachine. If the Hierarchy data is not displayed, then the Proactive Monitoring tree in the VMware app will not display as it depends on having hierarchy information available.

Validate that the Splunk App for VMware is installed correctly

You can use this dashboard to check that:

• you installed the app correctly.
• your data is timestamped correctly.
• your forwarders are set up correctly.
• you are collecting the correct data types.

Did you install the app correctly?

In the Splunk App for VMware, select Settings > App Install Health to get to this dashboard. On this dashboard you can:

• Validate the integrity of your data by examining the status of your environment.
• Check what you installed into your environment.
• See how your environment is configured to collect data.

Always check inventory, hierarchy, time, performance, and log data in the app for the given vCenter server and the associated ESXi hosts.

Views can take time to populate especially if you have a large environment collecting many different types of data. Wait for the dashboards to load the data before you start troubleshooting.

Is your data timestamped correctly

In the Sourcetypes last received status view, check the recent index time and the recent sent time to see if your data collection node or any of your ESXi hosts show up with an unacceptable time difference. If some do not, the clock on that host is most likely set incorrectly. If the time is set incorrectly, fix the time on all of your hosts for the solution to work correctly. If you reset the time on your data collection node, restart the forwarder inside it, wait for a few minutes, and then verify again that the time difference you see is within an acceptable time range.

When you have verified the time for the data collection node and all of your ESXi hosts, check that data collection works in the app for all of the different data types specified.

Are forwarders set up correctly

To check that you have correctly set up your forwarders to forward data:

1. In the Forwarder Appliance(s) over last 4 hours view, check that all the forwarder appliances that you have as part of the app are included in the list.
2. Select each forwarder appliance in the list individually, and check that the Forwarder Appliance summary displays data for each.

To check that you have correctly set up your vCenter servers:

1. In the vCenter Forwarder(s) count over last 4 hours view, check that all the vCenters from where you installed the vCenter add-on show up in the list.
2. Select each vCenter individually to ensure that the Virtual Center summary shows data for all vcenters.

Are you collecting the correct type of data

On the App Install health view, look at the sourcetypes last received status to check that the correct type of data ( inventory, hierarchy, performance, ESXi log data, tasks and events, vCenter log data) is collected, and that it is indexed recently and that it was sent within a recent timeframe.

For hierarchy and performance data, check that the relevant lookups have populated correctly. See Lookups in SA_HiearchyInventory Addon that must be populated and Lookups in SA_Performance Addon that must be populated. Also look at the Current Hierarchy Data and Current Entity Data views.

Run searches to validate performance data results

Run the searches below to validate performance data results.

• To check that data collection has continued for the last 15 minutes:

index=vmware sourcetype=vmware:perf

Four types of data are displayed - ClusterComputeResourcePerf, HostSystemPerf, ResourcePoolPerf, and VirtualMachinePerf.

• Shows a breakdown of all hosts that send performance data.

index=vmware sourcetype=vmware:perf | stats count by source

• Shows a breakdown of all hosts that send performance data and the types of data sent. ClusterComputeResourcePerf should only be returned by the Virtual Center hosts.

index=vmware sourcetype=vmware:perf | stats values(source) by host

Check ESXi log data by host

To check that ESXi log data is collected for each ESXi host monitored, run:

index=vmware sourcetype=vmware:esxilog:* | stats count by host

Check tasks and events data by host

• To display all the hosts (including vCenters) from which you receive task data, run:

index=vmware sourcetype=vmware:task | stats count by host

Check that all the hosts included in your environment are listed.

• To display all the events (including vCenters) from which you are receiving task data, run

index=vmware sourcetype=vmware:event | stats count by host

Check that all the events in your splunked environment are listed.

Check vCenter log data

• For all vCenter servers from which data is collected, look at Virtual center forwarding status to see that data is being received.
• Check that vCenter log data is collected correctly. Click on the vclog data sourcetype and drill down to get more detailed information.