Splunk® App for VMware (Legacy)

Installation Guide

On August 31, 2022, the Splunk App for VMware will reach its end of life. After this date, Splunk will no longer maintain or develop this product. The functionality in this app is migrating to a content pack in Data Integrations. Learn about the Content Pack for VMware Dashboards and Reports.
This documentation does not apply to the most recent version of Splunk® App for VMware (Legacy). For documentation on the most recent version, go to the latest release.

Troubleshoot Splunk App for VMware

1. Review the release notes to determine if the trouble you are experiencing is a known issue.

2. Consider enabling the troubleshooting logs on data collection nodes to facilitate root cause investigation. See Enable troubleshooting logs.

3. Review the following problems for advice on how to resolve them.


Problems to troubleshoot

Problem

Splunk App for VMware does not seem able to make read-only API calls to vCenter Server systems.

Cause

You do not have the appropriate vCenter Server service login credentials for each vCenter Server.

Resolution

Obtain vCenter Server service login credentials for each vCenter server. See Permissions in vSphere.

Problem

You have configured vCenter Server 5.0 in Splunk App for VMware, but no data is coming in.

Cause

vCenter Server 5.0 and 5.1 are missing WSDL files that are required for Splunk App for VMware to make API calls to vCenter Server.

  • reflect-message.xsd
  • reflect-types.xsd

Resolution

Install the missing VMware WSDL files as documented in the vSphere Web Services SDK WSDL workaround in the VMware documentation. Note that the programdata folder is typically a hidden folder.

Problem

The individual tests in the deployment section work, but the Splunk App for VMware main dashboard is empty.

Cause

This is typically caused by one of two issues:

  • Time synchronization issues between the indexer, DCN, and vCenter Server
  • Incorrect permissions assignments in Splunk Enterprise

Resolution

  • Make sure that you have assigned the correct roles to users of Splunk Enterprise.
User Role
admin user splunk_vmware_admin
all users of the Splunk App for VMware splunk_vmware_user


Problem

The DCNs are forwarding data using index=_internal tests, but Splunk App for VMware is not collecting any API data.

Cause

This is typically caused by one of two issues:

  • Network connectivity issues from the Scheduler to the DCNs.
  • You have not changed the DCN admin account password from its default value.

Resolution

  • In the Splunk for VMware App Settings page, verify the accuracy of the settings in the collection page.
  • Verify that the admin password for each DCN is not set to changeme.
  • Verify that each DCN has a fixed IP address. If Splunk App for VMware uses DCN host names instead of fixed IP addresses, verify that DNS lookups resolve to the correct IP addresses.


Problem

Splunk App for VMware works for 60 days, then stops.

Cause

The 60-day trial license for Splunk App for VMware has expired.

Resolution

Configure the DCN to join your Splunk Enterprise license pool.

splunk edit licenser-localslave -master_uri https://myhost:8089

Problem

Splunk App for VMware seems to be collecting only partial data. Hosts are missing, and so on.

Cause

There are insufficient DCNs to handle the data volume coming from the ESXi environment.

Resolution

  1. From the Settings page in Splunk App for VMware, review the list of hosts for each vCenter Server environment.
  2. Verify that each DCN polls information for up to 40 ESXi hosts and 1,000 virtual machines (30/750 is recommended), based on the specifications for a 4 core DCN (the one configured with the OVA for VMware). Based on this sizing, a site that pulls information from 200 hypervisors and 5,000 VMs needs at least 5 DCNs.
  3. Verify that the number of worker processes must be one fewer than the number of CPU cores the vCenter Server system granted to the DCN. For example, if the DCN has four CPU cores, the number of worker processes is three.
  4. In the Splunk App for VMware configuration pane for the DCN, make sure that the number of worker processes is one fewer than the number of CPU cores assigned to the machine.

    Problem

    The DCNs are not delivering data to the Splunk Enterprise indexers.

    Cause

    If the DCNs are configured correctly, this problem is typically the result of a connectivity issue.

    Resolution

    • Make sure that the DCN has an IP address and can resolve DNS.
    • Verify that no firewalls are preventing communication between the DCN and port 9997 on the indexers and that the Scheduler can connect to ports 8089 and 8008 on each DCN. On the search head, search index=_internal host=DCN-hostname-here.

    Problem

    In the Host Detail view of Splunk Enterprise version 6.0 or later, you see the following warning message:

    Events may not be returned in sub-second order due to search memory limits configured in limits.conf:[search]:max_rawsize_perchunk. See seasrch.log for more information.

    Or, in search.log of Splunk Enterprise version 6.0 or later, you see the following warning message: 

    02-06-2014 15:47:04.353 ERROR databasePartitionPolicy - Max Raw Size Limit Exceeded
02-06-2014 15:47:04.467 INFO  UnifiedSearch - Error in 'databasePartitionPolicy': Max Raw Size Limit Exceeded
02-06-2014 15:47:04.467 WARN  CursoredSearch - Events may be returned not in exact sub-second order: M=1368 > N=1250, where M is the number of events read in the 1390841799th second, and N is max number of events to read in a single span. Note that N was scaled back because we exceeded limits.conf:[search]:max_rawsize_perchunk value=100000000


    Cause

    A bug: non-surppressed error message (SOLNVMW-3587)

    Resolution

    1. On indexers, add the following configuration in the limits.conf file in the $SPLUNK_HOME/etc/system/local directory: 

    limits.conf
[search]
max_rawsize_perchunk = 800000000

    2. Restart the indexer.

    3. Test to you if still see the error message in Splunk App for VMware Host Detail view.


    Note that setting the value of max_rawsize_perchunk = 400000000 surppresses the warning message in the Host Detail view. However, in the search.log file, you will still see the following message: 

    02-07-2014 14:52:43.008 ERROR databasePartitionPolicy - Max Raw Size Limit Exceeded
02-07-2014 14:52:43.127 INFO  UnifiedSearch - Error in 'databasePartitionPolicy': Max Raw Size Limit Exceeded

    To mitigate the appearance of these error messages, set the max_rawsize_perchunk to at least 600000000.

    Problem

    Incomplete or no data coming from vCenters that are properly configured and connected to by a DCN. Data collection tasks are failing and/or connections between DCN and vCenter are closing before all data is transferred. This could be due to one of two issues.

    • The collection tasks taking longer than the vCenter and app are expecting.
    • Collection intervals are currently overloading your Data Collection Nodes (DCNs) and your vCenters.

    Resolution

    Change collection intervals in order to reduce the load on your Data Collection Nodes (DCNs) and your vCenters

    1. Change the time interval for your host inventory job.
      1. On the instance where your scheduler is running, navigate to \etc\apps\Splunk_TA_vmware\default\.
      2. Open the ta_vmware_collection.conf file.
      3. Change hostinv_interval and hostinv_expiration from the 900 second default to a larger number (maximum 2700 seconds). Keep hostinv_interval and hostinv_expiration at the same number of seconds.
      4. Save your changes and exit.
    2. Change the time interval for host performance data.
      1. On the instance where your scheduler is running, navigate to \etc\apps\Splunk_TA_vmware\local\.
      2. Open the ta_vmware_collection.conf file.
      3. Change hostvmperf_interval and hostvmperf_expiration from the 180 second default to a larger number (maximum 1200 seconds). Keep hostvmperf_interval and hostvmperf_expiration at the same number of seconds.
      4. Save your changes and exit.
    3. Increase the timeout period in the vpxd file on your vCenter.
      1. Open the vpxd.cfg file, located in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg file (C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg on Windows 2008) using a text editor.
      2. Add the below information in the <vpxd> tags:
      <heartbeat>
<notRespondingTimeout>180</notRespondingTimeout>
</heartbeat>
      1. Restart your VMware VirtualCenter Server service.






Last modified on 03 October, 2016
Collect VMware vCenter Server Linux Appliance log data   Upgrade from tsidx namespaces to data model acceleration

This documentation applies to the following versions of Splunk® App for VMware (Legacy): 3.3.1

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters