Known issues and workarounds

Known issues in Splunk for VMware 3.1

• When Installing VMware version 3.0.2 or later on the same Splunk instance as NetApp version 2.0.1, use the same version of SA-Hydra and SA-Utils on all components of the Distributed Collection Scheduler for both apps. (SOLNVMW-3770)

Get the latest version of SA-Hydra and SA-Utils from the Splunk App for VMware.

• Syslog data from syslog server not getting indexed. Syslog data sent from a syslog server, with a line breaking issue, in the following format is not indexed (by Splunk_TA_esxilogs). (SOLNVMW-3748)

0-dd6d170cebd6
' opID=SWI-7f741cda-80] [VpxLRO] -- FINISH task-internal-157304 --  -- vpxapi.VpxaService.fetchQuickStats -- 52bfdd7d-118e-3345-58e0-dd6d170cebd6
ete (processed 29730 bytes)

• Warning messages are displayed for ta_vmware_collection_worker modular inputs. To avoid these warnings, install SA-Utils and SA-Hydra on the indexers in your environment.(SOLNVMW-3674)
• Data collection nodes are not listed on the cluster search head. When the Splunk App for VMware is deployed in a cluster, all of the configuration tasks are performed on the master node. When the rest endpoint is called on the search head, no results are returned and the Data Collection Nodes panel in the App Install Health page (on the search head) can not populate with information about the data collection nodes. (SOLNVMW-3666)
• JqueryUI error with JS minification Off. When running the Splunk App for VMware on Splunk version 6.x, if JS minification is turned off in Splunk Web, the Collection Configuration dashboard freezes. (SOLNVMW-3429)
• Splunkd process is killed due to high memory usage on the indexers. (SOLNVMW-3379)

For a workaround, see "Workaround for high memory usage on Splunk indexers"

• For a search head pooled environment, in the App Install Health dashboard, the panel displaying the data collection node does not populate. (SOLNVMW-3360).
• License warning messages are displayed for all users. The messages are intended only for admin users with license edit privileges. Please ignore the messages if you are not an admin user. (SOLNVMW-3357), (SOLNVMW-3368)
• Splunk 5.0.4 and later displays a warning message to admins when restarting from the command line. This is as a result of a possible documentation error in splunk_for_vmware/default/app.conf. For more information, run 'splunk btool check --debug'. (SOLNVMW-3343).
• There is a known network connectivity issue with syslog data being sent from the ESXi hosts. The ESXi hosts can stop sending syslog data. The splunk App for Vmware implements a workaround to fix this. A script (a saved search) is run on the indexer on a 10 minute time schedule to check for data gaps and to reset the syslog process. If the indexer does not get data within that 10 minute period, then the script is automatically kicked off to reset syslog so that data collection can continue. 
• Error when user skips configuration of collection nodes.You must configure data collection to get data into the app. (SOLNVMW-3294).
• In certain cases inventory data gaps can be more than 4 hours. (SOLNVMW-3271)
• In certain cases, ESXi logs coming via Syslog are assigned an incorrect sourcetype. (SOLNVWM-3257).
• If you configure Syslog using intermediate forwarders and loghost values are not returned, the ESXi host becomes unresponsive to the API and no error message is displayed. (SOLNVMW-3245).
• The app has no control over resetting Syslog for hosts that are disconnected but once were connected to a vCenter. The host can still forward Syslog data. (SOLNVMW-3234).
• On IE9, IE9: The Threshold Configuration view's height grows unexpectedly as you enter text in the textbox (SOLNVMW-3168).
• In IE9 the vmware_task_event_details view does not refresh after entering text in the textbox. (SOLNVMW-3163).
• In a multi vCenter environment, a parent type node, like a cluster that has no children, will render a hover chart as if it were a leaf. (SOLNVMW-3119).
• A powered off vm can't lookup datastore information from time based lookup TimeDatastoreSummary. (SOLNVMW-3083).
• When you manually edit the sa_threshold.conf file the App is not updated to reflect the changes. Always update thresholds using the Configuration Thresholds page in the App. (SOLNVMW-2543)
• The VM Detail view (that shows the performance of a VM across migrations) is not displayed correctly when the TimeHierarchyVM lookup is missing migration data for the virtual machines. When the lookup is not updated correctly the summary data can not be correctly tied to the hierarchy. This breaks views such as VM Detail that show the performance of a VM across migrations. (SOLNVMW-2186).
• VMware-managed entity lists are sorted alphabetically, but do not take case into account. For example, all VMs with names starting with initial capital letters (A-Z) appear earlier in the lists than VMs with names that start with lower-case letters (a-z) (SOLNVMW-1817).
• At times, frequent VM migrations do not appear in the Virtual Machine Detail and VM Migration views (SOLNVMW-1458).
• Cleaning tsidx space fails due to the Splunk core bug "/services/data/indexes endpoint outputs blank value for tsidxstatshomepath attribute" Affected: v6.2.0/v6.1.1-v6.1.4; Fixed in v6.1.5 and v6.2.1.(SPL-86606)
• Autocomplete text box is empty and not populating data in views on Windows due to failing to run generateIIT.py. Note: SH on Windows is not QA-tested in this version. (TAG-8879)
• In Collection Configurations, vCenter authentication fails when special characters are used in password.(TAG-8211)

Workaround for high memory usage on Splunk indexers

This problem is caused by a memory issue resulting from a known issue in Splunk Enterprise - A search returning lots of large events with multikv applied can crash the indexer splunkd (SPL-74818). In the Splunk App for VMware, performance related events use the multikv field and event expansion command. This issue arises when you run a search on VMware performance data over a large timerange and select to run the search in the background.

We recommend that when searching VMware performance data, you do so using tsidx namespaces. For example, use a command such as:

| `tstats` avg(p_average_cpu_usage_percent) max(p_average_cpu_usage_percent) from vmw_perf_cpu_hostsystem where * moid=host-2833 host=vcenter01.splunk.com groupby _time span=2m | timechart minspan=2m avg(p_average_cpu_usage_percent) AS avgUsage max(p_average_cpu_usage_percent) AS maxUsage

We do not advise that you search the raw data, using sourcetype=vmware:perf*. However, if you intend to search on raw data you must use the workaround provided.

Pivot also exposes the issue if you create your own data models of VMware performance data, as the default behavior is to set it to "all time". The Splunk App for VMware does not ship with any data models. If you use data models, we recommend that you accelerate them to avoid this issue.

Workaround

You can workaround this issue by getting splunkd to search on smaller chunks of data by reducing the number assigned to max_rawsize_perchunk, the maximum raw size of results per call to search.

1. On all indexers and search heads, create Splunk_TA_vmware/local/limits.conf and add the following stanza:

   [search]

   max_rawsize_perchunk = 20000000

2. Restart Splunk.