Upgrade from tsidx namespaces to data model acceleration
Version 3.2.0 of the Splunk App for VMware replaces the use of tsidx namespacing with the use of data models and data model acceleration. Existing tsidx files will be not be deleted after the upgrade, and will not be utilized after the upgrade to version 3.2.0 of the Splunk App for VMware.
Previously (in versions 3.1.x and earlier), tsidx scheduling and storage was done on the search head. Starting in version 3.2.0, Data model acceleration is distributed and stored across your indexers. Spreading this task across your indexers instead of on your search head will promote more scalability across your installation.
See About data models to learn more.
Data model acceleration speeds up reporting for the entire set of attributes (fields) that you define in a data model. Data model acceleration creates summaries for the specific set of fields you and your Pivot users want to report on, accelerating the dataset represented by that collection of fields rather than a particular search against that dataset. It may take a few hours to create the data model acceleration depending on the amount of data that is historically indexed. Data that has not been accelerated yet (for example, live streaming data) will still be captured in queries by falling back to normal search for that data. Splunk Enterprise always process all summaries first, so accelerated data arrives faster.
See Accelerate data models to learn more.
Index storage size on your VMware indexes will remain unchanged with the implementation of data model acceleration. Previously (in versions 3.1.x and earlier), the age-based retention policy was 6 years, and the size-based retention policy was unlimited. Even if your raw data was deleted, tsidx data still existed, and performance and inventory charts were populated.
Starting in version 3.2.0, Data model acceleration has a 30-day retention policy, and does not have an established size-based retention policy, and can, by default, take up an unlimited amount of disk space. Data model acceleration is dependent on your raw data. So if your raw data has been deleted, the accelerated data cannot be preserved, and your performance and inventory charts cannot be populated.
Click here to learn how to set size-based retention policies.
In versions 3.1.x and earlier, summarized data (tsidx) used approximately .42 MB per day for each virtual machine. In version 3.2.0, data model acceleration uses approximately .57MB per day for each virtual machine.
Your deployment's Data Model acceleration properties are located in datamodels.conf on your indexers. See Data model acceleration configuration file reference in the Splunk App for VMware Configuration Guide to see the datamodels.conf default acceleration configuration.
Map existing tsidx namespaces to data model nodes
If you have built custom alerts, dashboards and/or reports using tsidx namespaces, use the table below to map your existing namespaces to nodes in the data models used by the VMware app.
|Namespace||Data model||Node name (not case sensitive)|
Manage deprecated namespace retention policies
$SPLUNK_HOME/etc/apps/SA-Utils/default/tsidx_retention.conf file specifies the default retention policy for all deprecated namespaces.
To set a policy for each of the specific deprecated namespaces, create a
$SPLUNK_HOME/etc/apps/<add-on-name>/local/tsidx_retention.conf file in each of the namespaces and configure the settings.
Edit your retention.conf file:
- Edit the
tsidx_retention.conffile to uncomment the deprecated namespaces that you want to use in the app.
- Change the values associated with those namespaces.
- You can keep the default values for the deprecated namespaces or modify the settings to values that work in your environment.
The app also runs a script that cleans up the tsidx namespaces. How the files are cleaned up depends on the retention policy defined for the namespace in the
TSIDX cleanup based on the file size cleans all tsidx files, but always leaves one file in that namespace. If you do not want to have one large tsidx file in the namespace, you can change the the maximum allowed size for the tsidx files.
Change the the maximum allowed size for the tsidx files:
- Open the
- Change the value of the parameter
optimize_max_size_mbto specify a size limit for a single tsidx file.
- Restart Splunk Enterprise.
Splunk App for VMware 3.1.2 and earlier includes the SA-VMW-Licensecheck package. Refer to the Installation Guide for instructions on how to remove this package to avoid benign licensing error messages.
Charts that only use summarized data
Versions 3.2.0 and above of the Splunk App for VMware and versions 2.1.0 and above of the Splunk App for NetApp DATA ONTAP use data model acceleration to gather performance and inventory data. This means that if your data collection is not accelerated, some charts and tables will not be populated.
Use the table below to see which charts and panels use only summarized data.
Dashboards using only summarized data
|Cluster Detail||VMWare > Proactive Monitoring > Entity Views > Cluster Detail||Warning panel (Visible only when Cluster is in Warning or Critical state)|
|Warning panel 1 (Visible only when Cluster is in Warning or Critical state)|
|Home Proactive Monitoring||VMWare > Home Proactive Monitoring||Virtual Machine Health -all gauges|
|Host System Health-all gauges|
|Performance of Hosts and VMs||VMWare > Performance and Capacity Planning > Performance of Hosts and VMs||Timechart|
|Capacity Planning (Hosts)||VMWare > Performance and Capacity Planning > Capacity Planning (Hosts)||Table - Host Performance|
|Displays timechart when clicked on any row of table|
|Capacity Planning Cluster||VMWare > Performance and Capacity Planning > Capacity Planning Cluster||Table - Cluster Performance|
|Displays timechart when clicked on any row of table|
|Capacity Forecasting||VMWare > Performance and Capacity Planning > Capacity Forecasting||CPU Usage Prediction Over Time (%)|
|Mem Usage Prediction Over Time (%)|
|Disk Usage Prediction Over Time (KBs/sec)|
|Capacity Planning for Clusters - CPU Headroom||VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - CPU Headroom||Capacity statistics for Selected Cluster in the last 24 hours|
|Currently used MHz and Total Capacity over time in the last 24 hours|
|VM's powered on in Selected Cluster in the last 24 hours|
|Capacity Planning for Clusters - Memory Headroom||VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - Memory Headroom||Powered on VMs memory usage in the cluster|
|Currently used GB and Total Capacity over time|
|Capacity statistics for Crest Cluster Cluster|
|Host System Detail||VMWare > Proactive Monitoring > Entity Views > Host System Detail||Warning panel (Visible only when Host is in Warning or Critical state)|
|Drop Down used for Timechart|
|Virtual Machine Detail||VMWare > Proactive Monitoring > Entity Views > Virtual Machine Detail||Drop Down used for Timechart|
|Virtual Machine Snapshots||VMWare > Performance and Capacity Planning > Virtual Machine Snapshots||Snapshots present on VM|
|Displays timechart when clicked on any row of table - Snapshots present on VM|
|Timechart-Snapshot space used on disk|
|Timechart- Number of Snapshots on datastore|
|Proactive Monitoring||VMWare > Proactive Monitoring > Proactive Monitoring||On chart pin board Graph|
|vCenter Detail Chart|
Reports using only summarized data
|Average CPU Utilization||VMWare > Knowledge Objects > Reports > Average CPU Utilization|
|Average Memory Provisioning||VMWare > Knowledge Objects > Reports > Average Memory Provisioning|
|Average VM CPU Ready||VMWare > Knowledge Objects > Reports > Average VM CPU Ready|
|Snapshots older than 14 days||VMWare > Knowledge Objects > Reports > Snapshots older than 14 days|
|Max Disk Latency per VM||VMWare > Knowledge Objects > Reports > Max Disk Latency per VM|
|Critical CPU Ready||VMWare > Knowledge Objects > Reports > Critical CPU Ready|
|Critical CPU Usage||VMWare > Knowledge Objects > Reports > Critical CPU Usage|
|Memory Utilization by Cluster||VMWare > Knowledge Objects > Reports > Memory Utilization by Cluster|
|Critical Disk Usage||VMWare > Knowledge Objects > Reports > Critical Disk Usage|
|Top Hosts with Ballooning||VMWare > Knowledge Objects > Reports > Top Hosts with Ballooning|
|CPU Usage by Cluster||VMWare > Knowledge Objects > Reports > CPU Usage by Cluster|
|CPU Usage by VC||VMWare > Knowledge Objects > Reports > CPU Usage by VC|
|CPU Ready by VC||VMWare > Knowledge Objects > Reports > CPU Ready by VC|
|Memory Usage by VC||VMWare > Knowledge Objects > Reports > Memory Usage by VC|
|Memory Ballooning by VC||VMWare > Knowledge Objects > Reports > Memory Ballooning by VC|
|Memory Swapped by VC||VMWare > Knowledge Objects > Reports > Memory Swapped by VC|
|Virtual Machine - Count by Disk Usage Severity||VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Disk Usage Severity|
|Virtual Machine - Count by Memory Usage Severity||VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Memory Usage Severity|
To identify whether your data collection has been accelerated:
- Navigate to the bottom of any chart panel, and click on the search icon to open a search string in a search view.
- Look for "summariesonly=true", and replace it with "summariesonly=false", then run the search. If your timechart begins to populate, then your data collection has been accellerated.
- Navigate to your data model settings and check progress of your data model acceleration.
Troubleshoot Splunk App for VMware
Set Splunk App for VMware trial license to work with remote license master
This documentation applies to the following versions of Splunk® App for VMware (Legacy): 4.0.4
Feedback submitted, thanks!