Upgrade from tsidx namespaces to data model acceleration

Version 3.2.0 of the Splunk App for VMware replaces the use of tsidx namespacing with the use of data models and data model acceleration. Existing tsidx files will be not be deleted after the upgrade, and will not be utilized after the upgrade to version 3.2.0 of the Splunk App for VMware.

Previously (in versions 3.1.x and earlier), tsidx scheduling and storage was done on the search head. Starting in version 3.2.0, Data model acceleration is distributed and stored across your indexers. Spreading this task across your indexers instead of on your search head will promote more scalability across your installation.

See About data models to learn more.

Data model acceleration speeds up reporting for the entire set of attributes (fields) that you define in a data model. Data model acceleration creates summaries for the specific set of fields you and your Pivot users want to report on, accelerating the dataset represented by that collection of fields rather than a particular search against that dataset. It may take a few hours to create the data model acceleration depending on the amount of data that is historically indexed. Data that has not been accelerated yet (for example, live streaming data) will still be captured in queries by falling back to normal search for that data. Splunk Enterprise always process all summaries first, so accelerated data arrives faster.

See Accelerate data models to learn more.

Index storage size on your VMware indexes will remain unchanged with the implementation of data model acceleration. Previously (in versions 3.1.x and earlier), the age-based retention policy was 6 years, and the size-based retention policy was unlimited. Even if your raw data was deleted, tsidx data still existed, and performance and inventory charts were populated.

Starting in version 3.2.0, Data model acceleration has a 30-day retention policy, and does not have an established size-based retention policy, and can, by default, take up an unlimited amount of disk space. Data model acceleration is dependent on your raw data. So if your raw data has been deleted, the accelerated data cannot be preserved, and your performance and inventory charts cannot be populated.

Click here to learn how to set size-based retention policies.

In versions 3.1.x and earlier, summarized data (tsidx) used approximately .42 MB per day for each virtual machine. In version 3.2.0, data model acceleration uses approximately .57MB per day for each virtual machine.

Your deployment's Data Model acceleration properties are located in datamodels.conf on your indexers. See Data model acceleration configuration file reference in the Configuration Guide to see the datamodels.conf default acceleration configuration.

Map existing tsidx namespaces to data model nodes

If you have built custom alerts, dashboards and/or reports using tsidx namespaces, use the table below to map your existing namespaces to nodes in the data models used by the VMware app.

Namespace	Data model	Node name (not case sensitive)
vmw_perf_cpu_hostsystem	VMWarePerformance	vmperformance.cpu.HostSystemCPU
vmw_perf_cpu_virtualmachine	VMWarePerformance	vmperformance.cpu.virtualmachinecpu
vmw_perf_datastore_virtualmachine	VMWarePerformance	vmperformance.datastore.virtualmachinedatastore
vmw_perf_disk_virtualmachine	VMWarePerformance	vmperformance.disk.virtualmachinedisk
vmw_perf_mem_virtualmachine	VMWarePerformance	vmperformance.memory.virtualmachinememory
vmw_perf_net_virtualmachine	VMWarePerformance	vmperformance.net.virtualmachinenet
vmw_perf_power_virtualmachine	VMWarePerformance	vmperformance.power.virtualmachinepower
vmw_perf_resdisk_virtualmachine	VMWarePerformance	vmperformance.resdisk. virtualmachineresdisk
vmw_perf_sys_virtualmachine	VMWarePerformance	vmperformance.sys.virtualmachinesys
vmw_perf_rescpu_virtualmachine	VMWarePerformance	vmperformance.rescpu.virtualmachinerescpu
vmw_perf_datastore_hostsystem	VMWarePerformance	Vmperformance.datastore.hostsystemdatastore
vmw_perf_disk_hostsystem	VMWarePerformance	vmperformance.disk.HostSystemDisk
vmw_perf_hbr_hostsystem	VMWarePerformance	vmperformance.hbr.hostsystemhbr
vmw_perf_mem_hostsystem	VMWarePerformance	vmperformance.memory.HostSystemMemory
vmw_perf_net_hostsystem	VMWarePerformance	vmperformance.net.hostsystemnet
vmw_perf_power_hostsystem	VMWarePerformance	vmperformance.power.hostsystempower
vmw_perf_resstorageadapter_hostsystem	VMWarePerformance	vmperformance.resstorageadapter.hostsystemresstorageadapter
vmw_perf_storageadapter_hostsystem	VMWarePerformance	vmperformance.storageadapter.hostsystemstorageadapter
vmw_perf_storagepath_hostsystem	VMWarePerformance	vmperformance.storagepath.hostsystemstoragepath
vmw_perf_sys_hostsystem	VMWarePerformance	vmperformance.sys.hostsystemsys
vmw_perf_rescpu_hostsystem	VMWarePerformance	vmperformance.rescpu.hostsystemrescpu
vmw_inv_datastore_virtualmachine	VMwareInventory	Inventory

Manage deprecated namespace retention policies

The $SPLUNK_HOME/etc/apps/SA-Utils/default/tsidx_retention.conf file specifies the default retention policy for all deprecated namespaces.

To set a policy for each of the specific deprecated namespaces, create a $SPLUNK_HOME/etc/apps/<add-on-name>/local/tsidx_retention.conf file in each of the namespaces and configure the settings.

Edit your retention.conf file:

1. Edit the tsidx_retention.conf file to uncomment the deprecated namespaces that you want to use in the app.
2. Change the values associated with those namespaces.
• You can keep the default values for the deprecated namespaces or modify the settings to values that work in your environment.

The app also runs a script that cleans up the tsidx namespaces. How the files are cleaned up depends on the retention policy defined for the namespace in the $SPLUNK_HOME/etc/apps/SA-VMW-Performance/default/tsidx_retention.conf file.

TSIDX cleanup based on the file size cleans all tsidx files, but always leaves one file in that namespace. If you do not want to have one large tsidx file in the namespace, you can change the the maximum allowed size for the tsidx files.

Change the the maximum allowed size for the tsidx files:

1. Open the $SPLUNK_HOME/etc/system/local/limit.conf file.
2. Change the value of the parameter optimize_max_size_mb to specify a size limit for a single tsidx file.
3. Restart Splunk Enterprise.

Remove SA-VMW-Licensecheck

Splunk App for VMware 3.1.2 and earlier includes the SA-VMW-Licensecheck package. Refer to the Installation Guide for instructions on how to remove this package to avoid benign licensing error messages.

Charts that only use summarized data

Versions 3.2.0 and above of the Splunk App for VMware and versions 2.1.0 and above of the Splunk App for NetApp DATA ONTAP use data model acceleration to gather performance and inventory data. This means that if your data collection is not accelerated, some charts and tables will not be populated.

Use the table below to see which charts and panels use only summarized data.

Dashboards using only summarized data

Dashboard	Path	Chart/Table/Panel
Cluster Detail	VMWare > Proactive Monitoring > Entity Views > Cluster Detail	Warning panel (Visible only when Cluster is in Warning or Critical state)
		Timechart
		Warning panel 1 (Visible only when Cluster is in Warning or Critical state)
Home Proactive Monitoring	VMWare > Home Proactive Monitoring	Virtual Machine Health -all gauges
		Host System Health-all gauges
Performance of Hosts and VMs	VMWare > Performance and Capacity Planning > Performance of Hosts and VMs	Timechart
Capacity Planning (Hosts)	VMWare > Performance and Capacity Planning > Capacity Planning (Hosts)	Table - Host Performance
		Displays timechart when clicked on any row of table
Capacity Planning Cluster	VMWare > Performance and Capacity Planning > Capacity Planning Cluster	Table - Cluster Performance
		Displays timechart when clicked on any row of table
Capacity Forecasting	VMWare > Performance and Capacity Planning > Capacity Forecasting	CPU Usage Prediction Over Time (%)
		Mem Usage Prediction Over Time (%)
		Disk Usage Prediction Over Time (KBs/sec)
Capacity Planning for Clusters - CPU Headroom	VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - CPU Headroom	Capacity statistics for Selected Cluster in the last 24 hours
		Currently used MHz and Total Capacity over time in the last 24 hours
		VM's powered on in Selected Cluster in the last 24 hours
Capacity Planning for Clusters - Memory Headroom	VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - Memory Headroom	Powered on VMs memory usage in the cluster
		Currently used GB and Total Capacity over time
		Capacity statistics for Crest Cluster Cluster
Host System Detail	VMWare > Proactive Monitoring > Entity Views > Host System Detail	Warning panel (Visible only when Host is in Warning or Critical state)
		Drop Down used for Timechart
		Timechart
Virtual Machine Detail	VMWare > Proactive Monitoring > Entity Views > Virtual Machine Detail	Drop Down used for Timechart
		Timechart
Virtual Machine Snapshots	VMWare > Performance and Capacity Planning > Virtual Machine Snapshots	Snapshots present on VM
		Displays timechart when clicked on any row of table - Snapshots present on VM
		Timechart-Snapshot space used on disk
		Timechart- Number of Snapshots on datastore
Proactive Monitoring	VMWare > Proactive Monitoring > Proactive Monitoring	On chart pin board Graph
		vCenter Detail Chart
		Cluster Detail
		Host Chart
		VM Chart

Reports using only summarized data

Panel/Chart	Path
Average CPU Utilization	VMWare > Knowledge Objects > Reports > Average CPU Utilization
Average Memory Provisioning	VMWare > Knowledge Objects > Reports > Average Memory Provisioning
Average VM CPU Ready	VMWare > Knowledge Objects > Reports > Average VM CPU Ready
Snapshots older than 14 days	VMWare > Knowledge Objects > Reports > Snapshots older than 14 days
Max Disk Latency per VM	VMWare > Knowledge Objects > Reports > Max Disk Latency per VM
Critical CPU Ready	VMWare > Knowledge Objects > Reports > Critical CPU Ready
Critical CPU Usage	VMWare > Knowledge Objects > Reports > Critical CPU Usage
Memory Utilization by Cluster	VMWare > Knowledge Objects > Reports > Memory Utilization by Cluster
Critical Disk Usage	VMWare > Knowledge Objects > Reports > Critical Disk Usage
Top Hosts with Ballooning	VMWare > Knowledge Objects > Reports > Top Hosts with Ballooning
CPU Usage by Cluster	VMWare > Knowledge Objects > Reports > CPU Usage by Cluster
CPU Usage by VC	VMWare > Knowledge Objects > Reports > CPU Usage by VC
CPU Ready by VC	VMWare > Knowledge Objects > Reports > CPU Ready by VC
Memory Usage by VC	VMWare > Knowledge Objects > Reports > Memory Usage by VC
Memory Ballooning by VC	VMWare > Knowledge Objects > Reports > Memory Ballooning by VC
Memory Swapped by VC	VMWare > Knowledge Objects > Reports > Memory Swapped by VC
Virtual Machine - Count by Disk Usage Severity	VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Disk Usage Severity
Virtual Machine - Count by Memory Usage Severity	VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Memory Usage Severity

To identify whether your data collection has been accelerated:

1. Navigate to the bottom of any chart panel, and click on the search icon to open a search string in a search view.
2. Look for "summariesonly=true", and replace it with "summariesonly=false", then run the search. If your timechart begins to populate, then your data collection has been accellerated.
3. Navigate to your data model settings and check progress of your data model acceleration.