Upgrade to Splunk App for VMware 3.2.0

These notes show how to upgrade to the Splunk App for VMware 3.2.0. See "Platform and Hardware Requirements" in this manual for supported Splunk Enterprise versions for this release. See "How to upgrade Splunk Enterprise" to upgrade to a new version of Splunk Enterprise.

Download the upgrade to Splunk App for VMware 3.2.0

Download the Splunk App for VMware version 3.2.0 from Splunk Apps to a location in your environment. See "Download Splunk App for VMware" in this manual.

Upgrade the Splunk search head

1. Stop the Distribution Collection Scheduler before you upgrade. You can do this by stopping the Splunk platform on your Splunk search head, or you can stop the Distribution collection scheduler in the Collection configuration page of the app.
2. Take the file splunk_app_vmware-<version>-<build_number>.zip that you downloaded from Splunkbase and put it in a temporary directory. This avoids overriding critical files.

   cp splunk_app_vmware-<version>-<build_number>.zip /tmp

3. Change to the /tmp directory, and unzip the app package.

   cd /tmp

   unzip splunk_app_vmware-<version>-<build_number>.zip

4. Copy the unzipped files and move into your $SPLUNK_HOME/etc/apps folder.

   cp -r etc/apps/* $SPLUNK_HOME/etc/apps/

5. Confirm that the following components exist in $SPLUNK_HOME/etc/apps:

   • SA-Utils/…
   • SA-Hydra/…
   • SA-Threshold/…
   • SA-VMW-HierarchyInventory/…
   • Splunk_TA_vcenter/…
   • Splunk_TA_vmware/…
   • Splunk_TA_esxilogs/…

6. Delete legacy add-ons.
   1. Edit the $SPLUNK_HOME/etc/apps/splunk_for_vmware/local/app.conf file and change the value of is_configured to 0. The app sets the value to 1, by default, to prevent the setup page displaying each time you launch it.
   2. Delete savedsearches.conf from $SPLUNK_HOME/etc/apps/SA-VMW-Performance/default/ if it still exists.
   3. Delete tsidx_retention.conf from $SPLUNK_HOME/etc/apps/SA-VMW-Performance/default/ if it still exists.
   For more information on legacy add-on removal, see the Manually remove legacy add-ons section below.
7. Remove the SA-VMW-Licensecheck folder, if it exists, from the $SPLUNK_HOME/etc/apps folder.
8. Restart your Splunk platform deployment, if you stopped it, or restart the distributed collection scheduler in the Collection Configuration page of the app.

Upgrade the Data Collection Node (DCN)

1. Stop Splunk.

2. Install splunk_forwarder_for_vmware-<version>-<build_number>.zip from the download package and put it in $SPLUNK_HOME.

3. Unzip this file (the data collection node components) from $SPLUNK_HOME. The file automatically unzips into the $SPLUNK_HOME/etc/apps directory.

4. Check that the data collection components SA-Utils, SA-Hydra, Splunk_TA_vmware, and Splunk_TA_esxilogs exist in $SPLUNK_HOME/etc/apps.

5. Delete the file $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local/hydra_job.conf

6. Start Splunk.

Validate the Splunk App for VMware upgrade

Validate that you correctly upgraded the Splunk App for VMware to the latest version and that the app can collect data.

For search head cluster deployments, delete savedsearches.conf and tsidx_retention.conf from shcluster/apps/SA-VMW-Performance/default/ on your deployer before applying the upgrade bundle. The deployer will restart all the search head cluster members after the upgrade is applied. If deployer does not restart the search head cluster members, perform a rolling restart.

1. Log in to the Splunk App for VMware on your search head.
2. When the app displays the Splunk for VMware Setup page, select the Delete all deprecated Add-ons checkbox under Disable/delete old add-ons. The app removes all legacy add-ons from the installation.
3. Under Delete old saved searches, select the Delete all deprecated saved searches checkbox. This removes saved searches of SA-VMW-Performance that are no longer in use. Note: If you have built any custom alerts on your saved searches, do not select the checkbox.
4. Restart Splunk Enterprise after saving configuration.

(Optional) For a clustering indexer environment, SA-Utils should be deleted from master-app folder if you need to do a bundle app upgrade.

Manually remove legacy add-ons

If you launched Splunk App for VMware but did not check Delete all deprecated Add-ons on the setup page, you can manually remove the legacy add-ons from your installation.

1. Stop Splunk Enterprise on your Splunk search head.

2. Delete the hydra_job.conf file in the $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local folder on the Splunk Search head.

3. Remove the SA-VMW-Licensecheck folder from the $SPLUNK_HOME/etc/apps folder on your Splunk search head. Do this for each server upon which you installed the Splunk App for VMware.

4. The below table shows the specific legacy add-ons, located in the $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local folder of the Splunk App for VMware, to delete when upgrading:

Component	2.5.0 or earlier	3.1.0 and 3.1.1
DA-VMW-HierarchyInventory	X
DA-VMW-LogEventTask	X
DA-VMW-Performance	X
SA-VMW-Licensecheck		X

If upgrading from versions 3.0.x or 3.1.2, 3.1.3 or 3.1.4, no applicable legacy add-ons need to be removed.  

5. Restart Splunk Enterprise.

Upgrade from tsidx namespaces to data model acceleration

Version 3.2.0 of the Splunk App for VMware replaces the use of tsidx namespacing with the use of data models and data model acceleration. Existing tsidx files will be not be deleted after the upgrade, and will not be utilized after the upgrade to version 3.2.0 of the Splunk App for VMware.

Previously (in versions 3.1.x and earlier), tsidx scheduling and storage was done on the search head. Starting in version 3.2.0, Data model acceleration is distributed and stored across your indexers. Spreading this task across your indexers instead of on your search head will promote more scalability across your installation.

See About data models to learn more.

Data model acceleration speeds up reporting for the entire set of attributes (fields) that you define in a data model. Data model acceleration creates summaries for the specific set of fields you and your Pivot users want to report on, accelerating the dataset represented by that collection of fields rather than a particular search against that dataset. It may take a few hours to create the data model acceleration depending on the amount of data that is historically indexed. Data that has not been accelerated yet (for example, live streaming data) will still be captured in queries by falling back to normal search for that data. Splunk Enterprise always process all summaries first, so accelerated data arrives faster.

See Accelerate data models to learn more.

Index storage size on your VMware indexes will remain unchanged with the implementation of data model acceleration. Previously (in versions 3.1.x and earlier), the age-based retention policy was 6 years, and the size-based retention policy was unlimited. Even if your raw data was deleted, tsidx data still existed, and performance and inventory charts were populated.

Starting in version 3.2.0, Data model acceleration has a 30-day retention policy, and does not have an established size-based retention policy, and can, by default, take up an unlimited amount of disk space. Data model acceleration is dependent on your raw data. So if your raw data has been deleted, the accelerated data cannot be preserved, and your performance and inventory charts cannot be populated.

Click here to learn how to set size-based retention policies.

In versions 3.1.x and earlier, summarized data (tsidx) used approximately .42 MB per day for each virtual machine. In version 3.2.0, data model acceleration uses approximately .57MB per day for each virtual machine.

Your deployment's Data Model acceleration properties are located in datamodels.conf on your indexers. See "Data model acceleration configuration file reference" to see the datamodels.conf default acceleration configuration.

Map existing tsidx namespaces to data model nodes

If you have built custom alerts, dashboards and/or reports using tsidx namespaces, use the table below to map your existing namespaces to nodes in the data models used by the VMware app.

Namespace	Data model	Node name (not case sensitive)
vmw_perf_cpu_hostsystem	VMWarePerformance	vmperformance.cpu.HostSystemCPU
vmw_perf_cpu_virtualmachine	VMWarePerformance	vmperformance.cpu.virtualmachinecpu
vmw_perf_datastore_virtualmachine	VMWarePerformance	vmperformance.datastore.virtualmachinedatastore
vmw_perf_disk_virtualmachine	VMWarePerformance	vmperformance.disk.virtualmachinedisk
vmw_perf_mem_virtualmachine	VMWarePerformance	vmperformance.memory.virtualmachinememory
vmw_perf_net_virtualmachine	VMWarePerformance	vmperformance.net.virtualmachinenet
vmw_perf_power_virtualmachine	VMWarePerformance	vmperformance.power.virtualmachinepower
vmw_perf_resdisk_virtualmachine	VMWarePerformance	vmperformance.resdisk. virtualmachineresdisk
vmw_perf_sys_virtualmachine	VMWarePerformance	vmperformance.sys.virtualmachinesys
vmw_perf_rescpu_virtualmachine	VMWarePerformance	vmperformance.rescpu.virtualmachinerescpu
vmw_perf_datastore_hostsystem	VMWarePerformance	Vmperformance.datastore.hostsystemdatastore
vmw_perf_disk_hostsystem	VMWarePerformance	vmperformance.disk.HostSystemDisk
vmw_perf_hbr_hostsystem	VMWarePerformance	vmperformance.hbr.hostsystemhbr
vmw_perf_mem_hostsystem	VMWarePerformance	vmperformance.memory.HostSystemMemory
vmw_perf_net_hostsystem	VMWarePerformance	vmperformance.net.hostsystemnet
vmw_perf_power_hostsystem	VMWarePerformance	vmperformance.power.hostsystempower
vmw_perf_resstorageadapter_hostsystem	VMWarePerformance	vmperformance.resstorageadapter.hostsystemresstorageadapter
vmw_perf_storageadapter_hostsystem	VMWarePerformance	vmperformance.storageadapter.hostsystemstorageadapter
vmw_perf_storagepath_hostsystem	VMWarePerformance	vmperformance.storagepath.hostsystemstoragepath
vmw_perf_sys_hostsystem	VMWarePerformance	vmperformance.sys.hostsystemsys
vmw_perf_rescpu_hostsystem	VMWarePerformance	vmperformance.rescpu.hostsystemrescpu
vmw_inv_datastore_virtualmachine	VMwareInventory	Inventory

Manage deprecated namespace retention policies

The $SPLUNK_HOME/etc/apps/SA-Utils/default/tsidx_retention.conf file specifies the default retention policy for all deprecated namespaces.

To set a policy for each of the specific deprecated namespaces, create a $SPLUNK_HOME/etc/apps/<add-on-name>/local/tsidx_retention.conf file in each of the namespaces and configure the settings.

Edit your retention.conf file:

1. Edit the tsidx_retention.conf file to uncomment the deprecated namespaces that you want to use in the app.
2. Change the values associated with those namespaces.
• You can keep the default values for the deprecated namespaces or modify the settings to values that work in your environment.

The app also runs a script that cleans up the tsidx namespaces. How the files are cleaned up depends on the retention policy defined for the namespace in the $SPLUNK_HOME/etc/apps/SA-VMW-Performance/default/tsidx_retention.conf file.

TSIDX cleanup based on the file size cleans all tsidx files, but always leaves one file in that namespace. If you do not want to have one large tsidx file in the namespace, you can change the the maximum allowed size for the tsidx files.

Change the the maximum allowed size for the tsidx files:

1. Open the $SPLUNK_HOME/etc/system/local/limit.conf file.
2. Change the value of the parameter optimize_max_size_mb to specify a size limit for a single tsidx file.
3. Restart Splunk Enterprise.

Remove SA-VMW-Licensecheck

Splunk App for VMware 3.1.2 and earlier includes the SA-VMW-Licensecheck package. Refer to the Installation Guide for instructions on how to remove this package to avoid benign licensing error messages.

Charts that only use summarized data

Versions 3.2.0 and above of the Splunk App for VMware and versions 2.1.0 and above of the Splunk App for NetApp DATA ONTAP use data model acceleration to gather performance and inventory data. This means that if your data collection is not accelerated, some charts and tables will not be populated.

Use the table below to see which charts and panels use only summarized data.

Dashboards using only summarized data

Dashboard	Path	Chart/Table/Panel
Cluster Detail	VMWare > Proactive Monitoring > Entity Views > Cluster Detail	Warning panel (Visible only when Cluster is in Warning or Critical state)
		Timechart
		Warning panel 1 (Visible only when Cluster is in Warning or Critical state)
Home Proactive Monitoring	VMWare > Home Proactive Monitoring	Virtual Machine Health -all gauges
		Host System Health-all gauges
Performance of Hosts and VMs	VMWare > Performance and Capacity Planning > Performance of Hosts and VMs	Timechart
Capacity Planning (Hosts)	VMWare > Performance and Capacity Planning > Capacity Planning (Hosts)	Table - Host Performance
		Displays timechart when clicked on any row of table
Capacity Planning Cluster	VMWare > Performance and Capacity Planning > Capacity Planning Cluster	Table - Cluster Performance
		Displays timechart when clicked on any row of table
Capacity Forecasting	VMWare > Performance and Capacity Planning > Capacity Forecasting	CPU Usage Prediction Over Time (%)
		Mem Usage Prediction Over Time (%)
		Disk Usage Prediction Over Time (KBs/sec)
Capacity Planning for Clusters - CPU Headroom	VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - CPU Headroom	Capacity statistics for Selected Cluster in the last 24 hours
		Currently used MHz and Total Capacity over time in the last 24 hours
		VM's powered on in Selected Cluster in the last 24 hours
Capacity Planning for Clusters - Memory Headroom	VMWare > Performance and Capacity Planning > Capacity Planning for Clusters - Memory Headroom	Powered on VMs memory usage in the cluster
		Currently used GB and Total Capacity over time
		Capacity statistics for Crest Cluster Cluster
Host System Detail	VMWare > Proactive Monitoring > Entity Views > Host System Detail	Warning panel (Visible only when Host is in Warning or Critical state)
		Drop Down used for Timechart
		Timechart
Virtual Machine Detail	VMWare > Proactive Monitoring > Entity Views > Virtual Machine Detail	Drop Down used for Timechart
		Timechart
Virtual Machine Snapshots	VMWare > Performance and Capacity Planning > Virtual Machine Snapshots	Snapshots present on VM
		Displays timechart when clicked on any row of table - Snapshots present on VM
		Timechart-Snapshot space used on disk
		Timechart- Number of Snapshots on datastore
Proactive Monitoring	VMWare > Proactive Monitoring > Proactive Monitoring	On chart pin board Graph
		vCenter Detail Chart
		Cluster Detail
		Host Chart
		VM Chart

Reports using only summarized data

Panel/Chart	Path
Average CPU Utilization	VMWare > Knowledge Objects > Reports > Average CPU Utilization
Average Memory Provisioning	VMWare > Knowledge Objects > Reports > Average Memory Provisioning
Average VM CPU Ready	VMWare > Knowledge Objects > Reports > Average VM CPU Ready
Snapshots older than 14 days	VMWare > Knowledge Objects > Reports > Snapshots older than 14 days
Max Disk Latency per VM	VMWare > Knowledge Objects > Reports > Max Disk Latency per VM
Critical CPU Ready	VMWare > Knowledge Objects > Reports > Critical CPU Ready
Critical CPU Usage	VMWare > Knowledge Objects > Reports > Critical CPU Usage
Memory Utilization by Cluster	VMWare > Knowledge Objects > Reports > Memory Utilization by Cluster
Critical Disk Usage	VMWare > Knowledge Objects > Reports > Critical Disk Usage
Top Hosts with Ballooning	VMWare > Knowledge Objects > Reports > Top Hosts with Ballooning
CPU Usage by Cluster	VMWare > Knowledge Objects > Reports > CPU Usage by Cluster
CPU Usage by VC	VMWare > Knowledge Objects > Reports > CPU Usage by VC
CPU Ready by VC	VMWare > Knowledge Objects > Reports > CPU Ready by VC
Memory Usage by VC	VMWare > Knowledge Objects > Reports > Memory Usage by VC
Memory Ballooning by VC	VMWare > Knowledge Objects > Reports > Memory Ballooning by VC
Memory Swapped by VC	VMWare > Knowledge Objects > Reports > Memory Swapped by VC
Virtual Machine - Count by Disk Usage Severity	VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Disk Usage Severity
Virtual Machine - Count by Memory Usage Severity	VMWare > Knowledge Objects > Reports > Virtual Machine - Count by Memory Usage Severity

To identify whether your data collection has been accellerated:

1. Navigate to the bottom of any chart panel, and click on the search icon to open a search string in a search view.
2. Look for "summariesonly=true", and replace it with "summariesonly=false", then run the search. If your timechart begins to populate, then your data collection has been accellerated.
3. Navigate to your data model settings and check progress of your data model acceleration.