Splunk® App for VMware (Legacy)

Configuration Guide

On August 31, 2022, the Splunk App for VMware will reach its end of life. After this date, Splunk will no longer maintain or develop this product. The functionality in this app is migrating to a content pack in Data Integrations. Learn about the Content Pack for VMware Dashboards and Reports.
This documentation does not apply to the most recent version of Splunk® App for VMware (Legacy). For documentation on the most recent version, go to the latest release.

Add, edit, and delete threshold settings

Manage threshold settings

The Configured Thresholds table lists the metrics that Splunk App for VMware collects that have default threshold values. All metrics listed on this page can be found in the VMware PerformanceManager or VMware VirtualMachineQuickStats unless it is a Splunk-defined metric.

Select Settings > Threshold Configuration to view this dashboard.

How to use thresholds

On this dashboard you can:

  • Add a new metric. Select new to add a row the existing metrics table. Set the field values with those of the new metric for which you want to collect data.
  • Enable or disable the metric, and select the entity type for the metric. Give it a name, a perf type and set warning and critical threshold values for it. Add a description. Click Save to save the metric to the server.
  • Delete a metric or group of metrics. Select the check box associated with the metric(s), then click Delete Selected.
  • Edit settings for a metric that exists. Click on the row corresponding to the metric, then toggle the Enable/Disable setting, edit the value fields for warning threshold and critical threshold values. Click Save.

When you enable or disable a metric data associated with that metric is collected by the app. You can:

  • Enable or disable a group of metrics: select the check box associated with each metric you want to enable, then click Enable Selected.
  • Enable or disable a single metric: Select the Enabled/Disabled toggle for the metric, then click Save.

Performance metrics reference table

The following performance metrics have default thresholds defined in Splunk App for VMware.

cpu

PerfType Metric name Entity Threshold Value Description
cpu p_average_cpu_usage_percent vm Critical > 90%
Warning > 75%
Virtual machine's average usage in percent.
host Critical > 90%,
Warning > 75%
Average usage of the host's cpu in percent.
p_summation_cpu_ready_millisecond vm Critical >2000
Warning >1000
Virtual machine's state waiting for cpu time measured in milliseconds.
host Critical > 2000
Warning >1000
Amount of time in milliseconds the host waited for cpu cycles.
p_average_cpu_demand_megaHertz vm Critical < 0
Warning < 0
The amount of cpu resources that a virtual machine would use if there was no cpu limit and no contention for cpu.Less than 0 indicates that the VM does not demand any CPU.
host Critical < 0
Warning < 0
The aggregate amount of cpu resources that all virtual machines would use if there was no cpu limit and no contention for cpu or cpu limit. Less than 0 indicates that none of the VMs on the host demand any CPU.
p_average_cpu_usagemhz_megaHertz vm Critical < 0
Warning < 0
The CPU usage, measure in megahertz. This is the amount of actively used vCPU. This is the hypervisor's view of the CPU usage, not the guest OSes version of the same metric. Less than 0 indicates the VM is using any CPU.
host Critical < 0
Warning < 0
This is the CPU usage measured in megahertz. This is the aggregate of CPU usage across all VMs on a host. Less than 0 indicates that none of the VMs on the host require CPU usage


mem

PerfType metric name Entity Threshold Value Description
mem p_average_mem_usage_percent vm Critical >= 75%
Warning >= 90%
Virtual machine's average usage in percent.
host Critical >= 75%
Warning
Average usage of the host's cpu in percent.
p_average_mem_active_kiloBytes vm Critical >95
Warning >75
A virtual machine's that is actively in use.
host Critical > 95
Warning >75
Average amount of all memory in active state by all virtual machines and the vpxd services.
p_average_mem_consumed_kiloBytes vm Critical >95
Warning >75
Virtual machine's memory - memory saved by memory sharing.
host Critical > 95
Warning > 75
Average amount of memory being consumed by the host. This includes all virtual machines and the overhead of the vmkernal.
p_average_mem_overhead_kiloBytes vm Critical > 95
Warning > 75
Memory used by vmware to actually power the virtual machine.
host Critical > 95
Warning> 75
The average overhead of all virtual machines and the overhead of the vSphere.
p_average_mem_granted_kiloBytes vm Critical > 95
Warning > 75
Physical memory that is mapped to the virtual machine. Does not include overhead memory.
host Critical > 95
Warning > 75
Average memory granted to all virtual machines and vSphere.
p_average_mem_vmmemctl_kiloBytes vm Critical > 10
Warning > 2
Amount of physical memory that is being reclaimed by the host through vmware's ballooning driver. Frequent ballooning is a sign of a host in stress.
host Critical > 10
Warning > 2
The sum of all vmmemctl values for all powered-on virtual machines. This value may be greater then the ballloon value of the host which is a sign of the kernel trying to have more virtual machines to release memory.
p_average_mem_swapin_kiloBytes vm Critical > 10
Warning > 0
Memory that's being read by the virtual machine from the hosts swap file. Any amount of swapping is a sign of a host in stress.
host Critical > 10
Warning > 0
Combined sum of all the swap-in values for all powered-on virtual machines.
p_average_mem_swapout_kiloBytes vm Critical >10
Warning > 0
The amount of memory the virtual machine has had to write to a swap file.
host Critical > 10
Warning > 0
Combined sum of all the swap-off values for all powered-on virtual machines.
p_average_mem_llSwaped_kiloBytes vm Critical > 5000
Warning > 0
Amount of memory from a virtual machine that has been swapped by the host. This is a host swapping and is always a sign of the host being in stress. Any time this threshold is triggered, the host has no memory, and can not reclaim it from the ballooning driver.
p_average_mem_llSwapUsed_kiloBytes host Critical >= 5000
Warning >= 0
Amount of memory from all virtual machine that has been swapped by the host. This is a host swapping and is always a sign of the host being in stress. Any time this threshold is triggered, the host has no memory, and can not reclaim it from the ballooning driver.


net

PerfType Splunk metric name Entity Threshold Value Description
net p_average_net_received_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Average kilobytes read across the virtual machine's virtual nic.
host Critical > 95%
Warning > 75%
Average amount of data in kilobytes received across the host's physical adapter.
p_average_net_transmitted_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Average kilobytes broadcasted across the virtual machine's virtual nic.
host Critical > 95%
Warning > 75%
Average amount of data in kilobytes broadcasted across the host's physical adapter.
p_average_net_usage_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Combined broadcast and received rates across all virtual NIC instances.
host Critical > 95%
Warning > 75%
Combined broadcast and received rates across all physical NIC instances.

disk

PerfType metric name Entity Threshold Value Description
disk p_average_disk_read_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Average read rate in kilobytes per second to the virtual disks attached.
p_average_disk_numberReadAveraged_number host Critical > 95%
Warning > 75%
Average kilobytes read from each LUN on the host.
p_average_disk_write_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Average write rate in kilobytes per second to the virtual disks attached.
p_average_disk_numberWriteAveraged_number host Critical > 95%
Warning > 75%
Average kilobytes written to each LUN on the host.
p_average_disk_usage_kiloBytesPerSecond vm Critical > 95%
Warning > 75%
Average I/O rate to the virtual disk.
host Critical > 95%
Warning > 75%
Average aggregated disk I/O for all virtual machines running on the host.
p_summation_disk_numberWrite_number vm Critical > 95%
Warning > 75%
Number of times the virtual machine wrote to it's virtual disk.
host Critical > 95%
Warning > 75%
Total number of writes to the target LUN.
p_summation_disk_numberRead_number vm Critical > 95%
Warning > 75%
Number of times the virtual machine read from it's virtual disk.
host Critical > 95%
Warning > 75%
Total number of reads from the target LUN.
p_latest_disk_maxTotalLatency_millisecond vm Critical > 30%
Warning > 15%
Time in milliseconds it took to process a SCSI command by the virtual machine.
host Critical > 30%
Warning > 15%
The sum in milliseconds of the kernel requests to the device.
p_average_disk_queueLatency_millisecond vm Critical > 5%
Warning > 1%
Time in milliseconds that a virtual machines request spent in a queue state.
host Critical > 5%
Warning > 1%
The sum in milliseconds a request spent in a queue state.
p_summation_disk_commandsAborted_number vm Critical > 2%
Warning > 0%
Number of commands that were aborted on the virtual machine.
host Critical > 2%
Warning > 0%
Number of commands that were aborted on the host.
p_summation_disk_busResets_number vm Critical > 2%
Warning > 0%
Number of SCSI-bus reset commands that were issued.
host Critical > 2%
Warning > 0%
Number of SCSI-bus reset commands that were issued.


inv

PerfType metric name Entity Threshold Value Description
inv PercentHighCPUVm vm Critical > 75
Warning > 50
This is a Splunk metric. The threshold is implemented on top of VMInvCpuMaxUsg. Used on the home_proactive_monitoring dashboard to to give a warning / critical level of vms that are in a "critical" state. This allows you to color the gauges based on the % of vm's in critical state out of the total number of vms.
PercentHighMemVm vm Critical > 75
Warning > 50
This is a Splunk metric.The threshold is implemented on top of VMInvMemMaxUsg. Used on the home_proactive_monitoring dashboard to give a warning / critical level of vms that are in a "critical" state. This allows you to color the gauges based on the % of vm's in critical state out of the total number of vms.
PercentHighSumRdyVm vm Critical > 75
Warning > 50
This is a Splunk metric.The threshold is implemented on top of SumRdy_ms. Used on the home_proactive_monitoring dashboard to give a warning / critical level of vms that are in a "critical" state. This allows you to color the gauges based on the % of vm's in critical state out of the total number of vms.
VMinvCpuMaxUsg vm Critical > 90
Warning > 75
This is a Splunk metric.This threshold is based on the max cpu that the host can give a vm. It is not the max of the reservations. If the vm is >= 100%, the vm is requesting more cpu then the host can allocate.
VMinvMemMaxUsg vm Critical > 90
Warning > 75
This is a Splunk metric.This is the a threshold that's based on the max mem that the host could give a vm. Not the max of the reservations. If the vm is >= 100%, the vm is requesting more mem then the host can allocate.
PercentHighBalloonHosts Host Critical > 75
Warning > 50
This is a Splunk metric. This threshold is a threshold on top of BalloonedMemory_MB. Used on the home_proactive_monitoring dashboard to give a warning / critical level of hosts that are in a "critical" state. This allows you to color the gauges based on the % of hosts in critical state out of the total number of hosts.
PercentHighSwapHosts Host Critical > 75
Warning > 50
This is a Splunk metric. This threshold is a threshold on top of SwappedMemory_MB. Used on the home_proactive_monitoring dashboard to give a warning / critical level of hosts that are in a "critical" state. This allows you to color the gauges based on the % of hosts in critical state out of the total number of vms.
PercentHighCPUHosts Host Critical > 75
Warning > 50
This is a Splunk metric. This threshold is a threshold on top of AvgUsg_pct. Used on the home_proactive_monitoring dashboard to give a warning / critical level of vms that are in a "critical" state. This allows you to color the gauges based on the % of hosts in critical state out of the total number of hosts.
BalloonedMemory_MB
(balloonedMemory)
Host Critical >= 10
Warning >= 2
This metric belongs to the VMware VirtualMachineQuickStats object type. Pulled from inventory data based on the reported vms that exist on the host at the time of collection. The threshold is based on the total amount of memory in MB that is reclaimed from all of the vms on that host.
SwappedMemory_MB
(swappedMemory)
Host Critical > 5
Warning > 0
This metric belongs to the VMware VirtualMachineQuickStats object type. Pulled from inventory data based on the reported vms that exist on the host at the time of collection. The threshold is based on the total amount of memory in MB that is being swapped from all vms on that host.
RemainingCapacity_GB
Datastore Critical <= 50
Warning <= 100
This is a Splunk metric. Changes state based on the remaining disk space in gigabytes on a datastore.
Overprovisioned_GB Datastore Critical > 95
Warning > 75
This is a Splunk metric. Changes state based on how much space is over-provisioned in gigabytes. Negative numbers are a representation of an under-provisioned datastore.

Complete list of VMware performance metrics

By default, Splunk App for VMware collects the complete set of VMware performance metrics. You can view the complete set of metrics in Splunk App for VMware on the Proactive Monitoring dashboard and on the Performance of Hosts and VMs dashboard.

You can read about the performance metrics in the VMware documentation at http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.PerformanceManager.html. Performance metrics are organized by VMware into the following categories.

Last modified on 22 June, 2016
Change data collection node capabilities and log levels   Enable and disable threshold settings

This documentation applies to the following versions of Splunk® App for VMware (Legacy): 3.3.0


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters