Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Download topic as PDF

Sample platform instrumentation searches

This topic introduces a few examples of analysis you can perform using Splunk Enterprise platform instrumentation. Read About Splunk Enterprise platform instrumentation for an introduction to the feature.

Aggregate median physical memory usage per search type

Use this search to find the median total physical memory used in MB, per search type (ad hoc, scheduled, report acceleration, data model acceleration, or summary indexing) for one host over the last hour:

index=_introspection host=<hostname> data.search_props.sid=* earliest=-1h | bin _time span=10s|stats latest(data.mem_used) as mem_used by data.search_props.sid, data.search_props.type, _time | stats sum(mem_used) as mem_used by data.search_props.sid, data.search_props.type, _time | timechart median(mem_used) by data.search_props.type

As a stacked column chart, this search produces a visualization that looks like this:

Phys mem per search type per host.png

Current disk usage per partition in use by Splunk Enterprise

Use this search to find the latest value of Splunk Enterprise disk usage per partition and instance:

| rest /services/server/status/partitions-space | eval usage = capacity - free | eval pct_usage = round(usage / capacity * 100, 2) | stats first(fs_type) as fs_type first(usage) as usage first(capacity) as capacity first(pct_usage) as pct_usage by mount_point, splunk_server

Disk usage per mount point.png

Median CPU usage for the main splunkd process for one host

Use this search to find the median CPU usage of the main splunkd process for one host over the last hour:

index=_introspection component=PerProcess host=<hostname> data.process=splunkd (data.args="-p * start" OR data.args="service") earliest=-1h | timechart median(data.pct_cpu) as cpu_usage(%)

Fill in "<hostname>" with the "host" metadata field associated with your instance, as recorded in inputs.conf's "host" property. As an area chart, this search produces something like this:

CPU usage splunkd.png

Median search concurrency by search mode for all instances

Use this search to find the median number of searches running at any given time, split by mode (historical, historical batch, real-time, or real-time indexed):

index=_introspection data.search_props.sid=* earliest=-1h | bin _time span=10s|stats dc(data.search_props.sid) as search_count by data.search_props.mode, _time | timechart median(search_count) by data.search_props.mode

Peak splunkweb file descriptor usage over time for one instance

index=_introspection component=PerProcess host="<hostname>" (data.process="python*" data.args="*/mrsparkle/root.py*") OR data.process=splunkweb | timechart max(data.fd_used) as fd_used

Fill in "<hostname>" with the "host" metadata field associated with your instance, as recorded in inputs.conf's "host" property.

Last modified on 07 August, 2019
Configure platform instrumentation
Contact Support

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters