Sample platform instrumentation searches

This topic introduces a few examples of analysis you can perform using Splunk Enterprise platform instrumentation. Read About Splunk Enterprise platform instrumentation for an introduction to the feature.

Aggregate median physical memory usage per search type

Use this search to find the median total physical memory used in MB, per search type (ad hoc, scheduled, report acceleration, data model acceleration, or summary indexing) for one host over the last hour:

index=_introspection host=<hostname> data.search_props.sid=* earliest=-1h | bin _time span=10s|stats latest(data.mem_used) as mem_used by data.search_props.sid, data.search_props.type, _time | stats sum(mem_used) as mem_used by data.search_props.sid, data.search_props.type, _time | timechart median(mem_used) by data.search_props.type

As a stacked column chart, this search produces a visualization that looks like this:

Current disk usage per partition in use by Splunk Enterprise

Use this search to find the latest value of Splunk Enterprise disk usage per partition and instance:

| rest /services/server/status/partitions-space | eval usage = capacity - free | eval pct_usage = round(usage / capacity * 100, 2) | stats first(fs_type) as fs_type first(usage) as usage first(capacity) as capacity first(pct_usage) as pct_usage by mount_point, splunk_server

Median CPU usage for the main splunkd process for one host

Use this search to find the median CPU usage of the main splunkd process for one host over the last hour:

index=_introspection component=PerProcess host=<hostname> data.process=splunkd (data.args="-p * start" OR data.args="service") earliest=-1h | timechart median(data.pct_cpu) as cpu_usage(%)

Fill in "<hostname>" with the "host" metadata field associated with your instance, as recorded in inputs.conf's "host" property. As an area chart, this search produces something like this:

Median search concurrency by search mode for all instances

Use this search to find the median number of searches running at any given time, split by mode (historical, historical batch, real-time, or real-time indexed):

index=_introspection data.search_props.sid=* earliest=-1h | bin _time span=10s|stats dc(data.search_props.sid) as search_count by data.search_props.mode, _time | timechart median(search_count) by data.search_props.mode

Peak splunkweb file descriptor usage over time for one instance

index=_introspection component=PerProcess host="<hostname>" (data.process="python*" data.args="*/mrsparkle/root.py*") OR data.process=splunkweb | timechart max(data.fd_used) as fd_used

Fill in "<hostname>" with the "host" metadata field associated with your instance, as recorded in inputs.conf's "host" property.