What does platform instrumentation log?

This topic describes the contents of log files that are tailed to populate the _introspection index. For the log files that populate _internal, see What Splunk logs about itself.

These log files comply with the Common Information Model (CIM). See the CIM add-on documentation for more information.

"Extra field" indicates a field that is not logged by default. Read more about configuring polling intervals and enabling this feature on a universal forwarder in Configure platform instrumentation.

resource_usage.log

Per-process resource usage data

Platform instrumentation exposes OS resource usage info for just Splunk software processes, broken down by process. Splunk processes include splunkd, splunkweb, Splunk search processes, splunkd-launched (fsck, splunk-optimize), and modular or scripted inputs launched on behalf of splunkd.

These fields are available:

• in the log file $SPLUNK_HOME/var/log/introspection/resource_usage.log
• in an indexer's _introspection index
• at the endpoint server/status/resource-usage/splunk-processes.

Data available for all Splunk software processes

You can view information about operating system resource utilization, broken down by Splunk process. Four fields here are "extra" fields, not logged by default. Read about populating extra fields in Configure platform instrumentation.

See the list of output fields at system/server/status/resource-usage/splunk-processes in the REST API Reference Manual.

Additional data available only for search processes

Splunk software can log all the above data for search processes (except args). In addition, it logs some additional information about search processes, in a subsection called search_props.

See the list of output fields at system/server/status/resource-usage/splunk-processes in the REST API Reference Manual. The search process fields are embedded within the larger process table, at the search_props entry.

Hostwide resource usage data

You can view host-level, dynamic CPU utilization and paging information.

These fields are available:

• in the log file resource_usage.log
• in an indexer's _introspection index
• at the endpoint server/status/resource-usage/hostwide.

See the list of output fields at system/server/status/resource-usage/hostwide in the REST API Reference Manual.

I/O statistics

Disk input-output usage statistics. The Splunk Enterprise iostats endpoint displays the most recent data. Historical data is logged to resource_usage.log.

Note that the statistics available here are usage statistics, not benchmarks.

See the list of output fields at server/status/resource-usage/iostats in the REST API Reference Manual.

Search infrastructure data

Unlike most data available under server/introspection, the search infrastructure data is logged in metrics.log and audit.log, which is indexed to _internal and _audit, respectively, and available in the file system at $SPLUNK_HOME/var/log/splunk. Read about metrics.log components in "About metrics.log."

server/introspection/search/dispatch

Provides vital statistics for distributed search framework, including details on search peer performance.

disk_objects.log

This disk object data is available in the log file $SPLUNK_HOME/var/log/introspection/disk_objects.log

Additionally, the latest snapshot of these field values is available at endpoints as itemized below.

server/info

Splunk Enterprise server configuration information (static server characteristics; dynamic characteristics go under server/status).

See the list of output fields at system/server/info in the REST API Reference Manual.

data/index-volumes

Lists the Splunk Enterprise volume(s).

See the list of output fields at data/index-volumes in the REST API Reference Manual.

data/index-volumes/{Name}

Characterizes persisted objects at the volume level.

See the list of output fields at index/data/index-volumes/{Name} in the REST API Reference Manual.

data/indexes-extended

Provides information about Splunk Enterprise index buckets.

See the list of output fields at index/data/indexes-extended in the REST API Reference Manual.

data/indexes-extended/{Name}

Provides bucket-level information for the specified index.

See the list of output fields at data/indexes-extended{Name} in the REST API Reference Manual.

server/status/dispatch-artifacts

Accesses search job information.

See the list of output fields at server/status/dispatch-artifacts in the REST API Reference Manual.

server/status/fishbucket

Accesses information about the private BTree database. Gives an idea of fishbucket growth. The fishbucket is a directory, $SPLUNK_DB/fishbucket/splunk_private_db/, that keeps a record about each file input. Most fundamentally, this record keeps track of how far into the file we've read, so that if splunkd is stopped and then restarted, it'll know where in each file input to resume reading.

See the list of output fields at server/status/fishbucket in the REST API Reference Manual

server/status/limits/search-concurrency

Search concurrency limits for a standalone Splunk Enterprise instance.

See the list of output fields at system/server/status/limits/search-concurrency in the REST API Reference Manual.

server/status/partitions-space

Helps track disk usage. These results show only partitions with Splunk disk objects (indexes, volumes, logs, fishbucket, search process artifacts) on them. There is a partitions event for each file system, and each event gives the respective file system type.

A file system (or "volume" in Windows) is a logical concept, identified on UNIX by a number called "device ID." A file system has the property of type (format). For example: ZFS, EXT3.

A partition is a physical concept, simply a chunk of hard drive (or solid state drive). All we know about a partition is its size. A file system can reside on multiple partitions. Splunk Enterprise does not report at the partition level.

See the list of output fields at server/status/partitions-space in the REST API Reference Manual.