About Splunk Enterprise platform instrumentation

Splunk Enterprise platform instrumentation refers to data that Splunk Enterprise logs and uses to populate the _introspection index. It generates data about your Splunk instance and environment and writes that data to log files to aid in reporting on system resource utilization and troubleshooting problems with your Splunk Enterprise deployment. You can also view the latest instrumentation data at REST endpoints.

Platform instrumentation is included in Splunk Enterprise as an add-on, sometimes referred to as the introspection_generator_addon.

Supported platforms

• Windows
  • x86-64: Server 2008, Server 2008 R2, Server 2012, Server 2016
  • x86-32: Server 2008, Server 2008 R2
• Linux
  • x86-64: 2.6 or later kernel
  • x86-32: 2.6 or later kernel
• Solaris
  • x86-64: 10, 11
  • SPARC: 10, 11

What data does Splunk Enterprise record in these introspection log files?

The introspection files contain data about:

• Operating system resource usage for Splunk Enterprise processes, broken down by process.
• Operating system resource usage for the entire host (i.e., all system and user processes).
• Disk object data.
• KV store performance data.

See "What data gets logged" for more information.

Where is this data written?

Events are written to two log files in $SPLUNK_HOME/var/log/introspection. Non-forwarders tail these log files and place results into the local _introspection index. Forwarders, which have no local indexes, forward these events to indexers.

The two log files are disk_objects.log and resource_usage.log. See "What gets logged" for a breakdown of what data goes into which file.

To find platform instrumentation events, qualify your searches:

• Find introspection data:

  index=_introspection

• To find introspection data from a forwarder or another instance in your deployment, qualify your search with the remote host name.

How does this feature affect my Splunk deployment?

If you are upgrading from a Splunk Enterprise version pre-6.1, expect the new log files to use a bit of disk space (an estimated 300 MB). The _introspection index's disk usage, on the other hand, varies from deployment to deployment.

Each log file has a maximum size of 25 Mb. You can change this limit in log.cfg. You can have up to six instances of each file, according to your log rotation policy. That is, resource_usage.log, resource_usage.log.1, ... resource_usage.log.5, and the same for disk_objects.log. Thus, the introspection log files by default can take up to 300 MB of disk space.

This feature is implemented as an auxiliary low-profile long-running process. This process is where resource usage (RU) introspection data is collected. Collecting disk object (DO) introspection data requires no extra I/O, as it leverages information that other parts of splunkd have already collected and cached.

See the upgrade docs in the Installation Manual for upgrade information.

See "Configure platform instrumentation" for instructions on tuning this feature.