Splunk® Enterprise

Troubleshooting Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.3 is no longer supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

About Splunk Enterprise platform instrumentation

Splunk Enterprise platform instrumentation refers to data that Splunk Enterprise logs and uses to populate the _introspection index. It generates data about your Splunk instance and environment and writes that data to log files to aid in reporting on system resource utilization and troubleshooting problems with your Splunk Enterprise deployment. You can also view the latest instrumentation data at REST endpoints.

Platform instrumentation is included in Splunk Enterprise as an add-on, sometimes referred to as the introspection_generator_addon.

Supported platforms

  • Windows
    • x86-64: Server 2008, Server 2008 R2, Server 2012, Server 2016
    • x86-32: Server 2008, Server 2008 R2
  • Linux
    • x86-64: 2.6 or later kernel
    • x86-32: 2.6 or later kernel
  • Solaris
    • x86-64: 10, 11
    • SPARC: 10, 11

What data does Splunk Enterprise record in these introspection log files?

The introspection files contain data about:

  • Operating system resource usage for Splunk Enterprise processes, broken down by process.
  • Operating system resource usage for the entire host (i.e., all system and user processes).
  • Disk object data.
  • KV store performance data.

See "What data gets logged" for more information.

Where is this data written?

Events are written to two log files in $SPLUNK_HOME/var/log/introspection. Non-forwarders tail these log files and place results into the local _introspection index. Forwarders, which have no local indexes, forward these events to indexers.

The two log files are disk_objects.log and resource_usage.log. See "What gets logged" for a breakdown of what data goes into which file.

To find platform instrumentation events, qualify your searches:

  • Find introspection data:

    index=_introspection

  • To find introspection data from a forwarder or another instance in your deployment, qualify your search with the remote host name.

How does this feature affect my Splunk deployment?

If you are upgrading from a Splunk Enterprise version pre-6.1, expect the new log files to use a bit of disk space (an estimated 300 MB). The _introspection index's disk usage, on the other hand, varies from deployment to deployment.

Each log file has a maximum size of 25 Mb. You can change this limit in log.cfg. You can have up to six instances of each file, according to your log rotation policy. That is, resource_usage.log, resource_usage.log.1, ... resource_usage.log.5, and the same for disk_objects.log. Thus, the introspection log files by default can take up to 300 MB of disk space.

This feature is implemented as an auxiliary low-profile long-running process. This process is where resource usage (RU) introspection data is collected. Collecting disk object (DO) introspection data requires no extra I/O, as it leverages information that other parts of splunkd have already collected and cached.

See the upgrade docs in the Installation Manual for upgrade information.

See "Configure platform instrumentation" for instructions on tuning this feature.

Last modified on 22 October, 2019
PREVIOUS
About access logs
  NEXT
What does platform instrumentation log?

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters