Configure platform instrumentation

This topic is about log files that are tailed to populate the _introspection index. Read about this feature in "About Splunk Enterprise platform instrumentation."

This topic helps you configure the default logging interval and enable or disable logging.

What is logged, and how frequently

Platform instrumentation is enabled by default on all Splunk Enterprise instances except for universal forwarders.

This table summarizes the default settings:

Instance type	Resource usage	Disk objects: indexes, bucket superdirectories, volumes, search dispatch artifacts	Disk objects: fishbucket, partitions
Universal forwarder	every 600 sec (disabled by default)	N/A (UFs do not have indexes)	every 600 sec
non-UFs	every 10 sec	every 600 sec	every 600 sec

See "What gets logged" for details about what data is logged.

Enable logging on a universal forwarder

The introspection generator add-on is disabled by default on a universal forwarder. To enable: in the forwarder's $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/app.conf, set

[install]
state = enabled

Enable the introspection generator add-on using deployment server

To facilitate the management of collecting introspection logs from Splunk Universal Forwarders, we will use the Splunk Deployment Server to enable the introspection generator add-on.

Prerequisites

The instructions require the use of a deployment server running Splunk Enterprise 6.2 or later. Additionally, you must have command line access to the deployment server host, as the changes cannot be completed using the Forwarder Management interface provided with the deployment server.

The introspection generator add-on is only available on Splunk Enterprise version 6.1 or later. All forwarder instances must be configured as deployment clients to a centralized deployment server.

Configure the introspection generator add-on on the deployment server

1. SSH into the deployment server.
2. Find the Splunk Enterprise installation path on the local machine. The default installation path is: /opt/splunk
3. Create a new folder: $SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon
4. Create a new folder: $SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon/local
5. Create an app.conf file under $SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon/local
6. Edit the app.conf file and enable the add-on by adding:

[install] 
state = enabled 

7. Save the changes. Review the changes to the app.conf file and the path as a validation step.

Review the excludeFromUpdate command

The excludeFromUpdate prevents the deployment server from overwriting the contents of defined folders in an app. For more examples, see the "serverclass.conf" in the Admin Manual.

For this task, we will use excludeFromUpdate to enable the introspection generator add-on, while preventing the deployment server from making any changes to the add-on by blocking it from overwriting the contents in the app/introspection_generator_addon/default and app/introspection_generator_addon/bin folders.

Update the serverclass.conf file, adding the app to a serverclass for deployment

1. Find the primary copy of the serverclass.conf file. The location and contents will vary between deployments, but some common locations are: $SPLUNK_HOME/etc/system/local/, and $SPLUNK_HOME/etc/apps/*/local. To use btool to find all serverclass.conf files referenced on the deployment server, run: ./splunk btool --debug serverclass list and review the output.

2. Create a new app definition for deploying the changes to the introspection generator add-on. This task is dependent upon the local environment and how the Splunk administrator has chosen to assign and manage apps deployed to forwarders. Many deployments use one serverclass definition to deploy and manage the most common apps for forwarders. For the purposes of this procedure, all universal forwarders are included under one encompassing serverclass named PrimaryForwarders.

3. Define the field excludeFromUpdate command at the app level.

[serverClass:PrimaryForwarders:app:introspection_generator_addon]
excludeFromUpdate = $app_root$/default, $app_root$/bin 
restartSplunkd = True 

4. Save the changes. Review the changes to the serverclass.conf file and the path as a validation step.

Reload the deployment server

1. Utilize your enterprise change control system to file the requirements and changes for this procedure.

2. Run ./splunk reload deploy-server to reload the deployment server and present the changes to all forwarder hosts at their next check-in interval. The command can be scripted to run on the deployment server after working hours.

Validate changes have been successfully deployed

Use the search head to validate the introspection logs are being forwarded. Example: index=_introspection host=<forwarder_host> | stats count by source, component

Populate "Extra" fields

Four fields (in per-process resource usage data) are not populated by default but can be turned on. See "What gets logged" for information.

In server.conf you can tell Splunk Enterprise to acquire the "Extra" fields by setting acquireExtra_i_data to true. For example:

[introspection:generator:disk_objects]
disabled = false
acquireExtra_i_data = true
collectionPeriodInSecs = 600

Increase the polling period

Why might you want to increase the polling period?

Search processes are polled every 10 seconds (600 seconds on a universal forwarder) by a low-profile process. For healthy Splunk Enterprise deployments, we do not expect this to cause any performance problems. But on a deployment that is already prone to performance problems such as a slow pooled search head environment, there might be some performance implications.

Configure by collection type

In server.conf you can increase the polling period by collection type (that is, resource usage data or disk object data).

The default settings (for anything other than a universal forwarder) are:

[introspection:generator:disk_objects]
disabled = false
acquireExtra_i_data = false
collectionPeriodInSecs = 600

[introspection:generator:resource_usage]
disabled = false
acquireExtra_i_data = false
collectionPeriodInSecs = 10

On a universal forwarder, the default resource usage collection period is 600 seconds.

Disable logging

It is possible to disable introspection logging, although in most cases, it's preferable to merely increase the polling interval.

Turn off all introspection logging

You can turn off all introspection collection (and subsequent logging) by disabling the Introspection Generator Add-On.

In the $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/app.conf file, set

[install]
state = disabled

Turn off introspection logging at the component level

In server.conf you can disable, enable, and configure collection by collection type. That is, resource usage data or disk object data.

The default settings are:

[introspection:generator:disk_objects]
disabled = false
acquireExtra_i_data = false
collectionPeriodInSecs = 600

[introspection:generator:resource_usage]
disabled = false
acquireExtra_i_data = false
collectionPeriodInSecs = 10

Run resource usage logging from the command line

If you've disabled this logging on your instance, you can still invoke the CLI command. To invoke, at the command line:

$ splunkd instrument-resource-usage [--debug] [--once] [--extra]

where the flags mean:

--debug: Set logging level to DEBUG (this can also be done via log-cmdline.cfg)

--once: Emit one set of introspection data, and then quit

--extra: This has the same effect as setting acquireExtra_i_data to true in the server.conf [introspection:generator:resource_usage] stanza. See "What gets logged" for which fields are not logged by default and require this flag.

Change the location of the _introspection index

In indexes.conf you can specify the _introspection index. The default location is in $SPLUNK_DB:

[_introspection]
homePath   = $SPLUNK_DB/_introspection/db
coldPath   = $SPLUNK_DB/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600