Configure platform instrumentation
This topic is about log files that are tailed to populate the _introspection
index. Read about this feature in "About Splunk Enterprise platform instrumentation."
This topic helps you configure the default logging interval and enable or disable logging.
What is logged, and how frequently
Platform instrumentation is enabled by default on all Splunk Enterprise instances except for universal forwarders.
This table summarizes the default settings:
Instance type | Resource usage | Disk objects: indexes, bucket superdirectories, volumes, search dispatch artifacts | Disk objects: fishbucket, partitions |
---|---|---|---|
Universal forwarder | every 600 sec (disabled by default) | N/A (UFs do not have indexes) | every 600 sec |
non-UFs | every 10 sec | every 600 sec | every 600 sec |
See "What gets logged" for details about what data is logged.
Enable logging on a universal forwarder
The introspection generator add-on is disabled by default on a universal forwarder. To enable:
in the forwarder's $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/app.conf
, set
[install] state = enabled
Enable the introspection generator add-on using deployment server
To facilitate the management of collecting introspection logs from Splunk Universal Forwarders, we will use the Splunk Deployment Server to enable the introspection generator add-on.
Prerequisites
The instructions require the use of a deployment server running Splunk Enterprise 6.2 or later. Additionally, you must have command line access to the deployment server host, as the changes cannot be completed using the Forwarder Management interface provided with the deployment server.
The introspection generator add-on is only available on Splunk Enterprise version 6.1 or later. All forwarder instances must be configured as deployment clients to a centralized deployment server.
Configure the introspection generator add-on on the deployment server
- SSH into the deployment server.
- Find the Splunk Enterprise installation path on the local machine. The default installation path is:
/opt/splunk
- Create a new folder:
$SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon
- Create a new folder:
$SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon/local
- Create an
app.conf
file under$SPLUNK_HOME/etc/deployment-apps/introspection_generator_addon/local
- Edit the
app.conf
file and enable the add-on by adding:
[install] state = enabled
7. Save the changes. Review the changes to the app.conf
file and the path as a validation step.
Review the excludeFromUpdate
command
The excludeFromUpdate
prevents the deployment server from overwriting the contents of defined folders in an app. For more examples, see the "serverclass.conf" in the Admin Manual.
For this task, we will use excludeFromUpdate
to enable the introspection generator add-on, while preventing the deployment server from making any changes to the add-on by blocking it from overwriting the contents in the app/introspection_generator_addon/default
and app/introspection_generator_addon/bin
folders.
Update the serverclass.conf
file, adding the app to a serverclass for deployment
1. Find the primary copy of the serverclass.conf
file. The location and contents will vary between deployments, but some common locations are: $SPLUNK_HOME/etc/system/local/
, and $SPLUNK_HOME/etc/apps/*/local
. To use btool to find all serverclass.conf
files referenced on the deployment server, run: ./splunk btool --debug serverclass list
and review the output.
2. Create a new app definition for deploying the changes to the introspection generator add-on. This task is dependent upon the local environment and how the Splunk administrator has chosen to assign and manage apps deployed to forwarders. Many deployments use one serverclass definition to deploy and manage the most common apps for forwarders. For the purposes of this procedure, all universal forwarders are included under one encompassing serverclass named PrimaryForwarders
.
3. Define the field excludeFromUpdate
command at the app level.
[serverClass:PrimaryForwarders:app:introspection_generator_addon] excludeFromUpdate = $app_root$/default, $app_root$/bin restartSplunkd = True
4. Save the changes. Review the changes to the serverclass.conf
file and the path as a validation step.
Reload the deployment server
1. Utilize your enterprise change control system to file the requirements and changes for this procedure.
2. Run ./splunk reload deploy-server
to reload the deployment server and present the changes to all forwarder hosts at their next check-in interval. The command can be scripted to run on the deployment server after working hours.
Validate changes have been successfully deployed
Use the search head to validate the introspection logs are being forwarded. Example: index=_introspection host=<forwarder_host> | stats count by source, component
Populate "Extra" fields
Four fields (in per-process resource usage data) are not populated by default but can be turned on. See "What gets logged" for information.
In server.conf you can tell Splunk Enterprise to acquire the "Extra" fields by setting acquireExtra_i_data
to true. For example:
[introspection:generator:disk_objects] disabled = false acquireExtra_i_data = true collectionPeriodInSecs = 600
Increase the polling period
Why might you want to increase the polling period?
Search processes are polled every 10 seconds (600 seconds on a universal forwarder) by a low-profile process. For healthy Splunk Enterprise deployments, we do not expect this to cause any performance problems. But on a deployment that is already prone to performance problems such as a slow pooled search head environment, there might be some performance implications.
Configure by collection type
In server.conf you can increase the polling period by collection type (that is, resource usage data or disk object data).
The default settings (for anything other than a universal forwarder) are:
[introspection:generator:disk_objects] disabled = false acquireExtra_i_data = false collectionPeriodInSecs = 600 [introspection:generator:resource_usage] disabled = false acquireExtra_i_data = false collectionPeriodInSecs = 10
On a universal forwarder, the default resource usage collection period is 600 seconds.
Disable logging
It is possible to disable introspection logging, although in most cases, it's preferable to merely increase the polling interval.
Turn off all introspection logging
You can turn off all introspection collection (and subsequent logging) by disabling the Introspection Generator Add-On.
In the $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/app.conf
file, set
[install] state = disabled
Turn off introspection logging at the component level
In server.conf you can disable, enable, and configure collection by collection type. That is, resource usage data or disk object data.
The default settings are:
[introspection:generator:disk_objects] disabled = false acquireExtra_i_data = false collectionPeriodInSecs = 600 [introspection:generator:resource_usage] disabled = false acquireExtra_i_data = false collectionPeriodInSecs = 10
Run resource usage logging from the command line
If you've disabled this logging on your instance, you can still invoke the CLI command. To invoke, at the command line:
$ splunkd instrument-resource-usage [--debug] [--once] [--extra]
where the flags mean:
--debug
: Set logging level to DEBUG (this can also be done via log-cmdline.cfg
)
--once
: Emit one set of introspection data, and then quit
--extra
: This has the same effect as setting acquireExtra_i_data
to true in the server.conf [introspection:generator:resource_usage]
stanza. See "What gets logged" for which fields are not logged by default and require this flag.
Change the location of the _introspection index
In indexes.conf you can specify the _introspection index. The default location is in $SPLUNK_DB
:
[_introspection] homePath = $SPLUNK_DB/_introspection/db coldPath = $SPLUNK_DB/_introspection/colddb thawedPath = $SPLUNK_DB/_introspection/thaweddb maxDataSize = 1024 frozenTimePeriodInSecs = 1209600
What does platform instrumentation log? | Sample platform instrumentation searches |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!