Indexing: Performance

This topic is a reference for the Performance dashboards under the Indexing menu in the Monitoring Console.

For more information about troubleshooting indexing performance problems, see Identify and triage indexing performance problems in the Troubleshooting Manual.

Indexing Performance: Deployment

The Indexing Performance: Deployment dashboard provides an overview of indexing performance across your Splunk Enterprise deployment.

In the Overview of Indexing Performance panel, total indexing rate is summed over all indexers.

In the Instances by Estimated Indexing Rate panel, the indexing rate is estimated because it uses metrics.log, which takes only the top ten results for each type by default. See About metrics.log in the Troubleshooting Manual.

Indexing Performance: Instance

The Indexing Performance: Instance dashboard contains panels about indexing performance on a per-indexer basis. These panels can provide more detailed insight into problems that surface in the deployment-wide dashboard.

The Splunk Enterprise Data Pipeline panel exposes decaying averages for queue sizes. The averages use data over the previous 15 minutes. This panel, along with the historical panel Median Fill Ratio of Data Processing Queues, helps you narrow down sources of indexing latency to a specific queue. Data starts at parsing and travels through the data pipeline to indexing at the end.

Here is an example of the Splunk Enterprise Data Pipeline panel for an instance with unhealthy queues:

In this example, although the parsing and aggregator queues have very high fill ratios, the problem is likely to be with processes in the typing queue. The typing queue is the first one that slows down, and data is backing up into the other two queues while waiting to get into the typing queue.

Indexing Performance: Advanced

The primary purpose of the Indexing Performance: Advanced dashboard is to provide information about pipeline set performance. You can use the dashboard to gain insight into the activity of the pipeline sets and their component pipelines. See Manage pipeline sets for index parallelization in Managing Indexers and Clusters of Indexers.

This dashboard is primarily of use when troubleshooting performance issues in consultation with Splunk Support. Without expert-level knowledge of the underlying processes, it can be difficult to interpret the information sufficiently to determine performance issue remediation.

This dashboard also includes panels with information on a few other advanced performance metrics, such as CPU usage. One of these panels, the Aggregate CPU Seconds Spent per Indexer Processor Activity panel, lets you "Split index service by subtask." The several index services are subtasks related to preparing for and cleaning up after indexing. For more information about the subtask categories, see the metrics.log topic in the Troubleshooting Manual.

Enable Splunk processor CPU profiling panels

The Indexing Performance: Advanced dashboard also includes panels that show CPU time metrics for Splunk event processors in the data pipeline, including Line Breaking, Aggregator, Regex, Metrics, and Metric Schema. You can use these panels to identify which processors and events are causing blocks in the pipeline. To populate these panels with data, you must enable the corresponding CPU profiling setting for each processor in limits.conf, as follows:

1. Edit $SPLUNK_HOME/etc/system/local/limits.conf.
2. Enable the CPU profiling settings for each processor, as shown:

   [default]
   regex_cpu_profiling = true
   agg_cpu_profiling = true
   msp_cpu_profiling = true
   mp_cpu_profiling = true
   lb_cpu_profiling = true
   

   For detailed information on processor CPU profiling settings, see limits.conf.

To view data in the metrics processor panel, you must ingest statsd or collectd data.

Troubleshoot these dashboards

The snapshot panels get data from Splunk REST endpoints for introspection. If snapshot panels lack data, check

• the system requirements for platform instrumentation.
• the pipelinesets setting in server.conf. When pipeline sets are used (that is, if pipelinesets is set to a value greater than 1), some panels of the Monitoring Console indexing performance dashboards will be blank.

The historical panels for these dashboards get data from metrics.log.