Splunk® Enterprise

Monitoring Splunk Enterprise

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.3 is no longer supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Access and customize health check

The monitoring console comes with preconfigured health checks. You can modify existing health checks and create or download new ones.

Use the health check

Find the health check at Monitoring Console > Health Check. Start the health check by clicking Start.

Each health check item runs a separate search. The searches run sequentially. When one search finishes, the next one starts. After all searches have completed, the results are sorted by severity: Error, Warning, Info, Success, or N/A.

Click a severity level at the top of the results to see only results with that severity level. Click a row to see more information, including suggested actions.

To run only some of the checks, filter by tag or category before clicking Start. From the monitoring console, you can run health checks that have been created in any app installed on your monitoring console node. Use the app drop-down list to filter health checks by app context.

Exclude a check

You can disable a specific check to prevent it from running when you click Start:

  1. Click Monitoring Console > Settings > Health Check Items.
  2. Locate the check you wish to disable in the list.
  3. Click Disable.
  4. Reload Monitoring Console > Health Check.

You can also filter the checks by group, app, tag, and category at the top of the page before clicking Start.

Modify an existing check

You can modify an existing check. For example, to modify the warning threshold for the Excessive physical memory usage check from 90% to 80%:

  1. Click Monitoring Console > Settings > Health Check Items.
  2. In the Excessive physical memory usage row, click Edit.
  3. Edit the Search and Description fields.
  4. (Optional) Rename the health check item to reflect your modification.
  5. Click Save.

The modifications are saved to your filesystem in $SPLUNK_HOME/etc/apps/splunk_monitoring_console/local/checklist.conf

Create a new health check

You can add a new health check item as follows:

  1. Click Monitoring Console > Settings > Health Check Items.
  2. Click New Health Check Item.
  3. Fill in the title and ID fields.
  4. (Optional) Choose an app context for this check. The default is monitoring console.
  5. Continue filling in the fields. Be sure to include a severity level in your search (| eval severity_level). Without this, the search returns results as N/A. See About searches for guidance filling in the Search field.
  6. (Optional) For Environments to exclude, select either Standalone or Distributed. Any other value in this field is ignored. See What can the monitoring console do? for information about standalone and distributed modes.
  7. Click Save.

The modifications are saved to your filesystem in $SPLUNK_HOME/etc/apps/<app_name>/local/checklist.conf on *nix or %SPLUNK_HOME%\etc\apps\<app_name>\local\checklist.conf on Windows. If you do not specify an app context, the modifications are saved in the splunk_monitoring_console app directory.

Search results format

In standalone mode, the search string generates the final result. In distributed mode, this search generates one row per instance in the result table.

The search results must be in the following format.

instance metric severity_level
<instance name> <metric number or string> <level number>

Severity level names correspond to values as follows.

Severity level name Severity level value
Error 3
Warning 2
Info 1
Success 0
N/A -1

Add a drilldown to a search or dashboard

You can also include a drilldown to another search or to a dashboard, for example a monitoring console dashboard, in your health check results.

To include a monitoring console dashboard drilldown:

  1. Choose an existing dashboard in the monitoring console that is relevant to the data you want to run a health check on. Choose a dashboard that has a drop-down list to choose an instance or machine.
  2. Inspect the URL using the drop-down list to see which parts of the URL are needed to specify the instance you want. Look for &form.splunk_server=$instance$ toward the end of the URL.
  3. Trim the URL to a URI that starts with /app/ and has a $ delimited variable name that is a column in the search results for your health check item. For example, /app/splunk_monitoring_console/distributed_search_instance?form.splunk_server=$search_head$

To include a search drilldown, find or create a search with a $ delimited variable in it. The variable must exist as a column name in the health check search results. For example, a drilldown of index=_internal $instance$ will work, as long as "instance" is a column name in the health check search.

Most likely, you want a drilldown search of the search you just ran. In that case, replace $rest_scope$ or $hist_scope$ with $instance$, where instance is a column name in the health check search. For example:

`dmc_set_index_internal` host=$instance$ earliest=-60m source=*splunkd.log* (component=AggregatorMiningProcessor OR component= LineBreakingProcessor OR component=DateParserVerbose) (log_level=WARN OR log_level=ERROR)

Proactively alert on health check conditions

Many health check items already have a corresponding platform alert. You can also turn an additional health check into an alert.

This table lists the health check items with corresponding platform alerts:

Health check Corresponding platform alert Condition
Indexing status Abnormal State of Indexer Processor Tests the current status of the indexer processor on indexer instances.
Excessive physical memory usage Critical System Physical Memory Usage Assesses system-wide physical memory usage and raises a warning for those servers where it is >90%.
Expiring or expired licenses Expired and Soon To Expire Licenses Checks for licenses that are expired or will expire within 2 weeks.
Missing forwarders Missing forwarders Checks for forwarders that have not connected to indexers for >15 minutes in the recent past.
Near-critical disk usage Near Critical Disk Usage Checks for 80% of the disk usage of partitions that Splunk Enterprise reads or writes to.
Saturation of event-processing queues Saturated Event-Processing Queues One or more of your indexer queues is reporting a fill percentage, averaged over the last 15 minutes, of 90% or more.
Distributed search health assessment Search Peer Not Responding Checks the status of the search peers (indexers) of each search head.

To create a new alert from a health check when a counterpart does not already exist:

  1. Run the health check.
  2. Click Open in search.
  3. Modify the search with a where clause.
  4. Save it as a new scheduled search with an alert action. For example, email the admin.

Export health check results

You can export the results from a health check item to your local machine to share with others.

To export results from a health check item:

  1. Run the health check.
  2. Click the row with the results you want to export.
  3. In the results table on the right, click Export.
  4. Choose the format of the results (XML, CSV, or JSON). You can also choose a file name and the number of results.
  5. Click Export.
Last modified on 03 October, 2019
PREVIOUS
Enable and configure platform alerts
  NEXT
Indexing: Performance

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters