Access and customize health check

The monitoring console comes with preconfigured health checks. You can modify existing health checks and create or download new ones.

Use the health check

Find the health check at Monitoring Console > Health Check. Start the health check by clicking Start.

Each health check item runs a separate search. The searches run sequentially. When one search finishes, the next one starts. After all searches have completed, the results are sorted by severity: Error, Warning, Info, Success, or N/A.

Click a severity level at the top of the results to see only results with that severity level. Click a row to see more information, including suggested actions.

To run only some of the checks, filter by tag or category before clicking Start. From the monitoring console, you can run health checks that have been created in any app installed on your monitoring console node. Use the app drop-down list to filter health checks by app context.

Exclude a check

You can disable a specific check to prevent it from running when you click Start:

1. Click Monitoring Console > Settings > Health Check Items.
2. Locate the check you wish to disable in the list.
3. Click Disable.
4. Reload Monitoring Console > Health Check.

You can also filter the checks by group, app, tag, and category at the top of the page before clicking Start.

Modify an existing check

You can modify an existing check. For example, to modify the warning threshold for the Excessive physical memory usage check from 90% to 80%:

1. Click Monitoring Console > Settings > Health Check Items.
2. In the Excessive physical memory usage row, click Edit.
3. Edit the Search and Description fields.
4. (Optional) Rename the health check item to reflect your modification.
5. Click Save.

The modifications are saved to your filesystem in $SPLUNK_HOME/etc/apps/splunk_monitoring_console/local/checklist.conf

Download health check updates

The Health Check page lets you download new health check items provided by the Splunk Health Assistant Add-on on splunkbase.

You can download new health check items as they become available directly through the monitoring console UI, as follows:

1. Click Settings > Monitoring Console > Health Check.
2. Click Update Health Checks. If this button is grayed out, then no health check updates are available.
3. Login to Splunkbase.
4. Click Accept and Continue to install the new health checks.
   This installs the latest version of the splunk_health_assistant_addon in $SPLUNK_HOME/etc/apps and automatically adds the new health checks to the Health Check page. No restart is required.
   
   You can now access your new health checks on the Health Check page in the Monitoring Console.

When health check updates become available, a notification message also appears in Splunk Web.

If you are unable to download health check updates directly through the monitoring console due to firewall restrictions, you can alternatively download the Splunk Health Assistant Add-on directly from splunkbase and install it using the command line. For installation instructions, see Manage app and add-on objects.

Create a new health check

You can add a new health check item as follows:

1. Click Monitoring Console > Settings > Health Check Items.
2. Click New Health Check Item.
3. Fill in the title and ID fields.
4. (Optional) Choose an app context for this check. The default is monitoring console.
5. Continue filling in the fields. Be sure to include a severity level in your search (| eval severity_level). Without this, the search returns results as N/A. See About searches for guidance filling in the Search field.
6. (Optional) For Environments to exclude, select either Standalone or Distributed. Any other value in this field is ignored. See What can the monitoring console do? for information about standalone and distributed modes.
7. Click Save.

The modifications are saved to your filesystem in $SPLUNK_HOME/etc/apps/<app_name>/local/checklist.conf on *nix or %SPLUNK_HOME%\etc\apps\<app_name>\local\checklist.conf on Windows. If you do not specify an app context, the modifications are saved in the splunk_monitoring_console app directory.

Search results format

In standalone mode, the search string generates the final result. In distributed mode, this search generates one row per instance in the result table.

The search results must be in the following format.

instance	metric	severity_level
<instance name>	<metric number or string>	<level number>

Severity level names correspond to values as follows.

Severity level name	Severity level value
Error	3
Warning	2
Info	1
Success	0
N/A	-1

Add a drilldown to a search or dashboard

You can also include a drilldown to another search or to a dashboard, for example a monitoring console dashboard, in your health check results.

To include a monitoring console dashboard drilldown:

1. Choose an existing dashboard in the monitoring console that is relevant to the data you want to run a health check on. Choose a dashboard that has a drop-down list to choose an instance or machine.
2. Inspect the URL using the drop-down list to see which parts of the URL are needed to specify the instance you want. Look for &form.splunk_server=$instance$ toward the end of the URL.
3. Trim the URL to a URI that starts with /app/ and has a $ delimited variable name that is a column in the search results for your health check item. For example, /app/splunk_monitoring_console/distributed_search_instance?form.splunk_server=$search_head$

To include a search drilldown, find or create a search with a $ delimited variable in it. The variable must exist as a column name in the health check search results. For example, a drilldown of index=_internal $instance$ will work, as long as "instance" is a column name in the health check search.

Most likely, you want a drilldown search of the search you just ran. In that case, replace $rest_scope$ or $hist_scope$ with $instance$, where instance is a column name in the health check search. For example:

`dmc_set_index_internal` host=$instance$ earliest=-60m source=*splunkd.log* (component=AggregatorMiningProcessor OR component= LineBreakingProcessor OR component=DateParserVerbose) (log_level=WARN OR log_level=ERROR)

Proactively alert on health check conditions

Many health check items already have a corresponding platform alert. You can also turn an additional health check into an alert.

This table lists the health check items with corresponding platform alerts:

Health check	Corresponding platform alert	Condition
Indexing status	Abnormal State of Indexer Processor	Tests the current status of the indexer processor on indexer instances.
Excessive physical memory usage	Critical System Physical Memory Usage	Assesses system-wide physical memory usage and raises a warning for those servers where it is >90%.
Expiring or expired licenses	Expired and Soon To Expire Licenses	Checks for licenses that are expired or will expire within 2 weeks.
Missing forwarders	Missing forwarders	Checks for forwarders that have not connected to indexers for >15 minutes in the recent past.
Near-critical disk usage	Near Critical Disk Usage	Checks for 80% of the disk usage of partitions that Splunk Enterprise reads or writes to.
Saturation of event-processing queues	Saturated Event-Processing Queues	One or more of your indexer queues is reporting a fill percentage, averaged over the last 15 minutes, of 90% or more.
Distributed search health assessment	Search Peer Not Responding	Checks the status of the search peers (indexers) of each search head.

To create a new alert from a health check when a counterpart does not already exist:

1. Run the health check.
2. Click Open in search.
3. Modify the search with a where clause.
4. Save it as a new scheduled search with an alert action. For example, email the admin.

Export health check results

You can export the results from a health check item to your local machine to share with others.

To export results from a health check item:

1. Run the health check.
2. Click the row with the results you want to export.
3. In the results table on the right, click Export.
4. Choose the format of the results (XML, CSV, or JSON). You can also choose a file name and the number of results.
5. Click Export.