Splunk® Enterprise

Monitoring Splunk Enterprise

Splunk Enterprise version 7.2 is no longer supported as of April 30, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.

Enable and configure platform alerts

Platform alerts are saved searches included in the monitoring console. Platform alerts notify Splunk Enterprise administrators of conditions that might compromise their deployment environment. When an alert is triggered, the Monitoring Console Overview page displays a notification. You can also view the alert and its results by going to Overview > Alerts > Managed triggered alerts.

The included platform alerts get their data from REST endpoints. Platform alerts are disabled by default.

Enable platform alerts

Prerequisite

Configure your Monitoring Console. See Single instance Monitoring Console setup steps or Multi-instance Monitoring Console setup steps depending on your deployment type.

  1. From the monitoring console Overview, click Triggered Alerts > Enable or Disable.
  2. Click the Enabled check box next to the alerts that you want to enable.

You can also set an alert action, such as an email notification.

Configure platform alerts and set alert actions

You can view and configure the default settings and parameters for platform alerts, including the following:

  • Alert thresholds, if applicable
  • Alert schedule
  • Suppression time
  • Alert actions (such as sending an email or starting a custom script)

To change an alert threshold, perform the following steps:

  1. From the Monitoring Console, click Overview > Alerts > Enable or Disable.
  2. Find the alert you want to configure and click Edit.
  3. Edit the threshold field to your desired value.
  4. Click Save.

To view and edit advanced settings like alert schedule, trigger conditions, and alert actions, perform the following steps:

  1. From the Monitoring Console, click Overview > Alerts > Enable or Disable.
  2. Find the alert you want to configure and click Advanced edit.
  3. Modify the settings, if desired.
  4. Click Save.

If you enable email notifications, make sure that you have defined a valid mail host in Settings > Server settings > Email settings.

For guidance on alert actions, see Set up alert actions in the Alerting Manual.

You can also view the complete list of default parameters for platform alerts in $SPLUNK_HOME/etc/apps/splunk_monitoring_console/default/savedsearches.conf. If you choose to edit configuration files directly, put the new configurations in a local directory instead of the default.

Never edit configuration files in the default directory.

Default platform alerts in the monitoring console

The following platform alerts are available by default in the monitoring console. To monitor your deployment with platform alerts, enable the individual alerts that you want.

Alert name Description For more information
Abnormal State of Indexer Processor Fires when one or more of your indexers reports an abnormal state. This abnormal state can be either throttled or stopped. For details on which indexer is in which abnormal state, and to begin investigating causes, see the monitoring console Indexing Performance: Deployment dashboard's Indexing Performance by Instance panel. For information about the dashboard, see Indexing performance dashboards in this manual.
Critical System Physical Memory Usage Fires when one or more instances exceeds 90% memory usage (by any process, Splunk software or otherwise). On most Linux distributions, this alert can trigger if the OS is engaged in buffers and filesystem caching activities. The OS releases this memory if other processes need it, so it does not always indicate a serious problem. For details on instance memory usage, navigate to the monitoring console Resource Usage: Deployment dashboard. For information about the dashboard, see Resource usage dashboards in this manual.
Expired and Soon To Expire Licenses Fires when you have licenses that have expired or will expire within two weeks. For information about your licenses and license usage, click Licensing in the monitoring console.
Missing forwarders Fires when one or more forwarders are missing. See the forwarders dashboards in the monitoring console.
Near Critical Disk Usage Fires when you have used 80% of your disk capacity. For more information about your disk usage, navigate to the three monitoring console Resource Usage dashboards and read Resource usage dashboards in this manual.
Saturated Event-Processing Queues Fires when one or more of your indexer queues reports a fill percentage, averaged over the last 15 minutes, of 90% or more. This alert can inform you of potential indexing latency. For more details about your indexer queues, navigate to the two monitoring console Indexing Performance dashboards and read Indexing performance dashboards in this manual.
Search Peer Not Responding Fires when any of your search peers (indexers) is unreachable. For the status of all your instances, see the monitoring console Instances view.
Total License Usage Near Daily Quota Fires when you have used 90% of your total daily license quota. For more information about your license usage, click Licensing in the monitoring console.

Longevity of platform alert search artifacts

In savedsearches.conf, the dispatch.ttl setting dictates that the searches from platform alerts keep search artifacts for four hours. But if an alert is triggered, its search artifact stays for seven days. This means that the link sent in an email to inspect the search results of a triggered alert expires in seven days (by default).


Next step

To set up health checks, see Access and customize health check. This step is optional.

Last modified on 02 August, 2023
Configure forwarder monitoring for the Monitoring Console   Access and customize health check

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters