Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure alerts in savedsearches.conf

You can create and configure alerts in savedsearches.conf. Before configuring an alert with savedsearches.conf, review the following topics in the Admin Manual.


Create or edit savedsearches.conf in the local directory:

$SPLUNK_HOME/etc/system/local/

For apps, create or edit savedsearches.conf in the custom application directory:

$SPLUNK_HOME/etc/apps/

These are the basic steps for defining alerts in savedsearches.conf:

  1. Create and save a report.
    You can set up an alert when you create the report or later add the alert configuration to savedsearches.conf.
  2. Schedule the report.
  3. Define the alert's triggering conditions.
  4. Configure alert actions.
    If you configure an email notification for the alert, configure the email notification settings in Settings. See Configure email notification settings.

Create a report

Create a report, either by saving a search or pivot as a report or by configuring a new stanza in savedsearches.conf. You also create a report when you save a search as an alert. See Create alerts from Splunk Web and Create and edit reports..

The savedsearches.conf file contains a stanza for each saved report. Attributes that define an alert for the saved report appear in this stanza. The following example shows the stanza for a saved search. Within the stanza are attributes defining the alert for this search.

[Too Many Errors Today]
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.to = Splunk250@example.com
action.email.useNSSubject = 1

alert.suppress = 0
alert.track = 0

counttype = number of events
quantity = 5
relation = greater than

# run every day at 14:00
cron_schedule = 0 14 * * *

#search for results in the last day
dispatch.earliest_time = -1d
dispatch.latest_time = now

display.events.fields = ["host","source","sourcetype","latitude"]
display.page.search.mode = verbose
display.visualizations.charting.chart = area
display.visualizations.type = mapping

enableSched = 1

request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = index=_internal " error " NOT debug source=*splunkd.log* earliest=-7d latest=now
disabled = 1

Schedule the report

You can schedule a report by editing savesearches.conf or by configuring the report in Splunk Web. This topic discusses editing savesearches.conf. See Schedule a report in the Reporting Manual for information on how to schedule a report from Splunk Web.

Schedule a report in savesearches.conf by adding the following attributes to the stanza for the report.

Attribute Type
Default
Description
enableSched string
false
Enable scheduling for the report.
cron_schedule text
The cron scheduler to run the report.

The following cron schedule runs the search for the report every 5 minutes:

*/5 * * * *

The following cron schedule specifies real-time for a per-result search or a rolling window.

* * * * *

See Cron notation for details.

dispatch.earliest
dispatch.latest
time modifier
Set the time window for the search.

For real-time searches:

  • Per event, use: rt, for example
    dispatch.earliest_time = rt
    dispatch.latest_time = rt
  • Rolling-window alerts use: rt-[#][unit], for example
    dispatch.earliest_time = rt-30m
    dispatch.latest_time = rt-0m

See Specify time modifiers in your search for more information.

max_concurrent integer
1
The maximum number of instances of the report that can run concurrently.

Configure basic and advanced alert conditions in savedsearches.conf

Two categories of conditions cn trigger an alert. You can configure both of these type of alerts in savedsearches.conf.

  • Basic conditional alerts
    Trigger alerts when the results of the search exceed the threshold for the number of events, sources, or hosts.
  • Advanced conditional alerts
    Trigger alerts based on the results of a conditional search that is evaluated against the results of the scheduled report. If the conditional search returns one or more events, the event triggers.

Configure a basic conditional alert

To configure a basic conditional alert in savedsearches.conf, use a combination of the following attributes:

Attribute Type
Default
Description
counttype text
Set the type of count for alerting.

Possible values:

  • always
    Default value for counttype. Triggers the alert each time the scheduled report runs. Use this value for per-result alerts. Per-result alerts are not conditional.
  • number of events
  • number of hosts
  • number of sources
  • custom
    Configure an advanced conditional alert.
relation string
Comparison factor between counttype and quantity.

Possible values:

  • greater than
  • less than
  • equal to
  • drops by
  • rises by
quantity integer
Numeric value that triggers the alert. Use with counttype and quantity.

For example, to trigger an alert if the results of a scheduled report rise by 25 between runs of the report, do the following:

counttype = number of events
relation = rises by
quantity = 25

The exception to using these settings together is to trigger an alert each time the scheduled report runs. In this case, use only the counttype attribute:

counttype = always

For more information, see Set up triggering conditions for a scheduled alert.

Configure an advanced conditional alert

To configure an advanced conditional alert in savedsearches.conf, use the following attribute.

Attribute Type
Default
Description
alert_condition string
A custom search string to trigger the alert.

The search string is a secondary search of the artifacts of the report job that determines whether to trigger an alert. The alert triggers when the secondary search yields a non-empty search result list.

If you specify alert_condition, do not use the other attributes for a basic conditional alert, counttype, relation, or quantity.

For example:

alert_condition = [search string]

For more information, see Set up triggering conditions for a scheduled alert.

Configure alert actions

Use the action attribute in savedsearches.conf to configure the following alert actions. Each alert action contains configurable parameters.


Parameter Description
action.email Send an email notification.
action.rss Write to an RSS feed.
action.script Run a script.

Global defaults for alert actions are in alert_actions.conf. You can override the defaults in savedsearches.conf.

Use the following syntax to enable or disable an action:

 
action.<action_name> = [0 | 1] # 0 disables, 1 enables the action

Use the following syntax to configure a parameter for an action:

 
action.<action_name>.<parameter> = <value>

Global defaults for all alert actions are configured in alert_actions.conf. You can override the defaults for a saved report in savedsearches.conf.

action.email

The action.email action sends email notifications when an alert triggers. The following example shows configuration parameters for action.email:

. . .
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.reportServerEnabled = 0
action.email.to = Splunk250@example.com
action.email.useNSSubject = 1
. . .
Parameter Type
Default
Description
action.email.to email list
Comma-delimited list of email addresses to notify.

You cannot define a default value for this in alert actions.conf.

action.email.from text
splunk
The from email address for the email notification.
action.email.subject text
Splunk Alert: $name$
The subject of the email notification.
action.email.sendresults boolean
false
Include search results in the email. The can be attached or included in the body of the email. See the action.email.inline parameter. Results include only the results from the base search. It does not include results from secondary conditional searches.
action.email.inline email list
Include results of the base search in the body of the email notification.
action.email.server text
localhost
The address of the SMTP server that sends the alert emails.
email.preprocess_results search string
empty string
Search string to preprocess results before sending the email notification. Use this parameter to filter unwanted fields.


action.rss

You can configure an alert to enable an RSS feed. When the alert triggers, the alert generates notification to the RSS feed. An alert must trigger at least once to generate the RSS feed. See Create an RSS feed for details.

Use action.rss to configure an RSS feed.

Attribute Type
Default
Description
action.rss boolean
0
Configures an RSS feed.

action.script

You can configure an alert to run a shell script. When the alert triggers the shell script runs.

Use action.script to configure an alert to run a script. See Configure scripted alerts for details.

Attribute Type
Default
Description
action.script boolean
0
Configure an alert to run a script when the alert triggers.
action.script.filename text
The filename of the shell script to run when the alert triggers.

Place the script in the following directory:

$SPLUNK_HOME/bin/scripts/
PREVIOUS
Triggered alerts
  NEXT
Configure a script for an alert action

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters