Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure alerts in savedsearches.conf

You can create and configure alerts in savedsearches.conf.

Before configuring an alert with savedsearches.conf, you can review the following topics in the Admin Manual.

Configuration file paths

Create or edit savedsearches.conf in the local directory:

$SPLUNK_HOME/etc/system/local/

For apps, create or edit savedsearches.conf in the custom application directory:

$SPLUNK_HOME/etc/apps/

Configure an alert

Here are the steps for defining alerts in savedsearches.conf Steps for defining alerts in Splunk Web are not included here.

  1. Create and save a search.
    You can save a search as an alert or add a new stanza to savedsearches.conf.
  2. Schedule the search.
  3. Define alert triggering.
  4. Configure alert actions.
    If you configure an email notification for the alert, configure the email notification settings in Settings. See Configure email notification settings.

Example savedsearches.conf stanza

The savedsearches.conf file contains a stanza for each saved search. The following example shows the stanza for a saved search. Within the stanza are alert attributes for the search.

[Too Many Errors Today]
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.to = address@example.com
action.email.useNSSubject = 1

alert.suppress = 0
alert.track = 0

counttype = number of events
quantity = 5
relation = greater than

# run every day at 14:00
cron_schedule = 0 14 * * *

#search for results in the last day
dispatch.earliest_time = -1d
dispatch.latest_time = now

display.events.fields = ["host","source","sourcetype","latitude"]
display.page.search.mode = verbose
display.visualizations.charting.chart = area
display.visualizations.type = mapping

enableSched = 1

request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = index=_internal " error " NOT debug source=*splunkd.log* earliest=-7d latest=now
disabled = 1

Schedule the search

Schedule a search in savesearches.conf by adding the following attributes to the stanza.

Attribute Type Default Description
enableSched Boolean false Enable scheduling for the report.
cron_schedule text Search cron schedule.

The following cron schedule runs the search every 5 minutes:

*/5 * * * *

The following cron schedule specifies a real-time search.

* * * * *

See Cron notation for more details.

dispatch.earliest
dispatch.latest
time modifier Set the time window for a real-time alert.
  • For per-event triggering, use: rt, for example
    dispatch.earliest_time = rt
    dispatch.latest_time = rt
  • For rolling time window triggering, use: rt-[#][unit], for example
    dispatch.earliest_time = rt-30m
    dispatch.latest_time = rt-0m

See Specify time modifiers in your search for more information.

max_concurrent integer 1 The maximum number of instances of the search that can run concurrently.

Configure basic and advanced alert conditions in savedsearches.conf

Two categories of conditions can trigger an alert. You can configure both of these type of alerts in savedsearches.conf.

  • Basic conditional alerts
    Trigger alerts when the results of the search exceed the threshold for the number of events, sources, or hosts.
  • Advanced conditional alerts
    Trigger alerts based on the results of a conditional search that is evaluated against the results of the scheduled report. If the conditional search returns one or more events, the event triggers.

Configure a basic conditional alert

To configure a basic conditional alert in savedsearches.conf, use a combination of the following attributes:

Attribute Type
Default
Description
counttype text
Set the type of count for alerting.

Possible values:

  • always
    Default value for counttype. Triggers the alert each time the scheduled report runs. Use this value for per-result alerts. Per-result alerts are not conditional.
  • number of events
  • number of hosts
  • number of sources
  • custom
    Configure an advanced conditional alert.
relation string
Comparison factor between counttype and quantity.

Possible values:

  • greater than
  • less than
  • equal to
  • drops by
  • rises by
quantity integer
Numeric value that triggers the alert. Use with counttype and quantity.

For example, to trigger an alert if the results of a scheduled report rise by 25 between runs of the report, do the following:

counttype = number of events
relation = rises by
quantity = 25

The exception to using these settings together is to trigger an alert each time the scheduled report runs. In this case, use only the counttype attribute:

counttype = always

For more information, see Set up triggering conditions for a scheduled alert.

Configure an advanced conditional alert

To configure an advanced conditional alert in savedsearches.conf, use the following attributes:

Attribute Type
Default
Description
alert_condition string
A custom search string to trigger the alert.

The search string is a secondary search of the artifacts of the report job that determines whether to trigger an alert. The alert triggers when the secondary search yields a non-empty search result list.

If you specify alert_condition, do not use the other attributes for a basic conditional alert, counttype, relation, or quantity.

For example:

alert_condition = [search string]

For more information, see Set up triggering conditions for a scheduled alert.

Configure an email alert action

Global defaults for all alert actions are configured in alert_actions.conf. You can override the defaults for a saved report in savedsearches.conf.

action.email

The action.email action sends email notifications when an alert triggers. The following example shows configuration parameters for action.email:

. . .
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.reportServerEnabled = 0
action.email.to = Splunk250@example.com
action.email.useNSSubject = 1
. . .
Parameter Type
Default
Description
action.email.to email list
Comma-delimited list of email addresses to notify.

You cannot define a default value for this in alert actions.conf.

action.email.from text
splunk
The from email address for the email notification.
action.email.subject text
Splunk Alert: $name$
The subject of the email notification.
action.email.sendresults boolean
false
Include search results in the email. The can be attached or included in the body of the email. See the action.email.inline parameter. Results include only the results from the base search. It does not include results from secondary conditional searches.
action.email.inline email list
Include results of the base search in the body of the email notification.
action.email.server text
localhost
The address of the SMTP server that sends the alert emails.
email.preprocess_results search string
empty string
Search string to preprocess results before sending the email notification. Use this parameter to filter unwanted fields.
PREVIOUS
Alert examples
  NEXT
Send SNMP traps to other systems

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters