Alerting Manual

Configure alerts in savedsearches.conf

You can create and configure alerts in savedsearches.conf.

Before configuring an alert with savedsearches.conf, you can review the following topics in the Admin Manual.

• About configuration files
• savedsearches.conf example.

Configuration file paths

Create or edit savedsearches.conf in the local directory:

$SPLUNK_HOME/etc/system/local/

For apps, create or edit savedsearches.conf in the custom application directory:

$SPLUNK_HOME/etc/apps/

Configure an alert

Here are the steps for defining alerts in savedsearches.conf Steps for defining alerts in Splunk Web are not included here.

1. Create and save a search.
   You can save a search as an alert or add a new stanza to savedsearches.conf.
2. Schedule the search.
3. Define alert triggering.
4. Configure alert actions.
   If you configure an email notification for the alert, configure the email notification settings in Settings. See Configure email notification settings.

Example savedsearches.conf stanza

The savedsearches.conf file contains a stanza for each saved search. The following example shows the stanza for a saved search. Within the stanza are alert attributes for the search.

[Too Many Errors Today]
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.to = address@example.com
action.email.useNSSubject = 1

alert.suppress = 0
alert.track = 0

counttype = number of events
quantity = 5
relation = greater than

# run every day at 14:00
cron_schedule = 0 14 * * *

#search for results in the last day
dispatch.earliest_time = -1d
dispatch.latest_time = now

display.events.fields = ["host","source","sourcetype","latitude"]
display.page.search.mode = verbose
display.visualizations.charting.chart = area
display.visualizations.type = mapping

enableSched = 1

request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = index=_internal " error " NOT debug source=*splunkd.log* earliest=-7d latest=now
disabled = 1

Schedule the search

Schedule a search in savesearches.conf by adding the following attributes to the stanza.

Attribute	Type	Default	Description
enableSched	Boolean	false	Enable scheduling for the report.
cron_schedule	text	–	Search cron schedule. The following cron schedule runs the search every 5 minutes: */5 * * * * The following cron schedule specifies a real-time search. * * * * * See Cron notation for more details.
dispatch.earliest dispatch.latest	time modifier	–	Set the time window for a real-time alert. For per-event triggering, use: rt, for example dispatch.earliest_time = rt dispatch.latest_time = rt For rolling time window triggering, use: rt-[#][unit], for example dispatch.earliest_time = rt-30m dispatch.latest_time = rt-0m See Specify time modifiers in your search for more information.
max_concurrent	integer	1	The maximum number of instances of the search that can run concurrently.

Configure basic and advanced alert conditions in savedsearches.conf

Two categories of conditions can trigger an alert. You can configure both of these type of alerts in savedsearches.conf.

• Basic conditional alerts
  Trigger alerts when the results of the search exceed the threshold for the number of events, sources, or hosts.

• Advanced conditional alerts
  Trigger alerts based on the results of a conditional search that is evaluated against the results of the scheduled report. If the conditional search returns one or more events, the event triggers.

Configure a basic conditional alert

To configure a basic conditional alert in savedsearches.conf, use a combination of the following attributes:

Attribute	Type	Default	Description
counttype	text	–	Set the type of count for alerting. Possible values: always Default value for counttype. Triggers the alert each time the scheduled report runs. Use this value for per-result alerts. Per-result alerts are not conditional. number of events number of hosts number of sources custom Configure an advanced conditional alert.
relation	string	–	Comparison factor between counttype and quantity. Possible values: greater than less than equal to drops by rises by
quantity	integer	–	Numeric value that triggers the alert. Use with counttype and quantity.

For example, to trigger an alert if the results of a scheduled report rise by 25 between runs of the report, do the following:

counttype = number of events
relation = rises by
quantity = 25

The exception to using these settings together is to trigger an alert each time the scheduled report runs. In this case, use only the counttype attribute:

counttype = always

For more information, see Set up triggering conditions for a scheduled alert.

Configure an advanced conditional alert

To configure an advanced conditional alert in savedsearches.conf, use the following attributes:

Attribute	Type	Default	Description
alert_condition	string	–	A custom search string to trigger the alert. The search string is a secondary search of the artifacts of the report job that determines whether to trigger an alert. The alert triggers when the secondary search yields a non-empty search result list. If you specify alert_condition, do not use the other attributes for a basic conditional alert, counttype, relation, or quantity.

For example:

alert_condition = [search string]

For more information, see Set up triggering conditions for a scheduled alert.

Configure an email alert action

Global defaults for all alert actions are configured in alert_actions.conf. You can override the defaults for a saved report in savedsearches.conf.

action.email

The action.email action sends email notifications when an alert triggers. The following example shows configuration parameters for action.email:

. . .
# send an email notification
action.email = 1
action.email.message.alert = The alert condition for '$name$' in the $app$ fired with $job.resultCount$ error events.
action.email.reportServerEnabled = 0
action.email.to = Splunk250@example.com
action.email.useNSSubject = 1
. . .

Parameter	Type	Default	Description
action.email.to	email list	–	Comma-delimited list of email addresses to notify. You cannot define a default value for this in alert actions.conf.
action.email.from	text	splunk	The from email address for the email notification.
action.email.subject	text	Splunk Alert: $name$	The subject of the email notification.
action.email.sendresults	boolean	false	Include search results in the email. The can be attached or included in the body of the email. See the action.email.inline parameter. Results include only the results from the base search. It does not include results from secondary conditional searches.
action.email.inline	email list	–	Include results of the base search in the body of the email notification.
action.email.server	text	localhost	The address of the SMTP server that sends the alert emails.
email.preprocess_results	search string	empty string	Search string to preprocess results before sending the email notification. Use this parameter to filter unwanted fields.