Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure alerts in savedsearches.conf

Configure alerts with savedsearches.conf. Use the $SPLUNK_HOME/etc/system/README/savedsearches.conf.example as an example, or create your own savedsearches.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see "About configuration files" in the Admin Manual.

Follow these steps:

1. Create a report.

2. Schedule the report.

3. Define the alert's triggering conditions (the conditions which, when met, cause the search to be triggered).

4. Configure alert actions (what happens when the search is triggered).

You can set up an alert at the time you create a report, or add the alert configurations to your report stanza later.

Note: You must have email enabled on your Splunk server for alerts to be sent out. Alternately, your Splunk server must be able to contact your email server. Configure email settings in the Splunk Web Settings menu.

Create a report

First, create a report, either by saving a search or pivot as a report (see the topic "Create and edit reports," in the Reporting Manual) or by configuring a new stanza in savedsearches.conf.

Schedule the report

Next, you can schedule your report. This means the report runs on a schedule that you specify. For example, you can arrange to have Splunk run the report every hour, every six hours, or each Monday at midnight.

If you want to schedule your report via Splunk Web, you'll have to access the Edit Schedule dialog through the Reports page. For more information, see the topic "Schedule reports," in the Reporting Manual.

To schedule a report via savesearches.conf, add the following attribute/value pairs to your report stanza:

userid = <integer>

  • UserId of the user who created this report.
    • Splunk needs this information to log who ran the report, and create editing capabilities in Splunk Web.
  • Possible values: Any Splunk user ID.
  • User IDs are found in $SPLUNK_HOME/etc/passwd.
    • Look for the first number on each line, right before the username.
    • For example 2:penelope....

enableSched = < 0 | 1 >

  • Set this to 1 to enable scheduling for the report.
  • Defaults to 0.

cron_sched = <cron string>

  • The cron schedule used to execute the report.
  • For example, */5 * * * * causes the search to execute every five minutes.

Note Cron scheduling lets you use standard cron notation to define your scheduled search interval. In particular, cron can accept this type of notation: 00,20,40 * * * *, which runs the report every hour at hh:00, hh:20, hh:40. Along the same lines, a cron of 03,23,43 * * * * runs the report every hour at hh:03, hh:23, hh:43. Splunk recommends that you schedule your reports so that they're staggered over time. This reduces system load. Running all of them (*/20) every 20 minutes means they would all launch at hh:00 (20, 40) and might slow your system every 20 min.

max_concurrent = <integer>

  • The maximum number of concurrent instances of the report that the scheduler is allowed to run.
  • Defaults to 1.

Configure alert conditions in savedsearches.conf

Next, configure alert conditions for the scheduled report. When Splunk runs a scheduled report, these are the conditions that trigger an alert action (such as an email) when they are met.

Alerts fit into two categories: basic conditional alerts and advanced conditional alerts.

  • Basic conditional alerts trigger alert actions when set thresholds in the number of events, sources, or hosts in your results are exceeded.
  • Advanced conditional alerts are based on the results of a conditional search that is evaluated against the results of the scheduled report. If the conditional search returns one or more events, the event is triggered.

Configure a basic conditional alert

Configure a threshold number of events, sources, or hosts. If the alert conditions are met when the report is run, Splunk notifies you via email or triggers a shell script. You can also set counttype = always if you want the alert action (such as an email) to be triggered each time the scheduled report runs.

counttype = <string>

  • Set the type of count for alerting.
  • Possible values: number of events, number of hosts, number of sources, and always.
  • Used in conjunction with the relation and quantity attributes, except when set to always.
  • Use counttype = always to trigger the alert action each time the scheduled report is run.

relation = <string>

  • How to compare against counttype.
  • Possible values: greater than, less than, equal to, drops by, rises by.

quantity = <integer>

  • Number to compare against the given counttype.

So if you have the following:

counttype = number of events
relation = rises by
quantity = 25

Splunk alerts you if your report results have risen by 25 since the last time the report ran.

For more information about configuring alert actions, see the "Configure alert actions" subtopic, below.

Configure an advanced conditional alert

If you'd rather configure an advanced conditional alert, you use the alert_condition attribute in place of counttype, relation, and quantity.

alert_condition = <string>

  • In <string>, enter a search. Splunk will evaluate this secondary search on the artifacts of the report job to determine whether to trigger an alert action.
  • Alert actions are triggered if this secondary search yields a non-empty search result list.

For an in-depth discussion of a use case for advanced conditional alerting over basic conditional alerting, see the "Set up triggering conditions for a scheduled alert" section of the "Define scheduled alerts" topic in the Alerting Manual. This topic discusses alert setup using Manager, but the underlying principles are the same.

For more information about configuring alert actions, see the following subtopic, "Configure alert actions."

Configure alert actions

You can configure three different kinds of alert actions--actions that happen when alert conditions are met--for your scheduled searches. These alert actions are notification by email, notification by RSS, and the triggering of a shell script.

To enable or disable an alert action for a particular scheduled, alerted search, add the following to the report definition in savedsearches.conf:

action.<action_name> = 0 | 1

  • Indicates whether the alert action is enabled or disabled for a particular report. Set to 0 (disabled) by default.
  • action_name can be email, script, or rss.

Global defaults for all alert actions are configured in alert_actions.conf. You can override these defaults at the individual report level in savedsearches.conf (or via Splunk Manager). If you don't need to override the alert action defaults, all you need to do is indicate which alert actions are enabled for a given scheduled search (see above).

To set a parameter for an alert action, the syntax is as follows:

action.<action_name>.<parameter> = <value>

The parameter options for each <action_name> are defined in the following sections.

Notification by email

Use the email action to have Splunk contact stakeholders when the scheduled report triggers an alert:

action.email = 1

The email action has a number of parameters. Defaults can be set for all of these parameters in alert_actions.conf, with the exception of the action.email.to parameter, which should be set for each scheduled report that uses the email alert action.

action.email.to = <email list>

  • The email addresses to which Splunk will send the email, arranged in a comma-delimited list.
  • This parameter is not set at the alert_actions.conf level. You must define it for every email alert action that you configure.

action.email.from = <email address>

  • The email address that is used as the sender's address.
  • Default is splunk@$LOCALHOST (or whatever is set for from in alert_actions.conf).

action.email.subject = <string>

  • The subject of the alert email.
  • Default is SplunkAlert-<reportname> (or whatever is set for subject in alert_actions.conf).

action.email.sendresults = <bool>

  • Specify whether to include the search results in the email. The results can be attached or included in the body of the email (see the action.email.inline parameter, below).
  • Default is false (or whatever is set for sendresults in alert_actions.conf).
  • Note: When you are using an advanced conditional alert, be aware that only the results of the original search are included with the email. The results of the triggering conditional search are discarded

action.email.inline = <true | false>

  • Specify whether the report results are included in the body of the alert mail.
  • Default is false (or whatever is set for inline in alert_actions.conf).

action.email.mailserver = <string>

  • The address of the MTA server that sends the alert emails.
  • Default is $LOCALHOST (or whatever is set for mailserver in alert_actions.conf).

action.email.preprocess_results = <search-string>

  • An optional search string to preprocess results before emailing them. Usually one would set this up to filter out unwanted internal fields.
  • Default is an empty string (or whatever is set for preprocess_results in alert_actions.conf).

Note: You can also arrange to have .pdf printouts of dashboards delivered by email on a set schedule. For more information, see "Create and deliver dashboard PDFs" in the Splunk Data Visualizations Manual.

There are settings for this feature in alert_actions.conf. For example, you can identify the URL of the PDF report server, and the report paper size and orientation.

The following is an example of what an email alert looks like:

Example alert email.png

Create an RSS feed

Use the rss action to have Splunk alert you via RSS when the scheduled report triggers an alert:

action.rss = 1

Whenever the alert conditions are met for a scheduled report that has Create an RSS feed selected, Splunk sends a notification out to its RSS feed. The feed is located at http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on localhost and uses port 8000, the correct link for the RSS feed would be http://localhost:8000/rss/errors_last15.

You can also access the RSS feed for a scheduled report through the Searches and Reports page in Settings. If a scheduled report has been set up to provide an RSS feed for alerting searches, when you look it up on the Searches and Reports page, you will see a RSS symbol in the RSS feed column:

Saved search RSS.png

You can click on this symbol to go to the RSS feed.

Note: The RSS feed for a scheduled report will not display any results until the report has run on its schedule and the alerting conditions that have been defined for it have been met. If you set the search up to alert each time it's run (by setting Perform actions to always), you'll see reports in the RSS feed after first time the report runs on its schedule.

Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular report job, but they can see the summarization displayed in the RSS feed, which includes the name of the report that was run and the number of results returned by the report job.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
        <title>Alert: errors last15</title>
        <description>Reports Feed for report errors last15</description>
            <title>errors last15</title>
            <description>Alert trigger: errors last15, results.count=123 </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>

Trigger a shell script

Use the script action to have Splunk run a shell script when the scheduled search triggers an alert:

action.script = 1

The script action has a filename parameter which is usually defined at the individual search level, although a default filename can also be set in alert_actions.conf:

action.script.filename = <script filename>

  • The filename of the shell script that you want Splunk to run. The script should live in $SPLUNK_HOME/bin/scripts/.

Example - Basic conditional alert configuration

This example is for a report titled "sudoalert." It runs a search for events containing the term "sudo" on a 12 minute interval. If a scheduled "sudoalert" run results in greater than 10 events, alert actions are triggered that send the results via email and post them to an RSS feed.

search = sudo
counttype = number of events
enableSched = 1
schedule = */12 * * * *
quantity = 10
relation = greater than
action.email = 1
action.email.to = me@work.org 
action.email.from = splunk@work.org
action.email.subject = Sudo Alert!
action.email.mailserver = mail@work.org
action.rss = 1

Enable summary indexing

Summary indexing is an additional kind of alert action that you can configure for any scheduled report. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar reports on a regular basis.

With summary indexing, you define a scheduled report that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the report it saves the results into a summary index that you've designated. You can then report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.

Note: Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

For more information about configuring summary index reports in savedsearches.conf, see "Configure summary indexes" in the Knowledge Manager Manual.

Review triggered alerts
Configure scripted alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters