Configure alerts in savedsearches.conf
Configure alerts with
savedsearches.conf. Use the
$SPLUNK_HOME/etc/system/README/savedsearches.conf.example as an example, or create your own
savedsearches.conf. Edit this file in
$SPLUNK_HOME/etc/system/local/, or your own custom application directory in
$SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see "About configuration files" in the Admin Manual.
Follow these steps:
1. Create a report.
3. Define the alert's triggering conditions (the conditions which, when met, cause the search to be triggered).
4. Configure alert actions (what happens when the search is triggered).
You can set up an alert at the time you create a report, or add the alert configurations to your report stanza later.
Note: You must have email enabled on your Splunk server for alerts to be sent out. Alternately, your Splunk server must be able to contact your email server. Configure email settings in the Splunk Web Settings menu.
Create a report
Schedule the report
Next, you can schedule your report. This means the report runs on a schedule that you specify. For example, you can arrange to have Splunk run the report every hour, every six hours, or each Monday at midnight.
If you want to schedule your report via Splunk Web, you'll have to access the Edit Schedule dialog through the Reports page. For more information, see the topic "Schedule reports," in the Reporting Manual.
To schedule a report via
savesearches.conf, add the following attribute/value pairs to your report stanza:
userid = <integer>
- UserId of the user who created this report.
- Splunk needs this information to log who ran the report, and create editing capabilities in Splunk Web.
- Possible values: Any Splunk user ID.
- User IDs are found in
- Look for the first number on each line, right before the username.
- For example
enableSched = < 0 | 1 >
- Set this to 1 to enable scheduling for the report.
- Defaults to 0.
cron_sched = <cron string>
- The cron schedule used to execute the report.
- For example,
*/5 * * * *causes the search to execute every five minutes.
Note Cron scheduling lets you use standard cron notation to define your scheduled search interval. In particular, cron can accept this type of notation: 00,20,40 * * * *, which runs the report every hour at hh:00, hh:20, hh:40. Along the same lines, a cron of 03,23,43 * * * * runs the report every hour at hh:03, hh:23, hh:43. Splunk recommends that you schedule your reports so that they're staggered over time. This reduces system load. Running all of them (*/20) every 20 minutes means they would all launch at hh:00 (20, 40) and might slow your system every 20 min.
max_concurrent = <integer>
- The maximum number of concurrent instances of the report that the scheduler is allowed to run.
- Defaults to 1.
Configure alert conditions in savedsearches.conf
Next, configure alert conditions for the scheduled report. When Splunk runs a scheduled report, these are the conditions that trigger an alert action (such as an email) when they are met.
Alerts fit into two categories: basic conditional alerts and advanced conditional alerts.
- Basic conditional alerts trigger alert actions when set thresholds in the number of events, sources, or hosts in your results are exceeded.
- Advanced conditional alerts are based on the results of a conditional search that is evaluated against the results of the scheduled report. If the conditional search returns one or more events, the event is triggered.
Configure a basic conditional alert
Configure a threshold number of events, sources, or hosts. If the alert conditions are met when the report is run, Splunk notifies you via email or triggers a shell script. You can also set
counttype = always if you want the alert action (such as an email) to be triggered each time the scheduled report runs.
counttype = <string>
- Set the type of count for alerting.
- Possible values:
number of events,
number of hosts,
number of sources, and
- Used in conjunction with the
quantityattributes, except when set to
counttype = alwaysto trigger the alert action each time the scheduled report is run.
relation = <string>
- How to compare against counttype.
- Possible values: greater than, less than, equal to, drops by, rises by.
quantity = <integer>
- Number to compare against the given counttype.
So if you have the following:
counttype = number of events relation = rises by quantity = 25
Splunk alerts you if your report results have risen by 25 since the last time the report ran.
For more information about configuring alert actions, see the "Configure alert actions" subtopic, below.
Configure an advanced conditional alert
If you'd rather configure an advanced conditional alert, you use the
alert_condition attribute in place of
alert_condition = <string>
<string>, enter a search. Splunk will evaluate this secondary search on the artifacts of the report job to determine whether to trigger an alert action.
- Alert actions are triggered if this secondary search yields a non-empty search result list.
For an in-depth discussion of a use case for advanced conditional alerting over basic conditional alerting, see the "Set up triggering conditions for a scheduled alert" section of the "Define scheduled alerts" topic in the Alerting Manual. This topic discusses alert setup using Manager, but the underlying principles are the same.
For more information about configuring alert actions, see the following subtopic, "Configure alert actions."
Configure alert actions
You can configure three different kinds of alert actions--actions that happen when alert conditions are met--for your scheduled searches. These alert actions are notification by email, notification by RSS, and the triggering of a shell script.
To enable or disable an alert action for a particular scheduled, alerted search, add the following to the report definition in
action.<action_name> = 0 | 1
- Indicates whether the alert action is enabled or disabled for a particular report. Set to
0(disabled) by default.
Global defaults for all alert actions are configured in
alert_actions.conf. You can override these defaults at the individual report level in
savedsearches.conf (or via Splunk Manager). If you don't need to override the alert action defaults, all you need to do is indicate which alert actions are enabled for a given scheduled search (see above).
To set a parameter for an alert action, the syntax is as follows:
action.<action_name>.<parameter> = <value>
The parameter options for each <code><action_name> are defined in the following sections.
Notification by email
action.email = 1
alert_actions.conf, with the exception of the
action.email.to parameter, which should be set for each scheduled report that uses the email alert action.
action.email.to = <email list>
- The email addresses to which Splunk will send the email, arranged in a comma-delimited list.
- This parameter is not set at the
alert_actions.conflevel. You must define it for every email alert action that you configure.
action.email.from = <email address>
- The email address that is used as the sender's address.
- Default is
splunk@$LOCALHOST(or whatever is set for
action.email.subject = <string>
- The subject of the alert email.
- Default is
SplunkAlert-<reportname>(or whatever is set for
action.email.sendresults = <bool>
- Specify whether to include the search results in the email. The results can be attached or included in the body of the email (see the
- Default is
false(or whatever is set for
- Note: When you are using an advanced conditional alert, be aware that only the results of the original search are included with the email. The results of the triggering conditional search are discarded
action.email.inline = <true | false>
- Specify whether the report results are included in the body of the alert mail.
- Default is
false(or whatever is set for
action.email.mailserver = <string>
- The address of the MTA server that sends the alert emails.
- Default is
$LOCALHOST(or whatever is set for
action.email.preprocess_results = <search-string>
- An optional search string to preprocess results before emailing them. Usually one would set this up to filter out unwanted internal fields.
- Default is an empty string (or whatever is set for
Note: You can also arrange to have .pdf printouts of dashboards delivered by email on a set schedule. For more information, see "Create and deliver dashboard PDFs" in the Splunk Data Visualizations Manual.
There are settings for this feature in
alert_actions.conf. For example, you can identify the URL of the PDF report server, and the report paper size and orientation.
The following is an example of what an email alert looks like:
Create an RSS feed
rss action to have Splunk alert you via RSS when the scheduled report triggers an alert:
action.rss = 1
Whenever the alert conditions are met for a scheduled report that has Create an RSS feed selected, Splunk sends a notification out to its RSS feed. The feed is located at
http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on
localhost and uses port 8000, the correct link for the RSS feed would be
You can also access the RSS feed for a scheduled report through the Searches and Reports page in Settings. If a scheduled report has been set up to provide an RSS feed for alerting searches, when you look it up on the Searches and Reports page, you will see a RSS symbol in the RSS feed column:
You can click on this symbol to go to the RSS feed.
Note: The RSS feed for a scheduled report will not display any results until the report has run on its schedule and the alerting conditions that have been defined for it have been met. If you set the search up to alert each time it's run (by setting Perform actions to always), you'll see reports in the RSS feed after first time the report runs on its schedule.
Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular report job, but they can see the summarization displayed in the RSS feed, which includes the name of the report that was run and the number of results returned by the report job.
Here's an example of the XML that generates the feed:
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0"> <channel> <title>Alert: errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Reports Feed for report errors last15</description> <item> <title>errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Alert trigger: errors last15, results.count=123 </description> <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate> </item> </channel> </rss>
Trigger a shell script
script action to have Splunk run a shell script when the scheduled search triggers an alert:
action.script = 1
script action has a
filename parameter which is usually defined at the individual search level, although a default filename can also be set in
action.script.filename = <script filename>
- The filename of the shell script that you want Splunk to run. The script should live in
Example - Basic conditional alert configuration
This example is for a report titled "sudoalert." It runs a search for events containing the term "sudo" on a 12 minute interval. If a scheduled "sudoalert" run results in greater than 10 events, alert actions are triggered that send the results via email and post them to an RSS feed.
[sudoalert] search = sudo counttype = number of events enableSched = 1 schedule = */12 * * * * quantity = 10 relation = greater than action.email = 1 action.email.to = firstname.lastname@example.org action.email.from = email@example.com action.email.subject = Sudo Alert! action.email.mailserver = firstname.lastname@example.org action.rss = 1
Enable summary indexing
Summary indexing is an additional kind of alert action that you can configure for any scheduled report. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar reports on a regular basis.
With summary indexing, you define a scheduled report that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the report it saves the results into a summary index that you've designated. You can then report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.
Note: Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.
For more information about configuring summary index reports in
savedsearches.conf, see "Configure summary indexes" in the Knowledge Manager Manual.
Review triggered alerts
Configure scripted alerts
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15