Set up alerts in savedsearches.conf
Configure alerts with savedsearches.conf. Use the
$SPLUNK_HOME/etc/system/README/savedsearches.conf.example as an example, or create your own
savedsearches.conf. Edit this file in
$SPLUNK_HOME/etc/system/local/, or your own custom application directory in
$SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.
Follow these steps:
1. Create a saved search.
2. Schedule the search.
3. Define alert conditions.
4. Configure alert actions.
You can set up an alert at the time you create a saved search, or add the alert configurations to your saved search stanza later.
Note: You must have email enabled on your Splunk server for alerts to be sent out. Alternately, your Splunk server must be able to contact your email server. Configure email settings in Manager.
Create a saved search
Schedule the search
Next, schedule your search. This means your search runs on a schedule that you specify. For example, you can arrange to have Splunk run your search every hour, or every day at midnight.
If you want to schedule your search via Splunk Web, you'll have to use the Create alert window or the Searches and reports page in Manager. For more information, see "Create an alert" in the User manual.
To schedule a search via
savesearches.conf, add the following attribute/value pairs to your saved search stanza:
userid = <integer>
- UserId of the user who created this saved search.
- Splunk needs this information to log who ran the search, and create editing capabilities in Splunk Web.
- Possible values: Any Splunk user ID.
- User IDs are found in
- Look for the first number on each line, right before the username.
- For example
enableSched = < 0 | 1 >
- Set this to 1 to enable schedule for search
- Defaults to 0.
cron_sched = <cron string>
- The cron schedule used to execute the search.
- For example,
*/5 * * * *causes the search to execute every five minutes.
Note Cron scheduling lets you use standard cron notation to define your scheduled search interval. In particular, cron can accept this type of notation: 00,20,40 * * * *, which runs the search every hour at hh:00, hh:20, hh:40. Along the same lines, a cron of 03,23,43 * * * * runs the search every hour at hh:03, hh:23, hh:43. Splunk recommends that you schedule your searches so that they're staggered over time. This reduces system load. Running all of them (*/20) every 20 minutes means they would all launch at hh:00 (20, 40) and might slow your system every 20 min.
max_concurrent = <integer>
- The maximum number of concurrent instances of this search the scheduler is allowed to run.
- Defaults to 1.
Configure alert conditions in savedsearches.conf
Next, configure alert conditions for the scheduled search. When Splunk runs a scheduled search, these are the conditions that trigger an alert action (such as an email) when they are met.
Alerts fit into two categories: basic conditional alerts and advanced conditional alerts.
- Basic conditional alerts trigger alert actions when set thresholds in the number of events, sources, or hosts in your results are exceeded.
- Advanced conditional alerts are based on the results of a conditional search that is evaluated against the results of the scheduled search. If the conditional search returns one or more events, the event is triggered.
Configure a basic conditional alert
Configure a threshold number of events, sources, or hosts. If the alert conditions are met when the search is run, Splunk notifies you via email or triggers a shell script. You can also set
counttype = always if you want the alert action (such as an email) to be triggered each time the scheduled search runs.
counttype = <string>
- Set the type of count for alerting.
- Possible values:
number of events,
number of hosts,
number of sources, and
- Used in conjunction with the
quantityattributes, except when set to
counttype = alwaysto trigger the alert action each time the scheduled search is run.
relation = <string>
- How to compare against counttype.
- Possible values: greater than, less than, equal to, drops by, rises by.
quantity = <integer>
- Number to compare against the given counttype.
So if you have the following:
counttype = number of events relation = rises by quantity = 25
Splunk alerts you if your search results have risen by 25 since the last time the search ran.
For more information about configuring alert actions, see the "Configure alert actions" subtopic, below.
Configure an advanced conditional alert
If you'd rather configure an advanced conditional alert, you use the
alert_condition attribute in place of
alert_condition = <string>
<string>, enter a search. Splunk will evaluate this secondary search on the artifacts of the saved search to determine whether to trigger an alert action.
- Alert actions are triggered if this secondary search yields a non-empty search result list.
For an in-depth discussion of a use case for advanced conditional alerting over basic conditional alerting, see "Create an alert" in the User Manual. This topic discusses alert setup using Manager, but the underlying principles are the same.
For more information about configuring alert actions, see the following subtopic, "Configure alert actions."
Configure alert actions
You can configure three different kinds of alert actions--actions that happen when alert conditions are met--for your scheduled searches. These alert actions are notification by email, notification by RSS, and the triggering of a shell script.
To enable or disable an alert action for a particular scheduled, alerted search, add the following to the search definition:
action.<action_name> = 0 | 1
- Indicates whether the alert action is enabled or disabled for a particular saved search. Set to
0(disabled) by default.
Global defaults for all alert actions are configured in
alert_actions.conf (or via Splunk Manager). You can override these defaults at the individual search level in
savedsearches.conf. If you don't need to override the alert action defaults, all you need to do is indicate which alert actions are enabled for a given scheduled search (see above).
To set a parameter for an alert action, the syntax is as follows:
action.<action_name>.<parameter> = <value>
The parameter options for each <code><action_name> are defined in the following sections.
Notification by email
action.email = 1
alert_actions.conf, with the exception of the
action.email.to parameter, which should be set for each scheduled search that uses the email alert action.
action.email.to = <email list>
- The email addresses to which Splunk will send the email, arranged in a comma-delimited list.
- This parameter is not set at the
alert_actions.conflevel. You must define it for every email alert action that you configure.
action.email.from = <email address>
- The email address that is used as the sender's address.
- Default is
splunk@$LOCALHOST(or whatever is set for
action.email.subject = <string>
- The subject of the alert email.
- Default is
SplunkAlert-<savedsearchname>(or whatever is set for
action.email.sendresults = <bool>
- Specify whether to include the search results in the email. The results can be attached or included in the body of the email (see the
- Default is
false(or whatever is set for
- Note: When you are using an advanced conditional alert, be aware that only the results of the original search are included with the email. The results of the triggering conditional search are discarded
action.email.inline = <true | false>
- Specify whether the search results are included in the body of the alert mail.
- Default is
false(or whatever is set for
action.email.mailserver = <string>
- The address of the MTA server that sends the alert emails.
- Default is
$LOCALHOST(or whatever is set for
action.email.preprocess_results = <search-string>
- An optional search string to preprocess results before emailing them. Usually one would set this up to filter out unwanted internal fields.
- Default is an empty string (or whatever is set for
Note: You can also arrange to have .pdf printouts of dashboards delivered by email on a set schedule. For more information, see "Schedule delivery of dashboard PDF printouts via email" in this manual.
There are settings for this feature in
alert_actions.conf. For example, you can identify the URL of the PDF report server, and the report paper size and orientation.
Important: Use of the .pdf printout feature requires the setup of the PDF Printer app on a central Linux host. If you don't have this set up, contact a system administrator. For more information see "Configure PDF printing for Splunk Web" in the Installation manual.
The following is an example of what an email alert looks like:
Create an RSS feed
rss action to have Splunk alert you via RSS when the scheduled search triggers an alert:
action.rss = 1
Whenever the alert conditions are met for a scheduled search that has Create an RSS feed selected, Splunk sends a notification out to its RSS feed. The feed is located at
http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on
localhost and uses port 8000, the correct link for the RSS feed would be
You can also access the RSS feed for a scheduled search through the Searches and reports page in Manager. If a scheduled search has been set up to provide an RSS feed for alerting searches, when you look it up on the Searches and reports page, you will see a RSS symbol in the RSS feed column:
You can click on this symbol to go to the RSS feed.
Note: The RSS feed for a scheduled search will not display any searches until the search has run on its schedule and the alerting conditions that have been defined for it have been met. If you set the search up to alert each time it's run (by setting Perform actions to always), you'll see searches in the RSS feed after first time the search runs on its schedule.
Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.
Here's an example of the XML that generates the feed:
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0"> <channel> <title>Alert: errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Saved Searches Feed for saved search errors last15</description> <item> <title>errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Alert trigger: errors last15, results.count=123 </description> <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate> </item> </channel> </rss>
Trigger a shell script
script action to have Splunk run a shell script when the scheduled search triggers an alert:
action.script = 1
script action has a
filename parameter which is usually defined at the individual search level, although a default filename can also be set in
action.script.filename = <script filename>
- The filename of the shell script that you want Splunk to run. The script should live in
Example - Basic conditional alert configuration
This example is for a saved search titled "sudoalert." It runs a search for events containing the term "sudo" on a 12 minute interval. If a scheduled "sudoalert" run results in greater than 10 events, alert actions are triggered that send the results via email and post them to an RSS feed.
[sudoalert] search = sudo counttype = number of events enableSched = 1 schedule = */12 * * * * quantity = 10 relation = greater than action.email = 1 action.email.to = firstname.lastname@example.org action.email.from = email@example.com action.email.subject = Sudo Alert! action.email.mailserver = firstname.lastname@example.org action.rss = 1
Enable summary indexing
Summary indexing is an additional kind of alert action that you can configure for any scheduled search. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.
With summary indexing, you define a scheduled search that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the search it saves the results into a summary index that you've designated. You can then search and report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.
Note: Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.
For more information about configuring summary index searches in
savedsearches.conf, see "Configure summary indexes" in the Knowledge Manager Manual.
How alerting works
Configure scripted alerts
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7