Set up alerts in savedsearches.conf

Set up alerts in savedsearches.conf

This page is currently a work in progress. Any information presented here may be incomplete or incorrect, and frequent near-term updates are expected.

Configure alerts with savedsearches.conf. Use the $SPLUNK_HOME/etc/system/README/savedsearches.conf.example as an example, or create your own savedsearches.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.

Follow these steps:

1. Create a saved search.

2. Schedule the search.

3. Define alert conditions.

4. Configure alert actions.

You can set up an alert at the time you create a saved search, or add the alert configurations to your saved search stanza later.

Note: You must have email enabled on your Splunk server for alerts to be sent out. Alternately, your Splunk server must be able to contact your email server. Configure email settings in Manager.

Create a saved search

First, set up a saved search, either via Splunk Web (see "Saving searches and sharing search results" in the User manual) or by configuring a new saved search stanza in savedsearches.conf.

Schedule the search

Next, schedule your search. This means your search runs on a schedule that you specify. For example, you can arrange to have Splunk run your search every hour, or every day at midnight.

If you want to schedule your search via Splunk Web, you'll have to use the Create alert window or the Searches and reports page in Manager. For more information, see "Create an alert" in the User manual.

To schedule a search via savesearches.conf, add the following attribute/value pairs to your saved search stanza:

userid = <integer>

• UserId of the user who created this saved search.
  • Splunk needs this information to log who ran the search, and create editing capabilities in Splunk Web.
• Possible values: Any Splunk user ID.
• User IDs are found in $SPLUNK_HOME/etc/passwd.
  • Look for the first number on each line, right before the username.
  • For example 2:penelope....

enableSched = < 0 | 1 >

• Set this to 1 to enable schedule for search
• Defaults to 0.

cron_sched = <cron string>

• The cron schedule used to execute the search.
• For example, */5 * * * * causes the search to execute every five minutes.

Note Cron scheduling lets you use standard cron notation to define your scheduled search interval. In particular, cron can accept this type of notation: 00,20,40 * * * *, which runs the search every hour at hh:00, hh:20, hh:40. Along the same lines, a cron of 03,23,43 * * * * runs the search every hour at hh:03, hh:23, hh:43. Splunk recommends that you schedule your searches so that they're staggered over time. This reduces system load. Running all of them (*/20) every 20 minutes means they would all launch at hh:00 (20, 40) and might slow your system every 20 min.

max_concurrent = <integer>

• The maximum number of concurrent instances of this search the scheduler is allowed to run.
• Defaults to 1.

Configure alert conditions in savedsearches.conf

Next, configure alert conditions for the scheduled search. When Splunk runs a scheduled search, these are the conditions that trigger an alert action (such as an email) when they are met.

Alerts fit into two categories: basic conditional alerts and advanced conditional alerts.

• Basic conditional alerts trigger alert actions when set thresholds in the number of events, sources, or hosts in your results are exceeded.
• Advanced conditional alerts are based on the results of a conditional search that is evaluated against the results of the scheduled search. If the conditional search returns one or more events, the event is triggered.

Configure a basic conditional alert

Configure a threshold number of events, sources, or hosts. If the alert conditions are met when the search is run, Splunk notifies you via email or triggers a shell script. You can also set counttype = always if you want the alert action (such as an email) to be triggered each time the scheduled search runs.

counttype = <string>

• Set the type of count for alerting.
• Possible values: number of events, number of hosts, number of sources, and always.
• Used in conjunction with the relation and quantity attributes, except when set to always.
• Use counttype = always to trigger the alert action each time the scheduled search is run.

relation = <string>

• How to compare against counttype.
• Possible values: greater than, less than, equal to, drops by, rises by.

quantity = <integer>

• Number to compare against the given counttype.

So if you have the following:

counttype = number of events
relation = rises by
quantity = 25

Splunk alerts you if your search results have risen by 25 since the last time the search ran.

For more information about configuring alert actions, see the "Configure alert actions" subtopic, below.

Configure an advanced conditional alert

If you'd rather configure an advanced conditional alert, you use the alert_condition attribute in place of counttype, relation, and quantity.

alert_condition = <string>

• In <string>, enter a search. Splunk will evaluate this secondary search on the artifacts of the saved search to determine whether to trigger an alert action.
• Alert actions are triggered if this secondary search yields a non-empty search result list.

For an in-depth discussion of a use case for advanced conditional alerting over basic conditional alerting, see "Create an alert" in the User Manual. This topic discusses alert setup using Manager, but the underlying principles are the same.

For more information about configuring alert actions, see the following subtopic, "Configure alert actions."

Configure alert actions

You can configure three different kinds of alert actions--actions that happen when alert conditions are met--for your scheduled searches. These alert actions are notification by email, notification by RSS, and the triggering of a shell script.

To enable or disable an alert action for a particular scheduled, alerted search, add the following to the search definition:

action.<action_name> = 0 | 1

• Indicates whether the alert action is enabled or disabled for a particular saved search. Set to 0 (disabled) by default.
• action_name can be email, script, or rss.

Global defaults for all alert actions are configured in alert_actions.conf (or via Splunk Manager). You can override these defaults at the individual search level in savedsearches.conf. If you don't need to override the alert action defaults, all you need to do is indicate which alert actions are enabled for a given scheduled search (see above).

To set a parameter for an alert action, the syntax is as follows:

action.<action_name>.<parameter> = <value>

The parameter options for each <code><action_name> are defined in the following sections.

Notification by email

Use the email action to have Splunk contact stakeholders when the scheduled search triggers an alert:

action.email = 1

The email action has a number of parameters. Defaults can be set for all of these parameters in alert_actions.conf, with the exception of the action.email.to parameter, which should be set for each scheduled search that uses the email alert action.

action.email.to = <email list>

• The email addresses to which Splunk will send the email, arranged in a comma-delimited list.
• This parameter is not set at the alert_actions.conf level. You must define it for every email alert action that you configure.

action.email.from = <email address>

• The email address that is used as the sender's address.
• Default is splunk@$LOCALHOST (or whatever is set for from in alert_actions.conf).

action.email.subject = <string>

• The subject of the alert email.
• Default is SplunkAlert-<savedsearchname> (or whatever is set for subject in alert_actions.conf).

action.email.sendresults = <bool>

• Specify whether to include the search results in the email. The results can be attached or included in the body of the email (see the action.email.inline parameter, below).
• Default is false (or whatever is set for sendresults in alert_actions.conf).
• Note: When you are using an advanced conditional alert, be aware that only the results of the original search are included with the email. The results of the triggering conditional search are discarded

action.email.inline = <true | false>

• Specify whether the search results are included in the body of the alert mail.
• Default is false (or whatever is set for inline in alert_actions.conf).

action.email.mailserver = <string>

• The address of the MTA server that sends the alert emails.
• Default is $LOCALHOST (or whatever is set for mailserver in alert_actions.conf).

action.email.preprocess_results = <search-string>

• An optional search string to preprocess results before emailing them. Usually one would set this up to filter out unwanted internal fields.
• Default is an empty string (or whatever is set for preprocess_results in alert_actions.conf).

Note: You can also arrange to have .pdf printouts of dashboards delivered by email on a set schedule. For more information, see "Schedule delivery of dashboard PDF printouts via email" in this manual.

There are settings for this feature in alert_actions.conf. For example, you can identify the URL of the PDF report server, and the report paper size and orientation.

Important: Use of the .pdf printout feature requires the setup of the PDF Printer app on a central Linux host. If you don't have this set up, contact a system administrator. For more information see "Configure PDF printing for Splunk Web" in the Installation manual.

The following is an example of what an email alert looks like:

Create an RSS feed

Use the rss action to have Splunk alert you via RSS when the scheduled search triggers an alert:

action.rss = 1

Whenever the alert conditions are met for a scheduled search that has Create an RSS feed selected, Splunk sends a notification out to its RSS feed. The feed is located at http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on localhost and uses port 8000, the correct link for the RSS feed would be http://localhost:8000/rss/errors_last15.

You can also access the RSS feed for a scheduled search through the Searches and reports page in Manager. If a scheduled search has been set up to provide an RSS feed for alerting searches, when you look it up on the Searches and reports page, you will see a RSS symbol in the RSS feed column:

You can click on this symbol to go to the RSS feed.

Note: The RSS feed for a scheduled search will not display any searches until the search has run on its schedule and the alerting conditions that have been defined for it have been met. If you set the search up to alert each time it's run (by setting Perform actions to always), you'll see searches in the RSS feed after first time the search runs on its schedule.

Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
    <channel>
        <title>Alert: errors last15</title>
        <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
        <description>Saved Searches Feed for saved search errors last15</description>
        <item>
            <title>errors last15</title>
            <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
            <description>Alert trigger: errors last15, results.count=123 </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>
        </item>
    </channel>
</rss>

Trigger a shell script

Use the script action to have Splunk run a shell script when the scheduled search triggers an alert:

action.script = 1

The script action has a filename parameter which is usually defined at the individual search level, although a default filename can also be set in alert_actions.conf:

action.script.filename = <script filename>

• The filename of the shell script that you want Splunk to run. The script should live in $SPLUNK_HOME/bin/scripts/.

Example - Basic conditional alert configuration

This example is for a saved search titled "sudoalert." It runs a search for events containing the term "sudo" on a 12 minute interval. If a scheduled "sudoalert" run results in greater than 10 events, alert actions are triggered that send the results via email and post them to an RSS feed.

[sudoalert]
search = sudo
counttype = number of events
enableSched = 1
schedule = */12 * * * *
quantity = 10
relation = greater than
action.email = 1
action.email.to = me@work.org 
action.email.from = splunk@work.org
action.email.subject = Sudo Alert!
action.email.mailserver = mail@work.org
action.rss = 1

Enable summary indexing

Summary indexing is an additional kind of alert action that you can configure for any scheduled search. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.

With summary indexing, you define a scheduled search that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the search it saves the results into a summary index that you've designated. You can then search and report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.

Note: Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

For more information about configuring summary index searches in savedsearches.conf, see "Configure summary indexes" in the Knowledge Manager Manual.

• Was this topic useful?
• Post a comment