Set alert conditions for scheduled searches

Set alert conditions for scheduled searches

After you schedule a search, you can configure a basic or advanced conditional alert for that search.

When the scheduled search runs and the alert conditions are met, alert actions, such as email or RSS notifications, are triggered. Email notifications typically explain which search triggered the alert; optionally they can also include search results (see below).

Specify alert conditions

You can set up basic and advanced conditional alerts.

Basic conditional alerts are triggered when set thresholds in the number of events, sources, or hosts in your results are exceeded. When the alert is triggered, Splunk notifies you via email or RSS, or by executing a custom script.

An advanced conditional alert is based on the results of a search that is evaluated against the results of the scheduled search. If those results meet specific conditions, then the alert is triggered.

You can also set up the search to alert every time it runs by setting Perform actions to always. This setting is typically only used for summary indexing (see below).

Define a basic conditional alert

Follow this procedure to define a basic conditional alert that notifies you of thresholds related to events, hosts, or sources.

1. In the Perform actions dropdown menu, select either if number of events, if number or sources, or if number of hosts. Choosing one of these three values causes the next two fields for basic conditional alerting to appear. (Alternatively, you can select always to have Splunk notify you each time the search is run. This can be handy if the search runs on an infrequent basis and you just want to see the results no matter what they are.)

2. Choose a comparison operation from the dropdown list that appears below the Perform actions field: is greater than, is less than, is equal to, rises by, or drops by.

3. In the field adjacent to the comparison operation dropdown, enter an integer for the alert threshold.

For example, you can set up an alert that sends out a notification if number of events returned by the search rises by a threshold of 10 events since the last time the search was run.

Define an advanced conditional alert

Advanced conditional alerting enables you to define alert conditions based on the results of a conditional search that is applied to the results returned by the scheduled search. Splunk triggers the alert if the conditional search returns at least one result.

By basing your alert conditions on a search, you can define specific conditions for triggering alerts and reduce the incidence of false positive alerts.

Follow this procedure to define an advanced conditional alert:

1. In the Perform action dropdown menu, select if custom condition is met. The Custom condition search field appears.

2. Enter your conditional search in the Custom condition search field.

Advanced conditional alert example

Lets say you're setting up an alert for this scheduled search:

This search returns the number of incorrect password entries associated with each user name.

What you want to do is arrange to have Splunk trigger the alert when the scheduled search finds more than 10 password failures for any given user. When the alert is triggered, an email containing the results of the triggering search gets sent to interested parties.

Now, it seems like you could simply append | search count > 10 to the original scheduled search:

Unfortunately, if you create a basic conditional alert based on this search, where an alert is triggered if the number of events returned is greater than 1, you won't get the behavior you desire. This is because this new search only returns user names that are associated with 10+ failed password entries--the actual count amounts are left out. When the alert is triggered and the results are emailed to stakeholders, you want the recipients to have a listing that matches each user name to the precise number of failed password attempts that it is associated with.

What you want to do is set Perform action to if custom condition is met and then place search count > 10 in the Custom condition search field. This conditional search runs against the results of the original scheduled search (failed password | stats count by user). With this, the alert is triggered only when the custom condition is met--when there are 1 or more user names associated with 10 failed password entries. But when it is triggered, the results of the original search--the list of user names and their failed password counts--is sent to stakeholders via email.

Specify alert actions

You can set up three different kinds of alert actions--actions that happen when alert conditions are met--for your scheduled searches. These alert actions are notification by email, notification by RSS, and the triggering of a shell script.

Notification by email

In the Alert actions section, select Send email to have Splunk contact stakeholders via email when a scheduled search triggers an alert. Enter one or more email addresses in the Email address(es) field, separated by commas.

To have Splunk send alert emails, you first need to configure the email alert settings. You can configure these settings in Manager, or by making changes to alert_actions.conf. You can also configure email alert settings for individual searches via savedsearches.conf. (For more information about .conf file management of saved searches and alert settings see "Set up alerts in savedsearches.conf" in the Admin Manual.)

In Splunk Web, navigate to Manager > System settings > Email alert settings. Here you can define the Email format and specify whether or not you want Splunk to Include results inline. If you don't see Email alert settings, you do not have permission to edit the settings. In this case, contact your Splunk Admin.

Send results in alert emails

You can arrange to have email alert notifications contain the results of the searches that trigger them. This works best when the search returns a truncated list (such as a list that returns the top 20 results) or a table. To do this, select Include results in email.

Include results in email ensures that the email includes the results either inline, or as an additional attachment. The method of inclusion is controlled via alert_actions.conf (at a global level) or savedsearches.conf (at an individual search level); for more information see "Set up alerts in savedsearches.conf" in the Admin Manual.

Note that Include results in email returns the results in text format as part of the body of the email, not as a separate email attachment.

To send a separate .pdf printout of the report results with the alert email, select Include PDF version of results.

Note: You can also arrange to have .pdf printouts of dashboards delivered by email on a set schedule. For more information, see "Schedule delivery of dashboard PDF printouts via email" in this manual.

Important: Use of the .pdf printout feature requires the setup of the PDF Printer app on a central Linux host. If you don't have this set up, contact a system administrator. For more information see "Configure PDF printing for Splunk Web" in the Installation manual.

The following is an example of what an email alert looks like:

Create an RSS feed

Select Create an RSS feed if you want to receive alerts via RSS.

Whenever the alert conditions are met for a scheduled search that has Create an RSS feed selected, Splunk sends a notification out to its RSS feed. The feed is located at http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on localhost and uses port 8000, the correct link for the RSS feed would be http://localhost:8000/rss/errors_last15.

You can also access the RSS feed for a scheduled search through the Searches and reports page in Manager. If a scheduled search has been set up to provide an RSS feed for alerting searches, when you look it up on the Searches and reports page, you will see a RSS symbol in the RSS feed column:

You can click on this symbol to go to the RSS feed.

Note: The RSS feed for a scheduled search will not display any searches until the search has run on its schedule and the alerting conditions that have been defined for it have been met. If you set the search up to alert each time it's run (by setting Perform actions to always), you'll see searches in the RSS feed after first time the search runs on its schedule.

Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
    <channel>
        <title>Alert: errors last15</title>
        <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
        <description>Saved Searches Feed for saved search errors last15</description>
        <item>
            <title>errors last15</title>
            <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
            <description>Alert trigger: errors last15, results.count=123 </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>
        </item>
    </channel>
</rss>

Trigger a shell script

If you want Splunk to trigger a shell script when the alert conditions are met, select Trigger a shell script and enter the file name of the script that you want Splunk to execute. For example, you may want an alert to trigger a script that generates an SNMP trap notification and sends it to another system such as a Network Systems Management console. Meanwhile, you could have a different alert that--when its alert conditions are met--triggers a script that calls an API, which in turn sends the triggering event to another system.

Note: For security reasons, all alert scripts must be placed in the $SPLUNK_HOME/bin/scripts directory. This is where Splunk will look for any script triggered by an alert.

Check out this excellent topic on troubleshooting alert scripts on the Splunk Community Wiki.

For more details on configuring alerts, including instructions for configuring alerts using savedsearches.conf, see the Admin Manual topic on alerts.

Specify fields to show in alerts through search language

When you receive alerts, Splunk includes all the fields in your search. Add the fields command to your saved search to specify which fields you want included and excluded.

• To eliminate a field, pipe your search to fields - $FIELDNAME.
• To add a field, pipe your search to fields + $FIELDNAME.

You can specify multiple fields to include and exclude in one string. For example, your Search field may be:

yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4

The alert you receive will exclude $FIELD1 and $FIELD2, but include $FIELD3 and $FIELD4.

Enable summary indexing

Summary indexing is an action that you can configure for any scheduled search. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.

With summary indexing, you define a scheduled search that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the search it saves the results into a summary index that you've designated. You can then search and report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.

If you want to enable summary indexing for this search, select Enable summary indexing. For the summary index to gather data on a regular interval, set Alert conditions to Always.

Note: Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

• Was this topic useful?
• Post a comment