Set up alert actions

Set up alert actions

This section provides more information about the various kinds of alert actions that you can enable for an alert. Your alert action choices are the same for all three alert actions.

You can enable email notification, the running of scripts, and the display of triggered alerts in Alert manager via the Actions step of the Create Alert dialog.

There are additional actions available for alerts in Manager. If you go to Manager > Searches and Reports and either define a new alert or open the detail page for an existing saved search upon which an alert is based, you will find that you can additionally enable RSS notification and turn on summary indexing for alerts.

For more information on how these alert actions work, see the sections below.

Note: This topic does not explain how to set up alerts. For a full overview of the alert creation process, see "Create an alert," in this manual.

Send an email

If you want Splunk to contact stakeholders when the alert is triggered, select Send email under Enable actions.

Under Addresses, enter a comma-separated list of email addresses to which the alert should be sent. Under Subject, supply a subject header for the email. By default it is set to be Splunk Alert: $name$. Splunk will replace $name$ with the saved search name.

Note: For your email notifications to work correctly, you first need to have your email alert settings configured in Manager. See the subsection "Configure email alert settings in Manager," below.

Send results in alert emails

When you're defining an alert, you can optionally arrange to have email alert notifications contain the results of the searches that trigger them. This works best when the search returns a single value, a truncated list (such as the result of a search that returns only the top 20 matching results) or a table.

In the Actions step of the Create Alert dialog, select Include results as and select either as CSV or inline. Splunk delivers inline results as part of the body of the alert email. If you select as CSV, Splunk will put the results in .CSV format and attach the file to the alert notification email.

The result inclusion method is controlled via alert_actions.conf (at a global level) or savedsearches.conf (at an individual search level); for more information see "Set up alerts in savedsearches.conf" in the Admin Manual.

If you have the PDF Report Server app set up you can alternatively arrange to have results sent as .PDF attachments. To arrange this go to Manager > Searches and Reports and open the detail page for the underlying search. In the Alert Actions section see that Enable is selected for Send email and then select Include PDF version of results. This setting will be unavailable if your PDF Report Server app is not installed.

Important: You cannot configure alerts to send results as PDF attachments until you install the PDF Report Server app on a central Linux host. If you aren't running this app, the as PDF option for Include search results will be unavailable. To get PDF printing working, contact a system administrator. For more information see "Configure PDF printing for Splunk Web" in the Installation manual.

You can also arrange to have PDF printouts of dashboards delivered by email on a set schedule. For more information, see "Schedule delivery of dashboard PDF printouts via email" in this manual.

The following is an example of what an email alert looks like when results are included inline (in the body of the email):

Configure email alert settings in Manager

Email alerting will not work if the email alert settings in Manager are not configured, or are configured incorrectly. You can define these settings at Manager > System settings > Email alert settings. Here you can define the Mail server settings (the mail host, username, password, and so on) and the Email format (link hostname, email subject and format, include inline results, and so on).

If you don't see System settings or Email alert settings in Manager, you do not have permission to edit the settings. In this case, contact your Splunk Admin.

You can also use configuration files to set up email alert settings. You can configure them for your entire Splunk implementation in alert_actions.conf, and you can configure them at the individual search level in savedsearches.conf. For more information about .conf file management of saved searches and alert settings see "Set up alerts in savedsearches.conf" in the Admin Manual.

Note: If you are sending search results as PDF attachments (see above), the link hostname field must be the search head hostname for the instance sending requests to a PDF Report Server. Set this option only if the hostname that is autodetected by default is not correct for your environment.

Run a script

If you want Splunk to run an alert script when the alert is triggered, select Run a script under Enable actions and enter the file name of the script that you want Splunk to execute.

For example, you may want an alert to run a script that generates an SNMP trap notification and sends it to another system such as a Network Systems Management console when its alerting conditions are met. Meanwhile, you could have a different alert that, when triggered, runs a script that calls an API, which in turn sends the triggering event to another system.

Note: For security reasons, all alert scripts must be placed in $SPLUNK_HOME/bin/scripts or $SPLUNK_HOME/etc/<AppName>/bin/scripts. This is where Splunk will look for any script triggered by an alert.

For detailed instruction on alert script configuration using savedsearches.conf in conjunction with shell script or batch file that you create, see "Configure scripted alerts" in the Admin Manual.

If you are having trouble with your alert scripts, check out this excellent topic on troubleshooting alert scripts on the Splunk Community Wiki.

Show triggered alerts in the Alert manager

If you want to have the Alert manager keep records of the triggered alerts related to a particular alert configuration, select the Show triggered alerts in Alert manager checkbox. The Alert manager will keep records of triggered alerts for the duration specified in the Expiration field on the Set Up Alert step of the Create Alert dialog box.

For more information about the Alert manager and how it is used, see the "Review triggered alerts" topic in this manual.

Give tracked alerts a severity level

On the Alert manager page, each alert is labeled with a Severity level that helps people know how important each alert is in relation to other alerts. For example, an alert that lets you know that a server is approaching disk capacity could be given a High label, while an alert triggered by a "disk full" error could have a Critical label.

You can choose from Info, Low, Medium, High, and Critical. The default is Medium.

Severity labels are informational in purpose and have no additional functionality. You can use them to quickly pick out important alerts from the alert listing on the Alert manager page. Get to the Alert manager page by clicking the Alerts link in the upper right-hand corner of the Splunk interface.

Alert action functionality available in Manager

If you create or update your alert in Manager > Searches and Reports you'll find addtional alert action options. For example, you can opt to have alert-triggering results sent to an RSS feed.

Create an RSS feed

If you want Splunk to post this alert to an RSS feed when it is triggered, select Enable next to Add to RSS on the detail page for the alerting search in Manager > Searches and Reports.

When an alert with the Add to RSS action enabled is triggered, Splunk sends a notification out to its RSS feed. The feed is located at http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on localhost and uses port 8000, the correct link for the RSS feed would be http://localhost:8000/rss/errors_last15.

You can also find links to the RSS feeds for alerting searches at Manager > Searches and reports. Searches that have Add to RSS enabled display an RSS symbol in the RSS feed column:

Click on this symbol to go to the RSS feed.

Note: An RSS feed for an alerting search won't display anything until the alert has been triggered at least once. If the alert is based on a scheduled search that is set to alert each time it is run (it has Perform actions to always), you'll see search information in the RSS feed after first time the search runs on its schedule.

Warning: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
    <channel>
        <title>Alert: errors last15</title>
        <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
        <description>Saved Searches Feed for saved search errors last15</description>
        <item>
            <title>errors last15</title>
            <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link>
            <description>Alert trigger: errors last15, results.count=123 </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>
        </item>
    </channel>
</rss>

Enable summary indexing

Summary indexing is an alert action that you can configure for any alert via Manager > Searches and Reports. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.

With summary indexing, you base an alert on a search that computes sufficient statistics (a summary) for events covering a slice of time. The search is set up so that each time it runs on its schedule, the search results are saved into a summary index that you designate. You can then run searches against this smaller (and thus faster) summary index instead of working with the much larger dataset from which the summary index receives its events.

To set up summary indexing for an alert, go to Manager > Searches and Reports, and either add a new saved search or open up the detail page for an existing search or alert. (You cannot set up summary indexing through the Create Alert window.) To enable the summary index to gather data on a regular interval, set its Alert condition to always and then select Enable under Summary indexing at the bottom of the view.

Note: There's more to summary indexing--you should take care how you construct the search that populates the summary index and in most cases special reporting commands should be used. Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

• Was this topic useful?
• Post a comment