Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Set up alert actions

This section provides more information about the various kinds of alert actions that you can enable for an alert. Your alert action choices are the same for all three alert types.

The actions you can enable include email notification, the running of scripts, and the display of triggered alerts in Alert manager via the alert actions page of the Save As Alert dialog.

There are additional actions available for alerts in Settings. If you go to Settings > Searches and Reports and either define a new alert or open the detail page for an existing report upon which an alert is based, you will find that you can additionally enable RSS notification and turn on summary indexing for alerts.

For more information on how these alert actions work, see the sections below.

Note: This topic does not explain how to set up alerts. For a full overview of the alert creation process, see "About alerts," in this manual.

Send an email

If you want Splunk to contact stakeholders when the alert is triggered, select Enable next to Send email.

Subject field

For the Subject field, supply a subject header for the email. By default, it is set to be Splunk Alert: $name$. Splunk will replace $name$ with the report name.

Splunk provides additional variables that you can use in the Subject field. They include, but are not limited to, the following:

Variable Description
$search$ The search that triggered the alert.
$alert.severity$ The severity level of the alert.
$results.count$ The number of results returned by the search.
$results.url$ A Splunk Web URL where users can view the results.
$results.file$ The absolute path to the results file.
$search_id$ The search ID of the job that triggered the alert.

You can find a full list of available variables in the savedsearches.conf specification file in the Admin Manual. All attributes displayed in savedsearches.conf can be used as variables in the Subject field of an email.

Addresses field

For the Addresses field, enter a comma-separated list of email addresses to which the alert should be sent.

Note: For your email notifications to work correctly, you first need to have your email alert settings configured in Settings. See the subsection "Configure email alert settings in Settings," below.

Send results in alert emails

When you're defining an alert, you can optionally arrange to have email alert notifications contain the results of the searches that trigger them. This works best when the search returns a single value, a truncated list (such as the result of a search that returns only the top 20 matching results) or a table.

In the alert actions page of the Save As Alert dialog, click one of the four buttons next to Include results to specify the output format of the results: None, Text, CSV, or PDF".

  • Text - Select to have Splunk deliiver the search results in the body of the alert email.
  • None - Means do not include the results.
  • CSV - Select this to have Splunk convert the results to .CSV format and attach the file to the alert notification email.
  • PDF - Select to have Splunk deliver the search results in the form of a PDF attachment.

The result inclusion method is controlled via alert_actions.conf (at a global level) or savedsearches.conf (at an individual search level); for more information see "Configure alerts in savedsearches.conf" in this manual.

60 saveasalert page2 enableactions listintrigger sendemail.png

The following is an example of what an email alert looks like when results are included inline (in the body of the email):


Configure email alert settings in Settings

Email alerting will not work if the email alert settings in Settings are not configured, or are configured incorrectly. You can define these settings at Settings > System settings > Email alert settings.

On the Email alert settings Settings page, you can define the Mail server settings (the mail host, security type, username, password, and so on) and the Email format (link hostname, email sender name, email subject header, and inline results format).

Finally, if you are sending results as PDF attachments (see above) you can determine the paper size and orientation of the PDF report under PDF Report Settings. You can also set the Remote PDF Report Server URL for the PDF Report Server App if you plan to use it.

Note: As of release 5.0, Splunk no longer requires the PDF Report Server App to generate search result PDFs, but it can still be used for printing dashboards that are built with Advanced XML. For more information see "Generate PDFs of your reports and dashboards" in the Reporting Manual.)

If you plan to use the PDF Report Server App, the Link hostname field must be the search head hostname for the instance sending requests to a PDF Report Server. Set this option only if Splunk improperly auto-detects the hostname for your environment.

Specify your choices and click Save to have all alerts use these settings for email actions.

If you don't see System settings or Email alert settings in Settings, you do not have permission to edit the settings. In this case, contact your Splunk administrator.

You can also use configuration files to set up email alert settings. You can configure them for your entire Splunk implementation in alert_actions.conf, and you can configure them at the individual search level in savedsearches.conf. For more information about .conf file management of reports and alert settings see "Configure alerts in savedsearches.conf" in this manual.

Run a script

If you want Splunk to run an alert script when the alert triggers, select Run a script under Enable actions and enter the file name of the script that you want Splunk to execute.

For example, you might want an alert to run a script that generates a Simple Network Management Protocol (SNMP) trap notification and sends it to another system such as a Network Systems Management console when its alerting conditions are met. Meanwhile, you could have a different alert that, when triggered, runs a script that calls an API, which in turn sends the triggering event to another system.

Note: For security reasons, all alert scripts must be placed in $SPLUNK_HOME/bin/scripts or $SPLUNK_HOME/etc/<AppName>/bin/scripts. Splunk looks in these two directories for any script triggered by an alert.

For detailed instruction on alert script configuration using savedsearches.conf in conjunction with shell script or batch file that you create, see "Configure scripted alerts" in this manual.

If you are having trouble with your alert scripts, check out this excellent topic on troubleshooting alert scripts on the Splunk Community Wiki.

Show triggered alerts in the Alert manager

If you want to have the Alert manager keep records of the triggered alerts related to a particular alert configuration, select the List in Triggered Alerts checkbox. The Alert manager will keep records of triggered alerts for the duration specified in the Expiration field in the Alert configuration for the specified alert.

For more information about the Alert manager and how to use it, see the "Review triggered alerts" topic in this manual.

Give tracked alerts a severity level

On the Alert manager page, each alert is labeled with a Severity level that helps people know how important each alert is in relation to other alerts. For example, an alert that lets you know that a server is approaching disk capacity could be given a High label, while an alert triggered by a "disk full" error could have a Critical label.

You can choose from Info, Low, Medium, High, and Critical. The default is Medium.

5.0-Alert Severity.jpg

Severity labels are informational in purpose and have no additional functionality. You can use them to quickly pick out important alerts from the alert listing on the Alert manager page. Get to the Alert manager page by clicking the Triggered Alerts link in the upper right-hand corner of the Splunk bar.

Alert action functionality available in Settings

If you create or update your alert in Settings > Searches and Reports you'll find additional alert action options. For example, you can opt to have alert-triggering results sent to an RSS feed.

Create an RSS feed

If you want Splunk to post this alert to an RSS feed when it triggers, select Enable next to Add to RSS on the detail page for the alerting search in Settings > Searches and Reports.

When an alert with the Add to RSS action enabled triggers, Splunk sends a notification out to its RSS feed. The feed is located at http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on localhost and uses port 8000, the correct link for the RSS feed would be http://localhost:8000/rss/errors_last15.

You can also find links to the RSS feeds for alerting searches at Settings > Searches and reports. Searches that have Add to RSS enabled display an RSS symbol in the RSS feed column:

Saved search RSS.png

Click on this symbol to go to the RSS feed.

Note: An RSS feed for an alerting search won't display anything until the alert has been triggered at least once. If the alert is based on a scheduled search that is set to alert each time it is run (it has Perform actions to always), you'll see search information in the RSS feed after first time the search runs on its schedule.

Caution: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
        <title>Alert: errors last15</title>
        <description>Reports Feed for report errors last15</description>
            <title>errors last15</title>
            <description>Alert trigger: errors last15, results.count=123 </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>

Specify fields to show in alerts through search language

When Splunk provides the results of the alerting search job (in an alert email, for example), it includes all the fields in those results. To have certain fields included in or excluded from the results, use the fields command in the base search for the alert.

  • To eliminate a field from the search results, pipe your search to fields - $FIELDNAME.
  • To add a field to the search results, pipe your search to fields + $FIELDNAME.

You can specify multiple fields to include and exclude in one string. For example, your Search field may be:

yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4

This generates an alert that excludes $FIELD1 and $FIELD2, but includes $FIELD3 and $FIELD4.

Enable summary indexing in Settings

Summary indexing is an action that you can configure for any alert via Settings > Searches and Reports. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.

With summary indexing, you base an alert on a search that computes sufficient statistics (a summary) for events covering a slice of time. The search is set up so that each time it runs on its schedule, the search results are saved into a summary index that you designate. You can then run searches against this smaller (and thus faster) summary index instead of working with the much larger dataset from which the summary index receives its events.

Note: You do not need to use summary indexing for searches that already benefit from report acceleration. For more information and a distinction between these two methods of speeding up slow running searches, see "About report acceleration and summary indexing" in the Knowledge Manager manual.

To set up summary indexing for an alert, go to Settings > Searches and Reports, and either add a new report or open up the detail page for an existing search or alert. (You cannot set up summary indexing through the Create Alert window.) To enable the summary index to gather data on a regular interval, set its Alert condition to always and then select Enable under Summary indexing at the bottom of the view.

Note: There's more to summary indexing--you should take care to properly construct the search that populates the summary index. In most cases special reporting commands should be used. Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

Update and expand alert functionality in Settings
Alert examples

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters