Set up alert actions
This section provides more information about the various kinds of alert actions that you can enable for an alert. Your alert action choices are the same for all three alert types.
The actions you can enable include email notification, the running of scripts, and the display of triggered alerts in Alert manager via the alert actions page of the Save As Alert dialog.
There are additional actions available for alerts in Settings. If you go to Settings > Searches and Reports and either define a new alert or open the detail page for an existing report upon which an alert is based, you will find that you can additionally enable RSS notification and turn on summary indexing for alerts.
For more information on how these alert actions work, see the sections below.
Note: This topic does not explain how to set up alerts. For a full overview of the alert creation process, see "About alerts," in this manual.
Send an email
If you want Splunk to contact stakeholders when the alert is triggered, select Enable next to Send email.
For the Subject field, supply a subject header for the email. By default, it is set to be
Splunk Alert: $name$. Splunk will replace
$name$ with the report name.
Splunk provides additional variables that you can use in the Subject field. They include, but are not limited to, the following:
||The search that triggered the alert.|
||The severity level of the alert.|
||The number of results returned by the search.|
||A Splunk Web URL where users can view the results.|
||The absolute path to the results file.|
||The search ID of the job that triggered the alert.|
You can find a full list of available variables in the
savedsearches.conf specification file in the Admin Manual. All attributes displayed in savedsearches.conf can be used as variables in the Subject field of an email.
For the Addresses field, enter a comma-separated list of email addresses to which the alert should be sent.
Note: For your email notifications to work correctly, you first need to have your email alert settings configured in Settings. See the subsection "Configure email alert settings in Settings," below.
Send results in alert emails
When you're defining an alert, you can optionally arrange to have email alert notifications contain the results of the searches that trigger them. This works best when the search returns a single value, a truncated list (such as the result of a search that returns only the top 20 matching results) or a table.
In the alert actions page of the Save As Alert dialog, click one of the four buttons next to Include results to specify the output format of the results: None, Text, CSV, or PDF".
- Text - Select to have Splunk deliiver the search results in the body of the alert email.
- None - Means do not include the results.
- CSV - Select this to have Splunk convert the results to .CSV format and attach the file to the alert notification email.
- PDF - Select to have Splunk deliver the search results in the form of a PDF attachment.
The result inclusion method is controlled via
alert_actions.conf (at a global level) or
savedsearches.conf (at an individual search level); for more information see "Configure alerts in savedsearches.conf" in this manual.
The following is an example of what an email alert looks like when results are included inline (in the body of the email):
Configure email alert settings in Settings
Email alerting will not work if the email alert settings in Settings are not configured, or are configured incorrectly. You can define these settings at Settings > System settings > Email alert settings.
On the Email alert settings Settings page, you can define the Mail server settings (the mail host, security type, username, password, and so on) and the Email format (link hostname, email sender name, email subject header, and inline results format).
Finally, if you are sending results as PDF attachments (see above) you can determine the paper size and orientation of the PDF report under PDF Report Settings. You can also set the Remote PDF Report Server URL for the PDF Report Server App if you plan to use it.
Note: As of release 5.0, Splunk no longer requires the PDF Report Server App to generate search result PDFs, but it can still be used for printing dashboards that are built with Advanced XML. For more information see "Generate PDFs of your reports and dashboards" in the Reporting Manual.)
If you plan to use the PDF Report Server App, the Link hostname field must be the search head hostname for the instance sending requests to a PDF Report Server. Set this option only if Splunk improperly auto-detects the hostname for your environment.
Specify your choices and click Save to have all alerts use these settings for email actions.
If you don't see System settings or Email alert settings in Settings, you do not have permission to edit the settings. In this case, contact your Splunk administrator.
You can also use configuration files to set up email alert settings. You can configure them for your entire Splunk implementation in
alert_actions.conf, and you can configure them at the individual search level in
savedsearches.conf. For more information about
.conf file management of reports and alert settings see "Configure alerts in savedsearches.conf" in this manual.
Run a script
If you want Splunk to run an alert script when the alert triggers, select Run a script under Enable actions and enter the file name of the script that you want Splunk to execute.
For example, you might want an alert to run a script that generates a Simple Network Management Protocol (SNMP) trap notification and sends it to another system such as a Network Systems Management console when its alerting conditions are met. Meanwhile, you could have a different alert that, when triggered, runs a script that calls an API, which in turn sends the triggering event to another system.
Note: For security reasons, all alert scripts must be placed in
$SPLUNK_HOME/etc/<AppName>/bin/scripts. Splunk looks in these two directories for any script triggered by an alert.
For detailed instruction on alert script configuration using
savedsearches.conf in conjunction with shell script or batch file that you create, see "Configure scripted alerts" in this manual.
If you are having trouble with your alert scripts, check out this excellent topic on troubleshooting alert scripts on the Splunk Community Wiki.
Show triggered alerts in the Alert manager
If you want to have the Alert manager keep records of the triggered alerts related to a particular alert configuration, select the List in Triggered Alerts checkbox. The Alert manager will keep records of triggered alerts for the duration specified in the Expiration field in the Alert configuration for the specified alert.
For more information about the Alert manager and how to use it, see the "Review triggered alerts" topic in this manual.
Give tracked alerts a severity level
On the Alert manager page, each alert is labeled with a Severity level that helps people know how important each alert is in relation to other alerts. For example, an alert that lets you know that a server is approaching disk capacity could be given a High label, while an alert triggered by a "disk full" error could have a Critical label.
You can choose from Info, Low, Medium, High, and Critical. The default is Medium.
Severity labels are informational in purpose and have no additional functionality. You can use them to quickly pick out important alerts from the alert listing on the Alert manager page. Get to the Alert manager page by clicking the Triggered Alerts link in the upper right-hand corner of the Splunk bar.
Alert action functionality available in Settings
If you create or update your alert in Settings > Searches and Reports you'll find additional alert action options. For example, you can opt to have alert-triggering results sent to an RSS feed.
Create an RSS feed
If you want Splunk to post this alert to an RSS feed when it triggers, select Enable next to Add to RSS on the detail page for the alerting search in Settings > Searches and Reports.
When an alert with the Add to RSS action enabled triggers, Splunk sends a notification out to its RSS feed. The feed is located at
http://[splunkhost]:[port]/rss/[saved_search_name]. So, let's say you're running a search titled "errors_last15" and have a Splunk instance that is located on
localhost and uses port 8000, the correct link for the RSS feed would be
You can also find links to the RSS feeds for alerting searches at Settings > Searches and reports. Searches that have Add to RSS enabled display an RSS symbol in the RSS feed column:
Click on this symbol to go to the RSS feed.
Note: An RSS feed for an alerting search won't display anything until the alert has been triggered at least once. If the alert is based on a scheduled search that is set to alert each time it is run (it has Perform actions to always), you'll see search information in the RSS feed after first time the search runs on its schedule.
Caution: The RSS feed is exposed to any user with access to the webserver that displays it. Unauthorized users can't follow the RSS link back to the Splunk application to view the results of a particular search, but they can see the summarization displayed in the RSS feed, which includes the name of the search that was run and the number of results returned by the search.
Here's an example of the XML that generates the feed:
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0"> <channel> <title>Alert: errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Reports Feed for report errors last15</description> <item> <title>errors last15</title> <link>http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh</link> <description>Alert trigger: errors last15, results.count=123 </description> <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate> </item> </channel> </rss>
Specify fields to show in alerts through search language
When Splunk provides the results of the alerting search job (in an alert email, for example), it includes all the fields in those results. To have certain fields included in or excluded from the results, use the
fields command in the base search for the alert.
- To eliminate a field from the search results, pipe your search to
fields - $FIELDNAME.
- To add a field to the search results, pipe your search to
fields + $FIELDNAME.
You can specify multiple fields to include and exclude in one string. For example, your Search field may be:
yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4
This generates an alert that excludes
$FIELD2, but includes
Enable summary indexing in Settings
Summary indexing is an action that you can configure for any alert via Settings > Searches and Reports. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.
With summary indexing, you base an alert on a search that computes sufficient statistics (a summary) for events covering a slice of time. The search is set up so that each time it runs on its schedule, the search results are saved into a summary index that you designate. You can then run searches against this smaller (and thus faster) summary index instead of working with the much larger dataset from which the summary index receives its events.
Note: You do not need to use summary indexing for searches that already benefit from report acceleration. For more information and a distinction between these two methods of speeding up slow running searches, see "About report acceleration and summary indexing" in the Knowledge Manager manual.
To set up summary indexing for an alert, go to Settings > Searches and Reports, and either add a new report or open up the detail page for an existing search or alert. (You cannot set up summary indexing through the Create Alert window.) To enable the summary index to gather data on a regular interval, set its Alert condition to always and then select Enable under Summary indexing at the bottom of the view.
Note: There's more to summary indexing--you should take care to properly construct the search that populates the summary index. In most cases special reporting commands should be used. Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.
Update and expand alert functionality in Settings
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15