Alerting Manual

Set up alert actions

You can enable the following alert actions:

• Send email notification.
  The email notification can include information related to the alert.
• Run scripts.
• Enable RSS notification for the alert.
• Enable summary indexing for alerts.
• Track the alert in Splunk Enterprise Settings.

Email notification

You can configure an alert to send an email notification to specified recipients when the alert triggers. The email notification is a multipart MIME message that includes both HTML and text parts.

You configure the email notification action for an alert when you save the alert from the Search page. You can also configure email notification from the Alerts Page and directly from a search command.

Before you can send an email notification, configure the email notification settings in Settings. See Configure email notification settings.

Email notification contexts

There are several contexts from which you can send email notifications. The email options available differ, depending on the context.

• Alert actions
  Send email notifications as an alert action from a search. Specify the notification from the Search Page, a listing in the Alerts Page, or directly from the search command.

• Scheduled report
  Configure email notifications for a scheduled report either from a listing in the Reports Page or from a report.

• Scheduled PDF delivery of dashboards
  Configure PDF delivery either from a listing in the Dashboards Page or from a dashboard.

This topic covers alert actions from a search job. See Schedule reports and Generate Dashboard PDFs for information on the other contexts for email notification.

Configure email notification for alerts

You configure email notifications from the Search Page when you save a search. You can also configure email notifications for an alert listed on the Alerts Page by editing an alert's actions. The procedure is the same as from the Search page.

After running a search from the Search page, save the search as an alert and configure email notification settings.

1. Run the search.
2. Select Save As > Alert.
3. Provide a Title and other information about the alert. Click Next.
4. Select Send Email.
   The Email Actions dialog box opens.
   
5. Specify the following:
   
   
   • To, CC, and BCC email recipients.
     Specify a comma-separated list of email recipients.
   • Priority
     Enforcement of priority depends on your email client.
   • Subject
   • Message
   • Include
     You can include the following items:
     
     Information about the search
       Link to the alert
       Search string
       Trigger condition
       Trigger time
     
     Information about search results
       Link to results
       Inline listing of results, as a table, raw events, or CSV file
       Results as a PDF attachment
       Results as a CSV attachment
6. Specify other alert actions.
   See Run a script and Create an RSS feed.
7. Click Save.

Send email notification from a search command

You can send email notifications directly from the sendemail search command. For example:

index=main | head 5 | sendemail to=example@splunk.com server=mail.example.com subject="Here is an email notification" message="This is an example message" sendresults=true inline=true format=raw sendpdf=true

See the sendemail command listing in the Search Reference for details.

Use tokens in email notifications

A token is a type of variable that represents data generated by a search job. Splunk Enterprise provides tokens that you can use to include information generated by a search in the fields of an email:

• To
• Cc
• Bcc
• Subject
• Message
• Footer

Access the value of a token with the following syntax:

$<token-name>$

For example, place the following token in the subject field of an email notification to reference the search ID of a search job.

Search results from $job.sid$

Tokens available for email notifications

This section lists common tokens you can use in email notifications. There are four categories of tokens that access data generated from a search. The context for using the tokens differ.

Category	Description	Context
Search metadata	Information about the search.	Alert actions from search Scheduled reports Scheduled PDF delivery of dashboards
Search results	Access results of a search	Alert actions from search Scheduled reports
Job information	Data specific to a search job	Alert actions from search Scheduled reports
Server information	Information about the Splunk Enterprise server	Alert actions from search Scheduled reports Scheduled PDF delivery of dashboards

In addition to the common tokens listed in this topic, the savedsearches.conf and alert_action.conf configuration files list attributes whose values are available from tokens. To access these attribute values, place the attribute between the '$' token delimiters. For example, to access the subject of an email notification, reference the following attribute listed in savedsearches.conf:

$action.email.subject$

Tokens that access search metadata

Common tokens that access information about a search. These tokens are available from the following contexts:

• Alert actions
• Scheduled reports
• Scheduled PDF delivery of dashboards

Here are some of the common tokens available.

Token	Description
$action.email.hostname$	Hostname of the email server.
$action.email.priority$	Priority of the search.
$app$	Name of the app containing the search.
$cron_schedule$	Cron schedule for the app.
$description$	Description of the search.
$name$	Name of the search.
$next_scheduled_time$	The next time the search runs.
$owner$	Owner of the search.
$results_link$	(Alert actions and scheduled reports only) Link to the search results.
$search$	The actual search.
$trigger_date$	(Alert actions only) The date that triggers the alert.
$trigger_time$	(Alert actions only) The scheduled time the alert runs.
$type$	Indicates if the search is from an alert, report, view, or the search command.
$view_link$	Link to view the saved report.
$alert.severity$	Severity level of the alert.
$alert.expires$	Time the alert expires.

Tokens available from results

From results, you use the result.<fieldname> token to access the first value of a specified field in search results. This token is available from the following contexts:

• Alert actions
• Scheduled reports

Token	Description
$result.fieldname$	Returns the first value for the specified field name from the first result in the search. The field name must be present in the search.

Tokens that access job information

Common tokens that access data specific to a search job, such as the search ID or messages generated by the search job. These tokens are available from the following contexts:

• Alert actions
• Scheduled reports

Token	Description
$job.earliestTime$	Initial time a search job starts.
$job.eventSearch$	Subset of the search that contains the part of the search before any transforming commands.
$job.latestTime$	Latest time recorded for the search job.
$job.messages$	List of error and debug messages generated by the search job.
$job.resultCount$	Number of results returned by the search job.
$job.runDuration$	Time, in seconds, that the search took to complete.
$job.sid$	Search ID.
$job.label$	Name given to the search job.

Tokens available from server

Common tokens that provide details available from your Splunk Enterprise server. They are available in the following contexts:

• Alert actions
• Scheduled reports
• Scheduled PDF delivery of dashboards

Token	Description
$server.build$	Build number of the Splunk Enterprise instance.
$server.serverName$	Server name hosting the Splunk Enterprise instance.
$server.version$	Version number of the Splunk Enterprise instance.

Deprecated email notification tokens

The following tokens from prior releases of Splunk Enterprise are deprecated.

Token	Description
$results.count$	(Deprecated) Use $job.resultCount$.
$results.url$	(Deprecated) Use $results_link$.
$results.file$	(Deprecated) No equivalent available.
$search_id$	(Deprecated) Use $job.id$.

Configure email notification settings

Before you send an email notification for an alert, configure the email notification settings. Configure email notifications by editing the alert_actions.conf configuration file or from Splunk Web.

To configure email alert settings from a configuration file, see alert_actions.conf.

Configure email alert settings from Splunk Web.

1. From Splunk Web, select Settings > System settings > Email settings.
2. Select Mail Server Settings:
   
   
   • Mail host
     The default is localhost. To schedule PDF delivery requires additional configuration of user roles. See User role configuration to schedule PDF delivery of dashboards.
   • Email security
   • Username
     Password
     User name and password are optional. You do not need to specify these fields to configure email notification.
3. Specify Email Format:
   
   
   • Link hostname
     The host name of the server from which to create URLs for outgoing results.
     
     This is also the search head host name for the instance sending requests to a PDF Report Server. Use the Remote PDF Report Server to print dashboards built with advanced XML. Set this option only if your environment improperly auto-detects the host name. See Dashboards and forms that use advanced XML.
   • Send emails as
     Add an email or string to specify the sender.
   • Email footer
     Text to be added as a footer to each email. You can specify tokens in the email footer. See Use tokens in email notifications.
4. Specify PDF Report Settings.
   
   
   • Report Paper Size
   • Report Paper Orientation
5. Click Save.

User role configuration to schedule PDF delivery of dashboards

For a user to schedule PDF delivery of dashboards, the user role must contain the following capabilities:

• schedule_search
• admin_all_objects
  This capability is required only if the mail host requires log-in credentials.

See About defining roles with capabilities.

Run a script for an alert action

You can run an alert script when a alert triggers. Select Run a script under Enable actions. Enter the file name of the script that you want to run.

For example, you can configure an alert to run a script that generates a Simple Network Management Protocol (SNMP) trap notification. The script sends the notification to another system such as a Network Systems Management console. You can configure a different alert that runs a script that calls an API, which in turn sends the triggering event to another system.

Note: For security reasons, place all alert scripts in either of the following locations:

• $SPLUNK_HOME/bin/scripts
• $SPLUNK_HOME/etc/apps/<AppName>/bin/scripts

For details on alert script configuration using savedsearches.conf with a shell script or batch file that you create, see "Configure scripted alerts" in this manual.

If you are having trouble with alert scripts, see Troubleshooting alert scripts on the Splunk Community Wiki.

Show triggered alerts in the Alert manager

Select the List in Triggered Alerts action to display triggered alerts in the Alert manager. The Alert manager lists details of triggered alerts for 24 hours or a specified duration. See "Review triggered alerts" in this manual.

Give tracked alerts a severity level

When listing a triggered alert, you can specify a Severity level. Severity levels are informational only. They let you group and highlight alerts in the Alert Manager according to the severity levels. You decide which level applies to the alert.

You can choose from the following severity levels. The default level is Medium.

• Info
• Low
• Medium
• High
• Critical

Create an RSS feed

• Note: The RSS feed alert action is not currently supported for search head clusters.

You can add an RSS feed for alert notifications. When the alert triggers, the alert generates notification to the RSS feed. An alert must trigger at least once to generate the RSS feed.

This alert action is available only from Settings.

1. Go to Settings > Searches, reports, and alerts.
2. Select the alert you are updating.
3. Scroll to Alert actions.
4. For Add to RSS, select Enable.
5. Return to Settings > Searches, reports, and alerts.
6. Click the RSS feed icon to subscribe to the feed.
   
   
   You are given several options to subscribe to the feed.

When an alert with the Add to RSS action triggers, it generates a notification to its RSS feed. The feed is located at:

http://[splunkhost]:[port]/[locale]/rss/[alert_name]

For example, here is the location for an RSS feed for an alert named "Errors in the last 24 hours", on a Splunk Enterprise instance using port 8000, and on a machine named "MyHost."

http://MyHost:8000/en-US/rss/Errors%20in%20the%20last%2024%20hours.

In Settings > Searches, reports, and alerts, click the RSS Feed icon to subscribe to the RSS feed.

Caution: The RSS feed is available to any user with access to the web server that displays the feed. Unauthorized users cannot follow the RSS link back to the Splunk Enterprise application to view the results of a specific search. But unauthorized users can see the summarization displayed in the RSS feed. The summarization includes the name of the search that was run and the number of results returned by the search.

This example shows the XML that generates the feed.

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
    <channel>
        <title>Alert: errors last15</title>
        <link>
          http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh
        </link>
        <description>Reports Feed for report errors last15</description>
        <item>
            <title>errors last15</title>
            <link>
              http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh
            </link>
            <description>
              Alert trigger: errors last15, results.count=123
            </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>
        </item>
    </channel>
</rss>

Specify fields to show in alerts through search language

The results of an alerting search job (in an alert email, for example) includes all the fields in those results. To include or exclude specific fields from the results, use the fields command in the base search for the alert.

• To eliminate a field from the search results, pipe your search to fields - $FIELDNAME.
• To add a field to the search results, pipe your search to fields + $FIELDNAME.

You can specify multiple fields in one string. The following search generates an alert that excludes $FIELD1 and $FIELD2, but includes $FIELD3 and $FIELD4.

yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4

Enable summary indexing in Settings

Summary indexing is an action that you can configure for any alert using Settings > Searches and Reports. Use summary indexing to perform analysis/reports on large amounts of data over long timespans. Typically this can be quite time consuming and a drain on performance if several users are running similar searches on a regular basis.

Caution: For summary indexing you typically use reporting commands to properly construct the search that populates the summary index. Before setting up a summary index, read "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

With summary indexing, you base an alert on a search that computes sufficient statistics (a summary) for events covering a slice of time. The search is set up so that each time it runs on its schedule, the search results are saved into a summary index that you designate. You can then run searches against this smaller (and thus faster) summary index instead of working with the much larger dataset from which the summary index receives its events.

Note: You do not need to use summary indexing for searches that already benefit from report acceleration. For more information and a distinction between these two methods of speeding up slow running searches, see "About report acceleration and summary indexing" in the Knowledge Manager manual.

To set up summary indexing for an alert, go to Settings > Searches and Reports, and either add a new report or open up the detail page for an existing search or alert. (You cannot set up summary indexing through the Create Alert window.) To enable the summary index to gather data on a regular interval, set its Alert condition to always and then select Enable under Summary indexing at the bottom of the view.