Alerting Manual

 


Set up alert actions

Set up alert actions

You can enable the following alert actions:

  • Send email notification.
    The email notification can include information related to the alert.
  • Run scripts.
  • Enable RSS notification for the alert.
  • Enable summary indexing for alerts.
  • Track the alert in Splunk Enterprise Settings.

Email notification

You can configure an alert to send an email notification to specified recipients when the alert triggers. The email notification is a multipart MIME message that includes both HTML and text parts.

You configure the email notification action for an alert when you save the alert from the Search page. You can also configure email notification from the Alerts Page and directly from a search command.

Before you can send an email notification, configure the email notification settings in Settings. See Configure email notification settings.

Email notification contexts

There are several contexts from which you can send email notifications. The email options available differ, depending on the context.

  • Alert actions
    Send email notifications as an alert action from a search. Specify the notification from the Search Page, a listing in the Alerts Page, or directly from the search command.
  • Scheduled report
    Configure email notifications for a scheduled report either from a listing in the Reports Page or from a report.
  • Scheduled PDF delivery of dashboards
    Configure PDF delivery either from a listing in the Dashboards Page or from a dashboard.

This topic covers alert actions from a search job. See Schedule reports and Generate Dashboard PDFs for information on the other contexts for email notification.

Configure email notification for alerts

You configure email notifications from the Search Page, when saving a search. You can also configure email notifications for an alert listed on the Alerts Page by editing an alert's actions. The procedure is the same as from the Search page.

After running a search from the Search page, save the search as an alert and configure email notification settings:

  1. Run the search.
  2. Select Save As > Alert.
  3. Provide a Title and other information about the alert. Click Next.
  4. Select Send Email.
    The Email Actions dialog opens.
    AlertEmailNotification.png
  5. Specify the following:

    • To, CC, and BCC email recipients.
      Specify a comma-separated list of email recipients.
    • Priority
      Enforcement of priority depends on your email client.
    • Subject
    • Message
    • Include items
      You can include the following items:

      Information about the search
        Link to the alert
        Search string
        Trigger condition
        Trigger time

      Information about search results
        Link to results
        Inline listing of results, as a table, raw events, or CSV file
        Results as a PDF attachment
        Results as a CSV attachment
  6. Specify other alert actions.
    See Run a script and Create an RSS feed.
  7. Click Save.

Send email notification from a search command

You can send email notifications directly from the sendemail search command. For example:

index=main | head 5 | sendemail to=example@splunk.com server=mail.example.com subject="Here is an email notification" message="This is an example message" sendresults=true inline=true format=raw sendpdf=true

See the sendemail command listing in the Search Reference for details.

Use tokens in email notifications

A token is a type of variable that represents data generated by a search job. Splunk Enterprise provides various tokens that you can use to include information generated by a search in the fields of an email:

  • To
  • Cc
  • Bcc
  • Subject
  • Message
  • Footer

Access the value of a token with the following syntax:

$<token-name>$

For example, place the following token in the subject field of an email notification to reference the search ID of a search job.

Search results from $job.sid$

Tokens available for email notifications

This section lists common tokens you can use in email notifications. There are four categories of tokens that access data generated from a search. The context for using the tokens differ.

Category Description Context
Search metadata Information about the search. Alert actions from search
Scheduled reports
Scheduled PDF delivery of dashboards
Search results Access results of a search Alert actions from search
Scheduled reports
Job information Data specific to a search job Alert actions from search
Scheduled reports
Server information Information about the Splunk Enterprise server Alert actions from search
Scheduled reports
Scheduled PDF delivery of dashboards

In addition to the common tokens listed in this topic, the savedsearches.conf and alert_action.conf configuration files list attributes whose values are available from tokens. To access these additional attribute values, place the attribute between the '$' token delimiters. For example, to access the subject of an email notification, reference the following attribute listed in savedsearches.conf:

$action.email.subject$

Tokens that access search metadata

Common tokens that access information about a search. These tokens are available from the following contexts:

  • Alert actions
  • Scheduled reports
  • Scheduled PDF delivery of dashboards

Here are some of the common tokens available.

Token Description
$action.email.hostname$ Hostname of the email server.
$action.email.priority$ Priority of the search.
$app$ Name of the app containing the search.
$cron_schedule$ Cron schedule for the app.
$description$ Description of the search.
$name$ Name of the search.
$next_scheduled_time$ The next time the search runs.
$owner$ Owner of the search.
$results_link$ (Alert actions and scheduled reports only) Link to the search results.
$search$ The actual search.
$trigger_date$ (Alert actions only) The date that triggers the alert.
$trigger_time$ (Alert actions only) The scheduled time the alert runs.
$type$ Indicates if the search is from an alert, report, view, or the search command.
$view_link$ Link to view the saved report.
$alert.severity$ Severity level of the alert.
$alert.expires$ Time the alert expires.

Tokens available from results

From results, you use the result.<fieldname> token to access the first value of a specified field in search results. This token is available from the following contexts:

  • Alert actions
  • Scheduled reports
Token Description
$result.fieldname$ Returns the first value for the specified field name from the first result in the search. The field name must be present in the search.

Tokens that access job information

Common tokens that access data specific to a search job, such as the search ID or messages generated by the search job. These tokens are available from the following contexts:

  • Alert actions
  • Scheduled reports
Token Description
$job.earliestTime$ Initial time a search job starts.
$job.eventSearch$ Subset of the search that contains the part of the search before any transforming commands.
$job.latestTime$ Latest time recorded for the search job.
$job.messages$ List of error and debug messages generated by the search job.
$job.resultCount$ Number of results returned by the search job.
$job.runDuration$ Time, in seconds, that the search took to complete.
$job.sid$ Search ID.
$job.label$ Name given to the search job.

Tokens available from server

Common tokens that provide details available from your Splunk Enterprise server. They are available in the following contexts:

  • Alert actions
  • Scheduled reports
  • Scheduled PDF delivery of dashboards
Token Description
$server.build$ Build number of the Splunk Enterprise instance.
$server.serverName$ Server name hosting the Splunk Enterprise instance.
$server.version$ Version number of the Splunk Enterprise instance.

Deprecated email notification tokens

The following tokens from prior releases of Splunk Enterprise are deprecated.

Token Description
$results.count$ (Deprecated) Use $job.resultCount$.
$results.url$ (Deprecated) Use $results_link$.
$results.file$ (Deprecated) No equivalent available.
$search_id$ (Deprecated) Use $job.id$.

Configure email notification settings

Before you send an email notification for an alert, configure the email notification settings. Configure email notifications by editing the alert_actions.conf configuration file or from Splunk Web.

To configure email alert settings from a configuration file, see alert_actions.conf.

Configure email alert settings from Splunk Web.

  1. From Splunk Web, select Settings > System settings > Email settings.
  2. Select Mail Server Settings:

    • Mail host
      The default is localhost. To schedule PDF delivery requires additional configuration of user roles. See User role configuration to schedule PDF delivery of dashboards.
    • Email security
    • Username
      Password
      Username and password are optional. You do not need to specify these fields to configure email notification.
  3. Specify Email Format:

    • Link hostname
      The hostname of the server from which to create URLs for outgoing results.

      This is also the search head hostname for the instance sending requests to a PDF Report Server. Use the Remote PDF Report Server to print dashboards built with advanced XML. Set this option only if Splunk improperly auto-detects the hostname for your environment. See Dashboards and forms that use advanced XML.
    • Send emails as
      The From field in the email.
    • Email footer
      Text to be added as a footer to each email. You can specify tokens in the email footer. See Use tokens in email notifications.
  4. Specify PDF Report Settings.

    • Report Paper Size
    • Report Paper Orientation
  5. Click Save.

User role configuration to schedule PDF delivery of dashboards

For a user to schedule PDF delivery of dashboards, the user role must contain the following capabilities:

  • schedule_search
  • admin_all_objects
    This capability is required only if the mail host requires log-in credentials.

See About defining roles with capabilities.

Run a script

You can run an alert script when a alert triggers. Select Run a script under Enable actions. Enter the file name of the script that you want to run.

For example, you can configure an alert to run a script that generates a Simple Network Management Protocol (SNMP) trap notification, and sends the notification to another system such as a Network Systems Management console. You can configure a different alert that runs a script that calls an API, which in turn sends the triggering event to another system.

Note: For security reasons, all alert scripts must be placed in either of the following locations:

  • $SPLUNK_HOME/bin/scripts
  • $SPLUNK_HOME/etc/<AppName>/bin/scripts

For detailed instruction on alert script configuration using savedsearches.conf in conjunction with shell script or batch file that you create, see "Configure scripted alerts" in this manual.

If you are having trouble with alert scripts, see this topic on troubleshooting alert scripts on the Splunk Community Wiki.

Show triggered alerts in the Alert manager

Select the List in Triggered Alerts action to display triggered alerts in the Alert manager. The Alert manager lists details of triggered alerts for 24 hours or a specified duration. See "Review triggered alerts" topic in this manual.

Give tracked alerts a severity level

When listing a triggered alert, you can specify a Severity level. Severity levels are informational only. They let you group and highlight alerts in the Alert Manager according to the severity levels. You decide which level applies to the alert.

You can choose from the following severity levels. The default level is Medium.

  • Info
  • Low
  • Medium
  • High
  • Critical

Create an RSS feed

You can add an RSS feed for alert notifications. When the alert triggers, the alert generates notification to the RSS feed. An alert must trigger at least once to generate the RSS feed.

This alert action is only available from Settings.

To post an alert to an RSS feed:

  1. Go to Settings > Searches, reports, and alerts.
  2. Select the alert you are updating.
  3. Scroll to Alert actions.
  4. For Add to RSS, select Enable.
  5. Return to Settings > Searches, reports, and alerts.
  6. Click the RSS feed icon to subscribe to the feed.

    Alert rss.png
    You are given several options to subscribe to the feed.


When an alert with the Add to RSS action triggers, it generates a notification to its RSS feed. The feed is located at:

http://[splunkhost]:[port]/[locale]/rss/[alert_name]

For example, here is the location for an RSS feed for an alert named "Errors in the last 24 hours", on a Splunk Enterprise instance using port 8000, and on a machine named "MyHost."

http://MyHost:8000/en-US/rss/Errors%20in%20the%20last%2024%20hours.

In Settings > Searches, reports, and alerts, click the RSS Feed icon to subscribe to the RSS feed.

Caution: The RSS feed is available to any user with access to the webserver that displays the feed. Unauthorized users cannot follow the RSS link back to the Splunk Enterprise application to view the results of a particular search. But unauthorized users can see the summarization displayed in the RSS feed. The summarization includes the name of the search that was run and the number of results returned by the search.

Here's an example of the XML that generates the feed:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
    <channel>
        <title>Alert: errors last15</title>
        <link>
          http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh
        </link>
        <description>Reports Feed for report errors last15</description>
        <item>
            <title>errors last15</title>
            <link>
              http://localhost:8000/app/search/@go?sid=scheduler_Z2d1cHRh
            </link>
            <description>
              Alert trigger: errors last15, results.count=123
            </description>
            <pubDate>Mon, 01 Feb 2010 12:55:09 -0800</pubDate>
        </item>
    </channel>
</rss>

Specify fields to show in alerts through search language

When Splunk provides the results of the alerting search job (in an alert email, for example), it includes all the fields in those results. To have certain fields included in or excluded from the results, use the fields command in the base search for the alert.

  • To eliminate a field from the search results, pipe your search to fields - $FIELDNAME.
  • To add a field to the search results, pipe your search to fields + $FIELDNAME.

You can specify multiple fields to include and exclude in one string. For example, your Search field may be:

yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4

This generates an alert that excludes $FIELD1 and $FIELD2, but includes $FIELD3 and $FIELD4.

Enable summary indexing in Settings

Summary indexing is an action that you can configure for any alert via Settings > Searches and Reports. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.

With summary indexing, you base an alert on a search that computes sufficient statistics (a summary) for events covering a slice of time. The search is set up so that each time it runs on its schedule, the search results are saved into a summary index that you designate. You can then run searches against this smaller (and thus faster) summary index instead of working with the much larger dataset from which the summary index receives its events.

Note: You do not need to use summary indexing for searches that already benefit from report acceleration. For more information and a distinction between these two methods of speeding up slow running searches, see "About report acceleration and summary indexing" in the Knowledge Manager manual.

To set up summary indexing for an alert, go to Settings > Searches and Reports, and either add a new report or open up the detail page for an existing search or alert. (You cannot set up summary indexing through the Create Alert window.) To enable the summary index to gather data on a regular interval, set its Alert condition to always and then select Enable under Summary indexing at the bottom of the view.

Note: There's more to summary indexing--you should take care to properly construct the search that populates the summary index. In most cases special reporting commands should be used. Do not attempt to set up a summary index until you have read and understood "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

This documentation applies to the following versions of Splunk: 6.1 , 6.1.1 , 6.1.2 , 6.1.3 , 6.1.4 View the Article History for its revisions.


Comments

In response to Orange, the preferred method to edit a search string for an alert is the following:

1. Go the Alerts page.
2. Select Open in Search for the alert you want to modify.
3. Modify the Search.
4. Run the Search.
5. Select Save.

To edit email actions:

1. Select the Alert from Alerts page.
2. For Actions, click Edit.
3. Click Send Email and modify the email actions.

Vgenovese
June 3, 2014

So we do have to change the search string on one page and the email action on another?

0range
May 30, 2014

You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!