Set up alerts

Set up alerts

An alert is comprised of

• conditions for triggering an alert
• actions to perform when the triggering conditions are met
• an optional schedule for performing the search.

You can specify that an alert be sent via email or RSS, or used to trigger a shell script. You can turn any saved search into an alert.

via SplunkWeb

You can set up an alert at the time you create a saved search, or you can enable an alert on any existing saved search you have permission to edit.

Set up an alert at the time you save a search:

1. Enter your search terms into the search bar and choose Save search... from the drop-down menu to the left of the search bar. The Save Search popup is displayed.

1. Fill in the fields to save your search and then click the Schedule & Alerts link at the top of the Save Search popup.

Follow the instructions in the Alert Options section below to continue.

Set up an alert on an existing saved search:

1. From the drop-down menu to the left of the search bar, choose Saved searches > Manage saves searches. This will launch the saved searches window.

1. In the table, locate the saved search that you want to turn into an alert.

1. Click enable in the Running column.

• • If you do not have permission to edit this search, the Running column will show *No*.
  • If there is already an alert defined for this saved search, it will either be Running or give the option to start it if you have the proper permissions.

To set up an alert, click the box next to Run this search on a schedule under Schedule & Alerts.

Follow the instructions in the Alert Options section below to continue.

View alert history

The alert history page shows what alerts have been triggered on the server since it was last initialized. To access, click the Admin link in the upper right hand corner and select the Saved Searches tab. Your alerts will show up in the Alert History column.

Alert Options

Schedule

Fill in the blanks to run the alert at regular intervals.

• Note: Too many searches running every minute or less can slow down the server.
• Click Advanced scheduling to specify cron-style scheduling.
  • Please note: Splunk supports most standard cron notation. For example:
  • enter */60 * * * 1-5 to run your search every minute, Monday through Friday.

Alert rules

• Fill in the blanks to configure the Alert to trigger. The final value is a digit, for example Create an alert if the number of events is greater than 3.

Alert actions

• Splunk can send email, create an RSS feed, or run a shell command when an alert triggers. There are multiple variables you can pass to an email or shell script.
• You may configure additional options through alert_actions.conf, including:
  • Set the maximum number of results sent out during an alert by configuring.
  • Which email address originates the alert email.
  • See alert_actions.conf for details.

Specify which fields to show

When you receive alerts, any fields included in your search will also be displayed. You can edit the saved search to change which fields are displayed in your alert.

To eliminate a field, pipe your search to fields - $FIELDNAME. To add a field, pipe your search to fields + $FIELDNAME. You can add or subtract any number of fields -- just separate them with a comma: fields - $FIELD1, $FIELD2 + $FIELD3, $FIELD4.

For example:

This search will keep the sourcetype field from appearing in your alerts.

via configuration files

You can configure the above aspects through savedsearches.conf. Copy savedsearches.conf from $SPLUNK_HOME/etc/bundles/default/ into $SPLUNK_HOME/etc/bundles/local/. Open $SPLUNK_HOME/etc/bundles/local/savedsearches.conf file. You can edit all of the fields below:

[<Splunk name>]
        * Name of the saved Splunk (header)
query = <string>
        * Actual query terms of this saved Splunk (eg. index::sampledata http NOT 500)
isglobal = <integer>
        * If isglobal is set to 1, everyone can see/use this Splunk
        Possible values: 1/0.
counttype = <string>
    * Which count to compare the quantity against relation
    Possible values: "number of events", "number of hosts",
        "number of sources", "number of sourcetypes"
relation = <string>
    * How to compare against count
    Possible values: "greater than", "less than", "equal to",
        "drops by", "rises by"
quantity = <integer>
    * Number to compare against the number returned for the given counttype
schedule = <string>
    * Cron style schedule (i.e. */12 * * * *)
action_script = <string>
    * Name of shell script to run
action_rss = <integer>
    * Whether or not to create an rss link
    Possible values: 1/0
action_email = <string>
    * Comma delimited list of email addresses to send to
sendresults = <integer>
    * Whether or not to send the results along with the email/shell script
    Possible values: 1/0

For example:

[sudoalert]
action_rss = 1
counttype = number of events
disabled = false
enableSched = 1
isGlobal = 0
quantity = 0
query = sudo
relation = greater than
schedule = */12 * * * *
sendresults = 0
userid = 1

Script options

Your alert can trigger a shell script.

Specify the name of the script file to run when a saved search triggers an alert. You don't need to provide the path to the script if the script is located under $SPLUNK_HOME/bin/scripts/.

Command line arguments passed to the script are:

$1 - number of events returned when search run
$2 - searched terms saved at the time saved search was created
$3 - fully qualified query string, the way query is run internally
$4 - saved search name
$5 - trigger reason for the alert
$6 - encoded http link to the saved search results
$7 - tags that have been saved against this saved search, if there are any.

For examples on how scripts can be configured to work with alerts, please see send SNMP traps and send syslog events.

• Was this topic useful?
• Post a comment