How alerting works
Alerts are searches you've configured to run on a schedule and send you their results. Use alerts to notify you of changes in your data, network infrastructure, file system or other devices you're monitoring. Alerts can be sent via email or RSS, or trigger a shell script. You can turn any saved search into an alert.
An alert consists of:
- a schedule for performing the search
- conditions for triggering an alert
- actions to perform when the triggering conditions are met
Set up an alert at the time you create a saved search, or enable an alert on any existing saved search. Configure both basic and advanced conditional alerts for searches by:
- Scheduling and defining alerts for saved searches through Splunk Web (if you have permission to edit them).
- Entering or updating saved search configurations in
savedsearches.conf. For more information, see "Set up alerts in savedsearches.conf", in this chapter.
Specify default email alert action settings
To specify email alert settings, go to Splunk Web and navigate to Manager > System settings > Email alert settings.
Under Mail server settings you can enter or update information related to the SMTP server that Splunk interacts with in order to send out alert emails. Identify the SMTP mail host server. Provide an authentication username/password if the SMTP server requires them. Optionally specify that Splunk use SSL or TLS when it communicates with the SMTP server. If you specify SSL, you must include the port number in the mail host server field; for example, "smtp.gmail.com:465".
Under Email format you can enter information about the format of the emails that Splunk sends. You can define the name that appears in the "sender" field (by default it is Splunk), and you can set up the format of the email subject line (by default it is Splunk Alert: $name$, where $name$ is the name of the search that the alert is based upon). You can also set at the Manager level the default email format for all alerts and whether or not alert emails provide inline results.
The Email format section also lets you provide the hostname that creates outgoing results URLs. But in most circumstances you can leave this field blank as Splunk can autodetect the proper hostname.
You can also set default settings for alert actions (including scripted alerts) by making changes directly to alert_actions.conf.
Note: If you add a clear text password in
alert_actions.conf, using the
auth_password attribute, restart Splunk. Splunk will encrypt the password upon restart.
PDF report settings
On the Email alert settings page in Manager (see the preceding subtopic), select Use PDF report server to open the PDF report settings section. This is where you enable the ability to have .pdf printouts of report results sent as attachments with alert emails.
Here you can define a remote PDF report server if necessary (you do not need to do this for a local PDF print server). You can also define the report paper size and orientation. Keep in mind that these settings are for all PDF reports. To set this stuff up at the individual alert level you should go to
savedsearches.conf and change the settings for specific saved search stanzas.
Note: You must have the PDF Printer app set up on a central Linux host before you can enable the PDF printing functionality here. For more information see "Configure PDF printing for Splunk Web" in the Installation manual.
Alerts can also trigger shell scripts. When you configure an alert, specify a script you've written. You can use this feature to send alerts to other applications. Learn more about configuring scripted alerts.
You can use scripted alerts to send syslog events, or SNMP traps.
When configuring alerts, keep the following in mind:
- Too many alerts/saved searches running at once may slow down your system -- depending on the hardware, 20-30 alerts running at once should be OK. If the searches your alerts are based on are complex, you should make the interval longer and spread the searches out more.
- Set a time frame for alerts that makes sense -- if the search takes longer than 4-5 minutes to run, don't set it to run every five minutes.
- You must have a mail server running on the LAN that the Splunk server can connect to. Splunk does not authenticate against the mail server.
- Read more about best practices for alert configuration on the Splunk Community Wiki, here.
Configure bloom filters
Set up alerts in savedsearches.conf
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7