Send SNMP traps to other systems
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Send SNMP traps to other systems
You can use Splunk as a monitoring tool to send SNMP alerts to other systems such as a Network Systems Management console.
If you're interested in sending SNMP traps on Windows, check this Community Wiki topic.
Configuration
Requirements
- Perl is required to run the script below.
- Net-SNMP package is required in order to use the
/usr/bin/snmptrapcommand - if you have another way of sending an SNMP trap from a shell script then modify as needed. - Admin access to the
$SPLUNK_HOME/bin/scriptsdirectory of your Splunk install. - For security reasons, scripts must reside in $SPLUNK_HOME/bin/scripts.
Create shell script
- Create
traphosts.plscript in your$SPLUNK_HOME/bin/scriptsdirectory.- For security reasons, scripts must reside in this directory. Create the directory if it doesn't already exist.
- Copy the code below into
sendsnmptrap.pl.
-
chmod +x sendsnmptrap.plto make it executable. - Change the
Host:Portof the SNMP trap handler, paths to external commandssplunkandsnmptrap, and the user/password if necessary. - The perl script will work on MS Windows systems with Perl. However, on some Windows systems, perl may not be installed, or perl scripts may not be configured to be directly executable via Splunk. In these cases, you may find it easier to send SNMP traps using a Windows CMD script.
#!/usr/bin/perl # # sendsnmptrap.pl: A script to for Splunk alerts to send an SNMP trap. # # Modify the following as necessary for your local environment # $hostPortSNMP = "qa-tm1:162"; # Host:Port of snmpd or other SNMP trap handler $snmpTrapCmd = "/usr/bin/snmptrap"; # Path to snmptrap, from http://www.net-snmp.org $TRAPOID = "1.3.6.1.4.1.27389.1.2"; # Object IDentifier for traps/notifications $OID = "1.3.6.1.4.1.27389.1.1"; # Object IDentifier for objects, Splunk Enterprise OID is 27389 # Parameters passed in from the alert. # $1-$9 is the positional parameter list. $ARGV[0] starts at $1 in Perl. $searchCount = $ARGV[0]; # $1 - Number of events returned $searchTerms = $ARGV[1]; # $2 - Search terms $searchQuery = $ARGV[2]; # $3 - Fully qualified query string $searchName = $ARGV[3]; # $4 - Name of saved search $searchReason = $ARGV[4]; # $5 - Reason saved search triggered $searchURL = $ARGV[5]; # $6 - URL/Permalink of saved search $searchTags = $ARGV[6]; # $7 - Always empty as of 4.1 $searchPath = $ARGV[7]; # $8 - Path to raw saved results in Splunk instance (advanced) # Send trap, with the the parameter list above mapping down into the OID. $cmd = qq/$snmpTrapCmd -v 2c -c public $hostPortSNMP '' $TRAPOID $OID.1 i $searchCount $OID.2 s "$searchTerms" $OID.3 s "$searchQuery" $OID.4 s "$searchName" $OID.5 s "$searchReason" $OID.6 s "$searchURL" $OID.7 s "$searchTags" $OID.8 s "$searchPath"/; system($cmd);
Configure your alert to call a shell script
- Create a saved search. Read about setting up saved searches for more information.
- Turn your saved search into an alert. Read about setting alert conditions from scheduled searches for more information.
- Set up your alert so that it calls your shell script by specifying the name of your script which resides in $SPLUNK_HOME/bin/scripts:
Here is an example of the script running, including what it returns:
[root@qa-tm1 ~]# snmptrapd -f -Lo
2007-08-13 16:13:07 NET-SNMP version 5.2.1.2 Started.
2007-08-13 16:14:03 qa-el4.splunk.com [172.16.0.121] (via UDP: [172.16.0.121]:32883) TRAP, SNMP v1, community public
SNMPv2-SMI::enterprises.27389.1 Warm Start Trap (0) Uptime: 96 days, 20:45:08.35
SNMPv2-SMI::enterprises.27389.1.1 = INTEGER: 7 SNMPv2-
SMI::enterprises.27389.1.2 = STRING: "sourcetype::syslog" SNMPv2-
SMI::enterprises.27389.1.3 = STRING: "search sourcetype::syslog starttime:12/31
/1969:16:00:00 endtime::08/13/2007:16:14:01" SNMPv2-SMI::enterprises.27389.1.4
= STRING: "SyslogEventsLast24" SNMPv2-SMI::enterprises.27389.1.5 = STRING:
"Saved Search [SyslogEventsLast24]: The number of hosts(7) was greater than 1"
SNMPv2-SMI::enterprises.27389.1.6 = STRING: "http://qa-el4:18000/?q=sourcetype
%3a%3asyslog%20starttimeu%3a%3a0%20endtimeu%3a%3a1187046841" SNMPv2-
SMI::enterprises.27389.1.7 = STRING: "/home/tet/inst/splunk/var/run/splunk
/SyslogEventsLast24"
2007-08-13 16:14:15 NET-SNMP version 5.2.1.2 Stopped.
This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 View the Article History for its revisions.

The SNMP scripts above won't work if you have complex search terms.
Lets say your saved search looks like this:
source=mylog/* server instance |rex Instance (?.*?)
This command won't execute and the SNMP trap will silently fail to be sent. This of course means that when something went totally wrong in your network you won't be alerted !
The quick fugly work around is to not send the search terms in the trap. I made the following change:
#$searchTerms = $ARGV[1]; # $2 - Search terms
#$searchQuery = $ARGV[2]; # $3 - Fully qualified query string
$searchTerms = "N/A"; # $2 - Search terms
$searchQuery = "N/A"; # $3 - Fully qualified query string
Totally ugly but it keeps traps from going into the black hole. the real fix should be to properly escape the content. As well as do things like length checks to make sure you don't piss off net-snmp.