How to use Splunk 2 Nagios

How to use Splunk 2 Nagios

For the most recent documentation on using Splunk with Nagios, see the Apps Wiki.

Nagios is an open source monitoring system designed to monitor resources using assorted plugins and SNMP. Out of the box, it supports monitoring of hosts and network services directly over tcp/ip, resource monitoring via SNMP scripts, and the framework required to build your own plugins. For information on writing custom plugins, refer to the development site for Nagios plugins: http://nagiosplug.sourceforge.net/

What Nagios logs

Nagios has a default log file called nagios.log kept in the base directory under var. If your base directory is /usr/pkg/nagios, the location is /usr/pkg/nagios/var/nagios.log .

Service and host events are logged to this main log file for historical purposes. Configure the log file name and location via the log_file directive in nagios.cfg.

To log messages to the syslog facility as well, set the use_syslog option to 1. Otherwise set it to 0.

Nagios also keeps several status files that it dynamically updates during operation. Splunk recommends parsing these status files both from Nagios itself from standalone scripts.

The status files are:

• downtime.dat
• status.dat

These are normally kept in the var directory under your Nagios base directory.

Install the Splunk2Nagios integration

Make sure you have the integration components found on SplunkBase

Make sure you have root permissions by either becoming root or using sudo.

Method 1 : convert the Nagios components manually

To convert the Nagios components by hand, do the following:

• Unpack splunk2nagios.tar.gz.

       tar xvfz splunk2nagios.tar.gz

• Change to the new directory.

       cd splunk2nagios

• Copy the .in files to .sh

     cp handle_live_splunk.in handle_live_splunk.sh

• Make the .in files executable.

     chmod 755 handle_live_splunk.sh

etc.

• Continue to the next section.

Method 2 : perform a modified installation of the Nagios components

• Unpack splunk2nagios.tar.gz.
• Change to the new directory

	cd splunk2nagios

• Run make.

The system prompts you for locations and users of nagios. When this completes, use the following instructions:

Modify scripts to work with Splunk 3.x

• Change the following stanza to point to your Splunk install directory:

if [ -z "${SPLUNK_HOME}" ] ; then
        SPLUNK_HOME=/opt/splunk  <-- If you installed to an alternate location, you will need to specify here.
fi

• Edit the following stanza and point the commands to where they are on your system:

DEFAULT_CAT_COMMAND="/bin/cat"
DEFAULT_GREP_COMMAND="/usr/bin/grep"
DEFAULT_SED_COMMAND="/usr/bin/sed"
DEFAULT_ECHO_COMMAND="/bin/echo"
DEFAULT_DATE_COMMAND="/bin/date"

• Change

SPLUNK_HTTP_PORT=`$CAT_COMMAND $SPLUNK_SEARCH_CONF | $GREP_COMMAND httpPort | $SED_COMMAND -e 's/.*<[a-zA-Z]*>\([0-9]*\)<\/[a-zA-Z]*>.*/\1/g'`

to

SPLUNK_HTTP_PORT=8000

• Change

SPLUNK_URL="${SPLUNK_BASEURL}:${SPLUNK_HTTP_PORT}/?events/?eventspage=1&num=20&q="

to

SPLUNK_URL="${SPLUNK_BASEURL}:${SPLUNK_HTTP_PORT}/q="

For splunk_host_notification.sh and splunk_service_notification, do the following :

• Modify SPLUNK_HOME= to your Splunk path by changing

SPLUNK_HTTP_PORT=`cat $SPLUNK_SEARCH_CONF | grep httpPort | sed -e 's/.*<[a-zA-Z]*>\([0-9]*\)<\/[a-zA-Z]*>.*/\1/g'`

to

SPLUNK_HTTP_PORT=8000 (or what ever port you set it to)

• Change

 SPLUNK_URL="${SPLUNK_BASEURL}:${SPLUNK_HTTP_PORT}/?events/?eventspage/?q=" 

  to

 SPLUNK_URL="${SPLUNK_BASEURL}:${SPLUNK_HTTP_PORT}/?q="

• Make sure mail and echo are in /bin/. If they are not, edit this line to reflect the correct path :

/bin/echo -e "${MESSAGEBODY}" | /bin/mail -s "${MESSAGESUBJECT}" "$8" 

• For check_splunk.sh
  • Make sure the following list of binaries are in their respective paths:

# Paths to utilities
DEFAULT_PS_COMMAND="/bin/ps"
DEFAULT_NETSTAT_COMMAND="/bin/netstat"
DEFAULT_CAT_COMMAND="/bin/cat"
DEFAULT_GREP_COMMAND="/bin/grep"
DEFAULT_SED_COMMAND="/bin/sed"
DEFAULT_WC_COMMAND="/usr/bin/wc"
DEFAULT_FIND_COMMAND="/usr/bin/find"
DEFAULT_LS_COMMAND="/bin/ls"
DEFAULT_HEAD_COMMAND="/usr/bin/head"
DEFAULT_TAIL_COMMAND="/usr/bin/tail"
DEFAULT_AWK_COMMAND="/usr/bin/awk"
DEFAULT_ECHO_COMMAND="/bin/echo"

• • Change SPLUNK_HOME to the location you've installed it in.
  • Make sure the base url is the proper IP address of your splunk server
  • Change the following line :

SPLUNKSEARCH_NAME="splunkSearch"

to

SPLUNKSEARCH_NAME="SplunkWeb.tac"

SEARCH_URL="${SPLUNK_BASEURL}:${HTTP_PORT}/?events/?eventspage=1&num=20&q=" 

to

SEARCH_URL="${SPLUNK_BASEURL}:${HTTP_PORT}/?q="

• • Create a SimpleSearchScript in $SPLUNK_HOME/bin that has the proper privs. This is a simple search that would be performed by running ./splunk search "foo bar NOT (Foo|Bar)"

    echo "foo bar NOT (Foo|Bar)" >> $SPLUNK_HOME/bin/SimpleSearchScript

Configure handle_live_splunk

Part of the Splunk2Nagios integration is the feature that sends Splunk alerts to Nagios.

To get alerts into Nagios from Splunk

• Make a directory in $SPLUNK_HOME/bin/ called scripts.
• Place the handle_live_splunk script into $SPLUNK_HOME/bin/scripts.
• For each Splunk alert that you want to monitor with Nagios, create a different handle_live_splunk.
• Edit the following lines for each version:

# Default Nagios variables
DEFAULT_NAGIOS_COMMAND_FILE="/usr/local/nagios/var/rw/nagios.cmd"
DEFAULT_NAGIOS_HOSTNAME="HOST THIS ALERT IS FROM"
DEFAULT_NAGIOS_SERVICEDESCRIPTION="Alert: Nagios"
DEFAULT_NAGIOS_RETURNCODE=2 <-- This is important.. 1 is warning, 2 critical 3 nothing
DEFAULT_NAGIOS_OUTPUTPREPEND="An Alert was just triggered!"

Within your service definition file, create a stanza like this :

define service{
        use                             local-service
        host_name                       HOST YOURE ALERTING FOM
        service_description             Alert: Nagios
        is_volatile                     0
        check_period                    none
        max_check_attempts              4
        normal_check_interval           10
        retry_check_interval            5
        contact_groups                  admins
        notification_options            w,u,c,r
        notification_interval           10
        notification_period             24x7
        check_command                   check_none
        active_checks_enabled           0
}

• Was this topic useful?
• Post a comment