Splunk-2-Smarts

Splunk-2-Smarts

Integrated IT Data Search with EMC Smarts and Splunk

Splunk-2-Smarts is an integrated module that provides seamless workflow and data integration between Splunk Professional and EMC Smarts SAM (Service Assurance Manager).

Live Splunk alerts can be forwarded to the Smarts console. Live Splunks are scheduled searches with optional alerting rules. When you locate specific event types or strings in log events that can cause service problems or require attention, such as an interfaces reset, save your search and make it live.

Splunk is the first place to turn when your monitoring system sends an alert or your users report a problem. Launch Splunk directly from the Smarts console to investigate and solve problems faster and avoid recurring problems.

System Requirements

• Splunk 2.2+ on any supported platform.
• EMC Smarts SAM Service Assurance Manager 6.5 or 7.0 on any supported platform.
• Smarts SNMP Trap Adapter

In addition you will need local administrator access to both the Splunk and the Smarts installed directories.

The right-click Client Tools are supported in the Smarts SAM fat client in both 6.5 and 7.0, you must upgrade to version 7.0 if you want this functionality in the Web Console applet.

These instructions are for Smarts SAM installed on Windows or Linux/Unix, with OS specific details as necessary. It has been tested with EMC Smarts SAM running on Windows Server 2003, and RedHat Linux ES 3.8, and it is anticipated to work with few changes on other supported Unix systems such as Solaris. Wherever you see the environment variable $SM_HOME or %SM_HOME% it means the Smarts SAM installation directory, which by default is usually:

• C:\InCharge6 or C:\InCharge7 on Windows
• /opt/InCharge6 or /opt/Incharge7 on Linux/Unix

Receive Live Splunk Alerts in Smarts SAM console

Live Splunks are scheduled searches with optional alerting rules. When you locate specific event types or strings in log events that can cause service problems or require attention, such as an interfaces reset, save your search and make it live.

Configure Smarts to Receive SNMP trap

Add the following to trap_mgr.conf, located in $SM_HOME/SAM/smarts/local/icoi/trap_mgr.conf on Linux/Unix, or %SM_HOME\SAM\smarts\local\icoi\trap_mgr.conf on Windows:

BEGIN_TRAP .1.3.6.1.4.1.27389.1 6 1
        ClassName:      Host
        InstanceName:   $V6$
        EventName:      Splunk: $V4$
        Severity:       2
        EventText:      Splunk URL: $V3$
        Expiration:     7200
        State:          NOTIFY
        ClearOnAcknowledge:     TRUE
        EventType:      DURABLE
        SysNameOrAddr:  $SYS$
        UnknownAgent:   CREATE
        UserDefined1:   Splunk Terms: $V2$
        UserDefined2:   Splunk Reason: $V5$
        UserDefined3:   Splunk URL: $V3$
        LogFile:        splunk.log
END_TRAP

This maps values from the SNMP trap to fields in Smarts SAM. Take care to use spaces instead of tab characters, and also make sure you are using proper CRLF if the host is Windows. You may wish to adjust certain values such as Expiration or even map fields differently as necessary for your local environment.

Restart the Smarts SNMP Trap Adapter in order for this to take effect as follows:

Linux/Unix

$SM_HOME/SAM/smarts/bin/sm_service stop ic-trapd-receiver

$SM_HOME/SAM/smarts/bin/sm_service start ic-trapd-receiver

Windows

%SM_HOME%\SAM\smarts\bin\sm_service.exe stop ic-trapd-receiver

%SM_HOME%\SAM\smarts\bin\sm_service.exe start ic-trapd-receiver

or use the Services GUI to restart "SMARTS SNMP Trap Adapter", or the following:

net stop "SMARTS SNMP Trap Adapter"

net start "SMARTS SNMP Trap Adapter"

The above is for version 6.5, for version 7.0 the service is called "EMC SMARTS SNMP Trap Adapter".

Test SNMP trap

This step is optional, you could also trigger a Live Splunk on your Splunk machine to test, by following the directions in the next section.

To test, login to the INCHARGE-SA Manager, then run the following command on the Smarts SAM machine:

$SM_HOME/SAM/smarts/bin/sm_snmp -d localhost -p 9000 trap localhost .1.3.6.1.4.1.27389.1 6 1 1 .1.3.6.1.4.1.27389.1.1 i 1 .1.3.6.1.4.1.27389.1.2 s "search terms here" .1.3.6.1.4.1.27389.1.3 s "http://email.demo.splunk.com/" .1.3.6.1.4.1.27389.1.4 s "test splunk" .1.3.6.1.4.1.27389.1.5 s "Number of events (1) is greater than 0." .1.3.6.1.4.1.27389.1.6 s "`hostname -s`" .1.3.6.1.4.1.27389.1.7 s "Splunk Test SNMP"

The trap should show up in the Notification Log - double click the event to see details.

Test SNMP

Useful log files to check for errors are:

$SM_HOME/SAM/smarts/local/logs/sm_trapd.log

$SM_HOME/SAM/smarts/local/logs/splunk.log

Send an SNMP Trap from a Live Splunk

Follow the instructions here to Send an SNMP Trap from a Live Splunk. You will need to modify the following line of the traphosts.pl script to point at the Host:Port of the machine that is running your Smarts SNMP Trap Adapter:

$hostPortSNMP = "snmphost:162";

The default is usually Port 9000, you can check to make sure by finding the file trapd.conf in one of two places:

$SM_HOME/SAM/smarts/conf/trapd/trapd.conf

or if it's been modified try here:

$SM_HOME/SAM/smarts/local/conf/trapd/trapd.conf

Launch Splunk from Smarts SAM Console

Configure Smarts to Launch Splunk

Splunk will be launched using the Client Tools functionality in Smarts. You will need to first install a script, and then configure your Smarts console to add a new Client Tool.

Linux/Unix

Download splunkit.sh or cut and paste the code that follows to the machine where you are running the Smarts console - and be sure to modify the BROWSER and SPLUNK_HOST variables for your local environment:

$SM_HOME/SAM/smarts/local/actions/client/splunkit.sh

#!/bin/sh
############################################################################
# This script launches Splunk, passing in the hostname and Last Notify time
# of the event in the Smarts console, and another parameter to limit the
# events Splunk finds to within N minutes of the Last Notify time.
############################################################################
# Modify the following BROWSER and SPLUNK_HOST variables as necessary
# for your local environment
BROWSER="/usr/local/firefox/firefox"
SPLUNK_HOST="SPLUNK_HOST:8000"
# Modify MINUTES variable to limit the events found to within that number
# of minutes from the Last Notify time
MINUTES="1"
#  Convert command line arguments to env vars
for i in "$@"
do
    # Remove " at offset 0 in string, and add " after first = in string
    j=`echo $i | sed 's/"//' | sed 's/=/="/'`
    # echo $i "--converted to--> |" $j "|"
    # Evaluate the assignment into this shell, i.e., execute
    # the modified string as if it had been typed on the command line
    eval $j
done
# Check for required parameters
if test -z "$SM_OBJ_InstanceName"; then
   echo Can not process request without SM_OBJ_InstanceName
   exit 1
fi
if test -z "$SM_OBJ_LastNotifiedAt"; then
   echo Can not process request without SM_OBJ_LastNotifiedAt
   exit 1
fi
# Convert Last Notify time format for use by Splunk
# Last Notify example: Mon Feb 26 13:12:51 PST 2007
# Splunk time example: 02/26/2007:13:12:51
CMONTH=`echo ${SM_OBJ_LastNotifiedAt} | awk '{print $2}'`
DAY=`echo ${SM_OBJ_LastNotifiedAt} | awk '{print $3}'`
TIME=`echo ${SM_OBJ_LastNotifiedAt} | awk '{print $4}'`
YEAR=`echo ${SM_OBJ_LastNotifiedAt} | awk '{print $6}'`
if [ "$CMONTH" = "Jan" ]; then 
   MONTH="01"
fi
if [ "$CMONTH" = "Feb" ]; then
   MONTH="02"
fi
if [ "$CMONTH" = "Mar" ]; then
   MONTH="03"
fi
if [ "$CMONTH" = "Apr" ]; then 
   MONTH="04"
fi
if [ "$CMONTH" = "May" ]; then 
   MONTH="05"
fi
if [ "$CMONTH" = "Jun" ]; then
   MONTH="06"
fi
if [ "$CMONTH" = "Jul" ]; then
   MONTH="07"
fi
if [ "$CMONTH" = "Aug" ]; then
   MONTH="08"
fi
if [ "$CMONTH" = "Sep" ]; then
   MONTH="09"
fi
if [ "$CMONTH" = "Oct" ]; then
   MONTH="10"
fi
if [ "$CMONTH" = "Nov" ]; then
   MONTH="11"
fi
if [ "$CMONTH" = "Dec" ]; then
   MONTH="12"
fi
ENDTIME="$MONTH%2F$DAY%2F$YEAR%3A$TIME"
# Launch Splunk
SPLUNK_CMD="http://$SPLUNK_HOST/?q=HOST%3A%3A$SM_OBJ_InstanceName%20endtime%3A%3A$ENDTIME%20searchtimespanminutes%3A%3A$MINUTES"
$BROWSER $SPLUNK_CMD &
echo launched $BROWSER on: `uname -a`

Windows

Download splunkit.cmd or cut and paste the code that follows to the machine where you are running the Smarts console - and be sure to modify the SPLUNK_HOST=SPLUNK_HOST:8000 variable to point to your Splunk server:

%SM_HOME%\SAM\smarts\local\actions\client\splunkit.cmd

@echo off
setlocal
REM  ###########################################################################
REM  This script launches Splunk, passing in the hostname and Last Notify time
REM  of the event in the Smarts console, and another parameter to limit the
REM  events Splunk finds to within N minutes of the Last Notify time.
REM  
REM  The web browser used is the default browser associated with
REM  the http: protocol on your system.
REM  ###########################################################################
REM  Modify the following SPLUNK_HOST variable to point to your Splunk
REM  installation.
set SPLUNK_HOST=SPLUNKHOST:8000
REM  Modify the following MINUTES variable to limit the events found to within
REM  that number of minutes from the Last Notify time
set MINUTES=1
REM  Process each command line argument in turn...
:CONVERT_TOOL_ARGS_TO_ENVARS
    if (%1) == () goto END__CONVERT_TOOL_ARGS_TO_ENVARS
        FOR /F "delims==, tokens=1,*" %%i in (%1) do set %%i=%%j
        REM  discard leading (processed) parameter
        shift
        goto CONVERT_TOOL_ARGS_TO_ENVARS
:END__CONVERT_TOOL_ARGS_TO_ENVARS
REM  Check for required parameters
if "%SM_OBJ_InstanceName%" == "" (
   echo Can not process request without SM_OBJ_InstanceName
   goto :ENDALL
)
if "%SM_OBJ_LastNotifiedAt%" == "" (
   echo Can not process request without SM_OBJ_LastNotifiedAt
   goto :ENDALL
)
REM  Convert Last Notify time format for use by Splunk
REM  Last Notify example: Mon Feb 26 13:12:51 PST 2007
REM  Splunk time example: 02/26/2007:13:12:51
for /F "usebackq tokens=2,3,4,6" %%A in (`echo %SM_OBJ_LastNotifiedAt%`) do (
   set mymonth=%%A
   set myday=%%B
   set mytime=%%C
   set myyear=%%D
)
if "%mymonth%"=="Jan" set mymonth=01
if "%mymonth%"=="Feb" set mymonth=02
if "%mymonth%"=="Mar" set mymonth=03
if "%mymonth%"=="Apr" set mymonth=04
if "%mymonth%"=="May" set mymonth=05
if "%mymonth%"=="Jun" set mymonth=06
if "%mymonth%"=="Jul" set mymonth=07
if "%mymonth%"=="Aug" set mymonth=08
if "%mymonth%"=="Sep" set mymonth=09
if "%mymonth%"=="Oct" set mymonth=10
if "%mymonth%"=="Nov" set mymonth=11
if "%mymonth%"=="Dec" set mymonth=12
set ENDTIME=%mymonth%%%2F%myday%%%2F%myyear%%%3A%mytime%
set SPLUNK_CMD=http://%SPLUNK_HOST%/?q=HOST%%3A%%3A%SM_OBJ_InstanceName%%%20endtime%%3A%%3A%ENDTIME%%%20searchtimespanminutes%%3A%%3A%MINUTES%
echo Splunking HOST::%SM_OBJ_InstanceName%, URL %SPLUNK_CMD%
start /b /min %SPLUNK_CMD%
goto :ENDALL
:ENDALL
endlocal
REM  end of file

Configure New Client Tool

Configure a new Client Tool in Smarts by logging in to the INCHARGE-SA Manager as admin or a user with admin privileges:

Login

Select from the menu Configure -> Global Manager Administration Console:

Configure Global Manager Administration Console

Navigate to INCHARGE-SA -> Tools -> Client, right-click on Client and select New Client Tool...:

Global Manager Administration Console

Name the tool SplunkIT, and leave the radio button on Create new, then click Next button to continue:

Client Tool Wizard - Name

In the next dialog choose the script splunkit.sh on Linux/Unix, or splunkit.cmd on Windows, from the dropdown.

Select the appropriate User Profiles for your environment - the example here shows admin-profile and default-profile, and click Next:

Client Tool Wizard - Program

For the context criteria set the Context object: to be ICIM_Notification and click Next:

Client Tool Wizard - Context Criteria

Likewise for the status criteria the Context object: is also ICIM_Notification, click Next:

Client Tool Wizard - Status Criteria

Click the OK button to exit creating the Tool:

Client Tool Wizard - OK

Finally make sure each of appropriate User Profiles has your new Tool listed under Client Tools - if not then click Modify List..:

- User Profiles

You must now close all Smarts console applications running on your machine and re-login in order to use your new SplunkIT tool.

Enable Client Tools in Web Console

Client Tools only work with the latest 7.01 version of Smarts SAM Web Console. The default URL for the Web Console is usually:

http://SMARTSHOST:8080/webconsole/InChargeApplet.html

To enable Client Tools in the Web Console you need to modify a property and restart the tomcat server:

Linux/Unix

vi $SM_HOME/smarts/tomcat/webapps/webconsole/webconsole.properties

Add or set the following property to true:

com.smarts.clientToolsInApplet=true

Restart the tomcat server:

$SM_HOME/smarts/bin/sm_service stop ic-business-dashboard

$SM_HOME/smarts/bin/sm_service stop ic-business-dashboard

Windows

edit %SM_HOME%\SAM\smarts\tomcat\webapps\webconsole\webconsole.properties

Add or set the following property to true:

com.smarts.clientToolsInApplet=true

Restart tomcat server:

%SM_HOME%\smarts\bin\sm_service.exe stop ic-business-dashboard

%SM_HOME%\smarts\bin\sm_service.exe start ic-business-dashboard

or use the Services GUI to restart "EMC SMARTS Servlet Engine", or the following:

net stop "EMC SMARTS Servlet Engine"

net start "EMC SMARTS Servlet Engine"

Launch Splunk

To launch Splunk from your Smarts SAM console, right-click on any event in a Notification Log or a Host in a Topology Browser and select Client Tools -> SplunkIT

A browser window will appear showing all log events indexed by Splunk for that Host:

SplunkIT

Documentation, Help and Support

An up-to-date Splunk User Manual and Admin Manual are available on our online Documentation. Technical support forums, FAQs and email contacts are available at Splunk Support.

EMC Smarts documentation, help and support are available at EMC Powerlink.

• Was this topic useful?
• Post a comment