Splunk® Enterprise

Developing Views and Apps for Splunk Web

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Setting up a scripted input

This section describes how to set up a scripted input for an app. To illustrate the setup, it uses an example script that polls a database and writes the results to a file. A more detailed version of this example is in Example script that polls a database. That topic provides details on the example, including code examples in Python and Java.

You can write any number and types of scripts in various scripting languages that perform various functions. This example shows the framework for a commonly found script. Adapt this framework according to your needs.

Script to poll a database

This example script does the following:

  • Runs at a regular interval.
  • Queries a database.
  • Writes the output to a file in a format optimized for Splunk indexing.
  • Splunk indexes the file containing the results of the queries.

Directory structure

Place scripts in the bin directory of your app:

$SPLUNK_HOME/etc/apps/<appName>/bin/

Here is the directory structure of the example script for this example. The directory structure for your app might differ.

Scripted Inputs directory.png

Script files

. . ./etc/apps/<appName>/bin/my_db_poll.py

This is the script that retrieves information from the database. This script does the following:

  • Queries the database and writes the query result to file.
  • Defines the format of output data.
  • Accesses a database using credentials stored in key.
  • Reads last_eventid to determine the next event to read from the database.
  • Queries the database at the next event and writes the output to a file.

. . ./etc/apps/<appName>/bin/starter_script.sh

Wrapper script that calls the my_db_poll.py script. In this example, it calls my_db_poll.py with the arguments needed to query the database.

In .../etc/apps/<appName>/default/inputs.conf, create a stanza that references this wrapper script. In this example, the stanza specifies how often to call the starter script to poll the database.

. . ./etc/apps/<appName>/bin/ip2int.py

A helper script to convert IP addresses from integer format to dotted format, and back. This is a type of helper script that formats data better for indexing. You often have helper scripts that aid the main script.

. . ./etc/apps/<appName>/bin/key

Text file containing username and password encoded in base64 using the python function base64.b64encode(). The Splunk Enterprise user has read and write access to this file.

Security for passwords is an issue when running scripts.

. . ./etc/apps/<appName>/bin/last_eventid

File containing a number for the last event received from the database. my_db_poll.py writes the last_eventid after querying the database. The Splunk user has read and write access to this file.

. . ./etc/apps/<appName>/bin/output.txt

A single event from the script, for reference. my_db_poll.py writes the actual output from querying the database to another directory.

. . ./etc/apps/<appName>/default/inputs.conf

Configure scripted data input in $SPLUNK_HOME/etc/<appName>/default/inputs.conf. Use the local directory for the app to overwrite behavior defined in the default directory. Here is an example:

[script://$SPLUNK_HOME/etc/apps/<appName>/bin/starter_script.sh]
disabled = true # change to false to start the input, requires restart
host = # enter hostname here
index = main
interval = 30 	 #frequency to run the script, in seconds
source = my_db
sourcetype = my_db_data


$SPLUNK_HOME/etc/system/local/props.conf

Configure properties for the script in the Splunk Enterprise system props.conf:

[my_db_data]
TIME_PREFIX=^[^\|]+\|
TIME_FORMAT=%Q
MAX_TIMESTAMP_LOOKAHEAD=10	#look ahead 10 characters
SHOULD_LINEMERGE=false

$SPLUNK_HOME/etc/system/local/transforms.conf

Define field transforms in transforms.conf:

[my_db_extractions]
DELIMS = "|"
FIELDS ="EventID","AlertTime","UserName",. . ."
PREVIOUS
Scripted inputs overview
  NEXT
Writing reliable scripts

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Comments

Hi all,<br />is possible to execute a script when my custom APP is launched?<br />Many thanks<br />Mauro

Mcompagnone
December 5, 2014

Thanks Joshdiakun. I made the correction you suggest.

Vgenovese
September 17, 2014

Correction needed:<br /><br />In the inputs.conf you define the following:<br />sourcetype = my_db_data<br /><br />However in props.conf you specify:<br />[my_db]<br /><br />The correction to props.conf should be:<br />[my_db_data] or [source::my_db]

Joshdiakun
September 15, 2014

Splunk1001, current Splunk architecture requires you to place scripts in $SPLUNK_HOME/etc/apps/appName/bin/

Vgenovese
December 10, 2013

In the script setup it mentions you can call a (DOS) batch script as the example shows .bat but the help here references only python. Is there any reason why you have to run a script inside of bin/scripts? With UAC on Win7 it's a real pain.

Splunk1001
December 9, 2013

I corrected a typo in the stanza header for inputs.conf. It now references starter_script.sh. As described in the docs, starter_script.sh is a wrapper that you would write to call my_db_poll.py. The actual coding you use in the script is outside the scope of these docs. This is just an example showing that you might want to use a starter script that can be changed without having to modify the python db_poll script.<br /><br />In a windows environment, to access a SQL server DB, you need to find a command line tool that can access the DB. In a UNIX environment, finding a binary to access a SQL server DB might be a bit harder.

Vgenovese
November 25, 2013

You are missing major steps, where is the starter_script.sh, I don't see any info on how to setup mssql

Neaves777
November 15, 2013

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters