Setting up a scripted input
This section describes how to set up a scripted input for an app. To illustrate the setup, it uses an example script that polls a database and writes the results to a file. A more detailed version of this example is in Example script that polls a database. That topic provides details on the example, including code examples in Python and Java.
Note: You can write any number and types of scripts in various scripting languages that perform various functions. This example shows the framework for a commonly found script. Adapt this framework according to your needs.
Script to poll a database
This example script does the following:
- Runs at a regular interval
- Queries a database
- Writes the output to a file in a format optimized for Splunk indexing
- Splunk indexes the file containing the results of the queries.
Place scripts in the bin directory of your app:
Here is the directory structure of the example script for this example. The directory structure for your app might differ.
+ . . . /etc/apps/<appName>/ | | + bin/ | | | + last_eventid | | | + key | | | + output.txt | | | + starter_script.sh | | | + my_db_poll.py | | | + ip2int.py | | + default/ | + inputs.conf | + app.conf | + . . .
. . ./etc/apps/<appName>/bin/my_db_poll.py
This is the script that retrieves information from the database. This script does the following:
- Queries the database and writes the query result to file
- Defines the format of output data
- Accesses a database using credentials stored in key
- Reads last_eventid to determine the next event to read from the database
- Queries the database at the next event and writes the output to a file
. . ./etc/apps/<appName>/bin/starter_script.sh
Wrapper script that calls the
my_db_poll.py script. In this example, it calls
my_db_poll.py with the arguments needed to query the database.
.../etc/apps/<appName>/default/inputs.conf, create a stanza that references this wrapper script. In this example, the stanza specifies how often to call the starter script to poll the database.
. . ./etc/apps/<appName>/bin/ip2int.py
A helper script to convert IP addresses from integer format to dotted format, and back. This is a type of helper script that formats data better for Splunk to index. You often have helper scripts that aid the main script.
. . ./etc/apps/<appName>/bin/key
Text file containing username and password encoded in base64 using the python function
base64.b64encode(). The Splunk user has read and write access to this file.
Security for passwords is an issue when running scripts.
. . ./etc/apps/<appName>/bin/last_eventid
File containing a number for the last event received from the database.
my_db_poll.py writes the last_eventid after querying the database. The Splunk user has read and write access to this file.
. . ./etc/apps/<appName>/bin/output.txt
A single event from the script, for reference.
my_db_poll.py writes the actual output from querying the database to another directory.
. . ./etc/apps/<appName>/default/inputs.conf
Configure scripted data input in
$SPLUNK_HOME/etc/<appName>/default/inputs.conf. Use the local directory for the app to overwrite behavior defined in the default directory. Here is an example:
[script://$SPLUNK_HOME/etc/apps/<appName>/bin/my_db_poll.py] disabled = true # change to false to start the input, requires restart host = # enter hostname here index = main interval = 30 #frequency to run the script, in seconds source = my_db sourcetype = my_db_data
Configure properties for the script in the Splunk system
[my_db] TIME_PREFIX=^[^\|]+\| TIME_FORMAT=%Q MAX_TIMESTAMP_LOOKAHEAD=10 #look ahead 10 characters SHOULD_LINEMERGE=false
Define field transforms in
[my_db_extractions] DELIMS = "|" FIELDS ="EventID","AlertTime","UserName",. . ."
Scripted inputs overview
Writing reliable scripts
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18