Get data from APIs and other remote data interfaces through scripted inputs
Splunk Enterprise can accept events from scripts that you provide. Scripted input is useful in conjunction with some Windows and *nix command-line tools, such as
top, and so on. You can use scripted input to get data from application program interfaces (APIs) and other remote data interfaces and message queues. You can then use commands like
iostat on that data to generate metrics and status data.
Note: This topic describes how to add scripted inputs that you've already written to your set of inputs. To learn how to develop scripted inputs, see "Build scripted inputs" in the Developing Views and Apps for Splunk Web manual.
You can configure scripted inputs from Splunk System or by editing inputs.conf.
Note: On Windows platforms, you can enable text-based scripts, such those in perl and python, with an intermediary Windows batch (
.bat) or PowerShell (
Caution: Scripts launched through scripted input inherit the Splunk Enterprise environment. Be sure to clear environment variables that can affect your script's operation. The only environment variable that's likely to cause problems is the library path (most commonly known as
LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD).
Splunk Enterprise logs any messages sent to the
stderr I/O channel by scripted inputs to
Add a scripted input in Splunk Web
To add a scripted input in Splunk Web:
A. Go to the Add New page
You add a scripted input from the Add New page in Splunk Web. You can get there through two routes:
- Splunk System
- Splunk Home
It doesn't matter which route you use to get there; the Add New page itself is the same either way.
Via Splunk System:
1. Click System in the upper left-hand corner of Splunk Web.
2. In the Data section of the System pop-up, click Data Inputs.
3. Click Scripts.
4. Click the New button to add an input.
Via Splunk Home:
1. Click the Add Data link in Splunk Home. This brings you to a page called "Data recipes".
2. Click the Run and collect the output of a script link to add an input.
B. Specify the scripted input
1. In the Command text box, specify the script command, including the path to the script.
2. In Interval, specify the interval in seconds between script runtimes. The default is 60 (seconds).
3. Enter a new Source name to override the default source value, if necessary.
Important: Consult Splunk support before changing this value.
4. To access other settings, check More settings. A number of additional settings appear. You can usually go with the defaults for these settings. If you want to set them explicitly, here's what they're for:
a. You can change the Host value, if necessary.
b. You can set the Source type. Source type is a default field added to events. Splunk Enterprise uses source type to determine processing characteristics, such as timestamps and event boundaries. For information on overriding automatic source typing, see "Override automatic source type assignment" in this manual.
c. You can set the Index for this input. Leave the value as "default", unless you have defined multiple indexes to handle different types of events. In addition to indexes for user data, Splunk Enterprise has a number of utility indexes, which also appear in this dropdown box.
5. Click Save.
Add a scripted input via inputs.conf
You add a scripted input in
inputs.conf by adding a
Here is the syntax for the
[script://$SCRIPT] <attrbute1> = <val1> <attrbute2> = <val2> ...
Note the following:
$SCRIPTis the fully-qualified path to the location of the script.
- As a best practice, put your script in the
bin/directory nearest the
inputs.confwhere your script is specified. For example, if you are configuring
$SPLUNK_HOME/etc/system/local/inputs.conf, place your script in
$SPLUNK_HOME/etc/system/bin/. If you're working on an application in
$SPLUNK_HOME/etc/apps/$APPLICATION/, put your script in
All attributes are optional. Here is the list of available attributes:
||* Indicates how often to execute the specified command. Specify either an integer value representing seconds or a valid cron schedule.
||* Sets the index where events from this input will be stored.
||* Sets the sourcetype key/field for events from this input.
||Splunk Enterprise picks a source type based on various aspects of the data. There is no hard-coded default.|
||* Sets the source key/field for events from this input.
||The input file path|
If you want the script to run continuously, write the script to never exit and set it on a short interval. This helps to ensure that if there is a problem the script gets restarted. Splunk Enterprise keeps track of scripts it has spawned and will shut them down upon exit.
Using a wrapper script
It is good practice to write a wrapper script for scripted inputs that use commands with arguments. In some cases, the command can contain special characters that Splunk Enterprise escapes when validating text entered in Splunk Web. This causes updates to a previously configured input to fail to save.
- Note: Characters that Splunk Enterprise escapes when validating text are those that should not be in paths, such as equals (
=) and semi-colon (
For example, the following scripted input is not correctly saved when edited in Splunk Web because Splunk Enterprise escapes the equals (=) sign in the parameter to the
[script://$SPLUNK_HOME/etc/apps/myApp/bin/myUtil.py file=my_datacsv] disabled = false
To avoid this problem, write a wrapper script that contains the scripted input. (Inputs updated by editing the conf file directly are not subject to this input validation.) For information on writing wrapper scripts, see "Scripted inputs overview" in the Developing Views and Apps for Splunk Web manual.
Example using inputs.conf
This example shows the use of the UNIX
top command as a data input source:
1. Create a new application directory. This example uses
$ mkdir $SPLUNK_HOME/etc/apps/scripts
2. All scripts should be run out of a
bin/ directory inside your application directory:
$ mkdir $SPLUNK_HOME/etc/apps/scripts/bin
3. This example uses a small shell script
$ #!/bin/sh top -bn 1 # linux only - different OSes have different parameters
4. Make sure the script is executable:
chmod +x $SPLUNK_HOME/etc/apps/scripts/bin/top.sh
5. Test that the script works by running it via the shell:
The script should send one
6. Add the script entry to
[script:///opt/splunk/etc/apps/scripts/bin/top.sh] interval = 5 # run every 5 seconds sourcetype = top # set sourcetype to top source = script://./bin/top.sh # set source to name of script
Note: You might need to modify props.conf:
- By default Splunk Enterprise breaks the single
topentry into multiple events.
- The easiest way to fix this problem is to tell the server to break only before something that does not exist in the output.
For example, adding the following to
$SPLUNK_HOME/etc/apps/scripts/default/props.conf forces all lines into a single event:
[top] BREAK_ONLY_BEFORE = <stuff>
Since there is no timestamp in the top output we need to tell Splunk Enterprise to use the current time. This is done in
props.conf by setting:
DATETIME_CONFIG = CURRENT
Set interval attribute to cron schedule
In the above example, you can also set the
interval attribute to a "cron" schedule by specifying strings like the following:
0 * * * *: Means run once an hour, at the top of the hour.
*/15 9-17 * * 1-5: Means run every 15 minutes from 9 am until 5 pm, on Monday to Friday.
15,35,55 0-6,20-23 1 */2 *: Means run at 15, 35, and 55 minutes after the hour, between midnight and 7 am and again between 8pm and midnight, on the first of every even month (February, April, June and so on).
For more information about setting cron schedules, read "CRONTAB(5) on the Crontab website.
Monitor changes to your file system
Find more things to monitor with crawl
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14