Getting Data In

 


Other ways to get stuff in

Get data from APIs and other remote data interfaces through scripted inputs

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Get data from APIs and other remote data interfaces through scripted inputs

Splunk can accept events from scripts that you provide. Scripted input is useful in conjunction with command-line tools, such as vmstat, iostat, netstat, top, etc. You can use scripted input to get data from APIs and other remote data interfaces and message queues. You can then use commands like vmstat and iostat on that data to generate metrics and status data.

Note: This topic describes how to add scripted inputs that you've already written to your set of Splunk inputs. To learn how to develop scripted inputs in the first place, see "Build scripted inputs" in the Developing Dashboards, Views, and Apps manual.

Lots of apps on Splunkbase provide scripted inputs for specific applications. You can find them on the Browse more apps tab in the Launcher.

You configure scripted inputs from Splunk Manager or by editing inputs.conf.

Note: On Windows platforms, you can enable text-based scripts, such those in perl and python, with an intermediary Windows batch (.bat) file.

Caution: Scripts launched through scripted input inherit Splunk's environment, so be sure to clear environment variables that can affect your script's operation. The only environment variable that's likely to cause problems is the library path (most commonly known as LD_LIBRARY_PATH on linux/solaris/freebsd).

Starting with release 4.2, any stderr messages generated by scripted inputs are logged to splunkd.log.

Add a scripted input in Splunk Web

To add a scripted input in Splunk Web:

A. Go to the Add New page

You add a scripted input from the Add New page in Splunk Web. You can get there through two routes:

  • Splunk Manager
  • Splunk Home

It doesn't matter which route you use to get there; the Add New page itself is the same either way.

Via Splunk Manager:

1. Click Manager in the upper right-hand corner of Splunk Web.

2. In the Data section of the Manager page, click Data Inputs.

3. Click Scripts.

4. Click the New button to add an input.

Via Splunk Home:

1. Click the Add Data link in Splunk Home. This brings you to a page called "Data recipes".

2. Click the Run and collect the output of a script link to add an input.

B. Specify the scripted input

1. In the Command text box, specify the script command, including the path to the script.

2. In Interval, specify the interval in seconds between script runtimes. The default is 60 (seconds).

3. Enter a new Source name to override the default source value, if necessary.

Important: Consult Splunk support before changing this value.

4. To access other settings, check More settings. A number of additional settings appear. You can usually go with the defaults for these settings. If you want to set them explicitly, here's what they're for:

a. You can change the Host value, if necessary.

b. You can set the Source type. Source type is a default field added to events. Source type is used to determine processing characteristics, such as timestamps and event boundaries. For information on overriding Splunk's automatic source typing, see "Override automatic source type assignment" in this manual.

c. You can set the Index for this input. Leave the value as "default", unless you have defined multiple indexes to handle different types of events. In addition to indexes for user data, Splunk has a number of utility indexes, which also appear in this dropdown box.

5. Click Save.

Add a scripted input via inputs.conf

You add a scripted input in inputs.conf by adding a [script] stanza.

Syntax

Here is the syntax for the [script] stanza:

[script://$SCRIPT] 
<attrbute1> = <val1>
<attrbute2> = <val2>
...

Note the following:

  • $SCRIPT is the fully-qualified path to the location of the script.
  • As a best practice, put your script in the bin/ directory nearest the inputs.conf where your script is specified. For example, if you are configuring $SPLUNK_HOME/etc/system/local/inputs.conf, place your script in $SPLUNK_HOME/etc/system/bin/. If you're working on an application in $SPLUNK_HOME/etc/apps/$APPLICATION/, put your script in $SPLUNK_HOME/etc/apps/$APPLICATION/bin/.

Attributes

All attributes are optional. Here is the list of available attributes:

interval = <integer>|<cron schedule>

  • Indicates how often to execute the specified command. Specify either an integer value representing seconds or a valid cron schedule.
  • Defaults to 60 seconds.
  • When a cron schedule is specified, the script is not executed on start up.
  • Splunk keeps one invocation of a script per instance. Intervals are based on when the script completes. So if you have a script configured to run every 10 minutes and the script takes 20 minutes to complete, the next run will occur 30 minutes after the first run.
  • For constant data streams, enter 1 (or a value smaller than the script's interval).
  • For one-shot data streams, enter -1. Setting interval to -1 will cause the script to run each time the splunk daemon restarts.

index = <string>

  • Set the index where events from this input will be stored.
  • The <string> is prepended with 'index::'.
  • Defaults to main, or whatever you have set as your default index.
  • For more information about the index field, see "How indexing works" in the Admin manual.

sourcetype = <string>

  • Sets the sourcetype key/field for events from this input.
  • Explicitly declares the source type for this data, as opposed to allowing it to be determined automatically. This is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing.
  • Sets the sourcetype key's initial value. The key is used during parsing/indexing, in particular to set the source type field during indexing. It is also the source type field used at search time.
  • The <string> is prepended with 'sourcetype::'.
  • If not set explicitly, Splunk picks a source type based on various aspects of the data. There is no hard-coded default.
  • For more information about source types, see "Why source types matter", in this manual.

source = <string>

  • Sets the source key/field for events from this input.
  • Note: Overriding the source key is generally not recommended. Typically, the input layer will provide a more accurate string to aid in problem analysis and investigation, accurately recording the file from which the data was retreived. Consider use of source types, tagging, and search wildcards before overriding this value.
  • The <string> is prepended with 'source::'.
  • Defaults to the input file path.

disabled = <true | false>

  • disabled is a boolean value that can be set to true if you want to disable the input.
    • Defaults to false.

If you want the script to run continuously, write the script to never exit and set it on a short interval. This helps to ensure that if there is a problem the script gets restarted. Splunk keeps track of scripts it has spawned and will shut them down upon exit.

Using a wrapper script

Generally, it is good practice to write a wrapper script for scripted inputs that use commands with arguments. In some cases the command can contain special characters that Splunk escapes when validating text entered in Splunk Web. This causes updates to a previously configured input to fail to save.

Note: Characters that Splunk escapes when validating text are those that should not be in paths, such as equals (=) and semi-colon (;).

For example, the following scripted input is not correctly saved when edited in Splunk Web because Splunk escapes the equals (=) sign in the parameter to the myUtil.py utility:

[script://$SPLUNK_HOME/etc/apps/myApp/bin/myUtil.py file=my_datacsv]
disabled = false

To avoid this problem, write a wrapper script that contains the scripted input. (Inputs updated by editing the conf file directly are not subject to this input validation.) For information on writing wrapper scripts, see "Scripted inputs overview" in the Developing Dashboards, Views, and Apps for Splunk Web manual.

Example using inputs.conf

This example shows the use of the UNIX top command as a data input source:

1. Create a new application directory. This example uses scripts/:

$ mkdir $SPLUNK_HOME/etc/apps/scripts

2. All scripts should be run out of a bin/ directory inside your application directory:

$ mkdir $SPLUNK_HOME/etc/apps/scripts/bin

3. This example uses a small shell script top.sh:

$ #!/bin/sh
top -bn 1  # linux only - different OSes have different parameters

4. Make sure the script is executable:

chmod +x $SPLUNK_HOME/etc/apps/scripts/bin/top.sh

5. Test that the script works by running it via the shell:

$SPLUNK_HOME/etc/apps/scripts/bin/top.sh

The script should send one top output.

6. Add the script entry to inputs.conf in $SPLUNK_HOME/etc/apps/scripts/default/:

[script:///opt/splunk/etc/apps/scripts/bin/top.sh]
interval = 5                     # run every 5 seconds
sourcetype = top                 # set sourcetype to top
source = script://./bin/top.sh   # set source to name of script

Note: You may need to modify props.conf:

  • By default Splunk breaks the single top entry into multiple events.
  • The easiest way to fix this problem is to tell the Splunk server to break only before something that does not exist in the output.

For example, adding the following to $SPLUNK_HOME/etc/apps/scripts/default/props.conf forces all lines into a single event:

[top]
BREAK_ONLY_BEFORE = <stuff>

Since there is no timestamp in the top output we need to tell Splunk to use the current time. This is done in props.conf by setting:

DATETIME_CONFIG = CURRENT

This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!