Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Get data from APIs and other remote data interfaces through scripted inputs

Splunk Enterprise can accept events from scripts that you provide. Scripted input is useful in conjunction with some Windows and *nix command-line tools, such as ipconfig, iostat, netstat, top, and so on. You can use scripted input to get data from application program interfaces (APIs) and other remote data interfaces and message queues. You can then use commands like vmstat and iostat on that data to generate metrics and status data. On Windows platforms, you can enable text-based scripts, such those in Perl and Python, with an intermediary Windows batch (.bat) or PowerShell (.ps1) file.

This topic describes how to add scripted inputs that you have already written. To learn how to write scripted inputs, see Build scripted inputs in the Developing Views and Apps for Splunk Web manual.

You can configure scripted inputs from the Settings menu or by editing inputs.conf.

When a scripted input launches a script, that script inherits the Splunk Enterprise environment. Clear any environment variables that can affect the operation of a script. The only environment variable that could cause problems is the library path (most commonly known as LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD).

Splunk Enterprise logs any messages that scripted inputs send to the stderr I/O channel to splunkd.log.

Add a scripted input in Splunk Web

Go to the Add New page

You can get there by two routes.

  • Splunk Home
  • Splunk Settings

By Splunk Settings:

  1. Click Settings in the upper right corner of Splunk Web.
  2. Click Data Inputs.
  3. Click Scripts.
  4. Click New to add an input.

By Splunk Home:

  1. Click the Add Data link in Splunk Home.
  2. Click Monitor to monitor a script on the local machine, or Forward to forward data from a script on a remote machine. Splunk Web displays the "Add Data - Select Source" page.
  3. In the left pane, locate and select Scripts.

Note: Forwarding data from scripted inputs requires additional setup.

Select the input source

  1. In the Script Path drop down, select the path where the script resides. Splunk Web updates the page to include a new drop down list, "Script Name."
  2. In the Script Name drop-down, select the script that you want to run. Splunk Web updates the page to populate the "Command" field with the script name.
  3. In the Command field, add any arguments needed to invoke the script.
  4. In the Interval field, enter the amount of time (in seconds) that Splunk Enterprise should wait before invoking the script.
  5. Optionally, In the Source Name Override field, enter a new source name to override the default source value, if necessary.
  6. Click Next.

Specify input settings

The Input Settings page lets you specify application context, default host value, and index. All of these parameters are optional. Learn more about setting the host value in "About hosts".

When you set the Host on this page, this only sets the host field in the resulting events. It does not direct Splunk Enterprise to look on a specific host on your network.

  1. Select the source type for the script. You can choose Select to pick from the list of available source types on the local machine, or "Manual" to enter the name of a source type.
  2. Select the appropriate Application context for this input.
  3. Set the Host name value. You have several choices for this setting.
  4. Set the Index that Splunk Enterprise should send data to. Leave the value as "default", unless you have defined multiple indexes to handle different types of events. In addition to indexes for user data, Splunk Enterprise has a number of utility indexes, which also appear in this drop down box.
  5. Click Review.

Review your choices

After specifying all your input settings, review your selections. Splunk Web lists all options you selected, including the type of monitor, the source, the source type, the application context, and the index.

  1. Review the settings.
  2. If they do not match what you want, click < to go back to the previous step in the wizard. Otherwise, click Submit.

Splunk Web displays the "Success" page.

Add a scripted input with inputs.conf

You add a scripted input in inputs.conf by adding a [script] stanza.

Syntax

The syntax for the [script] stanza follows:

[script://$SCRIPT] 
<attrbute1> = <val1>
<attrbute2> = <val2>
...
  • $SCRIPT is the full path to the location of the script.
  • $SCRIPT can also be a file path that ends in the .path suffix. This special suffix lets you use the stanza to point to another command or script that exists anywhere on the host filesystem. See Use the .path suffix to reference external scripts. The file that you reference in the stanza must heed the location restrictions that are described in "Where to place the scripts for scripted inputs" in this topic.

Where to place the scripts for scripted inputs

The script that you reference in $SCRIPT can only reside in one of the following places on the host file system:

  • $SPLUNK_HOME/etc/system/bin
  • $SPLUNK_HOME/etc/apps/<your_app>/bin
  • $SPLUNK_HOME/bin/scripts

As a best practice, put your script in the bin/ directory that is nearest the inputs.conf that calls your script on the host filesystem. For example, if you configure $SPLUNK_HOME/etc/system/local/inputs.conf, place your script in $SPLUNK_HOME/etc/system/bin/. If you work on an application in $SPLUNK_HOME/etc/apps/$APPLICATION/, put your script in $SPLUNK_HOME/etc/apps/$APPLICATION/bin/.

Attributes

All attributes are optional. Here is the list of available attributes:

Attribute Description Default
interval = <number>|<cron schedule> How often to execute the specified command. Specify either an integer value representing seconds or a valid cron schedule.

When you specify a cron schedule, the script does not execute on start up, but rather at the times that the cron schedule defines.

Splunk Enterprise keeps one invocation of a script per instance. Intervals are based on when the script completes. If you configure a script to run every 10 minutes and the script takes 20 minutes to complete, the next run will occur 30 minutes after the first run.

For constant data streams, enter 1 (or a value smaller than the script interval). For one-shot data streams, enter -1. Setting interval to -1 causes the script to run each time at start-up.

60 seconds
index = <string> The index where events from this input should be stored. Splunk Enterprise prepends the <string> with index::.

For more information about the index field, see "How indexing works" in the Managing Indexers and Clusters manual.

main, or whatever you have set as your default index.
sourcetype = <string> Sets the sourcetype key/field for events from this input. * The <string> is prepended with 'sourcetype::'.

Explicitly declares the source type for this data, as opposed to letting it be determined automatically. This is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing.

Sets the sourcetype key initial value. Splunk Enterprise uses this key is during parsing/indexing, in particular to set the source type field during indexing. It also uses the source type field at search time.

Splunk Enterprise picks a source type based on various aspects of the data. There is no hard-coded default.
source = <string> * Sets the source key/field for events from this input.
  • Note: Do not override the source key unless absolutely necessary. Typically, the input layer will provide a more accurate string to aid in problem analysis and investigation, accurately recording the file from which the data was retreived. Consider use of source types, tagging, and search wildcards before overriding this value.
  • Splunk Enterprise prepends <string> with source::.
The input file path
disabled = <true | false> Whether or not the input should run. Set to true if you want to disable the input. false

Run scripts continuously

If you want the script to run continuously, write the script to never exit and set it on a short interval. This helps to ensure that if there is a problem the script gets restarted. Splunk Enterprise keeps track of scripts it has spawned and shuts them down on exit.

Use a wrapper script

It is best practice to write a wrapper script for scripted inputs that use commands with arguments. In some cases, the command can contain special characters that the scripted input escapes when it validates text that you have entered in Splunk Web. This causes updates to a previously configured input to fail to save.

Splunk Enterprise escapes characters that should not be in paths, such as the equals sign (=) and semicolon (;) when it validates text. For example, the following scripted input is not correctly saved when you edit it in Splunk Web because the scripted input escapes the equals (=) sign in the parameter to the myUtil.py utility:

[script://$SPLUNK_HOME/etc/apps/myApp/bin/myUtil.py file=my_datacsv]
disabled = false

To avoid this problem, write a wrapper script that contains the scripted input, or use the special .path argument for the scripted input stanza name. For information on writing wrapper scripts, see Scripted inputs overview in the Developing Views and Apps for Splunk Web manual.

When you update scripted Inputs by editing inputs.conf directly, this validation does not occur.

Use the .path suffix to reference external scripts

As an alternative to writing a wrapper script, you can configure the scripted input to reference a script or executable that is anywhere on the host file system.

The script that you reference can have a single line that calls the script or executable that you want. You can use this file to call a runtime environment that is outside of the Splunk Enterprise environment. For example, if you have both Splunk Enterprise, which comes with Python, and a second installation of Python on the same host, you can use the .path method to reference the second Python installation.

  1. Use Splunk Web or edit inputs.conf and specify a scripted input stanza with a script name that ends in .path.
    [script://myfile.path]
    disabled = 0
    
  2. Place the file that you reference in the stanza in the appropriate directory, as described in Where to place the scripts for scripted inputs.
  3. Edit the file to specify the script or executable you want.
  4. /path/to/myscript -arg1 arg -arg2 arg
    

Examples of scripted inputs with inputs.conf

Unix top command

This example shows the use of the UNIX top command as a data input source:

  1. Create a new application directory. This example uses scripts/.
    $ mkdir $SPLUNK_HOME/etc/apps/scripts
    

  2. All scripts should be run out of a bin/ directory inside your application directory.
    $ mkdir $SPLUNK_HOME/etc/apps/scripts/bin
    

  3. This example uses a small shell script top.sh.
    $ #!/bin/sh
    top -bn 1  # linux only - different OSes have different parameters
    

  4. Make the script executable.
    chmod +x $SPLUNK_HOME/etc/apps/scripts/bin/top.sh
    

  5. Test that the script works by running it via the shell.
    $SPLUNK_HOME/etc/apps/scripts/bin/top.sh
    
    The script should send one top output.

  6. Add the script entry to inputs.conf in $SPLUNK_HOME/etc/apps/scripts/local/.
    [script:///opt/splunk/etc/apps/scripts/bin/top.sh]
    interval = 5                     # run every 5 seconds
    sourcetype = top                 # set sourcetype to top
    source = script://./bin/top.sh   # set source to name of script
    

    Note: You might need to modify props.conf:

    • By default Splunk Enterprise breaks the single top entry into multiple events.
    • The easiest way to fix this problem is to tell the server to break only before something that does not exist in the output.

    For example, adding the following to $SPLUNK_HOME/etc/apps/scripts/default/props.conf forces all lines into a single event:

    [top]
    BREAK_ONLY_BEFORE = <stuff>
    

    Since there is no timestamp in the top output, you must tell Splunk Enterprise to use the current time. Use props.conf and set the following:

    DATETIME_CONFIG = CURRENT
    

Reference an external script with the .path stanza

The following example uses the special .path stanza setting to reference an external build of Python to run a script on your host.

  1. Edit inputs.conf.
    [script://loglogs.path]
    disabled = 0
    
  2. Place or create loglogs.path in $SPLUNK_HOME/etc/system/bin.
  3. Edit loglogs.path to reference the external version of Python.
    /usr/bin/python logit.py --source /opt/files/my_files --target /opt/files/my_files/processed --logfile /opt/src/my_sources/logfiles
    

Set interval attribute to cron schedule

In the above example, you can also set the interval attribute to a "cron" schedule by specifying strings like the following:

0 * * * *: Means run once an hour, at the top of the hour.

*/15 9-17 * * 1-5: Means run every 15 minutes from 9 am until 5 pm, on Monday to Friday.

15,35,55 0-6,20-23 1 */2 *: Means run at 15, 35, and 55 minutes after the hour, between midnight and 7 am and again between 8pm and midnight, on the first of every even month (February, April, June and so on).

For more information about setting cron schedules, read CRONTAB(5) on the Crontab website.

PREVIOUS
Monitor changes to your file system
  NEXT
Find more data sources to monitor with crawl

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.0, 7.1.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters