Developing Dashboards, Views, and Apps for Splunk Web

 


Writing reliable scripts

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Writing reliable scripts

Here are some tips for creating reliable input scripts:

Environment variables

Clear environment variables that can affect your script's operation. One environment variable that is likely to cause problems is the library path. The library path is most commonly known as LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD. It is DYLD_LIBRARY_PATH on OS X, and LIBPATH on AIX.

If you are running external python software or using other python interpreters, consider clearing PYTHONPATH.

Caution: Changing PYTHONPATH may affect other installations of python.

On Windows platforms, the SPLUNKHOME environment variable is set for you. Avoid changing this environment variable. Changing this variable may interfere with the functioning of Splunk services.

Python version

For best results, use the version of Python available from your Splunk installation. Splunk uses this version to execute system scripts, so you should test your script using that version of Python.

Some Python libraries your script requires may not be available in Splunk's version of Python. In this case, you can copy the libraries to the same directory as the scripted input.

To run a script using the version of Python available from Splunk:

$SPLUNK_HOME/bin/splunk cmd python <your_script>.py

File paths in Python

Be careful when specifying platform-specific paths and relative paths.

Platform-specific paths

When writing scripts in Python, avoid hard coding platform-specific file paths. Instead specify file paths that can be interpreted correctly on Windows, UNIX, and Mac platforms. For example, the following Python code launches try.py, which is in the bin directory of your_app:

import os
import subprocess
 
# Edit directory names here if appropriate
if os.name == 'nt':
    ## Full path to your Splunk installation
    splunk_home = 'C:\Program Files\Splunk'
    ## Full path to python executable
    python_bin = 'C:\Program Files (x86)\Python-2.6-32bit\python.exe'
else:
    ## Full path to your Splunk installation
    # For some reason:
    #splunk_home = '/appl/opt/splunk_fwd/'
    # For a sensible OS:
    splunk_home = '/opt/splunk'

    ## Full path to python executable
    # For Mac OS X:
    #python_bin = '/Library/Frameworks/Python.framework/Versions/2.6/bin/python'
    # For a sensible filesystem:
    python_bin = '/usr/bin/python'

try_script = os.path.join(splunk_home, 'etc', 'apps', 'your_app', 'bin', 'try.py')
 
print subprocess.Popen([python_bin, try_script], stdout=subprocess.PIPE).communicate()[0]

Relative paths

Avoid using relative paths in scripts. Python scripts do not use the current directory when resolving relative paths. For example, on *nix platforms, relative paths are set relative to the root directory (/). The following example shows how to locate the extract.conf file, which is in the same directory as the script:

import os
import os.path

script_dirpath = os.path.dirname(os.path.join(os.getcwd(), __file__))

config_filepath = os.path.join(script_dirpath, 'extract.conf')

Format script output

Format the output of a script so Splunk can easily parse the data. Also, consider formatting data so it is more human-readable as well.

Use the Common Information Model

The Common Information Model is based on the idea that you can break down most log files into three components: fields, event type tags, and host tags.

With these three components you can set up log files in a way that makes them easily processable by Splunk and that normalizes noncompliant log files, forcing them to follow a similar schema. The Common Information model details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data.

For more information, see Understand and use the Common Information Model.

Timestamp formats

Time stamp the beginning of an event. There are several options for timestamp formats:

RFC-822, RFC-3339 These are standard timestamp formats for email headers and internet protocols. These formats provide an offset from GMT, and thus are unambiguous and more human-readable.

RFC-822 Tue, 15 Feb 2011 14:11:01 -0800
RFC-3339 2011-02-15 14:11:01-08:00

Note: RFC-822 and RFC-3339 formats can be handled with %z in a TIME_FORMAT setting.


UTC UTC formatting may not be as human-readable as some of the other options. If the timestamp is epoch time, no configuration is necessary. Otherwise, requires a configuration in props.conf that declares the input as TZ=UTC.

UTC
2011-02-15T14:11:01-05:00
2011-02-15T14:11:01Z
UTC converted to epoch time
1297738860


Multi-line data and field names

For multi-line data, find a way to separate events.

  • Write a distinctive initial line for a multi-line event.
  • Use a special end of event string to separate events. For example, use three newline characters to specify an end of an event. The event would then include any single or double newline characters.
  • For multi-line field values, place the field data inside quotes.
  • Use an equals (=) sign or other separator to expose name/value pairs. For example, key=value.
  • Configure Splunk to use other tokens that might already exist in the data.
  • Field names are case sensitive. For example the field names “message” and “Message” represent different fields. Be consistent when naming fields.

Write a setup screen to configure scripted inputs.

If you are packaging an app or add-on for distribution, consider creating a setup screen that allows users to interactively provide configuration settings for access to local scripted input resources. Include an input stanza for your script so setup.xml doesn’t require a custom endpoint. See Configure a setup screen for your app.

Refer to the *Nix and Windows apps for examples on using setup.xml pages to create a setup screen. These apps are available for download from Splunkbase.

Save state across invocations of the script

Scripts often need to checkpoint their work so subsequent invocations can pick up from where they left off. For example, save the last ID read from a database, mark the line and column read from a text file, or otherwise note the last input read. (See Example script that polls a database.)

You can check point either the Splunk index or the script. When check pointing data, keep in mind that the following things are not tied together as a transaction:

  • Writing out checkpoint files
  • Fully writing data into the pipe between the script and splunkd
  • splunkd completely writing out the data into the index

Thus, in the case of hard crashes, it’s hard to know if the data the script has acquired has been properly indexed. Here are some of the choices you have:

Search Splunk index One strategy is to have the scripted input search in the splunk index to find the last relevant event. This is reasonable in an infrequently-launched script, such as one that is launched every 5 or 10 minutes, or at launch time for a script which launches once and stays running indefinitely.

Maintain independent check point Because there is some delay between data being fed to splunk and the data becoming searchable, a frequently run scripted input must maintain its own checkpoint independent of the index.

Choose a scenario If the script always believes its own checkpoint, data may not be indexed on splunkd or system crash. If the index search is believed, some data may be indexed multiple times on splunkd or system crash. You need to choose which scenario you best fits your needs.

Accessing secured services

Use proper security measures for scripts that need credentials to access secured resources. Here are a few suggestions on how to provide secure access. However, no method is foolproof, so think carefully about your use case and design secure access appropriately:

  • Restrict which users can access the app or add-on on disk.
  • Create and use credentials specific to the script, with the minimum permissions required to access the data.
  • Avoid putting literal passwords in scripts or passing the password as a command line argument, making it visible to all local processes with operating system access.
  • Use Splunk to encrypt passwords. You can create an app set up page that allows users to enter passwords. (See Providing credentials to access scripts.) The user can enter a password in plain text, which Splunk stores in the credential stanza in apps.conf. Alternatively, you can specify a python script to securely provide access.
Caution: Splunk assembles a secret using locally available random seeds to encrypt passwords stored in configuration files. This method provides modest security against disclosure of passwords from admins with local disk read capability. However, it is not an adequate protection for privileged accounts.

Concurrency issues for scripted inputs

Be careful scheduling two copies of a script running at any given time. Splunk detects if another instance of the script is running, and does not launch a new instance if this is the case. For example, if you have a script scheduled to execute every 60 seconds, and a particular invocation takes 140 seconds, Splunk detects this and does not launch a new instance until after the long-running instance completes.

At times you may want to run multiple copies of a script, for example to poll independent databases. For these cases, design your scripts so they can handle multiple servers. Also, design your script so that multiple copies can exist (for example, use two app directories for script).

Alternatively, you could have separate scripts using the same sourcetype.

Troubleshooting scheduled scripts

Splunk logs exceptions thrown by scheduled scripts to the splunkd.log file, located here:

$SPLUNK_HOME/var/log/splunk/splunkd.log

Check splunkd.log first if expected events do not appear in the expected index after scheduling the scripted input.

Shutdown and restart issues

Keep these shutdown and restart issues in mind when designing your scripts:

Output at least one event at a time

This makes it easier to avoid reading a partial event if the script is terminated or crashes. Splunk expects events will complete in a timely manner, and has built-in time-outs to prevent truncated or incomplete events.

Configure the pipe fd as line-buffered, or write() full events at once. Be sure the events are flushed: line buffered/unbuffered/fflush()

Output relatively small batches of events

Fetching thousands of event over a few minutes and then outputting them all at once increases the risk of losing data due to a restart. Additionally, outputting small batches of events means your data is searchable sooner and improves script transparency.

This documentation applies to the following versions of Splunk: 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!