Writing reliable scripts

Here are some tips for creating reliable input scripts:

Environment variables

Clear environment variables that can affect your script's operation. One environment variable that is likely to cause problems is the library path. The library path is most commonly known as LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD. It is DYLD_LIBRARY_PATH on OS X, and LIBPATH on AIX.

If you are running external python software or using other python interpreters, consider clearing PYTHONPATH.

Caution: Changing PYTHONPATH may affect other installations of python.

On Windows platforms, the SPLUNK_HOME environment variable is set for you. Avoid changing this environment variable. Changing this variable may interfere with the functioning of Splunk Enterprise services.

Python version

For best results, use the version of Python available from your Splunk Enterprise installation. Splunk Enterprise uses this version to execute system scripts. Use this version of Python to test your scripts.

Some Python libraries that your script requires may not be available in the Splunk platform's version of Python. In this case, you can copy the libraries to the same directory as the scripted input.

To run a script using the version of Python available from Splunk Enterprise:

$SPLUNK_HOME/bin/splunk cmd python <your_script>.py

File paths in Python

Be careful when specifying platform-specific paths and relative paths.

Platform-specific paths

When writing scripts in Python, avoid hard coding platform-specific file paths. Instead specify file paths that can be interpreted correctly on Windows, UNIX, and Mac platforms. For example, the following Python code launches try.py, which is in the bin directory of your app, and has been made cross-compatible with Python 2 and Python 3 using python-future.:

from __future__ import print_function
import os
import subprocess
 
# Edit directory names here if appropriate
if os.name == 'nt':
    ## Full path to your Splunk installation
    splunk_home = 'C:\Program Files\Splunk'
    ## Full path to python executable
    python_bin = 'C:\Program Files (x86)\Python-2.7-32bit\python.exe'
else:
    ## Full path to your Splunk installation
    # For some reason:
    #splunk_home = '/appl/opt/splunk_fwd/'
    # For a sensible OS:
    splunk_home = '/opt/splunk'

    ## Full path to python executable
    # For Mac OS X:
    #python_bin = '/Library/Frameworks/Python.framework/Versions/2.7/bin/python'
    # For a sensible filesystem:
    python_bin = '/usr/bin/python'

try_script = os.path.join(splunk_home, 'etc', 'apps', 'your_app', 'bin', 'try.py')
 
print(subprocess.Popen([python_bin, try_script], stdout=subprocess.PIPE).communicate()[0])

Relative paths

Avoid using relative paths in scripts. Python scripts do not use the current directory when resolving relative paths. For example, on *nix platforms, relative paths are set relative to the root directory (/). The following example shows how to locate the extract.conf file, which is in the same directory as the script:

import os
import os.path

script_dirpath = os.path.dirname(os.path.join(os.getcwd(), __file__))

config_filepath = os.path.join(script_dirpath, 'extract.conf')

Format script output

Format the output of a script so Splunk software can easily parse the data. Also, consider formatting data so it is more human-readable as well.

Use the Common Information Model Add-on

The Common Information Model Add-on is based on the idea that you can break down most log files into three components: fields, event type tags, and host tags.

With these three components you can set up log files in a way that makes them easily processable and that normalizes non-compliant log files, forcing them to follow a similar schema. The Common Information Model Add-on organizes these fields and tags into categories and provides a separate data model for each category. You can use the CIM data models to test your data to ensure that it has been normalized correctly, and then report on it.

You can download the Common Information Model Add-on from Splunkbase here. For a more in-depth overview of the CIM Add-on, see the Common Information Model Add-on Manual.

Timestamp formats

Time stamp the beginning of an event. There are several options for timestamp formats:

RFC-822, RFC-3339

These are standard timestamp formats for email headers and internet protocols. These formats provide an offset from GMT, and thus are unambiguous and more human-readable. RFC-822 and RFC-3339 formats can be handled with %z in a TIME_FORMAT setting.

RFC-822 Tue, 15 Feb 2011 14:11:01 -0800

RFC-3339 2011-02-15 14:11:01-08:00

UTC

UTC formatting may not be as human-readable as some of the other options. If the timestamp is epoch time, no configuration is necessary. Otherwise, requires a configuration in props.conf that declares the input as TZ=UTC.

UTC

2011-02-15T14:11:01-05:00

2011-02-15T14:11:01Z

UTC converted to epoch time

1297738860

Multiline data and field names

For multiline data, find a way to separate events.

• Write a distinctive initial line for a multiline event.

• Use a special end of event string to separate events. For example, use three newline characters to specify an end of an event. The event then includes any single or double newline characters.

• For multiline field values, place the field data inside quotes.

• Use an equals sign, =, or other separator to expose name/value pairs. For example, key=value.

• Configure your Splunk Enterprise instance to use other tokens that might exist in the data.

• Field names are case sensitive. For example the field names "message" and "Message" represent different fields. Be consistent when naming fields.

Create a setup page to configure scripted inputs

If you are packaging an app or add-on for distribution, consider creating a setup page that allows users to interactively provide configuration settings for access to local scripted input resources. For more information, see Enable first-run configuration with setup pages in Splunk Cloud Platform or Splunk Enterprise on the Splunk Developer Portal.

Save state across invocations of the script

Scripts often need to checkpoint their work so subsequent invocations can pick up from where they left off. For example, save the last ID read from a database, mark the line and column read from a text file, or otherwise note the last input read. (See Example script that polls a database.)

You can check point either the index or the script. When check pointing data, keep in mind that the following things are not tied together as a transaction:

• Writing out checkpoint files
• Fully writing data into the pipe between the script and splunkd
• splunkd completely writing out the data into the index

Thus, in the case of hard crashes, it's hard to know if the data the script has acquired has been properly indexed. Here are some of the choices you have:

Search Splunk index One strategy is to have the scripted input search in the Splunk index to find the last relevant event. This is reasonable in an infrequently-launched script, such as one that is launched every 5 or 10 minutes, or at launch time for a script which launches once and stays running indefinitely.

Maintain independent check point Because there is some delay between data being fed to the Splunk platform and the data becoming searchable, a frequently run scripted input must maintain its own checkpoint independent of the index.

Choose a scenario If the script always believes its own checkpoint, data may not be indexed on splunkd or system crash. If the index search is believed, some data may be indexed multiple times on splunkd or system crash. You need to choose which scenario you best fits your needs.

Accessing secured services

Use proper security measures for scripts that need credentials to access secured resources. Here are a few suggestions on how to provide secure access. However, no method is foolproof, so think carefully about your use case and design secure access appropriately:

• Restrict which users can access the app or add-on on disk.
• Create and use credentials specific to the script, with the minimum permissions required to access the data.
• Avoid putting literal passwords in scripts or passing the password as a command line argument, making it visible to all local processes with operating system access.
• Use Splunk Enterprise to encrypt passwords. You can create an app set up page that allows users to enter passwords. See the setup page example with user credentials on the Splunk developer portal. The user can enter a password in plain text, which is stored in the credential stanza in apps.conf. Alternatively, you can specify a python script to securely provide access.

Caution: Splunk Enterprise assembles a secret using locally available random seeds to encrypt passwords stored in configuration files. This method provides modest security against disclosure of passwords from admins with local disk read capability. However, it is not an adequate protection for privileged accounts.

Concurrency issues for scripted inputs

Be careful scheduling two copies of a script running at any given time. Splunk Enterprise detects if another instance of the script is running, and does not launch a new instance if this is the case. For example, if you have a script scheduled to execute every 60 seconds, and a particular invocation takes 140 seconds, Splunk Enterprise detects this and does not launch a new instance until after the long-running instance completes.

At times you may want to run multiple copies of a script, for example to poll independent databases. For these cases, design your scripts so they can handle multiple servers. Also, design your script so that multiple copies can exist (for example, use two app directories for the script).

Alternatively, you could have separate scripts using the same source type.

Troubleshooting scheduled scripts

Splunk Enterprise logs exceptions thrown by scheduled scripts to the splunkd.log file, located here:

$SPLUNK_HOME/var/log/splunk/splunkd.log

Check splunkd.log first if expected events do not appear in the expected index after scheduling the scripted input.

Shutdown and restart issues

Keep these shutdown and restart issues in mind when designing your scripts:

Output at least one event at a time

This makes it easier to avoid reading a partial event if the script is terminated or crashes. Splunk Enterprise expects events to complete in a timely manner, and has built-in time-outs to prevent truncated or incomplete events.

Configure the pipe fd as line-buffered, or write() full events at once. Be sure the events are flushed: line buffered/unbuffered/fflush()

Output relatively small batches of events

Fetching thousands of event over a few minutes and then outputting them all at once increases the risk of losing data due to a restart. Additionally, outputting small batches of events means your data is searchable sooner and improves script transparency.