Writing reliable scripts
Here are some tips for creating reliable input scripts:
Clear environment variables that can affect your script's operation. One environment variable that is likely to cause problems is the library path. The library path is most commonly known as LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD. It is DYLD_LIBRARY_PATH on OS X, and LIBPATH on AIX.
If you are running external python software or using other python interpreters, consider clearing PYTHONPATH.
- Caution: Changing PYTHONPATH may affect other installations of python.
On Windows platforms, the SPLUNK_HOME environment variable is set for you. Avoid changing this environment variable. Changing this variable may interfere with the functioning of Splunk Enterprise services.
For best results, use the version of Python available from your Splunk Enterprise installation. Splunk Enterprise uses this version to execute system scripts. Use this version of Python to test your scripts.
Some Python libraries that your script requires may not be available in the Splunk platform's version of Python. In this case, you can copy the libraries to the same directory as the scripted input.
To run a script using the version of Python available from Splunk Enterprise:
$SPLUNK_HOME/bin/splunk cmd python <your_script>.py
File paths in Python
Be careful when specifying platform-specific paths and relative paths.
When writing scripts in Python, avoid hard coding platform-specific file paths. Instead specify file paths that can be interpreted correctly on Windows, UNIX, and Mac platforms. For example, the following Python code launches
try.py, which is in the
bin directory of your app:
import os import subprocess # Edit directory names here if appropriate if os.name == 'nt': ## Full path to your Splunk installation splunk_home = 'C:\Program Files\Splunk' ## Full path to python executable python_bin = 'C:\Program Files (x86)\Python-2.7-32bit\python.exe' else: ## Full path to your Splunk installation # For some reason: #splunk_home = '/appl/opt/splunk_fwd/' # For a sensible OS: splunk_home = '/opt/splunk' ## Full path to python executable # For Mac OS X: #python_bin = '/Library/Frameworks/Python.framework/Versions/2.7/bin/python' # For a sensible filesystem: python_bin = '/usr/bin/python' try_script = os.path.join(splunk_home, 'etc', 'apps', 'your_app', 'bin', 'try.py') print subprocess.Popen([python_bin, try_script], stdout=subprocess.PIPE).communicate()
Avoid using relative paths in scripts. Python scripts do not use the current directory when resolving relative paths. For example, on *nix platforms, relative paths are set relative to the root directory (
/). The following example shows how to locate the
extract.conf file, which is in the same directory as the script:
import os import os.path script_dirpath = os.path.dirname(os.path.join(os.getcwd(), __file__)) config_filepath = os.path.join(script_dirpath, 'extract.conf')
Format script output
Format the output of a script so Splunk software can easily parse the data. Also, consider formatting data so it is more human-readable as well.
Use the Common Information Model Add-on
The Common Information Model Add-on is based on the idea that you can break down most log files into three components: fields, event type tags, and host tags.
With these three components you can set up log files in a way that makes them easily processable and that normalizes non-compliant log files, forcing them to follow a similar schema. The Common Information Model Add-on organizes these fields and tags into categories and provides a separate data model for each category. You can use the CIM data models to test your data to ensure that it has been normalized correctly, and then report on it.
Time stamp the beginning of an event. There are several options for timestamp formats:
These are standard timestamp formats for email headers and internet protocols. These formats provide an offset from GMT, and thus are unambiguous and more human-readable. RFC-822 and RFC-3339 formats can be handled with
%z in a TIME_FORMAT setting.
Tue, 15 Feb 2011 14:11:01 -0800
UTC formatting may not be as human-readable as some of the other options. If the timestamp is epoch time, no configuration is necessary. Otherwise, requires a configuration in
props.conf that declares the input as TZ=UTC.
- UTC converted to epoch time
Multiline data and field names
For multiline data, find a way to separate events.
- Write a distinctive initial line for a multiline event.
- Use a special end of event string to separate events. For example, use three newline characters to specify an end of an event. The event then includes any single or double newline characters.
- For multiline field values, place the field data inside quotes.
- Use an equals sign,
=, or other separator to expose name/value pairs. For example,
- Configure your Splunk Enterprise instance to use other tokens that might exist in the data.
- Field names are case sensitive. For example the field names “message” and “Message” represent different fields. Be consistent when naming fields.
Write a setup screen to configure scripted inputs.
If you are packaging an app or add-on for distribution, consider creating a setup screen that allows users to interactively provide configuration settings for access to local scripted input resources. Include an input stanza for your script so setup.xml doesn’t require a custom endpoint. See Create a setup page for a Splunk app on the Splunk developer portal.
Refer to the *Nix and Windows apps for examples on using setup.xml pages to create a setup screen. These apps are available for download from Splunkbase.
Save state across invocations of the script
Scripts often need to checkpoint their work so subsequent invocations can pick up from where they left off. For example, save the last ID read from a database, mark the line and column read from a text file, or otherwise note the last input read. (See Example script that polls a database.)
You can check point either the index or the script. When check pointing data, keep in mind that the following things are not tied together as a transaction:
- Writing out checkpoint files
- Fully writing data into the pipe between the script and splunkd
- splunkd completely writing out the data into the index
Thus, in the case of hard crashes, it’s hard to know if the data the script has acquired has been properly indexed. Here are some of the choices you have:
Search Splunk index One strategy is to have the scripted input search in the Splunk index to find the last relevant event. This is reasonable in an infrequently-launched script, such as one that is launched every 5 or 10 minutes, or at launch time for a script which launches once and stays running indefinitely.
Maintain independent check point Because there is some delay between data being fed to the Splunk platform and the data becoming searchable, a frequently run scripted input must maintain its own checkpoint independent of the index.
Choose a scenario If the script always believes its own checkpoint, data may not be indexed on splunkd or system crash. If the index search is believed, some data may be indexed multiple times on splunkd or system crash. You need to choose which scenario you best fits your needs.
Accessing secured services
Use proper security measures for scripts that need credentials to access secured resources. Here are a few suggestions on how to provide secure access. However, no method is foolproof, so think carefully about your use case and design secure access appropriately:
- Restrict which users can access the app or add-on on disk.
- Create and use credentials specific to the script, with the minimum permissions required to access the data.
- Avoid putting literal passwords in scripts or passing the password as a command line argument, making it visible to all local processes with operating system access.
- Use Splunk Enterprise to encrypt passwords. You can create an app set up page that allows users to enter passwords. See the setup page example with user credentials on the Splunk developer portal. The user can enter a password in plain text, which is stored in the credential stanza in apps.conf. Alternatively, you can specify a python script to securely provide access.
- Caution: Splunk Enterprise assembles a secret using locally available random seeds to encrypt passwords stored in configuration files. This method provides modest security against disclosure of passwords from admins with local disk read capability. However, it is not an adequate protection for privileged accounts.
Concurrency issues for scripted inputs
Be careful scheduling two copies of a script running at any given time. Splunk Enterprise detects if another instance of the script is running, and does not launch a new instance if this is the case. For example, if you have a script scheduled to execute every 60 seconds, and a particular invocation takes 140 seconds, Splunk Enterprise detects this and does not launch a new instance until after the long-running instance completes.
At times you may want to run multiple copies of a script, for example to poll independent databases. For these cases, design your scripts so they can handle multiple servers. Also, design your script so that multiple copies can exist (for example, use two app directories for the script).
Alternatively, you could have separate scripts using the same source type.
Troubleshooting scheduled scripts
Splunk Enterprise logs exceptions thrown by scheduled scripts to the
splunkd.log file, located here:
splunkd.log first if expected events do not appear in the expected index after scheduling the scripted input.
Shutdown and restart issues
Keep these shutdown and restart issues in mind when designing your scripts:
Output at least one event at a time
This makes it easier to avoid reading a partial event if the script is terminated or crashes. Splunk Enterprise expects events to complete in a timely manner, and has built-in time-outs to prevent truncated or incomplete events.
Configure the pipe
fd as line-buffered, or
write() full events at once. Be sure the events are flushed:
Output relatively small batches of events
Fetching thousands of event over a few minutes and then outputting them all at once increases the risk of losing data due to a restart. Additionally, outputting small batches of events means your data is searchable sooner and improves script transparency.
Setting up a scripted input
Example script that polls a database
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2