Modular inputs overview
About modular inputs
In version 5.0, Splunk introduces Modular Inputs. Modular Inputs allows you to extend the Splunk framework to define a custom input capability. Splunk treats your custom input definitions as if they were part of Splunk's native inputs. The inputs appear automatically on the Splunk Manager > Data Inputs page. From a Splunk Web perspective, your users interactively create and update your custom inputs using Splunk manager, just as they do for Splunk native inputs.
Modular inputs provide the following features:
- Splunk Web automatically provides access to your custom defined inputs.
- You can define whether to launch a single instance or multiple instances. If single instance mode is enabled, each stanza defined in the script is run in the same instance. Otherwise, Splunk launches a separate instance for each stanza.
- You can provide validation for the inputs.
- You can package platform-specific versions of a script. For example you can include a Windows version, a Linux version, and an Apple (Darwin) version in your package.
- You can stream data as plain text, or new with modular inputs, as XML data.
- You can use Splunk REST endpoints to access your modular input scripts
- You can set permissions for these endpoints using Splunk capabilities.
Modular inputs vs. scripted inputs
Modular inputs are ideal for packaging and sharing technology-specific apps or any app that includes a scripted input. Modular inputs presented in Splunk Manager are easier for users to use and understand. You can capture key information without resorting to editing config files. Additionally, modular inputs provide runtime controls and allows you to stream XML to specify per event index-time settings.
The following table highlights the differences between modular inputs and scripted inputs:
|Feature||Scripted Inputs||Modular Inputs|
|Configuration|| Inline arguments
Separate, non-Splunk configuration
| Parameters defined in |
Splunk web fields treated as native Splunk inputs in Splunk Manager
|Specify event boundaries|| Yes
But with additional complexity in your script
XML streaming simplifies specifying event boundaries
|Single instance mode|| Yes
Requires manual implementation
|Multi-platform support||No|| Yes
You can package your script to include versions for separate platforms.
|Schedule runtime Intervals|| Yes
You can specify a cron schedule or otherwise specify when a script runs.
Not available with modular inputs in version 5
Requires manual implementation.
|Run as Splunk user|| Yes
You can specify which Splunk user can run the script.
All modular input scripts are run as Splunk system user.
|Custom REST endpoints||No|| Yes
Modular inputs can be accessed using REST.
|Endpoint permissions||N/A||Access implemented using Splunk capabilities|
Implement modular inputs
To implement modular inputs, you specify a custom input stream and Splunk configuration specifications. It begins with creating the script that streams data to Splunk for indexing. There are several requirements for your script to implement modular inputs. There are also optional procedures you can include in the script to enhance your implementation. You also have to create an input spec file for your script.
Here are the basic steps to create a modular input, with links to the documentation for each step:
- Create a modular input script
- Define a scheme for introspection
- Set up logging
- Set up external validation
- Create a modular input spec file
Here are some of the more advanced features you can implement for modular inputs:
- Enable, disable, and update modular input scripts
- Specify permissions for modular input scripts
- Implement data checkpoints
- Understand how Splunk reads the XML configuration
- Configuration layering for modular inputs
- Create a custom user interface
Developer tools and troubleshooting
Splunk provides some developer tools and troubleshooting tips to assist you in creating modular input scripts:
- REST API access
- Modular inputs configuration utility
- Inputs status endpoint
- Track a modular input script
Modular input examples
The Splunk documentation for modular inputs features two examples:
- Twitter example
This example streams JSON data from a Twitter source to Splunk for indexing.
- Amazon S3 online storage example
This example shows how to use modular inputs to index data from the Amazon S3 online storage web service.
The section Modular inputs examples in this manual provides a complete listing for the examples. The examples are also available for download from Splunkbase.
These examples use Python for the scripting language. However, you can use various other scripting languages to implement modular inputs.
Note: Splunk Universal Forwarder, unlike other Splunk instances, does not provide a Python interpreter. In this case, to run these examples you may need to install Python on the server if one is not already available.
Creating modular inputs with Splunk SDKs
Example script that polls a database
Create modular inputs
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18