Modular inputs overview
About modular inputs
A modular input is a Splunk Enterprise app or add-on that extends the Splunk Enterprise framework to define a custom input capability. Splunk Enterprise treats your custom input definitions as if they were Splunk Enterprise native inputs. The inputs appear automatically on the Settings > Data Inputs page. From a Splunk Web perspective, your users interactively create and update your custom inputs using Settings, just as they do for Splunk Enterprise native inputs.
Splunk Enterprise data sources
Splunk Enterprise has various ways to input data:
- Monitor files and directories
- Listen on TCP or UDP ports for network events
- Read the output from a script
The following are typical use cases for scripts. You can use traditional scripted inputs or modular inputs for these use cases.
- Stream results from a command, such as
- Query a database, web service, or API
- Reformat complex data
- Handle sensitive information more securely
- Handle special characters in inputs
Modular input features
Modular inputs provide the following features:
- Splunk Web automatically provides access to your custom defined inputs.
- You can provide validation for the inputs.
- You can package platform-specific versions of a script. For example you can include a Windows version, a Linux version, and an Apple (Darwin) version in your package.
- You can stream data as XML data, which allows you to annotate the script output. This gives you greater control of how Splunk Enterprise processes the data.
- You can use Splunk Enterprise REST endpoints to access your modular input scripts
- You can set permissions for these endpoints using Splunk Enterprise capabilities.
- You can define whether to launch a single instance or multiple instances. Single instance mode is useful when running in a single-threaded environment.
Modular inputs vs. scripted inputs
Modular inputs are ideal for packaging and sharing technology-specific apps or any app that includes a scripted input. Modular inputs presented in Splunk Enterprise Settings are easier for users to use and understand. You can capture key information without resorting to editing config files. Additionally, modular inputs provide runtime controls and allows you to stream XML to specify per event index-time settings.
The following table highlights the differences between modular inputs and scripted inputs:
|Feature||Scripted Inputs||Modular Inputs|
Separate, non-Splunk Enterprise configuration
|Parameters defined in |
Splunk Web fields treated as native inputs in Settings
|Specify event boundaries||Yes
But with additional complexity in your script
XML streaming simplifies specifying event boundaries
|Single instance mode||Yes
Requires manual implementation
You can package your script to include versions for separate platforms.
Requires manual implementation.
|Run as Splunk Entrprise user||Yes
You can specify which Splunk Enterprise user can run the script.
All modular input scripts are run as Splunk Enterprise system user.
|Custom REST endpoints||No||Yes
Modular inputs can be accessed using REST.
|Endpoint permissions||N/A||Access implemented using Splunk Enterprise capabilities|
Implement modular inputs
To implement modular inputs, you specify a custom input stream and configuration specifications. It begins with creating the script that streams data for indexing. There are several requirements for your script to implement modular inputs. There are also optional procedures you can include in the script to enhance your implementation. You also have to create an input spec file for your script.
Basic steps to create modular inputs
Here are the basic steps to create a modular input, with links to the documentation for each step:
- Create a modular input script
- Define a scheme for introspection
- Set up logging
- Set up external validation
- Create a modular input spec file
Here are some of the more advanced features you can implement for modular inputs:
- Enable, disable, and update modular input scripts
- Specify permissions for modular input scripts
- Implement data checkpoints
- Understand how Splunk reads the XML configuration
- Configuration layering for modular inputs
- Create a custom user interface
Developer tools and troubleshooting
Splunk provides some developer tools and troubleshooting tips to assist you in creating modular input scripts:
- REST API access
- Modular inputs configuration utility
- Inputs status endpoint
- Track a modular input script
Modular input examples
The Modular inputs basic example provides a basic, Hello World style, introduction to modular inputs.
Modular inputs examples provides two examples that detail advanced features.
- Twitter example
This example streams JSON data from a Twitter source to Splunk for indexing.
- Amazon S3 online storage example
This example shows how to use modular inputs to index data from the Amazon S3 online storage web service.
The section Modular inputs examples in this manual provides a complete listing for the examples. The examples are also available for download from Splunk Apps.
These examples use Python for the scripting language. However, you can use various other scripting languages to implement modular inputs.
Note: Splunk Universal Forwarder, unlike other Splunk instances, does not provide a Python interpreter. In this case, to run these examples install Python on the server if one is not already available.
Creating modular inputs with Splunk SDKs
Example script that polls a database
Modular inputs basic example
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15