Create custom data inputs for Splunk Cloud Platform or Splunk Enterprise
The Splunk platform handles many types of data, but when you need to get data from a data source or format that isn't natively supported by the Splunk platform, consider creating a custom data input.
Create a custom data input if you have any of the following objectives:
- Query a database, web service, or API
- Reformat complex data
- Stream results from a command, such as
vmstat
oriostat
- Remove or elide sensitive information from data before ingesting it
- Handle special characters in inputs
The most common types of custom data inputs are modular inputs and scripted inputs.
About modular inputs
A modular input is a Splunk app that defines a custom input capability. Because modular inputs are treated as native inputs, system administrators and end users can select and configure them like any other input in the Splunk platform.
Modular inputs are ideal for packaging and sharing technology-specific data sources. Modular inputs enable users to interact with key information using Splunk Web, without needing to edit configuration files. Modular inputs also provide runtime controls and allow the input to specify per-event index-time settings.
Modular inputs let you perform the following actions:
- Provide users with access to modular inputs on the Settings > Data Inputs page in Splunk Web.
- Validate modular input configurations.
- Package platform-specific versions of the modular input scripts. For example, you can include a Windows version, a Linux version, and MacOS version in one package.
- Stream data in XML format, allowing you to annotate the script output and manage how the data is processed.
- Use Splunk REST endpoints to access modular input scripts. You can use capability settings to manage endpoint permissions.
- Launch a single instance or multiple instances of a modular input.
To get started, see Overview of modular inputs for Splunk Cloud Platform or Splunk Enterprise.
About scripted inputs
A scripted input is an executable script in a Splunk app that feeds event data to a Splunk platform instance from nonstandard sources such as APIs, remote data interfaces, message queues, and Windows sources such as Active Directory, Windows Management Interface (WMI), and Registry. You can use shell scripts, Python scripts, or any other utility that can format and stream the data that you want to index.
Use a script to do the following tasks:
- Prepare data so that events and extracted fields can be properly parsed.
- Stream data to the Splunk platform for indexing. The script runs at a specified interval. The
stdout
data stream from the script is indexed. - Write data from a script to a file for indexing, similar to monitoring a file. You can configure your Splunk platform deployment to run the script at specific intervals rather than configuring an external method, such as a cron job or a Windows scheduled task, to start the script.
Typical use cases for scripted inputs include the following:
- Access data that is not available as an ordinary file
- Access data that cannot be pushed to a Splunk platform deployment from its source
- Stream data from command-line tools, such as
vmstat
andiostat
- Poll a database, web service, or API for specific data and process the results
- Reformat complex data to more easily parse the data into events and fields
- Maintain data sources with slow or resource-intensive startup procedures
- Provide special or complex handling for transient or unstable inputs
- Manage passwords and credentials in the script
- Creaate a wrapper script for command-line inputs that contain special characters
To get started, see Scripted input examples for Splunk Cloud Platform or Splunk Enterprise.
Feature comparison of modular and scripted inputs
The following table compares how modular and scripted inputs support different features.
Feature | Modular inputs | Scripted inputs |
---|---|---|
Checkpointing | Supported | Supported, but requires manual implementation |
Configuration | Parameters defined in inputs.conf Users can configure inputs using Splunk Web input Settings fields. Validation support | Inline arguments Separate configuration outside of the Splunk platform |
REST management API | Supported Access modular inputs using REST | Not supported |
Event boundaries | Supported XML streaming simplifies specifying event boundaries. | Available, but requires additional script complexity |
Multi-platform support | Supported Developers can package a modular input script to include versions for separate platforms. | Not supported |
REST endpoint authorization | Supported Use capabilities to control access. | n/a |
Run as user | Not supported All modular input scripts are run as the user using the Splunk instance. | Supported You can specify which user can run the script |
Single instance mode | Supported | Supported, but requires manual implementation |
Deployment considerations for custom data inputs
In a distributed deployment, the location where a user installs a custom data input depends on their Splunk Cloud Platform Experience (Classic or Victoria). In Classic Experience, custom data inputs run on the the Inputs Data Manager (IDM). If you deploy an app with a custom data input to the search head or indexer, the input does not run on these components. In Victoria Experience, custom data inputs run on the search head and don’t require the IDM.
As a best practice for public apps, include information about where to deploy the app in the app's listing on Splunkbase.
For more information about the different Splunk Cloud Platform Experiences, see Determine your Splunk Cloud Platform Experience in the Splunk Cloud Admin Manual.
For more information about where to deploy an app's files and knowledge objects, see How deployment architecture affects a Splunk app in Splunk Cloud Platform or Splunk Enterprise.
See also
- Monitor data in the Splunk Enterprise Getting Data In manual
- Get data from APIs and other remote data interfaces through scripted inputs in the Splunk Enterprise Getting Data In manual
- Use a wrapper script in the Splunk Enterprise Getting Data In manual