Configure Data Collection

In the Configure Data Collection section, configure the data inputs and setup page for your add-on. This page displays a list of data inputs that you have configured for your add-on. Data inputs determine how you will get data into your add-on:

• Using a REST API call
• Running a shell command script
• Using a modular input created from your own Python code

You can also use this section to configure the setup options for your add-on.

Once you have added data inputs to your add-on, you can manage them from this page. Data inputs you create for your add-on also appear on the Settings > Data Inputs page.

Notes

• Data inputs must be enabled so that Splunk can start indexing the data from them. You'll need indexed data if you want to perform field extractions and CIM mapping. Data inputs you create are enabled by default.
• For each data input you create, the Add-on Builder creates an input called "TAB_example", which is visible on the Settings > Data Inputs page. Do not modify this input, which is used by the Add-on Builder. Instead, if you want to change settings for the data input, access the options from the Configure Data Collection page in the Add-on Builder.

Add a data input using a REST API

Build the data collection for your add-on to gather data from a REST API that doesn’t require authentication or that uses authentication that can be requested statically in an HTTP header. For more complex authentication, use a modular input created from your own Python code.

1. On the Step 2: Configure Data Collection page, click Add Data.
2. Select Modular input using a REST API.
3. Enter a name for the sourcetype to create for this data input.
4. Enter an input name, title, and description.
5. Specify a collection interval indicating how often to retrieve data using this input, in seconds.
6. Specify a URL for a REST endpoint.
7. Select the REST method (GET or POST).
8. Optionally, specify REST headers and parameters for the REST call.

Note: You can pass parameter values to a variable in a REST URL, a REST header, a REST body. For more, see Use variables in the REST API and shell commands.

9. Click Test to run the REST command and test the data input configuration. The Output section indicates whether the command succeeded (sample data is displayed) or failed (the error message is displayed).
10. Click Save to save the data input.

The Add-on Builder returns to the main page of Step 2: Configure Data Collection. Your new data input is now displayed in the list.

Click Add Data to add another data input, or click Next to save your changes and proceed to the next section. You are prompted to restart Splunk Enterprise if you have added data inputs.

Add a data input using shell commands

Get data into your add-on by using a script of shell commands.

1. On the Step 2: Configure Data Collection page, click Add Data.
2. Select Modular input using shell commands.
3. Enter a name for the sourcetype to create for this data input.
4. Enter an input name, title, and description.
5. Specify a collection interval indicating how often to retrieve data using this input, in seconds.
6. Enter the shell commands.

Note: You can pass parameter values to a variable in a shell command. For more, see Use variables in the REST API and shell commands.

7. Click Test to run the shell command script and test the data input configuration. The Output section indicates whether the command succeeded (sample data is displayed) or failed (the error message is displayed).
8. Click Save to save your data input.

The Add-on Builder returns to the main page of Step 2: Configure Data Collection. Your new data input is now displayed in the list.

Click Add Data to add another data input, or click Next to save your changes and proceed to the next section. You are prompted to restart Splunk Enterprise if you have added data inputs.

Add a data input by writing your own code

Use this option to implement your own data collection mechanism using Python code.

1. On the Step 2: Configure Data Collection page, click Add Data.
2. Select Modular input using your own Python code.
3. Enter a name for the sourcetype to create for this data input.
4. Enter an input name, title, and description.
5. Specify a collection interval indicating how often to retrieve data using this input, in seconds.
6. Specify any variables for the modular input. At least one variable is required.
7. Click Save to save your data input.
8. When prompted to add your code now or later, click Add Now to add the code now. (To add code later, click Add Later, find your data input in the list on Step 2: Configure Data Collection, and click Code for that data input.)

The Add-on Builder opens a Code page, where you can edit the modular input code that has been auto-generated:

1. Edit the Python code to create your modular input by finding # TODO implement your own logic here, and inserting your Python code after that statement. (As an example, the auto-generated code contains commented-out code that generates a random number.)
2. Click Test to test your code. The Output section indicates whether the command succeeded (sample data is displayed) or failed (the error message is displayed).
3. Click Save to save your data input.

The Add-on Builder returns to the main page of Step 2: Configure Data Collection. Your new data input is now displayed in the list.

Click Add Data to add another data input, or click Next to save your changes and proceed to the next section. You are prompted to restart Splunk Enterprise if you have added data inputs.

Use variables in the REST API and shell commands

You can pass parameter values to a variable in a REST URL, a REST header, a REST body, and in a shell command. Following the Mako syntax, use the format ${varname} for the variable. Then, define a modular input variable for varname.

For example, let's say you want to use a URL such as http://localhost:5111/api/task/<taskID>, where "taskID" is a variable value. When you define the REST API input, use the URL "http://localhost:5111/api/${taskid}". Then, define a modular input variable "taskid" and assign an initial value, such as "t123". The resulting URL is "http://localhost:5111/api/task/t123".

Later, when your add-on is installed, go to Settings > Data inputs and find the data input you created. You'll need to create a new configuration for the input and when you do, you are prompted to specify a value for the "taskid" variable:

Configure setup options

If you need to prompt your users for information when the user first runs your add-on, you can configure setup options that are displayed to the user as a setup page. Use a setup page when:

• You need to authenticate users with a username and password, for example to authenticate a third-party API for a modular input.
• You need the user to provide proxy information, for example when you need to use a proxy to make outbound API calls from the add-on.

By including a setup page in your add-on, the user provides the information to run your add-on without having to edit a configuration file directly. Predefined setup options include credentials (such as a username and password) and proxy settings. You can also add any custom variables you want for your add-on.

The setup page is displayed the first time the user runs the add-on, if the add-on is visible. The "is_configured" setting in the $SPLUNK_HOME/etc/apps/TA_add-on_name/local/apps.conf file indicates whether the add-on has been configured for that user's installation.

Users can also open the setup page in Splunk Web by clicking the Manage Apps icon, then clicking Set up under the actions for your add-on.

Here's an example configuration in the Add-on Builder:

Here's the resulting setup page for this configuration in Splunk Web:

To configure setup options:

1. On the Step 2: Configure Data Collection page, click Configure Setup Info.
2. On the Step 2: Configure Data Collection > Configure Setup Info page, select the options you want to prompt users for:
• Select Add credentials to prompt for a username and password.
• Select Add proxy variables to prompt for proxy server settings, such as host, port, and proxy type.
• To prompt for custom settings, add custom variables under Add other variables: enter a variable name, description, and value type ("Boolean" for a checkbox, "String" for a string field, or "Password" for a masked string field).
3. Click Save to save your setup configuration and return to the Step 2: Configure Data Collection page.

Learn more

For more information, see the following Splunk Enterprise documentation:

• Source type naming conventions in the Splunk Add-ons manual.
• Modular inputs overview in the Developing Views and Apps for Splunk Web manual
• How to create modular inputs using Splunk SDK for Python on the Splunk Developer Portal
• Create a setup page for a Splunk app on the Splunk Developer Portal