Configure data collection using a REST API call

In Configure Data Collection, configure the data inputs and setup page for your add-on. This page displays a list of data inputs that you have configured for your add-on. Data inputs determine how you will get data into your add-on:

• Use a REST API call, described in this topic
• Run a shell command script
• Create a modular input from your own Python code

Once you have configured data inputs for your add-on, you can manage them from the Configure Data Collection page in Add-on Builder. Data inputs you configure for your add-on also appear on the Inputs page of the add-on.

• Data inputs must be enabled so that Splunk Enterprise can start indexing the data from them. You'll need indexed data if you want to perform field extractions and CIM mapping. Data inputs you configure are enabled by default.
• For each data input you configure, the Add-on Builder creates an input called "TAB_example", which is visible on the Inputs page of the add-on. Do not modify this input, which is used by the Add-on Builder. Instead, if you want to change settings for the data input, access the options from the Configure Data Collection page in Add-on Builder.

Users of your add-on should create data inputs from your configuration within the add-on. Do not direct users to use the Settings > Data inputs page in Splunk Enterprise. When users open your add-on, the Inputs page provides the option to create a new input.

Add a data input using a REST API

Build the data collection for your add-on to gather data from a REST API. A REST data input uses JSON as a data type, and supports basic authentication and API-based authentication. For advanced data collection, create a modular input by writing your own Python code.

1. On your add-on homepage, click the Configure Data Collection icon.
2. On the Configure Data Collection page, click Add Data.
   The Create Data Input wizard starts.
3. On the Choose Input Method page, click Modular input using a REST API.
4. On the Data Input Properties tab:
• Enter a name for the source type to create for this data input.
• Enter an input display name, internal name, and description.
• Specify a collection interval indicating how often to retrieve data using this input, in seconds.
5. Optionally, click the Data Input Parameters tab to define any parameters that are required for users to configure your data input.
   For each parameter, drag a field from the Component Library to the center panel, then specify its properties in the Property Editor.
7. Optionally, click the Add-on Setup Parameters tab to define any parameters that are required for users to configure your add-on.
• Select predefined options to prompt for account, proxy, or logging settings.
• Add custom inputs by dragging fields from the Component Library to the center panel panel. Select an input to specify its properties in the Property Editor.

For more about setup pages, see Create a setup page.

8. Click Next.
9. On the Data Input Definition tab under REST settings:
• Specify a REST URL for a REST endpoint.
• Select the REST method (GET or POST).
• If you are using the GET method, specify the names and values for the REST URL parameters.
  You can add parameters individually to the REST header or body, or add them directly to the URL. For details, see Pass values from data input parameters and Pass values from setup parameters.
• Optionally, specify the names and values for REST request headers.
• Optionally, if you are using the POST method, specify the names and values for the REST request body.
• If you added a Global Account field to your data input parameters, select Enable basic authentication if you want to include the username and password values in the HTTP header using HTTP Basic Authentication.



10. On the Data Input Definition tab under Data input parameters, enter a sample value for each data input parameter for testing purposes.
    To use a value from a data input parameter in the REST URL, use the format ${internal_name}, using the internal name defined for the parameter in the previous step. For details, see Pass values from data input parameters and Pass values from setup parameters.



11. On the Data Input Definition tab under Event extraction settings, optionally specify how to break the JSON payload into individual events.
    Use this setting when the results from the API return a JSON array of events, and you would like to break them into individual events before indexing them.
    Specify the JSON path to that array within the payload to use for event breaking. When the JSON path matches the array in the payload, the extracted events are highlighted in the test output. For more, see JSON path formats.



12. On the Data Input Definition tab under Checkpoint settings, if you want to set a checkpoint select Enable checkpointing.

A checkpoint allows you to read and index only the data that is new since the last time data was received. For more about checkpoints, including a sample configuration, see Use checkpoints for incremental data collection.

Specify the checkpoint settings:

• For Checkpoint parameter name, enter the REST parameter name to use as a checkpoint. Make sure you use the ${param_name} format to replace the checkpoint value in the REST URL definition.
• For Checkpoint field path, enter the JSON path to the field in the response payload to use as the checkpoint. For more about specifying the checkpoint field path, see JSON path formats.
• For Checkpoint initial value, enter the initial value to use for the checkpoint. Users can modify this value when creating a data input.
• If the timestamp format of the request is different from the response, specify values for Response timestamp format and Request timestamp format. If both timestamps use the same format, leave these fields empty. For more, see Normalize timestamp formats.



13. If you created a setup page, click the Add-on Setup Parameters tab to enter sample values for setup parameters for testing purposes.
14. Click the Data Input Definition tab, then click Test to run the REST command and test the data input configuration.
    The Output section indicates whether the REST command succeeded (sample data is displayed) or failed (the error message is displayed).



15. Click Save to save the data input, then click Finish.

The Add-on Builder displays a summary of the new data input, along with suggested next steps.

Add authentication to a data input

When you need to use authentication for a REST input, add a Global Setup input field to your data input parameters form:

The Global account settings option is automatically selected on your setup page.

When users first open your add-on, they set up their accounts on the add-on's Configuration page:

Then, users configure new inputs on the add-on's Inputs page, which displays the data input parameters you configured:

Use checkpoints for incremental data collection

When creating your REST API data input, you can use checkpoints to mark where data has already been read and indexed, indicating where to resume reading data the next time it is collected, thereby avoiding collecting the same data again. Checkpoints are useful for REST APIs that return a number of events per call. The checkpoint field is typically a timestamp, but you can specify any field. You do not need to use checkpoints with a REST API that returns a point-in-time value, such as metrics.

When using checkpoints, the REST API should do the following:

• Allow you to sort fields by the one you want to use for checkpoints. Otherwise, that field can't be used as a marker that indicates where new data begins.
• Be able to query data by the checkpoint field, starting with the initial checkpoint value.
• Allow timestamps in increments at least as small as the collection interval. For example, if your collection interval is 30 seconds, a timestamp in days would not be a useful checkpoint.

For an end-to-end walkthrough, see Walkthrough: Configure a data input with checkpoints on the Splunk Developer Portal.

Pass values from data input parameters

You can pass parameter values to a variable in a REST URL, a REST header, a REST body, and in a shell command. Following the Mako syntax, use the format ${varname} for the variable. Then, define a modular input variable for varname.

For example, let's say you want to use a URL such as http://localhost:5111/api/task/<taskID>, where <taskID> is a variable value.

When you define the REST API input, on the Inputs & Parameters step, click the Data Input Parameters tab and add an input variable for Task ID with the internal name "taskid":

Next, on the Define & Test step when you specify the REST URL, use "http://localhost:5111/api/${taskid}". To test it, use an initial value, such as "t123". The resulting REST URL is "http://localhost:5111/api/task/t123".

When users open your add-on, they can create instances of this data input on the Inputs page. The form displays the Task ID field as one of the data input parameters you configured:

Pass values from setup parameters

When referring to setup parameters, you must include a namespace string before the parameter name to get the value from the parameter. The following shows an example, where param_name is the parameter name:

${__settings__.additional_parameters.param_name}

For example, let's say you have a text input on your setup page that prompts the user for an API token, with the internal name "api_token". To pass the value of the API token to the REST call, use the following format:

api-key=${__settings__.additional_parameters.api_token}

We recommend you do not set account, proxy, or logging fields directly. You can access the global account values as follows:

${global_account.username}
{{global_account.username}}

${global_account.password}
{{global_account.password}}

JSON path formats

Use the following guidelines to form JSON paths when creating REST inputs.

Operator	Description
$	The root element to query.
*	Wildcard. Available anywhere a name or number is required.
..	Deep scan. Available anywhere a name is required.
.<name>	Dot-notated child.
['<name>']	Bracket-notated child.
[<number>]	Array index or indexes.

Example

Here is an example JSON response:

{
    "count": 251,
    "results": [
        {
            "isocode": "MZ",
            "country": "Mozambique"
        },
        {
            "isocode": "PG",
            "country": "Papua New Guinea"
        }
    ]
}

The following table shows different ways of accessing data in the JSON response.

JSON path	Returned events
$.count	251
$.results[*] or $.results	{"isocode": "MZ", "country": "Mozambique"} {"isocode": "PG","country": "Papua New Guinea"}
$.results[0]	{"isocode": "MZ", "country": "Mozambique"}
$.results[-1]	{"isocode": "PG","country": "Papua New Guinea"}
$.results[0].country	Mozambique
$['results'][0]['isocode']	MZ
$.results[0].*	MZ Mozambique
$..isocode	MZ PG

When the JSON path matches a field in the payload, the extracted value is highlighted in the test output.

Normalize timestamp formats

When you set a checkpoint, you only need to specify timestamp formats when the format of the request timestamp is different from the response timestamp. If these formats are different, use the following format for timestamps:

%Y-%m-%dT%H:%M:%S.%f+08:00

Additional notes:

• "0" left padding is added to months, days, hours, minutes, and seconds. To remove left padding, use "-". For example, use %-m.
• "0" right padding is added to milliseconds.
• Add-on Builder supports the time format as described in the Python function time_str2str for transforming a time string to a different format.
• The %Nf format is supported for to_format only.
• Due to a limitation in Python 2.x, the date-time formatting codes %z for "time zone name" and %Z for "time zone offset from GMT" are not supported.

End-to-end walkthroughs

For end-to-end walkthroughs, see the Splunk Developer Portal.

Learn more

For more information, see the following Splunk Enterprise documentation:

• How timestamp assignment works in the Getting Data In manual
• Source type naming conventions in the Splunk Add-ons manual
• Modular inputs overview in the Developing Views and Apps for Splunk Web manual
• How to create modular inputs using Splunk SDK for Python on the Splunk Developer Portal
• Create a setup page for a Splunk app on the Splunk Developer Portal