Configure data collection using a REST API call

Use the Configure Data Collection page to configure the data inputs and setup page for your add-on. The Configure Data Collection page provides a list of data inputs that are configured for your add-on. Data inputs determine the method used to get data into your add-on:

• a data input using a REST API call A REST API call
• A shell command script
• Modular input from your own Python code

Once you have configured data inputs for your add-on, you can manage them from the Configure Data Collection page in the Add-on Builder. You data inputs also appear on the add-on Inputs page.

Data inputs must be enabled for Splunk Enterprise to indexing their data. Indexed data is required in order to perform field extractions and data model mapping. Data inputs that you configure are enabled by default.

Your add-on users create data inputs within your add-on. When users open your add-on, the Inputs page provides the option to create a new input. Do not direct users to use the Settings > Data inputs page in Splunk Enterprise.

Add a data input using a REST API

Build the data collection for your add-on to gather data from a REST API. A REST data input uses JSON as a data type and supports basic authentication and API-based authentication. For advanced data collection, create a modular input by writing your own Python code.

1. On your add-on homepage, click the Configure Data Collection icon.
2. On the Configure Data Collection page, click New Input.
   The Create Data Input wizard starts.
3. On the Choose Input Method page, click Modular input using a REST API.
4. On the Data Input Properties tab:
• Enter a name for the source type to create for this data input.
• Enter an input display name, internal name, and description.
• Specify a collection interval indicating how often to retrieve data using this input, in seconds.
5. Optionally click the Data Input Parameters tab to define any parameters that are required for users to configure your data input. For each parameter, drag a field from the Component Library to the center panel, then specify properties in the Property Editor.
6. Optionally, click the Add-on Setup Parameters tab to define any parameters that are required for users to configure your add-on.
• Select predefined options to prompt for account, proxy, or logging settings.
• Add custom inputs by dragging fields from the Component Library to the center panel. Select an input to specify its properties in the Property Editor.

For more about setup pages, see Create a setup page.

7. Click Next.
8. On the Data Input Definition tab under REST settings:
• Specify a REST URL for a REST endpoint.
• Select the REST method (GET or POST).
• If you are using the GET method, specify the names and values for the REST URL parameters.
  You can add parameters individually to the REST header or body, or add them directly to the URL. For details, see Pass values from data input parameters and Pass values from setup parameters.
• Optionally specify the names and values for REST request headers.
• If you are using the POST method, optionally specify the names and values for the REST request body.
• If you added a Global Account field to your data input parameters, optionally select Enable basic authentication to include the username and password values in the HTTP header using HTTP Basic Authentication.
9. On the Data Input Definition tab under Data input parameters, enter a sample value for each data input parameter for testing purposes.
   To use a value from a data input parameter in the REST URL, use the format ${internal_name}, using the internal name defined for the parameter in the previous step. For details, see Pass values from data input parameters and Pass values from setup parameters.
10. On the Data Input Definition tab under Event extraction settings, optionally specify how to break the JSON payload into individual events.
    Use this setting when the results from the API return a JSON array of events, and you would like to break them into individual events before indexing them.
    Specify the JSON path to that array within the payload to use for event breaking. When the JSON path matches the array in the payload, the extracted events are highlighted in the test output. For more, see JSON path formats.
11. On the Data Input Definition tab under Checkpoint settings, optionally select Enable checkpointing to set a checkpoint .

A checkpoint allows you to read and index only the most recent data received (data received after the last most recent data receipt). For more about checkpoints, including a sample configuration, see Use checkpoints for incremental data collection.

Specify the checkpoint settings:

• For Checkpoint parameter name, enter the REST parameter name to use as a checkpoint. Make sure you use the ${param_name} format to replace the checkpoint value in the REST URL definition.
• For Checkpoint field path, enter the JSON path to the field in the response payload to use as the checkpoint. For more about specifying the checkpoint field path, see JSON path formats.
• For Checkpoint initial value, enter the initial value to use for the checkpoint. Users can modify this value when creating a data input.
• If the timestamp format of the request is different from the response, specify values for Response timestamp format and Request timestamp format. If both timestamps use the same format, leave these fields empty. For more, see Normalize timestamp formats.
12. If you created a setup page, click the Add-on Setup Parameters tab to enter sample values for setup parameters for testing purposes.
13. Click the Data Input Definition tab, then click Test to run the REST command and test the data input configuration.
    The Output section indicates whether the REST command succeeded (sample data is displayed) or failed (the error message is displayed).
14. Click Save to save the data input, then click Finish.

The Add-on Builder displays a summary of the new data input, along with suggested next steps.

Add authentication to a data input

When you need to use authentication for a REST input, add a Global Setup input field to your data input parameters form:

The Global account settings option is automatically selected on your setup page.

When users first open your add-on, they set up their accounts on the add-on's Configuration page:

Then, users configure new inputs on the add-on's Inputs page, which displays the data input parameters you configured:

Use checkpoints for incremental data collection

When you create your REST API data input, use checkpoints to mark where data has already been read and indexed so that you can resume reading data the next time it is collected. The checkpoint field is typically a timestamp, but you can specify any field. You do not need to use checkpoints with a REST API that returns a point-in-time value, such as metrics.

When using checkpoints, the REST API should do the following:

• Allow you to sort fields by the one you want to use for checkpoints to allow that field to indicate where new data begins.
• Query data by the checkpoint field, starting with the initial checkpoint value.
• Allow timestamps in increments at least as small as the collection interval.

Pass values from data input parameters

You can pass parameter values to a variable in a REST URL, a REST header, a REST body, and in a shell command. Following the Mako syntax, use the format ${varname} for the variable. Then, define a modular input variable for varname.

Pass values from setup parameters

When referring to setup parameters, include a namespace string before the parameter name on order to get the value from the parameter. The following shows an example, where param_name is the parameter name:

${__settings__.additional_parameters.param_name}

For example, let's say you have a text input on your setup page that prompts the user for an API token, with the internal name "api_token". To pass the value of the API token to the REST call, use the following format:

api-key=${__settings__.additional_parameters.api_token}

We recommend you do not set account, proxy, or logging fields directly. You can access the global account values as follows:

${global_account.username}
{{global_account.username}}

${global_account.password}
{{global_account.password}}

JSON path formats

Use the following guidelines to form JSON paths when creating REST inputs.

Operator	Description
$	The root element to query.
*	Wildcard. Available anywhere a name or number is required.
..	Deep scan. Available anywhere a name is required.
.<name>	Dot-notated child.
['<name>']	Bracket-notated child.
[<number>]	Array index or indexes.

Example

Here is an example JSON response:

{
    "count": 251,
    "results": [
        {
            "isocode": "MZ",
            "country": "Mozambique"
        },
        {
            "isocode": "PG",
            "country": "Papua New Guinea"
        }
    ]
}

The following table shows different ways of accessing data in the JSON response.

JSON path	Returned events
$.count	251
$.results[*] or $.results	{"isocode": "MZ", "country": "Mozambique"} {"isocode": "PG","country": "Papua New Guinea"}
$.results[0]	{"isocode": "MZ", "country": "Mozambique"}
$.results[-1]	{"isocode": "PG","country": "Papua New Guinea"}
$.results[0].country	Mozambique
$['results'][0]['isocode']	MZ
$.results[0].*	MZ Mozambique
$..isocode	MZ PG

When the JSON path matches a field in the payload, the extracted value is highlighted in the test output.

Normalize timestamp formats

When you set a checkpoint, you only need to specify timestamp formats if the format of the request timestamp is different from the response timestamp. in this case, use the following format for timestamps:

%Y-%m-%dT%H:%M:%S.%f+08:00

Additional notes:

• "0" left padding is added to months, days, hours, minutes, and seconds. To remove left padding, use "-". For example: %-m.
• "0" right padding is added to milliseconds.
• Add-on Builder supports the time format as described in the Python function time_str2str for transforming a time string to a different format.
• The %Nf format is supported for to_format only.

Learn more

For more information, see the following Splunk Enterprise documentation:

• How timestamp assignment works in the Getting Data In manual
• Source type naming conventions in the Splunk Add-ons manual
• Create custom data inputs for Splunk Cloud Platform or Splunk Enterprise on the Splunk Developer Portal
• How to create modular inputs using Splunk SDK for Python on the Splunk Developer Portal
• Create a setup page for a Splunk app on the Splunk Developer Portal