Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Automate indexed field extractions with HTTP Event Collector

When Splunk software indexes data, it parses the data stream into a series of events. As part of this process, it adds a number of fields to the event data. These fields include default fields that it adds automatically and any custom fields that you specify. The process of adding fields to events is known as field extraction. There are two types of field extraction, search-time field extraction and indexed field extraction. Indexed fields are incorporated into the index at index time and become part of the event data.

The indexed field extractions feature in HTTP Event Collector is available in Splunk Enterprise 6.5.0 and later, Splunk Light 6.5.0 and later, and the current releases of both Splunk Cloud and Splunk Light Cloud.

Previously, setting up custom fields created at index time required significant configuration steps, as described in Create custom fields at index time, that involve editing the props.conf, transforms.conf, and fields.conf files to add regular expression extractions. Now, you can use HTTP Event Collector to automate this process, using built-in support for indexed field extractions.

Indexed field extraction does not work with data sent to the REST endpoint.

Form HEC requests to trigger indexed field extractions

You can trigger indexed extractions of JavaScript Object Notation (JSON) fields in two ways--as part of the main event data or separate from the event data but still associated with the event.

Use nested JSON inside the "event" property

Assign the event property (at the top level of the JSON being sent to HEC) to a JSON object that contains the custom fields to be indexed, as key-value pairs. For example, the following "event" property, from within an HTTP request sent to the Splunk server, specifies two custom fields—"club" and "wins":

"event": {"club":"glee", "wins":["regionals","nationals"]}

In this example, the wins property has been set to a multi-value JSON array. The wins field will be assigned both the values in the array.

At the same level as the event property, you must also include a sourcetype property, and set it to a sourcetype that has indexed extraction enabled. You can use any sourcetype that has INDEXED_EXTRACTIONS set to JSON in the props.conf file, including built-in sourcetypes such as _json. For example:

"sourcetype":"_json"

Following is an example cURL command that sends an event to HEC on a Splunk server. In this case, the event data contains two custom fields that will be extracted at index time:

# Extracting JSON fields
curl -k https://mysplunkserver.example.com:8088/services/collector -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" -d '{"sourcetype": "_json", "event": {"club":"glee", "wins":["regionals","nationals"]}}'

Add a "fields" property at the top JSON level

Include the fields property at the top level of the JSON being sent to HEC—that is, at the same level as the event property. This specifies explicit custom fields that are separate from the main event data. This method is useful if you don't want to include the custom fields with the event data, but you want to be able to annotate the data with some extra information, such as where it came from. Using this method is also typically faster than the nested JSON method.

Be aware that you must send HEC requests containing the fields property to the /collector/event endpoint. Otherwise, they will not be indexed.

Assign the fields property to a JSON object that contains the custom fields to be indexed, as key-value pairs. For example, the following fields property, from within an HTTP request sent to the Splunk server, specifies two custom fields—club and wins:

"fields": {"club":"glee", "wins":["regionals","nationals"]}

Iin this example, the wins property has been set to a multi-value JSON array. The wins field will be assigned both the values in the array.

At the same level as the event and fields properties, you must also include a sourcetype property, and set it to a sourcetype that has indexed extractions enabled. You can use any sourcetype that has INDEXED_EXTRACTIONS set to JSON in the props.conf file, including built-in sourcetypes such as _json. For example:

"sourcetype":"_json"

Following is an example cURL command that sends an event to HEC on a Splunk server. In this case, the event data contains two custom fields that will be extracted at index time:

# Explicit JSON fields
curl -k https://mysplunkserver.example.com:8088/services/collector/event -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" -d '{"event": "Hello, McKinley High!", "sourcetype": "_json", "fields": {"club":"glee", "wins":["regionals","nationals"]}}'

Only strings can be used as field values.

Search for index-extracted fields

After the data is indexed, you can search for this event using indexed extraction ("double-colon") notation, as shown here:

sourcetype=_json club::glee

For more information about using extracted fields to retrieve events, see Use fields to retrieve events in the Splunk Enterprise Search Manual.

PREVIOUS
Format events for HTTP Event Collector
  NEXT
Send metrics to a metrics index

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.2.0, 7.2.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters