Automate indexed field extractions with HTTP Event Collector
When indexes data, it parses the data stream into a series of events. As part of this process, it adds a number of fields to the event data. These fields include default fields that it adds automatically and any custom fields that you specify. The process of adding fields to events is known as field extraction. There are two types of field extraction: search-time field extraction and indexed field extraction. Indexed fields are incorporated into the index at index time and become part of the event data.
Indexed field extraction doesn't work with data sent to the services/collector/raw endpoint. For more information, see services/collector/raw in the Splunk Enterprise REST API Reference Manual.
Form HEC requests to trigger indexed field extractions
You can trigger indexed extractions of JavaScript Object Notation (JSON) fields in two ways: as part of the main event
data or as separate from the event
data but still associated with the event.
Use nested JSON inside the event property
Assign the event
property, which is at the top level of the JSON being sent to HEC, to a JSON object that contains the custom fields you want to index as key-value pairs. For example, the following event
property, from within an HTTP request sent to HEC, specifies two custom fields: club
and wins
.
"event": {"club":"glee", "wins":["regionals","nationals"]}
In this example, the wins
property is a multi-value JSON array. The wins
field is assigned both the values in the array.
At the same level as the event
property, you must also include a sourcetype
property, and set it to a source type that has indexed extraction enabled. You can use any source type that has the INDEXED_EXTRACTIONS
setting configured to JSON
in the props.conf configuration file, including built-in source types such as _json
. See the following example:
"sourcetype":"_json"
Following is an example cURL command that sends an event to HEC on a instance. In this case, the event data contains two custom fields that are extracted at index time:
# Extracting JSON fields curl https://mysplunkserver.example.com:8088/services/collector -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" -d '{"sourcetype": "_json", "event": {"club":"glee", "wins":["regionals","nationals"]}}'
Add a fields property at the top JSON level
Include the fields
property at the top level of the JSON that you send to HEC, which is at the same level as the event
property. Adding this property specifies explicit custom fields that are separate from the main event
data. This method is useful if you don't want to include the custom fields with the event data, but you want to annotate the data with some extra information, such as where it came from. Using this method is also typically faster than the nested JSON method.
Be aware that you must send HEC requests containing the fields
property to the /collector/event endpoint. Otherwise, they aren't indexed.
Assign the fields
property to a JSON object that contains the custom fields to be indexed as key-value pairs. For example, the following fields
property, from within an HTTP request sent to the Splunk platform instance, specifies two custom fields:club
and wins
.
"fields": {"club":"glee", "wins":["regionals","nationals"]}
In this example, the wins
property is set to a multi-value JSON array. The wins
field is assigned both values in the array.
At the same level as the event
and fields
properties, you must also include a sourcetype
property and set it to a source type that has indexed extractions enabled. You can use any source type that has the INDEXED_EXTRACTIONS
setting configured to JSON
in the props.conf file, including built-in source types such as _json
. See the following example:
"sourcetype":"_json"
Following is an example cURL command that sends an event to HEC on a instance. In this case, the event data contains two custom fields that will be extracted at index time:
curl http://mysplunkserver.example.com:8088/services/collector/event -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" -d '{"event": {"my_event": "Hello, McKinley High!"}, "sourcetype": "_json", "fields": {"club":"glee", "wins":["regionals","nationals"]}}'
Only strings can be used as field values.
Search for index-extracted fields
After the data is indexed, you can search for this event using a double-colon ( :: ) indexed extraction notation, as shown here:
sourcetype=_json club::glee
For more information about using extracted fields to retrieve events, see Use fields to retrieve events in the Splunk Enterprise Search Manual.
Format events for HTTP Event Collector | Send metrics to a metrics index |
This documentation applies to the following versions of Splunk Cloud Platform™: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406
Feedback submitted, thanks!