Ingest Processor is currently released as a preview only and is not officially supported. See Splunk General Terms for more information. For any questions on this preview, please reach out to ingestprocessor@splunk.com.

Extract JSON fields from data using Ingest Processor

You can create a pipeline that extracts JSON fields from data. Field extractionlets you capture information from your data in a more visible way and configure further data processing based on those fields.

If you're sending data to Splunk Enterprise or Splunk Cloud Platform, be aware that some fields are extracted automatically during indexing. Additionally, be aware that indexing extracted fields can have an impact on indexing performance and search times. Consider the following best practices when configuring field extractions in your pipeline: Extract fields only as necessary. When possible, extract fields at search time instead. Avoid duplicating your data and increasing the size of your events. After extracting a value into a field, either remove the original value or drop the extracted field after you have finished using it to support a data processing action. For more information, see When Splunk software extracts fields in the Splunk Cloud Platform Knowledge Manager Manual.

Pipelines don't extract any fields by default. If a pipeline receives data from a data source that doesn't extract data values into fields, such as a universal forwarder without any add-ons, then the pipeline stores each event as a text string in a field named _raw.

Steps

To create a field extraction pipeline, use the Extract JSON fields from action in the pipeline editor to identify the field names you want to extract. Complete these steps to create a pipeline that receives data associated with a specific source type, optionally processes it, and sends that data to a destination.

1. From the home page on Splunk Cloud Platform, navigate to the Pipelines page and then select New pipeline, then Ingest Processor pipeline.
2. On the Get started page, select Blank pipeline, then Next.
3. On the Define your pipeline's partition page, do the following:
   1. Select how you want to partition your incoming data that you want to send to your pipeline. You can partition by source type, source, and host.
   2. Enter the conditions for your partition, including the operator and the value. Your pipeline will receive and process the incoming data that meets these conditions.
   3. Select Next to confirm the pipeline partition.
4. On the Add sample data page, do the following:
   1. Enter or upload sample data for generating previews that show how your pipeline processes data. The sample data must contain accurate examples of the values that you want to extract into JSON fields.
   2. Select Next to confirm the sample data that you want to use for your pipeline.
5. On the Select destination dataset page, select the name of the destination that you want to send data to. Then, do the following:
   1. If you selected a Splunk platform S2S or Splunk platform HEC destination, select Next.
   2. If you selected another type of destination, select Done and skip the next step.
6. (Optional) If you're sending data to a Splunk platform deployment, you can specify a target index:
   1. In the Index name field, select the name of the index that you want to send your data to.
   2. (Optional) In some cases, incoming data already specifies a target index. If you want your Index name selection to override previous target index settings, then select the Overwrite previously specified target index check box.
   3. Select Done.

   If you're sending data to a Splunk platform deployment, be aware that the destination index is determined by a precedence order of configurations.

7. (Optional) To generate a preview of how your pipeline processes data based on the sample data that you provided, select the Preview Pipeline icon (()). Use the preview results to validate your pipeline configuration.
8. Select the plus icon () in the Actions section, then select Extract JSON fields from.
9. Select the field that you want to extract from your data, then select Apply.
10. To extract multiple fields from your data, repeat steps 8-9.
11. To save your pipeline, do the following:
    1. Select Save pipeline.
    2. In the Name field, enter a name for your pipeline.
    3. (Optional) In the Description field, enter a description for your pipeline.
    4. Select Save. The pipeline is now listed on the Pipelines page, and you can now apply it, as needed.
12. To apply this pipeline, do the following:
    1. Navigate to the Pipelines page.
    2. In the row that lists your pipeline, select the Actions icon (()), and then select Apply.
    3. Select the pipelines that you want to apply, and then select Save.
    4. It can take a few minutes to finish applying your pipeline. During this time, all applied pipelines enter the Pending status.
    5. (Optional) To confirm that the Ingest Processor service has finished applying your pipeline, navigate to the Ingest Processor page and check if all affected Ingest Processors have returned to the Healthy status.
    6. Your applied pipelines can now process and route data as specified in the pipeline configuration. To generate a preview of how your pipeline processes data based on the sample data that you provided, select the Preview Pipeline icon (()).