Extracting fields in events data

You can extract fields in a data pipeline in different ways.

• Extract Timestamp extracts a timestamp from body using pre-defined rules.
• Rex performs field extractions using named capturing groups in Java regular expressions. Rex also creates a top-level field in your data where the value of your field extraction is placed.
• Extract regex also performs field extractions using named capturing groups in Java regular expressions, but outputs your fields as a map.
• Extract key value explicitly extracts field and value pairs.

Extract timestamps in your data

The Data Stream Processor has several rules to extract timestamps from the body field of your data. For the full list of rules, see Extract Timestamp.

1. From the Data Pipelines Canvas view, click on the + icon and add the Extract Timestamp function to your pipeline.
2. In the Extract timestamp function, complete the following fields:

   Field	Description	Example
   field	The name of the field to put the timestamp value in.	timestamp
   Rule	The rules to try matching your timestamp to.	date_timestamp(), other_timestamp()

3. With your Extract Timestamp function highlighted, click Start Preview to verify that the expression is working as expected.

   If no timestamp is matched, the created timestamp field returns the current epoch time.

If there isn't a rule for your timestamp yet, you can also extract a timestamp using the date and time scalar functions. See Date and Time.

Extract fields to create top-level fields in your data

To extract fields from your data, use the Rex function to extract a field with a Java regular expression and add that field as a top-level field in your data.

1. From the Data Pipelines Canvas view, click on the + icon and add the Parse regex function to your pipeline.
2. In the Parse Regex function, complete the following fields:

   Field	Description	Example
   Field	The field that you want to extract information from.	body
   Pattern	The Java regular expression that defines the information to match and extract from the specified field. You can include named capturing groups, as shown in the example.	/(?<ASA>%ASA-[\d]-[\d]{6})/

3. With your Parse Regex function highlighted, click Start Preview to verify that the expression is working as expected.

Extract fields as maps

To extract fields from your data and output the fields as a map, use the Extract regex scalar function.

1. From the Data Pipelines Canvas view, click on the + icon and add the Eval function to your pipeline.
2. In the Eval function, type asa=extract_regex(cast(body, "string"), /(?<ASA>ASA-\d-\d{6})/i).
   This uses a Java regular expression named capturing group to extract the ASA number from your event body and output it in a new field called "ASA" in your events. Note that this is the same pattern as the above Rex example.
3. With your Eval function highlighted, click Start Preview to verify that the expression is working as expected.

Extract explicit key-value pairs

To extract explicit key-value pairs in your data, use the Extract key value scalar function.

1. From the Data Pipelines Canvas view, click on the + icon and add the Eval function to your pipeline.
2. In the Eval function, type extracted_body=extract_key_value(cast(body, "string"), "=", " ").
   This extracts key-value pairs in the body field and puts the extracted key-value pairs in a field called extracted_body.
3. With your Eval function highlighted, click Start Preview to verify that the expression is working as expected.