Extract timestamps from event data using Ingest Processor

You can use a pipeline to extract timestamp fields and also convert those timestamps into specific formats. Extracting the Timestamp field lets you visualize events by time and convert timestamps into the appropriate format before sending it to a destination.

For example, if you have data that contains timestamps with multiple formats, you can convert the timestamp information from your data into a specific format directly from the pipeline editor. This is helpful when certain destinations require you to store timestamps using a particular format.

To convert a timestamp, you must write a SPL2 pipeline statement for extracting timestamps from your data and then convert your timestamp to normalize the data from those fields.

Steps

1. Navigate to the Pipelines page, then select New pipeline and then Ingest Processor pipeline.
2. On the Get started page, select Blank pipeline and then Next.
3. On the Define your pipeline's partition page, do the following:
1. Select how you want to partition your incoming data that you want to send to your pipeline. You can partition by source type, source, and host.
2. Enter the conditions for your partition, including the operator and the value. Your pipeline will receive and process the incoming data that meets these conditions.
3. Select 'Next to confirm the pipeline partition.
4. To specify a source type and some sample data for your pipeline, do the following:
1. On the Add sample data page, enter or upload sample data for generating previews that show how your pipeline processes data. The sample data must contain accurate examples of the values that you want to extract into fields. For example, the following sample events represent purchases made at a store at a particular time:

   E9FF471F36A91031FE5B6D6228674089, 72E0B04464AD6513F6A613AABB04E701, Credit Card, 7.7, 2023-01-13 04:41:00, 2023-01-13 04:45:00, -73.997292, 40.720982, 4532038713619608
   A5D125F5550BE7822FC6EE156E37733A, 08DB3F9FCF01530D6F7E70EB88C3AE5B, Credit Card,14, 2023-01-13 04:37:00, 2023-01-13 04:47:00, -73.966843,40.756741, 4539385381557252
   1E65B7E2D1297CF3B2CA87888C05FE43,F9ABCCCC4483152C248634ADE2435CF0, Game Card, 16.5, 2023-01-13 04:26:00, 2023-01-13 04:46:00, -73.956451, 40.771442

2. Select Next to confirm any sample data that you want to use for your pipeline.
5. On the Select a metrics destination page, select the name of the destination that you want to send metrics to.
6. (Optional) If you selected Splunk Metrics store as your metrics destination, specify the name of the target metrics index where you want to send your metrics.
7. On the Select a data destination page, select the name of the destination that you want to send logs to.
8. (Optional) If you selected a Splunk platform destination, you can configure index routing:
   1. Select one of the following options in the expanded destinations panel:

      Option	Description
      Default	The pipeline does not route events to a specific index. If the event metadata already specifies an index, then the event is sent to that index. Otherwise, the event is sent to the default index of the Splunk Cloud Platform deployment.
      Specify index for events with no index	The pipeline only routes events to your specified index if the event metadata did not already specify an index.
      Specify index for all events	The pipeline routes all events to your specified index.

   2. If you selected Specify index for events with no index or Specify index for all events, then from the Index name drop-down list, select the name of the index that you want to send your data to.
      If your desired index is not available in the drop-down list, then confirm that the index is configured to be available to the tenant and then refresh the connection between the tenant and the Splunk Cloud Platform deployment. For detailed instructions, see Make more indexes available to the tenant.

If you're sending data to a Splunk Cloud Platform deployment, be aware that the destination index is determined by a precedence order of configurations. See How does Ingest Processor know which index to send data to? for more information

9. Select Done to confirm the data destination.
   After you complete the on-screen instructions, the pipeline builder displays the SPL2 statement for your pipeline.
10. To generate a preview of how your pipeline processes data based on the sample data that you provided, select the Preview Pipeline icon ().
11. Select the plus icon () in the Actions section, then select Extract fields from _raw.
1. In the Regular expression field, specify one or more named capture groups using Regular Expression 2 (RE2) syntax. The name of the capture group determines the name of the extracted field, and the matched values determine the values of the extracted field. You can select named capture groups from the Insert from library list or enter named capture groups directly in the field.
   For example, to extract timestamps from the sample events described previously, select Timestamp from the Insert from library list. The resulting regular expression looks like this:

(?P<Timestamp_ISO8601>(?:\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[T ](?:2[0123]|[01]?[0-9]):?(?:[0-5][0-9])(?::?(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)

For more information about the supported regular expression syntax, see Regular expression syntax for Ingest Processor pipelines.

2. (Optional) By default, the regular expression matches are case sensitive. To make the matches case insensitive, uncheck the Match case check box.
3. Use the Events preview pane to validate your regular expression. The events in this pane are based on the last time that you generated a pipeline preview, and the pane highlights the values that match your regular expression for extraction.

A rex command is added to the SPL2 pipeline statement, and the new field appears in the Fields list.

4. (Optional) You can convert the timestamp to a different format before extraction. In the preview results panel, select the header of the Timestamp_ISO8601 field to make the Suggestions section appear. Select Convert _time from Timestamp_ISO8601.
5. Specify a Source timestamp format from the menu and select Apply.

The resulting Timestamp field stores the timestamp as the Unix format, but displays the timestamp to you in the format you specified. This ensures an accurate _time value even if you access the data from a different time zone. See the following pages for more information:

• How timestamp assignment works
• About searching with time

After converting timestamps from the _time field, Splunk best practice is to remove unnecessary timestamp fields from events after conversion. You can do this by adding the | fields - command to your SPL2 pipeline statement after your field extraction and conversion expressions. For example:

| fields -  Timestamp

12. To save your pipeline, do the following:
1. Select Save pipeline.
2. In the Name field, enter a name for your pipeline.
3. (Optional) In the Description field, enter a description for your pipeline.
4. Select Save.
13. To apply this pipeline, do the following:
1. Navigate to the Pipelines page.
2. In the row that lists your pipeline, select the Actions icon () and then select Apply.
3. Select the pipelines that you want to apply, and then select Save.

   You can only apply pipelines that are in the Healthy status.

It can take a few minutes for the Ingest Processor to finish applying your pipeline. During this time, all applied pipelines will have a Pending status. Once the operation is complete, the Pending Apply status icon () stops displaying beside the pipeline, and all affected pipelines transition from the Pending status back to the Healthy status. Refresh your browser to check if the Pending Apply status icon () no longer displays.