Hash fields using Ingest Processor

Create a pipeline that hashes specific fields in your data. When you hash a field, the Ingest Processor uses the selected hashing algorithm to compute a hash value or "digest" based on the original data values from that field. You can hash fields in order to obfuscate some of the data and prevent it from being directly human-readable.

Be aware that hashing alone might not be sufficient for anonymizing sensitive data or meeting compliance guidelines. Refer to your organization's compliance policies for more information.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Ingest Processor pipeline. Otherwise, all unprocessed data is dropped.

Supported hashing algorithms

Ingest Processor supports the following hashing algorithms:

Hashing algorithm	Value	Example SPL2
MD5	128-bit hash value	$pipeline = | from $source | eval <hashed_field> = md5(<original_field>) | into destination
SHA-1	160-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha1(<original_field>) | into destination
SHA-256	256-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha256(<original_field>) | into destination
SHA-512	512-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha512(<original_field>) | into destination

Prerequisites

Before starting to create a pipeline, confirm that the destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed, then you must add that destination to your tenant. See Add or manage destinations for more information.

Steps

Perform the following steps to create a pipeline that hashes an event field:

1. Create a pipeline.
2. Configure hashing in your pipeline.
3. Preview, save, and apply your pipeline.

Create a pipeline

Complete these steps to create a basic pipeline that receives a specific subset of the incoming data and then sends that data to a destination.

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline, and then select Next.
3. On the Define your pipeline's partition page, do the following:
1. Select the plus () icon next to Partition, or select the option that matches how you would like to create your partition in the Suggestions section.
2. In the Field field, specify the event field that you want the partitioning condition to be based on.
3. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
4. In the Operator field, select an operator for the partitioning condition.
5. In the Value field, enter the value that your partition should filter by to create the subset.
6. Select Apply.

You can create more conditions for a partition in a pipeline by selecting the plus () icon.

7. Once you have defined your partition, select Next.
4. (Optional) On the Add sample data page, enter or upload sample data for generating previews that show how your pipeline processes data.

   The sample data must be in the same format as the actual data that you want to process. See Getting sample data for previewing data transformations for more information.

5. Select Next to confirm the sample data that you want to use for your pipeline.
6. (Optional) On the Select a metrics destination page, select the name of any metrics destinations that you want to send data to, and then select Next.
7. On the Select destination dataset page, select the name of the destination that you want to send data to, and then select Next.

   If you're sending data to a Splunk platform deployment, be aware that the destination index is determined by a precedence order of configurations.

8. (Optional) Specify a target index as a field on each event. Click Done when complete.

You now have a simple pipeline that receives a specific subset of the incoming data and sends that data to a destination. In the next section, you'll configure this pipeline to hash an event field.

Configure hashing in your pipeline

During the previous step, you created a basic pipeline that receives a specific subset of data and then sends that data to a destination. The next step is to configure the pipeline to hash fields in the received events.

Be aware that after you hash an event field, the original plain text might still remain in other parts of the event. To hide the plain text, you must remove the field, mask the data, or perform both actions, as needed.

1. Select the plus icon () in the Actions section, then select Compute hash of.
2. In the Compute hash of a field dialog box, do the following:
   1. In the Source field field, specify the field containing the plain text values that you want to compute into hash values.
   2. Select the hashing algorithm that you want to use to compute the hash values.
   3. In the Target field field, enter the name of an event field where you want to store the hash values. You can specify an existing event field or the name of a new field that you want to add to your events. If you want to overwrite the original plain text values in the specified Source field with the hash values, then enter the same field as the Source field setting.
   4. When you have completed your configurations, click Apply.
3. If the original plain text values still exist in other parts of the event, then configure additional processing actions to remove or mask those values.
   • For information about dropping fields from events, see fields command overview in the SPL2 Search Reference.
   • For information about masking data values, see Filter and mask data Ingest Processor.

You now have a pipeline that hashes a selected field. In the next section, you'll verify that this pipeline processes data in the way that you expect and save it to be applied to an Ingest Processor.

Preview, save, and apply your pipeline

1. (Optional) Select the Preview Pipeline icon () to generate a preview that shows what your data looks like when it passes through the pipeline.
2. To save your pipeline, do the following:
   1. Select Save pipeline.
   2. In the Name field, enter a name for your pipeline.
   3. (Optional) In the Description field, enter a description for your pipeline.
   4. Select Save.

   The pipeline is now listed on the Pipelines page, and you can apply it as needed.

3. To apply this pipeline, do the following:
   1. Navigate to the Pipelines page.
   2. In the row that lists your pipeline, select the Actions icon () and then select Apply.
   3. Select the pipeline that you want to apply, and then select Save.

   It can take a few minutes for the Ingest Processor service to finish applying your pipeline. During this time, the pipeline enters the Pending Apply status (). Once the operation is complete, the Pending Apply status icon () stops displaying beside the pipeline. Refresh your browser to check if the icon no longer displays.

The Ingest Processor can now hash the specified field in the events that it receives.