Hash fields using an Edge Processor

You can create a pipeline that hashes specific fields in your data. When you hash a field, the Edge Processor uses the selected hashing algorithm to compute a hash value or "digest" based on the original data values from that field. You can hash fields in order to obfuscate some of the data and prevent it from being directly human-readable.

Be aware that hashing alone might not be sufficient for anonymizing sensitive data or meeting compliance guidelines. Refer to your organization's compliance policies for more information.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Edge Processors. Otherwise, all unprocessed data is dropped.

Supported hashing algorithms

Edge Processors support the following hashing algorithms:

Hashing algorithm	Value	Example SPL2
MD5	128-bit hash value	$pipeline = | from $source | eval <hashed_field> = md5(<original_field>) | into destination
SHA-1	160-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha1(<original_field>) | into destination
SHA-256	256-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha256(<original_field>) | into destination
SHA-512	512-bit hash value	$pipeline = | from $source | eval <hashed_field>= sha512(<original_field>) | into destination

Prerequisites

Before starting to create a pipeline, confirm that the destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed, then you must add that destination to your tenant. See Add or manage destinations for more information.

Steps

Perform the following steps to create a pipeline that hashes an event field:

1. Create a pipeline.
2. Configure hashing in your pipeline.
3. Preview, save, and apply your pipeline.

Create a pipeline

Complete these steps to create a basic pipeline that receives a specific subset of the incoming data and then sends that data to a destination.

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline and then select Next.
3. On the Define your pipeline's partition page, specify a subset of the data received by the Edge Processor for this pipeline to process. To do this, you must define a partition by completing these steps:
   1. Select the plus icon () next to Partition, or select the option that matches how you would like to create your partition in the Suggestions section.
   2. In the Field field, specify the event field that you want the partitioning condition to be based on.
   3. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
   4. In the Operator field, select an operator for the partitioning condition.
   5. In the Value field, enter the value that your partition should filter by to create the subset.
   6. Select Apply.

   You can create more conditions for a partition in a pipeline by selecting the plus icon ().

   7. Once you have defined your partition, select Next.
4. (Optional) On the Add sample data page, enter or upload sample data for generating previews that show how your pipeline processes data.

   The sample data must be in the same format as the actual data that you want to process. See Getting sample data for previewing data transformations for more information.

5. Select Next to confirm the sample data.
6. On the Select destination dataset page, select the name of the destination that you want to send data to. Then, do the following:
• If you selected a Splunk platform S2S or Splunk platform HEC destination, select Next.
• If you selected another type of destination, select Done and skip the next step.
7. (Optional) If you're sending data to a Splunk platform deployment, then on the Select a target index page, do the following to specify a target index:
   1. In the Index name field, select the name of the index that you want to send your data to.
   2. (Optional) In some cases, incoming data already specifies a target index. If you want your Index name selection to override previous target index settings, then select the Overwrite previously specified target index check box.
   3. Select Done.

   Be aware that the destination index is determined by a precedence order of configurations. See How does an Edge Processor know which index to send data to? for more information.

Configure hashing in your pipeline

During the previous step, you created a basic pipeline that receives a specific subset of data and then sends that data to a destination. The next step is to configure the pipeline to hash fields in the received events.

Be aware that after you hash an event field, the original plain text might still remain in other parts of the event. To hide the plain text, you must remove the field, mask the data, or perform both actions, as needed.

1. Select the plus icon () in the Actions section, then select Compute hash of.
2. In the Compute hash of a field dialog box, do the following:
   1. In the Source field field, specify the field containing the plain text values that you want to compute into hash values.
   2. Select the hashing algorithm that you want to use to compute the hash values.
   3. In the Target field field, enter the name of an event field where you want to store the hash values. You can specify an existing event field or the name of a new field that you want to add to your events. If you want to overwrite the original plain text values in the specified Source field with the hash values, then enter the same field as the Source field setting.
   4. When you have completed your configurations, click Apply.
3. If the original plain text values still exist in other parts of the event, then configure additional processing actions to remove or mask those values.
   • For information about dropping fields from events, see fields command overview in the SPL2 Search Reference.
   • For information about masking data values, see Filter and mask data using an Edge Processor.

You now have a pipeline that hashes a selected field. In the next section, you'll verify that this pipeline processes data in the way that you expect and save it to be applied to an Edge Processor.

Preview, save, and apply your pipeline

1. (Optional) Select the Preview Pipeline icon () to generate a preview that shows what your data looks like when it passes through the pipeline.
2. To save your pipeline, do the following:
   1. Select Save pipeline.
   2. In the Name field, enter a name for your pipeline.
   3. (Optional) In the Description field, enter a description for your pipeline.
   4. Select Save.

   If your pipeline is valid, the Edge Processor service prompts you to apply it to an Edge Processor.

3. To apply this pipeline to an Edge Processor, do the following:
   1. In the Apply pipeline prompt, select Yes, apply.
   2. Select the Edge Processors that you want to apply the pipeline to, and then select Save.

   You can only apply pipelines to Edge Processors that are in the Healthy status.

   It can take a few minutes for the Edge Processor service to finish applying your pipeline to an Edge Processor. During this time, the affected Edge Processors enter the Pending status. To confirm that the process completed successfully, do the following:

   • Navigate to the Edge Processors page. Then, verify that the Instance health column for the affected Edge Processors shows that all instances are back in the Healthy status.
   • Navigate to the Pipelines page. Then, verify that the Applied column for the pipeline contains a The pipeline is applied icon ().

The Edge Processor that you applied the pipeline to can now hash the specified field in the events that it receives.