Create field filters using Splunk Web

Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, tstats, typeahead, and walklex), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.

How to create field filters using Splunk Web

Field filters control visibility of data within events by redacting or obfuscating confidential information such as personal identifiable information (PII) and protected health information (PHI) data when users who are not exempt from the field filters run searches. This type of filter can remove fields from events returned by search results, or replace values of fields when those fields appear in the results. If you don't want unauthorized people to know that a field even exists, you can redact the field by deleting it from the search results. Alternatively, you can redact the field value by replacing it with a custom string, or you can replace the field value with a SHA-256 or SHA-512 (SHA-2 family) hash value that obfuscates the sensitive data. After you configure field filters, the settings then affect the results of searches run by all nonexempt users by filtering out specific indexed fields, including _raw fields.

Develop a plan for your field filters before you create them. See Plan for field filters in your organization.

Prerequisites

To create, view, edit, and delete field filters, you must be a member of the admin or sc_admin role. Custom roles aren't supported for managing field filters.

Steps

This section describes how to use Splunk Web to create a new field filter by assigning a value to a field. To create and manage field filters, follow these steps:

1. Select Settings and then Field filters from Users And Authentication.
2. If this is the first time you are creating a field filter, you are prompted to acknowledge that there are certain restricted commands that users can't use by default on searches across indexes that have field filters. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.
3. Select New field filter.

Now you have a choice of creating a field filter for an indexed or _raw field. The following steps describe how to:

• Create a field filter with limits for an indexed field.
• Create a field filter with limits for a _raw field using the sed (s) command.
• Create a field filter with limits for a _raw field using the sed (y) command.

If your indexed fields are generated from _raw, keep confidential information in indexed and _raw fields safe by creating field filters for your indexed fields and a field filter for the _raw field from which the indexed fields are extracted. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.

Create a field filter with limits for an indexed field

Create field filters for indexed fields that contain sensitive information that you want to redact or obfuscate.

1. In the Field filter and limit page, enter a name and optional description for your new field filter.
2. Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
3. In the Limit type field, select a specific host, source, or source type, which is used to limit the field filter and optimize performance. Only one limit type is supported per field filter.
4. If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
5. In the Field type field, select Indexed to create a field filter for an indexed field.
6. In the Indexed field field, enter the name of the field that you want to restrict with the field filter. For example, if you want to set a field filter on a sensitive field called account_number, the indexed field is account_number.
   You can't use wildcard characters to specify multiple fields in the fieldname because field filters treat wildcards like any other character.
7. Select Next.
8. In the Replacement method page, select the method you want to use with this field filter to protect sensitive data. The methods are described as follows:

   Method	Description
   Delete	At search time, removes the specified indexed field and its value from results of searches to which this filter is applied. The field is still visible to exempt roles because the field is still present in the index. Use this option if your sensitive data needs to be redacted with the highest form of privacy for legal or regulatory reasons.
   String	Replaces the value of the field with the specified string wherever the field appears in results of searches to which this filter is applied. Use this option if your sensitive data needs to be redacted with a high level of privacy while still indicating the existence of the field.
   SHA512	Computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-512 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference.
   SHA256	Computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-256 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference.

9. If you selected String as the replacement method, enter the string that you want to be used to replace the value of the field that this field filter will apply to.
10. Select Next.
11. In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter applies to all roles when they run searches to which this field filter applies.
12. Select Next.
13. Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.

    Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.

After you've created a field filter for your indexed field, make sure that there is also a field filter protecting the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.

Create a field filter with limits for a _raw field using the sed (s) command

The sed expression in a _raw field filter acts on searches to which the filter is applied. The sed expression replaces strings in search results that are matched by a regular expression (s).

If you create field filters for indexed fields that are generated from _raw, you should also create a field filter for the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.

The syntax for using the sed (s) command to replace strings in search results that are matched by a regular expression (s) is s/<regex>/<replacement>/<modifiers>. See About Splunk regular expressions ​in the Splunk Cloud Platform Knowledge Manager Manual.

When you create a _raw filter, make sure that the regular expression for that field filter doesn't conflict with an existing field filter.

Follow these steps to create a field filter for a _raw field using the sed (s) command to replace strings in search results that are matched by a regular expression.

1. In the Field filter and limit page, enter a name and optional description for your new field filter.
2. Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
3. In the Limit type field, optimize performance by limiting the field filter to a specific host, source, or source type. Only one limit type is supported per field filter.
4. If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
5. In the Field type field, select _raw to create a field filter for a _raw field.
6. Select Next.
7. In the Replacement method page, select s for the Sed command type that is used to specify a sed expression. The sed expression acts on searches to which this filter is applied and replaces strings in search results that are matched by a regular expression (s).
8. In the Regex value match field, enter the regular expression that you want this field filter to apply to searches.
9. In the Replacement field, enter a string that replaces the regular expression match. Use \<n> for back references, where <n> is a single digit.
10. In the Command modifier field, select Global or Number. Other sed modifiers (also known as flags) for the (s) command are not supported. The modifiers are described as follows:

    Modifier	Description
    Global	Globally replace all matches.
    Number	A number that specifies the position of the matching text to replace in the regular expression.

11. (Optional) If you selected Number, enter a number in the Modifier number field to specify the position of the matching text that should be replaced in the regular expression. For example, if you enter 2, the 2nd matching text in the string will be replaced when the field filter is applied to searches.
12. Select Next.
13. In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter will apply to all roles when they run searches to which this field filter applies.
14. Select Next.
15. Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.

    Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.

Create a field filter with limits for a _raw field using the sed (y) command

The sed expression in a _raw field filter acts on searches to which the filter is applied. The sed expression transliterates characters found in search results with corresponding characters provided by the sed expression (y).

If you create field filters for indexed fields that are generated from _raw, you should also create a field filter for the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.

The (y) command syntax transliterates the source characters in search results with corresponding destination characters that you provide in the expression. For example, /: y/abc/def/ replaces a with d, b with e, and c with f. This expression changes the string aaabbc to dddeef. See About Splunk regular expressions ​in the Splunk Cloud Platform Knowledge Manager Manual.

When you create a _raw filter, make sure that the regular expression for that field filter doesn't conflict with an existing field filter.

Follow these steps to create a field filter for a _raw field using the sed (y) command to transliterate characters found in search results with corresponding characters provided by the sed expression (y).

1. In the Field filter and limit page, enter a name and optional description for your new field filter.
2. Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
3. In the Limit type field, optimize performance by limiting the field filter to a specific host, source, or source type. Only one limit type is supported per field filter.
4. If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
5. In the Field type field, select _raw to create a field filter for a _raw field.
6. Select Next.
7. In the Replacement method page, select y for the Sed command type that is used to specify a sed expression. The sed expression acts on searches to which this filter is applied and transliterates characters found in search results with corresponding characters provided by the sed expression (y).
8. In the Source characters field, enter the source characters. Make sure that the Source characters and Replacement characters fields contain the same number of characters.
9. In the Replacement characters field, enter the characters that you want to replace in the sed expression.
10. Select Next.
11. In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter will apply to all roles when they run searches to which this field filter applies.
12. Select Next.
13. Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.

    Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.

Examples

The following examples show how you can use field filters to perform operations on fields in searches run by nonexempt roles.

1. Redact a field

At your organization, the indexed field patient_name is sensitive for security reasons. You have a role named managers, and you want people who belong to the managers role to be able to access the patient_name field in their search results. But, you don't want users with other roles to be able to see the patient_name field.

If you want to redact the field from the results of searches for everyone but people holding the managers role, create an indexed field filter for the patient_name field with the replacement method set to Delete and exempt the managers role. This removes the patient_name field from the results of searches, so people who aren't managers won't even know the field exists. The managers role that you exempted can still see the field in searches to which the field filter applies.

Removing a field and its value is the most secure method of protection because someone with malicious intentions can't deduce anything about the data if there is no trace of it in the search results.

2. Redact a field value

If you want to hide your field values, but not remove them from the search results entirely, you can set your field filter to replace the field value with a custom string. For example, if you want all users except for people who hold the staff role to see search results such as patient_name = XXXX, set a field filter to a custom string and exempt the staff role from the field filter.

Now when anyone who doesn't hold the staff role searches on events that include the patient_name field, the string XXX will appear in the results in place of the actual confidential information about the patient. Users who hold the staff role can still see the confidential data as usual.

3. Obfuscate a field value

You can also set your field filter to replace the field value with a hash value. For example, if you want all users to see a hash value instead of a patient's account number in search results, create a field filter for the patient_name field and set the replacement method to SHA256.

Now when someone searches on events that include the patient_account field, a hash value appears in the results in place of the actual confidential information about the patient.

4. Configure an indexed field and corresponding _raw field

To protect sensitive data in both indexed and _raw fields, create a field filter for an indexed field and a _raw field filter for the undefined field that is used to generate the indexed field. Follow these steps when creating a new field filter for the indexed field.

1. Specify the name of this field filter, for example, auth_filter.
2. Add a description for the field filter, such as Provide a field filter on authentication logs.
3. Specify the name of the target index, such as web_access_index.
4. Specify the limit type, such as source type.
5. Specify the name of source type limit, for example, corp_auth_log.
6. Select the indexed field type.
7. Specify the name of the indexed field, such as password.
8. Specify the action you want the field filter to perform, such as replace the field value with a string such as Redacted.
9. If a highly privileged role is exempt from this field filter, such as the "exempt" role, add the role to the list of roles without the filter.
10. The field filter summary page looks like this:

Create a field filter on the _raw field, so that _raw is treated as a simple indexed field and its value is redacted or hidden. Follow these steps to create a _raw field filter that corresponds to the indexed field filter you just created.

1. Specify the name of the field filter, for example, auth_raw_filter.
2. Add a description for the field filter, such as Provide a _raw field filter on authentication logs.
3. Specify the name of the target index, such as web_access_index.
4. Specify the limit type, such as source type.
5. Specify the name of the source type limit, for example, corp_auth_log.
6. Select the _raw field type. You don't need to specify the name of the _raw field because it is always called _raw.
7. Since a _raw field filter has an enhanced action that uses the Sed command to replace a partial string of the _raw field, use the s or y option. For example, use the s option to replace password=<my_pass_word> with password=REDACTED,
8. The field filter summary page looks like this:

Now the password field is protected with both an indexed field filter and a _raw field filter.

5. Create multiple field filters for the same field with different role exemptions

Say you have two roles called user and analyst that need different access to social security number data. For users, the value of the SSN field must be redacted. For analysts who need to run statistics on social security numbers, the SSN field must be hashed. To set up field filters and roles according to these requirements, you might do the following:

1. Create a field filter called ssn_redact that redacts the value of the indexed SSN field by replacing it with the string REDACTED.
2. Exempt the analyst role from the ssn_redact field filter, so analysts can still see the data.
3. Create a field filter called ssn_hash that sets the value of the indexed SSN field to SHA-256.
4. Exempt users from the ssn_hash field filter, so users can still see the redacted SSN field.

Now the SSN field is protected in the right way for each role and each role executes only one field filter.

6. Test a field filter that replaces a _raw field with a static value

This example shows you how to create and test a field filter that replaces the value for a _raw field called healthId with a Universally Unique Identifier (UUID) of 00000000-0000-0000-0000-000000000000.

1. In Splunk Web, select Settings and then Field filters from Users And Authentication.
2. Select New field filter.
3. In the Field filter and limit page, enter the name healthId FF for your new field filter and select Next.
4. In the Replacement method page, select s for the Sed command type
5. In the Regex value match field, enter the regular expression "healthId": "([^"]+)" that you want this field filter to apply to searches.
6. In the Replacement field, enter the string "healthId":"00000000-0000-0000-0000-000000000000" that replaces the regular expression match.
7. In the Command modifier field, select Global.
8. Select Next, and Next, and then Create.

Then, run the following search to test that the field filter is correctly replacing the value of the healthId field:

index="main" | head 1 | table healthId _raw

The search results look like this:

healthId	_raw
00000000- 0000-0000- 0000- 000000000000	{"user": "user-842", "eventTime": "2023-09-06T14:52:20", "healthId": "00000000-0000-0000-0000-000000000000", "field2": "7d92ee29-fd9b-4db3-bf5c-38dbbf821921", "field3": "2c85eb0d-4a1d-4c52-9a59-226b93d552e6"}

The replacement value for healthId is displayed in the healthId field, and the _raw field shows that healthId has been replaced with 00000000-0000-0000-0000-000000000000 in all places where it appears in the results.

7. Create two field filters for an indexed and _raw field

To make sure that both the indexed and source data for a field are protected, say you create the following two field filters, which mask the host= and user= fields for all users:

• A field filter for an indexed field called host that is set to replace the field and its value with the string unknown host.
• A field filter for a _raw field called user that uses the sed (s) command to replace the string that is matched by the regular expression s/user=[^ ]+/REMOVED-USER/.

Then, the host name and username are redacted when a user runs the following search:

index="main"

The search results might look like this:

Time	Event
2023-01-31 15:01:58.679	[timestamp=01-31-2023 15:01:58.679, REMOVED-USER action=search, info=granted REST: /search/ast] host = unknown host
2023-01-31 15:01:58.678	[timestamp=01-31-2023 15:01:58.678, REMOVED-USER action=search, info=granted REST: /search/timeparser] host = unknown host
2023-01-31 15:01:58.677	[timestamp=01-31-2023 15:01:58.677, REMOVED-USER action=edit_win_eventlogs, info=denied ] host = unknown host
2023-01-31 15:01:58.676	[timestamp=01-31-2023 15:01:58.676, REMOVED-USER action=edit_win_regmon, info=denied ] host = unknown host

8. Hash a field value and get the count

An important consideration to performing hash functions on field values using SHA encryption is that field filters preserve the uniqueness of the field. That means that even if you encrypt identifying information, such as user ID numbers, you can still get a count of the number of user IDs without exposing the value. For example, say you create a field filter that sets the value of the host field to a SHA-256 hash.

Then, you run the following search:

index=main | stats count by host

Your search results might look like this:

host	count
C1fe560f903b48c9681269132a783a9333414bbf7ac26bf41cebbaf79b6d	8223

Notice that even though there is only one hash value, the count of user IDs is 8223.

9. Redact PII and PHI data in search results

You can use field filters to redact sensitive fields such as Personal identifiable information (PII) and Protected Health Information (PHI) from searches of all of your event data. For example, perhaps your search results include the name and phone number for a patient named Alex Martin, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search patient_name=AlexMartin phone=+1-555-555-0199', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

If you want to replace patient_name=AlexMartin and phone=+1-555-555-0199 with custom strings, you would create two _raw field filters for users who don't have authorization to see that sensitive information. One _raw field filter might be called staffName and is set to s/patient_name=[^ ']+/REMOVED-NAME/. The other _raw field filter might be called staffPhone and is set to s/phone=[^ ']+/REMOVED-PHONE/.

Then, when a person who holds a role that is affected by these field filters run the following search:

index=main

The person now sees only search results for the patient with REMOVED_NAME REMOVED_PHONE, instead of patient_name=AlexMartin phone=+1-555-555-0199, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search REMOVED-NAME REMOVED-PHONE', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

The confidential data is no longer visible to all nonexempt roles that are affected by the field filters because the patient name and phone number have been replaced with custom strings.

See also

Protect PII, PHI, and other sensitive data with field filters