Use field filters in searches

Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, and tstats), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.

Examples of searches with field filters

1. Hash a field value and get the count

An important consideration to performing hash functions on field values using SHA encryption is that field filters preserve the uniqueness of the field. That means that even if you encrypt identifying information, such as user ID numbers, you can still get a count of the number of user IDs without exposing the value. For example, say you create a field filter that sets the value of the host field to a SHA-256 hash.

Then, you run the following search:

index=main | stats count by host

Your search results might look like this:

host	count
C1fe560f903b48c9681269132a783a9333414bbf7ac26bf41cebbaf79b6d	8223

Notice that even though there is only one hash value, the count of user IDs is 8223.

2. Redact PII and PHI data in search results

You can use field filters to redact sensitive fields such as Personal identifiable information (PII) and Protected Health Information (PHI) from searches of all of your event data. For example, perhaps your search results include the name and phone number for a patient named Alex Martin, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search patient_name=AlexMartin phone=+1-555-555-0199', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

If you want to replace patient_name=AlexMartin and phone=+1-555-555-0199 with custom strings, you would create two _raw field filters for users who don't have authorization to see that sensitive information. One _raw field filter might be called staffName and is set to s/patient_name=[^ ']+/REMOVED-NAME/. The other _raw field filter might be called staffPhone and is set to s/phone=[^ ']+/REMOVED-PHONE/.

Then, when a person who holds a role that is affected by these field filters run the following search:

index=main

The person now sees only search results for the patient with REMOVED_NAME REMOVED_PHONE, instead of patient_name=AlexMartin phone=+1-555-555-0199, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search REMOVED-NAME REMOVED-PHONE', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

The confidential data is no longer visible to all nonexempt roles that are affected by the field filters because the patient name and phone number have been replaced with custom strings.

3. Search for a redacted field

Searching for an index-time field might not return any results because the value of the filtered field is redacted at search time. For example, say you have an index-time field called password and you run the following search:

index=web_access_index sourcetype::corp_auth_log password=REDACTED

The search doesn't return any events because password=REDACTED is not indexed.

To search for the redacted field, run the following search:

index=web_access_index sourcetype::corp_auth_log | table _time, password, _raw | where password=REDACTED