Use field filters in searches

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone.

If your organization uses downstream configurations, such as accelerated data models, Splunk Enterprise Security (ES) detections using those data models, and user-level search-time field extractions, make sure that you plan around the implications of field filters on those configurations before deploying field filters in your environment. See READ THIS: Downstream impact of field filters.

If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, and tstats), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.

Walklex and typeahead in searches on indexes with field filters

Splunk platform supports searches that use search commands, as well as searches that use the walklex and typeahead commands on indexes protected by field filters. However, the behavior of walklex and typeahead differs from other search commands in important ways because of the way these generating commands access data.

See walklex and typeahead in the Splunk Platform Search Reference Manual.

Delete is always the field filter replacement method in searches that use typeahead

Regardless of the replacement method that is configured for a field filter, searches with the typeahead command always use delete as the replacement method when field filters are applied to searches. Removing indexed fields and their values from search results ensures that any matching field-value pairs are strictly removed from the results instead of being hashed or replaced, which provides the highest form of protection for sensitive data.

Field filters for _raw fields aren't applied in searches with walklex and typeahead

The walklex and typeahead commands aren't able to access _raw fields because they operate on tsidx files or merged lexicon (merged_lexicon.lex) files instead of raw event data. As a result, field filters applied to _raw fields don't have any effect on the results of searches that use the walklex and typeahead commands. This means that, as expected, a field filter on a _raw field doesn't protect the _raw data in searches that use walklex and typeahead.

Field filter limit type restrictions don't apply to searches with walklex and typeahead

Performance optimization for field filters that is based on limit types relies on fields such as source type, source, and host. Since search results that include walklex and typeahead might not include these fields, Splunk software doesn't apply limit type restrictions configured on field filters in searches with these two commands. This means that even if you create a field filter with a limit type for a host called vendor_sales, results from other hosts might still be filtered in searches with the walklex or typeahead command.

See Optimize field filter performance using Splunk Web.

Indexed fields are treated differently

In order to fully protect sensitive data, indexed field filters in searches with walklex and typeahead are applied to all related field-value pairs instead of index-time fields like searches with other commands.

For example, let's see what happens if we run the following search before a field filter is applied to search results:

|typeahead index=_audit prefix=action count=10

The search generates the following results:

content	count	operator
action	176142	false
"action.integration_url_override\\"	2	false
action.to/	2	false
action.url/	2	false
action/	2	false
"action=="	2	false
"action=accelerate_search"	2683	false
"action=add"	17318	false
"action=alert_"	214	false
"action=artifact_deleted"	7286	false

Now, let's see what happens to the search results if we add a field filter that replaces the action field and its value in the _audit index with the string masked. The same search generates the following results without any sensitive information:

content	count	operator
action	175366	false
"action.integration_url_override\\"	2	false
action.to/	2	false
action.url/	2	false
action/	2	false

Here's a similar example for the walklex command. Say you run the following search:

|walklex index=_audit type=all prefix=action |table term

The results look like this:

term
action
action=accelerate_search
action=alert_
action=artifact_deleted
action=change_authentication
action=create_token
action=dmc_manage_topology
action=edit_auto_ui_updates
action=edit_data_management_otelcollector

Now, let's apply the field filter that replaces the action field and its value in the _audit index with the string masked. The same search masks the key-value pairs, as shown in the following results:

action=masked

term
action
action=masked
action=masked
action=masked
action=masked
action=masked
action=masked
action=masked

Examples of searches with field filters

1. Hash a field value and get the count

An important consideration to performing hash functions on field values using SHA encryption is that field filters preserve the uniqueness of the field. That means that even if you encrypt identifying information, such as user ID numbers, you can still get a count of the number of user IDs without exposing the value. For example, say you create a field filter that sets the value of the host field to a SHA-256 hash.

Then, you run the following search:

index=main | stats count by host

Your search results might look like this:

host	count
C1fe560f903b48c9681269132a783a9333414bbf7ac26bf41cebbaf79b6d	8223

Notice that even though there is only one hash value, the count of user IDs is 8223.

2. Redact PII and PHI data in search results

You can use field filters to redact sensitive fields such as Personal identifiable information (PII) and Protected Health Information (PHI) from searches of all of your event data. For example, perhaps your search results include the name and phone number for a patient named Alex Martin, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search patient_name=AlexMartin phone=+1-555-555-0199', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

If you want to replace patient_name=AlexMartin and phone=+1-555-555-0199 with custom strings, you would create two _raw field filters for users who don't have authorization to see that sensitive information. One _raw field filter might be called staffName and is set to s/patient_name=[^ ']+/REMOVED-NAME/. The other _raw field filter might be called staffPhone and is set to s/phone=[^ ']+/REMOVED-PHONE/.

Then, when a person who holds a role that is affected by these field filters run the following search:

index=main

The person now sees only search results for the patient with REMOVED_NAME REMOVED_PHONE, instead of patient_name=AlexMartin phone=+1-555-555-0199, which looks like this:

Time

Event

2/17/23 3:57:04.597 PM

[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search REMOVED-NAME REMOVED-PHONE', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"]

The confidential data is no longer visible to all nonexempt roles that are affected by the field filters because the patient name and phone number have been replaced with custom strings.

3. Search for a redacted field

Searching for an index-time field might not return any results because the value of the filtered field is redacted at search time. For example, say you have an index-time field called password and you run the following search:

index=web_access_index sourcetype::corp_auth_log password=REDACTED

The search doesn't return any events because password=REDACTED is not indexed.

To search for the redacted field, run the following search:

index=web_access_index sourcetype::corp_auth_log | table _time, password, _raw | where password=REDACTED