Plan for field filters in your organization

Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, tstats, typeahead, and walklex), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes with field filters.

Planning considerations

Before you configure field filters in your organization, carefully plan how you want to implement field filters in the Splunk platform. Start at the highest level and decide which indexes you want to protect, then work your way down to the hosts, sources, or source types. Then, consider the fields you want to restrict, the field values you want to assign to those fields, the roles that you want exempted from the field filter, if any, and so on.

Consider the following areas as you plan your strategy for field filters.

• Create and manage your field filters
• Indexes
• Hosts, sources, or source types
• Fields and field values
• Number of field filters
• Naming field filters
• Roles and inherited roles
• Administrator permissions
• Optimize performance
• Downstream operations
• Search heads and indexers
• READ THIS: Restricted commands do not work in searches on indexes with field filters
• Migrate existing sed expressions from the SEDCMD setting
• Limitations on using field filters in your environment

Create and manage your field filters

How will you create and manage your field filters? Using Splunk Web is the easiest and most efficient way to configure field filters. But, depending on your organization's needs, you might decide to use Splunk platform REST API or, if you are using Splunk Enterprise, you might be more comfortable using configuration files.

You can create and manage field filters using one of the following methods:

• Use Splunk Web. See the following in this manual:
  • Create field filters using Splunk Web.
  • Optimize field filter performance using Splunk Web.
  • Exempt certain roles from field filters using Splunk Web.

• Update settings in the field_filters.conf file if you're using Splunk Enterprise. See the following in this manual:
  • Create field filters using configuration files.
  • Optimize field filter performance using configuration files.

• Use the Splunk platform REST API authorization/fieldfilters/{name} endpoint. You can also use the authorization/roles/{name} endpoint to exempt certain roles from field filters. See the following in Splunk Cloud Platform REST API Reference Manual:
  • authorization/fieldfilters
  • authorization/fieldfilters{name}
  • authorization/roles
  • authorization/roles{name}

Indexes

Which indexes contain the data you want to protect? For best performance and enhanced security, you must specify the searchable target index when you set up your field filters.

For more information about managing indexes and roles, see Create and manage roles with Splunk Web.

Hosts, sources, or source types

Which specific hosts, sources, or source types apply to the fields that you want to protect with field filters?

Fields and field values

Which sensitive data in your events do you need to protect? Are there certain indexed fields that contain personal identifiable information (PII), protected health information (PHI), or other confidential data that you need to keep private? You might consider creating a document that helps you keep track of all the fields and their values in one place.

Indexed and _raw field types

Field filters can apply to fields that are extracted at index time or later at search time. As a result, the sensitive fields you filter can be indexed fields, which are default fields that Splunk platform adds automatically, as well as any custom fields that you specify.

You can also use field filters to modify _raw fields, which are the source events that appear in search results. A _raw field is a type of default field that contains the original source data of events and is typically used as the basis for index-time and search-time field extractions. Your _raw fields contain the same confidential information that your indexed field filters safeguard. Therefore, if you have a field filter for an indexed field, but don't have a field filter for the _raw field that matches the events from which the indexed field was extracted, a user who isn't authorized to see the confidential information in the indexed field might still be able to see it in the _raw field. To keep all of your confidential data safe and ensure that your indexed field filters can't be circumvented, create a field filter that uses regular expressions to filter on each _raw field that extracts fields that your indexed field filters protect.

Another factor to consider is the implications of using indexed fields for downstream operations that depend on the value of the field that is changed by a field filter. See The sequence of search-time operations ​in the Splunk Cloud Platform Knowledge Manager Manual.

Replacement methods

Before you create your field filters, plan your field filters strategy and identify the specific fields that need to be deleted from the search results, or replaced with a custom string or hash. Your approach depends on the level of protection your data needs.

• If your data needs the highest form of privacy for legal or regulatory reasons, your plan might need to include redacting field values by deleting them or replacing them with a custom string. The delete option in Splunk Web removes the field and the custom string option replaces the field with a string value.
• If you plan to obfuscate your field values with SHA-256 or SHA-512 (SHA-2 family) hash functions, make sure that replacing field values with a hash offers the security your data requires.
• If you want to preserve the uniqueness of a field for statistical use cases, for example, when using the stats or timechart commands in searches, plan to replace the field value with a hash.

Number of field filters

Once you consider which fields you want to filter with field filters, decide how many field filters you need. If your organization has multiple fields per source type, you might need to configure multiple field filters for each source type.

Your list of field filters should include field filters for indexed fields containing sensitive information you want to protect, as well as field filters for the _raw fields that are used to generate those indexed fields. Typically there will be more field filters for indexed fields, and fewer field filters for _raw fields.

For performance reasons, the total number of field filters in your environment should not exceed 5,000.

Multiple field filters for the same field

If you need to create overlapping _raw field filters to ensure coverage of sensitive data, keep the following in mind.

• Multiple field filters for the same _raw field are allowed, but only one field filter is applied at a time.
• Multiple field filters for the same indexed field are allowed provided each field filter has a different target index.
• In the unlikely case there is a conflict between multiple field filters, the field filter that is last in lexicographical order takes precedence and is applied. For example, if aFieldFilter replaces the value of the field with the string A, and bFieldFilter replaces the value of the field with the string B, and cFieldFilter replaces the value of the field with a hash, then A is replaced with B, and B is hashed. The final field value that appears in the search results is a hash.
• Field filters for the same indexed field that have different replacements are treated as unique field filters. For example, if you have two field filters for the SSN field, and one field filter is limited by the source type and the other by the host, the field filters overlap, but are distinct.

Naming field filters

Give careful consideration to your naming conventions for your field filters. Your field filter naming convention is especially important if you want your field filters to be applied in a certain order. Since field filters are applied in lexicographical order, a field filter called 01-filter-ID executes before 02-filter-ID because the 01 prefix is applied before the 02 prefix. Therefore, the final value of the field is determined by 02-filter-ID.

Roles and inherited roles affected by field filters

Who does your sensitive data need to be protected from? Which users in your organization are authorized or not authorized to see your sensitive data? Identify which privileged roles need to be able to continue accessing sensitive data, and which roles should be restricted. You can control who sees what in your organization using field filters, so plan ahead for which field filters will be set for various roles in your organization. By default, field filters apply to all roles, unless you specify certain roles that are exempt from the field filter because they are authorized to access the confidential data. You specify exempt roles when you create your field filter in Splunk Web. See Create and manage roles with Splunk Web.

If you use inherited roles, consider role inheritance when you exempt roles from field filters. Role exemptions for field filters are inherited, which means that roles inherit the exemptions of their parent. For example, say the User role is exempt from a field filter, and another role called UserInherited inherits the User role. Because of role inheritance, all users who are assigned to the User role and UserInherited role are exempt from the field filter.

Administrator permissions

Who are the trusted administrators you want to oversee field filters in your organization? Make sure that the administrators who will view, create, edit, and delete field filters can be trusted with your most sensitive data.

To perform all actions on field filters, including viewing, creating, editing, and deleting field filters, your administrators must be a member of the predefined admin role, which includes the edit_field_filter capability by default.

If you don't want to use the predefined admin role to manage field filters, you can create your own custom roles instead. If you plan to use custom roles, make sure that the roles have the following capabilities:

• view_field_filter
• edit_field_filter
• change_authentication

Users with custom roles can view, create, and edit field filters, but they can't delete field filters; only users with the admin role are allowed to delete field filters.

Optimize performance

All field filters affect the performance of searches to some extent because of the extra processing overhead required to filter fields. As you plan how you will deploy field filters in your organization, keep the following performance considerations in mind:

• Minimize the number and complexity of field filters. The total number of field filters in your environment should not exceed 5,000.
• Limit each field filter to a specific target index, host, source, or source type. See Optimize field filter performance using Splunk Web.

Minimize the number and complexity of field filters

The performance impact of field filters depends on how each field filter is defined and how many there are. Search performance degrades as you increase the number and complexity of field filters. A few simple field filters that don't use sed expressions or SHA hash functions have minimal performance impact on searches, similar to that of filters with the eval command.

Limit each field filter to a specific target index, host, source, or source type

For optimal performance, limit each field filter to a specific target index, host, source, or source type when you create your field filter in Splunk Web, which avoids performance impacts on unrelated search results. For example, if you don't specify a host limit on a field filter, the field filter will be applied to all hosts in search results, which will result in inefficient searches. The target index is required when setting up your field filter. See Optimize field filter performance using Splunk Web.

Downstream operations

Field filters remove _raw and indexed fields, or replace their values, when those fields appear before the field filters operation at search time. As the fourth operation in the search-time operation sequence, field filters are processed before other operations that come later in the sequence. As a result, any operations performed after field filters in the sequence that depend on certain field values might break when expected field values are filtered with field filters. You might need to configure field filters using hash functions that preserve the statistical uniqueness of field values for later operations, or reevaluate search operations that are used together with field filters. See The sequence of search-time operations in the Splunk Cloud Platform Knowledge Manager Manual.

Search heads and indexers

If you have a distributed Splunk Enterprise environment and plan to run field filters on a search head, the versions of the search head and indexers must be compatible. For example, to configure field filters on a search head running Splunk Enterprise version 9.3.0, the indexers must also run Splunk Enterprise version 9.3.0 or higher. Your plans for rolling out field filters may need to include upgrading your search heads and indexers, so that all of your versions are in sync.

READ THIS: Restricted commands do not work in searches on indexes with field filters

When you deploy field filters in your organization, the following commands will no longer work in searches across any indexes in the organization:

• mpreview
• mstats
• tstats
• typeahead
• walklex

The Splunk platform disables these commands by default if field filters are present, and does not allow any users to run these commands, because they can return sensitive index information to which users restricted by field filters might not be allowed to access. Without the safeguard around restricted commands, your organization might be exposed to a potential security risk if someone with malicious intentions tries to use these commands to get around field filters.

Restricted commands workaround

If you want certain highly trusted roles to be able to use these restricted commands when field filters are in use, you must configure the roles to have the run_commands_ignoring_field_filter capability. Users with this capability can run restricted commands that return index information even when their role is affected by a field filter. To add capabilities to a role, see Define roles on the Splunk platform with capabilities.

Migrate existing sed expressions from the SEDCMD setting

If you use the SEDCMD setting to anonymize raw data at index time, you can now use field filters to remove the same type of sensitive data at search time instead. Just migrate the sed expression used with the SEDCMD setting by copying and pasting the sed expression into a new _raw field filter in Splunk Web. For information about the SEDCMD setting, see Anonymize data in the Splunk Cloud Platform Getting Data In.

Limitations on using field filters in your environment

Consider the following issues when setting up your environment to take advantage of field filters.

• Support for generating commands
• Add new fields
• Data model acceleration
• Federated search
• Summary indexes

Support for generating commands

When you set up searches with field filters, keep in mind that field filters do not support searches that use generating commands other than the search command. This means that the first command in the search string must not be another command besides search. For example, in the following search, the field filter is not applied because the generating command is inputcsv, not search:

| inputcsv ... | search ...

However, the field filter is applied in the following search:

index=main | ....

Technically, the search command comes first in the pipeline, but doesn't actually need to be specified because it is implied. See Generating commands in the Splunk Cloud Platform Search Reference.

Add new fields

Don't use field filters to add new fields at search time. Use calculated fields for this purpose. See About calculated fields in the Splunk Cloud Platform Knowledge Manager Manual.

Data model acceleration

Field filters don't support data model acceleration because the tstats command is a restricted command and can return sensitive data that a role with field filters might not be allowed to access. To allow certain highly trusted roles that are configured with field filters to continue to use accelerated data models, you must configure the role to have one of the following capabilities:

• The run_commands_ignoring_field_filter capability. Users with this capability can run commands that return index information even when their role is not exempt from a field filter.
• The admin_all_objects capability. This capability is very powerful. Users with this capability have access to all objects in the system. Use this capability with caution and only with the most trusted roles in your organization.

Roles that are not exempt from a field filter and are configured with one of these capabilities can view data acceleration as usual, but without field filters.

See Accelerate data models in the Splunk Cloud Platform Knowledge Manager Manual.

Federated search

Field filters can't be used to protect sensitive fields in remote datasets using federated search.

Summary indexes

Summary indexes generated by scheduled reports can expose sensitive index information that you might not want unprivileged roles to be able to access. If you collect sensitive summary index information that you want to filter with field filters, carefully consider which highly trusted roles can create those summary indexes.

By default, all reports run with the permissions of the report "owner", which is the person who created the report. To ensure that unprivileged users don't have access to sensitive data, set up your summary indexes using a role configured with field filters. For example, if your field filter applies to the support role, you must also use the support role when you set up your summary indexes.

See Determine whether to run reports as the report owner or report user in the Splunk Cloud Platform Reporting Manual and Create a summary index in Splunk Web in the Splunk Cloud Platform Knowledge Manager Manual.

Example

If the role you use to set up your scheduled searches is exempt from a field filter, but your users hold a different role that is restricted by the field filter, you might expose your confidential data to people who aren't allowed to access that information.

Consider what might happen if you set up a scheduled search using the admin role, which you have exempted from a field filter that replaces the value of the host name field with a SHA-256 hash. When a user holding a role called test that is not exempt from the field filter runs a search on the index, this is what they might see:

Time	Event
2/28/23 10:15:00.000 AM	02/28/2023 10:15:00 -0800, search_name=TestSummary, search_now=1646075700.000, info_min_time=1646072100.000, info_max_time=1646075700.000, info_search_time=1646075700.607, orig_host="test-mbp-571a7", action=upload_mmdb_files, psrsvd_v=1, psrsvd_gc=9, testField="testValue" host = c1fe560f903b48c96812691329a783a9333414bbf7ac26c26bf41cebbaf79b6d

Notice that the host name shows up in two places in the summary index. The value for the first instance of the host field is test-mbp-571a7, and the value for the second instance is a hash. In this case, using one role, the admin role, to set up the scheduled search and then running the search using a different role, the test role, that is configured with field filters, resulted in exposing the data that was supposed to be obfuscated with the hash value.

If the same test role that is configured with field filters is also used to set up the scheduled search, this is what the user who holds the test role sees when they run the search:

Time	Event
2/28/23 10:15:00.000 AM	02/28/2023 10:15:00 -0800, search_name=TestSummary, search_now=1646075700.000, info_min_time=1646072100.000, info_max_time=1646075700.000, info_search_time=1646075700.607, orig_host="c1fe560f903b48c96812691329a783a9333414bbf7ac26c26bf41cebbaf79b6d", action=upload_mmdb_files, psrsvd_v=1, psrsvd_gc=9, testField="testValue" host = c1fe560f903b48c96812691329a783a9333414bbf7ac26c26bf41cebbaf79b6d

Now both instances of the host field in the summary index are obfuscated with the hash value. As a result, sensitive data about the true value of the host name is no longer exposed to unprivileged users.

See also

Protect PII, PHI, and other sensitive data with field filters

Create field filters using Splunk Web

Optimize field filter performance using Splunk Web

Create and manage roles with Splunk Web