Create field filters using configuration files

Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, tstats, typeahead, and walklex), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.

How to create field filters using configuration files

If you don't want to use Splunk Web to create and manage your field filters, you can use configuration files instead. For example, you might choose to use configuration files if you want to automate the creation of field filters across a large deployment.

To maintain data integrity and security, do not use configuration files to exempt roles from field filters. Instead, use Splunk Web or Splunk platform REST API endpoints to exempt roles from field filters. See Protect PII, PHI, and other sensitive data with field filters.

You can only use configuration files to create and manage field filters in Splunk Enterprise.

Develop a plan for your field filters before you create them. See Plan for field filters in your organization.

To optimize performance by limiting your field filters to particular indexes, hosts, sources, or source types, see Optimize field filter performance using configuration files.

Prerequisites

• Have the necessary permissions to edit configuration files. Only users with file system access, such as system administrators, can edit configuration files.
• Know how to edit configuration files. Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
• Decide which directory to store configuration file changes in. There can be configuration files with the same name in your default, local, and app directories. See Where you can place (or find) your modified configuration files in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Steps

To create a field filter in the field_filters.conf file, follow these steps.

1. Open or create a local field_filters.conf file at $SPLUNK_HOME/etc/system/local if you are using *nix, or %SPLUNK_HOME%\etc\system\local if you are using Windows.
2. In the [<fieldFilterName>] stanza for the field filter that you want to create, change <fieldFilterName> to the name of the field filter you are creating. Field filter names can contain only alphanumeric characters and underscores "_". Spaces and special symbols are not allowed.
3. In the stanza for the new field filter, add the following new line for the optional description for your new field filter:

   description = <string description>
   

4. In the stanza for the new field filter, add a new line using Backus–Naur form (BNF) to assign operators for actions using the following BNF for <actions> syntax:
   

   action = <field> = <operator>
       <action> ::= <field> = <operator>  
       <operator> ::= null() | sha256() | sha512() | <string literal> | sed(<string literal>) 
       <field> ::= <string literal> 
   

   You can't use wildcard characters to specify multiple fields because field filters treat wildcards like any other character.

   See the following section for descriptions of the operators for actions.

5. To reload the field_filters.conf file so the changes take effect, type the following into your browser:

   http://<yoursplunkserver>:8000/en-US/debug/refresh?entity=admin/fieldfilters

If you create field filters for indexed fields that are generated from _raw, you should also create a field filter for the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.

Use the same replacement method for matching indexed and _raw field filters. For example, if a field filter for an indexed field uses a custom string XXX, then XXX should also be used to replace the field value for the matching _raw field.

Operators for actions

Operators for actions might be one of the following. Function names are case-sensitive.

Expression	Description
null()	Splunk software removes the field value from results of searches to which this filter is applied.
sha256()	Splunk software computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-256 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference.
sha512()	Splunk software computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-512 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference.
<string literal>	Splunk software replaces the fieldname value with the specified string wherever the field value appears in results of searches to which this filter is applied. A string literal is a sequence of characters enclosed in double quotation marks (" "). Use backslash ( \ ) to escape the \ and " characters in a string literal. For example, use \\ and \" .
sed(<string literal>)	For _raw fields. The sed expression acts on searches to which this filter is applied. The sed expression replaces strings in search results that are matched by a regular expression (s) or transliterates characters found in search results with corresponding characters provided by the sed expression (y). A string literal is a sequence of characters enclosed in double quotation marks (" "). Use backslash ( \ ) to escape the \ and " characters in a string literal. For example, use \\ and \" .

Restrict field filters to specific indexes

You must specify the searchable target indexes when you set up your field filters. To restrict field filters to specific indexes, set the index setting in field_filters.conf. If the index setting is empty, the field filter is invalid and doesn't apply to the field.

The field filter index setting doesn't support statements that include wildcards or the following operators: AND, OR.

Steps

To restrict a field filter to specific indexes, follow these steps.

1. Open or create a local field_filters.conf file at $SPLUNK_HOME/etc/system/local if you are using *nix, or %SPLUNK_HOME%/etc/system/local if you are using Windows.
2. In the [<fieldFilterName>] stanza for the field filter you want to modify, add a new line using Backus–Naur form (BNF) for the index setting using the following BNF for <actions> syntax:
   
   

   index =[index] 
   

   For example, to configure your field filter to filter events from the idx1 and idx2 indexes, update the index setting in the field_filters.conf file to index = idx1,idx2.

Examples

The following examples show how you can use configuration files to configure field filters that perform operations on fields in searches run by roles that are not exempt from field filters.

1. Redact a field

At your organization, the indexed field PatientName is sensitive for security reasons. You don't want any users to be able to see the PatientName field.

If you want to redact the field from the results of searches for all users, create an indexed field filter for the PatientName field with the operator for the action set to null(). This removes the PatientName field from the results of searches, so people won't even know the field exists. Removing a field and its value is the most secure method of protection because someone with malicious intentions can't deduce anything about the data if there is no trace of it in the search results.

To redact the PatientName field, add the following lines to your field_filters.conf file:

[filter_PatientName_indexed] 
action = "PatientName" = null() 
description = Redact the PatientName field.

2. Redact a field value

If you want to hide your field values, but not remove them from the search results entirely, you can set your field filter to replace the field value with a custom string. For example, if you want all users to see search results such as PatientName = XXXX, set a field filter to a custom string.

Now when anyone searches on events that include the PatientName field, the string XXX will appear in the results in place of the actual confidential information about the patient.

To redact the value of the PatientName field, add the following lines to your field_filters.conf file:

[filter_PatientName_indexed] 
action = "PatientName" = "XXXX" 
description = Redact the value of the PatientName field.

3. Obfuscate a field value

You can also set your field filter to replace the field value with a hash value. For example, if you want all users to see a hash value instead of a patient's account number in search results, create a field filter for the PatientName field and set the replacement method to SHA256.

Now when someone searches on events that include the PatientName field, a hash value appears in the results in place of the actual confidential information about the patient.

To obfuscate the value of the PatientName field with a SHA256 hash, add the following lines to your field_filters.conf file:

[filter_PatientName_indexed] 
action = "PatientName" = sha256() 
description = Obfuscate the PatientName field. 

4. Replace the component field from _raw

To ensure that no one can access sensitive information in _raw, add the following lines to your field_filters.conf file:

[filter_component_raw]
action = "_raw" = sed("s/\"component\":\"([^\"]+)\"/\"component\":\"REMOVED-COMP\"/g") 
description = Replace component field's value with REMOVED-COMP from _raw data.

See also

Protect PII, PHI, and other sensitive data with field filters

Create field filters using Splunk Web

Optimize field filter performance using configuration files