Include or exclude specific incoming data

You can use allow list and deny list rules to determine which files that the Splunk platform consumes or excludes when you monitor a directory or set of directories. You can also apply these settings to batch type monitoring inputs. When you define an allow list, only indexes the files you specify. When you define a deny list, the Splunk platform ignores the specified files and processes all other files. You define these filters in the input stanza in the inputs.conf configuration file. If you want to apply the same filters across all of the forwarders in a Splunk platform deployment, you can set up a deployment server to perform this task. See About deployment server and forwarder management in Updating Splunk Enterprise Instances.

You don't need to define both an allow list and a deny list in a configuration stanza. The settings are independent, but the deny list filter overrides the allow list filter if a file is in both lists. If you define both filters and a file matches them both, does not index that file.

The list rules use the regular expression syntax to define the match on the file name or path. The rules must be contained within a configuration stanza, such as [monitor://<path>]. The Splunk platform ignores filter lists that are not inside a stanza. When you define filter entries, you must use exact regular expression syntax.

Route and filter data

Instead of including or excluding your data inputs, you can filter specific events and send them to different queues or indexes.

Include files

Add the following line to your monitor stanza in the local/inputs.conf file for the app context that you defined the input.

whitelist = <your_custom regex>

For example, to monitor only files with the .log extension, make the following change:

[monitor:///mnt/logs]
    whitelist = \.log$

Include multiple files

You can include multiple files in one line, using the "|" (pipe, or "OR") operator. For example, to include file names that contain query.log OR my.log, add the following line to the inputs.conf file:

whitelist = query\.log$|my\.log$

Or, you can include only files that match exactly. See the following example:

whitelist = /query\.log$|/my\.log$

The dollar symbol ( $ ) anchors the regular expression to the end of the line. There is no space before or after the pipe ( | ) operator.

Exclude files

Add the following line to your monitor stanza in the /local/inputs.conf configuration file for the app context in which you defined the input.

blacklist = <your_custom_regex>

If you create a blacklist entry for each file you want to ignore, activates only the last filter.

Example 1: Exclude only files with a .txt extension

To ignore and not monitor only files with the txt extension, add the following line to the inputs.conf file:

[monitor:///mnt/logs]
    blacklist = \.txt$

Example 2: Exclude files with a .txt or .gz extension

To ignore and not monitor all files with either the .txt extension or the .gz extension, add the following line to the inputs.conf file:

[monitor:///mnt/logs]
    blacklist = \.(?:txt|gz)$

Example 3: Exclude an entire directory

To ignore entire directories beneath a monitor input, add the following line to the inputs.conf file:

[monitor:///mnt/logs]
    blacklist = archive|historical|\.bak$

This example configures to ignore all files under /mnt/logs/ within the archive or historical directories, and all files ending in the *.bak extension.

Example 4: Exclude a file whose name contains a string

To ignore files whose names contain a specific string, add the following line to the inputs.conf file:

[monitor:///mnt/logs]
   blacklist = 2009022[89]file\.txt$

This example ignores the webserver20090228file.txt and webserver20090229file.txt files under /mnt/logs/.

Example 5: Exclude Windows Event Code 4662 events whose Message field contains a specific value

To ignore Windows Event Code 4662 events whose Message field contains events with the value Account Name: "example account", add the following line to the inputs.conf file:

[WinEventLog:Security]
blacklist1 = EventCode = "4662" Message = "Account Name:\s+(example account)"