Include or exclude specific incoming data
You can use allow list and deny list rules to determine which files that the Splunk platform consumes or excludes when you monitor a directory or set of directories. You can also apply these settings to
batch type monitoring inputs. When you define an allow list, Splunk Enterprise only indexes the files you specify. When you define a deny list, the Splunk platform ignores the specified files and processes all other files. You define these filters in the
input stanza in the inputs.conf configuration file. If you want to apply the same filters across all of the forwarders in a Splunk platform deployment, you can set up a deployment server to perform this task. See About deployment server and forwarder management in Updating Splunk Enterprise Instances.
You don't need to define both an allow list and a deny list in a configuration stanza. The settings are independent, but the deny list filter overrides the allow list filter if a file is in both lists. If you define both filters and a file matches them both, Splunk Enterprise does not index that file.
The list rules use the regular expression syntax to define the match on the file name or path. The rules must be contained within a configuration stanza, such as
[monitor://<path>]. The Splunk platform ignores filter lists that are not inside a stanza. When you define filter entries, you must use exact regular expression syntax.
Route and filter data
Instead of including or excluding your data inputs, you can filter specific events and send them to different queues or indexes.
Add the following line to your
monitor stanza in the local/inputs.conf file for the app context that you defined the input.
whitelist = <your_custom regex>
For example, to monitor only files with the .log extension, make the following change:
[monitor:///mnt/logs] whitelist = \.log$
Include multiple files
You can include multiple files in one line, using the "|" (pipe, or "OR") operator. For example, to include file names that contain
query.log OR my.log, add the following line to the inputs.conf file:
whitelist = query\.log$|my\.log$
Or, you can include only files that match exactly. See the following example:
whitelist = /query\.log$|/my\.log$
The dollar symbol ( $ ) anchors the regular expression to the end of the line. There is no space before or after the pipe ( | ) operator.
Add the following line to your
monitor stanza in the
/local/inputs.conf configuration file for the app context in which you defined the input.
blacklist = <your_custom_regex>
If you create a
blacklist entry for each file you want to ignore, Splunk Enterprise activates only the last filter.
Example 1: Exclude only files with a .txt extension
To ignore and not monitor only files with the txt extension, add the following line to the inputs.conf file:
[monitor:///mnt/logs] blacklist = \.txt$
Example 2: Exclude files with a .txt or .gz extension
To ignore and not monitor all files with either the .txt extension or the .gz extension, add the following line to the inputs.conf file:
[monitor:///mnt/logs] blacklist = \.(?:txt|gz)$
Example 3: Exclude an entire directory
To ignore entire directories beneath a monitor input, add the following line to the inputs.conf file:
[monitor:///mnt/logs] blacklist = archive|historical|\.bak$
This example configures Splunk Enterprise to ignore all files under /mnt/logs/ within the archive or historical directories, and all files ending in the *.bak extension.
Example 4: Exclude a file whose name contains a string
To ignore files whose names contain a specific string, add the following line to the inputs.conf file:
[monitor:///mnt/logs] blacklist = 2009022file\.txt$
This example ignores the webserver20090228file.txt and webserver20090229file.txt files under /mnt/logs/.
Example 5: Exclude Windows Event Code 4662 events whose Message field contains a specific value
To ignore Windows Event Code 4662 events whose
Message field contains events with the value
Account Name: "example account", add the following line to the inputs.conf file:
[WinEventLog:Security] blacklist1 = EventCode = "4662" Message = "Account Name:\s+(example account)"
Specify input paths with wildcards
How the Splunk platform handles log file rotation
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.1.0, 8.1.1, 8.1.2, 8.1.3