Include or exclude specific incoming data

You can use allow list and deny list rules to specify which files to consume or exclude when you monitor a directory or set of directories. You can also apply these settings to batch type monitoring inputs. When you define an allow list, Splunk Enterprise only indexes the files you specify. When you define a deny list, the platform ignores the specified files and processes all other files. You define these filters in the input stanza in the inputs.conf configuration file. If you want to apply the same filters across all of the forwarders in a Splunk platform deployment, you can set up a deployment server to perform this task.

It is not necessary to define both an allow list and a deny list in a configuration stanza. The settings are independent. If you do define both filters and a file matches them both, Splunk Enterprise does not index that file, as the blacklist filter overrides the whitelist filter.

The list rules use the regular expression syntax to define the match on the file name or path. They must be contained within a configuration stanza, for example [monitor://<path>]. The Splunk platform ignores filter lists that are not inside a stanza. When you define filter entries, you must use exact regular expression syntax.

To learn more about regular expressions and how to build them, see the Regular-expressions.info (http://regular-expressions.info) website.

Route and filter data

Instead of including or excluding your data inputs, you can filter specific events and send them to different queues or indexes.

Include files

• Add the following line to your monitor stanza in the /local/inputs.conf file for the app context that you defined the input.
  

  whitelist = <your_custom regex>
  

  
  

  For example, to monitor only files with the .log extension:

  [monitor:///mnt/logs]
      whitelist = \.log$
  

Include multiple files

You can include multiple files in one line, using the "|" (pipe, or "OR") operator. For example, to include filenames that contain query.log OR my.log.

whitelist = query\.log$|my\.log$

Or, you can include only files that match exactly.

whitelist = /query\.log$|/my\.log$

The "$" anchors the regular expression to the end of the line. There is no space before or after the "|" operator.

Exclude files

• Add the following line to your monitor stanza in the /local/inputs.conf configuration file for the app context in which you defined the input.
  

  blacklist = <your_custom_regex>
  

If you create a blacklist entry for each file you want to ignore, Splunk Enterprise activates only the last filter.

Example: Exclude only files with a .txt extension

To ignore and not monitor only files with the .txt extension:

[monitor:///mnt/logs]
    blacklist = \.txt$

Example 2: Exclude files with either a .txt or .gz extension

To ignore and not monitor all files with either the .txt extension OR the .gz extension:

[monitor:///mnt/logs]
    blacklist = \.(?:txt|gz)$

Example 3: Exclude an entire directory

To ignore entire directories beneath a monitor input:

[monitor:///mnt/logs]
    blacklist = archive|historical|\.bak$

This example configures Splunk Enterprise to ignore all files under /mnt/logs/ within the "archive" or "historical" directories, and all files ending in *.bak.

Example 4: Blacklist a file whose name contains a string

To ignore files whose names contain a specific string:

[monitor:///mnt/logs]
   blacklist = 2009022[89]file\.txt$

This example ignores the webserver20090228file.txt and webserver20090229file.txt files under /mnt/logs/.

Example 5: Exclude Windows Event Code #4662 events whose "Message" field contains a specific value

To ignore Windows Event Code #4662 events whose Message field contains events with the value Account Name: "example account":

[WinEventLog:Security]
blacklist1 = EventCode = "4662" Message = "Account Name:\s+(example account)"

This example ignores Windows Security events with Event Code 4662 and Message values with Account Name: example account.