Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Monitor files and directories with inputs.conf

You can use inputs.conf to monitor files and directories with Splunk Enterprise. Inputs.conf provides the most configuration options for setting up a file monitor input. For Splunk Cloud, use Splunk Web to configure file monitoring inputs instead.

To configure an input to Splunk Enterprise, add a stanza to inputs.conf in $SPLUNK_HOME/etc/system/local/ or your own custom application directory in $SPLUNK_HOME/etc/apps/. These locations are on the machine that runs Splunk Enterprise.

You can configure multiple settings in an input stanza. If you do not specify a value for a setting, Splunk Enterprise uses the default for that setting. You can find the defaults for settings in $SPLUNK_HOME/etc/system/default/inputs.conf.

For more information about configuration files, see About configuration files.

Configure monitoring of files with inputs.conf

  1. On the machine that runs Splunk Enterprise, open a shell or command prompt.
  2. Change to the $SPLUNK_HOME/etc/system/local directory.
  3. (Optional) If inputs.conf does not exist, create the file.
  4. Open inputs.conf for editing with a text editor.
  5. Add a stanza that references the files or directories that you want to monitor. For example, to monitor the /var/log/messages file on a *nix system, specify:
    [monitor:///var/log/messages]
    disabled = 0
    

    To monitor C:\Windows\System32\WindowsUpdate.log on a Windows system, specify:

    [monitor://C:\Windows\System32\WindowsUpdate.log]
    disabled = 0
    
  6. (Optional) Add settings that further configure the input, depending on what you want the input to do. See Configuration settings later in this topic, or review the inputs.conf configuration specification file in the Admin Manual for additional settings.
    [monitor://path/to/file]
    disabled = 0
    setting1 = value
    setting2 = value
    ...
    
  7. Save the inputs.conf file and close it.
  8. Either restart Splunk Enterprise or reload the configuration by running the following command. Splunk Enterprise prompts you for credentials if you reload the configuration.
    ./splunk _internal call /services/data/inputs/monitor/_reload -auth
    

Configuration settings

You can use the following settings in both monitor and batch input stanzas.

Setting Description Default
host = <string> Sets the host key to a static initial value for this stanza. The input processor uses the key during parsing and indexing to set the host field and uses the field during searching. Splunk Enterprise prepends the <string> with host::. the IP address or fully-qualified domain name of the host where the data originated.
index = <string> Sets the index where events from this input will be stored. Splunk Enterprise prepends the <string> with index::.

For more information about the index field, see How indexing works in the Managing Indexers and Clusters manual.

main or whatever you set the default index to
sourcetype = <string> Sets the sourcetype key/field for events from this input. Explicitly declares the source type for this data, as opposed to letting Splunk Enterprise determine it automatically. This is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing.

Sets the sourcetype key initial value. Splunk Enterprise uses the key during parsing and indexing to set the source type field and uses the source type field during searching. Splunk Enterprise prepends the <string> with sourcetype::.

For more information about source types, see Why source types matter.

Splunk Enterprise picks a source type based on various aspects of the data. There is no default.
queue = parsingQueue | indexQueue Specifies where the input processor should deposit the events that it reads. Set to "parsingQueue" to apply props.conf and other parsing rules to your data. Set to "indexQueue" to send your data directly into the index. parsingQueue
_TCP_ROUTING = <tcpout_group_name>,<tcpout_group_name>,... Specifies a comma-separated list of tcpout group names. Use this setting to selectively forward your data to specific indexers by specifying the tcpout groups that the forwarder should use when forwarding the data.

Define the tcpout group names in outputs.conf in [tcpout:<tcpout_group_name>] stanzas.

the groups present in 'defaultGroup' in [tcpout] stanza in outputs.conf
host_regex = <regular expression> A regular expression that extracts host from the file name of each input. Specifically, Splunk Enterprise uses the first group of the regular expression as the host. the default "host =" setting, if the regular expression fails to match
host_segment = <integer> Sets the segment of the path as the host, using <integer> to determine the segment. For example, if host_segment = 2, host becomes the second segment of the path. Path segments are separated by the '/' character. the default "host =" setting, if the value is not an integer, or is less than 1

Monitor syntax

Monitor input stanzas configure Splunk Enterprise to watch all files in the <path> or the <path> itself if it represents a single file. You must specify the input type before the path, so add three slashes in the path if the path includes the root directory on *nix machines.

You can use wildcards for the path. See Specify input paths with wildcards.

[monitor://<path>]
<setting1> = <val1>
<setting2> = <val2>
...

The following are additional settings you can use when defining monitor input stanzas:

Setting Description Default
source = <string> Sets the source field for events from this input. You can use this setting when using the MonitorNoHandle input and want to set the source to the name of the file you are monitoring. Otherwise, do not override unless absolutely necessary. Consider use of source types, tagging, and search wildcards instead. The input layer usually provides a more accurate string to aid in problem analysis and investigation by accurately recording the file from which the data was retrieved.

Splunk Enterprise prepends the <string> with source::.

the input file path (except in the case of MonitorNoHandle, where it is MonitorNoHandle)
crcSalt = <string> Forces Splunk Enterprise to index files that have matching cyclic redundancy checks (CRCs). By default, the software only performs CRCs against the first few lines of a file. This behavior prevents indexing of the same file twice, even though you might have renamed it, such as with rolling log files. However, because the CRC counts only the first few lines of the file, it is possible for legitimately different files to have matching CRCs.)

If set, Splunk Enterprise adds string to the CRC. If set to <SOURCE>, Splunk Enterprise adds the full source path to the CRC. This ensures that each file being monitored has a unique CRC.

Use caution with this setting for rolling log files. It can lead to the log file being re-indexed after it has rolled.

This setting is case sensitive.

N/A
ignoreOlderThan = <time_window> Causes the input to stop checking files for updates if the file modification time (modtime) has passed the <time_window> threshold. This improves the speed of file tracking operations when monitoring directory hierarchies with large numbers of historical files (for example, when active log files share a directory with old files that no longer get writes).

Splunk Enterprise does not index files whose modification time falls outside <time_window> when it first attempts to monitor the file.

You must specify <number><unit>. For example, "7d" indicates one week. Valid units are "d" (days), "h" (hours), "m" (minutes), and "s" (seconds).

0 (disabled)
followTail = 0|1 If set to 1, monitoring begins at the end of the file (like *nix tail -f). This only applies to files the first time Splunk Enterprise attempts to monitor them. After that, Splunk Enterprise keeps track of the file using its internal file position records. 0
whitelist = <regular expression> If set, Splunk Enterprise only monitors files whose names match the specified regular expression. N/A
blacklist = <regular expression> If set, Splunk Enterprise does NOT monitor files whose names match the specified regular expression. N/A
alwaysOpenFile = 0 | 1 If set to 1, Splunk Enterprise opens a file to check if it has already been indexed. This is only useful for files that don't update their modification time.

Use this setting for monitoring files on Windows, and for Internet Information Server (IIS) logs.

Caution: Use of this setting increases load and slows down indexing.

N/A
recursive = true|false If set to false, Splunk Enterprise does not look into subdirectories that it finds within a monitored directory. true
time_before_close = <integer> The modification time delta required before Splunk Enterprise can close a file on End-of-file (EOF). Tells the system not to close files that have been updated in the past <integer> seconds. 3
followSymlink = true|false If false, Splunk Enterprise ignores symbolic links that it finds within a monitored directory. true

MonitorNoHandle syntax

On Windows systems only, the MonitorNoHandle input monitors files without using Windows file handles. This lets you read special log files like the Windows DNS server log files.

You must specify a valid path to a file when you use the MonitorNoHandle input stanza. You cannot specify a directory. If you specify a file that already exists, Splunk Enterprise does not index the existing data in the file. It only indexes new data that the system writes to the file.

MonitorNoHandle sets the source for files you monitor to MonitorNoHandle.To specify the file name as the source, you must use the source setting in the stanza for the MonitorNoHandle input for the file.

You can only configure MonitorNoHandle on a Windows machine with inputs.conf or the Command Line Interface (CLI). You cannot configure it in Splunk Web or on a *nix machine.

[MonitorNoHandle://<path>]
disabled=0
source = <path>
<setting1> = <val1>
<setting2> = <val2>
...

Batch syntax

Use batch to set up a one-time, destructive input of data from a source.

For continuous, non-destructive inputs, use monitor. Splunk enterprise deletes data that it has indexed with the batch monitor.

[batch://<path>]
move_policy = sinkhole
<setting1> = <val1>
<setting2> = <val2>
...

When you define batch inputs, you must include the setting move_policy = sinkhole. This loads the file destructively. Do not use the batch input type for files that you do not want to delete after indexing.

To ensure that Splunk Enterprise indexes new events when you copy over an existing file with new contents, set the CHECK_METHOD = modtime setting in props.conf for the input source. This checks the modification time of the file and re-indexes it when the time changes. Splunk Enterprise indexes the entire file, which can result in duplicate events.

Examples of monitor input stanzas

Single *nix file

This example stanza configures Splunk Enterprise to index the single file, /var/log/messages.

[monitor:///var/log/messages]
disabled = 0
sourcetype = unixlog

Single Windows directory

This Windows example configures Splunk Enterprise to monitor the directory, C:\Windows\Logs. and all the files in it.

[monitor://C:\Windows\Logs]
disabled = 0

Multiple Windows directories

This Windows example tells Splunk Enterprise to monitor all of the directories in C:\Windows\Debug.

[monitor://C:\Windows\Debug\*]
disabled = 0

Multiple *nix directories with a wildcard

This example configures Splunk Enterprise to monitor directories like /apache/foo/log, /apache/bar/log, etc.

[monitor:///apache/.../logs]

Multiple *nix files in one directory with a wildcard

This *nix example configures Splunk Enterprise to monitor multiple files in one directory, such as /apache/*.log.

[monitor:///apache/*.log]

MonitorNoHandle, single Windows file

This Windows example uses the MonitorNoHandle input to monitor a file that Windows has open for writing, such as C:\Windows\System32\DNS\DNS.log.

[MonitorNoHandle://<path>]
disabled = 0
source = <path>
<setting1> = <val1>
<setting2> = <val2>
...

Batch

This batch example loads and deletes all files from the directory system/flight815/.

[batch://system/flight815/*]
move_policy = sinkhole
PREVIOUS
Monitor files and directories with the CLI
  NEXT
Specify input paths with wildcards

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.2, 7.3.1, 8.0.0


Comments

Thank you for your comment, Yutaka1005. There was an error when we exported this manual for localization, and this topic was not included. We will pick it up again in the next release. The topic is exactly the same for 7.1.0, so you can refer to the Japanese version of the 7.1 manual to see the translation.

Cgales splunk, Splunker
February 28, 2019

In the following manual, which is the Japanese version of this manual, the section corresponding to this chapter is missing.

https://docs.splunk.com/images/d/d2/Splunk-7.2.0-Data_ja-JP.pdf

Yutaka1005
February 25, 2019

Hi Mykol j,

Thanks for your feedback. We could probably have more Windows examples in this page. I'll get some added shortly.

Malmoore, Splunker
January 26, 2018

Windows examples are terrible to non-existant. I have seen suggested from other Splunk answers it's something like:

[monitor://$WINDIR\temp\flight815\*]

But what about spaces?
For example, this is not working:
[monitor://$WINDIR\Program Files\Microsoft\Exchange Server\V15\Logging\Imap4]

Thanks,
Michael

Mykol j
January 23, 2018

Hi Reedmohn,

The correct usage is MonitorNoHandle.

I spoke with an engineer about the case and he said that was how they coded it. Presumably, there was no reason behind it.

Thanks,

Malmoore, Splunker
May 8, 2017

This doc uses both
monitorNoHandle and
MonitorNoHandle

Only the latter works, I believe.
Also: Why is this with a capital M, unlike everything else?

Reedmohn
May 8, 2017

It's a little strange to refer to these sinkhole inputs as "one time". It is certainly true that the moment you create the input the files there will be deleted. However it will continue indexing anything that is created or copied there! "one time" makes it sound like it's not an ongoing thing, and also sounds too much like the "oneshot" input, where it really is just a "one time" indexing deal with no ongoing effects. </2cents>

Sideview
November 1, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters