Splunk® Enterprise

Getting Data In

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Monitor files and directories

Splunk has two file input processors: monitor and upload. For the most part, you can use monitor to add all your data sources from files and directories. However, you might want to use upload to add one-time inputs, such as an archive of historical data.

You can add inputs to monitor or upload using any of these methods:

You can use the data preview feature to see how Splunk will index a file's data. See "Overview of data preview" for details.

How monitor works in Splunk

Specify a path to a file or directory and Splunk's monitor processor consumes any new data written to that file or directory. This is how you can monitor live application logs such as those coming from J2EE or .NET applications, Web access logs, and so on. Splunk continues to monitor and index the file or directory as new data appears. You can also specify a mounted or shared directory, including network file systems, so long as Splunk can read from the directory. If the specified directory contains subdirectories, Splunk recursively examines them for new files.

Splunk checks for the file or directory specified in a monitor configuration on Splunk start and restart. Subdirectories of monitored directories are scanned continuously. To add new inputs without restarting Splunk, use Splunk Web or the CLI. If you want Splunk to find potential new inputs automatically, use crawl.

When using monitor, note the following:

  • On most file systems, files can be read even as they are being written to. However, Windows file systems have the ability to prevent files from being read while they are being written, and some Windows programs may use these modes, though most do not.
  • Files or directories can be included or excluded via whitelists and blacklists.
  • Upon restart, Splunk continues processing files where it left off.
  • Splunk decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z.
  • If you add new data to an existing archive file, Splunk will re-index the entire file, not just the new data in the file. This can result in duplication of events.
  • Splunk detects log file rotation and does not process renamed files it has already indexed (with the exception of .tar and .gz archives; for more information see "Log file rotation" in this manual).
  • The entire dir/filename path must not exceed 1024 characters.
  • Splunk does not index files with a .splunk filename extension. This is because Splunk expects files with that extension to be metadata information files. If you need to index files with a .splunk extension, use the add oneshot CLI command.
  • Disabling or deleting a file-based input using the command line or Manager does not stop the input's files from being indexed. Rather, it stops files from being checked again, but all the initial content will be indexed. To stop all in-process data, you must restart the Splunk server.

Monitor inputs may overlap. So long as the stanza names are different, Splunk will treat them as independent stanzas and files matching the most specific stanza will be treated in accordance with its settings.

Why use upload or batch

To index a static file once, select Upload a local file or Index a file on the Splunk server in Splunk Web. The file will not be monitored on an ongoing basis.

You can also use the CLI add oneshot or spool commands for the same purpose. See Use the CLI for details.

Use the batch input type in inputs.conf to load files once and destructively. By default, Splunk's batch processor is located in $SPLUNK_HOME/var/spool/splunk. If you move a file into this directory, Splunk indexes it and then deletes it.

Note: For best practices on loading file archives, see "How to index different sized archives" on the Community Wiki.

What Splunk does with your data (and how to make it do it better)
Use Splunk Web

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters