How Splunk handles log file rotation

How Splunk handles log file rotation

Splunk recognizes when a file that it is monitoring (such as /var/log/messages) has been rolled (/var/log/messages1) and will not read the rolled file a second time.

How to work with log rotation into compressed files

Splunk does not identify compressed files produced by logrotate (such as bz2 or gz) as being the same as the uncompressed originals. This can lead to a duplication of data if those files are then monitored by Splunk.

To avoid this problem, you can choose from two approaches:

• Configure logrotate to move those files into a directory that Splunk is not monitoring.

• Set blacklist rules for archive filetypes to prevent Splunk from reading those files as new logfiles.

Example:

 blacklist = \.(gz|bz2|z|zip)$ 

Splunk recognizes the following archive filetypes: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z.

For more information on setting blacklist rules see "Whitelist or blacklist specific incoming data" in this manual.

On a related issue, if you add new data to an existing compressed archive such as a .gz file, Splunk will re-index the entire file, not just the new data in the file. This can result in duplication of events.

How Splunk recognizes log rotation

The monitoring processor picks up new files and reads the first and last 256 bytes of the file. This data is hashed into a begin and end cyclic redundancy check (CRC). Splunk checks new CRCs against a database that contains all the CRCs of files Splunk has seen before. The location Splunk last read in the file, known as the file's seekPtr, is also stored.

There are three possible outcomes of a CRC check:

1. There is no begin and end CRC matching this file in the database. This indicates a new file. Splunk will pick it up and consume its data from the start of the file. Splunk updates the database with the new CRCs and seekPtrs as the file is being consumed.

2. The begin CRC and the end CRC are both present, but the size of the file is larger than the seekPtr Splunk stored. This means that, while Splunk has seen the file before, there has been data added to it since it was last read. Splunk opens the file, seeks to the previous end of the file, and starts reading from there. In this way, Splunk will only grab the new data and not anything it has read before.

3. The begin CRC is present, but the end CRC does not match. This means that Splunk has previously read the file but that some of the material that it read has since changed. In this case, Splunk must re-read the whole file.

Important: Since the CRC start check is run against only the first 256 bytes of the file, it is possible for non-duplicate files to have duplicate start CRCs, particularly if the files are ones with identical headers. To handle such situations you can

• Use the initCrcLength attribute to increase the number of characters used for the CRC calculation, and make it longer than your static header.
• Use the crcSalt attribute when configuring the file in inputs.conf, as described in "Edit inputs.conf" in this manual. The crcSalt attribute ensures that each file has a unique CRC. You do not want to use this attribute with rolling log files, however, because it defeats Splunk's ability to recognize rolling logs and will cause Splunk to re-index the data.

• Was this topic useful?
• Post a comment