Hunk User Manual

Archive cold buckets to frozen

Data is aged in Splunk Enterprise locally on every indexer. The way you configure your index determines the data size or age at which the data to moves to the next state (hot, warm, cold, frozen) and is ultimately deleted.

Once you configure a Splunk index to archive data with Hunk, the archiving of Splunk indexes runs on a schedule that is determined is globally on the Splunk search head.

When both processes occur, a disconnect can occur between the indexer's local processes and the Hunk archiving process. As a result, the indexers can delete a bucket before it's been archived by Hunk's archiving process.

To avoid buckets from being deleted you can use the the splunk_archiver app script on the local indexer process. This script shifts the responsibility of deleting buckets from the indexer to the global Hunk archiving process, so only use this script for indexes that are being archived by Hunk.

How it works:

1. The script moves buckets that are about to be frozen away from Splunk so that it is not deleted.

2. Hunk archives the data.

3. Once the data is archived, Hunk deletes the data.

Consider the Script as a fallback and not your primary hook for archiving. This script buys you more time when either your system is receiving data faster than normal, or when the archiving storage layer is down, so that you'll have more time to archive bucket. To facilitate this further, for each archive index you can set your vix.output.buckets.older.than = seconds as low as possible, so that buckets are archived as quickly as possible.

Configure the cold bucket to roll to frozen

Note the following if you are using Hunk's coldToFrozenSh.script:

  • The script must be installed on each stanza which configures an index that is being archived by Hunk.
  • All the search peers to the Hunk search head must have the script installed as well. You can do each peer manually or use the deployer for search head clusters. See Configure search head clustering.
  • The script must be removed from any index for which you disable archiving by Hunk. Otherwise, the script will continue to run and the data will overfill your existing disk space because there is no archive to receive that data (and thus it will not get deleted).
  • Do not add this script to any indexers that are not configured to archive data by Hunk.

For each Splunk index, use the provided script located in $SPLUNK_HOME/etc/apps/splunk_archiver/bin/ and named to archive your cold data to frozen. This path may very depending upon your configuration path. For example:

[<index name>]
coldToFrozenScript = "$SPLUNK_HOME/etc/apps/splunk_archiver/bin/"
Last modified on 03 December, 2015
Archiving Splunk Enterprise indexes to Amazon S3   Configure Hunk to read Hadoop Archive (HAR) files

This documentation applies to the following versions of Hunk®(Legacy): 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11

Was this topic useful?

You must be logged into in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters