Archive cold buckets to frozen
Data is aged in Splunk Enterprise locally on every indexer. The way you configure your index determines the data size or age at which the data to moves to the next state (hot, warm, cold, frozen) and is ultimately deleted.
Once you configure a Splunk index to archive data with Hunk, the archiving of Splunk indexes runs on a schedule that is determined is globally on the Splunk search head.
When both processes occur, a disconnect can occur between the indexer's local processes and the Hunk archiving process. As a result, the indexers can delete a bucket before it's been archived by Hunk's archiving process.
To avoid buckets from being deleted you can use the the
coldToFrozen.sh script on the local indexer process. This script shifts the responsibility of deleting buckets from the indexer to the global Hunk archiving process, so only use this script for indexes that are being archived by Hunk.
How it works:
1. The script moves buckets that are about to be frozen away from Splunk so that it is not deleted.
2. Hunk archives the data.
3. Once the data is archived, Hunk deletes the data.
coldToFrozen.sh Script as a fallback and not your primary hook for archiving. This script buys you more time when either your system is receiving data faster than normal, or when the archiving storage layer is down, so that you'll have more time to archive bucket. To facilitate this further, for each archive index you can set your
vix.output.buckets.older.than = seconds as low as possible, so that buckets are archived as quickly as possible.
Configure the cold bucket to roll to frozen
Note the following if you are using Hunk's
- The script must be installed on each stanza which configures an index that is being archived by Hunk.
- All the search peers to the Hunk search head must have the script installed as well. You can do each peer manually or use the deployer for search head clusters. See Configure search head clustering.
- The script must be removed from any index for which you disable archiving by Hunk. Otherwise, the script will continue to run and the data will overfill your existing disk space because there is no archive to receive that data (and thus it will not get deleted).
- Do not add this script to any indexers that are not configured to archive data by Hunk.
For each Splunk index, use the provided script located in
$SPLUNK_HOME/etc/apps/splunk_archiver/bin/ and named
coldToFrozen.sh to archive your cold data to frozen. This path may very depending upon your configuration path. For example:
[<index name>] coldToFrozenScript = "$SPLUNK_HOME/etc/apps/splunk_archiver/bin/coldToFrozen.sh"
Archiving Splunk Enterprise indexes to Amazon S3
Configure Hunk to read Hadoop Archive (HAR) files
This documentation applies to the following versions of Hunk®(Legacy): 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11
Feedback submitted, thanks!