Automate archiving

Automate archiving

Set up Splunk to archive your data automatically as it ages. To do this, configure indexes.conf to call archiving scripts located in $SPLUNK_HOME/bin. Edit this file in $SPLUNK_HOME/etc/system/local/, or in your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work. Do not edit the copy in default.

Note: By default, Splunk deletes ALL frozen data. To avoid losing your data, you must specify a valid coldToFrozenScript in $SPLUNK_HOME/etc/system/local/indexes.conf (or your own custom app directory in $SPLUNK_HOME/etc/apps/).

Sign your archives

Splunk supports archive signing; configuring this allows you to verify integrity when you restore an archive.

Use Splunk's index aging policy to archive

Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which correspond to file directory locations. Data starts out in the hot database $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db_hot. Then, data moves through the warm database $SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database $SPLUNK_HOME/var/lib/splunk/defaultdb/colddb.

Finally, data reaches the frozen state. Splunk erases frozen index data once it is older than frozenTimePeriodinSecs in indexes.conf. The coldToFrozenScript (also specified in indexes.conf) runs just before the frozen data is erased. The default script simply writes the name of the directory being retired, e.g. /opt/splunk/var/lib/splunk/defaultdb/colddb, to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log.

Add the following to $SPLUNK_HOME/etc/system/local/indexes.conf:

[<index>]
coldToFrozenScript = <script> $DIR

• [<index>]
  • Specify which index to archive.
• coldToFrozenScript = <script>
  • Specify the archiving script to use by changing <script>.
  • Define <script> paths relative to $SPLUNK_HOME/bin.
  • $DIR is substituted with the path to the index data bucket that is being frozen. <script> should expect the path to be passed to it as an argument.
  • Splunk ships with two default archiving scripts that you can use.
  • Note: Rename and then modify these scripts to set the archive location for your installation. By default, the location is set to opt/tmp/myarchive.
  • compressedExport.sh: Export with tsidx files compressed as gz.
  • flatfileExport.sh: Export as a flat text file.

Note: Either rename the script you use or move it to another location (and specify that location in indexes.conf) to avoid having changes overwritten when you upgrade Splunk.
Note: These are example scripts and should not be applied to a Production instance without editing to suit your environment and extensive testing.

• Windows users use this notation: coldToFrozenScript = <script> "$DIR"
  • <script> can be directory path to either:
  • WindowsCompressedExport.bat (download the example script here).
  • WindowsFlatfileExport.bat (download the example script here).

Note: Either rename the script you use or move it to another location (and specify that location in indexes.conf) to avoid having changes overwritten when you upgrade Splunk.
Note: These are example scripts and should not be applied to a Production instance without editing to suit your environment and extensive testing.

• Was this topic useful?
• Post a comment