Archive indexed data
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Archive indexed data
Set up Splunk to archive your data automatically as it ages. To do this, configure indexes.conf to call archiving scripts located in
$SPLUNK_HOME/bin. Edit a copy of
$SPLUNK_HOME/etc/system/local/, or in your own custom application directory in
$SPLUNK_HOME/etc/apps/. Do not edit the copy in
$SPLUNK_HOME/etc/system/default. For information on configuration files and directory locations, see "About configuration files".
Caution: By default, Splunk deletes all frozen data. To avoid losing your data, you must specify a valid
For detailed information on data storage in Splunk, see "How Splunk stores indexes".
Sign your archives
Splunk supports archive signing; configuring this allows you to verify integrity when you restore an archive.
Use Splunk's index aging policy to archive
Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which correspond to file directory locations. Data starts out in the hot database, located as subdirectories under
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/. Then, data moves through the warm database, also located as subdirectories under
$SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database
Finally, data reaches the frozen state. Splunk erases frozen index data once it is older than
coldToFrozenScript (also specified in
indexes.conf) runs just before the frozen data is erased. The default script simply writes the name of the directory being erased to the log file
$SPLUNK_HOME/var/log/splunk/splunkd_stdout.log. If you want to archive frozen data rather than delete it, you'll need to substitute your own archiving script.
To substitute your own script, add the following stanza to
[<index>] coldToFrozenScript = <script>
Note the following:
<index>specifies which index to archive.
<script>specifies the archiving script.
- Define the
<$script>path relative to
$SPLUNK_HOME/bin. The script needs to be located in that directory or a subdirectory.
- Define the
Splunk ships with two archiving scripts in the
$SPLUNK_HOME/bin directory. You can modify these (or you can create your own):
compressedExport.sh: Export with tsidx files compressed as gz.
flatfileExport.sh: Export as a flat text file (not recommended for current performance and resource issues -- it can take a long time, and use a lot of ram, 2-3GB, while running).
Note: If using one of these scripts, modify it to specify the archive location for your installation. By default, the location is set to
opt/tmp/myarchive. Also, rename the script or move it to another location to avoid having changes overwritten when you upgrade Splunk. These are example scripts and should not be applied to a production instance without editing to suit your environment and testing extensively.
As a best practice, make sure the script completes as quickly as possible, so that Splunk doesn't end up waiting for the return indicator. For example, if you want to archive to a slow volume, set the script to copy the buckets to a temporary location on the same (fast) volume as the index. Then use a separate script, outside Splunk, to move the buckets from the temporary location to their destination on the slow volume.
For more information on the
coldToFrozen script, see "Configure archive signing".
Windows users should use the following notation:
[<index>] coldToFrozenScript = <script> "$DIR"
Note: Enclose the directory name with double quotes if it contains a space.
<script>, you can use one of these example scripts:
WindowsCompressedExport.bat. Download the example script here.
WindowsFlatfileExport.bat(not recommended for current performance and resource issues -- it can take a long time, and use a lot of ram, 2-3GB, while running). Download the example script here.
Note: Rename the script or move it to another location to avoid having changes overwritten when you upgrade Splunk. These are example scripts and should not be applied to a production instance without editing them to suit your environment and testing extensively.
The following configuration will archive main index frozen buckets in the directory
1. Place the
WindowsFlatfileExport.bat or </WindowsCompressedExport.bat</code> in
2. Change the parameter for "dest_base" in
[main] coldToFrozenScript = "C:\Program Files\Splunk\share\splunk\WindowsCompressedExport.bat" "$DIR"
4. Restart Splunk.