Archive indexed data

Archive indexed data

Set up Splunk to archive your data automatically as it ages. To do this, configure indexes.conf to call archiving scripts located in $SPLUNK_HOME/bin. Edit a copy of indexes.conf in $SPLUNK_HOME/etc/system/local/, or in your own custom application directory in $SPLUNK_HOME/etc/apps/. Do not edit the copy in $SPLUNK_HOME/etc/system/default. For information on configuration files and directory locations, see "About configuration files".

Caution: By default, Splunk deletes all frozen data. To avoid losing your data, you must specify a valid coldToFrozenScript in indexes.conf.

For detailed information on data storage in Splunk, see "How Splunk stores indexes".

Sign your archives

Splunk supports archive signing; configuring this allows you to verify integrity when you restore an archive.

Use Splunk's index aging policy to archive

Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which correspond to file directory locations. Data starts out in the hot database, located as subdirectories under $SPLUNK_HOME/var/lib/splunk/defaultdb/db/. Then, data moves through the warm database, also located as subdirectories under $SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database $SPLUNK_HOME/var/lib/splunk/defaultdb/colddb.

Finally, data reaches the frozen state. Splunk erases frozen index data once it is older than frozenTimePeriodinSecs in indexes.conf. The coldToFrozenScript (also specified in indexes.conf) runs just before the frozen data is erased. The default script simply writes the name of the directory being erased to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log. If you want to archive frozen data rather than delete it, you'll need to substitute your own archiving script.

To substitute your own script, add the following stanza to $SPLUNK_HOME/etc/system/local/indexes.conf:

[<index>]
coldToFrozenScript = <script>

Note the following:

• <index> specifies which index to archive.
• <script> specifies the archiving script.
  • Define the <$script> path relative to $SPLUNK_HOME/bin. The script needs to be located in that directory or a subdirectory.

Splunk ships with two archiving scripts in the $SPLUNK_HOME/bin directory. You can modify these (or you can create your own):

• compressedExport.sh: Export with tsidx files compressed as gz.
• flatfileExport.sh: Export as a flat text file (not recommended for current performance and resource issues -- it can take a long time, and use a lot of ram, 2-3GB, while running).

Note: If using one of these scripts, modify it to specify the archive location for your installation. By default, the location is set to opt/tmp/myarchive. Also, rename the script or move it to another location to avoid having changes overwritten when you upgrade Splunk. These are example scripts and should not be applied to a production instance without editing to suit your environment and testing extensively.

As a best practice, make sure the script completes as quickly as possible, so that Splunk doesn't end up waiting for the return indicator. For example, if you want to archive to a slow volume, set the script to copy the buckets to a temporary location on the same (fast) volume as the index. Then use a separate script, outside Splunk, to move the buckets from the temporary location to their destination on the slow volume.

For more information on the coldToFrozen script, see "Configure archive signing".

Windows Users

Windows users should use the following notation:

[<index>]
coldToFrozenScript = <script> "$DIR"

Note: Enclose the directory name with double quotes if it contains a space.

For <script>, you can use one of these example scripts:

• WindowsCompressedExport.bat. Download the example script here.
• WindowsFlatfileExport.bat (not recommended for current performance and resource issues -- it can take a long time, and use a lot of ram, 2-3GB, while running). Download the example script here.

Note: Rename the script or move it to another location to avoid having changes overwritten when you upgrade Splunk. These are example scripts and should not be applied to a production instance without editing them to suit your environment and testing extensively.

Examples

The following configuration will archive main index frozen buckets in the directory D:\MyArchive:

1. Place the WindowsFlatfileExport.bat or </WindowsCompressedExport.bat</code> in %SPLUNK_HOME%\share\splunk.

2. Change the parameter for "dest_base" in WindowsFlatfileExport.bat or WindowsCompressedExport.bat.

set dest_base=D:\MyArchive

3. In %SPLUNK_HOME%\etc\system\local\inputs.conf specify:

[main]
coldToFrozenScript = "C:\Program Files\Splunk\share\splunk\WindowsCompressedExport.bat" "$DIR"

4. Restart Splunk.

• Was this topic useful?
• Post a comment