Documentation > Splunk > Admin Manual > Automate archiving

Admin Manual

 


How Splunk Works
Getting Started
Data Inputs
Data Distribution
Indexing
Timestamps
Fields
Hosts
Source Types
Event Types
Transaction Types
Search
Distributed Search
Security
Data Management
Deployment Server
Performance Tuning
Configuration Files
Applications
Reference
Troubleshooting

Automate archiving

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Contents

Automate archiving

Set up Splunk to archive your data automatically. To do this, configure indexes.conf to call archiving scripts located in $SPLUNK_HOME/bin.


Use Splunk's index aging policy to archive

Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which also correspond to file directory locations. Data starts out in the hot database $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db_hot. Then, data moves through the warm database -- a new directory in $SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database $SPLUNK_HOME/var/lib/splunk/defaultdb/colddb.


Finally, data reaches the frozen state. Splunk erases frozen index data once frozenTimePeriodinSecs in indexes.conf is reached. The coldToFrozenScript value specified in indexes.conf runs just before the frozen data is erased. The default script simply writes the name of the directory being retired, e.g. /opt/splunk/var/lib/splunk/defaultdb/colddb, to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log.


From $SPLUNK_HOME/etc/bundles/local/indexes.conf: 


coldToFrozenScript = echo $DIR

Note: coldToFrozenScript must take two arguments -- the script or command and then $DIR.


Archiving scripts

If you want to keep retired data, change coldToFrozenScript to specify a script to archive your data. Splunk ships with two archiving scripts in $SPLUNK_HOME/bin. Use one of these, or create your own and place it in $SPLUNK_HOME/bin.


Note: Modify these scripts to set the archive location for your installation. By default, the location is set to opt/tmp/myarchive.


Splunk's two archiving scripts are:


compressedExport.sh

This script exports files with the tsidx files compressed as gz. 


#!/bin/sh
gzip $1/*.tsidx
cp -r $1 /opt/tmp/myarchive  #replace this with your archive directory

flatfileExport.sh

This script exports files as a flat text file. 


#!/bin/sh
exporttool $1 ${1}/index.export meta::all
rm -rf ${1}/*.data
rm -rf ${1}/rawdata
rm -rf ${1}/*.tsidx
cp -r $1 /opt/tmp/myarchive  #replace this with your archive directory
Retrieved from "http://docs.splunk.com/Documentation/Splunk/3.2.5/Admin/AutomateArchiving"

This documentation applies to the following versions of Splunk: 3.2 , 3.2.1 , 3.2.2 , 3.2.3 , 3.2.4 , 3.2.5 , 3.2.6 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!