Configure archive signing
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Contents
Configure archive signing
Use archive signing to sign your Splunk data as it is archived (moved from colddb to frozen). This lets you verify integrity when you restore an archive. Configure the size of the slice by setting your archiving policies.
How archive signing works
Data is archived from the colddb to frozen when either:
- the size of your index reaches a maximum that you specify.
- data in your index reaches a certain age.
Specify archiving policies to define how your data is archived.
Splunk ships with two standard scripts, but you may use your own. Data is archived from the colddb to frozen with a coldToFrozen script that you specify. The coldToFrozen script tells Splunk how to format your data (gz, raw, etc..), and where to archive it. Archive signing happens after the coldToFrozen script formats your data into its archive format, and then the data is moved to the archive location that you specified according to your archive policy.
An archive signature is a hash signature of all the data in the data slice.
To invoke archive signing, use the standalone signtool utility. Add signtool -s <path_of_archive> to the coldToFrozen script anywhere after the data formatting lines, but before the lines that copy your data to your archive. See the section below on configuring coldToFrozen scripts.
Verify archived data signatures
Splunk verifies archived data signatures automatically upon restoring. You can verify signatures manually by using signtool -v <path_to_archive>.
Configure coldToFrozen scripts
Configure any coldToFrozen script by adding a line for the signtool utility.
Note: If you use a standard Splunk archiving script, either rename the script or move it to another location (and specify that location in indexes.conf) to avoid having changes overwritten when you upgrade Splunk.
Standard Splunk archiving scripts
The two standard archiving scripts that are shipped with Splunk are shown below with archive signing.
Splunk's two archiving scripts are:
compressedExport.sh
This script exports files with the tsidx files compressed as gz.
#!/bin/sh gzip $1/*.tsidx signtool -s <path_to_archive> # replace this with the path to the archive you want signed cp -r $1 /opt/tmp/myarchive #replace this with your archive directory
flatfileExport.sh
This script exports each splunk 'source' event stream as a flat text file.
#!/bin/sh
exporttool $1 ${1}/index.export
rm -rf ${1}/*.data
rm -rf ${1}/rawdata
rm -rf ${1}/*.tsidx
signtool -s <path_to_archive> # replace this with the path to the archive you want signed
cp -r $1 /opt/tmp/myarchive #replace this with your archive directory
Note: flatfileExport.sh is currently not recommended for performance and resource issues we hope to address in the future. It can take a long time (tens of minutes to hours), and use a lot of ram, 2-3GB, while running.
Your own custom scripts
You can also use your own scripts to move data from cold to frozen.
Sign or verify your data slices
Use signtool, located in $SPLUNK_HOME/bin, to sign data slices as they are archived or verify the integrity of an archive.
Syntax
To sign:
signtool [- s | -- sign] archive_path
To verify:
signtool [-v | --verify] archive_path
This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 View the Article History for its revisions.
Comments
Best practice :
Splunk may be slowed by calling the coldToFrozen script, waiting for the return signal before manipulating the buckets.
you should make your script the faster possible,
By example : if you want to archive files to slow volumes.
- use the script to move the file out of splunk to a temp folder on the same volume (faster) then return success.
- use another script outside of splunk to do your file archive from the temp folder to your final storage.
Added text for ykherian's comment here:
http://www.splunk.com/base/Documentation/4.1.6/Admin/Automatearchiving