Set retirement policy

Set retirement policy

You can set retirement and archiving policy by controlling the size of indexes or the age of data in the indexes. Splunk indexes go through four stages:

• Hot - open for writing. Only one of these for each index. Searchable.
• Warm - data rolled from hot. There are many warm indexes. Searchable.
• Cold - data rolled from warm. There are many cold indexes. Searched only when the search specifies a time range included in these files.
• Frozen - data rolled from cold. Eligible for deletion.

Caution: All index locations must be writable.

The size, location and age of these files are controlled by indexes.conf. Before modifying any configuration file, read about configuration files.

Remove files beyond a certain size

Find this entry in indexes.conf and set it to it new value (in megabytes)

maxTotalDataSizeMB = <non-negative number> (500000)
   * The maximum size of an index. If an index grows bigger than this the oldest data is frozen out.

Example:

[main]
maxTotalDataSizeMB = 2500000

You will need to restart the server for the new setting to take effect. It may take up to 40 minutes for Splunk to move events out of the index to conform to the new policy, during which you may see high CPU usage.

Note: ensure your values are in the correct units. For a quick calculator, you can do basic unit conversions with Google:

Search Google for "50000 megabytes in gigabytes"

Remove files beyond a certain age

Set the variable frozenTimePeriodinSecs in indexes.conf to the number of seconds after which indexed data should be erased. The example below configures Splunk to cull old events from its index when they become more than 180 days old. The default value is approximately 6 years.

[main]
frozenTimePeriodInSecs = 15552000

You will need to restart the server for the new setting to take effect.

Note: ensure your values are in the correct units. For a quick calculator, you can do basic unit conversions with Google:

Search Google for "15552000 seconds in days"

Note: Splunk ages out data by buckets. Specifically, when the most recent data in a particular bucket reaches the configured age, the entire bucket is rolled. If you are indexing a large volume of events, bucket size is less a concern for retirement policy because they fill quickly. You can adjust the bucket size by setting maxDataSize in indexes.conf smaller so they roll faster. But more, smaller buckets take more time to search than fewer, larger buckets. To get the results you are after, you will have to experiment a bit for the right size. Due to the structure of the index, there isn't a direct relationship between time and data size.

• Was this topic useful?
• Post a comment