Disk Usage

Disk Usage

There are three ways to set limits on how Splunk uses disk space.

Set a minimum free disk space

The Splunk Server's web interface lets you set a minimum amount of disk space it will keep free on the disk where indexed data is stored. If the limit is reached, the server will pause indexing data until more space becomes available.

Two caveats for this setting:

• The Splunk Server will not clear any of its own disk space under this method. It will simply wait for more space to become available.
• Some events may be lost if they are not written to file during the paused period.

Web interface

• Click Admin in the upper left corner of the web interface.
• Click Server -> Settings
• Under the Basic Setting section, find the control labeled, "Pause indexing if free disk space falls below ___ MB"
• Enter your desired minimum free disk space in megabytes.
  • Click Save at the bottom of the page.

You will need to restart the server for the new setting to take effect.

Command line interface

# splunk set minfreemb 200000

# splunk restart

Configuration file

$SPLUNK_HOME/etc/myinstall/splunkd.xml

Edit the file $SPLUNK_HOME/etc/myinstall/splunkd.xml to modify the minFreeSpace// parameter. Set it to the number of megabytes you wish to keep free. The example below would configure Splunk to pause indexing whenever free disk space goes under 2,000 megabytes, and to start again when more space becomes available. Some events may be lost if they are not written to file during the paused period.

<processor name="diskusage" plugin="diskusageprocessor">
    <config>
        <minFreeSpace>2000</minFreeSpace>
        <pollingFrequency>100000</pollingFrequency>
    </config>
 </processor>

You will need to restart the server for the new setting to take effect.

Remove files beyond a certain size

$SPLUNK_HOME/etc/myinstall/pluginConfs/multiIndexer.xml

Find the entry in multiIndexer.xml for the main index. Set the variable maxTotalDataSizeMB to the number of megabytes beyond which the oldest indexed data should be erased. The example below would configure Splunk to cull old events from its index whenever it goes over 200,000 megabytes.

<database>
      <name>main</name>
      ....
      <maxTotalDataSizeMB>200000</maxTotalDataSizeMB> 
     ...
</database>

You will need to restart the server for the new setting to take effect. It will take some time, up to 30 or 40 minutes, for Splunk to move events out of the index to conform to the new policy, during which you may see high CPU usage.

Remove files beyond a certain age

$SPLUNK_HOME/etc/myinstall/pluginConfs/multiIndexer.xml

Find the entry in multiIndexer.xml for the main index. Set the variable frozenTimePeriodinSecs to the number of seconds after which indexed data should be erased. The example below would configure Splunk to cull old events from its index when they become more than 180 days old. The default value is approximately 6 years

<database>
      <name>main</name>
      ....
      <frozenTimePeriodInSecs>15552000</frozenTimePeriodInSecs> 
     ...
</database>

You will need to restart the server for the new setting to take effect.

Note Ensure your values are in the correct units. For a quick calculator, you can do basic unit conversions with Google:

Search Google for "50000 megabytes in gigabytes"

Recover free space

If you find that Splunk's default settings for disk management consume too much free space modify the $SPLUNK_HOME/etc/myinstall/pluginConfs/multiIndexer.xml file appropriately and restart the server. On startup. splunkd will read the new configuration and start moving events out of the index as appropriate. Depending on how many events are involved, this can take some time (30 minutes or more.) Splunk will continue to operate normally but you will notice a higher CPU load while it does the index maintenance. There is no longer a need to manually roll (!++cmd++::roll) to free up disk space.

How to archive old data

Based on your Disk Usage Policy Splunk will rotate old data out of the index to free up disk space. The last stage of a data's life span in Splunk is the frozen state. Splunk will definitely erase frozen index data. The coldToFrozenScript specified in the $SPLUNK_HOME/etc/myinstall/pluginConfs/multiIndexer.xml script gets run just before erasing them. The default value, echo $DIR, writes the name of the directory being retired, e.g. /opt/splunk/var/lib/splunk/defaultdb/colddb, to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log. If you want to keep retired data, use a cp command or a script here.

Find the entry in multiIndexer.xml for the index that you want to archive. Set the variable coldToFrozenScript to cp command or a script. The example below would configure Splunk to copy old events from its main index to a long-term storage location after 180 days.

  <databases>
    <database>
      <name>main</name>
      ...
      <frozenTimePeriodInSecs>15552000</frozenTimePeriodInSecs>
      ...
      <coldToFrozenScript>cp -r $DIR /LongTermParking/logdata/Splunk/ </coldToFrozenScript>
      ...
    </database>

• Was this topic useful?
• Post a comment