Splunk® Enterprise

Capacity Planning Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.0 is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Estimate your storage requirements

This topic describes how to estimate the size of your Splunk Enterprise index, so that you can plan your storage capacity requirements.

When Splunk Enterprise indexes your data, it creates two main types of files: the "rawdata" file that contains the original data in compressed form and the index files that point to this data. (It also creates a few metadata files, which don't consume much space.) With a little experimentation, you can estimate how much index disk space you will need for a given amount of incoming data.

Typically, the compressed rawdata file is 10% the size of the incoming, pre-indexed raw data. The associated index files range in size from approximately 10% to 110% of the rawdata file. The number of unique terms in the data affect this value.

Depending on the data's characteristics, you might want to tune your segmentation settings, as described in About segmentation in the Getting Data In Manual.

The best way to get an idea of your space needs is to experiment by indexing a representative sample of your data, and then checking the sizes of the resulting directories in $SPLUNK_HOME/var/lib/splunk/defaultdb.

On *nix systems, follow these steps

Once you've indexed your data sample:

1. Go to $SPLUNK_HOME/var/lib/splunk/defaultdb/db.

2. Run du -ch hot_v* and look at the last total line to see the size of the index.

On Windows systems, follow these steps

1. Download the du utility from Microsoft TechNet.

2. Extract du.exe from the downloaded ZIP file and place it into your %SYSTEMROOT% or %WINDIR% folder.

Note: You can also place it anywhere in your %PATH%.

3. Open a command prompt.

4. Once there, go to %SPLUNK_HOME%\var\lib\splunk\defaultdb\db.

5. Run del %TEMP%\du.txt & for /d %i in (hot_v*) do du -q -u %i\rawdata | findstr /b "Size:" >> %TEMP%\du.txt.

6. Open the %TEMP%\du.txt file. You will see Size: n, which is the size of each rawdata directory found.

7. Add these numbers together to find out how large the compressed persisted raw data is.

8. Next, run for /d %i in (hot_v*) do dir /s %i, the summary of which is the size of the index.

9. Add this number to the total persistent raw data number.

This is the total size of the index and associated data for the sample you have indexed. You can now use this to extrapolate the size requirements of your Splunk Enterprise index and rawdata directories over time.

Answers

Have questions? Visit Splunk Answers to see what questions and answers other Splunk users had about data sizing.

Last modified on 10 June, 2020
PREVIOUS
How Splunk Enterprise calculates disk storage
  NEXT
Distribute indexing and searching

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters