Estimate your storage requirements
This topic describes how to estimate the size of your Splunk index, so that you can plan your storage capacity requirements.
When Splunk indexes your data, it creates two main types of files: the rawdata file containing the original data in compressed form and the index files that point to this data. (It also creates a few metadata files, which don't consume much space.) With a little experimentation, you can estimate how much index disk space you will need for a given amount of incoming data.
Typically, the compressed rawdata file is approximately 10% the size of the incoming, pre-indexed raw data. The associated index files range in size from approximately 10% to 110% of the rawdata file. This value is affected strongly by the number of unique terms in the data. Depending on the data's characteristics, you might want to tune your segmentation settings, as described in "About segmentation".
The best way to get an idea of your space needs is to experiment by indexing a representative sample of your data, and then checking the sizes of the resulting directories in
On *nix systems, follow these steps
Once you've indexed your sample:
1. Go to
du -shc hot_v*/rawdata to determine how large the compressed persisted raw data is.
This is the persisted data to which the items in the index point. Typically, this file's size is about 10% of the size of the sample data set you indexed.
du -ch hot_v* and look at the last
total line to see the size of the index.
4. Add the values you get together.
On Windows systems, follow these steps
1. Download the du utility from Microsoft TechNet.
du.exe from the downloaded ZIP file and place it into your
Note: You can also place it anywhere in your
3. Open a command prompt.
4. Once there, go to
del %TEMP%\du.txt & for /d %i in (hot_v*) do du -q -u %i\rawdata | findstr /b "Size:" >> %TEMP%\du.txt.
6. Open the
%TEMP%\du.txt file. You will see
Size: n, which is the size of each
rawdata directory found.
7. Add these numbers together to find out how large the compressed persisted raw data is.
8. Next, run
for /d %i in (hot_v*) do dir /s %i, the summary of which is the size of the index.
9. Add this number to the total persistent raw data number.
This is the total size of the index and associated data for the sample you have indexed. You can now use this to extrapolate the size requirements of your Splunk index and
rawdata directories over time.
Have questions? Visit Splunk Answers to see what questions and answers other Splunk users had about data sizing.
High availability reference architecture
Splunk architecture and processes
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7