Splunk® Enterprise

Search Manual

Download manual as PDF

Download topic as PDF

Calculate sizes of dynamic fields

This search determines which fields in your events, without any prior knowledge of field names and number of events, consume the most disk space.

Scenario

index=_internal earliest=-15m latest=now
| fieldsummary 
| rex field=values max_match=0 "value\":\"(?<values>[^\"]*)\","
| mvexpand values 
| eval bytes=len(values)
| rex field=field "^(?!date|punct|host|hostip|index|linecount|source|sourcetype|timeendpos|timestartpos|splunk_server)(?<FieldName>.*)"
| stats count sum(bytes) as SumOfBytesInField values(values) as Values max(bytes) as MaxFieldLengthInBytes by FieldName
| rename count as NumberOfValuesPerField
| eventstats sum(NumberOfValuesPerField) as TotalEvents sum(SumOfBytesInField) as TotalBytes
| eval PercentageOfTotalEvents=round(NumberOfValuesPerField/TotalEvents*100,2)
| eval PercentageOfTotalBytes=round(SumOfBytesInField/TotalBytes*100,2)
| eval ConsumedMB=SumOfBytesInField/1024/1024
| eval TotalMB=TotalBytes/1024/1024
| table FieldName NumberOfValuesPerField SumOfBytesInField ConsumedMB PercentageOfTotalBytes PercentageOfTotalEvents
| addcoltotals labelfield=FieldName label=Totals
| sort - PercentageOfTotalEvents


CalculateSizeofFields report.png

Walkthough

1. The example begins with a search to retrieve all events in index=_internal within the last 15 minutes.

index=_internal earliest=-15m latest=now

Note: You can replace this with any search string and timerange.

2. Next, the fieldsummary command creates a summary of all the fields in previously retrieved events.

... | fieldsummary

This looks something like this:

Fieldsummary example1.png

3. The values of each field are extracted with a regex into a multivalue field, values, and then expanded. The length of each value is calculated in bytes.

| rex field=values max_match=0 "value\":\"(?<values>[^\"]*)\","
| mvexpand values 
| eval bytes=len(values)

4. The values of the field are extracted with another regex, with some exceptions.

| rex field=field "^(?!date|punct|host|hostip|index|linecount|source|sourcetype|timeendpos|timestartpos|splunk_server)(?<FieldName>.*)"

5.

| stats count sum(bytes) as SumOfBytesInField values(values) as Values max(bytes) as MaxFieldLengthInBytes by FieldName
| rename count as NumberOfValuesPerField

6.

| eventstats sum(NumberOfValuesPerField) as TotalEvents sum(SumOfBytesInField) as TotalBytes

7.

| eval PercentageOfTotalEvents=round(NumberOfValuesPerField/TotalEvents*100,2)
| eval PercentageOfTotalBytes=round(SumOfBytesInField/TotalBytes*100,2)
| eval ConsumedMB=SumOfBytesInField/1024/1024
| eval TotalMB=TotalBytes/1024/1024

8.

| table FieldName NumberOfValuesPerField SumOfBytesInField ConsumedMB PercentageOfTotalBytes PercentageOfTotalEvents
| addcoltotals labelfield=FieldName label=Totals
| sort - PercentageOfTotalEvents
PREVIOUS
Monitor and alert on Windows disk usage
 

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 7.0.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters