When Splunk software extracts fields
Field extraction at index time
Caution: Do not add custom fields to the set of default fields that Splunk software extracts and indexes at index time. Adding to this list of fields can slow indexing performance and search times, because each indexed field increases the size of the searchable index. Indexed fields are also less flexible, because whenever you make changes to your set of indexed fields, you must re-index your entire dataset. See Index time versus search time in the Managing Indexers and Clusters manual.
Field extraction at search time
When field discovery is enabled, Splunk software:
- Identifies and extracts the first 100 fields that it finds in the event data that match obvious
key=valuepairs. This 100 field limit is a default that you can modify by editing the
limits.conf, if you have Splunk Enterprise.
- Extracts any field explicitly mentioned in the search that it might otherwise have found though automatic extraction, but is not among the first 100 fields identified.
- Performs custom field extractions that you have defined, either through the Field Extractor, the Extracted Fields page in Settings, configuration file edits, or search commands such as
When field discovery is disabled, Splunk software extracts:
- Any field explicitly mentioned in the search.
- The default and indexed fields mentioned above.
- Any custom field extraction that has the
CAN_OPTIMIZEparameter set to true in
Splunk software discovers fields other than default fields and fields explicitly mentioned in the search string only when you:
- Run a non-transforming search in the Smart search mode.
- Run any search in the Verbose search mode.
See Set search mode to adjust your search experience in the Search Manual.
For an explanation of search time and index time, see Index time versus search time in the Managing Indexers and Clusters manual.
Example of automatic field extraction
This is an example of how Splunk software automatically extracts fields without user help, as opposed to custom field extractions, which follow event-extraction rules that you define.
Say you search on
sourcetype, a default field that Splunk software extracts for every event at index time. If your search is
for the past 24 hours, Splunk software returns every event with a sourcetype of
veeblefetzer in that time range. From this set of events, Splunk software extracts the first 100 fields that it can identify on its own. And it performs extractions of custom fields, based on configuration files. All of these fields appear in the fields sidebar when the search is complete.
Now, if a name/value combination like
userlogin=fail appears for the first time 25,000 events into the search, and
userlogin isn't among the set of custom fields that you've preconfigured, it likely is not among the first 100 fields that Splunk software finds on its own.
However, if you change your search to
then Splunk software finds and returns all events including both the
userlogin field and a
sourcetype value of
veeblefetzer. It will be available in the field sidebar along with the other fields extracted for this search.
Use default fields
About regular expressions with field extractions
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 8.0.0, 8.0.1