When Splunk software extracts fields

Fields are extracted at index time and again at search time. After you run a search, fields extracted for that search are listed in the fields sidebar.

Field extraction at index time

At index time, Splunk software extracts a small set of default fields for each event, including host, source, and sourcetype. Default fields are common to all events. See Use default fields.

Splunk software can also extract custom indexed fields at index time. These are fields that you have explicitly configured for index-time extraction.

Caution: Do not add custom fields to the set of default fields that Splunk software extracts and indexes at index time. Adding to this list of fields can slow indexing performance and search times, because each indexed field increases the size of the searchable index. Indexed fields are also less flexible, because whenever you make changes to your set of indexed fields, you must re-index your entire dataset. See Index time versus search time in the Managing Indexers and Clusters manual.

Field extraction at search time

At search time, Splunk software can extract additional fields, depending on its Search Mode setting and whether that setting enables field discovery given the type of search being run.

When field discovery is enabled, Splunk software:

• Identifies and extracts the first 100 fields that it finds in the event data that match obvious key=value pairs. This 100 field limit is a default that you can modify by editing the [kv] stanza in limits.conf, if you have Splunk Enterprise.
• Extracts any field explicitly mentioned in the search that it might otherwise have found through automatic extraction, but is not among the first 100 fields identified.
• Performs custom field extractions that you have defined, either through the Field Extractor, the Extracted Fields page in Settings, configuration file edits, or search commands such as rex.

When field discovery is disabled, Splunk software extracts:

• Any field explicitly mentioned in the search.
• The default and indexed fields mentioned above.
• Any custom field extraction that has the CAN_OPTIMIZE parameter set to true in transforms.conf.

Splunk software discovers fields other than default fields and fields explicitly mentioned in the search string only when you:

• Run a non-transforming search in the Smart search mode.
• Run any search in the Verbose search mode.

See Set search mode to adjust your search experience in the Search Manual.

For an explanation of search time and index time, see Index time versus search time in the Managing Indexers and Clusters manual.

Example of automatic field extraction

This is an example of how Splunk software automatically extracts fields without user help, as opposed to custom field extractions, which follow event-extraction rules that you define.

Say you search on sourcetype, a default field that Splunk software extracts for every event at index time. If your search is

sourcetype=veeblefetzer

for the past 24 hours, Splunk software returns every event with a sourcetype of veeblefetzer in that time range. From this set of events, Splunk software extracts the first 100 fields that it can identify on its own. And it performs extractions of custom fields, based on configuration files. All of these fields appear in the fields sidebar when the search is complete.

Now, if a name/value combination like userlogin=fail appears for the first time 25,000 events into the search, and userlogin isn't among the set of custom fields that you've preconfigured, it likely is not among the first 100 fields that Splunk software finds on its own.

However, if you change your search to

sourcetype=veeblefetzer userlogin=*

then Splunk software finds and returns all events including both the userlogin field and a sourcetype value of veeblefetzer. It will be available in the field sidebar along with the other fields extracted for this search.