Overview of search-time field extraction

Overview of search-time field extraction

This topic provides a brief overview of Splunk Web field extraction methods.

As you use Splunk, you will encounter situations that require the creation of new fields that will be additions to the set of fields that Splunk automatically extracts for you at index time and search time.

As a knowledge manager, you'll be managing field extractions for the rest of your team. In many cases you'll be defining fields that Splunk has not identified on its own, in effort to make your event data more useful for searches, reports, and dashboards. However, you may also want to define field extractions as part of an event data normalizaton strategy, where you redefine existing fields and create new ones in an effort to reduce redundancies and increase the overall usability of the fields available to other Splunk users on their team. (For more information, see "Understand and use the Common Information Model," in this manual.)

If you find that you need to create additional search-time field extractions, you have a number of ways to go about it. Splunk Web provides a variety of search-time field extraction methods. The search language also enables you to create temporary field extractions. And you can always add and maintain field extractions by way of configuration file edits.

For a detailed discussion of search-time field addition using methods based in Splunk Web, see "Extract and add new fields" in the User manual. We'll just summarize the methods in this subtopic and provide links to topics with in-depth discussions and examples.

Use interactive field extraction to create new fields

You can create custom fields dynamically using the interactive field extractror (IFX) in Splunk Web. IFX enables you to quickly turn any search into a field extracting regular expression. You use IFX on the local indexer. For more information about using IFX, see "Extract fields interactively in Splunk Web" in the User manual.

Note: IFX is especially useful if you are not familiar with regular expression syntax and usage, because it will generate field extraction regexes for you (and enable you to test them).

To access IFX, run a search and then select "Extract fields" from the dropdown that appears beneath timestamps in the field results. IFX enables you to extract only one field at a time (although you can edit the regex it generates later to extract multiple fields).

Use Splunk Manager to add and maintain field extractions

You can use the Field extractions and Field transformations pages in Splunk Manager to review, edit, and create extracted fields.

The Field extractions page

The Field extractions page shows you the search-time field extractions in props.conf. You can edit existing extractions and create new ones. The Field extractions page allows you to review, update, and create field extractions. You can use it to create and manage both basic "inline" search-time extractions (extractions that are defined entirely within props.conf) and more advanced search-time extractions that reference a field transformation component in transforms.conf. You can define field transformations in Manager through the Field transformations page (see below).

In Splunk Web, you navigate to the Field extractions page by selecting Manager > Fields > Field extractions.

For more information, see "Use the Field extractions page in Manager".

The Field transformations page

You can also use Manager to create more complex search-time field extractions that involve a transform component in transforms.conf. To do this, you couple an extraction from the Field extractions page with a field transform on the Field transformations page.

The Field transformations page displays search-time field transforms that have been defined in transforms.conf. Field transforms work with extractions set up in props.conf to enable advanced field extractions. With transforms, you can define field extractions that

• Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform for multiple field extractions).
• Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction).
• Use a regular expression to extract fields from the values of another field (also referred to as a "source key").

In Splunk Web, you navigate to the Field transformations page by selecting Manager > Fields > Field transformations.

For more information, see "Use the Field transformations page in Manager".

Configure field extractions in props.conf and transforms.conf

You can also create and maintain field extractions by making edits directly to props.conf and transforms.conf. If this sounds like your kind of thing--and it may be, especially if you are an old-timey Splunk user, or just prefer working at the configuration file level of things, you can find all the details in "Create and maintain search-time extractions through configuration files," in this manual.

It's important to note that the configuration files do enable you to do more things with search-time field extractions than Manager currently does. For example with the config files you can you can set up:

• Delimiter-based field extractions.
• Extractions for multivalue fields.
• Extractions of fields with names that begin with numbers or underscores (normally not allowed unless key cleaning is disabled).
• Formatting of extracted fields.

Use search commands to create field extractions

Splunk provides a variety of search commands that facilitate the extraction of fields in different ways. Here's a list of these commands:

• The rex search command performs field extractions using a Perl regular expression with named groups named groups that you include in the search string.
• The extract (or kv, for "key/value") search command extracts field/value pairs from search results. If you use extract without specifying any arguments, Splunk extracts fields using field extraction stanzas that have been added to props.conf. You can use extract to test any field extractions that you plan to add manually through conf files, to see if they extract field/value information as expected.
• Use multikv to extract field/value pairs from multiline, tabular-formatted events. It creates a new event for each table row and derives field names from the table title.
• xmlkv enables you to extract field/value pairs from xml-formatted event data, such as transactions from webpages.
• kvform extracts field/value pairs from events based on predefined form templates that describe how the values should be extracted. These templates are stored in $SPLUNK_HOME/etc/system/form/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.../form. For example, if form=sales_order, Splunk matches all of the events it processes against that form in an effort to extract values. When Splunk encounters an event with error_code=404, it looks for a sales_order.form file.

For details about how these commands are used, along with examples, see either the Search Reference or the "Extract and add new fields" topic in the User manual.

• Was this topic useful?
• Post a comment