Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Overview of search-time field extraction

This topic provides a brief overview of Splunk Web field extraction methods.

As you use Splunk Enterprise, you will encounter situations that require the creation of new fields that will be additions to the set of fields that Splunk Enterprise automatically extracts for you at index time and search time.

As a knowledge manager, you'll be managing field extractions for the rest of your team. In many cases you'll be defining fields that Splunk Enterprise has not identified on its own, in effort to make your event data more useful for searches, reports, and dashboards. However, you may also want to define field extractions as part of an event data normalization strategy, where you redefine existing fields and create new ones in an effort to reduce redundancies and increase the overall usability of the fields available to other Splunk Enterprise users on their team.

For more information about normalizing your fields, see the product documentation for the Common Information Model Add-on. You can download the CIM Add-on here.

If you find that you need to create additional search-time field extractions, you have a number of ways to go about it. Splunk Web provides a variety of search-time field extraction methods. The search language also enables you to create temporary field extractions. And you can always add and maintain field extractions by way of configuration file edits.

For a detailed discussion of search-time field addition using methods based in Splunk Web, see "About fields" in this manual. We'll just summarize the methods in this subtopic and provide links to topics with in-depth discussions and examples.

Use interactive field extraction to create new fields

You can create custom fields dynamically using the interactive field extractor in Splunk Web. IFX enables you to quickly turn any search into a field extracting regular expression. You use IFX on the local indexer. For more information about using IFX, see "Extract fields interactively with IFX" in this manual.

Note: IFX is especially useful if you are not familiar with regular expression syntax and usage, because it will generate field extraction regexes for you (and enable you to test them).

To access IFX, run a search and then select Extract fields from the dropdown that appears beneath timestamps in the field results. IFX enables you to extract only one field at a time (although you can edit the regex it generates later to extract multiple fields).

Use Splunk Web to add and maintain field extractions

You can use the Field extractions and Field transformations pages in Splunk Web to review, edit, and create extracted fields.

The Field extractions page

The Field extractions page shows you the search-time field extractions in props.conf. You can edit existing extractions and create new ones. The Field extractions page allows you to review, update, and create field extractions. You can use it to create and manage both basic "inline" search-time extractions (extractions that are defined entirely within props.conf) and more advanced search-time extractions that reference a field transformation component in transforms.conf. You can define field transformations through the Field transformations page (see below).

In Splunk Web, you navigate to the Field extractions page by selecting Settings > Fields > Field extractions.

For more information, see "Use the Field extractions page in Splunk Web".

The Field transformations page

You can also use Splunk Web to create more complex search-time field extractions that involve a transform component in transforms.conf. To do this, you couple an extraction from the Field extractions page with a field transform on the Field transformations page.

The Field transformations page displays search-time field transforms that have been defined in transforms.conf. Field transforms work with extractions set up in props.conf to enable advanced field extractions. With transforms, you can define field extractions that

  • Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform for multiple field extractions).
  • Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction).
  • Use a regular expression to extract fields from the values of another field (also referred to as a "source key").

In Splunk Web, you navigate to the Field transformations page by selecting Settings > Fields > Field transformations.

For more information, see "Use the Field transformations page in Splunk Web".

Configure field extractions in props.conf and transforms.conf

You can also create and maintain field extractions by making edits directly to props.conf and transforms.conf. If this sounds like your kind of thing--and it may be, especially if you are an old-timey Splunk Enterprise user, or just prefer working at the configuration file level of things, you can find all the details in "Create and maintain search-time extractions through configuration files," in this manual.

It's important to note that the configuration files do enable you to do more things with search-time field extractions than Splunk Web currently does. For example, with the config files you can set up:

  • Delimiter-based field extractions.
  • Extractions for multivalue fields.
  • Extractions of fields with names that begin with numbers or underscores (normally not allowed unless key cleaning is disabled).
  • Formatting of extracted fields.

Use search commands to create field extractions

Splunk Enterprise provides a variety of search commands that facilitate the extraction of fields in different ways. Here's a list of these commands:

  • The rex search command performs field extractions using a Perl regular expression with named groups named groups that you include in the search string.
  • The extract (or kv, for "key/value") search command extracts field/value pairs from search results. If you use extract without specifying any arguments, Splunk Enterprise extracts fields using field extraction stanzas that have been added to props.conf. You can use extract to test any field extractions that you plan to add manually through conf files, to see if they extract field/value information as expected.
  • Use multikv to extract field/value pairs from multiline, tabular-formatted events. It creates a new event for each table row and derives field names from the table title.
  • xmlkv enables you to extract field/value pairs from xml-formatted event data, such as transactions from webpages.
  • kvform extracts field/value pairs from events based on predefined form templates that describe how the values should be extracted. These templates are stored in $SPLUNK_HOME/etc/system/form/, or your own custom app directory in $SPLUNK_HOME/etc/apps/.../form. For example, if form=sales_order, Splunk Enterprise matches all of the events it processes against that form in an effort to extract values. When Splunk Enterprise encounters an event with error_code=404, it looks for a sales_order.form file.

For details about how these commands are used, along with examples, see either the Search Reference or the "Extract fields with search commands" topic in the Search Manual.

Last modified on 03 November, 2014
About fields
Extract fields interactively with IFX

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters