Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Use the Field extractions page

Use the Field extractions page in Settings to manage search-time field extractions. There are three methods by which you can add search-time field extractions. You can:

  • Use the field extractor to create extractions. This method is relatively easy and does not require you to understand how regular expressions work.
  • Make direct edits to props.conf, if you have Splunk Enterprise.
  • Add new field extractions with the Field extractions page (see below).

The Field extractions page enables you to:

  • Review the overall set of search-time extractions that you have created or which your permissions enable you to see, for all Apps in your Splunk deployment.
  • Create new search-time field extractions.
  • Update permissions for field extractions. Field extractions created through the field extractor and the Field extractions page are initially only available to their creators until they are shared with others.
  • Delete field extractions, if your app-level permissions enable you to do so, and if they are not default extractions that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.

If you have "write" permissions for a particular search-time field extraction, the Field extractions page enables you to:

  • Update its regular expression, if it is an inline transaction.
  • Add or delete named extractions that have been defined in transforms.conf or the Field transactions page in Splunk Web, if it uses transactions.

Note: You cannot manage index-time field extractions in Splunk Web. We don't recommend that you change your set of index-time field extractions, but if you find that you must do so, you have to modify your props.conf and transforms.conf configuration files manually. For more information about index-time field extraction configuration, see "Configure index-time field extractions" in the Getting Data In Manual.

Navigate to the Field extractions page by selecting Settings > Fields > Field extractions.

Review search-time field extractions in Splunk Web

To better understand how the Field extractions page displays your field extraction, it helps to understand how field extractions are set up in your props.conf and transforms.conf files.

Field extractions can be set up entirely in props.conf, in which case they are identified on the Field extractions page as inline field extractions. But some field extractions include a transforms.conf component, these types of field extractions are called transform field extraction. To create/edit that component of the field extraction via Splunk Web, you use the Field Transforms page in Splunk Web.

For more information about transforms and the Field Transforms page, see "Manage field transforms" in this manual.

For more information about field extraction setup directly in the props.conf and transforms.conf files see "Create and maintain search-time field extractions through configuration files" in this manual.

Name column

The Name column in the Field extractions page displays the overall name (or "class") of the field extraction. The field extraction format is:

<spec> : [EXTRACT-<class> | REPORT-<class>]

  • <spec> can be:
    • <sourcetype>, the source type of an event.
    • host::<host>, where <host> is the host for an event.
    • source::<source>, where <source> is the source for an event.

EXTRACT-<class> field extractions are extractions that are wholly defined in props.conf (in other words, they do not reference a transform in transforms.conf). They are created automatically by field extractions made through IFX and certain search commands. If you have Splunk Enterprise, you can also add them by making direct updates to the props.conf file. This kind of extraction is always associated with a field-extracting regular expression. On the Field extractions page, this regex appears in the Extraction/Transform column.

REPORT-<class> field extractions reference field transform stanzas in transforms.conf. This is where their field-extracting regular expressions are located. On the Field extractions page, the referenced field transform stanza is indicated in the "Extraction/Transform" column.

You can work with transforms in Splunk Web through the Field Transformations page. For more information see "Use the Field Transformations page in Splunk Web" in this manual.

Type column

There are two field extraction types: inline and transforms.conf.

  • Inline extractions always have EXTRACT-<class> configurations. They are identified as such because they are entirely defined within props.conf; they do not reference external field transforms.
  • Uses transform extractions always have REPORT-<class> name configurations. As such they reference field transforms in transforms.conf. You can define field transforms directly in transforms.conf or via Splunk Web using the Field transformations page.

Extraction Transform column

In the Extraction/Transform column, Splunk Web displays different things depending on the field extraction Type.

  • For inline extraction types, Splunk Web displays the regular expression that Splunk software uses to extract the field. The named group (or groups) within the regex show you what field(s) it extracts.
For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regex by using it in a search with the rex search command.
  • In the case of Uses transform extraction types, Splunk Web displays the name of the transforms.conf field transform stanza (or stanzas) that the field extraction is linked to through props.conf. A field extraction can reference multiple field transforms if you want to apply more than one field-extracting regex to the same source, source type, or host. This can be necessary in cases where the field or fields that you want to extract appear in two or more very different event patterns.

For example, the Expression column could display two values for a Uses transform extraction: access-extractions and ip-extractions. These may appear in props.conf as:

[access_combined]
REPORT-access = access-extractions, ip-extractions
In this example, access-extractions and ip-extractions are both names of field transform stanzas in transforms.conf. To work with those field transforms through Splunk Web, go to the Field transforms page.

Add new field extractions

Click the New button at the top of the Field extractions page to add a new field extraction. The Add New page appears.

If you know how field extractions are set up in props.conf, you should find this to be pretty simple.

All of the fields described below are required.

  1. Define a Destination app context for the field extraction. By default it will be the app context you are currently in.
  2. Give the field extraction a Name, using underscores for spaces between words. In props.conf this is the <class> value for an EXTRACT or REPORT field extraction type. Note: <class> values do not have to follow field name syntax restrictions (see note below). You can use characters other than a-z, A-Z, and 0-9, and spaces are allowed. In addition <class> values are not subject to key cleaning.
  3. Define the sourcetype, source, or host to which the extraction applies. Select sourcetype, source, or host and enter the value. This maps to the <spec> value in props.conf.
  4. Define the extraction type.
    If you select Uses transform, enter the transform(s) involved in the Extraction/Transform field, separated by commas. The transforms can then be created or updated via the Field transforms page.
    If you select Inline, enter the regular expression used to extract the field (or fields) in the Extraction/Transform field. For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regex by using it in a search with the rex search command. Splunk also maintains a list of useful third-party tools for writing and testing regular expressions.

Important: The capturing groups in your regex must identify field names that only contain alpha-numeric characters or underscores.

  • Valid characters for field names are a-z, A-Z, 0-9, or _ .
  • Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk Enterprise's internal variables.
  • International characters are not allowed.

Splunk Enterprise applies the following "key cleaning" rules to all extracted fields, either by default or through a custom configuration:

  • All characters that are not in a-z, A-Z, and 0-9 ranges are replaced with an underscore (_).
  • All leading underscores and 0-9 characters are removed from extracted field names.

To disable this behavior for a specific field extraction, you have to manually modify both props.conf and transforms.conf. For more information, see "Create and maintain search-time field extractions through configuration files" in this manual.

Note: You cannot turn off key cleaning for inline field extractions (field extractions that do not require a field transform component) without editing the extraction stanza in props.conf.

Example - Add a new error code field

This shows how you would define an extraction for a new err_code field. The field can be identified by the occurrence of device_id= followed by a word within brackets and a text string terminating with a colon. The field should be extracted from events related to the testlog source type.

In props.conf this extraction would look like:

[testlog]
EXTRACT-errors = device_id=\[w+\](?<err_code>[^:]+)

Here's how you would set that up through the Add new field extractions page:

300p

Note: You can find a version of this example in "Create and maintain search-time field extractions" topic in this manual, which shows you how to set up field extractions using the props.conf file.

Update existing field extractions

To edit an existing field extraction, click its name in the Name column.

EditFieldExtractions.png

This takes you to a details page for that field extraction. In the Extraction/Transform field what you can do depends on the type of extraction that you are working with.

  • If the field extraction is an inline extraction, you can edit the regular expression it uses to extract fields.
  • If the field extraction uses one or more transforms, you can update the transform or transforms involved (put them in a comma-separated list if there is more than one.) The transforms can then be created or updated via the Field transforms page.

The field extraction depicted above is uses three transforms named wel-message, wel-eq-kv, and wel-col-kv. To find out more about how these transforms are set up, navigate to Settings > Fields > Field Transformations or go straight to transforms.conf.

Note: Uses transform field extractions must include at least one valid transforms.conf field extraction stanza name.

Update field extraction permissions

When a field extraction is created through an inline method (such as IFX or a search command) it is initially only available to its creator. To make it so that other users can use the field extraction, you need to update its permissions. To do this, locate the field extraction on the Field extractions page and select its Permissions link. This opens the standard permission management page used in Splunk Web for knowledge objects.

On this page you can set up role-based permissions for the field extraction, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Splunk Web, see "Manage knowledge object permissions," in this manual.

Delete field extractions in Splunk Web

You can delete field extractions if your permissions enable you to do so. You will not be able to delete default field extractions (extractions that were delivered with the product and which are stored in the "default" directory of an app).

  1. Navigate to Settings > Fields > Field extractions.
  2. Click Delete for the field extraction you want to remove.

Note: Take care when deleting objects that have downstream dependencies. For example, if your field extraction is used in a search that in turn is the basis for an event type that is used by five other saved searches (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that extraction from the system. For more information about deleting knowledge objects, see Disable or delete knowledge objects.

PREVIOUS
Field Extractor: Save step
  NEXT
Use the Field transformations page

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2


Comments

Hi,

Thanks for your comment! You do not need to restart Splunk after configuring a field extraction. This is because searches run in a separate process that reloads configurations. You can find more info on when you need to restart your Splunk instance in this topic in the Admin manual: https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Configurationfilechangesthatrequirerestart

Myu splunk, Splunker
March 24, 2017

I'm fairly certain you need to restart splunk after configuring a field extraction, but that doesn't seem to be mentioned anywhere on this page.

Dfqobvbkmnpi
March 23, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters