Use the Field Extractions page in Manager
Use the Field Extractions page in Manager to manage search-time field extractions that have been added to
props.conf. There are three methods by which you can add search-time field extractions to
props.conf. You can:
- Use the Interactive Field Extractor, or IFX, to create extractions.
- Make direct edits to
- Add new field extractions with the Field Extractions page (see below).
The Field Extractions page enables you to:
- Review the overall set of search-time extractions that you have created or which your permissions enable you to see, for all Apps in your instance of Splunk.
- Create new search-time field extractions.
- Update permissions for field extractions. Field extractions created through the Interactive Field Extractor and the Field extractions page are initially only available to their creators until they are shared with others.
- Delete field extractions, if your app-level permissions enable you to do so, and if they are not default extractions that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.
If you have "write" permissions for a particular search-time field extraction, the Field extractions page enables you to:
- Update its regular expression, if it is an inline transaction.
- Add or delete named extractions that have been defined in
transforms.confor the Field transactions page in Manager, if it uses transactions.
Note: You cannot manage index-time field extractions via Manager. We don't recommend that you change your set of index-time field extractions, but if you find that you must do so, you have to modify your
transforms.conf configuration files manually. For more information about index-time field extraction configuration, see "Configure index-time field extractions" in the Getting Data In Manual.
Navigate to the Field extractions page by selecting Manager > Fields > Field extractions.
Review search-time field extractions in Manager
To better understand how the Field extractions page in Manager displays your field extraction, it helps to understand how field extractions are set up in your
Field extractions can be set up entirely in
props.conf, in which case they are identified on the Field extractions page as inline field extractions. But some field extractions include a
transforms.conf component called a field transform. To create/edit that component of the field extraction via Splunk Web, you use the Field Transforms page in Manager.
For more information about transforms and the Field Transforms page, see "Manage field transforms" in this manual.
For more information about field extraction setup directly in the props.conf and transforms.conf files see "Add fields at search time" in this manual.
The Name column in the Field extractions page displays the overall name (or "class") of the field extraction, as it appears in
props.conf, the field extraction format is:
<spec> : [EXTRACT-<class> | REPORT-<class>]
<sourcetype>, the source type of an event.
<host>is the host for an event.
<source>is the source for an event.
EXTRACT-<class> field extractions are extractions that are wholly defined in
props.conf (in other words, they do not reference a transform in transforms.conf). They are created automatically by field extractions made through IFX and certain search commands. You can also add them by making direct updates to the
props.conf file. This kind of extraction is always associated with a field-extracting regular expression. On the Field Extractions page, this regex appears in the Extraction/Transform column.
REPORT-<class> field extractions reference field transform stanzas in
transforms.conf. This is where their field-extracting regular expressions are located. On the Field Extractions page, the referenced field transform stanza is indicated in the "Extraction/Transform" column.
You can work with transforms in Manager through the Field Transformations page. For more information see "Use the Field Transformations page in Manager" in this manual.
There are two field extraction types: inline and transforms.conf.
- Inline extractions always have
EXTRACT-<class>configurations. They are identified as such because they are entirely defined within
props.conf; they do not reference external field transforms.
- Uses transform extractions always have
REPORT-<class>name configurations. As such they reference field transforms in
transforms.conf. You can define field transforms directly in
transforms.confor via Manager using the Field transformations page.
Extraction Transform column
In the Extraction/Transform column, Manager displays different things depending on the field extraction Type.
- For inline extraction types, Manager displays the regular expression that Splunk uses to extract the field. The named group (or groups) within the regex show you what field(s) it extracts.
- For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regex by using it in a search with the rex search command. Splunk also maintains a list of useful third-party tools for writing and testing regular expressions.
- In the case of Uses transform extraction types, Manager displays the name of the
transforms.conffield transform stanza (or stanzas) that the field extraction is linked to through
props.conf. A field extraction can reference multiple field transforms if you want to apply more than one field-extracting regex to the same source, source type, or host. This can be necessary in cases where the field or fields that you want to extract appear in two or more very different event patterns.
For example, the Expression column could display two values for a Uses transform extraction: access-extractions and ip-extractions. These may appear in
REPORT-access = access-extractions, ip-extractions
- In this example,
ip-extractionsare both names of field transform stanzas in
transforms.conf. To work with those field transforms through Manager, go to the Field transforms page.
Add new field extractions
Click the New button at the top of the Field extractions page to add a new field extraction. The Add New page appears.
If you know how field extractions are set up in
props.conf, you should find this to be pretty simple.
All of the fields described below are required.
1. Define a Destination app context for the field extraction. By default it will be the app context you are currently in.
2. Give the field extraction a Name, using underscores for spaces between words. In
props.conf this is the
<class> value for an EXTRACT or REPORT field extraction type. Note:
<class> values do not have to follow field name syntax restrictions (see the "Important" note below). You can use characters other than a-z, A-Z, and 0-9, and spaces are allowed. In addition
<class> values are not subject to "key cleaning."
3. Define the sourcetype, source, or host to which the extraction applies. Select sourcetype, source, or host and enter the value. This maps to the
<spec> value in
4. Define the extraction type.
- If you select Uses transform, enter the transform(s) involved in the Extraction/Transform field, separated by commas. The transforms can then be created or updated via the Field transforms page.
- If you select Inline, enter the regular expression used to extract the field (or fields) in the Extraction/Transform field. For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regex by using it in a search with the rex search command. Splunk also maintains a list of useful third-party tools for writing and testing regular expressions.
Important: The capturing groups in your regex must identify field names that only contain alpha-numeric characters or underscores.
- Valid characters for field names are a-z, A-Z, 0-9, or _ .
- Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk's internal variables.
- International characters are not allowed.
Splunk applies the following "key cleaning" rules to all extracted fields, either by default or through a custom configuration:
- All characters that are not in a-z, A-Z, and 0-9 ranges are replaced with an underscore (_).
- All leading underscores and 0-9 characters are removed from extracted field names.
To disable this behavior for a specific field extraction, you have to manually modify both
transforms.conf. For more information, see "Create and maintain search-time field extractions through configuration files" in this manual.
Note: You cannot turn off key cleaning for inline field extractions (field extractions that do not require a field transform component) without editing the extraction stanza in
Example - Add a new error code field
This shows how you would define an extraction for a new
err_code field. The field can be identified by the occurrence of
device_id= followed by a word within brackets and a text string terminating with a colon. The field should be extracted from events related to the
testlog source type.
props.conf this extraction would look like:
[testlog] EXTRACT-errors = device_id=\[w+\](?<err_code>[^:]+)
Here's how you would set that up through the Add new field extractions page:
Note: You can find a version of this example in "Create and maintain search-time field extractions" topic in this manual, which shows you how to set up field extractions using the
Update existing field extractions
To edit an existing field extraction, click its name in the Name column.
This takes you to a details page for that field extraction. In the Extraction/Transform field what you can do depends on the type of extraction that you are working with.
- If the field extraction is an inline extraction, you can edit the regular expression it uses to extract fields.
- If the field extraction uses one or more transforms, you can update the transform or transforms involved (put them in a comma-separated list if there is more than one.) The transforms can then be created or updated via the Field transforms page.
The field extraction depicted above is uses three transforms named
wel-col-kv. To find out more about how these transforms are set up, navigate to Manager > Fields > Field Transformations or go straight to
Note: Uses transform field extractions must include at least one valid
transforms.conf field extraction stanza name.
Update field extraction permissions
When a field extraction is created through an inline method (such as IFX or a search command) it is initially only available to its creator. To make it so that other users can use the field extraction, you need to update its permissions. To do this, locate the field extraction on the Field extractions page and select its Permissions link. This opens the standard permission management page used in manager for knowledge objects.
On this page you can set up role-based permissions for the field extraction, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Manager, see "Manage knowledge object permissions," in this manual.
Delete field extractions
On the Field extractions page in Manager, you can delete field extractions if your permissions enable you to do so. You won't be able to delete default field extractions (extractions that were delivered with the product and which are stored in the "default" directory of an app).
Click Delete for the field extraction that you want to remove.
Note: Take care when deleting objects that have downstream dependencies. For example, if your field extraction is used in a search that in turn is the basis for an event type that is used by five other saved searches (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that extraction from the system. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.
Extract fields interactively with IFX
Use the Field transformations page in Manager
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18