Use the Field transformations page
The Field transformations page in Settings lets you manage transform field extractions, which reside in
transforms.conf. Field transforms can be created either through direct edits to
transforms.conf or by addition through the Field transformations page.
Note: Every field transform has at least one field extraction component. But "inline" field extractions do not need to have a field transform component.
The Field transformations page enables you to:
- Review the overall set of field transforms that you have created or which your permissions enable you to see, for all Apps in your Splunk deployment.
- Create new search-time field transforms. For more information about situations that call for the use of field transforms, see "When to use the Field transformations page," below.
- Update permissions for field transforms. Field transforms created through the Field transformations page are initially only available to their creators until they are shared with others. You can only update field transform permissions if you own the transform, or if your role's permissions enable you to do so.
- Delete field transforms, if your app-level permissions enable you to do so, and if they are not default field transforms that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.
If you have "write" permissions for a particular field transform, the Field transformations page enables you to:
- Update its regular expression and change the key the regular expression applies to.
- Define or update the field transform format.
Navigate to the Field transformations page by selecting Settings > Fields > Field transformations.
Why set up a field transform for a field extraction?
While you can define most search-time field extractions entirely within
props.conf (or the Field extractions page in Splunk Web), some advanced search-time field extractions require a
transforms.conf component called a field transform. This component can be defined and managed through the Field transforms page.
You set up search-time field extractions with a field transform component when you need to:
- Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform that is referenced by multiple field extractions). If you find yourself using the same regex to extract fields for different sources, source types, and hosts, you may want to set it up as a transform. Then, if you find that you need to update the regex, you only have to do so once, even though it is used by more than one field extraction.
- Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction). This is sometimes necessary in cases where the field or fields that you want to extract from a particular source/source type/host appear in two or more very different event patterns.
- Use a regular expression to extract fields from the values of another field (also referred to as a "source key"). For example, you might pull a string out of a
urlfield value, and have that be a value of a new field.
You can do more things with search-time field transforms (such as setting up delimeter based field extractions and configuring extractions for multivalue fields) if you configure them directly within
transforms.conf. See the section on field transform setup in "Create and maintain search-time field extractions through configuration files" in this manual for more information.
Note: All index-time field extractions are coupled with one or more field transforms. You cannot manage index-time field extractions in Splunk Web, however--you have to use the
transforms.conf configuration files. We don't recommend that you change your set of index-time field extractions under normal circumstances, but if you find that you must do so, see "Create custom fields at index-time" in the Getting Data In manual.
Review and update search-time field transforms in Splunk Web
To better understand how the Field transformations page in Splunk Web displays your field transforms, it helps to understand how search-time field extractions are set up in your
A typical field transform looks like this in
[banner] REGEX = /js/(?<license_type>[^/]*)/(?<version>[^/]*)/login/(?<login>[^/]*) SOURCE_KEY = uri
This transform matches its regex against
uri field values, and extracts three fields as named groups:
props.conf, that transform is matched to the source
.../banner_access_log* like so:
[source::.../banner_access_log*] REPORT-banner = banner
This means the regex is only matched to
uri fields in events coming from the
But you can match it to other sources, sourcetypes, and hosts if necessary. This is something you can't do with inline field extractions (field extractions set up entirely within
Note: By default, transforms are matched to a
SOURCE_KEY value of
_raw, in which case their regexes are applied to the entire event, not just fields within that event.
The Name column
The Name column of the Field transformations page displays the names of the search-time field transforms that your permissions enable you to see. These names are the actual stanza names for field transforms in
transforms.conf. The transform example presented above would appear in the list of transforms as
Click on a transform name to see the detail information for that particular transform.
Reviewing and editing transform details
The details page for a field transform enables you to view and update its regular expression, key, and event format. Here's the details page for the
banner transform that we described at the start of this subtopic:
If you have the permissions to do so, you can edit the regex, key, and event format. Keep in mind that these edits can affect multiple field extractions defined in
props.conf and the Field extractions page, if the transform has been applied to more than one source, sourcetype, or host.
Create a new field transform
To create a new field transform:
1. First, navigate to the Field transformations page and click the New button.
2. Identify the Destination app for the field transform, if it is not the app you are currently in.
3. Give the field transform a Name.
- This equates to the stanza name for the transform on
transforms.conf. When you save this transform this is the name that appears in the Name column on the Field transformations page. (This is a required field.)
4. Enter a Regular expression for the transform.
5. (Optional) Define a Key for the transform.
- This corresponds to the
transforms.conf. By default it is set to
_raw, which means the regular expression is applied to entire events.
- To have the regular expression be applied to values of a specific field, replace
_rawwith the name of that field. You can only use fields that are present when the field transform is executed.
6. (Optional) Specify the Event format.
- This corresponds to the
transforms.conf. You use
$nto indicate groups captured by the regular expression. For example, if the regular expression you've designed captures two groups, you could have a Format set up like this:
$1::$2, where the first group is the field name, and the second group is the field value. Or you could set Format up as
username::$1 userid::$2, which means the regular expression extracts the values for the
useridfields. The Format field defaults to <transform_stanza_name>::$1.
7. (Optional) Select Create multivalue fields if the same field can be extracted from your events more than once.
- This causes Splunk software to extract the field as a single multivalue field.
8. (Optional) Select Automatically clean field names to ensure that the extracted fields have valid names.
- Leading underscore characters and 0-9 numerical characters are removed from field names, and characters other than those falling within the a-z, A-Z, and 0-9 ranges in field names are replaced with underscores. See "Regular expression syntax and usage."
Regular expression syntax and usage
Important: The capturing groups in your regular expression must identify field names that contain alpha-numeric characters or an underscore.
- Valid characters for field names are a-z, A-Z, 0-9, or _ .
- Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk software internal variables.
- International characters are not allowed.
When Automatically clean field names is selected for a field transform, Splunk software applies the following "key cleaning" rules to field names extracted by that transform:
- All characters that are not in a-z, A-Z, and 0-9 ranges are replaced with an underscore (_).
- All leading underscores and 0-9 characters are removed from the name.
Note: You cannot turn off key cleaning for inline field extractions (field extractions that do not require a field transform component).
Example - Extract both field names and their corresponding field values from an event
You can use the Event format attribute in conjunction with a properly designed regular expression to set up a field transform that extracts both a field name and its corresponding field value from each matching event.
Here's an example, using a transform that is delivered with Splunk software.
bracket-space field transform has a regular expression that finds field name/value pairs within brackets in event data. It will reapply this regular expression until all of the matching field/value pairs in an event are extracted.
As we stated earlier in this topic, field transforms are always associated with a field extraction. On the Field Extractions page in Splunk Web, you can see that the
bracket-space field transform is associated with the
Update field transform permissions
When a field transform is first created, by default it is only available to its creator. To make it so that other users can use the field transform, you need to update its permissions. To do this, locate the field transform on the Field transformations page and select its Permissions link. This opens the standard permission management page used in Splunk Web for knowledge objects.
On this page you can set up role-based permissions for the field transform, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Splunk Web, see "Manage knowledge object permissions," in this manual.
Delete field transforms
On the Field transformations page in Splunk Web, you can delete field transforms if your permissions enable you to do so.
Click Delete for the field extraction that you want to remove.
Note: Take care when deleting knowledge objects that have downstream dependencies. For example, if the field extracted by your field transform is used in a search that in turn is the basis for an event type that is used by five other reports (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that transform from the system. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.
Use the Field extractions page
Configure custom fields at search time
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2