Use the Field transformations page

The Field transformations page in Settings lets you manage transform field extractions, which reside in transforms.conf. Field transforms can be created either through direct edits to transforms.conf or by addition through the Field transformations page.

Every field transform has at least one field extraction component.

The Field transformations page enables you to:

• Review the overall set of field transforms that you have created or which your permissions enable you to see, for all Apps in your Splunk deployment.
• Create new search-time field transforms. For more information about situations that call for the use of field transforms, see "When to use the Field transformations page," below.
• Update permissions for field transforms. Field transforms created through the Field transformations page are initially only available to their creators until they are shared with others. You can only update field transform permissions if you own the transform, or if your role's permissions enable you to do so.
• Delete field transforms, if your app-level permissions enable you to do so, and if they are not default field transforms that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see Disable or delete knowledge objects in this manual.

If you have "write" permissions for a particular field transform, the Field transformations page enables you to:

• Update its regular expression and change the key the regular expression applies to.
• Define or update the field transform format.

Navigate to the Field transformations page by selecting Settings > Fields > Field transformations.

Why set up a field transform for a field extraction?

While you can define most search-time field extractions entirely within props.conf or the Field extractions page in Splunk Web, some advanced search-time field extractions require a transforms.conf component called a field transform. These search-time field extractions are called transform field extractions and can be defined and managed through the Field transforms page.

Use a search-time field extractions with a field transform component when you need to:

• Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform that is referenced by multiple field extractions). If you find yourself using the same regex to extract fields for different sources, source types, and hosts, you may want to set it up as a transform. Then, if you find that you need to update the regex, you only have to do so once, even though it is used by more than one field extraction.
• Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction). This is sometimes necessary in cases where the field or fields that you want to extract from a particular source/source type/host appear in two or more very different event patterns.
• Use a regular expression to extract fields from the values of another field (also referred to as a "source key"). For example, you might pull a string out of a url field value, and have that be a value of a new field.

You can do more things with search-time field transforms (such as setting up delimiter based field extractions and configuring extractions for multi-value fields) if you configure them directly within transforms.conf. See the section on field transform setup in Configure advanced extractions with field transforms.

Note: All index-time field extractions are coupled with one or more field transforms. You cannot manage index-time field extractions in Splunk Web, however--you have to use the props.conf and transforms.conf configuration files. We don't recommend that you change your set of index-time field extractions under normal circumstances, but if you find that you must do so, see Create custom fields at index-time in the Getting Data In manual.

Review and update search-time field transforms in Splunk Web

To better understand how the Field transformations page in Splunk Web displays your field transforms, it helps to understand how search-time field extractions are set up in your props.conf and transforms.conf files.

A typical field transform looks like this in transforms.conf:

[banner]
REGEX = /js/(?<license_type>[^/]*)/(?<version>[^/]*)/login/(?<login>[^/]*)
SOURCE_KEY = uri

This transform matches its regex against uri field values, and extracts three fields as named groups: license_type, version, and login.

In props.conf, that transform is matched to the source .../banner_access_log* like so:

[source::.../banner_access_log*]
REPORT-banner = banner

This means the regex is only matched to uri fields in events coming from the .../banner_access_log source. But you can match it to other sources, sourcetypes, and hosts if necessary.

Note: By default, transforms are matched to a SOURCE_KEY value of _raw, in which case their regexes are applied to the entire event, not just fields within that event.

The Name column

The Name column of the Field transformations page displays the names of the search-time field transforms that your permissions enable you to see. These names are the actual stanza names for field transforms in transforms.conf. The transform example presented above would appear in the list of transforms as banner.

Click on a transform name to see the detail information for that particular transform.

Reviewing and editing transform details

The details page for a field transform enables you to view and update its regular expression, key, and event format. Here's the details page for the banner transform that we described at the start of this subtopic:

If you have the permissions to do so, you can edit the regex, key, and event format. Keep in mind that these edits can affect multiple field extractions defined in props.conf and the Field extractions page, if the transform has been applied to more than one source, sourcetype, or host.

Create a new field transform

Prerequisites

• Regular expressions and field name syntax for information about field-extracting regular expressions.
• About default fields (host, source, source type, and more) for information about hosts, sources, and sourcetypes.
• Configure custom fields at search time for information on different types of field extraction.

Splunk Cloud Platform does not support saving field transformations to the 000-self-service app using the Splunk Web UI. Doing so can overwrite existing field transformations in transforms.conf. All apps starting with a 3-digit prefix, such as 100-whisper, 100-whisper-common, 100-whisper-searchhead, and so on, are for internal Splunk use only and not intended for use by customers.

Steps

1. Select Settings > Fields to navigate to the Fields manager page.
2. Select Field transformations > New to navigate to the FIelds transformations page.
3. Identify the Destination app for the field transform, if it is not the app you are currently in.
4. Give the field transform a Name.

   This equates to the stanza name for the transform on transforms.conf. When you save this transform this is the name that appears in the Name column on the Field transformations page.

5. Enter a Regular expression for the transform.

   See Regular expressions and field name syntax.

6. (Optional) Define a Key for the transform.

   This corresponds to the SOURCE_KEY option in transforms.conf. By default it is set to _raw, which means the regular expression is applied to entire events.

   To have the regular expression be applied to values of a specific field, replace _raw with the name of that field. You can only use fields that are present when the field transform is executed.

7. (Optional) Specify the Event format.

   This corresponds to the FORMAT option in transforms.conf. You use $n to indicate groups captured by the regular expression. For example, if the regular expression you've designed captures two groups, you could have a Format set up like this: $1::$2, where the first group is the field name, and the second group is the field value. Or you could set Format up as username::$1 userid::$2, which means the regular expression extracts the values for the username and userid fields. The Format field defaults to <transform_stanza_name>::$1.

8. (Optional) Select Create multivalue fields if the same field can be extracted from your events more than once.

   This causes Splunk software to extract the field as a single multivalue field.

9. (Optional) Select Automatically clean field names to ensure that the extracted fields have valid names.

   Leading underscore characters and 0-9 numerical characters are removed from field names, and characters other than those falling within the a-z, A-Z, and 0-9 ranges in field names are replaced with underscores.

Example - Extract both field names and their corresponding field values from an event

You can use the Event format attribute in conjunction with a properly designed regular expression to set up a field transform that extracts both a field name and its corresponding field value from each matching event.

Here's an example, using a transform that is delivered with Splunk software.

The bracket-space field transform has a regular expression that finds field name/value pairs within brackets in event data. It will reapply this regular expression until all of the matching field/value pairs in an event are extracted.

As we stated earlier in this topic, field transforms are always associated with a field extraction. On the Field Extractions page in Splunk Web, you can see that the bracket-space field transform is associated with the osx-asl:REPORT-asl extraction.

Update field transform permissions

When a field transform is first created, by default it is only available to its creator. To make it so that other users can use the field transform, you need to update its permissions. To do this, locate the field transform on the Field transformations page and select its Permissions link. This opens the standard permission management page used in Splunk Web for knowledge objects.

On this page you can set up role-based permissions for the field transform, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Splunk Web, see Manage knowledge object permissions.

Delete field transforms

On the Field transformations page in Splunk Web, you can delete field transforms if your permissions enable you to do so.

Click Delete for the field extraction that you want to remove.

Note: Take care when deleting knowledge objects that have downstream dependencies. For example, if the field extracted by your field transform is used in a search that in turn is the basis for an event type that is used by five other reports (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that transform from the system. For more information about deleting knowledge objects, see Disable or delete knowledge objects.