Use the Field transformations page in Manager

Use the Field transformations page in Manager

The Field transformations page in Manager enables you to manage the field transform components of search-time field extractions, which reside in transforms.conf. Field transforms can be created either through direct edits to transforms.conf or by addition through the Field transformations page.

Note: Every field transform has at least one field extraction component. But "inline" field extractions do not need to have a field transform component.

The Field transformations page enables you to:

• Review the overall set of field transforms that you have created or which your permissions enable you to see, for all Apps in your instance of Splunk.
• Create new search-time field transforms. For more information about situations that call for the use of field transforms, see "When to use the Field transformations page," below.
• Update permissions for field transforms. Field transforms created through the Field transformations page are initially only available to their creators until they are shared with others. You can only update field transform permissions if you own the transform, or if your role's permissions enable you to do so.
• Delete field transforms, if your app-level permissions enable you to do so, and if they are not default field transforms that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see "Curate Splunk knowledge with Manager" in this manual.

If you have "write" permissions for a particular field transform, the Field transformations page enables you to:

• Update its regular expression and change the key the regular expression applies to.
• Define or update the field transform format.

Navigate to the Field transformations page by selecting Manager > Fields > Field transformations.

Why set up a field transform for a field extraction?

While you can define most search-time field extractions entirely within props.conf (or the Field extractions page in Manager), some advanced search-time field extractions require a transforms.conf component called a field transform. This component can be defined and managed through the Field transforms page.

You set up search-time field extractions with a field transform component when you need to:

• Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform that is referenced by multiple field extractions). If you find yourself using the same regex to extract fields for different sources, source types, and hosts, you may want to set it up as a transform. Then, if you find that you need to update the regex, you only have to do so once, even though it is used by more than one field extraction.
• Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction). This is sometimes necessary in cases where the field or fields that you want to extract from a particular source/source type/host appear in two or more very different event patterns.
• Use a regular expression to extract fields from the values of another field (also referred to as a "source key"). For example, you might pull a string out of a url field value, and have that be a value of a new field.

You can do more things with search-time field transforms (such as setting up delimeter based field extractions and configuring extractions for multivalued fields) if you configure them directly within transforms.conf. See the section on field transform setup in "Create and maintain search-time field extractions through configuration files" in this manual for more information.

Note: All index-time field extractions are coupled with one or more field transforms. You cannot manage index-time field extractions via Manager, however--you have to use the props.conf and transforms.conf configuration files. We don't recommend that you change your set of index-time field extractions under normal circumstances, but if you find that you must do so, see "Create custom fields at index-time" in the Getting Data In manual.

Review and update search-time field transforms in Manager

To better understand how the Field transformations page in Manager displays your field transforms, it helps to understand how search-time field extractions are set up in your props.conf and transforms.conf files.

A typical field transform looks like this in transforms.conf:

[banner]
REGEX = /js/(?<license_type>[^/]*)/(?<version>[^/]*)/login/(?<login>[^/]*)
SOURCE_KEY = uri

This transform matches its regex against uri field values, and extracts three fields as named groups: license_type, version, and login.

In props.conf, that transform is matched to the source .../banner_access_log* like so:

[source::.../banner_access_log*]
REPORT-banner = banner

This means the regex is only matched to uri fields in events coming from the .../banner_access_log source.

But you can match it to other sources, sourcetypes, and hosts if necessary. This is something you can't do with inline field extractions (field extractions set up entirely within props.conf).

Note: By default, transforms are matched to a SOURCE_KEY value of _raw, in which case their regexes are applied to the entire event, not just fields within that event.

The Name column

The Name column of the Field transformations page displays the names of the search-time field transforms that your permissions enable you to see. These names are the actual stanza names for field transforms in transforms.conf. The transform example presented above would appear in the list of transforms as banner.

Click on a transform name to see the detail information for that particular transform.

Reviewing and editing transform details

The details page for a field transform enables you to view and update its regular expression, key, and event format. Here's the details page for the banner transform that we described at the start of this subtopic:

If you have the permissions to do so, you can edit the regex, key, and event format. Keep in mind that these edits can affect multiple field extractions defined in props.conf and the Field extractions page, if the transform has been applied to more than one source, sourcetype, or host.

Create a new field transform

To create a new field transform:

1. First, navigate to the Field transformations page and click the New button.

2. Identify the Destination app for the field transform, if it is not the app you are currently in.

3. Give the field transform a Name. This equates to the stanza name for the transform on transforms.conf. When you save this transform this is the name that appears in the Name column on the Field transformations page. (This is a required field.)

4. Enter a Regular expression for the transform. (This is a required field.)

5. Optionally define a Key for the transform. This corresponds to the SOURCE_KEY option in transforms.conf. By default it is set to _raw, which means the regex is applied to entire events. To have the regex be applied to values of a specific field, replace _raw with the name of that field. You can only use fields that are present when the field transform is executed.

6. Optionally specify the Event format. This corresponds to the FORMAT option in transforms.conf. You use $n to indicate groups captured by the regex. For example, if the regex you've designed captures two groups, you could have a Format set up like this: $1::$2, where the first group is the field name, and the second group is the field value. Or you could set Format up as username::$1 userid::$2, which means the regex extracts the values for the username and userid fields. The Format field defaults to <transform_stanza_name>::$1.

Regular expression syntax and usage

For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regex by using it in a search with the rex search command. Splunk also maintains a list of useful third-party tools for writing and testing regular expressions.

Important: The capturing groups in your regex must identify field names that contain alpha-numeric characters or an underscore.

• Valid characters for field names are a-z, A-Z, 0-9, or _ .
• Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk's internal variables.
• International characters are not allowed.

Splunk applies the following "key cleaning" rules to all extracted fields when they are extracted at search-time, either by default or through a custom configuration:

1. All characters that are not in a-z, A-Z, and 0-9 ranges are replaced with an underscore (_).

2. When key cleaning is enabled (it is enabled by default), Splunk removes all leading underscores and 0-9 characters from extracted fields.

You can disable key cleaning for a particular field transform by setting CLEAN_KEYS=false in its stanza on transforms.conf.

Note: You cannot turn off key cleaning for inline field extractions (field extractions that do not require a field transform component).

Example - Extract both field names and their corresponding field values from an event

You can use the Event format attribute in conjunction with a properly designed regular expression to set up a field transform that extracts both a field name and its corresponding field value from each matching event.

Here's an example, using a transform that is delivered with Splunk.

The bracket-space field transform has a regular expression that finds field name/value pairs within brackets in event data. It will reapply this regular expression until all of the matching field/value pairs in an event are extracted.

As we stated earlier in this topic, field transforms are always associated with a field extraction. On the Field Extractions page in Manager, you can see that the bracket-space field transform is associated with the osx-asl:REPORT-asl extraction.

Update field transform permissions

When a field transform is first created, by default it is only available to its creator. To make it so that other users can use the field transform, you need to update its permissions. To do this, locate the field transform on the Field transformations page and select its Permissions link. This opens the standard permission management page used in Manager for knowledge objects.

On this page you can set up role-based permissions for the field transform, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Manager, see "Curate Splunk knowledge with Manager," in this manual.

Delete field transforms

On the Field transformations page in Manager, you can delete field transforms if your permissions enable you to do so.

Click Delete for the field extraction that you want to remove.

Note: Take care when deleting knowledge objects that have downstream dependencies. For example, if the field extracted by your field transform is used in a search that in turn is the basis for an event type that is used by five other saved searches (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that transform from the system. For more information about deleting knowledge objects, see "Curate Splunk knowledge with Manager" in this manual.

• Was this topic useful?
• Post a comment