Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Use the Field transformations page

The Field transformations page in Settings lets you manage transform field extractions, which reside in transforms.conf. Field transforms can be created either through direct edits to transforms.conf or by addition through the Field transformations page.

Note: Every field transform has at least one field extraction component. But "inline" field extractions do not need to have a field transform component.

The Field transformations page enables you to:

  • Review the overall set of field transforms that you have created or which your permissions enable you to see, for all Apps in your Splunk deployment.
  • Create new search-time field transforms. For more information about situations that call for the use of field transforms, see "When to use the Field transformations page," below.
  • Update permissions for field transforms. Field transforms created through the Field transformations page are initially only available to their creators until they are shared with others. You can only update field transform permissions if you own the transform, or if your role's permissions enable you to do so.
  • Delete field transforms, if your app-level permissions enable you to do so, and if they are not default field transforms that were delivered with the product. Default knowledge objects cannot be deleted. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.

If you have "write" permissions for a particular field transform, the Field transformations page enables you to:

  • Update its regular expression and change the key the regular expression applies to.
  • Define or update the field transform format.

Navigate to the Field transformations page by selecting Settings > Fields > Field transformations.

Why set up a field transform for a field extraction?

While you can define most search-time field extractions entirely within props.conf (or the Field extractions page in Splunk Web), some advanced search-time field extractions require a transforms.conf component called a field transform. This component can be defined and managed through the Field transforms page.

You set up search-time field extractions with a field transform component when you need to:

  • Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform that is referenced by multiple field extractions). If you find yourself using the same regex to extract fields for different sources, source types, and hosts, you may want to set it up as a transform. Then, if you find that you need to update the regex, you only have to do so once, even though it is used by more than one field extraction.
  • Apply more than one field-extracting regular expression to the same source, source type, or host (in other words, apply multiple field transforms to the same field extraction). This is sometimes necessary in cases where the field or fields that you want to extract from a particular source/source type/host appear in two or more very different event patterns.
  • Use a regular expression to extract fields from the values of another field (also referred to as a "source key"). For example, you might pull a string out of a url field value, and have that be a value of a new field.

You can do more things with search-time field transforms (such as setting up delimeter based field extractions and configuring extractions for multivalue fields) if you configure them directly within transforms.conf. See the section on field transform setup in "Create and maintain search-time field extractions through configuration files" in this manual for more information.

Note: All index-time field extractions are coupled with one or more field transforms. You cannot manage index-time field extractions in Splunk Web, however--you have to use the props.conf and transforms.conf configuration files. We don't recommend that you change your set of index-time field extractions under normal circumstances, but if you find that you must do so, see "Create custom fields at index-time" in the Getting Data In manual.

Review and update search-time field transforms in Splunk Web

To better understand how the Field transformations page in Splunk Web displays your field transforms, it helps to understand how search-time field extractions are set up in your props.conf and transforms.conf files.

A typical field transform looks like this in transforms.conf:

[banner]
REGEX = /js/(?<license_type>[^/]*)/(?<version>[^/]*)/login/(?<login>[^/]*)
SOURCE_KEY = uri

This transform matches its regex against uri field values, and extracts three fields as named groups: license_type, version, and login.

In props.conf, that transform is matched to the source .../banner_access_log* like so:

[source::.../banner_access_log*]
REPORT-banner = banner

This means the regex is only matched to uri fields in events coming from the .../banner_access_log source.

But you can match it to other sources, sourcetypes, and hosts if necessary. This is something you can't do with inline field extractions (field extractions set up entirely within props.conf).

Note: By default, transforms are matched to a SOURCE_KEY value of _raw, in which case their regexes are applied to the entire event, not just fields within that event.

The Name column

The Name column of the Field transformations page displays the names of the search-time field transforms that your permissions enable you to see. These names are the actual stanza names for field transforms in transforms.conf. The transform example presented above would appear in the list of transforms as banner.

Click on a transform name to see the detail information for that particular transform.

Reviewing and editing transform details

The details page for a field transform enables you to view and update its regular expression, key, and event format. Here's the details page for the banner transform that we described at the start of this subtopic:

NewFieldTransformation.png

If you have the permissions to do so, you can edit the regex, key, and event format. Keep in mind that these edits can affect multiple field extractions defined in props.conf and the Field extractions page, if the transform has been applied to more than one source, sourcetype, or host.

Create a new field transform

To create a new field transform:

1. First, navigate to the Field transformations page and click the New button.

2. Identify the Destination app for the field transform, if it is not the app you are currently in.

3. Give the field transform a Name.

This equates to the stanza name for the transform on transforms.conf. When you save this transform this is the name that appears in the Name column on the Field transformations page. (This is a required field.)

4. Enter a Regular expression for the transform.

See "Regular expression syntax and usage."

5. (Optional) Define a Key for the transform.

This corresponds to the SOURCE_KEY option in transforms.conf. By default it is set to _raw, which means the regular expression is applied to entire events.
To have the regular expression be applied to values of a specific field, replace _raw with the name of that field. You can only use fields that are present when the field transform is executed.

6. (Optional) Specify the Event format.

This corresponds to the FORMAT option in transforms.conf. You use $n to indicate groups captured by the regular expression. For example, if the regular expression you've designed captures two groups, you could have a Format set up like this: $1::$2, where the first group is the field name, and the second group is the field value. Or you could set Format up as username::$1 userid::$2, which means the regular expression extracts the values for the username and userid fields. The Format field defaults to <transform_stanza_name>::$1.

7. (Optional) Select Create multivalue fields if the same field can be extracted from your events more than once.

This causes Splunk software to extract the field as a single multivalue field.

8. (Optional) Select Automatically clean field names to ensure that the extracted fields have valid names.

Leading underscore characters and 0-9 numerical characters are removed from field names, and characters other than those falling within the a-z, A-Z, and 0-9 ranges in field names are replaced with underscores. See "Regular expression syntax and usage."

Regular expression syntax and usage

For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test your regular expression by using it in a search with the rex search command.

Important: The capturing groups in your regular expression must identify field names that contain alpha-numeric characters or an underscore.

  • Valid characters for field names are a-z, A-Z, 0-9, or _ .
  • Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk software internal variables.
  • International characters are not allowed.

When Automatically clean field names is selected for a field transform, Splunk software applies the following "key cleaning" rules to field names extracted by that transform:

  • All characters that are not in a-z, A-Z, and 0-9 ranges are replaced with an underscore (_).
  • All leading underscores and 0-9 characters are removed from the name.

Note: You cannot turn off key cleaning for inline field extractions (field extractions that do not require a field transform component).

Example - Extract both field names and their corresponding field values from an event

You can use the Event format attribute in conjunction with a properly designed regular expression to set up a field transform that extracts both a field name and its corresponding field value from each matching event.

Here's an example, using a transform that is delivered with Splunk software.

The bracket-space field transform has a regular expression that finds field name/value pairs within brackets in event data. It will reapply this regular expression until all of the matching field/value pairs in an event are extracted.

BracketSpaceExample.png

As we stated earlier in this topic, field transforms are always associated with a field extraction. On the Field Extractions page in Splunk Web, you can see that the bracket-space field transform is associated with the osx-asl:REPORT-asl extraction.

Update field transform permissions

When a field transform is first created, by default it is only available to its creator. To make it so that other users can use the field transform, you need to update its permissions. To do this, locate the field transform on the Field transformations page and select its Permissions link. This opens the standard permission management page used in Splunk Web for knowledge objects.

On this page you can set up role-based permissions for the field transform, and determine whether it is available to users of one specific App, or globally to users of all Apps. For more information about managing permissions with Splunk Web, see "Manage knowledge object permissions," in this manual.

Delete field transforms

On the Field transformations page in Splunk Web, you can delete field transforms if your permissions enable you to do so.

Click Delete for the field extraction that you want to remove.

Note: Take care when deleting knowledge objects that have downstream dependencies. For example, if the field extracted by your field transform is used in a search that in turn is the basis for an event type that is used by five other reports (two of which are the foundation of dashboard panels), all of those other knowledge objects will be negatively impacted by the removal of that transform from the system. For more information about deleting knowledge objects, see "Disable or delete knowledge objects" in this manual.

PREVIOUS
Use the Field extractions page
  NEXT
Configure custom fields at search time

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters