Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Configure extractions of multivalue fields with fields.conf

Multivalue fields are fields that can appear multiple times in an event and have a different value for each appearance. One of the more common examples of multivalue fields is that of email address fields, which typically appears two to three times in a single sendmail event--once for the sender, another time for the list of recipients, and possibly a third time for the list of Cc addresses, if one exists. If all of these fields are labeled identically (as "AddressList," for example), they lose meaning that they might otherwise have if they're identified separately as "From", "To", and "Cc".

Multivalue fields are parsed at search time, which enables you to process the resulting values in the search pipeline. Search commands that work with multivalue fields include makemv, mvcombine, mvexpand, and nomv. For more information on these and other commands see the topic on manipulating multivalue fields in the Search Manual. The complete command reference is in the Search Reference manual.

Use the TOKENIZER key to configure multivalue fields in fields.conf. TOKENIZER uses a regular expression to tell Splunk software how to recognize and extract multiple field values for a recurring field in an event. If you have Splunk Enterprise, you edit fields.conf in $SPLUNK_HOME/etc/system/local/, or your own custom app directory in $SPLUNK_HOME/etc/apps/.

For more information on configuration files in general, see "About configuration files" in the Admin manual.

For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test regexes by using them in searches with the rex search command.

Configure a multivalue field via fields.conf

If you have Splunk Enterprise, you can define a multivalue field by adding a stanza for it in fields.conf. Then add a line with the TOKENIZER key and a corresponding regular expression that shows how the field can have multiple values.

Note: If you have other attributes to set for a multivalue field, set them in the same stanza underneath the TOKENIZER line. See the fields.conf topic in the Admin manual for more information. 

[<field name 1>]
TOKENIZER = <regular expression>

[<field name 2>]
TOKENIZER = <regular expression>
  • <regular expression> should indicate how the field in question can take on multiple values.
  • TOKENIZER defaults to empty. When TOKENIZER is empty, the field can only take on a single value.
  • Otherwise the first group is taken from each match to form the set of field values.
  • The TOKENIZER key is used by the where, timeline, and stats commands. It also provides the summary and XML outputs of the asynchronous search API.

Note: Tokenization of indexed fields (fields extracted at index time) is not supported. If you have set INDEXED=true for a field, you cannot also use the TOKENIZER key for that field. You can use a search-time extraction defined in props.conf and transforms.conf to break an indexed field into multiple values.

Example

Say you have a poorly formatted email log file where all of the addresses involved are grouped together under AddressList: 

From:          sender@splunkexample.com
To:            recipient1@splunkexample.com, recipient2@splunkexample.com, recipient3@splunkexample.com
CC:            cc1@splunkexample.com, cc2@splunkexample.com, cc3@splunkexample.com
Subject:       Multivalue fields are out there!
X-Mailer:      Febooti Automation Workshop (Unregistered)
Content-Type:  text/plain; charset=UTF-8
Date:          Wed, 3 Nov 2014 17:13:54 +0200
X-Priority:    3 (normal)

This example from $SPLUNK_HOME/etc/system/README/fields.conf.example breaks email fields To, From, and CC into multiple values. 

[To]
TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w)

[From]
TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w)

[Cc]
TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w)
PREVIOUS
Example transform field extraction configurations 		  NEXT
Define calculated fields

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12

Comments

@Spammenot66: Thanks for your recent Splunk documentation feedback about multivalued field configuration in Splunk Web.

If you’re just trying to create multivalued fields, you can try setting them up through "Settings > Fields > Field Transformations,” as described in http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Managefieldtransforms

* First you enter a regular expression that extracts a field that appears multiple times in an event, but with different values.
* Then you select "Create multivalued fields.” This makes the extracted field a multivalue field.

Keep in mind that all you are doing here is setting up a field transform. It has to be tied to a field extraction that you create at "Settings > Fields > Field Extractions” http://docs.splunk.com/Documentation/Splunk/6.0.1/Knowledge/Managesearch-timefieldextractions

Mness, Splunker
March 14, 2016

Is this possible to do using SPLUNK Web? or access to fields.conf for manual edit is absolutely necessary?

Spammenot66
March 13, 2016

I added an example event to better illustrate how you can use fields.conf to set up multivalued field extractions.<br /><br />However, as I researched this I realized that this sort of extraction is probably better configured in props.conf and transforms.conf using the MV_ADD parameter in transforms.conf. Here's an example that shows you how this parameter is used: http://docs.splunk.com/Documentation/Splunk/6.0beta/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Handling_events_with_multivalued_fields<br /><br />It looks like MV_ADD can effectively replace the usage of fields.conf for most cases of multivalued field extraction. See this link for more examples of MV_ADD: http://answers.splunk.com/search/?q=MV_ADD. <br /><br />I'll update the documentation around this after I do a bit more research.

Mness
August 25, 2014

Thanks for the document. <br /><br />Please post a sample log entry with the article. I will be really handy to understand what is the target text being processed. Not sure if this documents needs to be referred with any other complimentary resources.

Fromdev
August 22, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters