Configure multivalue fields with fields.conf
Multivalue fields are fields that can appear multiple times in an event and have a different value for each appearance. One of the more common examples of multivalue fields is that of email address fields, which typically appears two to three times in a single sendmail event--once for the sender, another time for the list of recipients, and possibly a third time for the list of Cc addresses, if one exists. If all of these fields are labeled identically (as "AddressList," for example), they lose meaning that they might otherwise have if they're identified separately as "From", "To", and "Cc".
Splunk Enterprise parses multivalue fields at search time, and enables you to process the values in the search pipeline. Search commands that work with multivalue fields include makemv, mvcombine, mvexpand, and nomv. For more information on these and other commands see the topic on manipulating multivalue fields in the Search Manual. The complete command reference is in the Search Reference manual.
TOKENIZER key to configure multivalue fields in fields.conf.
TOKENIZER uses a regular expression to tell Splunk Enterprise how to recognize and extract multiple field values for a recurring field in an event. Edit
$SPLUNK_HOME/etc/system/local/, or your own custom app directory in
For more information on configuration files in general, see "About configuration files" in the Admin manual.
For a primer on regular expression syntax and usage, see Regular-Expressions.info. You can test regexes by using them in searches with the rex search command. Splunk Enterprise also maintains a list of useful third-party tools for writing and testing regular expressions.
Configure a multivalue field via fields.conf
Define a multivalue field by adding a stanza for it in
fields.conf. Then add a line with the
TOKENIZER key and a corresponding regular expression that shows how the field can have multiple values.
Note: If you have other attributes to set for a multivalue field, set them in the same stanza underneath the
TOKENIZER line. See the fields.conf topic in the Admin manual for more information.
[<field name 1>] TOKENIZER = <regular expression> [<field name 2>] TOKENIZER = <regular expression>
<regular expression>should indicate how the field in question can take on multiple values.
TOKENIZERdefaults to empty. When
TOKENIZERis empty, the field can only take on a single value.
- Otherwise the first group is taken from each match to form the set of field values.
TOKENIZERkey is used by the where, timeline, and stats commands. It also provides the summary and XML outputs of the asynchronous search API.
Note: Tokenization of indexed fields (fields extracted at index time) is not supported. If you have set
INDEXED=true for a field, you cannot also use the
TOKENIZER key for that field. You can use a search-time extraction defined in
transforms.conf to break an indexed field into multiple values.
Say you have a poorly formatted email log file where all of the addresses involved are grouped together under
From: email@example.com To: firstname.lastname@example.org, email@example.com, firstname.lastname@example.org CC: email@example.com, firstname.lastname@example.org, email@example.com Subject: Multivalue fields are out there! X-Mailer: Febooti Automation Workshop (Unregistered) Content-Type: text/plain; charset=UTF-8 Date: Wed, 3 Nov 2014 17:13:54 +0200 X-Priority: 3 (normal)
This example from
$SPLUNK_HOME/etc/system/README/fields.conf.example breaks email fields
CC into multiple values.
[To] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w) [From] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w) [Cc] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w)
Create and maintain search-time field extractions through configuration files
Define calculated fields
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.3.0, 6.3.1, 6.3.1511, 6.3.2, 6.3.3, 6.3.4, 6.4.0