Configure extractions of multivalue fields with fields.conf
A multivalue field is a field that contains more than one value. One of the more common examples of multivalue fields is that of email address fields, which typically appear two to three times in a single sendmail event—once for the sender, another time for the list of recipients, and possibly a third time for the list of Cc addresses, if one exists.
A multivalue fields occurs when there are multiple To
or Cc
recipients. A multivalue field might also occur if all of the fields are labeled identically, such as AddressList
. The fields lose meaning that they might otherwise have if they're identified separately as From
, To
, and Cc
.
Multivalue fields are parsed at search time, which enables you to process the resulting values in the search pipeline. Search commands that work with multivalue fields include makemv, mvcombine, mvexpand, and nomv. For more information on these and other commands see Manipulate and evaluate fields with multiple values in the Search Manual. The complete command reference is in the Search Reference manual.
Use the TOKENIZER setting to define a multivalue field in fields.conf
You can use the TOKENIZER
setting to define a multivalue field in fields.conf
. At search time, TOKENIZER
uses a regular expression to tell the Splunk platform how to recognize and extract multiple field values for a recurring field in an event.
The TOKENIZER
setting is used by the where
, timeline
, and stats
commands. It also provides the summary and XML outputs of the asynchronous search API.
Tokenization of indexed fields (fields extracted at index time) is not supported. If you have set INDEXED=true
for a field, you cannot also use the TOKENIZER
setting for that field. You can use a transform extraction defined in props.conf
and transforms.conf
to break an indexed field into multiple values.
Prerequisites
- Review the TOKENIZER multivalue field configuration syntax.
- See fields.conf in the Admin Manual to learn how the
fields.conf
file works. - For an overview of configuration file usage in the Splunk platform, see About configuration files in the Admin Manual.
- For a primer on regular expression syntax and usage, see About Splunk regular expressions. You can test regular expressions by using them in searches with the rex search command.
Steps
- Open the
fields.conf
file that you want to edit.
If you have Splunk Enterprise, you editfields.conf
in$SPLUNK_HOME/etc/system/local/
, or your own custom app directory in$SPLUNK_HOME/etc/apps/
. - Add a stanza for the multivalue field. The stanza name should be the name of the field.
- Add a line in the stanza that matches the
TOKENIZER
setting with a regular expression that is designed to capture multiple values for a field. - Optional. If you have other attributes to set for the multivalue field, set them in the same stanza underneath the
TOKENIZER
line. - Save your changes to the file.
TOKENIZER multivalue field configuration syntax
[<field name 1>] TOKENIZER = <regular expression> [<field name 2>] TOKENIZER = <regular expression>
<regular expression>
should be designed to capture multiple values for a field. For example, if a field name is followed by a list of email addresses, the regular expression should be able to extract each individual address as a separate value of the field without capturing delimiters like commas and spaces.TOKENIZER
defaults to empty. WhenTOKENIZER
is empty, the field can only take on a single value.- When
TOKENIZER
is not empty, the first group is taken from each match to form the set of field values. TOKENIZER
separates the multiple values of a field with the following delimiter characters:\n
.
Example
You start with a poorly formatted email log file where all of the addresses involved are grouped together under AddressList
. Here is a sample from that log file.
From: sender@splunkexample.com To: recipient1@splunkexample.com, recipient2@splunkexample.com, recipient3@splunkexample.com CC: cc1@splunkexample.com, cc2@splunkexample.com, cc3@splunkexample.com Subject: Multivalue fields are out there! X-Mailer: Febooti Automation Workshop (Unregistered) Content-Type: text/plain; charset=UTF-8 Date: Wed, 3 Nov 2017 17:13:54 +0200 X-Priority: 3 (normal)
This example from $SPLUNK_HOME/etc/system/README/fields.conf.example
breaks email fields To
, From
, and CC
into multiple values.
[To] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w) [From] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w) [Cc] TOKENIZER = (\w[\w\.\-]*@[\w\.\-]*\w)
Because the TOKENIZER process adds a \n
delimiter between each value it extracts for a field, the multiple values for To
in the sample event for this example will display like this: recipient1@splunkexample.com\nrecipient2@splunkexample.com\nrecipient3@splunkexample.com
.
Example transform field extraction configurations | About calculated fields |
This documentation applies to the following versions of Splunk® Enterprise: 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!