Process events with ingest-time eval
You can create ingest-time eval expressions to process data before indexing occurs.
An ingest-time eval is a type of transform that evaluates an expression at index-time. Ingest-time eval provides much of the same functionality provided by search-time eval. The primary difference is that an ingest-time eval processes event data prior to indexing and the new fields and values that result from the evaluation are sent to indexers.
For more information on search-time eval expressions, see Use the eval command and functions in the Search Manual.
You can use ingest-time eval expressions to create new fields and perform a wide range of operations on incoming data, including mathematical, statistical, and cryptographic functions. See Evaluation functions in the Search Reference.
If you have access to the Edge Processor solution, you can also use Edge Processors to process data using eval expressions. All eval expressions in Edge Processors process event data before indexing. See About the Edge Processor solution in the Use Edge Processors manual.
Why use ingest-time eval?
Ingest-time eval provides an alternative to ingest-time transformations that are difficult or impossible with regular expressions alone, such as normalizing metrics data. See Example of targeted log-to-metrics conversions in the Metrics manual.
Through ingest-time eval you can set up ingest-time lookups, which enable you to enrich your data with lookup fields as it is ingested, and before it is indexed. If you have certain lookups that are performed on almost all of your events, you may want to set them up as ingest-time lookups. See Reduce lookup overhead with ingest-time lookups.
Ingest-time eval also gives you more direct control over index-time fields. For example, you can use ingest-time eval to control exactly how an index-time field is stored in the rawdata journal of a Splunk Enterprise index. For more information, see How the indexer stores indexes in Managing Indexers and Clusters of Indexers.
Ingest-time eval syntax and usage
Ingest-time eval takes a similar format to the search-time | eval
command. For more information, see eval
in the Search Reference.
An ingest-time eval stanza in transforms.conf
contains an INGEST_EVAL
expression. For example:
[eval1] INGEST_EVAL= field3=length (_raw) *2
You can also chain multiple comma-separated INGEST_EVAL
expressions, for example:
[eval2] INGEST_EVAL= field4=_time, field5=field4+1
For detailed usage information and examples of INGEST_EVAL
, see transforms.conf.
The following search-time eval functions are not currently supported at index-time with INGEST_EVAL: mvfilter
, mvmap
, searchmatch
, now
, and commands
.
Search-time calculated fields that use the EVAL-fieldname
setting in props.conf
are not available.
Data processing that occurs before indexing with ingest-time eval can impact performance.
Configure an ingest-time eval transform
You configure eval-based transforms the same way you configure other index-time transforms, using a transforms.conf
file that contains the transform stanza, in conjunction with a props.conf
file that references it. You must also configure a fields.conf
file on the search head to enable searching of newly indexed eval fields.
To process event data with ingest-time eval, configure the following files:
Configure transforms.conf
To configure transforms.conf
for ingest-time eval, follow these steps:
- Create a
transforms.conf
file in the$SPLUNK_HOME/etc/system/local
directory. - Add an ingest-time eval stanza that specifies the
INGEST_EVAL
expression. For example, the followingINGEST_EVAL
expression creates a new field calledeval_user
and populates the field with the lowercase version of the values in theusername
field:[myeval] INGEST_EVAL = eval_user=lower(username)
Configure props.conf
To configure props.conf
for ingest-time eval, follow these steps:
- Create a
props.conf
in the$SPLUNK_HOME/etc/system/local
directory. - Add a stanza that specifies the data you want to process, such as <my_sourcetype>, and references the ingest-time eval stanza in
transforms.conf
. For example:[my_sourcetype] TRANSFORMS = myeval
Ingest-eval transforms require a sourcetype stanza in props.conf
You can mix eval-based transforms and regex-based transforms in
props.conf
in any order. The order in which you list the transforms determines when the transforms run relative to other stanzas intranforms.conf
. For example,TRANSFORMS = eval1,regex1,eval2,regex2
runs four differenttransforms.conf
stanzas in that specific order.
Configure fields.conf
To configure fields.conf
to enable search of ingest-time eval fields, do the following:
- On the search head, create a
fields.conf
file in the$SPLUNK_HOME/etc/system/local
directory. - Add a stanza that references the newly indexed field created by the
INGEST_EVAL
expression, as follows:[eval_user] INDEXED = True
For more information on how to configure index-time transforms, see Define additional indexed fields.
Examples
For basic and extended examples of eval expressions, see eval
in the Search Reference.
Extract fields from files with structured data | Reduce lookup overhead with ingest-time lookups |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!