Process events with ingest-time eval
Splunk Enterprise users can create ingest-time eval expressions to process data before indexing occurs.
An ingest-time eval is a type of transform that evaluates an expression at index-time. Ingest-time eval provides much of the same functionality provided by search-time eval. The primary difference is that an ingest-time eval processes event data prior to indexing and the new fields and values that result from the evaluation are sent to indexers.
For more information on search-time eval expressions, see Use the eval command and functions in the Search Manual.
You can use ingest-time eval expressions to create new fields and perform a wide range of operations on incoming data, including mathematical, statistical, and cryptographic functions. See Evaluation functions in the Search Reference.
Why use ingest-time eval?
Ingest-time eval provides an alternative to ingest-time transformations that are difficult or impossible with regular expressions alone, such as normalizing metrics data. See Example of targeted log-to-metrics conversions in the Metrics manual.
Through ingest-time eval you can set up ingest-time lookups, which enable you to enrich your data with lookup fields as it is ingested, and before it is indexed. If you have certain lookups that are performed on almost all of your events, you may want to set them up as ingest-time lookups. See Reduce lookup overhead with ingest-time lookups.
Ingest-time eval also gives you more direct control over index-time fields. For example, you can use ingest-time eval to control exactly how an index-time field is stored in the rawdata journal of a Splunk Enterprise index. For more information, see How the indexer stores indexes in Managing Indexers and Clusters of Indexers.
Ingest-time eval syntax and usage
Ingest-time eval takes a similar format to the search-time
| eval command. For more information, see
eval in the Search Reference.
An ingest-time eval stanza in
transforms.conf contains an
INGEST_EVAL expression. For example:
[eval1] INGEST_EVAL= field3=length (_raw) *2
You can also chain multiple comma-separated
INGEST_EVAL expressions, for example:
[eval2] INGEST_EVAL= field4=_time, field5=field4+1
For detailed usage information and examples of
INGEST_EVAL, see transforms.conf.
The following search-time eval functions are not currently supported at index-time with INGEST_EVAL:
Search-time calculated fields that use the
EVAL-fieldname setting in
props.conf are not available.
Data processing that occurs before indexing with ingest-time eval can impact performance.
Configure an ingest-time eval transform
You configure eval-based transforms the same way you configure other index-time transforms, using a
transforms.conf file that contains the transform stanza, in conjunction with a
props.conf file that references it. You must also configure a
fields.conf file on the search head to enable searching of newly indexed eval fields.
To process event data with ingest-time eval, configure the following files:
transforms.conf for ingest-time eval, follow these steps:
- Create a
transforms.conffile in the
- Add an ingest-time eval stanza that specifies the
INGEST_EVALexpression. For example, the following
INGEST_EVALexpression creates a new field called
eval_userand populates the field with the lowercase version of the values in the
[myeval] INGEST_EVAL = eval_user=lower(username)
props.conf for ingest-time eval, follow these steps:
- Create a
- Add a stanza that specifies the data you want to process, such as <my_sourcetype>, and references the ingest-time eval stanza in
transforms.conf. For example:
[my_sourcetype] TRANSFORMS = myeval
Ingest-eval transforms require a sourcetype stanza in props.conf
You can mix eval-based transforms and regex-based transforms in
props.confin any order. The order in which you list the transforms determines when the transforms run relative to other stanzas in
tranforms.conf. For example,
TRANSFORMS = eval1,regex1,eval2,regex2runs four different
transforms.confstanzas in that specific order.
fields.conf to enable search of ingest-time eval fields, do the following:
- On the search head, create a
fields.conffile in the
- Add a stanza that references the newly indexed field created by the
INGEST_EVALexpression, as follows:
[eval_user] INDEXED = True
For more information on how to configure index-time transforms, see Define additional indexed fields.
For basic and extended examples of eval expressions, see
eval in the Search Reference.
Extract fields from files with structured data
Reduce lookup overhead with ingest-time lookups
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 9.0.0, 9.0.1, 9.0.2, 9.0.3