Process events with ingest-time eval

An ingest-time eval is a type of transform that evaluates an expression at index-time. Ingest-time eval provides much of the same functionality provided by search-time eval. The primary difference is that an ingest-time eval processes event data before indexing occurs and new fields and values that result from the evaluation are sent to indexers.

For more information on search-time eval expressions, see Use the eval command and functions in the Search Manual.

You can use ingest-time eval expressions to create new fields and perform a wide range of operations on incoming data, including mathematical, statistical, and cryptographic functions. See Evaluation functions in the Search Reference.

Why use ingest-time eval?

Ingest-time eval provides an alternative to ingest-time transformations that are difficult or impossible with regular expressions alone, such as normalizing metrics data. See Example of targeted log-to-metrics conversions in the Metrics manual.

Ingest-time eval also gives you more direct control over index-time fields. For example, you can use ingest-time eval to control exactly how an index-time field is stored in the rawdata journal of a Splunk Enterprise index. For more information, see How the indexer stores indexes in Managing Indexers and Clusters of Indexers.

Ingest-time eval syntax and usage

Ingest-time eval takes a similar format to the search-time | eval command. For more information, see eval in the Search Reference.

An ingest-time eval stanza in transforms.conf contains an INGEST_EVAL expression. For example:

[eval1]
INGEST_EVAL= field3=length (_raw) *2

You can also chain multiple comma-separated INGEST_EVAL expressions, for example:

[eval2]
INGEST_EVAL= field4=_time, field5=field4+1

For detailed usage information and examples of INGEST_EVAL, see transforms.conf.

Search-time calculated fields that use the EVAL-fieldname setting in props.conf are not available.

Data processing that occurs before indexing with ingest-time eval can impact performance.

Configure an ingest-time eval transform

You configure eval-based transforms the same way you configure other index-time transforms, using a transforms.conf file that contains the transform stanza, in conjunction with a props.conf file that references it. You must also configure a fields.conf file on the search head to enable searching of newly indexed eval fields.

To process event data with ingest-time eval, configure the following files:

Configure transforms.conf

To configure transforms.conf for ingest-time eval, follow these steps:

1. Create a transforms.conf file in the $SPLUNK_HOME/etc/system/local directory.
2. Add an ingest-time eval stanza that specifies the INGEST_EVAL expression. For example, the following INGEST_EVAL expression creates a new field called eval_user and populates the field with the lowercase version of the values in the username field:

   [myeval]
   INGEST_EVAL = eval_user=lower(username)
   

Configure props.conf

To configure props.conf for ingest-time eval, follow these steps:

1. Create a props.conf in the $SPLUNK_HOME/etc/system/local directory.
2. Add a stanza that specifies the data you want to process, such as <my_sourcetype>, and references the ingest-time eval stanza in transforms.conf. For example:

   [my_sourcetype]
   TRANSFORMS = myeval
   

   Ingest-eval transforms require a sourcetype stanza in props.conf

   You can mix eval-based transforms and regex-based transforms in props.conf in any order. The order in which you list the transforms determines when the transforms run relative to other stanzas in tranforms.conf. For example, TRANSFORMS = eval1,regex1,eval2,regex2 runs four different transforms.conf stanzas in that specific order.

Configure fields.conf

To configure fields.conf to enable search of ingest-time eval fields, do the following:

1. On the search head, create a fields.conf file in the $SPLUNK_HOME/etc/system/local directory.
2. Add a stanza that references the newly indexed field created by the INGEST_EVAL expression, as follows:

   [eval_user]
   INDEXED = True
   

For more information on how to configure index-time transforms, see Define additional indexed fields.

Examples

For basic and extended examples of eval expressions, see eval in the Search Reference.