Process events with ingest-time eval

You can create ingest-time eval expressions to process data before indexing occurs.

An ingest-time eval is a type of transform that evaluates an expression at index-time. Ingest-time eval provides much of the same functionality provided by search-time eval. The primary difference is that an ingest-time eval processes event data prior to indexing and the new fields and values that result from the evaluation are sent to indexers.

For more information on search-time eval expressions, see Use the eval command and functions in the Search Manual.

You can use ingest-time eval expressions to create new fields and perform a wide range of operations on incoming data, including mathematical, statistical, and cryptographic functions. See Evaluation functions in the Search Reference.

If you have access to the Edge Processor solution, you can also use Edge Processors to process data using eval expressions. All eval expressions in Edge Processors process event data before indexing. See About the Edge Processor solution in the Use Edge Processors manual.

Why use ingest-time eval?

Ingest-time eval provides an alternative to ingest-time transformations that are difficult or impossible with regular expressions alone, such as normalizing metrics data. See Example of targeted log-to-metrics conversions in the Metrics manual.

Through ingest-time eval you can set up ingest-time lookups, which enable you to enrich your data with lookup fields as it is ingested, and before it is indexed. If you have certain lookups that are performed on almost all of your events, you may want to set them up as ingest-time lookups. See Reduce lookup overhead with ingest-time lookups.

Ingest-time eval also gives you more direct control over index-time fields. For example, you can use ingest-time eval to control exactly how an index-time field is stored in the rawdata journal of a Splunk Enterprise index. For more information, see How the indexer stores indexes in Managing Indexers and Clusters of Indexers.

Ingest-time eval syntax and usage

Ingest-time eval takes a similar format to the search-time | eval command. For more information, see eval in the Search Reference.

An ingest-time eval stanza in transforms.conf contains an INGEST_EVAL expression. For example:

[eval1]
INGEST_EVAL= field3=length (_raw) *2

You can also chain multiple comma-separated INGEST_EVAL expressions, for example:

[eval2]
INGEST_EVAL= field4=_time, field5=field4+1

For detailed usage information and examples of INGEST_EVAL, see transforms.conf.

The following search-time eval functions are not currently supported at index-time with INGEST_EVAL: mvfilter, mvmap, searchmatch, now, and commands.

Search-time calculated fields that use the EVAL-fieldname setting in props.conf are not available.

Data processing that occurs before indexing with ingest-time eval can impact performance.

Configure an ingest-time eval transform

You configure eval-based transforms the same way you configure other index-time transforms, using a transforms.conf file that contains the transform stanza, in conjunction with a props.conf file that references it. You must also configure a fields.conf file on the search head to enable searching of newly indexed eval fields.

To process event data with ingest-time eval, configure the following files:

Configure transforms.conf

To configure transforms.conf for ingest-time eval, follow these steps:

1. Create a transforms.conf file in the $SPLUNK_HOME/etc/system/local directory.
2. Add an ingest-time eval stanza that specifies the INGEST_EVAL expression. For example, the following INGEST_EVAL expression creates a new field called eval_user and populates the field with the lowercase version of the values in the username field:

   [myeval]
   INGEST_EVAL = eval_user=lower(username)
   

Configure props.conf

To configure props.conf for ingest-time eval, follow these steps:

1. Create a props.conf in the $SPLUNK_HOME/etc/system/local directory.
2. Add a stanza that specifies the data you want to process, such as <my_sourcetype>, and references the ingest-time eval stanza in transforms.conf. For example:

   [my_sourcetype]
   TRANSFORMS = myeval
   

   Ingest-eval transforms require a sourcetype stanza in props.conf

   You can mix eval-based transforms and regex-based transforms in props.conf in any order. The order in which you list the transforms determines when the transforms run relative to other stanzas in tranforms.conf. For example, TRANSFORMS = eval1,regex1,eval2,regex2 runs four different transforms.conf stanzas in that specific order.

Configure fields.conf

To configure fields.conf to enable search of ingest-time eval fields, do the following:

1. On the search head, create a fields.conf file in the $SPLUNK_HOME/etc/system/local directory.
2. Add a stanza that references the newly indexed field created by the INGEST_EVAL expression, as follows:

   [eval_user]
   INDEXED = True
   

For more information on how to configure index-time transforms, see Define additional indexed fields.

Examples

For basic and extended examples of eval expressions, see eval in the Search Reference.