Splunk® Enterprise

Search Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Built-in optimization

The Splunk software includes some built-in optimizations that analyze and process your searches for maximum efficiency.

One goal of these optimizations is to filter results as early as possible. Filtering reduces the amount of data that needs to be processed. Early filtering avoids unnecessary lookups to enrich events that will not end up in the final search results. Early filtering avoids unnecessary eval calculations on events that will not end up in the final results.

Another goal is to process as much as possible in parallel on the indexers. The built-in optimizations can reorder search processing, so that as many commands as possible are run in parallel on the indexers before sending the search results to the search head for final processing.

Predicate pushdown

Predicate pushdown is an optimization that analyzes a search and reorders the search processing so that predicates are processed as early as possible.

Predicates act as filters to remove unnecessary events as your search is processed. The earlier the predicates are applied when a search is processed, the more efficient the use of system resources. If a predicate is applied early, when events are fetched, only the events that match the predicate are fetched. If the same predicate is applied later in the search, then resources are wasted fetching more events than are needed.

Types of predicates

There are two types of predicates, simple predicate and complex predicate.

  • A simple predicate is a single conditional of the form field = value.
  • A complex predicate is a combination of several conjunctions and disjunctions. For example Field1 = Value1 OR Field2 = Value 2 AND Field3 = Value 3. A complex predicate can be split apart to push down part or all of the predicate.


In Splunk SPL, there are two commands that perform predicate-based filtering, where and search.

An example of using the where command for filtering is:

index="_internal" | where bytes > 10

An example of using the search command for filtering is:

index="_internal" | search bytes > 10

Search-based predicates are a subset of where based predicates. In other words, search-based predicates can be replaced by where-based predicates. However, where-based predicates cannot always be replaced by search-based predicates.

Example of simple predicate pushdown

You perform the following search.

sourcetype=access* (status=401 OR status=403) | lookup usertogroup user OUTPUT group | where src_category="email_server"

This image shows three blocks in a row. Each block represents a command in the search. The commands are Search, Lookup, and Where. The Where command is last.

The sourcetype=access* (status=401 OR status=403) portion of the search retrieves 50,000 events. The lookup is performed on all 50,000 events. Then the where command is applied and filters out events that are not src_category="email_server" . The result is that 45,000 events are discarded and 5,000 events are returned in the search results.

If the search criteria used with the where command is applied before the lookup, more events are filtered out of the results. Only 5,000 events are retrieved from disk before the lookup is performed.

With predicate pushdown, the search is reordered for processing. The where command is eliminated by moving the search criteria src_category="email_server" before the first pipe.

sourcetype=access* (status=401 OR status=403) src_category="email_server" | lookup usertogroup user OUTPUT group

This image shows two blocks in a row. Each block represents a command in the search. The commands are Search (with Where) and Lookup.

Example of complex predicate pushdown

Consider the following search fragment.

... | eval x=if(isnull(x) OR x=="", "missing", x) | where x = "hello"

In this situation, the eval command is assigning a default value when a field is null or empty. This is a common pattern in Common Information Model (CIM) datamodels.

The built-in optimization reorganizes the search before processing to move the where command before the eval command.

... | where x = "hello" | eval x=if(isnull(x) OR x=="", "missing", x)

Analyze search optimizations

You can analyze the impact of the built-in optimizations by using the Job Inspector.

Determine search processing time

You can determine how long a search took to run.

  1. Run your search.
  2. From the search action buttons, select Job, Inspect job.
  3. In the Job Inspector window, review the message at the top of the window. The message is something like "The search has completed and returned X results by scanning X events in X seconds." Make note of the number of seconds to complete the search.
  4. Close the Job inspector window.
  5. In the Search bar, add |noop search_optimization=false to the end of your search. This turns off the built-in optimizations for this search.
  6. Run the search.
  7. Inspect the job and compare the message at the top of the Job Inspector window with the previous message. This message specifies how many seconds the search processing took without the built-in optimizations.

View the optimized search

You can compare your original search with the optimized search that is created when the built-in optimizations are turned on.

  1. Run your search.
  2. From the search action buttons, select Job, Inspect job.
  3. In the Job Inspector window, expand the Search job properties section and scroll to the normalizedSearch property. This property shows the internal search that is created based on your original search.
  4. Scroll to the optimizedSearch property. This property shows the search that is created, based on the normalizedSearch, when the built-in optimizations are applied.

Optimization settings

By default, the built-in optimizations are turned on.

Turn off optimization for a specific search

In some very limited situations, the optimization that is built into the search processor might not optimize a search correctly. In these situations, you can troubleshoot the problem by turning off the search optimization for that specific search.

Turning off the search optimization enables you to determine if the cause of unexpected or limited search results is the search optimization.

You turn off the built-in optimizations for a specific search by using the noop command.

You add noop search_optimization=false at the end of your search. For example:

| datamodel Authentication Successful_Authentication search | where Authentication.user = "fred" | noop search_optimization=false

Turn off all optimizations

You can turn off all of the built-in optimizations for all users.

Prerequisites

  • Only users with the admin role, or a role with equivalent permissions, can turn off the built-in optimizations.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Steps

  1. Open the local limits.conf file for the Search app. For example, $SPLUNK_HOME/etc/apps/<app_name>/local.
  2. Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
  3. Under the [search_optimization] stanza, set enabled to false.

See also

PREVIOUS
Write better searches
  NEXT
About retrieving events

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters