Built-in optimization
The Splunk software includes built-in optimizations that analyze and process your searches for maximum efficiency.
One goal of these optimizations is to filter results as early as possible. Filtering reduces the amount of data to process for the search. Early filtering avoids unnecessary lookups and evaluation calculations for events that are not part of the final search results.
Another goal is to process as much as possible in parallel on the indexers. The built-in optimizations can reorder search processing, so that as many commands as possible are run in parallel on the indexers before sending the search results to the search head for final processing. You can see the reordered search using the Job Inspector. See Analyze search optimizations.
Predicate optimization
Predicates act as filters to remove unnecessary events as your search is processed. The earlier in the search pipeline that a predicate is applied, the more efficient is the use of system resources. If a predicate is applied early, when events are fetched, only the events that match the predicate are fetched. If the same predicate is applied later in the search, then resources are wasted fetching more events than are necessary.
There are several built-in optimizations that focus on predicates:
- Predicate merge
- Predicate pushdown
- Predicate splitting
Types of predicates
There are two types of predicates, simple predicates and complex predicates.
- A simple predicate is a single conditional of the form
field = value
. - A complex predicate is a combination of several conjunctions (AND and OR) and disjunctions (NOT). For example,
Field1 = Value1 OR Field2 = Value2 AND Field3 = Value3
.
A complex predicate can be merged or split apart to optimize a search.
In Splunk SPL, there are two commands that perform predicate-based filtering, where
and search
.
An example of using the where
command for filtering is:
index="_internal" | where bytes > 10
An example of using the search
command for filtering is:
index="_internal" | search bytes > 10
Search-based predicates are a subset of where-based predicates. In other words, search-based predicates can be replaced by where-based predicates. However, where-based predicates cannot be replaced by search-based predicates.
Predicate merge
The predicate merge optimization takes two predicates and merges the predicates into one predicate. For example, consider the following search:
index=main | search a > 10 AND fieldA = "New"
There are two search commands in this example. The search
command before the pipe, which is implied at the beginning of every search, and the explicit search
command after the first pipe. With the predicate merge optimization, the predicates in the second search
command are merged with the predicates in the first search
command. For example:
index=main AND a > 10 AND fieldA = "New"
In some cases, predicates used with the where
command can also be merged. For example, consider the following search:
index=main | where fieldA = "New"
With the predicate merge optimization, the predicates specified with the where
command are merged with the predicates specified with the search
command.
index=main AND fieldA = "New"
Issues with field extractions and predicate merge
An inline field extraction requires special handling if the regular expression pattern extracts a subtoken. The field must be set to indexed=false
in the fields.conf
file. See Example inline field extraction configurations in the Knowledge Manager Manual.
Consider the following sample event:
Mon Apr 17 16:08:16 2017 host=10.10.1.1 Login name=John SUCCESS FRANCE
You create an extracted field called country
that uses the following regular expression:
SUCCESS\s+?\S{3}
SUCCESS
matches the characters SUCCESS literally and is case sensitive.\s
matches any whitespace character (space, tab, new line).+?
is a quantifier that matches between one and unlimited times, the fewest needed to match the pattern.\S
matches any non-whitespace character.{3}
is a quantifier that matches exactly 3 non-whitespace characters
For the sample event, the following regular expression extracts the first 3 characters of the word FRANCE or FRA
. The extraction FRA
is a subtoken of the indexed term FRANCE.
When you search using the extracted field, for example:
index=main | search country=FRA
The search is optimized, using the predicate merge optimizer:
index=main country=FRA
However, the search returns no results because FRA
is not part of the index. FRANCE
is the indexed term.
To overcome this issue, you must add the following setting to the fields.conf
file:
[country] indexed=false
Alternatively, you can disable the built-in optimizations. See Optimization settings.
Predicate pushdown
The predicate pushdown optimization analyzes a search and reorders the search processing so that predicates are processed as early as possible.
Example of a simple predicate pushdown
You perform the following search:
sourcetype=access* (status=401 OR status=403) | lookup usertogroup user OUTPUT group | where src_category="email_server"
The sourcetype=access* (status=401 OR status=403)
portion of the search retrieves 50,000 events. The lookup
is performed on all 50,000 events. Then the where
command is applied and filters out events that are not src_category="email_server"
. The result is that 45,000 events are discarded and 5,000 events are returned in the search results.
If the search criteria used with the where
command is applied before the lookup
, more events are filtered out of the results. Only 5,000 events are retrieved from disk before the lookup
is performed.
With predicate pushdown, the search is reordered for processing. The where
command is eliminated by moving the search criteria src_category="email_server"
before the first pipe.
sourcetype=access* (status=401 OR status=403) src_category="email_server" | lookup usertogroup user OUTPUT group
Example of a complex predicate pushdown
Consider the following search fragment:
... | eval x=if(isnull(x) OR x=="", "missing", x) | where x = "hello"
In this situation, the eval
command is assigning a default value when a field is null or empty. This is a common pattern in Common Information Model (CIM) data models.
The built-in optimization reorganizes the search criteria before processing the search. The where
command is moved before the eval
command.
... | where x = "hello" | eval x=if(isnull(x) OR x=="", "missing", x)
Predicate splitting
Predicate splitting is the action of taking a predicate and dividing, or splitting, the predicate into smaller predicates. The predicate splitting optimizer can then move the smaller predicates, when possible, to an earlier place in the search.
Consider the following search:
index="_internal" component = "SearchProcess" | eval a = (x + y) | where a > 200 AND x < 10
Because field a
is generated as part of the eval
command, it cannot be processed earlier in the search. However, field x
exists in the events and can be processed earlier. The predicate splitting optimization separates the components of the where
portion of the search. The search is reordered to process eligible components earlier.
index="_internal" component = "SearchProcess" | where x < 10 | eval a = (x + y) | where a > 200
The portion of the where
command x < 10
is moved to before the eval
command. This move reduces the number of results that the eval
command must process.
Projection elimination
This optimization analyzes your search and determines if any of the generated fields specified in the search are not used to produce the search results. If generated fields are identified that can be eliminated, an optimized version of the search is run. Your search syntax remains unchanged.
For example, consider the following search:
index=_internal | eval c = x * y / z | stats count BY a, b
The c
field that is generated by eval c = x * y / z
is not used in the stats
calculation. The c
field can be eliminated from the search.
Your search syntax remains:
index=_internal | eval c = x * y / z | stats count BY a, b
But the optimized search that is run is:
index=_internal | stats count BY a, b
Here is another example:
| tstats count FROM datamodel=internal_audit_logs
For buckets whose data models are not built, this expands to the following fallback search:
| search ((index=* OR index=_*) index=_audit) | eval nodename = "Audit" | eval is_searches=if(searchmatch("(action=search NOT dmauditsearch)"),1,0), is_not_searches=1-is_searches, is_modify=if(searchmatch("(action=edit_user OR action=edit_roles OR action=update)"),1,0), is_not_modify=1-is_modify | eval nodename = if(nodename == "Audit" AND searchmatch("(action=search NOT dmauditsearch)"), mvappend(nodename, "Audit.searches"), nodename) | eval is_realtime=case(is_realtime == 0, "false", is_realtime == 1, "true", is_realtime == "N/A", "false"), search_id=replace(search_id,"'",""), search=replace(search,"'",""), search_type=case((id LIKE "DM_%" OR savedsearch_name LIKE "_ACCELERATE_DM%"), "dm_acceleration", search_id LIKE "scheduler%", "scheduled", search_id LIKE "rt%", "realtime", search_id LIKE "subsearch%", "subsearch", (search_id LIKE "SummaryDirector%" OR search_id LIKE "summarize_SummaryDirector%"), "summary_director", 1=1, "adhoc") | rename is_realtime AS Audit.searches.is_realtime search_id AS Audit.searches.search_id search AS Audit.searches.search search_type AS Audit.searches.search_type | eval nodename = if(nodename == "Audit" AND searchmatch("(action=edit_user OR action=edit_roles OR action=update)"), mvappend(nodename, "Audit.modify"), nodename) | rename action AS Audit.action info AS Audit.info object AS Audit.object operation AS Audit.operation path AS Audit.path user AS Audit.user exec_time AS Audit.exec_time result_count AS Audit.result_count savedsearch_name AS Audit.savedsearch_name scan_count AS Audit.scan_count total_run_time AS Audit.total_run_time is_searches AS Audit.is_searches is_not_searches AS Audit.is_not_searches is_modify AS Audit.is_modify is_not_modify AS Audit.is_not_modify | addinfo type=count label=prereport_events | stats count
This search can be optimized to this syntax:
| search ((index=* OR index=_*) index=_audit) | stats count
Eliminating unnecessary generated fields, or projections, leads to better search performance.
Event types and tags optimization
Event typing and tagging can take a significant amount of search processing time. This is especially true with Splunk Enterprise Security where there are many event types and tags defined. The more event types and tags that are defined in the system, the greater the annotation costs.
The built-in event types and tags optimization applies to transforming searches and data model acceleration searches.
Transforming searches
When a transforming search is run without the built-in optimization, all of the event types and tags are uploaded from the configuration system to the search processor. This upload happens whether the search requires the event types and tags or not. The search processing attempts to apply the event types and tags to each event.
For example, consider the following search:
index=_internal | search tag=foo | stats count by host
This search needs only the tag foo
. But without the optimization, all of the event types and tags are uploaded.
The built-in optimization solves this problem by analyzing the search and only uploading the event types and tags that are required. If no event types or tags are needed, none are uploaded which improves search performance.
For transforming searches, this optimization is controlled by several settings in the limits.conf
file, under the [search_optimization::required_field_values]
stanza. These settings are turned on by default. There is no need to change these settings, unless you need to troubleshoot an issue.
Data model acceleration searches
Another problem that can be solved by the event type and tags optimization is specifically targeted at data model acceleration. With a configuration setting in the datamodels.conf
file, you can specify a list of tags that you want to use with the data model acceleration searches. You specify the list under the [dm.name]
stanza with the tags_whitelist
setting.
For detailed information about the tags_whitelist
setting, including usage examples, see Set a tag whitelist for better data model performance in the Knowledge Manager Manual.
Analyze search optimizations
You can analyze the impact of the built-in optimizations by using the Job Inspector.
Determine search processing time
You can use the Job Inspector to determine whether the built-in search optimizations are helping your search to complete faster.
- Run your search.
- From the search action buttons, select Job > Inspect job.
- In the Job Inspector window, review the message at the top of the window. The message is similar to "The search has completed and returned X results by scanning X events in X seconds." Make note of the number of seconds to complete the search.
- Close the Job inspector window.
- In the Search bar, add
|noop search_optimization=false
to the end of your search. This turns off the built-in optimizations for this search. - Run the search.
- Inspect the job and compare the message at the top of the Job Inspector window with the previous message. This message specifies how many seconds the search processing took without the built-in optimizations.
View the optimized search
You can compare your original search with the optimized search that is created when the built-in optimizations are turned on.
- Run your search.
- From the search action buttons, select Job, Inspect job.
- In the Job Inspector window, expand the Search job properties section and scroll to the normalizedSearch property. This property shows the internal search that is created based on your original search.
- Scroll to the optimizedSearch property. This property shows the search that is created, based on the normalizedSearch, when the built-in optimizations are applied.
Optimization settings
By default, the built-in optimizations are turned on.
Turn off optimization for a specific search
In some very limited situations, the optimization that is built into the search processor might not optimize a search correctly. In these situations, you can troubleshoot the problem by turning off the search optimization for that specific search.
Turning off the search optimization enables you to determine if the cause of unexpected or limited search results is the search optimization.
You turn off the built-in optimizations for a specific search by using the noop
command.
You add noop search_optimization=false
at the end of your search. For example:
| datamodel Authentication Successful_Authentication search | where Authentication.user = "fred" | noop search_optimization=false
Turn off all optimizations
You can turn off all of the built-in optimizations for all users.
- Splunk Cloud Platform
- To change the built-in optimizations, request help from Splunk Support. If you have a support contract, file a new case using the Splunk Support Portal at Support and Services. Otherwise, contact Splunk Customer Support.
- Splunk Enterprise
- To change the built-in optimizations, follow these steps.
- Prerequisites
- Only users with file system access, such as system administrators, can change the built-in optimizations using configuration files.
- Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.
- Steps
- Open or create a local limits.conf file for the Search app at
$SPLUNK_HOME/etc/apps/search/local
.- Open or create a local limits.conf file for the Search app at
- Under the
[search_optimization]
stanza, set enabled=false.- Under the
See also
Write better searches | Search normalization |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12
Feedback submitted, thanks!