Splunk® Enterprise

Search Manual

Download manual as PDF

Download topic as PDF

Quick tips for optimization

The key to fast searching is to limit the data that needs to be pulled from disk to an absolute minimum. Filter the data as early as possible in the search, so that processing is done on the minimum amount of data necessary.

Limit the data from disk

The techniques to limit the amount of data retrieved from disk range from setting a narrow time window, being as specific as possible, and retrieving the smallest number of events necessary.

Narrow the time window

One of the most effective ways to limit the data that is pulled off from disk is to limit the time range. Use the time range picker or specify time modifiers in your search to identify the smallest window of time necessary for your search.

If you need to see data from only the last hour, do not use the default time range of Last 24 hours.


If you must use a broad time range, such as Last week or All time, then use other techniques to limit the amount of data retrieved from disk.

Specify the index, source, or source type

Understanding how your data is organized is important to optimizing your searches. Take the time to learn which indexes contain your data, the sources of your data, and the source types. Knowing this information about your data helps you narrow down your searches.

  1. Run the following search.

    source=*

    This search not optimized, but it does provide an opportunity for you to learn about the data you have access to.

  2. In the Selected fields list, click on each type of field and look at the values for host, source, and sourcetype.
  3. In the Interesting fields list, click on the index field. Look at the names of the indexes that you have access to.


Whenever possible, specify the index, source, or source type in your search. When Splunk software indexes data, it automatically tags each event with a number of fields. The index, source, and source type fields are added automatically to each event as default fields. A default field is an indexed field that the Splunk software recognizes in your event data at search time. The host, source, and source type fields describe where the event originated.

Be specific

Use the most specific terms in your search that you can. If possible, avoid using wildcard characters.

For example, instead of using a wildcard character for a keyword:

*error

Use the specific keyword:

fatal_error

Here is another example.

Instead of using a wildcard character for field values:

status=404 OR status=5*

Specify each value:

status=404 OR status=500 OR status=503

Combine a source type or an index with one or more field-value pairs. For example:

sourcetype=access_* status=200 action=purchase

This search retrieves events from only your web access logs. A wildcard character, access_*, is used in the field value to match any Apache web access source type. The source types can be access_common, access_combined, or access_combined_wcookie. Two specific field-value pairs are included in the search, status=200 and action=purchase.

Limit the number of events retrieved

You can specify a limit to the number of events retrieved by using the head command. The head command retrieves only the most recent N events for a historical search, or the first N captured events for a realtime search.

Limiting the number of events retrieved is useful in several situations:

  • You are creating a search and want to determine if you are retrieving the correct events
  • You need only a subset or sample set of events for your search

For example:

sourcetype=access_* | head 1000 ...

Avoid using NOT expressions

More resources are used tracking NOT expressions than if you specify what you are looking for. Where ever possible, avoid using NOT expressions. For example, instead of using a string of NOT or != expressions:

(NOT host=d NOT host=e)

or

(host!=d OR host!=e)

Use the specific terms you are searching for:

(host=a OR host=b OR host=c).

To learn more, see Difference between NOT and !=.

Filter as soon as possible

Filter results as soon as possible before performing calculations. You can use field-value pairs and commands to filter results.

Use field-value pairs before the first pipe

Field-value pairs are indexed. Specifying field-value pairs before the first pipe is an efficient way to filter out events.

For example, in the following search the term status=404 is in a separate search:

ERROR | search status=404

Move the term status=404 before the first pipe:

ERROR status=404

Here is another example.

The second search includes the term clientip="10.0.0.0/8". There is no reason to wait to filter on that term.

ERROR | stats sum(bytes) as sum by clientip | search sum >1048576 AND clientip="10.0.0.0/8"

Move the term clientip="10.0.0.0/8" to filter out all other clientip addresses before the stats command.

ERROR clientip="10.0.0.0/8" | stats sum(bytes) by clientip | search sum > 1048576

Use filtering commands before calculating commands

Use filtering commands, such as where, before commands that perform calculations, such as eval.

For example, this search has a where command after the eval command. The search does not require the results of the eval command before the where command is run.

field1=value | eval KB=bytes/1024 | where field2=field3

Move the where command to filter the results before the eval command is processed:

field1=value | where field2=field3 | eval KB=bytes/1024

Filter unnecessary fields from search results

You can remove unnecessary fields from the search results by using commands such as fields.

Use non-streaming commands as late as possible

Postpone commands like dedup, sort, and stats as late as possible in your search. These commands are referred to as non-streaming commands. Before these commands can run, the entire result set must be returned. For example, the results cannot be sorted until all of the results are available.

For an explanation about the differences between streaming and non-streaming commands, see Types of commands.
For a list of of commands by type, see Command types in the Search Reference.

Other techniques for search optimization

There are a few other techniques that you can use to optimize your searches.

  • Use Fast Mode to increase the speed of searches by reducing the event data that the search returns. See Search modes.

See also

PREVIOUS
About search optimization
  NEXT
Write better searches

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters