Types of commands
As you learn about Splunk SPL, you might hear the terms streaming, generating, and transforming used to describe the types of search commands. This topic explains what these terms mean and lists the commands that fall into each category.
There are four broad categorizations for all the search commands: distributable streaming, centralized streaming, generating, transforming.
For a complete list of commands that are in each type, see Command types in the Search Reference.
Streaming and non-streaming commands
A streaming command operates on each event as it is returned by a search. Essentially one event in and one (or no) event out.
For example, the
eval command can create a new field,
full_name, to contain the concatenation of the value in the
first_name field, a space, and the value in the
... | eval full_name = first_name." ".last_name
eval command evaluates each event without considering the other events.
A non-streaming command requires the events from all of the indexers before the command can operate on the entire set of events.
For example, before the
sort command can begin to sort the events, the entire set of events must be received by the
sort command. Other examples of non-streaming commands include
Non-streaming commands force the entire set of events to the search head. This requires a lot of data movement and a loss of parallelism.
For information on how to mitigate the cost of non-streaming commands, see Write better searches in this manual.
Differences between types
|Distributable streaming||Centralized streaming||Event based non-streaming||Transforming|
|Can run on indexers||Y||N||N||N|
|Can output before final input||Y||Y||N||N|
|Outputs events if inputs are events||Y||Y||Y||N|
A streaming command operates on each event returned by a search. A distributable streaming command is a command that can be run on the indexer, which improves processing time. The other commands in a search determine if the distributable streaming command is run on the indexer:
- If all of the commands before the distributable streaming command can be run on the indexer, the distributable streaming command is run on the indexer.
- If any one of the commands before the distributable streaming command must be run on the search head, the remaining commands in the search must be run on the search head. When the search processing moves to the search head, it cannot move back to the indexer.
Distributable streaming commands can be applied to subsets of indexed data in a parallel manner. For example, the
rex command is streaming. It extracts fields and adds them to events at search time.
For a complete list of distributable streaming commands, see Streaming commands in the Search Reference.
A centralized streaming command applies a transformation to each event returned by a search. But unlike distributable streaming commands, a centralized streaming command only works on the search head. You might also hear the term "stateful streaming" to describe these commands.
A generating command fetches information from the indexes, without any transformations. Generating commands are either event-generating (distributable or centralized) or report-generating. Depending on which type the command is, the results are returned in a list or a table.
Generating commands do not expect or require an input. Generating commands are usually invoked at the beginning of the search and with a leading pipe. That is, there cannot be a search piped into a generating command. The exception to this is the search command, because it is implicit at the start of a search and does not need to be invoked.
For a complete list of generating commands, see Generating commands in the Search Reference.
A transforming command orders the results into a data table, that is, it "transforms" the specified cell values for each event into numerical values that Splunk software can use for statistical purposes. Transforming commands are not streaming. Also, they are required to transform search result data into the data structures required for visualizations such as column, bar, line, area, and pie charts.
For more information about transforming commands and their role in create statistical tables and chart visualizations, see About transforming commands and searches in the this manual.
For a complete list of transforming commands, see Transforming commands in the Search Reference.
There are a handful of commands that do not fit into these categories. These commands are non-transforming, not distributable, and not streaming: sort, eventstats, some modes of dedup, and some modes of cluster.
Types of searches
Search with Splunk Web, CLI, or REST API
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3