How distributable and non-distributable commands work in Splunk Analytics for Hadoop (and what works best)

Splunk Analytics for Hadoop reaches End of Life on January 31, 2025.

Distributable search commands are the most effective commands in Splunk Analytics for Hadoop reports because they can be distributed to search heads and virtual indexes. Generally, non-distributable commands only work on local indexes and are not as effective on virtual indexes.

You can create searches across different index types that use both distributable and non-distributable commands as long as you keep in mind that these such a search returns all data from the local indexes but limited data from the virtual indexes.

This topic discusses the types of commands that work best with Splunk Analytics for Hadoop and commands that should be reserved for use with the Splunk Enterprise local directories.

Smart mode searches

Search modes control the amount or type of data that the search returns.

Smart mode is the default and recommended setting for VIX searches. It maintains search behavior based on whether your search contains transforming commands. When searching virtual indexes we recommend that you search in smart mode, as it is more efficient.

If you use verbose mode to search a VIX, note that Splunk Analytics for Hadoop does not start a MapReduce job for that search. This is because verbose mode searches search for all events as well as any reports that you might be running. The benefits of MapReduce jobs in that case are minimal and in some cases can have a negative impact on your searches.

To learn more about Splunk Enterprise search modes, see In the Search Manual:

• Set search mode to adjust your search experience

Distributable commands

Distributable commands are commands that can be run on a local indexer but can also be distributed to search heads and virtual indexes. They run on the indexer in Enterprise and the DataNode/TaskTracker.

Commands that work best with virtual indexes are:

• Distributable streaming commands: This is any streaming command that operates on each event returned by a search. Distributable streaming commands include:
  • bin (if it's called with an explicit span)
  • convert
  • eval
  • extract (kv)
  • fields
  • lookup (if not local=t)
  • mvexpand
  • multikv
  • rename
  • regex
  • replace
  • rex
  • search
  • strcat
  • tags
  • typer
  • where

• Distributable generating commands: Event-generating commands that are distributable return an events list or a table of results. Generating commands are usually invoked at the beginning of the search and with a leading pipe. There cannot be a search piped into a generating command. (The exception is the search command, because it is implicit at the start of a search and does not need to be invoked.) Distributable event-generating commands include:
  • search
  • metadata

Non-distributable commands

Non-distributable commands (also referred to as non-streaming commands) require all data to come back to the local indexer. They are not particularly effective commands for searching virtual indexes.

Non-streaming commands are best reserved for when part of your searching involves local indexes in some capacity. Searches run across local and virtual indexes that use non-streaming commands will be applied to local indexes but not the virtual indexes included in the search.

Types of non-distributable or non-streaming commands are:

• Centralized streaming commands: These commands are sometimes referred to as "stateful streaming" commands and include:
  • head
  • streamstats
  • Some modes of dedup
  • Some modes of cluster

• Transforming streaming commands: A transforming command orders events into values that Splunk can use for statistical purposes and include:
  • chart
  • timechart
  • stats
  • top
  • rare
  • contingency
  • highlight
  • typer
  • addtotals when it is used to calculate column totals

• Non-distributable Generating commands: Generating commands that are either centralized event-generating or report-generating do not work on virtual indexes. You cannot export data from any searches that contain a reporting command.
  • Centralized event-generating commands include:
    • loadjob
    • inputcsv
    • inputlookup
  • Report-generating commands include:
    • dbinspect
    • datamodel
    • metadata
    • pivot
    • tstats

Other commands

There are a handful of commands that do not fit into these categories. These commands are non-reporting, not distributable, and not streaming: sort, eventstats, some modes of dedup, and some modes of cluster.