Splunk® Enterprise

Admin Manual

Download manual as PDF

Download topic as PDF

Configuration parameters and the data pipeline

Data goes through several phases as it transitions from raw input to searchable events. This process is called the data pipeline and consists of four phases:

Each phase of the data pipeline relies on different configuration file parameters. Knowing which phase uses a particular parameter allows you to identify where in your Splunk deployment topology you need to set the parameter.

What the data pipeline looks like

This diagram outlines the data pipeline:

Datapipeline1 60.png

The Distributed Deployment manual describes the data pipeline in detail, in "How data moves through Splunk: the data pipeline".

How Splunk Enterprise components correlate to phases of the pipeline

One or more Splunk Enterprise components can perform each of the pipeline phases. For example, a universal forwarder, a heavy forwarder, or an indexer can perform the input phase.

Data only goes through each phase once, so each configuration belongs on only one component, specifically, the first component in the deployment that handles that phase. For example, say you have data entering the system through a set of universal forwarders, which forward the data to an intermediate heavy forwarder, which then forwards the data onwards to an indexer. In that case, the input phase for that data occurs on the universal forwarders, and the parsing phase occurs on the heavy forwarder.

Data pipeline phase Components that can perform this role
Input indexer
universal forwarder
heavy forwarder
Parsing indexer
heavy forwarder
light/universal forwarder (in conjunction with the INDEXED_EXTRACTIONS attribute only)
Indexing indexer
Search indexer
search head

Where to set a configuration parameter depends on the components in your specific deployment. For example, you set parsing parameters on the indexers in most cases. But if you have heavy forwarders feeding data to the indexers, you instead set parsing parameters on the heavy forwarders. Similarly, you set search parameters on the search heads, if any. But if you aren't deploying dedicated search heads, you set the search parameters on the indexers.

For more information, see "Components and the data pipeline" in the Distributed Deployment Manual.

How configuration parameters correlate to phases of the pipeline

This is a non-exhaustive list of configuration parameters and the pipeline phases that use them. By combining this information with an understanding of which Splunk component in your particular deployment performs each phase, you can determine where to configure each setting.

For example, if you are using universal forwarders to consume inputs, you need to configure inputs.conf parameters on the forwarders. If, however, your indexer is directly consuming network inputs, you need to configure those network-related inputs.conf parameters on the indexer.

The following items in the phases below are listed in the order Splunk applies them (ie LINE_BREAKER occurs before TRUNCATE).

Input phase

  • inputs.conf
  • props.conf
    • CHARSET
    • NO_BINARY_CHECK
    • CHECK_METHOD
    • CHECK_FOR_HEADER
    • PREFIX_SOURCETYPE
    • sourcetype
  • wmi.conf
  • regmon-filters.conf

Structured parsing phase

  • props.conf
    • INDEXED_EXTRACTIONS, and all other structured data header extractions

Parsing phase

  • props.conf
    • LINE_BREAKER, TRUNCATE, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings
    • TIME_PREFIX, TIME_FORMAT, DATETIME_CONFIG (datetime.xml), TZ, and all other time extraction settings and rules
    • TRANSFORMS which includes per-event queue filtering, per-event index assignment, per-event routing
    • SEDCMD
    • MORE_THAN, LESS_THAN
  • transforms.conf
    • stanzas referenced by a TRANSFORMS clause in props.conf
    • LOOKAHEAD, DEST_KEY, WRITE_META, DEFAULT_VALUE, REPEAT_MATCH

Indexing phase

  • props.conf
    • SEGMENTATION
  • indexes.conf
  • segmenters.conf

Search phase

  • props.conf
    • EXTRACT
    • REPORT
    • LOOKUP
    • KV_MODE
    • FIELDALIAS
    • EVAL
    • rename
  • transforms.conf
    • stanzas referenced by a REPORT clause in props.conf
    • filename, external_cmd, and all other lookup-related settings
    • FIELDS, DELIMS
    • MV_ADD
  • lookup files in the lookups folders
  • search and lookup scripts in the bin folders
  • search commands and lookup scripts
  • savedsearches.conf
  • eventtypes.conf
  • tags.conf
  • commands.conf
  • alert_actions.conf
  • macros.conf
  • fields.conf
  • transactiontypes.conf
  • multikv.conf

Other configuration settings

There are some settings that don't work well in a distributed Splunk environment. These tend to be exceptional and include:

  • props.conf
    • CHECK_FOR_HEADER, LEARN_MODEL, maxDist. These are created in the parsing phase, but they require generated configurations to be moved to the search phase configuration location.
PREVIOUS
List of configuration files
  NEXT
Back up configuration information

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters