Documentation > Splunk > Developing Dashboards, Views, and Apps for Splunk Web > Understanding modules and processors

Developing Dashboards, Views, and Apps for Splunk Web

 


Overview
User Interface
Data Inputs
Data Outputs
Management

Understanding modules and processors

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Contents

Understanding modules and processors

Default processors and pipelines

The universal pipeline, or the parsing pipeline, is where events are input, processed and output to the indexing pipeline.


Modules, pipelines and queues

Below is a list of the processors, in order executed, that make up the default universal pipeline. You can see this in the pipeline "parsingPipeline" in $SPLUNK_HOME/etc/myinstall/splunkd.xml file.


readerIn : Queue Input processor, data comes in here


utf8 : UTF8 processor


linebreaker : Line Breaking


aggregator : Line Merging/Date Extraction


regexreplacement : Regex Extraction


typing : Event Typing


clusterer : Meta Event Creation


sendOut : Queue Output, data sent to next pipeline


Pipeline data keys

Changing the default parsing and indexing sequence

The processors that make up what is called the Universal pipeline can be reconfigured or replaced by creating a new module with your custom processing pipeline.


The Splunk processor loading architecture supports the ability to insert a processor before, after or instead of another processor. If you are extending splunk by providing your own processor or wish to change the processing pipelines defined in splunkd.xml it is recommended that you define your processor in a new module and use the "insertBefore", "insertAfter", or "replace" attribute.


For example, lets suppose you wanted to add a processor "replaceProcessor" just before the "indexer" processor:


  1. Create a new module directory in $SPLUNK_HOME/etc/modules/replaceProcessor
  1. Create a config.xml file in the new module directory
  1. Define a module configuration with a list of processors - note that for this case you do not use the <pipeline> ... </pipeline> tags because you are not defining a new pipeline but changing an existing one.
  1. Define your processor(s) in the config.xml file and specify the action and target and target pipeline.

Here is the config.xml for an example replaceProcessor (uses an example urlencodeprocessor.) It will insert the "replaceProcessor" before the "indexer" processor in the "indexerPipe" pipeline: 


<module>
   <processor name="replaceProcessor" plugin="urlencodeProcessor" action="insertBefore" target="indexer" pipelineTarget="indexerPipe" >
       <config></config>
    </processor>
</module>

Adding custom processors

Custom processors can provide data input and handling not available by other methods. You can add to the existing default processing or replace it by changing the configuration of Splunk pipelines.

Retrieved from "http://docs.splunk.com/Documentation/Splunk/3.0.1/Developer/UnderstandingModulesAndProcessors"

This documentation applies to the following versions of Splunk: 3.0 , 3.0.1 , 3.0.2 , 3.1 , 3.1.1 , 3.1.2 , 3.1.3 , 3.1.4 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!