How data moves through the Edge Processor solution

Data moves through the Edge Processor solution as follows:

1. A tool, machine, or piece of software in your network generates data such as event logs or traces.
2. An agent, such as a Splunk forwarder, receives the data and then sends it to an Edge Processor. Alternatively, the device or software that generated the data can send it to an Edge Processor without using an agent.
3. The Edge Processor filters and transforms the data, and then sends the resulting processed data to a destination such as an indexer then into a Splunk index.

By default, Edge Processors route processed data to destinations based on any pipelines you applied. If there are no applicable pipelines, then unprocessed data is either dropped or routed to the default destination specified in the configuration setting of the Edge Processor. For more information about how data moves through an Edge Processor, see Partitions.

If you don't specify a default destination, then Edge Processors drop unprocessed data.

As the Edge Processor receives and processes data, it measures metrics indicating the volume of data that was received, processed, and sent to a destination. These metrics are stored in the _metrics index of the Splunk Cloud Platform deployment that is connected to your tenant. The Edge Processor service surfaces the metrics in the dashboard, providing detailed overviews of the amount of data that is moving through the system.

Partitions

Each Edge Processor instance merges the received data into an internal dataset before processing and routing that data. A partition is a subset of data that is selected for processing by a pipeline. Each pipeline that you apply to an Edge Processor creates a partition from the internal dataset. For information about how to specify a partition when creating a pipeline, see Create pipeline for Edge Processors.

The partitions that you create and the configuration of your Edge Processor determines how the Edge Processor routes the received data and whether any data is dropped:

• The data that the Edge Processor receives is defined as processed or unprocessed based on whether there is at least one partition for that data. For example, if your Edge Processor receives Windows event logs and Linux audit logs, but you only applied a pipeline for Windows event logs, then those Windows event logs are selected in a partition and considered to be processed while the Linux audit logs are considered to be unprocessed.
• Each pipeline creates a partition of the incoming data based on specified conditions, and only processes data that meets those conditions. Any data that does not meet those conditions is considered to be unprocessed.
• If you configure your pipeline to filter the processed data, the data that is filtered out gets dropped.
• If you configure your Edge Processor to have a default destination, then the unprocessed data goes to that default destination.
• If you do not set a default destination, then the unprocessed data is dropped.

The following is a diagram of the Edge Processor data pathway.

See also

• Create pipelines for Edge Processors
• How the Edge Processor solution transforms data
• System architecture of the Edge Processor solution