Route internal logs from forwarders using an Edge Processor

When you configure a forwarder to send data to an Edge Processor, the forwarded data includes internal logs about how the forwarder is functioning. The Edge Processor treats these internal logs as unprocessed data, and either routes or drops the logs based on the Default destination on the Edge Processor. To prevent internal logs from being dropped or to send them to a destination other than the default destination, you can create a pipeline that routes internal logs from forwarders to a destination of your choice.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Edge Processors. Otherwise, all unprocessed data is dropped. See Add an Edge Processor.

Typically, internal logs from forwarders are sent to internal indexes in the Splunk platform such as _internal or _introspection. For more information about internal indexes, see About managing indexes in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

To route internal logs to their originally intended internal index in your Splunk platform deployment, configure your pipeline to use a Splunk platform S2S destination. Sending the internal logs from an Edge Processor to an indexer using the Splunk-to-Splunk (S2S) protocol allows the logs to be routed to the index specified in their metadata. For more information about how data from Edge Processors gets routed to an index, see How does an Edge Processor know which index to send data to?

Prerequisites

Before starting to create a pipeline, make sure that the destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed on that page, then you must add that destination to your tenant. See Add or manage destinations for more information.

Steps

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline and then select Next.
3. On the Define your pipeline's partition screen, specify a subset of the data received by the Edge Processor for this pipeline to process. To do this, you must define a partition by completing these steps:
1. Select the plus icon () next to Partition.
2. In the Add partition dialog box, configure these settings:

   Option	Enter or select the following
   Field	index
   Action	Keep
   Operator	.* match
   Value	^_.*

3. Select Apply to save your settings and close the Add partition dialog box. Then, select Next.
4. On the Add sample data screen, select Skip.
5. Select the name of the destination that you want to send data to.

   If you want to route the internal logs to their originally intended internal index in your Splunk platform deployment, then you must select a Splunk platform S2S destination. Sending the internal logs from an Edge Processor to an indexer using the S2S protocol allows the logs to be routed to the index specified in their metadata. To filter for Splunk platform S2S destinations, open the All kinds drop-down list and select Splunk platform S2S.

6. (Optional) If you selected a Splunk platform S2S or Splunk platform HEC destination, you can configure index routing:
   1. Select one of the following options in the expanded destinations panel:

      Option	Description
      Default	The pipeline does not route events to a specific index. If the event metadata already specifies an index, then the event is sent to that index. Otherwise, the event is sent to the default index of the Splunk platform deployment.
      Specify index for events with no index	The pipeline only routes events to your specified index if the event metadata did not already specify an index.
      Specify index for all events	The pipeline routes all events to your specified index.

      If you selected a Splunk platform S2S destination and you want to route the internal logs to their originally intended internal index, then select Default.

   2. If you selected Specify index for events with no index or Specify index for all events, then in the Index name field, select or enter the name of the index that you want to send your data to.

   Be aware that the destination index is determined by a precedence order of configurations. See How does an Edge Processor know which index to send data to? for more information.

7. Select Done to confirm the data destination.
   After you complete the on-screen instructions, the pipeline builder displays the SPL2 statement for your pipeline.
8. To save your pipeline, do the following:
   1. Select Save pipeline.
   2. In the Name field, enter a name for your pipeline.
   3. (Optional) In the Description field, enter a description for your pipeline.
   4. Select Save.

   The pipeline is now listed on the Pipelines page, and you can now apply it to Edge Processors as needed.

9. To apply this pipeline to an Edge Processor, do the following:
   1. Navigate to the Pipelines page.
   2. In the row that lists your pipeline, select the Actions icon () and then select Apply/Remove.
   3. Select the Edge Processors that you want to apply the pipeline to, and then select Save.

   You can only apply pipelines to Edge Processors that are in the Healthy status.

It can take a few minutes for the Edge Processor service to finish applying your pipeline to an Edge Processor. During this time, all Edge Processors that the pipeline is applied to enter the Pending status. To confirm that the process completed successfully, do the following:

• Navigate to the Edge Processors page. Then, verify that the Instance health column for the affected Edge Processors shows that all instances are back in the Healthy status.
• Navigate to the Pipelines page. Then, verify that the Applied column for the pipeline contains a The pipeline is applied icon ().

The Edge Processor that you applied the pipeline to can now send internal logs from forwarders to the destination specified in the pipeline.