Splunk Cloud Platform

Use Edge Processors

How the destination for Edge Processor works

In order to send data from an Edge Processor to a storage location such as an index or an Amazon S3 bucket, you must define the location as a destination in the Edge Processor service. Each destination contains the connection information necessary for allowing an Edge Processor to send data to a given location.

The steps for adding a destination to the Edge Processor service varies depending on whether the destination is part of the Splunk Cloud Platform deployment that's connected to your cloud tenant:

You can confirm the destinations that are available by checking the Destinations page, and view additional details about a given destination by selecting it on the Destinations page.

What happens to my data if a destination becomes unavailable?

Edge Processors currently provide no data delivery guarantees. However, to help prevent data loss, the Edge Processor instance holds data in a queue if it is unable to send data to a destination or if it receives more data than it can send. If the queue fills up before the destination is available again, then the Edge Processor back pressures the data until it is ready to be sent to the destination and will continue to attempt to put data in the queue unless the Edge Processor needs to restart or shut down. If the Edge Processor instance shuts down or restarts while data is being sent, data cannot be written to a persistent queue which can cause data loss.

Queued data is stored on the hard drive of the Edge Processor host. By default, the queue is configured to hold up to 10000 batches of events. Depending on which receiver you use, each batch can contain various amounts of events ranging from 1 to 128 events. The amount of data contained and how quickly the queue fills up varies depending on the rate at which the Edge Processor is receiving data.

If your pipeline uses either the branch or route command and one of the queues for your destination is full, then data may be delivered more than once for the other healthy destinations causing data duplication.

Once the destination is available again, the Edge Processor sends the queued events to the destination. It might take some time for newer data to be processed by an Edge Processor as the data in the queue is prioritized first. If you want to adjust the size of the queue, see the solution instructions in An Edge Processor fails to send data, and logs a "Dropping data because sending_queue is full" error.

Last modified on 25 September, 2024
Obtain TLS certificates for data sources and Edge Processors   Add or manage destinations

This documentation applies to the following versions of Splunk Cloud Platform: 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403, 9.2.2406 (latest FedRAMP release)


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters