Troubleshoot the Edge Processor solution

Review this page if you are having difficulties with sending data through the Edge Processor solution. If the problem that you're experiencing is not described on this page, you can find more information by doing the following:

• Review the list of known issues in the product. See Known issues.
• Check the logs associated with your Edge Processor. See View logs for the Edge Processor solution.

If the problem persists, contact your Splunk representative for assistance. To help expedite the support process, you can generate a diagnostic report and send it to your Splunk representative.

Generate a diagnostic report for an Edge Processor instance

You can run the edge_diagnostic tool to generate a diagnostic report for a specific Edge Processor instance. The diagnostic report contains logs about the performance and activity of the Edge Processor instance. Include this report when contacting your Splunk representative for assistance.

1. On the host machine of your Edge Processor instance, download the package containing the edge_diagnostic tool.

   curl -o 'edge-diagnostic-linux.tar.gz' 'https://beam.scs.splunk.com/acies/diagnostic/edge-diagnostic-linux.tar.gz'

2. Extract the edge_diagnostic tool into <install_directory>, where <install_directory> is the installation directory of the Edge Processor instance.

   tar -xf edge-diagnostic-linux.tar.gz -C <install_directory>

3. Navigate to the installation directory.

   cd <install_directory>

4. Validate the edge_diagnostic tool by running the following command and confirming whether the returned value matches the expected value:

   sha256sum edge_diagnostic_linux_amd64

   The expected value is:

   1494759b52a478283ee22fede32ab87259bb7e608d8d97823a814eda34c504ea
   

   If these values don't match, that indicates that the edge_diagnostic tool is invalid. This problem can occur if the package download did not complete as expected. If this occurs, delete the package and the extracted edge_diagnostic tool, and then try again from step 1.

5. Use the edge_diagnostic tool to generate a diagnostic report. By default, the tool generates a file named edge-diag-<host_name>-<timestamp>.tar.gz, where <host_name> is the host name of the machine that the Edge Processor instance is running on and <timestamp> is a timestamp indicating when the file was generated. You can use the -out option to specify a different file name.
   • To generate a diagnostic report using the default settings, run this command:

     ./edge_diagnostic_linux_amd64

   • To specify a different output file name, run this command, where <file_name> is the name that you want to use:

     ./edge_diagnostic_linux_amd64 -out <file_name>.tar.gz

   The edge_diagnostic tool generates a diagnostic report file in the current directory. While the tool is running, it returns INFO logs about its status.

6. (Optional) If any unexpected behavior occurs with the edge_diagnostic tool, you can get more information about the status of the tool by running it with the -verbose option. This option causes the tool to print DEBUG level logs about its status while generating the diagnostic report.

   ./edge_diagnostic_linux_amd64 -verbose

When contacting your Splunk representative for assistance, send them a copy of the generated file.

Edge Processor installation returns a "401 Unauthorized" error

When you try to install an Edge Processor instance, the installation fails. Additionally, when you review the <install_directory>/var/log/supervisor.log file on the host machine where you attempted the installation, you see an error message similar to the following:

2024/03/26 06:52:16 token refresh failed, will retry: sign token: {"status_code":401,"status":"401 Unauthorized"}

Cause

The security token that's required during the installation process is expired. This problem can happen because the system clock on the host machine is set to an incorrect time, causing the token to expire prematurely. The token is valid for 5 minutes only.

Solution

Make sure that the system clock on the host machine is correct, and synchronize it to a Network Time Protocol (NTP) server. Refer to the documentation for your operating system for more information.

An Edge Processor is not detecting my forwarder

If the Data sources pane in the detailed view of your Edge Processor does not include the data from your forwarder, or the pane displays a "No data sources to display" message, check the time range that's specified in the Metrics drop-down list. Confirm that your forwarders were sending data during that time window. If necessary, increase the time range and try again.

If data sources are still missing, then on the machine where your forwarder is installed, confirm the following:

1. The outputs.conf file defines a target group for the Edge Processor.
2. The outputs.conf, inputs.conf, and props.conf files don't define any advanced routing or filtering settings that would prevent data from being forwarded to the target group for the Edge Processor.
3. The forwarder is running and has an "active forward" sending data to the Edge Processor. To confirm this, run this command from the $SPLUNK_HOME/bin directory:

   splunk list forward-server

An Edge Processor instance that was previously "Healthy" is now "Disconnected"

When viewing details about an Edge Processor, you notice that an instance that previously had the Healthy status now has the Disconnected status.

Cause

The Edge Processor service lost contact with that instance. This problem can happen for a variety of reasons, such as the host machine of the instance going down or the communication between the instance and the Edge Processor service getting blocked.

This problem can also happen if you remove an Edge Processor instance from its host machine using any method other than the uninstallation command provided in the Edge Processor service. In this case, the service fails to detect that the instance has been removed.

Solution

To determine the root cause of the problem, check the logs for the supervisor that's associated with the disconnected Edge Processor instance.

1. Log in to the host machine of the disconnected Edge Processor instance.
2. Navigate to the <install_directory>/var/log directory, where <install_directory> is the installation directory of the Edge Processor instance.
3. Review the logs in the supervisor.log file.

If the Edge Processor instance was removed using a method other than the uninstallation command, then you need to reinstall the instance and then uninstall it using the appropriate command. Using the command provided in the Edge Processor service allows the service to detect this change and stop listing the instance in the Edge Processor details.

1. Reinstall the instance:
   1. On the Edge Processors page, select the Edge Processor that has Disconnected instance.
   2. In the panel that contains your Edge Processor details, select Manage instances.
   3. Select the Install/uninstall tab, and then expand the Step 1: Run commands to install/uninstall instances section.
   4. Select Install to view the commands for downloading and installing an Edge Processor instance on a Linux machine, and then select Copy to clipboard.
   5. On the machine that previously hosted the removed instance, open a command-line interface in a directory of your choice and then paste and run the commands. When the installation is complete, you will see the following message:

      splunk-edge.service - Splunk edge starter
          Loaded: loaded (/etc/systemd/system/splunk-edge.service, enabled)
          Active: active (running)

      This command contains sensitive information about your cloud environment. Do not share this command with anyone except your Splunk representative or trusted members in your organization.

   6. Verify that your instance is healthy by checking its status on the Edge Processors page or in the detailed view of your Edge Processor. It may take up to 1 minute for the status to change to Healthy.
2. Uninstall the instance using the command provided in the Edge Processor service:
   1. In the Manage instances panel for your Edge Processor, on the Install/uninstall tab, expand the Step 1: Run commands to install/uninstall instances section and then select Uninstall.
   2. To copy the command for uninstalling an instance from a Linux machine, select Copy to clipboard.
   3. On the host machine where you reinstalled the instance, open a command-line interface in a directory of your choice and then paste and run the command.

An Edge Processor instance is in the "Warning" status

When viewing details about an Edge Processor, you notice that an instance is in the Warning status.

Cause

This Edge Processor instance is consuming more resources than what is acceptable based on the CPU threshold and Memory threshold settings configured in the Edge Processor service.

Solution

First, confirm that the CPU threshold and Memory threshold settings are configured to allow a reasonable amount of resource usage. The recommended amount is 80% of the total allocated resources.

1. On the Edge Processors page, select Shared settings.
2. Select the Other settings tab.
3. Check the values specified in the CPU threshold and Memory threshold fields.
4. If you want to adjust those settings, do the following:
   1. Select Edit.
   2. In the CPU threshold field, enter the percentage of the total allocated CPU power that an Edge Processor instance can use before it enters the Warning state.
   3. In the Memory threshold field, enter the percentage of the total allocated memory that an Edge Processor instance can use before it enters the Warning state.
   4. Select Save to save your changes, and then select Edge Processors to return to the Edge Processor management page.

If the problem persists, then try the following solutions.

Reduce CPU usage per instance by scaling up the Edge Processor

Add more instances to the Edge Processor. Having more instances associated with the Edge Processor reduces the amount of CPU processing power that any one instance needs to consume.

For more information, see Add more instances to an Edge Processor.

Reinstall the instance on a machine that has more memory

Install a new Edge Processor instance on a host machine that has more memory available. Then, uninstall the instance that has the Warning status. Make sure to update your data sources to start sending data to the new instance and stop sending data to the uninstalled instance.

For more information, see Add more instances to an Edge Processor and Uninstall an Edge Processor instance.

An Edge Processor instance is in the "Error" status

When viewing details about an Edge Processor, you notice that an instance is in the Error status.

Cause

The Error status indicates that something is wrong with the Edge Processor instance, but it is still in contact with the Edge Processor service. This problem can occur for a variety of reasons. For example, this status can occur if the Edge Processor is configured incorrectly, or if an internal component is stuck in a restart loop.

To get more information about the root cause of an Error status, view the logs for the instance. The logs are located on the host machine of the instance in <install_directory>/var/log/edge.log, where <install_directory> is the installation directory of the Edge Processor instance.

For example, the following messages in the logs indicate that mutually authenticated TLS (mTLS) or shared settings for your Edge Processors are not configured correctly:

Log message	Cause of problem
"failed to read client private key from path open <PEM_file_path>: no such file or directory"	The private key file that you specified in the Server private key field is invalid.
"tls: private key does not match public key"	The private key and server certificate files that you specified in the Server private key and Server certificate fields are not part of the same key pair.
"connecting socket: Connection refused"	The port you configured to use is not available.

Solution

The solution varies depending on the root cause of the problem.

For example, if the logs for the instance indicate that mTLS has not been configured correctly, then review the mTLS settings for your Edge Processor and re-upload the private key or certificate files. If the logs indicate a misconfigured port, configure your settings to use an available port. For more information, see Obtain TLS certificates for data sources and Edge Processors and Add an Edge Processor.

An Edge Processor instance is stuck in the "Pending" status

When viewing details about an Edge Processor, you notice that an instance has been in the Pending status for longer than 2 minutes. The Pending status is supposed to change to the Healthy status once the Edge Processor service finishes applying configuration changes to the instance, which typically takes a few minutes.

Cause

The Edge Processor service is not working as expected. Something is preventing the service from completing the configuration changes on your instance.

Solution

Contact your Splunk representative for assistance.

My data is missing from a destination

If you suspect that data is missing, first confirm that the Edge Processor is sending the volume of data that you expect.

1. Log in to your cloud tenant.
2. Navigate to the Data management page.
3. Select View debug logs to troubleshoot the system and then select the Run icon ().
4. Confirm that there haven't been any errors recently that would lead to data loss.
5. If there were no recent errors, navigate to the Edge Processors page.
6. Select the Edge Processor that you want. The Edge Processor details page opens.
7. Confirm that the Outbound data shown in the Pipelines section matches the data volume that you expect.

If the data volume shown does not match what you expect, then check your pipeline configurations to make sure that you are not accidentally filtering out data that you want to keep. If the data volume shown matches what you expect but you cannot find some data, follow these troubleshooting steps.

Cause: The Edge Processor queue has filled up

Edge Processors currently provide no data delivery guarantees. However, to help prevent data loss, the Edge Processor holds data in a queue if the destination that it is sending data to is unavailable or if it receives more data than it can send. If the queue fills up before the destination is available again, then data loss still occurs as the Edge Processor starts dropping any additional data that it receives. To avoid losing data due to a full queue, you can increase the size of the queue.

Queued data is stored on the hard drive of the Edge Processor host. By default, the queue is configured to hold up to 10000 batches of data. The amount of data contained in each batch and how quickly the queue fills up varies depending on the rate at which the Edge Processor is receiving data. For example, if the Edge Processor receives a large amount of data over a short period of time, then the queue fills up quickly with larger batches of data. Typically, a queue size of 10000 holds at least 3 minutes of processing data.

You can adjust the queue size of your Edge Processor if you are sending data to a Splunk platform S2S destination. To adjust the queue size, perform the following steps.

1. In a browser, log in to the Splunk Cloud Console.
2. Select More Options and select Settings.
3. Select the Copy to clipboard icon beside the Access Token field.
4. From the command line, set the following environment variables.
   1. Set TOKEN to be the access token that you copied in step 3.
   2. Set TENANT to be the name of your tenant.
   3. Set API_URL to be https://<tenant>.api.scs.splunk.com.
   4. Set DATASET_NAME to be shared.pipelines.<destination_name>. For example, if you'd like to increase the size of the queue for your destination named default_splunk_cloud_destination, then your DATASET_NAME will be shared.pipelines.default_splunk_cloud_destination.
   5. Run the following command to modify the queue size. Replace <updated_queue_size> with the maximum number of data batches that you want the queue to hold.

      curl --location --request PATCH "$API_URL/$TENANT/search/v3alpha1/datasets/$DATASET_NAME"\
       --header "Authorization: Bearer $TOKEN"\
       --header "Content-Type: application/json"\
       --data-raw '{ "sendQueueSize": <updated_queue_size> }'
      

5. Refresh your pipelines so that they use the updated sendQueueSize setting. See Refresh a pipeline for more information.

Cause: You are sending data to an index that doesn't exist yet

If you've specified a destination index that doesn't exist in the Splunk Cloud Platform, then your data is sent to the index that you specified in the lastchanceindex stanza in your indexes.conf file.

Cause: You are sending data to an index that you don't have permission to view

Your permissions can vary depending on the index privileges. Perform the following steps to update your permissions so you can view the index that data is being sent to.

1. Work with your Splunk administrator to add the index to the list of indexes that your role has permission to search. See Create and manage roles with Splunk Web in the Securing Splunk Cloud Platform manual.
2. Refresh the connection between your cloud tenant and your Splunk Cloud Platform deployment. See Make more indexes available to the tenant.

My data is not sending to the indexer

When sending data to your destination, you don't see your data but receive the following error:

{"level":"INFO","time":"2023-09-15T17:14:17.396Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"XYZ","commit":"6a7d87e4","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S2S/acies_logs","errReason":"failed to connect to <SEARCH-HEAD-IP>:<PORT>: dial tcp <SEARCH-HEAD-IP>:<PORT>: connect: connection refused"}

Cause

The host machine is not configured to connect to the indexer.

Solution

The Edge Processors in your network must be able to communicate with the Edge Processor service in the cloud through specific URLs. For more information see Firewall settings in this topic.

My data is not being processed as expected

When you try to preview a pipeline, the preview results area displays a "No results" message or data that looks incorrect.

Alternatively, when you view the data that was sent from a pipeline to a destination, you notice that the data looks incorrect.

Cause

Reasons why a pipeline might not process data as expected include, but are not limited to, the following:

• The inbound stream of data is not being broken into events correctly. Data must be pre-processed into distinct events before being processed by a pipeline.
• The pipeline is not configured correctly.
• The pipeline preview is for the wrong destination.

Solution

For pipelines with multiple destinations, check to see if you are previewing the correct destination. If not, run the pipeline preview by selecting the Preview Pipeline icon () then select the destination name in the Preview drop-down list.

If this is not the case, make sure that event breaking and merging has been configured correctly for the source type of the data that you want to process.

1. Navigate to the Source types page.
2. Look for a source type with a name that matches the value of the sourcetype field in the data that you want to process.
   • If the source type exists, select it to view its configuration details. Confirm that the event breaking and merging behavior is configured correctly for the data that you want to process.
   • If the source type does not exist, then add it to the Edge Processor service.

   For more information about the configuration settings for source types, see Add source types for Edge Processors.

If the problem persists after you've verified the source type configuration, then complete the following steps to verify that the processing commands in your pipeline are configured correctly.

1. If you don't already have your pipeline open for editing, do the following:
   1. Navigate to the Pipelines page.
   2. On the Pipelines page, in the row that lists the pipeline you want to verify, select the Actions icon () and then select Edit.
2. From the side panel of the pipeline builder, select Sample data.
3. Enter or upload sample data that matches the inbound data that you want this pipeline to process, and then select Apply. You can use text strings that represent raw data or CSV values that represent parsed, field-extracted data. See Getting sample data for previewing data transformations for more information.
4. To generate a preview of what your data looks like after being processed by the pipeline, select the Preview Pipeline icon ().
5. Verify that the preview results match how you want the pipeline to process your data. If the results do not match, or the preview cannot be generated, then make sure that the SPL2 statement of your pipeline is written correctly and contains only supported SPL2 commands. See Edge Processor pipeline syntax for more information.

When I edit a pipeline, I receive a system message saying that my pipeline has changed and I need to update the partition and "where" command configuration

When you open a pipeline for editing, the Edge Processor service displays the following system message:

Splunk has released a software update that affects how filtering clauses in pipelines are interpreted. This change can impact how Edge Processors determine which data to drop or send to the default destination. Update the partition and "where" command configurations in this pipeline as needed, and then save your changes.

Cause

This message appears because your pipeline is affected by a feature update that was released on January 22, 2024. See Updates to partitioning and filtering behavior in Edge Processor pipelines for more information.

The pipeline is still valid. However, the data processing behavior of the pipeline will be changed by the feature update after you save your changes to the pipeline.

Typically, the Edge Processor service automatically adjusts the configuration of your pipeline so that the pre-existing data processing behavior is preserved even after the feature update takes effect. However, in some cases, this adjustment fails. This system message indicates that the automatic adjustment failed, so you need to review and manually update the configuration of your pipeline to ensure that it continues to work as intended.

Solution

1. Review the SPL2 statement of the pipeline, and identify any where clauses that immediately follow the from $source command.
   For example, in the following pipeline, host="buttercup" and source="test_server" are where clauses that immediately follow the from $source command:

   $pipeline = | from $source | where host="buttercup" AND source="test_server" | eval index="my_test_index" | where result="success" | into $destination;
   

   Before the feature update that was released on January 22, 2024, Edge Processors interpreted these where clauses as partition conditions, so any data excluded by these where clauses got sent to the Edge Processor's default destination. Now, Edge Processors interpret them as filters in the main body of the pipeline, so the excluded data gets dropped instead.

2. To retain the original data processing behavior of your pipeline, do the following:
   1. Update the partition of the pipeline to include the where clauses identified in step 1.
   2. Delete the where clauses identified in step 1 from the SPL2 statement.
3. (Optional) Preview your pipeline using sample data to confirm that it works as expected.
4. Save your changes.

See Example 2: The pipeline configuration is not automatically adjusted for more information.

An Edge Processor is not connecting to the Splunk Cloud Platform

After you set up your Edge Processor and try to connect to the destination, you receive the following errors:

{"level":"INFO","time":"2023-09-09T03:43:32.794Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"XYZ","commit":"de064ae8","version":"1.0.0","kind":"exporter","data_type":"metrics","name":"S2S/shared.pipelines.acies_test_play_default_destination_aug_2023","errReason":"failed to connect to <SEARCH-HEAD-IP>:<PORT>: dial tcp <SPLUNK-IP>:<PORT>: connect: connection timed out"}

{"level":"INFO","time":"2023-09-11T09:25:18.533Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"acf798510cce","commit":"de064ae8","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S2S/acies_logs","errReason":"failed to connect to <SEARCH-HEAD-IP>:<PORT>: dial tcp <SEARCH-HEAD-IP>:<PORT>: i/o timeout"}

{"level":"WARN","time":"2023-09-15T09:59:18.156Z","location":"batchprocessor@v0.83.0/batch_processor.go:258","message":"Sender failed","service":"edge-processor","hostname":"XYZ","commit":"6a7d87e4","version":"1.0.0","kind":"processor","name":"batch","error":"sending_queue is full"}

Cause

The machine that the Edge Processor is hosted on is not allowed to connect to the Splunk Cloud Platform IP, range, and port.

Solution

Configure your device allow access to connect to the Splunk Cloud Platform IP, range, and port. See Installation requirements for Edge Processors for more information.

Edge Processor installation displays "certificate has expired" error

When installing an Edge Processor instance, the instance fails to connect with the Edge Processor service, and you receive the following error message about an expired certificate:

splunk@xxx:/splunk/ep/splunk-edge/bin$ ./splunk-edge run
2023/09/12 20:22:51 instance not onboarded - attempting to do so...
Error: provision service principal: POST https://xxx.api.scs.splunk.com/xxx/teleport/v1alpha2/instances/onboard: Post "https://xxx.api.scs.splunk.com/xxx/teleport/v1alpha2/instances/onboard": x509: certificate has expired or is not yet valid: current time 2023-09-12T20:22:51Z is after 2021-09-30T14:01:15Z

Cause

The operating system of the host machine for this Edge Processor instance is using CA certificates that are out-of-date, preventing the host machine from connecting with the Edge Processor service.

Solution

Update your CA certificates for the operating system or host and deploy your Edge Processor nodes with a more recent operating system distribution.

When I send data to the Splunk platform, chunks of data from different events are intertwined together after indexing

When you search your Splunk platform deployment for data that you routed through an Edge Processor, the search returns events that incorrectly contain chunks of data from different events. This problem occurs even though your pipelines are configured correctly and the pipeline previews show data that looks correct.

Cause

This problem occurs if the host_segment attribute is configured in the inputs.conf file for multiple forwarders, and the host_segment settings are causing those forwarders to use the same host value. Typically, data from different forwarders is associated with different host values.

If your Edge Processors are routing data from multiple forwarders and the data from those forwarders are associated with the same host, source, and sourcetype values, then indexers treat that data as pieces of the same event and the data is intertwined as a result.

Solution

For each forwarder that is sending data to an Edge Processor, open the inputs.conf file and check the host_segment setting. Make sure that none of your forwarders have been configured to send data using the same host value.

For more information, see the following pages:

• Set the event host with the host_segment attribute in the Splunk Cloud Platform Getting Data In manual.
• inputs.conf in the Splunk Enterprise Admin Manual.

An Edge Processor fails to receive data through a HEC data source and returns a 401 HTTP status response

Your Edge Processor is not receiving data from your HEC data source. You receive a 401 HTTP status response from your Edge Processor HEC data ingestion.

Cause

The HEC token authentication feature is turned on but you have not provided the correct token in your HEC data source.

Solution

If you do not want to use the HEC token authentication feature, you can turn it off in the shared Edge Processor settings. See Configure shared Edge Processor settings for more information.

If you want to continue using the HEC token authentication feature, do the following:

• Confirm that you have entered the correct token in the shared Edge Processor settings.
• Make sure that the token is configured in your HEC data source HTTP header. For example:
  

  Authorization: Splunk <token> 

An Edge Processor fails to send data through HEC and logs a 403 error

Your Edge Processor has a pipeline that sends data to a Splunk platform HEC destination, but that data is not arriving in the Splunk platform as expected. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see an error message containing the HTTP 403 code. For example:

"error":"HTTP 403 \"Forbidden\""

For information about other error codes and messages returned by HEC endpoints, see Possible error codes in the Getting Data In manual.

Cause

The HEC token that the Edge Processor is using to send data to the Splunk platform is invalid. Reasons why a HEC token might be invalid include, but are not limited to, the following:

• The token is turned off in the Splunk platform.
• The token was entered incorrectly in the original HTTP request or in the Splunk platform HEC destination settings.

Solution

First, confirm which HEC token your Edge Processor is using to send the data:

• If the data was originally transmitted to the Edge Processor through an HTTP request that specifies a HEC token in the Authorization header, then this token is used when the Edge Processor sends the data to the Splunk platform.
• Otherwise, the HEC token specified in the Splunk platform HEC destination is used.

Then, confirm the status of the HEC token in the Splunk platform deployment:

1. Log in to the Splunk platform deployment where the HEC token is configured.
2. In Splunk Web, select Settings, then Data inputs.
3. Select HTTP Event Collector.
4. Confirm that the HTTP Event Collector page lists your HEC token, and that the status of the token is Enabled.
5. If the status of the token is Disabled, then turn it on by selecting Enable in the Actions column.

An Edge Processor fails to send data through HEC and logs an "Incorrect index" error

Your Edge Processor has a pipeline that sends data to a Splunk platform HEC destination, but that data is not arriving in the Splunk platform as expected. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see an error message containing the phrase "Incorrect index".

For information about other error codes and messages returned by HEC endpoints, see Possible error codes in the Getting Data In manual.

Cause

The HEC token that the Edge Processor is using to send data to the Splunk platform doesn't have access to the destination index specified for the data.

Solution

First, confirm which HEC token your Edge Processor is using to send the data:

• If the data was originally transmitted to the Edge Processor through an HTTP request that specifies a HEC token in the Authorization header, then this token is used when the Edge Processor sends the data to the Splunk platform.
• Otherwise, the HEC token specified in the Splunk platform HEC destination is used.

Then, update the index permission settings on the HEC token:

1. Log in to the Splunk platform deployment where the HEC token is configured.
2. In Splunk Web, select Settings, then Data inputs.
3. Select HTTP Event Collector.
4. Select the token that your Edge Processor is using to send data.
5. In the Select Allowed Indexes control, select remove all for the Selected indexes pane. When no indexes are selected, the HEC token allows data to be sent to any index in the Splunk platform deployment.
6. Select Save.

Destinations associated with the connected Splunk Cloud Platform deployment are not working as expected

When you try to send data to a Splunk platform S2S destination that has the Tenant paired property in the Kind field, you encounter errors and the Edge Processor fails to send data to that destination.

Cause

If the Kind field in a destination has the Tenant paired property, that destination is available to Edge Processors through a connection named scpbridge. Reasons why this destination might not work as expected include, but are not limited to, the following:

• The scpbridge connection settings are incorrect. This scenario can occur if the credentials of the service account have changed since the last time the connection was updated, or if the Splunk Cloud Platform deployment has been updated in a way that changes its connection information.
• An index that was previously available as part of this connection has been deleted or changed in the Splunk Cloud Platform deployment.

For more information about the scpbridge connection, see First-time setup instructions for the Edge Processor solution and Send data from Edge Processors to the Splunk Cloud Platform deployment connected to your tenant.

Solution

To verify or update your scpbridge connection settings, complete the following steps.

1. In your cloud tenant, select the Settings icon () and then select System connections.
2. On the System connections page, confirm the status of the scpbridge connection.
   • If the Status: connected icon () displays beside the connection name, then the connection settings are valid. In this case, the problem is more likely to be due to permissions issues in the service account or a change in the indexes that are available in the Splunk Cloud Platform deployment. See the next set of instructions in this section for more guidance.
   • If the Status: disconnected icon () displays instead, then the connection settings are invalid and you need to correct them.
3. To verify the connection settings, select the Edit icon () on the scpbridge connection and then do the following:
   • Confirm that the Hostname, Management port, and Service account username values are correct for the Splunk Cloud Platform deployment that you want your cloud tenant to be connected to.
   • Confirm that the password for the service account hasn't been changed since the scpbridge connection was last updated.
4. If you need to update any connection settings, change the settings as needed and then select Apply.

If you are unable to send data from an Edge Processor to an index that is associated with the scpbridge connection, make sure that the index is still available in the connected Splunk Cloud Platform deployment and accessible by the scpbridge connection.

1. Log in to the connected Splunk Cloud Platform deployment using your admin credentials.
2. Confirm that the index has not been deleted from the deployment.
3. Confirm that the service account used by the scpbridge connection has permission to access the index:
   1. In the Settings menu, in the Users and authentication section, select Roles.
   2. In the row that lists the role used by your service account, select Edit > Edit.

      This role and service account were created during the initial setup of the Edge Processor solution. See First-time setup instructions for the Edge Processor solution for more information.

   3. On the 3. Indexes tab, make sure that the Included check box is selected for your index. If that check box is not already selected, then select it and select Save.
4. In your cloud tenant, refresh the connection to your Splunk Cloud Platform deployment:
   1. Select the Settings icon () and then select System connections.
   2. On the scpbridge connection, select the Refresh icon ().

When I try to delete the "scpbridge" connection, an error occurs

On the System connections page, when you select the Delete icon () on the scpbridge connection, the Edge Processor service returns an error message indicating that the connection could not be deleted.

Cause

This error message appears because the scpbridge connection cannot be deleted after it is created. The Edge Processor solution uses this connection to store and read logs and metrics from Edge Processors, and cannot operate correctly without this connection.

For more information about the scpbridge connection, see First-time setup instructions for the Edge Processor solution and Send data from Edge Processors to the Splunk Cloud Platform deployment connected to your tenant.

Solution

You cannot delete the scpbridge connection. However, if necessary, you can update the connection settings to connect the Edge Processor solution to a different Splunk Cloud Platform deployment.

To update the scpbridge connection settings, do the following:

1. In your cloud tenant, select the Settings icon () and then select System connections.
2. On the System connections page, select the Edit icon () on the scpbridge connection.
3. Update your connection settings as needed and then select Apply.

Syslog data fails to reach an Edge Processor

Your Edge Processor is not receiving syslog data from your syslog data source. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see error messages from "input/tcp" or "input/udp" or "input/syslog".

Cause

This problem occurs when you use the UDP transport protocol to send data. This is a known issue with the UDP transport protocol that is not unique to the Edge Processor solution. UDP does not provide any data guarantees, so when you try to send syslog data to an Edge Processor through UDP, the data might not be delivered successfully.

Solution

If possible, send your syslog data through the TCP transport protocol to validate the connection. If UDP is needed, keep sending syslog data until data reaches Edge Processor. See Configure your device to send syslog data to an Edge Processor for instructions.

My data is not appearing in an Amazon S3 bucket

When you send data to an Amazon S3 destination, you receive the following error:

{"level":"INFO","time":"2023-09-15T10:04:09.397Z","location":"exporterhelper/queued_retry.go:423","message":"Exporting failed. Will retry the request after interval.","service":"edge-processor","hostname":"XYZ,"commit":"6a7d87e4","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S3","error":"operation error S3: PutObject, https response error StatusCode: 403, RequestID: <ID>, HostID: <ID>, api error Access Denied.","interval":"31.154004002s"}

Cause

The Edge Processor is not configured correctly to send data to an Amazon S3 destination.

Solution

Check that you have fulfilled the prerequisites to send data to an Amazon S3 bucket from your Edge Processor. See Send data from Edge Processors to Amazon S3 for more information.