Troubleshoot the Edge Processor solution

Review this page if you are having difficulties with sending data through the Edge Processor solution. If the problem that you're experiencing is not described on this page, you can find more information by doing the following:

• Review the list of known issues in the product. See Known issues.
• Check the logs associated with your Edge Processor. See View logs for the Edge Processor solution.

If the problem persists, contact your Splunk representative for assistance. To help expedite the support process, you can generate a diagnostic report and send it to your Splunk representative.

Generate a diagnostic report for an Edge Processor instance

You can run the edge_diagnostic tool to generate a diagnostic report for a specific Edge Processor instance. The diagnostic report contains logs about the performance and activity of the Edge Processor instance. Include this report when contacting your Splunk representative for assistance.

1. On the host machine of your Edge Processor instance, download the package containing the edge_diagnostic tool.

   curl -o 'edge-diagnostic-linux.tar.gz' 'https://beam.scs.splunk.com/acies/diagnostic/edge-diagnostic-linux.tar.gz'

2. Extract the edge_diagnostic tool into <install_directory>, where <install_directory> is the installation directory of the Edge Processor instance.

   tar -xf edge-diagnostic-linux.tar.gz -C <install_directory>

3. Navigate to the installation directory.

   cd <install_directory>

4. Validate the edge_diagnostic tool by running the following command and confirming whether the returned value matches the expected value:

   sha256sum edge_diagnostic_linux_amd64

   The expected value is:

   64adc244b921b6841cd0bee468321029ad9ad689f92efdd18b4a00c776989786
   

   If these values don't match, that indicates that the edge_diagnostic tool is invalid. This problem can occur if the package download did not complete as expected. If this occurs, delete the package and the extracted edge_diagnostic tool, and then try again from step 1.

5. Use the edge_diagnostic tool to generate a diagnostic report. By default, the tool generates a file named edge-diag-<host_name>-<timestamp>.tar.gz, where <host_name> is the host name of the machine that the Edge Processor instance is running on and <timestamp> is a timestamp indicating when the file was generated. You can use the -out option to specify a different file name.
   • To generate a diagnostic report using the default settings, run this command:

     ./edge_diagnostic_linux_amd64

   • To specify a different output file name, run this command, where <file_name> is the name that you want to use:

     ./edge_diagnostic_linux_amd64 -out <file_name>.tar.gz

   The edge_diagnostic tool generates a diagnostic report file in the current directory. While the tool is running, it returns INFO logs about its status.

6. (Optional) If any unexpected behavior occurs with the edge_diagnostic tool, you can get more information about the status of the tool by running it with the -verbose option. This option causes the tool to print DEBUG level logs about its status while generating the diagnostic report.

   ./edge_diagnostic_linux_amd64 -verbose

When contacting your Splunk representative for assistance, send them a copy of the generated file.

Edge Processor installation returns a "401 Unauthorized" error

When you try to install an Edge Processor instance, the installation fails. Additionally, when you review the <install_directory>/var/log/supervisor.log file on the host machine where you attempted the installation, you see an error message similar to the following:

2024/03/26 06:52:16 token refresh failed, will retry: sign token: {"status_code":401,"status":"401 Unauthorized"}

Cause

The security token that's required during the installation process is expired. This problem can happen because the system clock on the host machine is set to an incorrect time, causing the token to expire prematurely. The token is valid for 5 minutes only.

Solution

Make sure that the system clock on the host machine is correct, and synchronize it to a Network Time Protocol (NTP) server. Refer to the documentation for your operating system for more information.

Edge Processor installation displays "certificate has expired" error

When installing an Edge Processor instance, the instance fails to connect with the Edge Processor service, and you receive the following error message about an expired certificate:

splunk@xxx:/splunk/ep/splunk-edge/bin$ ./splunk-edge run
2023/09/12 20:22:51 instance not onboarded - attempting to do so...
Error: provision service principal: POST https://xxx.api.scs.splunk.com/xxx/teleport/v1alpha2/instances/onboard: Post "https://xxx.api.scs.splunk.com/xxx/teleport/v1alpha2/instances/onboard": x509: certificate has expired or is not yet valid: current time 2023-09-12T20:22:51Z is after 2021-09-30T14:01:15Z

Cause

The operating system of the host machine for this Edge Processor instance is using CA certificates that are out-of-date, preventing the host machine from connecting with the Edge Processor service.

Solution

Update your CA certificates for the operating system or host and deploy your Edge Processor nodes with a more recent operating system distribution.

An Edge Processor instance that was previously "Healthy" is now "Disconnected"

When viewing details about an Edge Processor, you notice that an instance that previously had the Healthy status now has the Disconnected status.

Cause

The Edge Processor service lost contact with that instance. This problem can happen for a variety of reasons, such as the host machine of the instance going down or the communication between the instance and the Edge Processor service getting blocked.

This problem can also happen if you remove an Edge Processor instance from its host machine using any method other than the uninstallation command provided in the Edge Processor service. In this case, the service fails to detect that the instance has been removed.

Solution

To determine the root cause of the problem, check the logs for the supervisor that's associated with the disconnected Edge Processor instance.

1. Log in to the host machine of the disconnected Edge Processor instance.
2. Navigate to the <install_directory>/var/log directory, where <install_directory> is the installation directory of the Edge Processor instance.
3. Review the logs in the supervisor.log file.

If the Edge Processor instance was removed using a method other than the uninstallation command, then you need to reinstall the instance and then uninstall it using the appropriate command. Using the command provided in the Edge Processor service allows the service to detect this change and stop listing the instance in the Edge Processor details.

1. Reinstall the instance:
   1. On the Edge Processors page, select the Edge Processor that has Disconnected instance.
   2. In the panel that contains your Edge Processor details, select Manage instances.
   3. Select the Install/uninstall tab, and then expand the Step 1: Run commands to install/uninstall instances section.
   4. Select Install to view the commands for downloading and installing an Edge Processor instance on a Linux machine, and then select Copy to clipboard.
   5. On the machine that previously hosted the removed instance, open a command-line interface in a directory of your choice and then paste and run the commands. When the installation is complete, you will see the following message:

      splunk-edge.service - Splunk edge starter
          Loaded: loaded (/etc/systemd/system/splunk-edge.service, enabled)
          Active: active (running)

      This command contains sensitive information about your cloud environment. Do not share this command with anyone except your Splunk representative or trusted members in your organization.

   6. Verify that your instance is healthy by checking its status on the Edge Processors page or in the detailed view of your Edge Processor. It may take up to 1 minute for the status to change to Healthy.
2. Uninstall the instance using the command provided in the Edge Processor service:
   1. In the Manage instances panel for your Edge Processor, on the Install/uninstall tab, expand the Step 1: Run commands to install/uninstall instances section and then select Uninstall.
   2. To copy the command for uninstalling an instance from a Linux machine, select Copy to clipboard.
   3. On the host machine where you reinstalled the instance, open a command-line interface in a directory of your choice and then paste and run the command.

An Edge Processor instance is in the "Warning" status

When viewing details about an Edge Processor, you notice that an instance is in the Warning status.

Cause

This Edge Processor instance is consuming more resources than what is acceptable based on the CPU threshold and Memory threshold settings configured in the Edge Processor service.

Solution

First, confirm that the CPU threshold and Memory threshold settings are configured to allow a reasonable amount of resource usage. The recommended amount is 80% of the total allocated resources.

1. On the Edge Processors page, select Shared settings.
2. Select the Other settings tab.
3. Check the values specified in the CPU threshold and Memory threshold fields.
4. If you want to adjust those settings, do the following:
   1. Select Edit.
   2. In the CPU threshold field, enter the percentage of the total allocated CPU power that an Edge Processor instance can use before it enters the Warning state.
   3. In the Memory threshold field, enter the percentage of the total allocated memory that an Edge Processor instance can use before it enters the Warning state.
   4. Select Save to save your changes, and then select Edge Processors to return to the Edge Processor management page.

If the problem persists, then try the following solutions.

Reduce CPU usage per instance by scaling up the Edge Processor

Add more instances to the Edge Processor. Having more instances associated with the Edge Processor reduces the amount of CPU processing power that any one instance needs to consume.

For more information, see Add more instances to an Edge Processor.

Reinstall the instance on a machine that has more memory

Install a new Edge Processor instance on a host machine that has more memory available. Then, uninstall the instance that has the Warning status. Make sure to update your data sources to start sending data to the new instance and stop sending data to the uninstalled instance.

For more information, see Add more instances to an Edge Processor and Uninstall an Edge Processor instance.

An Edge Processor instance is in the "Error" status

When viewing details about an Edge Processor, you notice that an instance is in the Error status.

Cause

The Error status indicates that something is wrong with the Edge Processor instance, but it is still in contact with the Edge Processor service. This problem can occur for a variety of reasons. For example, this status can occur if the Edge Processor is configured incorrectly, or if an internal component is stuck in a restart loop.

To get more information about the root cause of an Error status, view the logs for the instance. The logs are located on the host machine of the instance in <install_directory>/var/log/edge.log, where <install_directory> is the installation directory of the Edge Processor instance.

For example, the following messages in the logs indicate that mutually authenticated TLS (mTLS) or shared settings for your Edge Processors are not configured correctly:

Log message	Cause of problem
"failed to read client private key from path open <PEM_file_path>: no such file or directory"	The private key file that you specified in the Server private key field is invalid.
"tls: private key does not match public key"	The private key and server certificate files that you specified in the Server private key and Server certificate fields are not part of the same key pair.
"connecting socket: Connection refused"	The port you configured to use is not available.
"status" : "failed"	There is an error in your most recent configuration. The Edge Processor is running a past configuration.

Solution

The solution varies depending on the root cause of the problem.

For example, if the logs for the instance indicate that mTLS has not been configured correctly, then review the mTLS settings for your Edge Processor and re-upload the private key or certificate files. If the logs indicate a misconfigured port, configure your settings to use an available port. If the logs indicate a failed configuration, compare a previous successful configuration to your most recent configuration to find the root cause. For more information, see Obtain TLS certificates for data sources and Edge Processors and Add an Edge Processor.

An Edge Processor instance is stuck in the "Pending" status

When viewing details about an Edge Processor, you notice that an instance has been in the Pending status for longer than 2 minutes. The Pending status is supposed to change to the Healthy status once the Edge Processor service finishes applying configuration changes to the instance, which typically takes a few minutes.

Cause

The Edge Processor service is not working as expected. Something is preventing the service from completing the configuration changes on your instance.

Solution

Contact your Splunk representative for assistance.

The "Received data" pane is missing some data

You have configured one or more data sources to send data to your Edge Processor. However, when you open the detailed view of your Edge Processor and check the Received data pane, you notice that some data is missing, or the pane shows a "No data to display" message.

Cause

Possible causes for this problem include the following:

• The Received data pane is displaying information from a time range that does not include the data you're looking for.
• The scpbridge connection, which pairs the Edge Processor service with a Splunk Cloud Platform deployment, has been configured incorrectly. This issue can prevent the Received data metrics from being communicated as expected.

Solution

Start by confirming that the Received data pane is displaying information from the correct time range. Do the following:

1. On the Edge Processors page, in the row that lists your Edge Processor, select the Actions icon () and then select Open.
2. Check the time range that's specified in the Metrics drop-down list. Make sure that it's set to a time range during which your data source was sending data to the Edge Processor.

If the data is still missing, then confirm whether the scpbridge connection is configured correctly. Do the following:

1. In your cloud tenant, select the Settings icon () and then select System connections.
2. On the System connections page, confirm the status of the scpbridge connection.
   • If the Status: connected icon () displays beside the connection name, then the connection settings are valid. In this case, the problem is more likely to be due to permissions issues in the service account. See step 5 for instructions on confirming the permissions of the service account.
   • If the Status: disconnected icon () displays instead, then the connection settings are invalid and you need to correct them.
3. To verify the connection settings, select the Edit icon () on the scpbridge connection and then do the following:
   • Confirm that the Host name, Management port, and Service account username values are correct for the Splunk Cloud Platform deployment that you want your cloud tenant to be connected to.
   • Confirm that the password for the service account hasn't been changed since the scpbridge connection was last updated.
4. If you need to update any connection settings, change the settings as needed and then select Apply.
5. To confirm that the service account used by the scpbridge connection has the necessary permissions, do the following:
   1. Log in to the connected Splunk Cloud Platform deployment using your admin credentials.
   2. Confirm that the service account used by the scpbridge connection has permission to access all internal indexes:
      1. In the Settings menu, in the Users and authentication section, select Roles.
      2. In the row that lists the role used by your service account, select Edit > Edit.

         This role and service account were created during the initial setup of the Edge Processor solution. See First-time setup instructions for the Edge Processor solution for more information.

      3. On the 3. Indexes tab, make sure that the Included check box is selected in the _* (All internal indexes) row. If that check box is not already selected, then select it and select Save.
   3. In your cloud tenant, refresh the connection to your Splunk Cloud Platform deployment:
      1. Select the Settings icon () and then select System connections.
      2. On the scpbridge connection, select the Refresh icon ().

If the Received data panel still does not display the expected information, then your data might be failing to reach your Edge Processor. See An Edge Processor is not receiving the expected data for additional troubleshooting guidance.

An Edge Processor is not receiving the expected data

The Received data pane in the detailed view of your Edge Processor is missing some data, or the pane displays a "No data to display" message.

Before proceeding with this troubleshooting guidance, confirm the following:

• The Received data pane is displaying information from a correct time range.
• The scpbridge connection is configured correctly.

See The "Received data" pane is missing some data on this page for more information.

Cause

Possible causes for this problem include the following:

• The data source or the Edge Processor is not configured correctly.
• Network requirements for allowing the data source and Edge Processor to communicate have not been met.

Solution

Check the configuration settings for the data source and the Edge Processor. Make sure of the following:

• The data source meets the requirements described in this documentation manual and is configured correctly. See the Get data into Edge Processors chapter for more information.
• The Receiver settings in your Edge Processor are configured correctly. For example, if your data source is a syslog device, then you must select the Syslog check box in the Receiver settings area. See Add an Edge Processor for more information.
• On the host machine of your Edge Processor, the firewall settings and ports are configured correctly. See Network requirements for more information.

For additional troubleshooting guidance for specific data sources, see the following sections on this page:

• An Edge Processor is not receiving data from a forwarder
• Syslog data fails to reach an Edge Processor

An Edge Processor is not receiving data from a forwarder

You've configured a universal forwarder or heavy forwarder to send data to an Edge Processor, but the data is not reaching the Edge Processor.

Before proceeding with this troubleshooting guidance, confirm that this problem is not caused by incorrect Receiver settings in your Edge Processor or by misleading information in the Received data pane. See The "Received data" pane is missing some data and An Edge Processor is not receiving the expected data on this page for more information.

Cause

Possible causes for this problem include the following:

• The forwarder is not configured correctly.
• Network requirements for allowing the forwarder and Edge Processor to communicate have not been met.

Solution

Start by making sure that your forwarder meets the requirements described in Get data from a forwarder into an Edge Processor and is configured correctly. In particular, confirm the following:

1. The outputs.conf file defines a target group for the Edge Processor.
2. The outputs.conf, inputs.conf, and props.conf files don't define any advanced routing or filtering settings that would prevent data from being forwarded to the target group for the Edge Processor.
3. The forwarder is running and has an "active forward" sending data to the Edge Processor. To confirm this, run this command from the $SPLUNK_HOME/bin directory:

   splunk list forward-server

If the forwarder configuration is correct but the problem persists, then use the netcat tool to confirm whether network issues are preventing the forwarder and the Edge Processor from communicating. To do this, you must first have the netcat tool installed on both the forwarder host machine and the Edge Processor host machine.

1. On the host machine of your Edge Processor, use netcat to start listening for incoming data on the receiver port for Splunk forwarders. Do the following:
   1. Confirm the port number that your Edge Processor is using to listen for forwarded data. In the Edge Processor service, on the Edge Processors page, select Shared settings and then check the port number specified in the Splunk forwarders section.
   2. On the Edge Processor host machine, run the following command, where <forwarder_port> is the port number that you confirmed during the previous step.

      nc -1 <forwarder_port>
      

2. On the host machine of your forwarder, send the message "hello world" to the host machine of the Edge Processor, and use netcat to confirm if the message was sent successfully. To do this, run the following command, where <edge_processor_host> is the IP address of the Edge Processor host machine and <forwarder_port> is the port number that you confirmed during step 1a:

   echo "hello world" | nc -v <edge_processor_host> <forwarder_port>
   

   If the forwarder host machine successfully sends "hello world" to the Edge Processor host machine, then the following messages are returned:

   • In the terminal of the forwarder host: "Connection to <edge_processor_host> port <forwarder_port> [tcp/de-noc] succeeded!"
   • In the terminal of the Edge Processor host: "hello world"

If you do not see a successful result, that indicates that the forwarder is unable to communicate with the Edge Processor due to network problems. To resolve these problems, confirm that the firewall settings and ports on your Edge Processor host are configured correctly. See Network requirements for more information.

An Edge Processor fails to receive data through a HEC data source and returns a 401 HTTP status response

Your Edge Processor is not receiving data from your HEC data source. You receive a 401 HTTP status response from your Edge Processor HEC data ingestion.

Cause

The HEC token authentication feature is turned on but you have not provided the correct token in your HEC data source.

Solution

If you do not want to use the HEC token authentication feature, you can turn it off in the shared Edge Processor settings. See Configure shared Edge Processor settings for more information.

If you want to continue using the HEC token authentication feature, do the following:

• Confirm that you have entered the correct token in the shared Edge Processor settings.
• Make sure that the token is configured in your HEC data source HTTP header. For example:
  

  Authorization: Splunk <token> 

Syslog data fails to reach an Edge Processor

Your Edge Processor is not receiving syslog data from your syslog data source. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see error messages from "input/tcp" or "input/udp" or "input/syslog".

Cause

This problem occurs when you use the UDP transport protocol to send data. This is a known issue with the UDP transport protocol that is not unique to the Edge Processor solution. UDP does not provide any data guarantees, so when you try to send syslog data to an Edge Processor through UDP, the data might not be delivered successfully.

Solution

If possible, send your syslog data through the TCP transport protocol to validate the connection. If UDP is needed, keep sending syslog data until data reaches Edge Processor. See Configure your device to send syslog data to an Edge Processor for instructions.

My data is missing from a destination

You've configured your Edge Processor to send processed data to a destination, but the data is missing from that destination.

Before proceeding with this troubleshooting guidance, make sure that your Edge Processor is successfully receiving the data. See View data flow information about an Edge Processor.

Cause

Possible causes for this problem include the following:

• The pipelines that are applied to the Edge Processor are not configured correctly.
• The destinations used in the applied pipelines are not configured correctly.
• The Edge Processor's data queue is full.
• Network requirements for allowing the Edge Processor to communicate with the destination have not been met.

Solution

To determine the root cause of the problem, start by checking the Edge Processor logs for error messages. Do the following:

1. In the Edge Processor service, navigate to the Edge Processors page.
2. In the row that lists your Edge Processor, select the Actions icon () and then select View debug logs.
3. To retrieve the Edge Processor logs, select a suitable time range and then select the Run icon ().
4. Check the returned logs for any errors that indicate data loss, network connectivity issues, a full "sending_queue", or problems with the destination configuration. For additional troubleshooting guidance for specific error messages, see the following sections on this page:
   • An Edge Processor is not connecting to the Splunk Cloud Platform
   • An Edge Processor fails to send data, and logs a "Dropping data because sending_queue is full" error
   • An Edge Processor fails to send data through HEC and logs a 403 error
   • An Edge Processor fails to send data through HEC and logs an "Incorrect index" error
   • My data is not appearing in an Amazon S3 bucket

If the logs do not seem to contain any relevant errors, then do the following to confirm that the Edge Processor is sending the volume of data that you expect:

1. On the Edge Processors page, in the row that lists your Edge Processor, select the Actions icon () and then select Open.
2. From the Metrics drop-down list, select the time range for the Edge Processor information that you want to check.
3. In the Pipelines pane, confirm whether the Outbound data column displays the data volume that you expect.

If the data volume shown does not match what you expect, then the pipelines applied to the Edge Processor might not be configured correctly. Check your pipeline configurations to make sure that you are not accidentally filtering out data that you want to keep.

If the problem persists after you check the outbound data volume and pipeline configurations, then the root cause of the problem might be due to how data is being routed inside the destination after it leaves the Edge Processor, or due to network connectivity issues.

• If you're sending data to a Splunk platform deployment, then see My data is missing from an index for further troubleshooting guidance.
• Otherwise, contact your Splunk representative for assistance.

My data is missing from an index

You've configured an Edge Processor to send data to an index in Splunk Cloud Platform or Splunk Enterprise, but the data is missing from the index.

Before continuing with this troubleshooting guidance, make sure that the problem is not due to misconfigured destinations, pipeline filtering, or other specific errors. See My data is missing from a destination on this page for more information.

Cause

Possible causes for this problem include the following:

• Your data is being routed in unexpected ways after it reaches the Splunk platform deployment.
• Network requirements for allowing the Edge Processor to communicate with the Splunk platform deployment have not been met.

Solution

Start by checking the configuration settings that determine how indexers route data to an index, and making sure that your data is being routed to the expected index. Do the following:

• Review the configurations described in How does an Edge Processor know which index to send data to? and make sure that your data is being routed to the index that you expect.
• Make sure that the target index exists in the Splunk platform deployment. If you attempt to send data to an index that doesn't exist in the deployment, then your data is sent to the index specified by the lastchanceindex property in the indexes.conf file.

If these routing configurations are correct but the problem persists, then use the netcat tool to confirm whether network issues are preventing the Edge Processor and Splunk platform deployment from communicating. To do this, you must first have the netcat tool installed on the Edge Processor host machine.

Use the netcat tool to connect from the Edge Processor host machine to the Splunk platform deployment. Run the following command, where <indexer> is the IP address or host name of an indexer in your Splunk platform deployment:

nc -v <indexer> 9997

If the connection is successful, then the following message is returned:

Connection to <indexer> port 9997 [tcp/palace-6] succeeded!

If you do not see a successful result, that indicates that the Edge Processor is unable to communicate with the indexer due to network problems. To resolve these problems, confirm that the firewall settings and ports on your Edge Processor host are configured correctly. See Network requirements for more information.

An Edge Processor is not connecting to the Splunk Cloud Platform

You've configured an Edge Processor to send data to an index in Splunk Cloud Platform or Splunk Enterprise, but the data is missing from the index. Additionally, when you check the Edge Processor logs, you see error messages such as the following:

This "connection refused" error message:

{"level":"INFO","time":"2023-09-15T17:14:17.396Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"XYZ","commit":"6a7d87e4","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S2S/acies_logs","errReason":"failed to connect to <SPLUNK-IP>:<PORT>: dial tcp <SPLUNK-IP>:<PORT>: connect: connection refused"}

This "connection timed out" error message:

{"level":"INFO","time":"2023-09-09T03:43:32.794Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"XYZ","commit":"de064ae8","version":"1.0.0","kind":"exporter","data_type":"metrics","name":"S2S/shared.pipelines.acies_test_play_default_destination_aug_2023","errReason":"failed to connect to <SPLUNK-IP>:<PORT>: dial tcp <SPLUNK-IP>:<PORT>: connect: connection timed out"}

This "i/o timeout" error message:

{"level":"INFO","time":"2023-09-11T09:25:18.533Z","logger":"DNSResolvingClientProvider.Peer","location":"v2/peer.go:387","message":"fail to connect","service":"edge-processor","hostname":"acf798510cce","commit":"de064ae8","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S2S/acies_logs","errReason":"failed to connect to <SPLUNK-IP>:<PORT>: dial tcp <SPLUNK-IP>:<PORT>: i/o timeout"}

Cause

The Edge Processor is unable to communicate with the Splunk platform deployment due to network problems.

Solution

Make sure that the firewall settings and ports on your Edge Processor host are configured correctly. See Network requirements for more information.

An Edge Processor fails to send data, and logs a "Dropping data because sending_queue is full" error

You've configured an Edge Processor to send data to a destination, but the data is missing from that destination. When you view the Edge Processor logs, you see an error message similar to the following:

{"level":"ERROR","time":"2024-03-27T06:27:33.628Z","location":"exporterhelper/queue_sender.go:196","message":"Dropping data because sending_queue is full. Try increasing queue_size.","service":"edge-processor","hostname":"EUAWS00LNX0215","commit":"958c65c0","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S2S/shared.pipelines.default_splunk_cloud_destination","dropped_items":69,"callstack":"<callstack_details>"}

Cause

The Edge Processor is dropping data because its data queue is full.

When a destination is unavailable or an Edge Processor receives more data than it can send out, the Edge Processor holds data in a queue. If this queue fills up before the problem resolves, then the Edge Processor starts dropping any additional data that it receives. For more information about the queue, see What happens to my data if a destination becomes unavailable?

Solution

If you're sending data to a Splunk platform S2S destination, you can increase the size of the queue to avoid losing data due to a full queue. Otherwise, contact your Splunk representative for assistance.

To adjust the queue size for a Splunk platform S2S destination, do the following:

1. In a browser, log in to the Splunk Cloud Console.
2. Select More Options and select Settings.
3. Select the Copy to clipboard icon beside the Access Token field.
4. From the command line, set the following environment variables.
   1. Set TOKEN to be the access token that you copied in step 3.
   2. Set TENANT to be the name of your tenant.
   3. Set API_URL to be https://<tenant>.api.scs.splunk.com.
   4. Set DATASET_NAME to be shared.pipelines.<destination_name>. For example, if you'd like to increase the size of the queue for your destination named default_splunk_cloud_destination, then your DATASET_NAME will be shared.pipelines.default_splunk_cloud_destination.
   5. Run the following command to modify the queue size. Replace <updated_queue_size> with the maximum number of data batches that you want the queue to hold.

      curl --location --request PATCH "$API_URL/$TENANT/search/v3alpha1/datasets/$DATASET_NAME"\
       --header "Authorization: Bearer $TOKEN"\
       --header "Content-Type: application/json"\
       --data-raw '{ "sendQueueSize": <updated_queue_size> }'
      

5. Refresh your pipelines so that they use the updated sendQueueSize setting. See Refresh a pipeline for more information.

An Edge Processor fails to send data through HEC and logs a 403 error

Your Edge Processor has a pipeline that sends data to a Splunk platform HEC destination, but that data is not arriving in the Splunk platform as expected. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see an error message containing the HTTP 403 code. For example:

"error":"HTTP 403 \"Forbidden\""

For information about other error codes and messages returned by HEC endpoints, see Possible error codes in the Getting Data In manual.

Cause

The HEC token that the Edge Processor is using to send data to the Splunk platform is invalid. Reasons why a HEC token might be invalid include, but are not limited to, the following:

• The token is turned off in the Splunk platform.
• The token was entered incorrectly in the original HTTP request or in the Splunk platform HEC destination settings.

Solution

First, confirm which HEC token your Edge Processor is using to send the data:

• If the data was originally transmitted to the Edge Processor through an HTTP request that specifies a HEC token in the Authorization header, then this token is used when the Edge Processor sends the data to the Splunk platform.
• Otherwise, the HEC token specified in the Splunk platform HEC destination is used.

Then, confirm the status of the HEC token in the Splunk platform deployment:

1. Log in to the Splunk platform deployment where the HEC token is configured.
2. In Splunk Web, select Settings, then Data inputs.
3. Select HTTP Event Collector.
4. Confirm that the HTTP Event Collector page lists your HEC token, and that the status of the token is Enabled.
5. If the status of the token is Disabled, then turn it on by selecting Enable in the Actions column.

An Edge Processor fails to send data through HEC and logs an "Incorrect index" error

Your Edge Processor has a pipeline that sends data to a Splunk platform HEC destination, but that data is not arriving in the Splunk platform as expected. When you view the logs in the <install_directory>/var/log/edge.log file on the host machine of the Edge Processor, you see an error message containing the phrase "Incorrect index".

For information about other error codes and messages returned by HEC endpoints, see Possible error codes in the Getting Data In manual.

Cause

The HEC token that the Edge Processor is using to send data to the Splunk platform doesn't have access to the destination index specified for the data.

Solution

First, confirm which HEC token your Edge Processor is using to send the data:

• If the data was originally transmitted to the Edge Processor through an HTTP request that specifies a HEC token in the Authorization header, then this token is used when the Edge Processor sends the data to the Splunk platform.
• Otherwise, the HEC token specified in the Splunk platform HEC destination is used.

Then, update the index permission settings on the HEC token:

1. Log in to the Splunk platform deployment where the HEC token is configured.
2. In Splunk Web, select Settings, then Data inputs.
3. Select HTTP Event Collector.
4. Select the token that your Edge Processor is using to send data.
5. In the Select Allowed Indexes control, select remove all for the Selected indexes pane. When no indexes are selected, the HEC token allows data to be sent to any index in the Splunk platform deployment.
6. Select Save.

My data is not appearing in an Amazon S3 bucket

You've configured an Edge Processor to send data to an Amazon S3 bucket, but the data is missing from the bucket. Additionally, when you check the Edge Processor logs, you see the following error message:

{"level":"INFO","time":"2023-09-15T10:04:09.397Z","location":"exporterhelper/queued_retry.go:423","message":"Exporting failed. Will retry the request after interval.","service":"edge-processor","hostname":"XYZ,"commit":"6a7d87e4","version":"1.0.0","kind":"exporter","data_type":"logs","name":"S3","error":"operation error S3: PutObject, https response error StatusCode: 403, RequestID: <ID>, HostID: <ID>, api error Access Denied.","interval":"31.154004002s"}

Cause

The Edge Processor is not configured correctly to send data to an Amazon S3 destination.

Solution

Check that you have fulfilled the prerequisites to send data to an Amazon S3 bucket from your Edge Processor. See Send data from Edge Processors to Amazon S3 for more information.

My data is not being processed as expected

When you try to preview a pipeline, the preview results area displays a "No results" message or data that looks incorrect.

Alternatively, when you view the data that was sent from a pipeline to a destination, you notice that the data looks incorrect.

Cause

Reasons why a pipeline might not process data as expected include, but are not limited to, the following:

• The inbound stream of data is not being broken into events correctly. Data must be pre-processed into distinct events before being processed by a pipeline.
• The pipeline is not configured correctly.
• The pipeline preview is for the wrong destination.

Solution

For pipelines with multiple destinations, check to see if you are previewing the correct destination. If not, run the pipeline preview by selecting the Preview Pipeline icon () then select the destination name in the Preview drop-down list.

If this is not the case, make sure that event breaking and merging has been configured correctly for the source type of the data that you want to process.

1. Navigate to the Source types page.
2. Look for a source type with a name that matches the value of the sourcetype field in the data that you want to process.
   • If the source type exists, select it to view its configuration details. Confirm that the event breaking and merging behavior is configured correctly for the data that you want to process.
   • If the source type does not exist, then add it to the Edge Processor service.

   For more information about the configuration settings for source types, see Add source types for Edge Processors.

If the problem persists after you've verified the source type configuration, then complete the following steps to verify that the processing commands in your pipeline are configured correctly.

1. If you don't already have your pipeline open for editing, do the following:
   1. Navigate to the Pipelines page.
   2. On the Pipelines page, in the row that lists the pipeline you want to verify, select the Actions icon () and then select Edit.
2. From the side panel of the pipeline builder, select Sample data.
3. Enter or upload sample data that matches the inbound data that you want this pipeline to process, and then select Apply. You can use text strings that represent raw data or CSV values that represent parsed, field-extracted data. See Getting sample data for previewing data transformations for more information.
4. To generate a preview of what your data looks like after being processed by the pipeline, select the Preview Pipeline icon ().
5. Verify that the preview results match how you want the pipeline to process your data. If the results do not match, or the preview cannot be generated, then make sure that the SPL2 statement of your pipeline is written correctly and contains only supported SPL2 commands. See Edge Processor pipeline syntax for more information.

When I send data to the Splunk platform, chunks of data from different events are intertwined together after indexing

When you search your Splunk platform deployment for data that you routed through an Edge Processor, the search returns events that incorrectly contain chunks of data from different events. This problem occurs even though your pipelines are configured correctly and the pipeline previews show data that looks correct.

Cause

This problem occurs if the host_segment attribute is configured in the inputs.conf file for multiple forwarders, and the host_segment settings are causing those forwarders to use the same host value. Typically, data from different forwarders is associated with different host values.

If your Edge Processors are routing data from multiple forwarders and the data from those forwarders are associated with the same host, source, and sourcetype values, then indexers treat that data as pieces of the same event and the data is intertwined as a result.

Solution

For each forwarder that is sending data to an Edge Processor, open the inputs.conf file and check the host_segment setting. Make sure that none of your forwarders have been configured to send data using the same host value.

For more information, see the following pages:

• Set the event host with the host_segment attribute in the Splunk Cloud Platform Getting Data In manual.
• inputs.conf in the Splunk Enterprise Admin Manual.

When I edit a pipeline, I receive a system message saying that my pipeline has changed and I need to update the partition and "where" command configuration

When you open a pipeline for editing, the Edge Processor service displays the following system message:

Splunk has released a software update that affects how filtering clauses in pipelines are interpreted. This change can impact how Edge Processors determine which data to drop or send to the default destination. Update the partition and "where" command configurations in this pipeline as needed, and then save your changes.

Cause

This message appears because your pipeline is affected by a feature update that was released on January 22, 2024. See Updates to partitioning and filtering behavior in Edge Processor pipelines for more information.

The pipeline is still valid. However, the data processing behavior of the pipeline will be changed by the feature update after you save your changes to the pipeline.

Typically, the Edge Processor service automatically adjusts the configuration of your pipeline so that the pre-existing data processing behavior is preserved even after the feature update takes effect. However, in some cases, this adjustment fails. This system message indicates that the automatic adjustment failed, so you need to review and manually update the configuration of your pipeline to ensure that it continues to work as intended.

Solution

1. Review the SPL2 statement of the pipeline, and identify any where clauses that immediately follow the from $source command.
   For example, in the following pipeline, host="buttercup" and source="test_server" are where clauses that immediately follow the from $source command:

   $pipeline = | from $source | where host="buttercup" AND source="test_server" | eval index="my_test_index" | where result="success" | into $destination;
   

   Before the feature update that was released on January 22, 2024, Edge Processors interpreted these where clauses as partition conditions, so any data excluded by these where clauses got sent to the Edge Processor's default destination. Now, Edge Processors interpret them as filters in the main body of the pipeline, so the excluded data gets dropped instead.

2. To retain the original data processing behavior of your pipeline, do the following:
   1. Update the partition of the pipeline to include the where clauses identified in step 1.
   2. Delete the where clauses identified in step 1 from the SPL2 statement.
3. (Optional) Preview your pipeline using sample data to confirm that it works as expected.
4. Save your changes.

See Example 2: The pipeline configuration is not automatically adjusted for more information.

Destinations associated with the connected Splunk Cloud Platform deployment are not working as expected

When you try to send data to a Splunk platform S2S destination that has the Tenant paired property in the Kind field, you encounter errors and the Edge Processor fails to send data to that destination.

Cause

If the Kind field in a destination has the Tenant paired property, that destination is available to Edge Processors through a connection named scpbridge. Reasons why this destination might not work as expected include, but are not limited to, the following:

• The scpbridge connection settings are incorrect. This scenario can occur if the credentials of the service account have changed since the last time the connection was updated, or if the Splunk Cloud Platform deployment has been updated in a way that changes its connection information.
• An index that was previously available as part of this connection has been deleted or changed in the Splunk Cloud Platform deployment.

For more information about the scpbridge connection, see First-time setup instructions for the Edge Processor solution and Send data from Edge Processors to the Splunk Cloud Platform deployment connected to your tenant.

Solution

To verify or update your scpbridge connection settings, complete the following steps.

1. In your cloud tenant, select the Settings icon () and then select System connections.
2. On the System connections page, confirm the status of the scpbridge connection.
   • If the Status: connected icon () displays beside the connection name, then the connection settings are valid. In this case, the problem is more likely to be due to permissions issues in the service account or a change in the indexes that are available in the Splunk Cloud Platform deployment. See the next set of instructions in this section for more guidance.
   • If the Status: disconnected icon () displays instead, then the connection settings are invalid and you need to correct them.
3. To verify the connection settings, select the Edit icon () on the scpbridge connection and then do the following:
   • Confirm that the Host name, Management port, and Service account username values are correct for the Splunk Cloud Platform deployment that you want your cloud tenant to be connected to.
   • Confirm that the password for the service account hasn't been changed since the scpbridge connection was last updated.
4. If you need to update any connection settings, change the settings as needed and then select Apply.

If you are unable to send data from an Edge Processor to an index that is associated with the scpbridge connection, make sure that the index is still available in the connected Splunk Cloud Platform deployment and accessible by the scpbridge connection.

1. Log in to the connected Splunk Cloud Platform deployment using your admin credentials.
2. Confirm that the index has not been deleted from the deployment.
3. Confirm that the service account used by the scpbridge connection has permission to access the index:
   1. In the Settings menu, in the Users and authentication section, select Roles.
   2. In the row that lists the role used by your service account, select Edit > Edit.

      This role and service account were created during the initial setup of the Edge Processor solution. See First-time setup instructions for the Edge Processor solution for more information.

   3. On the 3. Indexes tab, make sure that the Included check box is selected for your index. If that check box is not already selected, then select it and select Save.
4. In your cloud tenant, refresh the connection to your Splunk Cloud Platform deployment:
   1. Select the Settings icon () and then select System connections.
   2. On the scpbridge connection, select the Refresh icon ().

When I try to delete the "scpbridge" connection, an error occurs

On the System connections page, when you select the Delete icon () on the scpbridge connection, the Edge Processor service returns an error message indicating that the connection could not be deleted.

Cause

This error message appears because the scpbridge connection cannot be deleted after it is created. The Edge Processor solution uses this connection to store and read logs and metrics from Edge Processors, and cannot operate correctly without this connection.

For more information about the scpbridge connection, see First-time setup instructions for the Edge Processor solution and Send data from Edge Processors to the Splunk Cloud Platform deployment connected to your tenant.

Solution

You cannot delete the scpbridge connection. However, if necessary, you can update the connection settings to connect the Edge Processor solution to a different Splunk Cloud Platform deployment.

To update the scpbridge connection settings, do the following:

1. In your cloud tenant, select the Settings icon () and then select System connections.
2. On the System connections page, select the Edit icon () on the scpbridge connection.
3. Update your connection settings as needed and then select Apply.