Troubleshooting the Splunk SOAR Automation Broker

You might need to troubleshoot an issue related to your Splunk SOAR Automation Broker. This topic will help you:

• Download and find the log files for the Automation Brokers on your Docker host.
• Match those log files to the correct Automation Broker.
• Explain how to rotate the encryption keys for an Automation Broker.

Download Splunk SOAR logs

When you troubleshoot issues relating to the Automation Broker, you will need to download the logs from Splunk SOAR first.

1. Log in to your Splunk SOAR instance.
2. In Splunk SOAR, click Home > System Health > Debugging.
3. Click the Download Logs button, then select either Recent Logs or All Logs.
4. Splunk SOAR downloads a file named phantom-logs-<type>-<YYYY-MM-DD-24HR>.tgz where type is either "recent" or "all".
5. Extract the phantom-logs file

   tar -xvf <phantom-logs file>

Match the Automation Broker to its log file

You'll need to know which log file goes with which automation broker.

Automation Broker logs are named with the pattern broker_broker-<UUID>_localsplunk.log. To match a log with its automation broker, you need the UUID of the automation broker.

1. From either the /shared_data/broker directory on your Docker host or the /broker directory on the Automation Broker docker container, read the file brokerd.conf. See Change Automation Broker settings by editing brokerd.conf in Customize the Splunk SOAR Automation Broker.
2. Find the broker_uuid entry.
3. The matching log is named broker_broker-<UUID you found in brokerd.conf>_localsplunk.log and is found in the /opt/phantom/private/logs-<hash>/phantom/ directory created when you extracted your downloaded logs.

Rotate the encryption keys for the Splunk SOAR Automation Broker

Splunk SOAR automatically rotates encryption keys every thirty (30) days for normal operations. However, you can manually rotate the encryption keys for the Splunk SOAR Automation Broker.

• If pairing with your Splunk SOAR instance failed during installation.
• If the encryption keys are about to or have reached their scheduled expiration date.
• If actions are failing on the Automation Broker and errors about 'missing scs_identity' appear in the automation broker's log.

Method 1: Use the Splunk SOAR user interface

1. Log in to your Splunk SOAR instance using an account with permissions to manage system settings.
2. In Splunk SOAR, click Home > Administration, then navigate to Product Settings and select Automation Broker.
3. Locate the automation broker for which you want to rotate the encryption keys in the list, and click the ellipsis ( ... ) menu.
4. Select Rotate encryption keys.
5. (Conditional) If the credentials for this Automation Broker have expired, select Get new credentials. This opens a modal dialog with new authentication credentials for your Splunk SOAR Automation Broker.
   1. (Conditional) Copy the credentials from the dialog.
   2. (Conditional) Run the docker command to update the credentials for the Automation Broker. Podman users would user podman instead of docker in the example command.
      

docker exec -ti <container_id> python3 /splunk/broker/bin/update_creds.py --new-creds "<copied_credentials>"
 docker <container_id> restart

Method 2: Use the Splunk SOAR REST API

From a terminal session, use cURL to send the rotate parameter the REST API /rest/automation_proxy/<id> endpoint.
Include the following items:

• A user account with permissions to manage system settings.
• The host (and optionally the port) of your Splunk SOAR instance.
• The numeric ID of the automation broker.

Example:

curl -X POST -u <username>:<password> -H "Content-Type: application/json" 'http://<host>:<port>/rest/automation_proxy/<id>' --data '{"rotate": true}'

Example response:

{
  "success": true,
  "id": 5,
  "message": "Rotated encryption keys, rotated on prem token"
}

See REST Automation Broker in REST API Reference for Splunk SOAR for more information.

Turn off TLS certificate verification for lab or test environments

You can turn off strict TLS certificate validation in the Splunk SOAR Automation broker. This would allow you to use a self-signed certificate in your testing or lab environment.

There are two environment variables for turning off certificate validation.

• PHANTOM_HTTPS_STRICT_TLS_AUTODETECT - Turns or or off auto-detection of strict TLS checking. Valid values are 0 for "off" or 1 for "on." Default is 1.
• PHANTOM_HTTPS_STRICT_TLS - Turns on or off strict TLS checking. Valid values are 0 for "off" or 1 for "on."

Both variables must be set to 0 in order to turn off strict TLS certificate validation. Default is 1.

If you see an error message about certificates signed by an unknown authority, you may need to disable TLS certificate validation for your testing environment.

[+] Running 0/0
⠿ automation_broker Error                                                                                                                                                                                                                               0.0s
Error response from daemon: Get "<my soar instance url>/v2/": x509: certificate signed by unknown authority

If you need to turn off the Splunk SOAR Automation Broker's strict TLS certificate verification and are using docker or podman compose to start your containers, do these steps before launching the Splunk SOAR Automation Broker's container:

1. Open the file docker-compose.yml in a text editor.
2. Add two environment variables to the environment: section:
   

   - PHANTOM_HTTPS_STRICT_TLS_AUTODETECT=0
   - PHANTOM_HTTPS_STRICT_TLS=0
   

3. Save and exit the docker-compose.yml file
4. Follow the rest of the installation instructions for either Docker or Podman hosts. See either Install Splunk Automation Broker on a Docker host or Install Splunk Automation Broker on a Podman host in this manual.

If you are starting the Splunk SOAR Automation Broker using an image file and the docker run or podman run commands, then you can pass environment variables to the Automation Broker on the command line. See Install Splunk Automation Broker when you cannot use DockerHub

Example:

docker run --env PHANTOM_BASE_URL=<PHANTOM_BASE_URL> --env PHANTOM_HTTPS_STRICT_TLS_AUTODETECT=0 --env PHANTOM_HTTPS_STRICT_TLS=0 -v <path/to/your/data_directory>/splunk_data -d phantomsaas/automation_broker:<version>

Add an insecure Docker registry for lab or testing environments

Your testing environment may not be able to connect to some internet resources. You can add an insecure registry to your Docker configuration for your testing environment.

The instructions in this section are not secure and should not be used in production environments.

You can add the registry using Docker Desktop, or manually by editing the Docker daemon configuration file.

Add the insecure registry using Docker Desktop.

1. Start Docker Engine GUI.
2. Open Settings from the gear icon in the upper right of interface.
3. Select Docker Engine.
4. Modify the insecure-registries field. Use the IP address of the registry you wish to use.

   { "insecure-registries": ["<IP_ADDRESS>"] }

5. Click the button marked Apply & restart.
6. You may need to set the permissions on the /splunk_data directory.

   chown 1000:1000 <path/to/your/data_directory>/splunk_data
   chmod 770 <path/to/your/data_directory>/splunk_data
   

   You are responsible for setting the appropriate filesystem permissions on this directory to prevent unauthorized access.

Add the insecure registry manually:

1. Create or edit the Docker daemon configuration file. On Linux systems it is typically found at /etc/docker/daemon.json. Check the Docker docs for other operating systems.
   1. Modify the insecure-registries field. Use the IP address of the registry you wish to use.

      { "insecure-registries": ["<IP_ADDRESS>"] }

   2. Restart the Docker daemon.

      sudo systemctl restart docker

2. Set the permissions on the /splunk_data directory used by your Splunk SOAR Automation Broker.

   chown 1000:1000 <path/to/your/data_directory>/splunk_data
   chmod 770 <path/to/your/data_directory>/splunk_data
   

   You are responsible for setting the appropriate filesystem permissions on this directory to prevent unauthorized access.