Troubleshoot Splunk Secure Gateway performance issues

If you're experiencing connection issues with Splunk Secure Gateway, see Troubleshoot Secure Gateway Connection Issues.

If you're using a proxy, make sure to open port 443 outbound to prod.spacebridge.spl.mobi to enable Splunk Secure Gateway. See Splunk Secure Gateway requirements for more information.

Troubleshoot error codes

Refer to the following table if you receive an error code in Splunk Secure Gateway:

Error code	Meaning	What to do
301	Invalid Splunk platform instance URL	If you're using another service to reach Spacebridge, make sure your Splunk platform instance URL is correct in the configuration.
303	Redirect Error	If you're using another service to reach Spacebridge, make sure your Splunk platform instance URL is correct in the configuration.
305	Proxy Required	Make sure your proxy is properly configured. See Use a proxy server with Splunk Secure Gateway to learn how to configure a proxy for Splunk Secure Gateway.
400	Unable to complete request	Try again later or contact your Splunk administrator.
401	Authentication error	There's an error with the authentication process. Try clearing your cache, or see Troubleshoot registration issues and make sure your credentials are valid. Or Contact Splunk support and see Troubleshoot performance issues to troubleshoot further.
403	You don't have the right permissions.	See Configure Splunk Secure Gateway permissions.
405	Method not allowed.	Splunk Secure Gateway doesn't accept this request. You can Contact Splunk support.
404	Page not found	Make sure the URL is valid and try again.
406	Request not acceptable	Splunk Secure Gateway received an unexpected request. Modify your request and try again.
407	Proxy authentication required	Authenticate with your proxy and try again. See Use a proxy server with Splunk Secure Gateway for more information about using a proxy with Splunk Secure Gateway
408	Request timeout	Your request timed out. Try again.
409	Entity already exists	Create a different entity.
410	Page no longer exists	The URL no longer exists. Remove references to this URL.
413	Request payload is too large	Try the request again with a smaller payload
429	You've made too many requests	Splunk Secure Gateway cannot handle that many requests at a time. Lower your number of requests and try again.
431	Request header is too large	Make the request header smaller and try again.
500	The app is able to make the request to Spacebridge, but it encountered an internal error.	Contact Splunk support and see Troubleshoot performance issues to troubleshoot further.
502	Invalid response	Try again or Contact Splunk support.
503	The app is unable to connect to Spacebridge. The connection is unavailable or the Splunk Secure Gateway modular inputs aren't working.	Make sure you have internet connection, or Contact Splunk support.
504	Server timed out	Try again.
510	Not enough information	Provide more information and try again.
511	Network authentication required	Authenticate to network connection and try again.

Troubleshoot registration issues

If you or a Splunk admin changes your Splunk user credentials, your device becomes unregistered.

To register your device again, use Splunk Secure Gateway. An admin must enable token authentication so you can access the Connected Experiences apps again.

If token authentication isn't enabled, users aren't able to access the Connected Experiences apps. Enable token authentication so users can register their devices again after their credentials have been changed.

See Enable token authentication in the Splunk Enterprise Securing the Splunk Platform manual to learn how to enable token authentication. See Register your mobile device to a Splunk platform instance in the Use Splunk Secure Gateway manual for registration steps.

Troubleshoot performance issues

You can use the troubleshooting dashboards in Splunk Secure Gateway to run tests or you can use the command line.

Use the Splunk Secure Gateway troubleshooting dashboards

Splunk Secure Gateway version 2.5.4 and higher comes with troubleshooting dashboards to help identify possible issues. To view the troubleshooting dashboards, navigate to Dashboards > Troubleshooting Dashboards in Splunk Secure Gateway.

Splunk Support might ask you to use the following dashboards during the troubleshooting process.

Troubleshooting dashboard	How to use
Request tracing	You can troubleshoot and trace a particular request. Select a time range and a request ID to get troubleshooting logs and a stack trace. Or, you can get a request ID from the Connected Experiences mobile apps. See the following documentation to learn how to view logs and get a request ID from the Connected Experiences mobile apps: View or share logs to troubleshoot Splunk Mobile View or share logs to troubleshoot Splunk TV View or share logs to troubleshoot Splunk AR
Subscription Tracing	You can troubleshoot and trace a particular panel that isn't loading. Select a time range and a panel ID to get troubleshooting logs and a stack trace. Or, you can get a panel ID from the Connected Experiences mobile apps. See the following documentation to learn how to view logs and get a request ID from the Connected Experiences mobile apps: View or share logs to troubleshoot Splunk Mobile View or share logs to troubleshoot Splunk TV View or share logs to troubleshoot Splunk AR
Single Value Test	A single-value panel that tests if a simple dashboard loads in Splunk Secure Gateway. It helps identify possible causes for issues during troubleshooting.
End-to-End Websocket Test	Initiate a sample request to run an end-to-end websocket test for Splunk Secure Gateway, device authentication, and Spacebridge messaging. Select Single Request or Subscription Message and identify the request type. This dashboard requires JSON Web Tokens (JWT) to be enabled.
KV Store Collections Status	If you're running into issues with the Connected Experiences apps, Splunk Support might ask you to use this dashboard to check the KV Store status. This dashboard provides details about the KV Store collections status.

Use the command line to run a modular input test

To determine if the Splunk Secure Gateway modular inputs are running, run the following command at the command line for your operating system. The following example responses indicate that the modular inputs are running.

Operating System	Command	Response
Linux	$ ps ax | grep modular_input	25468 ? Ss 0:24 python /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_subscription_modular_input.py 26201 ? Ssl 0:28 python /opt/splunk/etc/apps/splunk_secure_gateway/bin/secure_gateway_modular_input.py
Windows	index=_internal sourcetype="secure_gateway_app_internal_log" ssg_subscription_modular_input OR secure_gateway_modular_input "Running"	INFO [secure_gateway_modular_input.app] [secure_gateway_modular_input] [do_run] [938] Running Splunk Secure Gateway modular input on search head, shard_id=splunkenterprise INFO [ssg_subscription_modular_input.app] [ssg_subscription_modular_input] [do_run] [977] Running Subscription Manager modular input on search head

Disable and re-enable the Splunk Secure Gateway modular inputs

If the Splunk Secure Gateway modular inputs aren't running, there might be a requirements issue. See Get Splunk Secure Gateway to make sure your Splunk platform meets the Splunk Secure Gateway requirements. Or you can disable and re-enable the following Splunk Secure Gateway modular inputs in Splunk Web in Settings > Data Inputs:

• Splunk Secure Gateway
• Splunk Secure Gateway Subscription Processor
• Splunk Secure Gateway Mobile Alerts TTL
• Splunk Secure Gateway Role Based Notification Manager
• Splunk Secure Gateway Metrics Collector
• Splunk Secure Gateway Subscription Clean Up
• Splunk Secure Gateway Registered Users List
• Splunk Secure Gateway Deleting Expired Tokens
• Splunk Secure Gateway Migration Modular Input

Troubleshoot data request issues

See the following suggestions to troubleshoot data request issues.

Unable to see a dashboard or panel

Configure the appropriate dashboard permissions to make dashboards visible in the Connected Experiences apps. See Configure Splunk Secure Gateway and dashboard permissions for more information about configuring dashboard permissions for the Connected Experiences apps. Apps without these permissions don't appear in the Connected Experiences apps.

If you can't find a dashboard you're looking for and you're sure that it contains only supported visualizations, make sure the app that the dashboard comes from is selected in the App Selection tab. See Select which Splunk apps to show dashboards from in the Connected Experiences apps.

If the dashboard you're looking for isn't on the dashboard list or a panel isn't loading, the dashboard may contain an unsupported visualization or configuration. For a list of supported visualizations and dashboard configurations, see Visualization support for the Connected Experiences apps in the Splunk Secure Gateway Release Notes manual.

Unable to see dashboard list

If you can't see your list of dashboards, you might be running into a service connection error. See Troubleshoot Splunk Secure Gateway Connection Issues in the Splunk Secure Gateway Use Splunk Secure Gateway manual.

Unable to send alerts to mobile devices from Splunk Enterprise Security

To send alerts from Splunk Enterprise Security, add splunk_secure_gateway to the update_es Application Inclusion List field in the App Imports Update page of Splunk Enterprise Security.

See Enable Splunk Enterprise Security mobile alerting for more information.

Expired license error

Splunk Secure Gateway requires a Splunk Cloud or Splunk Enterprise license. You might receive a notice saying that one of your licenses has expired. This notice appears when any of your Splunk licenses has expired. Splunk Secure Gateway functionality might not be affected if your Splunk platform license is still valid.

If you're experiencing issues with Splunk Secure Gateway and receive this notice, make sure your Splunk platform license is still valid.