Splunk Cloud Platform

Admin Config Service Manual

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Troubleshoot ACS error messages

The table lists common ACS API error messages and suggested troubleshooting steps you can take to investigate and resolve these issues.

Feature Error message Troubleshooting steps
All ACS endpoints 401-unauthorized: call not properly authenticated Possible causes:
  • JWT token has expired or is not valid yet. Check the Expires On and Not Before token fields. If token has expired, create a new valid token and try again.
  • Make sure to specify the JWT token value and not the token-id value in the Auth header.
  • If attempting to target a search head, the token may be valid, but not for the target search head. When targeting a search head, you must authenticate with a JWT token created on that specific search head. You can create such a token using the search head prefix with the ACS tokens endpoint. For more information, see Targeting a search head.
401-unauthorized","message":"failed to parse token Possible causes:
  • Incomplete or invalid JWT token specified in Auth header.
  • Make sure to specify the JWT token value and not the token-id value in the Auth header.
403-forbidden: operation not supported for splunk stack, please refer to the API documentation for limitations and more detail Check ACS feature compatibility with your Splunk Cloud Platform deployment Experience. See ACS compatibility matrix.
403-forbidden: insufficient permission(s) to perform this operation Check RBAC table to ensure the user has the right capability(s) to access the endpoint. See Required ACS capabilties.
403-forbidden: user does not have the required capability: [x] Check RBAC table to ensure the user has the right capability(s) to access the endpoint. See Required ACS capabilties.
403-forbidden: user does not have one of the required roles: [x1, x2] User must have one of the listed roles to access the endpoint.
404-stack-not-found Make sure stack name is correct. Each Splunk Cloud Platform deployment is identified by the stack-name, which is the prefix of the deployment's URL. For example, if your deployment's URL is "https://my-company-name.splunkcloud.com" the stack-name is "my-company-name".
409-object-already-exists An object with the same name already exists. To update the object, use PATCH or PUT based on the specific ACS API feature.
500-internal-server-error Make sure your deployment has one or more separate search heads or a search head cluster. ACS is not supported on single instance deployments.
503-service-unavailable
"message": "service temporarily unavailable, please try again later."
Multiple concurrent requests can cause the service to become unavailable. Retry your request at a later time.
424-failed-dependency: A deployment task is still in progress. Please try again later. On Classic Experience, if a previous POST/PUT/DELETE/PATCH request is in progress, subsequent requests cannot be made. Retry your request when previous request completes. See Retry failed operations in Splunk Cloud Platform.
Configure IP allow lists 400-bad-request cannot remove all access to stack's <feature> This check prevents users from removing all subnets for a certain feature. It ensures there's at least one subnet, preventing users from cutting off all access to their stack for ingestion.
400-bad-request missing access feature parameter You must specify the {feature} parameter with the ACS request, for example search-api, hec,s2s, and so on. For more information, see Determine IP allow list use case.
400-bad-request
"message": "SH allowlist count of 242 exceeds maximum allowed count of 230"
Make sure the aggregate number of IP subnet rules for the IP allowlist feature group (search heads, indexers, IDMs, or single instance) does not exceed 230. For more information, see IP allow list behavior and IP subnet limits.
400-bad-request subnet overlaps splunk's reserved IP blocks Refer to Splunk Cloud Platform's reserved IP block:

"54.147.174.149/32"
"3.210.45.80/32"
"50.19.33.107/32"
"18.208.12.214/32"
"35.168.6.234/32"
"54.227.109.89/32"

400-bad-request cannot delete splunk's reserved subnet
Configure outbound ports 400-bad-request invalid port number provided. Please retry with a port number within range 1-65535 Determine correct port number and try again with a port number between 1-65535. Not including ports 443, 9997, 2525.
400-bad-request
"message": "outbound rules for port 443 may not be added."


Manage app permissions 404-app-not-found: {app} app not found User inputted a nonexistent app or user's role does not have read permission.
400-bad-request: Read or write permissions list may not be empty. User inputted an empty array as one of the arguments to PATCH.
400-bad-request: invalid role provided.
<name of offending role> role does not exist.
User inputted an invalid role as an argument in PATCH. Role may be an existing role that the user does not have access to.
400-bad-request: Invalid roles set. * cannot include other roles. User inputted *, meaning all roles, alongside other roles to PATCH. Remove the duplicate roles.
400-bad-request: Invalid role set. Roles with write permission must have read permission. User inputted *, meaning all roles, alongside other roles to PATCH. Remove the duplicate roles.
Manage HTTP Event Collector (HEC) tokens 404-hec-not-found If returned by GET request after creating new hec-token, token creation is still in progress. It might take a few minutes for hec-token creation process to complete. Repeat GET request until status code is 200.
424-failed-dependency
"message": "Cannot update HEC tokens because the previous deployment task has failed. Retry the latest deployment task using the following ACS API retry endpoint.
If returned after running an HEC token management operation, send a POST request to the deployment/retry endpoint to retry the failed operation. See Retry failed operations in Splunk Cloud Platform.
Manage indexes 403-forbidden internal index names cannot be used Refer to list of internal indexes:

"Main"
"History"
"Splunklogger"
"Summary"
"lastchanceindex"

403-forbidden internal index names cannot be deleted
400-bad-request internal index names are not allowed for defaultIndex
404-index-not-found If returned by GET request after creating new index, index creation is still in progress. It might take a few minutes for the index creation process to complete. Repeat GET request until status code is 200.
Manage limits.conf configurations 403-forbidden - not allowed to access stanza Specified limits.conf stanza is not supported by ACS. For more information on limits.conf error messages, see Manage limits.conf configurations.
Manage private apps 400-bad-request This app has failed AppInspect validation Possible causes:
  • Appinspect validation failures. Verify the failure count in the Appinspect report.
  • Appinspect version 3.3.0 and higher adds a check_package_compression validation check. Make sure to use the -z flag to compress the file when tarring an app package. For example: tar -zcvf acs_test_app.
  • Incorrect Appinspect tags. Make sure to specify the correct tags, as follows:
    • For Victoria Experience:
      • Version 8.2.2111 and lower, correct tags are 'included_tags="cloud,self-service"'.
      • Version 8.2.2112-8.2203, correct tag is 'included_tags="private_app"'.
      • Version 9.0.2205 and higher, no tag required when sending POST request to app/validate endpoint. But make sure to specify query parameter ?included_tag=private_victoria, when sending GET requests to the app/validate/status/{request_id} and app/report/{request_id} endpoints. See Validate your private app (Victoria Experience).
    • For Classic Experience:
      • Version 8.2.2203 and lower, the correct tag is 'included_tags="private_app"'.
      • Version 9.0.2205 and higher, no tag required when sending POST requests to app/validate endpoint. But make sure to specify query parameter ?included_tag=private_classic, when sending GET requests to the app/validate/status/{request_id} and app/report/{request_id} endpoints. See Validate your private app (Classic Experience).
  • Splunk Cloud Platform versions prior to 8.2.2111 might have manual checks failing Appinspect validation.
  • Check if the user doesn't override value of private_app_vetting_global in server.conf
  • Make sure the App-inspect token is not expired. It expires in about ~1 day. Try with a fresh token.
  • If using Github private-app-demo or ACS CLI, Victoria stack must be on version 8.2.2112 or highger.
400-bad-request "message":"AppUpload_InspectValidation: Unable to retrieve AppInspect validation result. Try again later. Possible causes:
  • Incorrect Appinspect tags. Make sure to specify the correct tags, as follows:
    • For Victoria Experience:
      • Version 8.2.2111 and lower, correct tags are 'included_tags="cloud,self-service"'
      • Version 8.2.2112-8.2203, correct tag is 'included_tags="private_app"'.
      • Version 9.0.2205 and higher, no tag required when sending POST request to app/validate endpoint. But make sure to specify query parameter ?included_tag=private_victoria, when sending GET requests to the app/validate/status/{request_id} and app/report/{request_id} endpoints. See Validate your private app (Victoria Experience).
    • For Classic Experience:
      • Version 8.2.2203 and lower, the correct tag is 'included_tags="private_app"'.
      • Version 9.0.2205 and higher, no tag required when sending POST request to app/validate endpoint. But make sure to specify query parameter ?included_tag=private_classic, when sending GET requests to the app/validate/status/{request_id} and app/report/{request_id} endpoints. See Validate your private app (Classic Experience).
400-bad-request Extract app information from the package failed Possible causes:
  • Occurs when your file is not added to the app package or you specify an incorrect path. Make sure to specify a valid app installation package.
  • Invalid app-package file format. App package format must be .tgz, .tar.gz, .spl or .tar file.
Web Application Firewall (WAF) related errors 403-forbidden A 403 response with awselb in response header indicates the request was blocked by WAF. Possible fixes:
  • For private apps:
    • Make sure to specify the complete file path, for example /Users/<username>/downloads/app.tgz without ../../ in the path.
    • Try to untar and tar the file again. For example tar -xvf {app_file}.tgz and COPYFILE_DISABLE=1 tar cvf {app_file}.tgz {app_file}.
  • Make sure to specify User Agent Header.
  • Send request again on different machine and network.

If still blocked, contact Splunk Support for assistance.

429-too-many-request Check the request rate. How many requests are you sending? Have you exceeded the rate limit of 300 reqs/min?

For further assistance with ACS errors, contact Splunk Support.

Last modified on 08 March, 2024
PREVIOUS
Administer Splunk Cloud Platform using the ACS CLI
 

This documentation applies to the following versions of Splunk Cloud Platform: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308 (latest FedRAMP release), 9.1.2312


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters