Troubleshoot SAML SSO
Following are some common issues that you can encounter when you use Security Assertion Markup Language (SAML) as an authentication scheme with the Splunk platform. Read this topic to learn how to resolve those issues and ensure the security of your Splunk platform instance.
Changes to logging categories for the SAML authentication scheme
For version 8.1.0 of Splunk Enterprise and version 8.0.2007 of Splunk Cloud Platform and higher, various logging categories for the SAML authentication scheme have changed. Following is a summary of the changes:
- The `
AuthenticationManagerSAMLcategory name has been changed to
AuthenticationManagerSAMLcategory no longer handles logging for SAML scripted authentication extensions. Instead, they have their own category,
By default, these categories continue to provide logs at the INFO logging level. If you use Splunk Cloud Platform, contact support for information on how to change the levels on your instance. To learn more about logging levels and how to change them in Splunk Enterprise, see Enable debug logging in the Splunk Enterprise Troubleshooting Manual
Error message: SAML fails to verify assertions
When the Splunk platform cannot verify SAML assertions, you will see the following error message:
Failed to verify the assertion - The 'Audience' field in the saml response from the IdP does not match the configuration
- The SAML error appears in the splunkd.log on the search head. You can see the complete error message by running a search on that search head:
index=_internal sourcetype=splunkd SAML error
You should see something like the following:
09-18-2017 14:58:06.939 +0000 ERROR Saml - Failed to verify the assertion - The 'Audience' field in the saml response from the IdP does not match the configuration, Error details=Expected=https://<instance_name>.com, found=https://<wrong_instance_name>.com/
- Using a text editor, edit the
$SPLUNK_HOME/etc/system/local/authentication.confconfiguration file and add or replace the
entityIdthat appeared in the error message in Step 1.
[saml] entityId= https://<instance_name>.com/ (found from ERROR message)
authentication.conffrom Splunk Web at Settings > Access Controls > Authentication Method > Reload Authentication configuration
Error message: Leaf certificate does not match
If the signature certificate on the Splunk platform instance does not match the certificate that the IdP uses to sign SAML messages, you receive the following message:
No leaf certificate matched one from the assertion
If your signature verification certificate is a self-signed certificate:
Confirm that the certificate specified in the
idpCertPath setting in
authentication.conf is the same as the certificate the IdP uses to sign SAML messages. You can use OpenSSL to determine the details of the certificate that the Splunk platform uses for signature verification.
For example, the following command:
openssl x509 -in etc/auth/idpCerts/idpCert.pem -text -noout | grep 'Serial\|Issuer:\|Subject:'
produces information like the following:
Serial Number: 1478287046063 (0x15830c635af) Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, OU=Splunk Service, CN=5165ffd1bf1a0363c8a5cd8062337fb4 Subject: C=US, ST=CA, L=San Francisco, O=Splunk, OU=Splunk Service, CN=5165ffd1bf1a0363c8a5cd8062337fb4
If the signature verification certificate is part of a certificate chain
Confirm that the signing certificates match and are consistently named. For example, a simple chain would have three files in the following order:
- the root Certificate Authority (CA), for example: "
- the intermediate certificate, for example: "
- the leaf certificate or the signing certificate, for example: "
In this example, confirm that the "
cert_3.pem" (the leaf) is the same certificate that the IdP uses to sign responses.
If you have multiple chains, or chains with more than one intermediate CA
In most cases, the certificate chain consists of a single root certificate, a single intermediate certificate, and a single signing certificate. However, you might have multiple chains configured, or more than one intermediate CA.
If you have multiple chains configured, structure your certificate chain as follows:
$SPLUNK_HOME/etc/auth/idpCerts idpCertChain_1 idpCertChain_2 $SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_1 cert_1.pem cert_2.pem cert_3.pem $SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_2 cert_1.pem cert_2.pem cert_3.pem
If you have more than one intermediate CA
If you have more than one intermediate CA, structure your certificate chain as follows:
$SPLUNK_HOME/etc/auth/idpCerts idpCertChain_1 $SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_1 cert_1.pem cert_2.pem cert_3.pem cert_4.pem cert_5.pem
Error message: Attribute query request error
If the Splunk platform is not able to process attribute query requests (AQRs) properly, you will see a message like the following:
ERROR AuthenticationProviderSAML - Requesting user info from ID returned an error. Error in Attribute query request, AttributeQueryTransaction err=Cannot resolve hostname, AttributeQueryTransaction descr=Error resolving: Name or service not known, AttributeQueryTransaction statusCode=502
- Confirm that the
cipherSuiteis specified correctly in the SAML stanza. For example:
cipherSuite = TLSv1+MEDIUM:@STRENGTH
cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
- Confirm that all Simple Object Access Protocol (SOAP) password requirements are met.
- Confirm that your SSL settings for SAML are configured correctly in
If the Splunk platform instance cannot retrieve AQR attributes in the AuthRequest, you receive a message like the following:
ERROR AuthenticationProviderSAML - Attribute query request failed. Status code=urn:oasis:names:tc:SAML:2.0:status:UnknownPrincipal, Status msg=No attributes found for requested subject
- Make sure that the
realNameattributes are mapped to be returned back as part of
AuthnRequestand the Attribute Query Request.
Error message: SAML user missing roles
If the Splunk platform instance cannot correctly determine roles for a SAML-based user, you see the following message:
ERROR UserManagerPro - user="samluser1" had no roles
Confirm that the
rolemap_SAML stanza in the authentication.conf file contains the correct role mapping with ";" at the end of each role name.
User cannot login
User cannot log in after successful assertion validation. No valid Splunk role is found in the local mapping or in the assertion.
- Confirm that the
rolemap_SAMLstanza in the authentication.conf file contains proper mapping between roles returned from the IdP and the appropriate Splunk role.
- Confirm there are no spaces between, before, or after each role as defined in
authentication.conf. For example:
user = User;Employee
User cannot access SAML login page
Authentication is configured as SAML and the settings appear to be correct, but the login screen shows the page for Splunk authentication instead.
- Confirm that in the
appServerPortsis set to a valid port and not '0'.
- Confirm that the
web.conffile does not contain a value for
Error message: Failed to validate SAML logout response
When you log out of Splunk Enterprise or Splunk Cloud, you see the following error message:
Failed to validate SAML logout response received from IdP
This might be caused by IdPs that expect the Splunk platform to preserve uppercase letters in usernames. You can change the username to lowercase in the IdP or configure the IdP to accept the lowercase version of a username.
Cannot authenticate users for CLI commands
Unable to authenticate SSO users for CLI commands
The Splunk platform must use native authentication for CLI and REST API commands. This is because the user password is never sent in the SAML assertion. You can add the SAML users as native Splunk users.
Warning message: Found difference in original & scoped XML after signature verification
When the Splunk platform receives SAML XML from the IdP that contains whitespace, comments, or attributes that it does not require, it removes these elements from the XML as part of scoping the XML to meet the format requirements that the Splunk platform expects. When this happens, the Splunk platform logs the following warning message:
WARN Saml [23564 webui] - Found difference in original & scoped XML after signature verification
This is a benign message outside of any errors that the IdP returns at the same time. If you see this message in conjunction with an error, investigate and resolve the cause of the error first.
Additionally, you can configure your IdP so that it returns SAML XML without whitespace, comments, or unnecessary attributes.
Best practices for using SAML as an authentication scheme for single-sign on
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.1.0, 9.1.1