Troubleshoot SAML SSO

Following are some common issues that you can encounter when you use Security Assertion Markup Language (SAML) as an authentication scheme with the Splunk platform. Read this topic to learn how to resolve those issues and ensure the security of your Splunk platform instance.

Changes to logging categories for the SAML authentication scheme

For version 8.1.0 of Splunk Enterprise and version 8.0.2007 of Splunk Cloud Platform and higher, various logging categories for the SAML authentication scheme have changed. Following is a summary of the changes:

• The `AuthenticationManagerSAML category name has been changed to AuthenticationProviderSAML.
• The AuthenticationManagerSAML category no longer handles logging for SAML scripted authentication extensions. Instead, they have their own category, AuthenticationProviderScripted.

By default, these categories continue to provide logs at the INFO logging level. If you use Splunk Cloud Platform, contact support for information on how to change the levels on your instance. To learn more about logging levels and how to change them in Splunk Enterprise, see Enable debug logging in the Splunk Enterprise Troubleshooting Manual

Error message: SAML fails to verify assertions

When the Splunk platform cannot verify SAML assertions, you will see the following error message:

Failed to verify the assertion - The 'Audience' field in the saml response 
from the IdP does not match the configuration

Mitigation

1. The SAML error appears in the splunkd.log on the search head. You can see the complete error message by running a search on that search head:

   index=_internal sourcetype=splunkd SAML error

   You should see something like the following:

   09-18-2017 14:58:06.939 +0000 ERROR Saml - Failed to verify the assertion - The 'Audience' 
   field in the saml response from the IdP does not match the configuration, Error 
   details=Expected=https://<instance_name>.com, found=https://<wrong_instance_name>.com/

2. Using a text editor, edit the $SPLUNK_HOME/etc/system/local/authentication.conf configuration file and add or replace the entityId that appeared in the error message in Step 1.

   [saml]
   entityId= https://<instance_name>.com/ (found from ERROR message) 

3. Reload authentication.conf from Splunk Web at Settings > Access Controls > Authentication Method > Reload Authentication configuration

Error message: Leaf certificate does not match

If the signature certificate on the Splunk platform instance does not match the certificate that the IdP uses to sign SAML messages, you receive the following message:

No leaf certificate matched one from the assertion

Mitigation

If your signature verification certificate is a self-signed certificate:

Confirm that the certificate specified in the idpCertPath setting in authentication.conf is the same as the certificate the IdP uses to sign SAML messages. You can use OpenSSL to determine the details of the certificate that the Splunk platform uses for signature verification.

For example, the following command:

openssl x509 -in etc/auth/idpCerts/idpCert.pem -text -noout |  grep 'Serial\|Issuer:\|Subject:'

produces information like the following:

Serial Number: 1478287046063 (0x15830c635af)

Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, OU=Splunk Service, 
CN=5165ffd1bf1a0363c8a5cd8062337fb4

Subject: C=US, ST=CA, L=San Francisco, O=Splunk, OU=Splunk Service, 
CN=5165ffd1bf1a0363c8a5cd8062337fb4

If the signature verification certificate is part of a certificate chain

Confirm that the signing certificates match and are consistently named. For example, a simple chain would have three files in the following order:

• the root Certificate Authority (CA), for example: "cert_1.pem"
• the intermediate certificate, for example: "cert_2.pem"
• the leaf certificate or the signing certificate, for example: "cert_3.pem"

In this example, confirm that the "cert_3.pem" (the leaf) is the same certificate that the IdP uses to sign responses.

If you have multiple chains, or chains with more than one intermediate CA

In most cases, the certificate chain consists of a single root certificate, a single intermediate certificate, and a single signing certificate. However, you might have multiple chains configured, or more than one intermediate CA.

If you have multiple chains configured, structure your certificate chain as follows:

$SPLUNK_HOME/etc/auth/idpCerts
idpCertChain_1 idpCertChain_2
$SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_1
cert_1.pem  cert_2.pem  cert_3.pem
$SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_2
cert_1.pem  cert_2.pem  cert_3.pem

If you have more than one intermediate CA

If you have more than one intermediate CA, structure your certificate chain as follows:

$SPLUNK_HOME/etc/auth/idpCerts
idpCertChain_1
$SPLUNK_HOME/etc/auth/idpCerts/idpCertChain_1
cert_1.pem cert_2.pem cert_3.pem cert_4.pem cert_5.pem

Error message: Attribute query request error

If the Splunk platform is not able to process attribute query requests (AQRs) properly, you will see a message like the following:

ERROR AuthenticationProviderSAML - Requesting user info from ID returned an error. Error in Attribute
query request, AttributeQueryTransaction err=Cannot resolve hostname, AttributeQueryTransaction 
descr=Error resolving: Name or service not known, AttributeQueryTransaction statusCode=502

Mitigation

• Confirm that the cipherSuite is specified correctly in the SAML stanza. For example:
  • cipherSuite = TLSv1+MEDIUM:@STRENGTH
  • cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

• Confirm that all Simple Object Access Protocol (SOAP) password requirements are met.

• Confirm that your SSL settings for SAML are configured correctly in authentication.conf.

If the Splunk platform instance cannot retrieve AQR attributes in the AuthRequest, you receive a message like the following:

ERROR AuthenticationProviderSAML - Attribute query request failed. Status 
code=urn:oasis:names:tc:SAML:2.0:status:UnknownPrincipal, Status msg=No attributes found for 
requested subject

Mitigation

• Make sure that the role, mail, and realName attributes are mapped to be returned back as part of AuthnRequest and the Attribute Query Request.

Error message: SAML user missing roles

If the Splunk platform instance cannot correctly determine roles for a SAML-based user, you see the following message:

ERROR UserManagerPro - user="samluser1" had no roles

Mitigation

Confirm that the rolemap_SAML stanza in the authentication.conf file contains the correct role mapping with ";" at the end of each role name.

User cannot login

User cannot log in after successful assertion validation. No valid Splunk role is found in the local mapping or in the assertion.

Mitigation

• Confirm that the rolemap_SAML stanza in the authentication.conf file contains proper mapping between roles returned from the IdP and the appropriate Splunk role.

• Confirm there are no spaces between, before, or after each role as defined in authentication.conf. For example:

user = User;Employee

User cannot access SAML login page

Authentication is configured as SAML and the settings appear to be correct, but the login screen shows the page for Splunk authentication instead.

Mitigation

• Confirm that in the web.conf configuration file, appServerPorts is set to a valid port and not '0'.
• Confirm that the web.conf file does not contain a value for trustedIP.

Error message: Failed to validate SAML logout response

When you log out of Splunk Enterprise or Splunk Cloud, you see the following error message:

Failed to validate SAML logout response received from IdP

Mitigation

This might be caused by IdPs that expect the Splunk platform to preserve uppercase letters in usernames. You can change the username to lowercase in the IdP or configure the IdP to accept the lowercase version of a username.

Cannot authenticate users for CLI commands

Unable to authenticate SSO users for CLI commands

Mitigation

The Splunk platform must use native authentication for CLI and REST API commands. This is because the user password is never sent in the SAML assertion. You can add the SAML users as native Splunk users.

Warning message: Found difference in original & scoped XML after signature verification

When the Splunk platform receives SAML XML from the IdP that contains whitespace, comments, or attributes that it does not require, it removes these elements from the XML as part of scoping the XML to meet the format requirements that the Splunk platform expects. When this happens, the Splunk platform logs the following warning message:

WARN  Saml [23564 webui] - Found difference in original & scoped XML after signature verification

Mitigation

This is a benign message outside of any errors that the IdP returns at the same time. If you see this message in conjunction with an error, investigate and resolve the cause of the error first.

Additionally, you can configure your IdP so that it returns SAML XML without whitespace, comments, or unnecessary attributes.