Test and troubleshoot TLS connections

After you configure TLS on your Splunk Platform instance endpoints, test them as soon as possible to ensure that they work properly and that the instances communicate using the certificates you have configured.

As a protocol, TLS is very descriptive when it encounters errors in configuration. A good sign that you have performed your TLS configurations properly is the lack of error messages that appear on startup of the Splunk daemon and during connections to other Splunk instances, services, and processes.

When you do encounter problems with TLS connections, the problems arise in one of the following areas:

• Host names or DNS misconfigurations. TLS certificates require valid hostnames and a working domain name service to function properly. If you haven't specified the correct hostnames in the certificates, or your DNS isn't working, machines won't be able to use the certificates to connect to other machines.
• Certificate misconfigurations. It is very important that TLS certificates be configured correctly. If you generated your own certificates you must ensure that the private key you used to create the certificate is also included in the certificate file. You must reference and include, where applicable, the correct certificate authority certificate. If you use a certificate chain, the certificates within the chain must be in the correct order, beginning with the server certificate, including any intermediate certificates, and final the certificate authority certificate.
• Configuration problems. Confirm that you reference the correct Splunk configuration file as well as the correct stanza within the file for the service that you want to secure. Confirm that settings within the TLS stanzas reference the correct files, the files are in the correct format, usually the privacy-enhanced mail format, and the files themselves are formatted properly, with the correct certificate and private key tags around each individual certificate text blob. If you use a password for your certificates, ensure that you have configured the Splunk platform with the passwords for those certificates.

In general, a TLS error usually does not prevent the splunk daemon from starting. This means that you can review the splunkd.log file during startup to see where TLS connection errors happen. You can also use Splunk Web or the CLI to run a search that shows the status of your TLS connections. The following search shows a list of all connections that the instance attempted to make using TLS:

index=_internal source=*metrics.log* group=tcpin_connections | dedup hostname | table _time hostname version sourceIp destPort ssl

You can also view the splunkd.log file directly to validate and troubleshoot your configuration. The splunkd.log file is located at $SPLUNK_HOME/var/log/splunk/splunkd.log.

Following is how a successful connection might appear in splunkd.log on an indexer:

TcpInputProc - Connection in cooked mode from 10.1.12.111
TcpInputProc - Valid signature found
TcpInputProc - Connection accepted from 10.1.12.111

Following is how a successful connection might appear in the splunkd.log file on a forwarder:

TcpOutputProc - attempting to connect to 10.1.12.112:9997...
TcpOutputProc - Connected to 10.1.12.112:9997

Troubleshoot TLS connections

1. From a command or shell prompt, run the following command to test your certificates:

   openssl s_client -connect {server}:{port}
   

   
   where server is the name of the instance to which you want to connect and <port> is the network port for the service on the instance that you want to test, or example 8000 For Splunk Web, 8089 for the management port,, 9997 or 9998 for forwarding data, etc.
   

   The command returns a message similar to the following with a good certificate:

   Verify return code: 0 (ok)

2. On Splunk indexers and forwarders, review the $SPLUNK_HOME/var/log/splunk/splunkd.log file for errors.
   • On indexers, look for messages from the TCP input processor (TcpInputProc).
   • On forwarders, check for messages from the TCP output processor (TcpOutputProc).
3. If necessary, increase the logging level of the appropriate processors on the indexer and the forwarder by editing the $SPLUNK_HOME/etc/log.cfg file.
   • On the forwarder, set category.TcpOutputProc=DEBUG.
   • On the indexer set category.TcpInputProc=DEBUG.

   Restart Splunk Enterprise for these changes to take effect, then observe the start-up sequence for the pertinent component. This method reveals most configuration issues explicitly.

4. Check the TLS configuration using the splunk btool command as follows:
   • On the indexer, run the following:
     $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
   • On the forwarder, run the following:
     $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

Common problems with TLS and TLS certificates

Following are some common problems that you can

• The path to the server certificate file set as the value of the serverCert setting in the inputs.conf file is wrong, or the instance cannot read the file. This results in an error similar to the following:

ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory

• The password to the RSA private key that is in the server certificate file is wrong. This generates an error similar to the following:

ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key that is in the file with the command:

openssl rsa -in /opt/splunk/etc/auth/server.pem -text

On Windows, you can manually test the password of the RSA key using the following command:

openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text