Troubleshoot your forwarder to indexer authentication

1. Test your certificates:

openssl s_client -connect {server}:{port}

Port 8000, 8060, 8089, 9998, etc.

A good certificate will return the following or something similar:

Verify return code: 0 (ok)

2. Check $SPLUNK_HOME/var/log/splunk/splunkd.log (indexer and forwarder) for errors. On the indexer, check for the messages from the TCP input processor TcpInputProc. On the forwarder, check the messages from the TCP output processor TcpOutputProc.

3. Increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.

On the forwarder, set category.TcpOutputProc=DEBUG, on the indexer set category.TcpInputProc=DEBUG.

4. Restart Splunk Enterprise for these to take effect and observe the start-up sequence for the pertinent component. Most configuration issues are explicitly revealed by this method.

5. Check the SSL configuration using btool as follows:

On the indexer :

$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug

On the forwarder :

$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

Common problems

• The path to the server certificate file set as the value of serverCert in inputs.conf is wrong, or the file cannot be read. This will generate the following error :

12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory

• The password to the RSA private key contained in the server certificate file is wrong.

12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file with the comand:

# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

On Windows, you can manually test the password of the RSA key using the following command:

>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text