Preparing custom certificates for use with KV store

When you upgrade to Splunk Enterprise 9.4 or higher and KV store server version 7.0 or higher, you must pass a Certificate Authority (CA) verification check. After these upgrades, KV store does not function properly unless you have the appropriate CA certificates.

In a single-instance Splunk Enterprise deployment, the CA verification check is only performed against itself (that single server). In a clustered deployment, however, the CA verification check is performed against itself and all KV store peers.

Complete the following steps to prepare and verify your certificates. This process prevents certificate-related issues after your upgrade to Splunk Enterprise 9.4 or higher.

1. To identify which certificates your KV store is currently using, enter one of the following commands in the command-line interface (CLI):

   These commands apply only to standard Windows and Unix operating system. If you are using a FIPS or Common Criteria operating system, then do not use these commands. Use the commands in the next step instead.

   • Unix:

     $SPLUNK_HOME/bin/splunk cmd btool server list kvstore
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd btool server list kvstore
     

In the command's output, if sslVerifyServerCert is set to true, make a note of the value on the kvstore stanza and skip the next step. However, if sslVerifyServerCert is not present or it's set to false, use the commands in the next step instead.

2. If you are using a FIPS or Common Criteria operating system, or if the output in the previous step showed sslVerifyServerCert is set to false, then enter one of the following commands in the CLI. Otherwise, skip this step.
   • Unix:

     $SPLUNK_HOME/bin/splunk cmd btool server list sslConfig
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd btool server list sslConfig
     

   Make a note of the value on the the kvstore stanza in the command's output.

3. From the same output that you used in the first two steps, make a note of the file path for the following parameters:
   • serverCert
   • sslRootCAPath

   If sslRootCAPath is not present, you can note down the file path after caCertFile instead in its place. This option is deprecated. sslRootCAPath takes precedence over caCertFile. Do not use caTrustStore.

4. Using the file paths you noted in the previous step, enter the following command into the CLI:
   
   • Unix:

     $SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
     

5. To verify that your certificate is properly signed, ensure the output from the previous step matches the following example response, where instead of this example file path, it shows the file path to your server certificate:

   $SPLUNK_HOME/etc/auth/server.pem: OK
   

   If you see any output other than OK, you are missing one or more CA certificates from your sslRootCAPath or caCertFile file. You must locate these certificates and append them to the existing sslRootCAPath or caCertFile file, then run the command again until you receive an OK status.

   The sslRootCAPath file could be located on a deployment server, a search head cluster member, a cluster manager, or a third party server. If you edit this file, ensure you edit it in the correct location for your deployment.

Troubleshooting your certificates

See the following documentation for more information about troubleshooting your certificates:

• How to prepare TLS certificates for use with the Splunk platform in the Security Splunk Enterprise manual.
• Configure TLS certificates for inter-Splunk communication in the Securing Splunk Enterprise manual.
• Securing the Splunk platform with TLS in the Splunk Lantern Customer Success Center.