Preparing custom certificates for use with KV store

When you upgrade to Splunk Enterprise 9.4 or higher and KV store server version 7.0 or higher, you must pass a Certificate Authority (CA) verification check. After these upgrades, KV store does not function properly unless you have the appropriate CA certificates.

In a single-instance Splunk Enterprise deployment, the CA verification check is only performed against itself (that single server). In a clustered deployment, however, the CA verification check is performed against itself and all KV store peers.

Complete the following steps to prepare and verify your certificates. This process prevents certificate-related issues after your upgrade to Splunk Enterprise 9.4 or higher.

1. To identify which certificates your KV store is currently using, enter one of the following commands in the command-line interface (CLI):
   • Unix:

     $SPLUNK_HOME/bin/splunk cmd btool server list kvstore
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd btool server list kvstore
     

2. In the command's output, if sslVerifyServerCert is set to true, then skip directly to Step 4. However, if sslVerifyServerCert is not present or it's set to false, discard the previous command's output, and use the commands in Step 3 instead.
3. If the output from Step 1 showed sslVerifyServerCert is set to false, then enter one of the following commands in the CLI. Otherwise, the following commands do not apply to you. Skip this step, and and go directly to Step 4.
   
   If you are using a FIPS or Common Criteria operating system, the following commands do not apply to you. Skip this step, and and go directly to Step 4.
   • Unix:

     $SPLUNK_HOME/bin/splunk cmd btool server list sslConfig
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd btool server list sslConfig
     

4. From whichever command was appropriate for your use case, make a note of the file path for the following parameters:
   • serverCert
   • sslRootCAPath

   If sslRootCAPath is not present, you can note down the file path after caCertFile instead in its place. This option is deprecated. sslRootCAPath takes precedence over caCertFile. Do not use caTrustStore.

5. Using the file paths you noted in the previous step, enter the following command into the CLI:
   
   • Unix:

     $SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
     

   • Windows:

     $SPLUNK_HOME\bin\splunk.exe cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
     

6. To verify that your certificate is properly signed, ensure the output from the previous step matches the following example response, where instead of this example file path, it shows the file path to your server certificate:

   $SPLUNK_HOME/etc/auth/server.pem: OK
   

   If you see any output other than OK, you are missing one or more CA certificates from your sslRootCAPath or caCertFile file. You must locate these certificates and append them to the existing sslRootCAPath or caCertFile file, then run the command again until you receive an OK status.

   The sslRootCAPath file could be located on a deployment server, a search head cluster member, a cluster manager, or a third party server. If you edit this file, ensure you edit it in the correct location for your deployment.

Troubleshooting your certificates

See the following documentation for more information about troubleshooting your certificates:

• How to prepare TLS certificates for use with the Splunk platform in the Security Splunk Enterprise manual.
• Configure TLS certificates for inter-Splunk communication in the Securing Splunk Enterprise manual.
• Securing the Splunk platform with TLS in the Splunk Lantern Customer Success Center.