Preparing custom certificates for use with KV store
When you upgrade to Splunk Enterprise 9.4 or higher and KV store server version 7.0 or higher, you must pass a Certificate Authority (CA) verification check. After these upgrades, KV store does not function properly unless you have the appropriate CA certificates.
In a single-instance Splunk Enterprise deployment, the CA verification check is only performed against itself (that single server). In a clustered deployment, however, the CA verification check is performed against itself and all KV store peers.
Complete the following steps to prepare and verify your certificates. This process prevents certificate-related issues after your upgrade to Splunk Enterprise 9.4 or higher.
-
To identify which certificates your KV store is currently using, enter one of the following commands in the command-line interface (CLI):
These commands apply only to standard Windows and Unix operating system. If you are using a FIPS or Common Criteria operating system, then do not use these commands. Use the commands in the next step instead.
-
Unix:
$SPLUNK_HOME/bin/splunk cmd btool server list kvstore
-
Windows:
$SPLUNK_HOME\bin\splunk.exe cmd btool server list kvstore
-
Unix:
-
If you are using a FIPS or Common Criteria operating system, or if the output in the previous step showed
sslVerifyServerCert
is set tofalse
, then enter one of the following commands in the CLI. Otherwise, skip this step.-
Unix:
$SPLUNK_HOME/bin/splunk cmd btool server list sslConfig
-
Windows:
$SPLUNK_HOME\bin\splunk.exe cmd btool server list sslConfig
Make a note of the value on the the
kvstore
stanza in the command's output. -
Unix:
-
From the same output that you used in the first two steps, make a note of the file path for the following parameters:
serverCert
sslRootCAPath
If
sslRootCAPath
is not present, you can note down the file path aftercaCertFile
instead in its place. This option is deprecated.sslRootCAPath
takes precedence overcaCertFile
. Do not usecaTrustStore
. -
Using the file paths you noted in the previous step, enter the following command into the CLI:
-
Unix:
$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
-
Windows:
$SPLUNK_HOME\bin\splunk.exe cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
-
Unix:
-
To verify that your certificate is properly signed, ensure the output from the previous step matches the following example response, where instead of this example file path, it shows the file path to your server certificate:
$SPLUNK_HOME/etc/auth/server.pem: OK
If you see any output other than
OK
, you are missing one or more CA certificates from yoursslRootCAPath
orcaCertFile
file. You must locate these certificates and append them to the existingsslRootCAPath
orcaCertFile
file, then run the command again until you receive anOK
status.The
sslRootCAPath
file could be located on a deployment server, a search head cluster member, a cluster manager, or a third party server. If you edit this file, ensure you edit it in the correct location for your deployment.
In the command's output, if sslVerifyServerCert
is set to true
, make a note of the value on the kvstore
stanza and skip the next step. However, if sslVerifyServerCert
is not present or it's set to false
, use the commands in the next step instead.
Troubleshooting your certificates
See the following documentation for more information about troubleshooting your certificates:
- How to prepare TLS certificates for use with the Splunk platform in the Security Splunk Enterprise manual.
- Configure TLS certificates for inter-Splunk communication in the Securing Splunk Enterprise manual.
- Securing the Splunk platform with TLS in the Splunk Lantern Customer Success Center.
Upgrade the KV store server version | Manage the KV store workload with read-only mode |
This documentation applies to the following versions of Splunk® Enterprise: 9.4.2
Feedback submitted, thanks!