Configure TLS certificates for inter-Splunk communication
You can use transport layer security (TLS) certificates to secure connections between various Splunk components. This is known as inter-Splunk communication. Typically, Splunk processes communicate on the TCP/IP network port 8089. This port is also known as the Splunk management port.
The certificates you use can replace the default certificates that Splunk provides. You can either obtain certificates from a certificate authority, or create and sign them yourself.
Prerequisites for configuring TLS certificates for inter-Splunk communication
Before you can secure communications between Splunk indexers and forwarders, you must have the following:
- One or more TLS certificates.
- You can either obtain third party certificates from a certificate authority, or create and sign them yourself
- After you get the certificates, you must prepare the certificates for use with Splunk platform instances
- The certificates must be in Privacy-Enhanced Mail format and comply with the x.509 public key certificate standard
- You must have a private key file for each certificate file.
- The key files that come with the certificates must comply with the Rivest-Shamir-Adelman (RSA) security algorithm.
- Administrative access to the Splunk Enterprise instances that you want to secure
- An understanding of Splunk configuration files
Configure TLS certificates for communications on the Splunk management port
You can configure TLS certificates on Splunk services that use the management network port.
When you configure Splunk Enterprise to use TLS certificates, upon restart, it changes the file permissions on the certificates so that only the user that Splunk Enterprise runs as has full access. This is by design, in line with security industry standards, and cannot be changed.
- Open a shell or command prompt.
- Using this prompt or file system management tools, copy the server certificate and the certificate authority public certificate into an accessible directory on the instance where you want to configure certificates. For example, you can move the files to a destination directory of
$SPLUNK_HOME/etc/auth/mycerts/
. - Use a text editor to open the
$SPLUNK_HOME/etc/system/local/server.conf
configuration file for editing. - In the server.conf file, configure the instance to use the server certificate. Add the following stanzas and settings to the file.
Setting/stanza name Data type Description [sslConfig] n/a Sets up TLS/SSL configurations for inter-Splunk communication. caTrustStore (Optional) string The certificate authority (CA) trust store to use when validating a network connection over TLS. The setting has two values: - "splunk" which is the trust store that the Splunk platform uses for certificates, as defined by the value for the 'sslRootCAPath' setting. When you specify this setting, the Splunk platform uses the certificates in this trust store to validate TLS connections. See the entry for that setting later in this table for information on configuring the 'sslRootCAPath' setting. This is the default value for 'caTrustStore'.
- "os" which is the CA trust store that the operating system uses. When you specify this setting, the Splunk platform uses the certificate in the trust store location that the operating system which runs the Splunk platform instance defines. Currently, there is support for Linux and Windows operating systems only. There is no support for macOS operating system trust stores.
You can specify both of these values by separating them with a comma. When you do this, the Splunk platform looks in both the Splunk and OS trust stores for certificates to validate TLS connections.If the Splunk platform finds an identical TLS certificate in both trust stores, it uses the certificate in the Splunk trust store to validate the TLS connection.
caTrustStorePath (Optional, required if 'caTrustStore' has a value of "os") string The certificate authority (CA) trust store path that the operating system uses, and that the Splunk platform is to use to find certificates for validating TLS connections. Currently, there is support for Linux operating systems only for this setting. There is no support for macOS, and Windows stores its certificates in the Windows Registry. There is no default value for this setting, so you must give it one if you give 'caTrustStore' a value of "os".
The Splunk platform uses this setting if you give the 'caTrustStore' setting its default value of "os", or "splunk,os". It does not use this setting if you give the 'caTrustStore' setting a value of only "splunk". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in the Splunk trust store to validate the TLS connection.If you give 'caTrustStore' a value of "os" but don't specify a value for this setting, the Splunk platform doesn't attempt to look for certificates in the operating system trust store. It only looks for certificates in the Splunk trust store, and logs an error to the splunkd.log log file.
serverCert string The location of the server certificate on the Splunk platform instance. This is the certificate that the machine uses to support inbound connections over TLS/SSL. You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myServerCert.pem
, or you can use a relative path, such asetc/auth/mycerts/myServerCert.pem
and the instance uses the Splunk platform instance installation directory.sslPassword (Optional) string The password that you entered when you created the certificate, if you created a password. requireClientCert (Optional) Boolean Whether or not the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed. A value of "true" means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate. Configure this setting to "true" if you want your receivers to require authentication with certificates. When both the forwarder and receiver have a "true" value for this setting, mutually authenticated TLS or mTLS is active. sslVersions (Optional) comma-separated list The list of SSL versions that the receiver supports. The Splunk platform supports the following versions for SSL and TLS: "ssl3", "tls1.0", "tls1.1", and "tls1.2". cipherSuite (Optional) string The list of cipher suite strings that the TLS/SSL sessions are to use. sslCommonNameToCheck
(Optional except in certain circumstances)comma-separated list A list of one or more common names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the common names in this setting value matches the common name in the certificate of the connecting client, the receiving instance declines the connection as not authorized. sslAltNameToCheck
(Optional except in certain circumstances)comma-separated list A list of one or more alternate names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the alternate names in this setting value matches the alternate name in the certificate of the connecting client, the receiving instance declines the connection as not authorized. sslRootCAPath string The location of the Splunk CA certificate trust store. The Splunk platform uses this path to find certificates to validate TLS connections depending on the value of the 'caTrustStore' setting.
The Splunk platform uses this setting if you give the 'caTrustStore' setting its default value of "splunk" or "splunk,os". It does not use this setting if you give the 'caTrustStore' setting a value of only "os". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in this trust store to validate the TLS connection. - Save the server.conf file and close it.
- Using the CLI, restart the splunkd process:
# $SPLUNK_HOME/bin/splunk restart splunkd
Configuration file examples for configuring TLS certificates on the Splunk management port
Following is an example of a server.conf configuration file on a search head and search peer. The configuration is as follows:
- The search head uses a certificate that is in the
/opt/splunk/etc/auth/mycerts
directory called mySHServerCert.pem - The instance uses a certificate authority certificate located at
/opt/splunk/etc/auth/mycerts/myCACertificate.pem
- The server certificate was created with a password "mySHCertificatePassword"
- The instance requires a certificate from clients that connect to it to ensure mutually authenticated TLS (mTLS)
- The instance checks incoming certificates to ensure that the Common Name field in the certificate contains either "splunk1.mycompany.com" or "splunk2.mycompany.com"
[sslConfig] enableSplunkdSSL = true sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem serverCert = /opt/splunk/etc/auth/mycerts/mySHServerCert.pem sslPassword = mySHCertificatePassword requireClientCert = true sslVersions = *,-ssl2 sslCommonNameToCheck = splunk1.mycompany.com,splunk2.mycompany.com
If you supply a password for your server certificate in the server.conf file by providing a value for the sslPassword
setting, the Splunk platform encrypts that password from clear text when you restart the Splunk platform instance.
Manage certificate authority (CA) certificate trust stores
On Splunk Enterprise, you can manage the trust stores that the Splunk platform uses for storage of certificate authority (CA) certificates.
The Splunk platform has access to two separate trust stores:
- The Splunk trust store. The Splunk platform uses this trust store for CA certificates to secure connections between Splunk services by default. The
sslRootCAPath
setting controls where this trust store is on the Splunk platform instance. - The operating system trust store. Every operating system has a location where you can store CA certificates.
To configure where the Splunk platform looks for CA certificates to use for network connections using TLS, use the caTrustStore
and caTrustStorePath
settings in the server.conf configuration file. The Splunk platform looks in either the Splunk trust store or the operating system trust store for your CA certificates, or both, if you specify both values for the caTrustStore
setting. See the table earlier in this topic for more information on these settings and their default values.
If you specify the "os" value for the caTrustStore
setting, then you must also specify a value for the caTrustStorePath
setting. This configures the Splunk platform to locate the directory on the operating system where the OS stores its CA certificates.
Next steps
Confirm that the forwarder and indexer configurations work properly. See Test and troubleshoot TLS connections.
Configure Splunk indexing and forwarding to use TLS certificates | Configure Splunk Web to use TLS certificates |
This documentation applies to the following versions of Splunk® Enterprise: 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!