Configure TLS certificates for inter-Splunk communication

You can use transport layer security (TLS) certificates to secure connections between various Splunk components. This is known as inter-Splunk communication. Typically, Splunk processes communicate on the TCP/IP network port 8089. This port is also known as the Splunk management port.

The certificates you use can replace the default certificates that Splunk provides. You can either obtain certificates from a certificate authority, or create and sign them yourself.

Prerequisites for configuring TLS certificates for inter-Splunk communication

Before you can secure communications between Splunk indexers and forwarders, you must have the following:

1. One or more TLS certificates.
   • You can either obtain third party certificates from a certificate authority, or create and sign them yourself
   • After you get the certificates, you must prepare the certificates for use with Splunk platform instances
   • The certificates must be in Privacy-Enhanced Mail format and comply with the x.509 public key certificate standard
   • You must have a private key file for each certificate file.
   • The key files that come with the certificates must comply with the Rivest-Shamir-Adelman (RSA) security algorithm.
2. Administrative access to the Splunk Enterprise instances that you want to secure
3. An understanding of Splunk configuration files

Configure TLS certificates for communications on the Splunk management port

You can configure TLS certificates on Splunk services that use the management network port.

When you configure Splunk Enterprise to use TLS certificates, upon restart, it changes the file permissions on the certificates so that only the user that Splunk Enterprise runs as has full access. This is by design, in line with security industry standards, and cannot be changed.

1. Open a shell or command prompt.
2. Using this prompt or file system management tools, copy the server certificate and the certificate authority public certificate into an accessible directory on the instance where you want to configure certificates. For example, you can move the files to a destination directory of $SPLUNK_HOME/etc/auth/mycerts/.
3. Use a text editor to open the $SPLUNK_HOME/etc/system/local/server.conf configuration file for editing.
4. In the server.conf file, configure the instance to use the server certificate. Add the following stanzas and settings to the file.

   Setting/stanza name	Data type	Description
   [sslConfig]	n/a	Sets up TLS/SSL configurations for inter-Splunk communication.
   caTrustStore (Optional)	string	The certificate authority (CA) trust store to use when validating a network connection over TLS. The setting has two values: "splunk" which is the trust store that the Splunk platform uses for certificates, as defined by the value for the 'sslRootCAPath' setting. When you specify this setting, the Splunk platform uses the certificates in this trust store to validate TLS connections. See the entry for that setting later in this table for information on configuring the 'sslRootCAPath' setting. This is the default value for 'caTrustStore'. "os" which is the CA trust store that the operating system uses. When you specify this setting, the Splunk platform uses the certificate in the trust store location that the operating system which runs the Splunk platform instance defines. Currently, there is support for Linux and Windows operating systems only. There is no support for macOS operating system trust stores. You can specify both of these values by separating them with a comma. When you do this, the Splunk platform looks in both the Splunk and OS trust stores for certificates to validate TLS connections. If the Splunk platform finds an identical TLS certificate in both trust stores, it uses the certificate in the Splunk trust store to validate the TLS connection.
   caTrustStorePath (Optional, required if 'caTrustStore' has a value of "os")	string	The certificate authority (CA) trust store path that the operating system uses, and that the Splunk platform is to use to find certificates for validating TLS connections. Currently, there is support for Linux operating systems only for this setting. There is no support for macOS, and Windows stores its certificates in the Windows Registry. There is no default value for this setting, so you must give it one if you give 'caTrustStore' a value of "os". The Splunk platform uses this setting if you give the 'caTrustStore' setting its default value of "os", or "splunk,os". It does not use this setting if you give the 'caTrustStore' setting a value of only "splunk". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in the Splunk trust store to validate the TLS connection. If you give 'caTrustStore' a value of "os" but don't specify a value for this setting, the Splunk platform doesn't attempt to look for certificates in the operating system trust store. It only looks for certificates in the Splunk trust store, and logs an error to the splunkd.log log file.
   serverCert	string	The location of the server certificate on the Splunk platform instance. This is the certificate that the machine uses to support inbound connections over TLS/SSL. You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myServerCert.pem, or you can use a relative path, such as etc/auth/mycerts/myServerCert.pem and the instance uses the Splunk platform instance installation directory.
   sslPassword (Optional)	string	The password that you entered when you created the certificate, if you created a password.
   requireClientCert (Optional)	Boolean	Whether or not the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed. A value of "true" means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate. Configure this setting to "true" if you want your receivers to require authentication with certificates. When both the forwarder and receiver have a "true" value for this setting, mutually authenticated TLS or mTLS is active.
   sslVersions (Optional)	comma-separated list	The list of SSL versions that the receiver supports. The Splunk platform supports the following versions for SSL and TLS: "ssl3", "tls1.0", "tls1.1", and "tls1.2".
   cipherSuite (Optional)	string	The list of cipher suite strings that the TLS/SSL sessions are to use.
   sslCommonNameToCheck (Optional except in certain circumstances)	comma-separated list	A list of one or more common names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the common names in this setting value matches the common name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.
   sslAltNameToCheck (Optional except in certain circumstances)	comma-separated list	A list of one or more alternate names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the alternate names in this setting value matches the alternate name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.
   sslRootCAPath	string	The location of the Splunk CA certificate trust store. The Splunk platform uses this path to find certificates to validate TLS connections depending on the value of the 'caTrustStore' setting. The Splunk platform uses this setting if you give the 'caTrustStore' setting its default value of "splunk" or "splunk,os". It does not use this setting if you give the 'caTrustStore' setting a value of only "os". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in this trust store to validate the TLS connection.

5. Save the server.conf file and close it.
6. Using the CLI, restart the splunkd process:

   # $SPLUNK_HOME/bin/splunk restart splunkd
   

Configuration file examples for configuring TLS certificates on the Splunk management port

Following is an example of a server.conf configuration file on a search head and search peer. The configuration is as follows:

• The search head uses a certificate that is in the /opt/splunk/etc/auth/mycerts directory called mySHServerCert.pem
• The instance uses a certificate authority certificate located at /opt/splunk/etc/auth/mycerts/myCACertificate.pem
• The server certificate was created with a password "mySHCertificatePassword"
• The instance requires a certificate from clients that connect to it to ensure mutually authenticated TLS (mTLS)
• The instance checks incoming certificates to ensure that the Common Name field in the certificate contains either "splunk1.mycompany.com" or "splunk2.mycompany.com"

[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySHServerCert.pem
sslPassword = mySHCertificatePassword
requireClientCert = true
sslVersions = *,-ssl2
sslCommonNameToCheck = splunk1.mycompany.com,splunk2.mycompany.com

If you supply a password for your server certificate in the server.conf file by providing a value for the sslPassword setting, the Splunk platform encrypts that password from clear text when you restart the Splunk platform instance.

Manage certificate authority (CA) certificate trust stores

On Splunk Enterprise, you can manage the trust stores that the Splunk platform uses for storage of certificate authority (CA) certificates.

The Splunk platform has access to two separate trust stores:

• The Splunk trust store. The Splunk platform uses this trust store for CA certificates to secure connections between Splunk services by default. The sslRootCAPath setting controls where this trust store is on the Splunk platform instance.
• The operating system trust store. Every operating system has a location where you can store CA certificates.

To configure where the Splunk platform looks for CA certificates to use for network connections using TLS, use the caTrustStore and caTrustStorePath settings in the server.conf configuration file. The Splunk platform looks in either the Splunk trust store or the operating system trust store for your CA certificates, or both, if you specify both values for the caTrustStore setting. See the table earlier in this topic for more information on these settings and their default values.

If you specify the "os" value for the caTrustStore setting, then you must also specify a value for the caTrustStorePath setting. This configures the Splunk platform to locate the directory on the operating system where the OS stores its CA certificates.

Next steps

Confirm that the forwarder and indexer configurations work properly. See Test and troubleshoot TLS connections.