Configure Splunk indexing and forwarding to use TLS certificates

You can use transport layer security (TLS) certificates to secure connections between forwarders and indexers.

The certificates you use can replace the default certificates that Splunk provides. You can either obtain certificates from a certificate authority, or create and sign them yourself.

Do not use these instructions to configure secure forwarding of data to a Splunk Cloud Platform instance. Instead, download and use the Splunk Cloud Universal Forwarder Credentials package and install it on your forwarding infrastructure. For details, see Install and configure the Splunk Cloud Platform universal forwarder credentials package in the Universal Forwarder Manual.

Prerequisites for configuring Splunk indexing and forwarding using TLS certificates

Before you can secure communications between Splunk indexers and forwarders, you must have the following:

1. One or more TLS certificates.
   • You can either obtain third party certificates from a certificate authority, or create and sign them yourself
   • After you get the certificates, you must prepare the certificates for use with Splunk platform instances
   • The certificates must be in Privacy-Enhanced Mail format and comply with the x.509 public key certificate standard
   • You must have a private key file for each certificate file.
   • The key files that come with the certificates must be in RSA security format.

Configure TLS certificates for indexers

You can configure indexers to use TLS certificates. The certificates you configure on indexers control how the indexer receives data from a forwarder.

When you configure Splunk Enterprise to use TLS certificates, upon restart, it changes the file permissions on the certificates so that only the user that Splunk Enterprise runs as has full access. This is by design, in line with security industry standards, and cannot be changed.

1. Open a shell or command prompt.
2. Using this prompt or file system management tools, copy the server certificate and the certificate authority public certificate into an accessible directory on the indexer where you want to configure certificates. For example, you can move the files to a destination directory of $SPLUNK_HOME/etc/auth/mycerts/.
3. Use a text editor to open the $SPLUNK_HOME/etc/system/local/inputs.conf configuration file for editing.
4. In the inputs.conf file, configure the indexer to use the server certificate. Add the following stanzas and settings to the file.

   Setting/stanza name	Data type	Description
   [splunktcp-ssl:<port>]	stanza	Defines a TCP network input to receive data over TLS/SSL on the port you specify.
   [SSL]	stanza	Defines the TLS/SSL settings for all inputs that you define for this instance.
   serverCert	string	The location of the server certificate on the Splunk platform instance. This is the certificate that the machine uses to support inbound connections over TLS/SSL. You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myServerCert.pem, or you can use a relative path, such as etc/auth/mycerts/myServerCert.pem and the instance uses the Splunk platform instance installation directory.
   sslPassword (Optional)	string	The password that you entered when you created the certificate, if you created a password. Do not configure this setting if you did not specify a password when you created your certificates.
   requireClientCert (Optional except in certain cases) If you want to use the Certificate Assist helper package for the Splunk Assist monitoring service, then this is a required setting.	Boolean	Whether or not the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed. A value of "true" means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate. Configure this setting to "true" if you want your receivers to require authentication with certificates. When both the forwarder and receiver have a "true" value for this setting, mutually authenticated TLS or mTLS is active.
   sslVersions (Optional)	comma-separated list	The list of SSL versions that the receiver supports. The Splunk platform supports the following versions for SSL and TLS: "ssl3", "tls1.0", "tls1.1", and "tls1.2".
   cipherSuite (Optional)	string	The list of cipher suite strings that the TLS/SSL sessions are to use.
   sslCommonNameToCheck (Optional except in certain cases)	comma-separated list	A list of one or more common names, or fully-qualified host names, upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the common names in this setting value matches the common name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.
   sslAltNameToCheck (Optional except in certain cases)	comma-separated list	A list of one or more alternate names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the alternate names in this setting value matches the alternate name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.

5. Save the inputs.conf file and close it.
6. On indexers that do not run on Windows, open the $SPLUNK_HOME/etc/system/local/server.conf configuration file for editing.
7. Add the following text to establish the location of the certificate authority certificate.
   

   [sslConfig]
   sslRootCAPath = <Absolute path to the CA certificate. The default value is $SPLUNK_HOME/etc/auth/cacert.pem>
   

8. Save the server.conf file and close it.
9. Using the CLI, restart the splunkd process:

   # $SPLUNK_HOME/bin/splunk restart splunkd
   

Configuration file examples for configuring TLS certificates on receiving indexers

Following is an example of an inputs.conf configuration file on a receiving indexer. The configuration is as follows:

• The indexer uses a certificate that is located in the /opt/splunk/etc/auth/mycerts directory called myServerCert.pem
• The server certificate was created with a password "myCertificatePassword"
• The indexer checks incoming certificates to ensure that the Common Name field in the certificate contains either "indexer1.mycompany.com" or "indexer2.mycompany.com"

[splunktcp-ssl:9997]
disabled=0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myServerCert.pem
sslPassword = myCertificatePassword
requireClientCert = true
sslVersions = *,-ssl2
sslCommonNameToCheck = indexer1.mycompany.com,indexer2.mycompany.com

If you supply a password for your server certificate in the inputs.conf file by providing a value for the sslPassword setting, the Splunk platform encrypts that password from clear text when you restart the Splunk platform instance.

The server.conf configuration file establishes and references the location of the certificate authority certificate:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem

Configure TLS certificates for forwarders

The certificates you configure on forwarders control how the forwarder connects to the indexer and how it communicates with the indexer to send data. If you configure the receiver to require a client certificate, you must configure the forwarder to present that client certificate when it connects to the indexer.

1. Copy the new certificate and the certificate authority certificate files into an accessible folder on the forwarders you want to configure. For example, you can use a destination folder of $SPLUNK_HOME/etc/auth/mycerts on the forwarder.
2. Using a text editor, open the $SPLUNK_HOME/etc/system/local/outputs.conf file for editing.
3. Use the following settings to define the [tcpout] stanza in the file to configure the forwarder to use the certificate.

   Setting/stanza name	Data type	Description
   [tcpout:<name>]	n/a	Defines an output group to send data to a receiver.
   server	string	The hostname or IP address and port on which to connect securely to forward data.
   clientCert	string	The location of the client certificate on the forwarder. This is the certificate that the forwarder uses to connect to the receiving indexer over TLS. You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myClientCertificate.pem, or you can use a relative path, such as etc/auth/mycerts/myClientCertificate.pem and the instance uses the Splunk platform instance installation directory.
   useClientSSLCompression (Optional)	Boolean	Whether or not the forwarder performs TLS compression when it connects with a receiver. The default value of "true" means that the client uses TLS compression. A value of "false" means the client doesn't use compression. Disabling compression, particularly with TLS, can increase bandwidth usage.
   sslPassword (Optional)	string	Same as the setting in the inputs.conf configuration file
   sslVerifyServerCert (Optional)	Boolean	Whether or not, upon connection to a receiver, the forwarder confirms that the receiver has a valid TLS server certificate. A value of "true" means that the forwarder checks for a valid server certificate upon connection, then checks the common or alternate names against the names in the server certificate against the names in the values for the 'sslCommonNameToCheck' and 'sslAltNameToCheck' settings on the forwarder. If there is no match against the common or alternate names, the forwarder aborts the connection to the receiver as not authorized.
   sslVerifyServerName (Optional)	Boolean	Whether or not, upon connection to a receiver, the forwarder confirms that the valid TLS certificate that the receiver presents contains the host name of the receiver in the common name or subject alternate name field of the certificate. A value of "true" means that the forwarder checks for a host name match in the certificate that the receiver presents. If the host name in the certificate does not match the receiver host name, the forwarder aborts the connection to the receiver as not authorized.
   sslCommonNameToCheck (Optional)	comma-separated list	Same as the setting in the inputs.conf configuration file, except that you must give the 'sslVerifyServerCert' setting a value of "true" in the outputs.conf configuration file and the forwarder does the certificate verification.
   sslAltNameToCheck (Optional)	comma-separated list	Same as the setting in the inputs.conf configuration file, except that you must give the 'sslVerifyServerCert' setting a value of "true" in the outputs.conf configuration file and the forwarder does the certificate verification.
   cipherSuite (Optional)	comma-separated list	Same as the setting in the inputs.conf configuration file.

4. Save the outputs.conf file and close it.
5. On forwarders that do not run on Windows, open the server.conf configuration file for editing.
6. Add the following stanza and settings to the file:
   

   [sslConfig]
   sslRootCAPath = <absolute path to the certificate authority certificate>
   

7. Save the server.conf file and close it.
8. Restart the splunkd process.
   

   $SPLUNK_HOME/bin/splunk restart splunkd
   

Configuration file examples for configuring TLS certificates on forwarders

Following is an example of an outputs.conf configuration file on a forwarder. In this example:

• The forwarder uses a certificate that is located in the /opt/splunk/etc/auth/mycerts directory
• The forwarder certificate has a password of "myCertificatePassword"
• The forwarder uses TLS compression
• The forwarder requires that the receiving indexer present a certificate, and that that certificate contain a common name of indexer1.mycompany.com or indexer2.mycompany.com, or a subject alternate name of indexer3.mycompany.com

[tcpout:group1]
server=10.1.1.197:9997
disabled = 0
clientCert = /opt/splunk/etc/auth/mycerts/myClientCert.pem
useClientSSLCompression = true
sslPassword = myCertificatePassword
sslCommonNameToCheck = indexer1.mycompany.com,indexer2.mycompany.com
sslAltNameToCheck = indexer3.mycompany.com
sslVerifyServerCert = true 

When you make edits to the $SPLUNK_HOME/etc/system/local/outputs.conf configuration file to install certificates, if you supply a password for your server certificate, Splunk Enterprise encrypts that password from cleartext when you restart Splunk Enterprise.

The server.conf configuration file establishes and references the location of the certificate authority certificate. This certificate is the trust anchor to verify the indexer certificate in a TLS connection. You must configure this on the forwarder even though it is the client:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem

Send data over TLS from a forwarder to more than one indexer

If you need to forward data securely to multiple indexers, complete the following procedure:

1. On the forwarder where you want to send data to multiple indexers, use the "Configure forwarders to use a signed certificate" procedure earlier in this topic to open and make changes to the outputs.conf configuration file.
2. In the target output group definition stanza for the forwarder, add a host:port line for each indexer to which you want to send data over TLS. Separate multiple entries with commas.
3. Save the outputs.conf file and close it.
4. Restart the forwarder.

The following example outputs.conf file uses the same certificate for the forwarders as it does the indexers:

[tcpout]

[tcpout:group1]
server = 10.1.12.112:9997,10.1.12.111:9999
# multiple servers: 10.1.12.112:9997, 10.1.12.111:9999
disabled = 0
clientCert = $SPLUNK_HOME/etc/auth/client.pem
useClientSSLCompression = true 
# Defaults to the value set in the useClientSSLCompression 
# setting set in server.conf.
sslPassword = <password for the client certificate>
sslCommonNameToCheck = indexercn.example.org
sslVerifyServerCert = true

Forward data over TLS to multiple indexers using certificates with different common names

If you have created one server certificate for each indexer and you have set a unique sslCommonNameToCheck or sslAltNameToCheck in each indexer certificate to be checked by the forwarders, you must configure one [tcpout-server://host:port] configuration stanza for each indexer in the outputs.conf file. This action lets you specify which name to check for each indexer.

Next steps

Confirm that the forwarder and indexer configurations work properly. See Test and troubleshoot TLS connections.