Configure TLS certificate host name validation for secured connections between Splunk software components

If you have configured your Splunk platform instances to use transport layer security (TLS) certificates for secure network connections with one another, you can also configure the instances to verify host names in the certificates to ensure that the machines that the instances communicate with are who they say they are. This added configuration step improves security across your entire Splunk Cloud Platform forwarding tier and Splunk Enterprise deployment. The procedures in this topic are valid for both Splunk Cloud Platform forwarding tier and Splunk Enterprise instances.

For more information about the security updates, how they operate, and the modes that affect how they work, see Security updates.

Introduction

TLS certificate host name validation consists of three separate configurations:

TLS certificate requirement configuration

The server.conf configuration file controls the TLS certificate requirement. You can only configure TLS certificate requirements by editing the configuration file and specifying the appropriate setting and value. This means you can only perform this configuration on Splunk Enterprise, or on collection and forwarding infrastructure for Splunk Cloud Platform that you manage. You must enable the TLS certificate requirement for the certificate host name validation to work.

The sslVerifyServerCert setting controls the TLS certificate requirement feature. When you give this setting a value of "true", the Splunk platform instance requires that any Splunk platform instance to which it connects provides a valid TLS certificate before that connection can complete. If the connected instance does not provide a valid certificate, the requirement check fails and the connection terminates.

If the requirement check succeeds, and you have also configured TLS certificate host name validation, then the validation check happens, as explained in the next section of this topic.

TLS certificate host name validation configuration

The server.conf configuration file also controls TLS certificate host name validation. Like with the certificate requirement, you can only configure validation by editing the configuration file and specifying the appropriate setting and value. You can only perform this configuration on Splunk Enterprise, or on collection and forwarding infrastructure for Splunk Cloud Platform that you manage.

In nearly all cases, the sslVerifyServerName setting controls the TLS certificate host name validation feature. When you give this setting a value of "true", the connecting Splunk platform instance verifies that the TLS certificate that it received is valid and has either a Common Name (CN) or Subject Alternative Name (SAN) X.509 cryptography standard certificate field that matches the host name of the instance that sent the certificate. This verification must happen before the connection can complete. If the connected server provides a certificate that does not meet these criteria, the validation check fails and the connection terminates.

Before validation can start, the connected instance must first pass the TLS certificate requirement check.

TLS certificate host name validation configuration for the Splunk CLI

The Splunk CLI has a slightly different setting name for TLS certificate host name validation. When you run the CLI to connect to either a local or remote Splunk platform instance, the CLI uses the cliVerifyServerName setting to determine whether or not it is to verify host names on the TLS certificates it receives.

A value of "true" for this setting, like the sslVerifyServerName setting, means that the CLI performs TLS hostname validation. The validation works the same as validation does for other Splunk platform instances.

After you turn on TLS hostname validation for the CLI, you can temporarily disable it by using the --no-host-name-check CLI argument.

Prerequisites to enabling TLS certificate host name validation

You must have the following items before you can enable TLS certificate host name validation:

• All Splunk platform instances where you want to enable TLS certificate host name validation must run version 9.0.0 or higher.
• You must have already secured your Splunk platform instances with valid, current certificates that you either created or obtained from a third party. The certificates cannot be the ones that Splunk ships with Splunk platform installation packages. See "What is a valid certificate?" later in this topic for specifics on what a valid certificate is.
• You must have already installed the certificates on all Splunk platform instances in your deployment. The configuration for each instance must already reference the correct certificates.
• You must have already enabled TLS certificate requirements for each instance in your deployment. TLS certificate host name validation doesn't work if your instances do not require TLS certificates in the first place.

What is a valid certificate?

A valid certificate is one that satisfies all of the following criteria:

• It must not be one of the default certificates that come with the Splunk platform installation packages.
• It must be in privacy enhanced mail (PEM) format. Validation doesn't work with certificates that are in other formats.
• It must be a full certificate chain. Validation doesn't work with only a leaf certificate.
• It must contain any intermediate certificates, along with the root and server certificate, where applicable.
• It must be valid within its date range. Expired certificates and certificates whose validity has not yet come into force do not work.
• It must contain a valid Common Name (CN) or Subject Alternative Name (SAN) X.509 certificate standard field.
  • Either of those fields must contain a value that matches the host name of the machine that serves the certificate to the connecting client.

Configure TLS certificate host name validation

Before you attempt to configure TLS certificate host name validation, confirm you have met all the requirements. Then, choose the procedure from the following list for the service or instance type you want to secure with certificate host name validation.

You can only configure certificate host name validation using configuration files. It is not possible to configure this using Splunk Web.

Configure TLS host name validation for Splunk-to-Splunk communication

Follow this procedure to secure Splunk-to-Splunk communication between instances like indexers, search heads, clusters, and deployment and license servers.

1. Confirm that you have installed the certificates on all your Splunk platform instances.
2. On one of the instances, edit the $SPLUNK_HOME/etc/system/local/server.conf configuration file.
3. In the server.conf file, add the following settings and values to enable TLS certificate validation:
   

   [sslConfig]
   # turns on TLS certificate requirements
   sslVerifyServerCert = true
   # turns on TLS certificate host name validation
   sslVerifyServerName = true
   serverCert = <path to your server certificate>
   

4. Copy the configuration to the remainder of your Splunk platform instances.
   

   You might want to use a deployment server to deliver configurations to other Splunk platform instances.

5. Restart the Splunk platform instances.
6. Test the instance to confirm that it uses the certificate and that TLS host name validation works as you expect.

Configure TLS host name validation for Splunk Python modules

Follow this procedure if you want to enable TLS certificate host name validation for Python version 3 modules. TLS host name validation does not work with Python version 2 and lower modules.

1. Confirm that you have installed the certificates on all your Splunk platform instances.
2. On one of the instances, edit the $SPLUNK_HOME/etc/system/local/server.conf configuration file.
3. In the server.conf file, add the following settings and values to enable TLS certificate validation:
   

   [pythonSslClientConfig]
   sslVerifyServerCert = true
   sslVerifyServerName = true
   

4. Edit the $SPLUNK_HOME/etc/system/local/web.conf configuration file.
5. In the web.conf file, add the following settings and values to specify the server certificate location:
   

   [settings]
   serverCert = <path to your server certificate>
   

6. Edit the $SPLUNK_HOME/etc/splunk-launch.conf configuration file.
7. In the splunk-launch.conf file, add the following settings and values to enable Python module verification:
   

   PYTHONHTTPSVERIFY = 1
   

8. Distribute the configuration files to the remainder of your Splunk platform instances.
9. Restart the Splunk platform instances.
10. Test the instance to confirm that it uses the certificate and that TLS host name validation works as you expect. You can review the splunkd.log and python.log log files and, if necessary, change the default logging level temporarily.

Configure TLS host name validation for the App Key Value Store service

Follow this procedure if you want to enable TLS certificate host name validation for App Key Value Store. TLS host name validation only works for search head clusters that use App Key Value Store.

1. Confirm that you have installed the certificates on all your Splunk platform instances.
   

   For App Key Value Store, certificates must contain an Organization (O), Organizational Unit (OU), or Domain Component (DC).

2. On one of the instances, edit the $SPLUNK_HOME/etc/system/local/server.conf configuration file.
3. In the server.conf file, add the following settings and values to enable TLS certificate validation:
   

   [kvstore]
   sslVerifyServerCert = true
   sslVerifyServerName = true
   serverCert = <path to your server certificate>
   

4. Distribute the configuration file to the remainder of your Splunk platform instances.
5. Restart the Splunk platform instances.
6. Test the instance to confirm that it uses the certificate and that TLS host name validation works as you expect. You can review the splunkd.log and mongod.log log files and, if necessary, change the default logging level temporarily.

Configure TLS host name validation for the Splunk CLI

Follow this procedure if you want to enable TLS certificate host name validation for the Splunk CLI. You must perform the procedure on any instance where you use the CLI to connect to a Splunk platform instance.

1. Confirm that you have installed the certificates on all your Splunk platform instances.
2. On an instance where you plan to use the CLI, edit the $SPLUNK_HOME/etc/system/local/server.conf configuration file.
3. In the server.conf file, add the following settings and values to enable TLS certificate validation:
   

   [sslConfig]
   # turns on TLS certificate host name validation
   cliVerifyServerName = true
   # Reference the file that contains all root certificate authority certificates combined together
   sslRootCAPath = <path to you server certificate>
   

4. Save the configuration file.
5. Distribute the configuration file to all Splunk platform instances where you want to run the CLI.

You do not need to restart the Splunk platform instances. The Splunk CLI picks up the changed configuration when you run it.

6. Test the instance to confirm that it uses the certificate and that TLS host name validation works as you expect. You can review the output of the CLI command.
   

   To temporarily disable TLS certificate host name validation from the CLI, supply --no-server-name-check as an argument to the CLI command.

Configure universal forwarder management security

Follow this procedure to improve security on universal forwarders by changing the accessibility of the management port.

1. Where applicable, confirm that you have installed TLS certificates on all your universal forwarders.
2. On one of the forwarders, edit the $SPLUNK_HOME/etc/system/local/server.conf configuration file.
3. In the server.conf file, add the following settings and values:
   

   [httpServer]
   disableDefaultPort = true
   

4. Edit the $SPLUNK_HOME/etc/system/local/web.conf configuration file.
5. In the web.conf file, add the following settings and values to force the universal forwarder to accept requests on the management port locally:
   

   [settings]
   mgmtHostPort = localhost:8089 
   # if 'mgmtport' is not already set to "localhost" or is blank
   

6. Edit the $SPLUNK_HOME/etc/splunk-launch.conf configuration file.
7. In the splunk-launch.conf file, add the following settings and values to force the universal forwarder to accept inbound network requests from 127.0.0.1::
   

   SPLUNK_BINDIP = 127.0.0.1 
   # if 'SPLUNK_BINDIP' is not already set to "localhost"
   

8. Save the configuration files.
9. Distribute the configuration files to the remainder of your Splunk platform instances.
10. Restart the Splunk platform instances.
11. Test the forwarder to confirm that it does not allow connections to the management port from other machines.