Security updates

Read this topic to learn about the security improvements to Splunk Cloud Platform and how you can further secure your deployments with these updates.

See the "Last modified" information at the end of this topic to determine if the topic has been updated since you last reviewed it.

Introduction

Splunk Cloud Platform now offers additional options that let you and Splunk operate deployments more securely.

The updates center around the following themes:

• The addition of Transport Layer Security (TLS) certificate validation and other improvements when Splunk platform instances make secure connections
  • In particular, the Splunk daemon (splunkd), all Python modules that the Splunk platform uses,, the Splunk CLI, and App Key Value Store now support TLS certificate validation
• Upgrades to the OpenSSL software toolkit that facilitates secure connections between Splunk platform instances
• Improvements to universal forwarder security, including the limiting of access to the UF management port to only the local machine
• Upgrades to Splunk deployment client and server authentication logic, including restrictions on how and with what those instances can communicate

The table in the "Summary of changes" section of this topic provides a summary of the changes, and the following sections provide additional details.

All of these changes come in one of two operational modes. To understand the modes and how they affect your Splunk platform deployment, see Understand warning mode versus enforcement mode for security updates later in this topic.

Summary of changes

The following table lists a summary of the changes, the Splunk platforms on which the changes ship, the enforcement mode in which they currently operate, and links to procedures on how to configure Splunk software to enforce the changes.

Change	Description	Introduced in	Current mode	Learn more
TLS hostname validation	Splunk platform instances verify the hostname in the TLS certificate they receive when they connect to other Splunk platform instances.	Splunk Cloud Platform (SCP) 8.2.2202	Warning	Learn more
Python module TLS connection hardening	Python modules on Splunk platform instances always validate TLS connections.	SCP 8.2.2202	Warning	Learn more
OpenSSL version update	Splunk upgrades the version of OpenSSL to version 1.0.2zd.	SCP 8.2.2202	N/A	N/A

Details of security updates

A detailed listing of the changes follows. Follow the links in the "Summary of changes" table for details on how to activate the improvements in your environment.

Transport Layer Security

Splunk platform components use Transport Layer Security (TLS) to connect securely to one another and internal and external APIs. Each connection uses a TLS certificate to establish the secure connection. The certificates verify that the Splunk platform instances that make the connection are who they say they are. TLS certificates can be configured and validated for nearly all Splunk platform instance types, including indexers, indexer clusters, search heads, search head cluster nodes, deployers, forwarders, deployment servers, license servers, and App Key Value Store.

Most of these improvements center around how the Splunk platform handles TLS certificates. Prior to these updates, Splunk platform instances did not specifically verify the host name information within the TLS certificate that they received when they connected to other instances or APIs. This additional verification step must happen to ensure that the machine to which the instances connected is who it claims to be, and prevents what is known as a "machine-in-the-middle" attack. This is a cyberattack where a malicious actor can position a machine between the two Splunk platform instances on the network and intercept messages between them without either knowing that they aren't actually communicating with each other.

Before the certificate verification can work, Splunk platform instances must be configured to require that the instances that they connect to present a valid certificate prior to the connection completing. This certificate requirement option is different from the host name verification option, and has been available for some time. Many customers already require their Splunk platform instances to present certificates to ensure increased security in their Splunk platform deployments.

Beginning with the 8.2.2202 release of Splunk Cloud Platform, you can configure Splunk platform instances that you manage to perform host name validation upon connecting to other Splunk platform instances and receiving TLS certificates.

Information on valid certificates
A valid certificate is one that satisfies all of the following criteria:

• It must not be one of the default certificates that come with the Splunk platform installation packages. Validation doesn't work with the default certificates.
• It must be in privacy enhanced mail (PEM) format. Validation doesn't work with certificates in other formats.
• It must be a full certificate chain. Validation doesn't work with only a leaf certificate.
• It must contain any intermediate certificates, along with the root and server certificate, where applicable.
• It must be valid within their date range. Expired certificates and certificates whose validity has not yet come into force don't work.
• It must use a valid Common Name (CN) or Subject Alternate Name (SAN) X.509 certificate field.
• Either of those fields must match the host name that serves the certificate to the connecting client.

If the certificate doesn't meet all of these criteria, then validation, and the connection, fails. Because individual Splunk platform instances must make many network connections to share data, search jobs, artifacts, knowledge bundles, and other Splunk-related information, failed connections can cause problems with Splunk deployments.

Many of the security changes are disabled initially to prevent problems with downtime in your environment. This is the case with TLS certificate validation. On Splunk Enterprise deployments, and on data collection and forwarding tiers for Splunk Cloud Platform deployments that you manage, as the Splunk administrator, you can enable the feature after you confirm that the TLS certificates in your deployment have the proper configuration. In later versions of both Splunk Enterprise and Splunk Cloud Platform, Splunk will enable the enforcement of certificate validation.

TLS certificate validation does not work with the default certificates that come with the Splunk platform installation package. You must either generate or obtain certificates from a third party and install them in all of the instances of your Splunk platform deployment. You must then configure the instances to require that they get a valid TLS certificate from other instances upon connection, and that those certificates pass the host name validation check.

Follow the documentation links from this page to the appropriate procedures to perform enablement of TLS certificate validation. You can work with Splunk Support of Professional Services for the best course of action depending on the size of your deployment.

OpenSSL security toolkit

The version of OpenSSL that Splunk platform instances use to secure connections between other Splunk platform instances is now 1.0.2zd. This version of OpenSSL fixes several vulnerabilities in TLS, including how TLS deals with both server and client certificates. It improves the overall security of your Splunk platform deployment.

Splunk does not expect this change to affect your deployment negatively.

Understand warning mode versus enforcement mode for security updates

The security updates that this topic discusses come in two operation modes. These modes control how the Splunk platform reacts to items in your deployment that do not meet updated Splunk security best practices.

Warning mode

In Warning mode, Splunk platform instances log problems with configurations that do not align with updated security best practices. Enforcement of the updates does not occur in this mode. Your Splunk Cloud Platform or Splunk Enterprise deployment operates as it currently does, only generating additional logs about the out-of-specification configurations. At any time, you can review the logs to see where your deployment is out of compliance and subsequently reconfigure the instances to bring them into compliance and enable enforcement of the updates.

Enforcement mode

In Enforcement mode, Splunk platform instances enforce the new security changes. If your configuration does not meet the updated best practices for security, then connections between Splunk platform instances, processes, and APIs can fail, which can cause the deployment to experience breaking problems and downtime.

When Splunk ships versions of the Splunk platform that enable enforcement mode, problems can begin immediately after an upgrade if you have not enabled certificate requirements and TLS host name validation. Ensure that your Splunk platform instances have valid certificates, and that TLS certificate requirements and certificate host name validation are on. You cannot use the default certificates that come with Splunk platform instance installation packages - you must either generate or obtain certificates from a third party.

When and how each mode affects your Splunk platform deployment

Some of the security updates start in enforcement mode by default. Splunk does not expect security updates that initially start in enforcement mode to negatively affect Splunk Cloud Platform or Splunk Enterprise deployments. You can monitor your deployment to see whether enforcement mode affects your deployment. The "Summary of changes" table that appears earlier in this topic lists the current operating mode for each security update.

All other changes initially begin in warning mode. Changes that begin in warning mode give you time to configure your Splunk Cloud Platform or Splunk Enterprise deployment to begin enforcing the security updates. To do this, see Configure your Splunk platform deployment to enforce security updates later in this topic. Eventually, Splunk will update these changes to operate in enforcement mode by default.

Steps to address the security update warnings and errors

When Splunk updates Splunk Cloud Platform, security update warnings and errors appear in the splunkd.log file, and for App Key Value Store, the mongodb.log log file. You can review either of the files themselves for the warnings, or you can search the _internal index from Splunk Web.

The warnings and errors that you see depend on the mode each security update is currently in, the action that the Splunk platform instance is trying to perform, the Splunk service or command that performs the action, and the current state of your configuration. You might see errors that are similar to, but not exactly like, the ones shown in this table.

Current state	What you see	What you need to do
You have the proper certificates in place and you have enabled both the requirement for TLS certificates and TLS certificate host name validation.	Nothing	Nothing
You have the proper certificates in place and have enabled the requirement for TLS certificates, but haven't yet turned on TLS certificate host name validation.	Nothing	Enable TLS certificate host name validation.
You have the proper certificates in place, but haven't yet enabled TLS certificate requirements or certificate host name validation.	An error message similar to the following, when the instance starts up, in the SSLOptions channel of the splunkd.log file: sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security	Enable TLS certificate requirements and certificate host name validation.
You use the default certificates that come with the Splunk platform installation packages, and haven't yet turned on TLS certificate requirements or certificate host name validation.	An error message similar to the following, when the instance starts up, in the SSLOptions channel of the splunkd.log file: sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security	Obtain and install certificates. Then enable TLS certificate requirements and certificate host name validation.
You use the default certificates that come with the Splunk platform installation packages, or haven't yet turned on TLS certificate requirements or certificate host name validation, and try to connect to another Splunk instance using the CLI.	An error message similar to the following: WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Login failed	Obtain and install certificates. Then enable TLS certificate requirements and certificate host name validation. Then, try the CLI command again.
You use the default certificates that come with the Splunk platform installation packages, haven't yet turned on TLS certificate requirements or certificate host name validation, and later upgrade to a release that enables certificate validation by default	For connections between Splunk platform instances, TLS connection errors similar to the following: 2022-03-04T20:41:00.231Z E NETWORK [NetworkInterfaceASIO-Replication-0] SSL peer certificate validation failed: self signed certificate in certificate chain 2022-03-04T20:41:00.231Z W ASIO [NetworkInterfaceASIO-Replication-0] Failed to validate peer certificate during SSL handshake: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain 2022-03-04T20:41:00.231Z I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 10.202.5.121:8191 - SSLHandshakeFailed: SSLHandshakeFailed 2022-03-04T20:41:00.231Z I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 10.202.5.121:8191 due to failed operation on a connection For connections from the CLI, errors similar to the following: ERROR: certificate validation: self signed certificate in certificate chain ERROR: certificate validation: self signed certificate in certificate chain Couldn't complete HTTP request: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed Downtime risk. Splunk platform instances won't be able to connect with each other.	Obtain and install certificates. Then enable TLS certificate requirements and certificate host name validation. Then, if using the CLI, try the CLI command again.

Configure your Splunk platform deployment to enforce security updates

When Splunk upgrades your Splunk Cloud Platform instance to version 8.2.2202 and higher, the changes that this topic describes are available for you to enable. Splunk has already enabled some of the changes.

To configure the Splunk platform deployment to use enforcement mode for all changes, do the following:

1. Ensure you have valid certificates. If you don't, get or create them.
2. Install the certificates on all instances in the Splunk platform deployment that you manage, if you have not already.
3. Follow the "Learn More" links in the "Summary of changes" table to procedures that describe how to enable certificate validation for each update.

For further information

See the following topics for more information:

• For general information on using TLS to secure your Splunk platform deployments, see About securing the Splunk platform with TLS.
• For procedures on how to enable TLS certificate host name validation, see Enable TLS certificate host name validation.