Security updates

When Splunk released Splunk Enterprise version 9.0, it included several significant security functionality improvements. Splunk Enterprise versions higher than 9.0 also includes the improvements that came with version 9.0. If you upgrade to a version that is higher than 9.0 from a version that is lower than 9.0, these improvements are now available for you.

Read this topic to learn about the security functionality improvements in the Splunk Enterprise and Splunk Cloud platforms and how you can further secure your deployments with these improvements.

See the "Last modified" information at the end of this topic to determine if the topic has been updated since you last reviewed it.

Introduction

Splunk Enterprise and Splunk Cloud Platform now offer additional options that let you and Splunk operate deployments more securely.

The improvements center around the following themes:

• The addition of Transport Layer Security (TLS) certificate host name validation and other improvements when Splunk platform instances make secure connections
  • In particular, the Splunk daemon (splunkd), all Python modules that the Splunk platform uses, the Splunk CLI, and App Key Value Store now support TLS certificate validation
• Improvements to universal forwarder security, including the limiting of access to the UF management port to only the local machine
• Upgrades to Splunk deployment client and server authentication logic, including restrictions on how and with what those instance types can communicate

The table in the "Summary of changes" section of this topic provides a summary of the changes, and the following sections provide additional details.

All of these changes come in one of two operational modes. To understand the modes and how they affect your Splunk platform deployment, see Understand warning mode versus enforcement mode for security updates later in this topic.

Summary of changes

The following table lists a summary of the changes, the Splunk platforms on which the changes ship, the enforcement mode in which they currently operate, and links to procedures on how to configure Splunk software to enforce the changes.

Change	Description	Introduced in	Addresses	Current mode	Learn more
TLS certificate host name validation	Splunk platform instances verify the hostname in the TLS certificate they receive when they connect to other Splunk platform instances.	Splunk Cloud Platform (SCP) 8.2.2202, Splunk Enterprise (SE) 9.0	SVD-2022-0602, SVD-2022-0603	Warning	Learn more
Python module TLS connection hardening	Python modules on Splunk platform instances always validate TLS connections.	SCP 8.2.2202, SE 9.0	SVD-2022-0601	Warning	Learn more
TLS certificate host name validation using Splunk CLI	The Splunk Command Line Interface (CLI) validates TLS certificates for any connections it makes to other Splunk platform instances.	SCP 8.2.2203, SE 9.0	SVD-2022-0606	Warning	Learn more
Universal forwarder security	Universal forwarders always validate connections from other Splunk platform instances.	SE 9.0	SVD-2022-0605	Enforcement	Learn more
	Universal forwarders now support the least-privileged user model for operations on Linux machines.	SE 9.0	--	Warning	Learn more
Deployment client and server authentication logic	Communication protocols between deployment clients and servers, including handshake, subscription, and heartbeat logic, change.	SE 9.0	--	Warning	Learn more
	Deployment clients require authentication to download or execute forwarder bundles.	SE 9.0	SVD-2022-0607	Warning	Learn more
	Deployment clients require authentication to download or execute forwarder bundles.	SE 9.0	SVD-2022-0608	Enforcement	Learn more
Risky search command safeguard improvements	You must assign additional capabilities to run some risky and custom search commands.	SE 9.0	SVD-2022-0604	Enforcement	Learn more

Details of security updates

A detailed listing of the changes follows. Follow the links in the "Summary of changes" table for details on how to activate the improvements in your environment.

Transport Layer Security

Splunk platform components use Transport Layer Security (TLS) to connect securely to one another and internal and external APIs. Each connection uses a TLS certificate to establish the secure connection. The certificates verify that the Splunk platform instances that make the connection are who they say they are. TLS certificates can be configured and validated for nearly all Splunk platform instance types, including indexers, indexer clusters, search heads, search head cluster nodes, deployers, forwarders, deployment servers, license servers, and App Key Value Store.

Most of these improvements center around how the Splunk platform handles TLS certificates. Prior to these updates, Splunk platform instances did not specifically verify the host name information within the TLS certificate that they received when they connected to other instances or APIs. This additional verification step must happen to ensure that the machine to which the instances connected is who it claims to be, and prevents what is known as a "machine-in-the-middle" attack. This is a cyberattack where a malicious actor can position a machine between the two Splunk platform instances on the network and intercept messages between them without either knowing that they aren't actually communicating with each other.

Before the certificate verification can work, Splunk platform instances must be configured to require that the instances that they connect to present a valid certificate prior to the connection completing. This certificate requirement option is different from the host name verification option, and has been available for some time. Many customers already require their Splunk platform instances to present certificates to ensure increased security in their Splunk platform deployments.

Beginning with the 8.2.2202 release of Splunk Cloud Platform and version 9.0 of Splunk Enterprise, you can configure Splunk platform instances to perform host name validation upon connecting to other Splunk platform instances and receiving TLS certificates.

Many of the security changes are disabled initially to prevent problems with downtime in your environment. This is the case with TLS certificate validation. On Splunk Enterprise deployments, and on data collection and forwarding tiers for Splunk Cloud Platform deployments that you manage, as the Splunk administrator, you can enable the feature after you confirm that the TLS certificates in your deployment have the proper configuration. In later versions of both Splunk Enterprise and Splunk Cloud Platform, Splunk will enable the enforcement of certificate validation.

TLS certificate host name validation does not work with the default certificates that come with Splunk Enterprise. You must either generate or obtain certificates from a third party and install them in all of the instances of your Splunk platform deployment. You must then configure the instances to require that they get a valid TLS certificate from other instances upon connection, and that those certificates pass the host name validation check.

Follow the documentation links from this page to the appropriate procedures to perform enablement of TLS certificate validation. You can work with Splunk Support of Professional Services for the best course of action depending on the size of your deployment.

Deployment client and server authentication logic

The authentication, subscription, and heartbeat logic for deployment clients and servers has changed. No communication between instances happens without TLS certificate validation. The way that deployment clients subscribe to deployment server channels has also changed, and deployment clients ignore certain types of messages that do not specifically come from deployment servers.

Deployment client and server protocol updates begin in warning mode. Splunk does not expect this change to affect your deployment negatively.

Universal forwarder security

Universal forwarders now have improved security. Beginning with Splunk Enterprise version 9.0, forwarders now require a configuration change to accept inbound connections to their management port from other machines. Additionally, when universal forwarders act as deployment clients, they receive the same deployment-client-to-deployment-server authentication, subscription, and heartbeat logic changes that other deployment clients receive.

Universal forwarder security updates, with the exception of changes to deployment server logic, begin in enforcement mode. Splunk does not expect this change to affect your deployment negatively.

Risky search command safeguard improvements

Splunk has added some capabilities that you must assign to users who need to run the dump search command and any custom search commands. Dashboards that currently use these commands might break if the user that created the dashboards does not hold a role with these capabilities.

OpenSSL security toolkit

The version of OpenSSL that Splunk platform instances use to secure connections between other Splunk platform instances is now 1.0.2zd. This version of OpenSSL fixes several vulnerabilities in TLS, including how TLS deals with both server and client certificates. It improves the overall security of your Splunk platform deployment.

Splunk does not expect this change to affect your deployment negatively.

What is a valid TLS certificate?

A valid certificate is one that satisfies all of the following criteria:

• It must not be one of the default certificates that come with the Splunk platform installation packages. Validation doesn't work with the default certificates.
• It must be a regular text file that is in privacy enhanced mail (PEM) format. Validation doesn't work with certificates that are in other formats.
• It must be a full certificate chain. Validation doesn't work with only a leaf certificate.
• It must contain any intermediate certificates, along with the root and server certificate, where applicable.
• It must be valid within its date range. Expired certificates and certificates whose validity has not yet come into force don't work.
• It must use a valid Common Name (CN) or Subject Alternate Name (SAN) X.509 certificate standard field.
  • Either of those fields must contain a value that matches the host name of the machine that serves the certificate to the connecting client.

If the certificate doesn't meet all of these criteria, then validation, and the connection, fails. Because individual Splunk platform instances must make many network connections to share data, search jobs, artifacts, knowledge bundles, and other Splunk-related information, failed connections can cause problems with Splunk deployments.

Understand warning mode versus enforcement mode for security updates

The security updates that this topic discusses come in two operation modes. These modes control how the Splunk platform reacts to items in your deployment that do not meet updated Splunk security best practices.

Warning mode

In Warning mode, Splunk platform instances log problems with configurations that do not align with updated security best practices. Enforcement of the updates does not occur in this mode. Your Splunk Cloud Platform or Splunk Enterprise deployment operates as it currently does, only generating additional logs about the out-of-specification configurations. At any time, you can review the logs to see where your deployment is out of compliance and subsequently reconfigure the instances to bring them into compliance and enable enforcement of the updates.

If you activate and use the Splunk Assist monitoring service, it can identify the instances in your deployment that are out of compliance and provide tools to help you get them into compliance. See Introduction to Splunk Assist for additional information about the service and the details it provides.

Enforcement mode

In Enforcement mode, Splunk platform instances enforce the new security changes. If your configuration does not meet the updated best practices for security, then connections between Splunk platform instances, processes, and APIs can fail, which can cause the deployment to experience breaking problems and downtime.

When Splunk ships versions of the Splunk platform that enable enforcement mode, problems can begin immediately after an upgrade if you have not enabled certificate requirements and TLS host name validation. Ensure that your Splunk platform instances have valid certificates, and that TLS certificate requirements and certificate host name validation are on. You cannot use the default certificates that come with Splunk platform instance installation packages - you must either generate your own certificates or obtain them from a third party.

When and how each mode affects your Splunk deployment

Some of the security updates start in enforcement mode by default. Splunk does not expect security updates that initially start in enforcement mode to negatively affect Splunk Cloud Platform or Splunk Enterprise deployments. You can monitor your deployment to see whether enforcement mode affects your deployment. The "Summary of changes" table that appears earlier in this topic lists the current operating mode for each security update.

All other changes initially begin in warning mode. Changes that begin in warning mode give you time to configure your Splunk Cloud Platform or Splunk Enterprise deployment to begin enforcing the security updates. To do this, see "Configure your Splunk platform deployment to enforce security updates" later in this topic. Eventually, Splunk will update these changes to operate in enforcement mode by default.

Steps to address the security update warnings and errors

When you update Splunk Enterprise or Splunk updates Splunk Cloud Platform, security update warnings and errors appear in the splunkd.log file, and for App Key Value Store, the mongodb.log log file. You can review either of the files themselves for the warnings, or you can search the _internal index from Splunk Web.

The warnings and errors that you see depend on the operating mode each security update is currently in, the action that the Splunk platform instance is trying to perform, the Splunk service or command that performs the action, and the current state of your configuration. You might see errors that are similar to, but not exactly like, the ones that appear in this table.

Current state	What the logs say	What you need to do
You have the proper certificates in place and you have enabled both the requirement for TLS certificates and TLS certificate host name validation.	Nothing	Nothing
You have the proper certificates in place and have enabled the requirement for TLS certificates, but haven't yet turned on TLS certificate host name validation.	Nothing	Enable TLS certificate host name validation.
You have the proper certificates in place, but haven't yet enabled TLS certificate requirements or certificate host name validation.	An error message similar to the following, when the instance starts up, in the SSLOptions channel of the splunkd.log file: sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security	Enable TLS certificate requirements and certificate host name validation.
You use the default certificates that come with the Splunk platform installation packages, and haven't yet turned on TLS certificate requirements or certificate host name validation.	An error message similar to the following, when the instance starts up, in the SSLOptions channel of the splunkd.log file: sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security	Obtain and install valid certificates. Then enable TLS certificate requirements and certificate host name validation.
You use the default certificates that come with the Splunk platform installation packages, or haven't yet turned on TLS certificate requirements or certificate host name validation, and try to connect to another Splunk instance using the CLI.	An error message similar to the following: WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Login failed	Obtain and install valid certificates. Then enable TLS certificate requirements and certificate host name validation. Then, try the CLI command again.
You use the default certificates that come with the Splunk platform installation packages, haven't yet turned on TLS certificate requirements or certificate host name validation, and later upgrade to a release that enables certificate validation by default	For connections between Splunk platform instances, TLS connection errors similar to the following: 2022-03-04T20:41:00.231Z E NETWORK [NetworkInterfaceASIO-Replication-0] SSL peer certificate validation failed: self signed certificate in certificate chain 2022-03-04T20:41:00.231Z W ASIO [NetworkInterfaceASIO-Replication-0] Failed to validate peer certificate during SSL handshake: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain 2022-03-04T20:41:00.231Z I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 10.202.5.121:8191 - SSLHandshakeFailed: SSLHandshakeFailed 2022-03-04T20:41:00.231Z I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 10.202.5.121:8191 due to failed operation on a connection For connections from the CLI, errors similar to the following: ERROR: certificate validation: self signed certificate in certificate chain ERROR: certificate validation: self signed certificate in certificate chain Couldn't complete HTTP request: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed Downtime risk. Splunk platform instances won't be able to connect with each other.	Obtain and install valid certificates. Then enable TLS certificate requirements and certificate host name validation. Then, if using the CLI, try the CLI command again.

Configure your Splunk platform deployment to enforce security updates

When you upgrade to Splunk Enterprise version 9.0, or Splunk upgrades your Splunk Cloud Platform instance to version 8.2.2202 and higher, the changes that this topic describes are available for you to enable. Splunk has already enabled some of the changes.

To configure the Splunk platform deployment to use enforcement mode for all changes, do the following:

1. Ensure you have valid certificates. If you don't, get or create them. See the following topics for instructions:
   • To create and sign your own certificates, see How to self-sign certificates.
   • To get certificates from a third party, see How to obtain certificates signed by a third-party for inter-Splunk communication.
2. Install the certificates on all instances in your Splunk platform deployment, if you have not already.
3. Follow the "Learn More" links in the "Summary of changes" table to procedures that describe how to enable certificate validation for each update.

For further information

See the following topics for more information:

• For general information on using TLS to secure your Splunk platform deployments, see About securing the Splunk platform with TLS.
• For procedures on how to enable TLS certificate host name validation, see Enable TLS certificate host name validation.