Configure secure communications between Splunk instances with updated cipher suite and message authentication code

Version 7.2 of Splunk Enterprise, Splunk Light, and the universal forwarder introduces a new cipher suite and message authentication code (MAC), that it uses for data encryption and secure communications between Splunk software instances. The cipher suite and MAC replace the current cipher that Splunk software has used for these types of communications.

The new cipher suite uses a stronger, more secure mechanism for encrypting and decrypting file-based data. It uses a separate MAC to handle node authentication during communications between Splunk instances. Previously, the legacy cipher handled both types of operations.

By default, version 7.2 and higher of Splunk software uses the new cipher suite and MAC for these types of communication. They have also been configured to use the existing legacy cipher for backward compatibility in both data encryption and node authentication with versions lower than 7.2.

This backward compatibility lets you plan upgrades for your Splunk and universal forwarder instances while maintaining secure communications between the instances. Later, after you have upgraded all instances in your deployment to version 7.2 or higher, you can disable the legacy cipher and your Splunk instances continue internal communications using only the new cipher suite and MAC.

The main points for understanding how to use the ciphers are:

• Versions of on-premises Splunk software lower than 7.2 do not have the new cipher suite and MAC, and cannot have either integrated within them.
• You must run version 7.2 or higher to take advantage of the stronger encryption capabilities that the new cipher suite and MAC offer.
• Splunk-to-Splunk communication happens using either the legacy cipher or the new cipher suite/MAC, not both
• A Splunk instance that uses only the legacy cipher cannot communicate at all with a Splunk instance that uses only the new cipher suite and MAC.

Configure cipher usage and decryption with configuration files

By default, versions of Splunk software 7.2 and higher use both the legacy cipher and new cipher suite and MAC for communications between instances, depending on the version of the instance they communicate with:

• For backward compatibility, when 7.2 and higher instances of Splunk software communicate with instances of Splunk software lower than 7.2, they use only the legacy cipher.
• When 7.2 and higher instances communicate with other 7.2 and higher instances, they use only the new cipher suite and MAC.

You must configure the cipher settings with the server.conf configuration file on instances of Splunk that are version 7.2 and higher only. You cannot configure cipher settings on versions lower than 7.2. You also cannot configure the ciphers in Splunk Web.

After you make configuration changes, you must restart Splunk instances for the changes to take effect.

Configure which ciphers the Splunk instance uses

1. In the $SPLUNK_HOME/etc/system/local directory, create server.conf if it does not already exist. Do not create this file in $SPLUNK_HOME/etc/default as this file gets overwritten whenever you upgrade.
2. Use a text editor to open the file.
3. Create a [node_auth] stanza and add the following block of text to it depending on your specific needs:
   

   One or more instances runs lower than 7.2	All instances run 7.2 or higher
   [node_auth] signatureVersion = v1,v2	[node_auth] signatureVersion = v2

4. Save the file and close it.
5. Restart the Splunk instance. The instance initiates Splunk-to-Splunk communications using the ciphers that you specified.

Configure legacy cipher decryption options

1. In the $SPLUNK_HOME/etc/system/local directory, create server.conf if it does not already exist. Do not create this file in $SPLUNK_HOME/etc/default/ as this file gets overwritten whenever you upgrade.
2. Use a text editor to open the file.
3. Under the [general] stanza, add the legacyCiphers setting, based on the versions of Splunk software that run in your Splunk deployment:

   One or more instances runs lower than 7.2	All instances run 7.2 or higher
   legacyCiphers = decryptOnly	legacyCiphers = disabled

4. Save the file and close it.
5. Restart the Splunk instance. The instance decrypts configurations in accordance with the legacyCiphers setting.

Update the splunk.secret key file on instances to use the new cipher

The splunk.secret file controls the encryption of various elements of a Splunk deployment, including SSL and LDAP passwords for configuration files, to ensure secure communications between Splunk instances. In Splunk Enterprise version 7.2.2 and higher, you can rotate the splunk.secret file so that it triggers the re-encryption of file-based passwords with the new cipher that was introduced in Splunk Enterprise version 7.2.

The splunk.secret file can be recreated in one of the following ways:

• By using a CLI command.
• By making an HTTP POST request to a REpresentational State Transfer (REST) endpoint.

The procedure differs slightly based on the type of Splunk Enterprise instance you run. The following section describes the procedure you must run to successfully update the splunk.secret key file.

If you update the splunk.secret key file, any tokens on the Splunk instance become invalid immediately. You cannot undo this change. After you update the key file, issue new authentication tokens to parties who previously had valid tokens and need to regain access.

If the authentication tokens feature is in use, but is temporarily disabled when you rotate the splunk.secret file, then you must manually delete the invalidated tokens after you re-enable the tokens feature.

See Set up authentication with tokens for information on authentication tokens and how to create and manage them.

Update the key file on a single instance of Splunk Enterprise with the CLI

• From a shell, command prompt, or PowerShell window, run the following command:

splunk rotate splunk-secret

Splunk Enterprise creates a new splunk.secret file and re-encrypts all secure configurations using the new file, based on the new cipher suite.

Update the key file on a single instance of Splunk Enterprise using REST

• Make the following HTTP POST request to the /services/server/security/splunk-secret/rotate REST endpoint:

curl -u <splunk username>:<splunk password> https://<splunk server>:<management port>/services/server/security/splunk-secret/rotate -X POST

Update the key file on a search head cluster (SHC) with the CLI

Prior to updating the splunk.secret on a search head cluster, confirm the following conditions are true for the cluster:

• The cluster is healthy, meaning that the search head cluster captain and all its peers are up, and that configuration replication occurs regularly. The CLI command fails if it detects that the cluster is not healthy.

You can check configuration status with the splunk show shcluster-status --verbose CLI command and view the last_conf_replication entry in the output of that command.

• The cluster is not in a rolling restart or upgrade. The CLI command fails if it detects that the cluster is in either of these states.
• All peers in the cluster run the same version of Splunk Enterprise. The CLI command fails if it detects any version mismatches among the peers.

The cluster captain always determines the shared splunk.secret key file for a cluster. If you attempt to rotate the key file as a cluster peer becomes a captain, a situation can occur where the newly-elected captain pushes the original key file to the rest of the cluster peers. Confirming that the cluster is healthy before starting a key file rotation prevents this possibility.

If you run a SHC deployer instance to distribute configurations, that instance does not use the same splunk.secret file as the other SHC members. If you push pre-encrypted configurations to SHC members using the deployer, you must first re-encrypt those configuration with the new shared splunk.secret file before you distribute them with the SHC deployer.

Consider running the key file update process in a maintenance window. When you update the key file on a search head cluster, the cluster peers remove and then re-add themselves to the cluster using the updated key file, and any scheduled searches that the cluster has configured are not available during this period. The amount of time that scheduled searches are not available depends on the size of the cluster.

splunk rotate shcluster-splunk-secret

The search head cluster captain generates the new key file and re-encrypts its passwords with that file. As peers report in to the captain at standard intervals, the captain directs each peer to remove and re-add itself to the cluster. When the peers reconnect, the captain then sends the new key file to each peer, which then subsequently re-encrypt their passwords with the new key file.

Update the key file on a search head cluster using REST

Before attempting to update the key file on a search head cluster, read the information in "Update the key file on a search head cluster (SHC) with the CLI" for important prerequisites to making the update.

curl -u <splunk username>:<splunk password> https://<splunk server>:<management port>/services/shcluster/captain/control/control/rotate-splunk-secret -X POST

Troubleshoot cipher version mismatches

Splunk instances that run a version lower than 7.2 cannot communicate securely with instances that run version 7.2 and higher if the higher version has been configured to use the new cipher suite and MAC only. Lower version instances cannot use the new cipher suite at all. A 7.2 or higher version instance must be configured to use at least the legacy cipher to communicate with lower version instances.

If a lower version Splunk instance attempts to connect to a higher version instance that has only been configured to use the new cipher suite and MAC, the instance that makes the connection logs the following error in $SPLUNK_HOME/var/log/splunk/splunkd.log:

ERROR HttpClientRequest - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'

ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:default-autolb-group] in 'outputs.conf' matches the same setting  under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://ronnie:8090/services/indexer_discovery http_code=502 http_response="Unauthorized"]

To fix the problem, do one of the following:

• If the connecting Splunk instance runs a version lower than 7.2, upgrade the instance to 7.2 or higher, if possible.
• Similarly, if the connecting instance runs version 7.2 or higher and the instance being connected to runs a version lower than 7.2, upgrade the lower versioned instance to 7.2 or higher, if possible.
• If the connecting instance runs a version lower than 7.2 and the instance being connected to runs version 7.2 or higher, edit server.conf on the 7.2 or higher instance and confirm that the signatureVersion setting is set to v1,v2 and the legacyCiphers setting is set to decryptOnly. These are the defaults for 7.2 or higher instances.