Configure secure communications between Splunk instances with updated cipher suite and message authentication code

Version 7.2 of Splunk Enterprise, Splunk Light, and the universal forwarder introduces a new cipher suite and message authentication code (MAC), that it uses for data encryption and secure communications between Splunk software instances. The cipher suite and MAC replace the current cipher that Splunk software has used for these types of communications.

The new cipher suite uses a stronger, more secure mechanism for encrypting and decrypting file-based data. It uses a separate MAC to handle node authentication during communications between Splunk instances. Previously, the legacy cipher handled both types of operations.

By default, version 7.2 and higher of Splunk software uses the new cipher suite and MAC for these types of communication. They have also been configured to use the existing legacy cipher for backward compatibility in both data encryption and node authentication with versions lower than 7.2.

This backward compatibility lets you plan upgrades for your Splunk and universal forwarder instances while maintaining secure communications between the instances. Later, after you have upgraded all instances in your deployment to version 7.2 or higher, you can disable the legacy cipher and your Splunk instances continue internal communications using only the new cipher suite and MAC.

The main points for understanding how to use the ciphers are:

• Versions of on-premises Splunk software lower than 7.2 do not have the new cipher suite and MAC, and cannot have either integrated within them.
• You must run version 7.2 or higher to take advantage of the stronger encryption capabilities that the new cipher suite and MAC offer.
• Splunk-to-Splunk communication happens using either the legacy cipher or the new cipher suite/MAC, not both
• A Splunk instance that uses only the legacy cipher cannot communicate at all with a Splunk instance that uses only the new cipher suite and MAC.

Configure cipher usage and decryption with configuration files

By default, versions of Splunk software 7.2 and higher use both the legacy cipher and new cipher suite and MAC for communications between instances, depending on the version of the instance they communicate with:

• For backward compatibility, when 7.2 and higher instances of Splunk software communicate with instances of Splunk software lower than 7.2, they use only the legacy cipher.
• When 7.2 and higher instances communicate with other 7.2 and higher instances, they use only the new cipher suite and MAC.

You must configure the cipher settings with the server.conf configuration file on instances of Splunk that are version 7.2 and higher only. You cannot configure cipher settings on versions lower than 7.2. You also cannot configure the ciphers in Splunk Web.

After you make configuration changes, you must restart Splunk instances for the changes to take effect.

Configure which ciphers the Splunk instance uses

1. In the $SPLUNK_HOME/etc/system/local directory, create server.conf if it does not already exist. Do not create this file in $SPLUNK_HOME/etc/default as this file gets overwritten whenever you upgrade.
2. Use a text editor to open the file.
3. Create a [node_auth] stanza and add the following block of text to it depending on your specific needs:
   

   One or more instances runs lower than 7.2	All instances run 7.2 or higher
   [node_auth] signatureVersion = v1,v2	[node_auth] signatureVersion = v2

4. Save the file and close it.
5. Restart the Splunk instance. The instance initiates Splunk-to-Splunk communications using the ciphers that you specified.

Configure legacy cipher decryption options

1. In the $SPLUNK_HOME/etc/system/local directory, create server.conf if it does not already exist. Do not create this file in $SPLUNK_HOME/etc/default/ as this file gets overwritten whenever you upgrade.
2. Use a text editor to open the file.
3. Under the [general] stanza, add the legacyCiphers setting, based on the versions of Splunk software that run in your Splunk deployment:

   One or more instances runs lower than 7.2	All instances run 7.2 or higher
   legacyCiphers = decryptOnly	legacyCiphers = disabled

4. Save the file and close it.
5. Restart the Splunk instance. The instance decrypts configurations in accordance with the legacyCiphers setting.

Troubleshoot cipher version mismatches

Splunk instances that run a version lower than 7.2 cannot communicate securely with instances that run version 7.2 and higher if the higher version has been configured to use the new cipher suite and MAC only. Lower version instances cannot use the new cipher suite at all. A 7.2 or higher version instance must be configured to use at least the legacy cipher to communicate with lower version instances.

If a lower version Splunk instance attempts to connect to a higher version instance that has only been configured to use the new cipher suite and MAC, the instance that makes the connection logs the following error in $SPLUNK_HOME/var/log/splunk/splunkd.log:

ERROR HttpClientRequest - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'

ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:default-autolb-group] in 'outputs.conf' matches the same setting  under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://ronnie:8090/services/indexer_discovery http_code=502 http_response="Unauthorized"]

To fix the problem, do one of the following:

• If the connecting Splunk instance runs a version lower than 7.2, upgrade the instance to 7.2 or higher, if possible.
• Similarly, if the connecting instance runs version 7.2 or higher and the instance being connected to runs a version lower than 7.2, upgrade the lower versioned instance to 7.2 or higher, if possible.
• If the connecting instance runs a version lower than 7.2 and the instance being connected to runs version 7.2 or higher, edit server.conf on the 7.2 or higher instance and confirm that the signatureVersion setting is set to v1,v2 and the legacyCiphers setting is set to decryptOnly. These are the defaults for 7.2 or higher instances.