Use SSL for secure intra-Splunk communication

Use SSL for secure intra-Splunk communication

Overview

SSL is turned on by default for intra-Splunk communications over the Splunk management port (default 8089). Of particular note, distributed search often performs better with SSL enabled because of its built-in data compression.

The main scenarios for communication over the management port are:

• Distributed search: communicating between indexers and search heads.
• Deployment server: communicating between deployment servers and clients.
• Splunk Web: communicating between Splunk Web and indexers or search heads (Splunk Web to splunkd).
• Splunk CLI: communicating between the CLI and a Splunk instance.

Note: This section only describes how to use SSL for communication over the management port. To turn on SSL for browser to Splunk Web communication, see "Use SSL (HTTPS) for secure access to Splunk Web". To use SSL to communicate between forwarders and receivers, see "Use SSL to encrypt and authenticate data from forwarders".

Configure SSL

To make changes to SSL settings, edit server.conf.

When the Splunk server is turned on for the first time, the server generates a certificate for that instance. This certificate is stored in the $SPLUNK_HOME/etc/auth/ directory by default.

Change SSL settings by editing $SPLUNK_HOME/etc/system/local/server.conf. Edit this file in $SPLUNK_HOME/etc/system/local/ or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.

The SSL configuration attributes are located in [sslConfig]. These are some of the most typical attributes, but refer to server.conf for the full set:

[sslConfig]
enableSplunkdSSL = true
sslKeysfile = server.pem
sslKeysfilePassword = password
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
certCreateScript = $SPLUNK_HOME/bin/genSignedServerCert.py

Note the following:

• enableSplunkdSSL. Set to true to enable SSL for intra-Splunk communications. Defaults to true.
• sslKeysfile. The certificate for this Splunk instance (created on Splunk start-up by default, if the certCreateScript tag is present). The path to the keyfile is relative to the caPath setting. If your keyfile is located outside $SPLUNK_HOME, you must specify the full (absolute) path.
• sslKeysfilePassword. The password for the pem file store. Set to password by default.
• caCertFile. The name of the certificate authority file.
• caPath. The path where the Splunk certificates are stored. Default is $SPLUNK_HOME/etc/auth.
• certCreateScript. The script for creating and signing server certificates.

With the default script enabled, Splunk will generate a certificate in the caPath directory on start-up.

Deactivate SSL

To deactivate SSL, set enableSplunkdSSL to FALSE. This will disable SSL.

Note: Running splunkd without SSL is not generally recommended. Distributed search will often perform better with SSL enabled.

Disable SSLv2

To disable SSLv2 and tell the HTTP server to only accept connections from SSLv3 clients, set the supportSSLV3Only attribute to true. By default, this setting is false.

Distribute key files to distributed search peers

To learn how to distribute key files to distributed search peers, look in the section on configuring distributed search in the Distributed Deployment manual: "Distribute the key files".

Generate a new root certificate

By default, all Splunk servers use the same root certificate. This allows Splunk instances to connect to each other out of the box.

Important: The default Splunk root certificate (which can be found in $SPLUNK_HOME/etc/auth/ca.pem) uses a private key that every other user of Splunk in the world has access to. Possession of a certificate authority's private key will allow attackers to generate certificates that are signed by the trusted authority, which would defeat attempts to control authentication via PKI. This is only important if you wish to use SSL authentication functionality.

The script $SPLUNK_HOME/bin/genRootCA.sh (%SPLUNK_HOME%\bin\genRootCA.bat on Windows) allows you to create a root certificate to be used in creating subsequent server and web certificates. Run this script when you want to regenerate the certificates Splunk uses. It generates cacerts.pem (public key) and ca.pem (public/private password protected PEM). When you run it, it checks to see if certs are already in place, and if they are, prompts you to overwrite them. It then wraps these files into an X509-formatted cert. Distribute cacerts.pem to clients as desired and keep ca.pem in a secure location.

genRootCA.sh example for *nix

The following example generates a new root certificate and private key pair at $SPLUNK_HOME/etc/auth/ca.pem.

Note: if Splunk is installed anywhere but /opt/splunk, you will need to set the environment variable OPENSSL_CONF to the path to your Splunk installation's openssl.cnf.

$ export OPENSSL_CONF=$SPLUNK_HOME/openssl/openssl.cnf
$ cd $SPLUNK_HOME
$ ./bin/genRootCA.sh  -d ./etc/auth/
There is ca.pem in this directory. If you choose to replace the CA, then 
splunk servers will require new certs signed by this CA before they can 
interact with it.
Do you wish to replace the CA ? [y/N]
y
rm: cacert.pem: No such file or directory
This script will create a root CA
It will output two files. ca.pem cacert.pem
Distribute the cacert.pem to all clients you wish to connect to you.
Keep ca.pem for safe keeping for signing other clients certs
Remember your password for the ca.pem you will need to later to sign other 
client certs
Your root CA will expire in 10 years
Generating a 1024 bit RSA private key
..++++++
...........++++++
writing new private key to 'cakey.pem'
-----
Signature ok
subject=/C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
Getting Private key
subject= /C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
issuer= /C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
notBefore=Apr 22 16:40:09 2010 GMT
notAfter=Apr 19 16:40:09 2020 GMT

genRootCA.bat example for Windows

The following example generates a new root certificate and private key pair at %SPLUNK_HOME%\etc\auth. Make sure that the OPENSSL_CONF environment variable points to the Splunk installation's openssl.cnf. Also note that path followed by the -d option, which specifies the destination directory for the generated key pair, is a DOS-style path and does not contain spaces.

>cd "c:\Program Files\Splunk\bin"
>set OPENSSL_CONF=c:\Program Files\Splunk\openssl.cnf

To make sure the environment variable is entered correctly run:

>set OPENSSL_CONF

It should return:

c:\Program Files\splunk\bin>set OPENSSL_CONF
OPENSSL_CONF=c:\Program Files\Splunk\openssl.cnf

c:\Program Files\splunk\bin>

Run the following command:

C:\Program Files\Splunk\bin>splunk.exe cmd cmd.exe /c genRootCA.bat 
  -d c:\progra~1\Splunk\etc\auth

The output is the following:

C:\Program Files\Splunk\bin>splunk.exe cmd cmd.exe /c genRootCA.bat 
-d c:\progra~1\Splunk\etc\auth
There is ca.pem in this directory. If you choose to replace the CA 
then splunk servers will require new certs signed by this CA before 
they can interact with it.
Do you wish to replace the CA ? [y/N]
y
Deleting certs cacert.pem and ca.pem
del /f /q cacert.pem
del /f /q ca.pem
This script will create a root CA.
It will output two files: ca.pem cacert.pem.
Distribute the cacert.pem to all clients you wish to connect to you.
Keep ca.pem for safe keeping for signing other clients certs.
Remember your password for the ca.pem you will need to later to sign 
other client certs.
Your root CA will expire in 10 years.
"C:\Program Files\Splunk\bin\openssl.exe" req -newkey rsa:1024 -passout 
pass:password -subj /countryName=US/stateOrProvinceName=CA/localityName=
SanFrancisco/organizationName=SplunkInc/commonName=SplunkCA/organizationName=
SplunkUser/ -sha1 -keyout cakey.pem -out careq.pem

Loading 'screen' into random state - done
Generating a 1024 bit RSA private key
.....................................................................++++++
.......++++++
writing new private key to 'cakey.pem'
-----
"C:\Program Files\Splunk\bin\openssl.exe" x509 -req -in careq.pem -passin 
pass:password -sha1 -extensions v3_ca -signkey cakey.pem -out cacert.pem -days 3650
Loading 'screen' into random state - done
Signature ok
subject=/C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
Getting Private key
Create root cert ca.pem from cacert.pem and cakey.pem

cacert.pem

cakey.pem

"C:\Program Files\Splunk\bin\openssl.exe" x509 -subject -issuer -dates -noout 
-in ca.pem
subject= /C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
issuer= /C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
notBefore=Jul 15 21:54:14 2010 GMT
notAfter=Jul 12 21:54:14 2020 GMT

Generate a new signed certificate and private key pair

By default, all Splunk servers use a certificate signed by the common root certificate discussed above. This allows Splunk instances to connect to each other out of the box.

Important: Splunk STRONGLY recommends that you DO NOT use the default self-signed certificate. Use of these default certificate will not result in confidential transmission of data.

genSignedServerCert.sh example for *nix

On *nix platforms, $SPLUNK_HOME/bin/genSignedServerCert.sh allows you to create a new private key and server certificate using the current Splunk root certificate.

This shell script is a wrapper for the Python script that Splunk runs to generate certificates when you start it for the first time. This script creates a CSR (certificate signing request), self-signs it, and outputs a signed private key and certificate pair.

The following example will generate a new private key and new server certificate for the server example.splunk.com, which is signed against the local Splunk root certificate:

$ cd $SPLUNK_HOME
$ ./bin/genSignedServerCert.sh -d ./etc/auth/ -n server2 -c example.splunk.com -p

* Create certificate server2.pem signed by the root CA
* Store the server2.pem key file locally with your client/server application
* Enter a secret pass phrase when requested
* The pass phrase is used to access server2.pem in your application
* Enter the application's host name as the Common Name when requested
* Enter the root CA pass phrase (Getting CA Private Key) to sign the key file
* The key file will expire after one year or sooner when the root CA expires
Generating a 1024 bit RSA private key
...........................++++++
....................++++++
writing new private key to 'server2.pemkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CA
Locality Name (eg, city) []:SanFrancisco
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Splunk Inc.
Organizational Unit Name (eg, section) []:Security
Common Name (eg, YOUR name) []:example.splunk.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Signature ok
subject=/C=US/ST=CA/L=SanFrancisco/O=Splunk Inc./OU=Security/CN=example.splunk.com
Getting CA Private Key
subject= /C=US/ST=CA/L=SanFrancisco/O=Splunk Inc./OU=Security/CN=example.splunk.com
issuer= /C=US/ST=CA/L=SanFrancisco/O=SplunkInc/CN=SplunkCA/O=SplunkUser
notBefore=Apr 22 17:20:31 2010 GMT
notAfter=Apr 21 17:20:31 2013 GMT

genSignedServerCert.py example for Windows

On Windows, run genSignedServercert.py:

C:\Program Files\Splunk\bin>splunk cmd python "c:\Program Files\splunk\bin\
  gensignedservercert.py" -d "c:\Program Files\Splunk\etc\auth" -n server2 
  -c win2008.splunk.com -p
* Create certificate server2.pem signed by the root CA
* Store the server2.pem key file locally with your client/server application
* Enter a secret pass phrase when requested
* The pass phrase is used to access server2.pem in your application
* Enter the application's host name as the Common Name when requested
* Enter the root CA pass phrase (Getting CA Private Key) to sign the key file
* The key file will expire after one year or sooner when the root CA expires
Loading 'screen' into random state - done
Generating a 1024 bit RSA private key
.................++++++
......................................................++++++
writing new private key to 'server2key.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
Verify failure
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CA
Locality Name (eg, city) []:San Francisco
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Splunk, Inc.
Organizational Unit Name (eg, section) []:Splunk Customer Support
Common Name (eg, YOUR name) []:Splunk Support
Email Address []:support@splunk.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:<password>
An optional company name []:
Loading 'screen' into random state - done
Signature ok
subject=/C=US/ST=CA/L=San Francisco/O=Splunk, Inc./OU=support/CN=splunksupport/
emailAddress=support@splunk.com
Getting CA Private Key
subject= /C=US/ST=CA/L=San Francisco/O=Splunk, Inc./OU=support/CN=splunksupport/
emailAddress=support@splunk.com
issuer= /C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=
support@splunk.com
notBefore=Jun 14 19:28:27 2010 GMT
notAfter=Jun 13 19:28:27 2013 GMT

Generate a CSR (Certificate Signing Request)

If your organization requires that your Splunk deployment use a certificate signed by an external CA, or you otherwise want to use certificates signed by a root certificate other than the default Splunk authority, you can generate the CSR to send to the CA.

For *nix

On *nix platforms:

1. Run the following command:

  openssl req -new -key [certificate name].pem -out [certificate name].csr

2. You are prompted for the following X.509 attributes of the certificate:

• Country Name: Use the two-letter code without punctuation for country; for example, US or GB.
• State or Province: Spell out the state completely. Do not abbreviate the state or province name; for example, California.
• Locality or City: The Locality is the city or town name; for example, Oakland. Do not abbreviate; for example, Los Angeles (not LA), Saint Louis (not St. Louis).
• Company: If your company or department contains &, @, or any other non-alphanumeric symbol that requires you to use the shift key, you must spell out the symbol or omit it. For example, Fflanda & Rhallen Corporation must be either Fflanda Rhallen Corporation or Fflanda and Rhallen Corporation.
• Organizational Unit: This field is optional, but you can specify it to help identify certificates registered to an organization. The Organizational Unit (OU) field is the name of the department or organization unit making the request. To skip the OU field, press Enter.
• Common Name: The Common Name is the Host + Domain Name; for example, www.company.com or company.com. This must exactly match the host name of the server where you intend to deploy the certificate.

This creates a private key (<certificate name>.key), which is stored locally on your server, and a CSR (<certificate name>.csr), which contains the public key associated with the private key. You can then use this information to request a signed certificate from an external CA.

To copy and paste the information into your CA's enrollment form, open the .csr file in a text editor and save it as a .txt file.

Note: Do not use Microsoft Word. It can insert extra hidden characters that alter the contents of the CSR.

For Windows

On Windows, the procedure is similar to the method described for *nix platforms, but it requires an extra step to set the ENV variable OPENSSL_CONF:

1. Open up a Command Prompt window and navigate to $SPLUNK_HOME\bin

2. Set the OPENSSL_CONF ENV variable:

   C:\Program Files\Splunk\bin>set OPENSSL_CONF=C:\Program Files\Splunk\openssl.cnf

3. Verify the variable has been set correctly:

  echo %OPENSSL_CONF%

4. Run the command to generate the CSR:

  openssl.exe req -new -key "C:\Program Files\Splunk\etc\auth\server.pem" 
    -out server.csr -passin pass:password

5. You are prompted for the following X.509 attributes of the certificate:

• Country Name: Use the two-letter code without punctuation for country; for example, US or GB.
• State or Province: Spell out the state completely. Do not abbreviate the state or province name; for example, California.
• Locality or City: The Locality is the city or town name; for example, Oakland. Do not abbreviate; for example, Los Angeles (not LA), Saint Louis (not St. Louis).
• Company: If your company or department contains &, @, or any other non-alphanumeric symbol that requires you to use the shift key, you must spell out the symbol or omit it. For example, Fflanda & Rhallen Corporation must be either Fflanda Rhallen Corporation or Fflanda and Rhallen Corporation.
• Organizational Unit: This field is optional, but you can specify it to help identify certificates registered to an organization. The Organizational Unit (OU) field is the name of the department or organization unit making the request. To skip the OU field, press Enter.
• Common Name: The Common Name is the Host + Domain Name; for example, www.company.com or company.com. This must exactly match the host name of the server where you intend to deploy the certificate.

This creates a private key (<certificate name>.key), which is stored locally on your server, and a CSR (<certificate name>.csr), which contains the public key associated with the private key. You can then use this information to request a signed certificate from an external CA.

To copy and paste the information into your CA's enrollment form, open the .csr file in a text editor and save it as a .txt file.

Note: Do not use Microsoft Word; it can insert extra hidden characters that alter the contents of the CSR.

Require a client certificate for Splunk Web to splunkd communication

To specify that any HTTPS client that connects to splunkd must have a certificate signed by the Splunk certificate authority, set requireClientCert=true in server.conf.

Warning: If you set requireClientCert=true, both Splunk Web and splunkd must use certificates provided by the same Root CA. Otherwise, Splunk Web will not be able to communicate with splunkd.

See the server.conf spec file for details on this attribute.

• Was this topic useful?
• Post a comment