Configure Splunk Enterprise for Common Criteria

After you install the Splunk Enterprise instance and SELinux policy, as described in Install the Common Criteria-compliant Splunk Enterprise and SELinux policy packages, you can then configure the instance.

Set the splunk user

You must perform the tasks in this topic as the "splunk" user. The "splunk" user is is the user under which your Splunk Enterprise application runs. If you create or modify any files in the installation as the root user or any other user, the splunk user cannot access Splunk Enterprise, which can cause unexpected behavior.

Run the following commands to become the "splunk" user:

su - splunk  
export SPLUNK_HOME=/opt/splunk
export SPLUNK_ETC=/etc/opt/splunk

Generate or obtain Common Criteria-compliant security certificates

Splunk Enterprise in Common Criteria mode does not generate any cryptographic keys or certificates. Use OpenSSL or any other key or certificate generation tool to generate self-signed certificates. These certificates must be Federal Information Process Standards (FIPS)-compliant. You can also get certificates issued by certificate authorities (CAs) such as Verisign/GlobalSign. The certificates must be in privacy-enhanced mail (PEM) format.

If you use the Splunk-generated default certificates, Splunk Enterprise will not be able to communicate over the network. The CLI, as well Splunk Web, will not function. Splunk Enterprise logs any errors in splunkd.log.

List of certificates and keys

Provide certificates and keys for Splunk Enterprise to work in Common Criteria mode. Some of these certificates are optional, depending on whether you need the functionality. The details of these attributes can be found in /etc/opt/splunk/system/README/*.conf.spec.

Configuration file	Configuration stanza name	Setting name	Notes
server.conf	[sslConfig]	serverCert
server.conf	[sslConfig]	sslRootCAPath
server.conf	[kvstore]	serverCert
web.conf	[settings]	serverCert
audit.conf	[auditTrail]	privateKey	Provided in the installation step before installing the SELinux policy package
audit.conf	[auditTrail]	privateKey	Provided in the installation step before installing the SELinux policy package
audit.conf	[auditTrail]	publicKey	Provided in the installation step before installing the SELinux policy package
distsearch.conf	[tokenExchKeys]	publicKey
distsearch.conf	[tokenExchKeys]	privateKey
inputs.conf	[SSL]	serverCert	Needed only if using splunktcp-ssl for getting input from forwarders
outputs.conf	[SSL]	serverCert	Needed if this is a forwarder configuration
outputs.conf	[tcpout]	clientCert	CRLs: must store CRL files under /etc/opt/splunk/auth/crl directory. Look at README in that directory

Update Splunk Enterprise configuration files with Common Criteria-compliant settings

Update or create the /etc/opt/splunk/system/local.conf files with these settings. The paths shown in these samples are for illustration and can be different if desired.

server.conf

[general]
requireBootPassphrase = true
allowRemoteLogin = never

[sslConfig]
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384
# Note: ECDHE-ECDSA-AES256-SHA384 equates to ECDHE-ECDSA-AES256-CBC-SHA384 as defined in the Security Target.
sendStrictTransportSecurityHeader = true
serverCert = <absolute_path_to_server_certificate>
sslAltNameToCheck = <comma separated list of SSL alternate names>
sslCommonNameList = <comma separated list of Common Namess>
# On RHEL 6.5, this will be typically '/etc/pki/tls/certs/ca-bundle.crt'.
# For any additional CAs that need to be trusted, append them to this file.
sslRootCAPath = <path to OS root cert store>
sslVerifyServerCert = true
sslVersions = tls1.2
sslVersionsForClient = tls1.2

[kvstore]
serverCert = <absolute path to App Key Value Store certificate>

[applicationsManagement]
allowInternetAccess = false

web.conf

[settings]
cipherSuite= ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384
enableSplunkWebSSL = 1
privKeyPath = <absolute path to encrypted private key>
serverCert = <absolute path to public certificate>
sslVersions = tls1.2

authentication.conf

[secrets]
disabled = false

alert_actions.conf

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384
pdf.html_image_rendering = false
sslAltNameToCheck = <comma separated list_of SSL alternate names>
sslCommonNameToCheck = <comma separated list of common names>
sslVerifyServerCert = true 
sslVersions = tls1.2
use_tls = 1

inputs.conf

#Use only if configuring Splunk Enterprise as an Indexer, which can receive data from forwarders.
[SSL]
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384
requireClientCert = true
serverCert = <absolute path to server certificate>
sslAltNameToCheck = <comma separated list of SSL alternate names>
sslCommonNameToCheck = <comma separated list of common names>
sslVersions = tls1.2

outputs.conf

[tcpout]
defaultGroup = group1
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384
clientCert = <absolute path to client certificate>
sslAltNameToCheck = <comma separated list of SSL alternate names>
sslCommonNameToCheck = <comma separated list of common names>
sslVerifyServerCert = true
sslVersions = tls1.2
useClientSSLCompression = true

Enable Common Criteria mode for Splunk Enterprise

To activate Common Criteria mode for Splunk Enterprise, modify the /etc/opt/splunk/splunk-launch.conf configuration file.

SPLUNK_COMMON_CRITERIA=1
SPLUNK_FIPS=1

# Do not generate python byte code
PYTHONDONTWRITEBYTECODE=1

Create a Splunk Enterprise administrative account

When Splunk Enterprise first starts, it prompts you to create an administrative account. You must create this account, or your cannot log in.

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: 

1. Type in the username that you want the software to create, for example admin.
2. The software then prompts:

   Password must contain at least:
      * 8 total printable ASCII character(s).
   Please enter a new password:
   

   
   Type in a password that meets the shown password requirements.
3. Re-enter the password you chose in the previous step. The software creates the account and continues start-up.

Simplify environment setup with .bashrc inclusion

Include these lines in the /home/splunk/.bashrc shell configuration file so that your Common Criteria-compliant environment is setup properly when using the Splunk CLI.

export SPLUNK_ETC=/etc/opt/splunk
export DBUS_SESSION_BUS_ADDRESS=$(awk '{ print $1}' /tmp/dbus-address)
export DBUS_SESSION_BUS_PID=$(awk '{ print $2}' /tmp/dbus-address)
export PATH=/usr/bin:$PATH
. /opt/splunk/bin/setSplunkEnv

Generate a Common Criteria-Compliant "splunk.secret" file

The Splunk Enterprise deterministic random bit generator (DRBG) must be seeded with sufficient entropy. Use the rdrand-gen (RdRand) tool to generate a seed file that Splunk Enterprise can use to generate a Common Criteria-compliant splunk.secret file.

Run the following commands:

rdrand-gen -v -n 1M -m reseed_delay -o $SPLUNK_HOME/rdrand.bin
runcon -u system_u -t splunk_t -r system_r splunk gen-cc-splunk-secret --rand-path $SPLUNK_HOME/rdrand.bin --bytes-to-read 384

Initialize Secret Storage

1. Before starting Splunk Enterprise, start the dbus process. The dbus daemon communicates with GNOME keyring.

   /home/splunk/run_dbus.sh
   

2. Check that the the dbus process is running with the splunk_dbusd_t SELinux context

   ps auxZ | grep dbus
   
   You will see an output similar to:
   unconfined_u:system_r:splunk_dbusd_t:s0 splunk 28563 0.0  0.0 31680 872 ?      Ssl  14:36   0:00 dbus-daemon --session --print-pid --print-address --fork
   

3. Source the .bashrc so that the environment variables for dbus are are set up.

    source /home/splunk/.bashrc

4. Initialize the secret storage password:

   runcon -u system_u -t splunk_t -r system_r splunk secret-storage --unlock
   

5. To see list of keys available for secret storage:

   runcon -u system_u -t splunk_t -r system_r splunk secret-storage --spec
   

Add secrets to Secret Storage

Run the following command to add secrets to the GNOME keyring:

runcon -u system_u -t splunk_t -r system_r splunk secret-storage --write --no-prompt <conf-file> <stanza-name> <setting-name> <passphrase>

where:

<conf-file> is the configuration file (for example, server.conf) <stanza-name> is the name of a stanza in the file (for example, sslConfig) <setting-name> is the name of a setting (for example, sslKeysfilePassword) <passphrase> is the passphrase to use

List of Secrets

<conf-file>, <stanza-name>, <attribute-name>
alert_actions.conf, [email], auth_password
audit.conf, [auditTrail], privateKeyPassphrase
distsearch.conf, [tokenExchKeys], privateKeyPassphrase
inputs.conf, [SSL], sslPassword
outputs.conf, [tcpout], sslPassword
server.conf, [sslConfig], sslPassword
server.conf, [kvstore], sslPassword
web.conf, [settings], sslPassword

An example: [splunk@qa-cc-rhel65-03 ~]$ runcon -u system_u -t splunk_t -r system_r splunk secret-storage --write --no-prompt server sslConfig sslKeysfilePassword password

Start Splunk Enterprise and validate your configuration

1. Start Splunk Enterprise.

   /home/splunk/run_splunk.sh
   

2. Confirm that Splunk Enterprise is running with the splunk_t SELinux context.

   ps auxZ | grep splunk
   

3. Review the /opt/splunk/var/log/splunk/splunkd.log file. Look for a message similar to the following that indicates that Splunk Enterprise is running in common criteria mode:

    ServerConfig - Splunk is starting in Common Criteria Mode. 

   
   Both splunkd and splunkweb should work normally in Common Criteria mode.
4. Confirm that you have a valid Splunk Enterprise license installed. See Types of Splunk Enterprise licenses.

Using Splunk Enterprise in Common Criteria Mode

• Run Splunk CLI commands as the 'splunk' user, When you run the commands, prepend the SELinux run-on command to set the proper context within SELinux.

  runcon -u system_u -t splunk_t -r system_r splunk <cli_cmd>
  

• To stop Splunk Enterprise, use the provided stop_splunk.sh script:

  /home/splunk/stop_splunk.sh
  

• If you need to stop dbus, use the provided stop_dbus.sh script and run the following commands:

  /home/splunk/stop_dbus.sh
  pkill gnome-keyring
  rm /tmp/dbus-address
  

Update certificate revocation list information

Splunk Enterprise expects to find the certificate revocation lists (CRLs) for revocation-checking in the $SPLUNK_ETC/auth/crl directory. It expects these lists to be in privacy enhanced mail (PEM) format. Splunk provides a script as an example of how you can automate the update of CRLs. Any other mechanism that downloads the CRL files under the designated location will work as well. The example script expects the user to provide a list of URLs, one per line, which are CRLs for the certificates Splunk Enterprise will use.

An example file follows:

 $ cat crl.txt 
http://pki.google.com/GIAG2.crl
http://g.symcb.com/crls/gtglobal.crl

The following bash script reads through the crl.txt file, download the CRL files into the $SPLUNK_ETC/auth/crl directory, and converts it into PEM format if necessary.

#!/bin/bash

# NOTE: Only applicable for Splunk version 6.4.x and higher, while running in Common Criteria mode.
# This script is provided as an example for downloading the CRL files in a location
# Splunk expects it to be. Any other mechanism which updates CRL files should work.

# The user can run the script one time OR setup a cron job to run it periodically (say every 30 min).
# The script cleans out ALL existing CRL files (*.crl, *.pem) and then downloads the new versions.

# Example invocation: /home/splunk/update_crl.sh /home/splunk/crl.txt /etc/opt/splunk/auth/crl

if [ "$#" -ne 2 ]; then
	echo "Usage: $0 <crllist_file_absolute_path> <crl_download_location_absolute_path>"
	exit 1
fi

PWD=`pwd`

filename=$1
crl_dir=$2

if [ ! -f "$filename" ] || [ ! -d "$crl_dir" ] || [[ "$filename" != /* ]] || [[ $crl_dir != /* ]]; then
	echo "Both the crllist_file and crl_download_location must exist and be specified as absolute paths."
	exit 2
fi

# go to $crl_dir
cd $crl_dir
# remove older CRL files if present
rm -rf *.crl *.pem

while read -r line || [[ -n "$line" ]]; do
	url=$line
	wget $url
	if [ "$?" -ne 0 ]; then
		echo "Failed to download CRL file: $url"
	fi
done < "$filename"

# For each file except README in this dir, check if the file is in DER format.
# If yes, then convert to PEM and remove the corresponding CRL file.
for f in ./*
do
	if [ $f != "./README" ];then
		# use openssl from the OS itself
		openssl crl -in $f -text -noout &> /dev/null
		if [ "$?" -ne 0 ]; then #DER format, must convert to PEM
			openssl crl -inform der -in $f -out $f.pem 
			if [ "$?" -ne 0 ]; then
				echo "Failed to convert DER format CRL file ($f) into PEM format. Splunk will not use this CRL file"
			fi
			rm $f
		fi
	fi
done

#revert to old pwd
cd $PWD

Let's say, you have saved this script under /home/splunk/update_crl.sh. Set the appropriate SELinux context and file-permissions on this file.

chown splunk:splunk update_crl.sh crl.txt
chcon -u system_u -r object_r -t initrc_exec_t update_crl.sh
chmod 755 update_crl.sh

You can set up a cron job to execute this script periodically and update CRL files which Splunk Enterprise uses. Run the following command as the 'splunk' user,

$ crontab -e 

This opens a cron configuration file for 'splunk' user. Add this line to this file to update CRL info every 15 minutes.

*/15 * * * *  /home/splunk/update_crl.sh /home/splunk/crl.txt /etc/opt/splunk/auth/crl &> /dev/null

This updates the CRL files. To load the updated CRL info into Splunk Enterprise, you can simply run a search in Splunk Web: | rest /services/server/security/crl/_reload

To automate, save this search as the 'admin' user. You can update the /etc/opt/splunk/users/admin/search/local/savedsearches.conf file to add the search, similar to the following:

[Reload CRL information]
display.general.type = statistics
display.page.search.tab = statistics
display.visualizations.show = 0
request.ui_dispatch_app = search
request.ui_dispatch_view = search
enableSched = 1
#update every minute (change if necessary)
cron_schedule=* * * * *
search = | rest /services/server/security/crl/_reload

You can check that CRL info is updating in Splunk Enterprise on a regular basis by enabling 'DEBUG' logging for the 'X509' component. Messages similar to the following appear when CRL info is reloaded in splunkd.log:

09-08-2016 15:50:14.452 -0700 DEBUG X509 - Successfully added (/etc/opt/splunk/auth/crl/GIAG2.crl.pem) to the revocation store.
09-08-2016 15:50:14.452 -0700 DEBUG X509 - Successfully added (/etc/opt/splunk/auth/crl/gtglobal.crl.pem) to the revocation store.
09-08-2016 15:50:14.452 -0700 DEBUG X509 - Added 2 CRL files from the CRL directory(/etc/opt/splunk/auth/crl) to the revocation store.