Configure and install certificates in Splunk Enterprise for Splunk Log Observer Connect

You can set up SSL certificates to secure the connection between Splunk Enterprise and Splunk Log Observer Connect.

The procedures that you must follow depend on the type of certificates you intend to use to secure communications between Splunk Enterprise and Splunk Log Observer Connect. You have two options to obtain these certificates:

• Create and sign the certificates yourself. This is the fastest method to get certificates, but web browsers don't recognize them as secure because they are not from a verified certificate authority and don't appear in browser certificate stores.
• Get signed certificates from a third party. This option is the most secure method to get certificates, and doesn't result in browser validation problems, but involves a third party and potentially a cost to obtain the certificates.

After you receive or generate the certificates, you install them into your Splunk Enterprise deployment. You can then connect securely to the Splunk Enterprise instance using Splunk Log Observer Connect.

Prerequisites for configuring and installing certificates in Splunk Enterprise for Splunk Log Observer Connect

Before you can install certificates into Splunk Enterprise, you must have the following:

• A decision on the type of certificate with which you want to secure the Splunk Enterprise-to-Splunk Log Observer connection: Self-signed, or third party
• Experience using a shell prompt on *nix machines
• An understanding of the $SPLUNK_HOME installation directory hierarchy, in particular, where to put configuration files within the hierarchy

Generate and self-sign a certificate to secure the Splunk Enterprise management port

If you decide to create and sign your own certificates to secure the Splunk Enterprise management port for use with Splunk Log Observer Connect, follow at least Sections 1 through 4 of this procedure. After you have your signed certificate, you can then secure the management port and connect to the instance from Splunk Observability Cloud.

When you sign your own certificates, web browsers do not have the certificate in their certificate stores by default. This causes the browser to notify you that the connection is not secure.

1. Generate a new root certificate to be your Certificate Authority

Before you can sign a certificate, you must create a root certificate that acts as the certificate authority (CA) for signing certificates.

1. Open a shell prompt on the machine that runs Splunk Enterprise.
2. Create a new directory on your Splunk Enterprise installation to host the Splunk Log Observer Connect certificates and keys. For example, $SPLUNK_HOME/etc/auth/sloccerts.
   

   mkdir $SPLUNK_HOME/etc/auth/sloccerts

   Use a different directory than the default of $SPLUNK_HOME/etc/auth/splunkweb for your new certificates. This action prevents you from overwriting existing certificates and ensures that you can use the certificates that ship with Splunk Enterprise in the $SPLUNK_HOME/etc/auth/splunkweb directory for other Splunk components.

3. Change to the new directory you created.
   

   cd $SPLUNK_HOME/etc/auth/sloccerts

4. Use the OpenSSL program to generate a new RSA private key.
   

   $SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048
   

5. When the OpenSSL command prompts you, enter a password. After you create the password, the private key file myCAPrivateKey.key appears in your directory. This is the root certificate private key.
6. Generate a certificate signing request using the root certificate private key myCAPrivateKey.key. In the command, replace <your-domain> with the URL of your Splunk Enterprise instance.
   

   $SPLUNK_HOME/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr
   

7. Provide the password you created to the private key myCAPrivateKey.key. A new certificate signing request (CSR) file myCACertificate.csr appears in your directory.
8. Use the CSR file to generate a new root certificate and sign it with your private key. In the first command that follows, replace <your-domain> with the URL of your Splunk Enterprise instance.
   

   echo -e "# ssl-extensions-x509.cnf\n[v3_ca]\nbasicConstraints = CA:FALSE\nkeyUsage = digitalSignature, keyEncipherment\nsubjectAltName = DNS:<url-of-splunk-enterprise-instance>" > ssl-extensions-x509.cnf
   
   $SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -signkey myCAPrivateKey.key -extensions v3_ca -extfile ./ssl-extensions-x509.cnf -out myCACertificate.pem -days 3650
   

9. When the OpenSSL command prompts you, provide for the password to the private key myCAPrivateKey.key. A new certificate myCACertificate.pem appears in your directory. This is your public certificate.

2. Create a new private key

The private key is what you use in combination with the public certificate to sign the certificate you will use to secure the connection between Splunk Enterprise and Splunk Log Observer Connect.

1. Use the OpenSSL command to generate a new private key.

   $SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out mySplunkWebPrivateKey.key 2048
   

2. When the OpenSSL command prompts you, enter a password. After you type in the password, a new key file, mySplunkWebPrivateKey.key appears in your directory.
3. Remove the password from your key. You must do this because Splunk Web does not support password-protected private keys.

   $SPLUNK_HOME/bin/splunk cmd openssl rsa -in mySplunkWebPrivateKey.key -out mySplunkWebPrivateKey.key
   

   
   

   You can verify that you successfully removed the password with the following command. If successful, you can read the contents of the certificate without having to provide a password.

   $SPLUNK_HOME/bin/splunk cmd openssl rsa -in mySplunkWebPrivateKey.key -text
   

3. Create and sign a server certificate

Now that you have both the private key and the public certificate, you can then create the server certificate which you then use to secure The Splunk Enterprise to Splunk Log Observer Connect connection.

1. Create a new certificate signature request using your private key mySplunkWebPrivateKey.key.
   

   $SPLUNK_HOME/bin/splunk cmd openssl req -new  -key mySplunkWebPrivateKey.key -out mySplunkWebCert.csr
   

   
   The CSR file mySplunkWebCert.csr appears in your directory.
2. Sign the CSR with the root certificate private key myCAPrivateKey.key.
   

   $SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in mySplunkWebCert.csr -CA myCACertificate.pem -extensions v3_ca -extfile ./ssl-extensions-x509.cnf -CAkey myCAPrivateKey.key -CAcreateserial -out mySplunkWebCert.pem -days 1095
   

3. When the OpenSSL binary prompts you, provide the password to the root certificate private key myCAPrivateKey.key. The certificate mySplunkWebCert.pem appears in your directory. This is your server certificate.

4. Combine the server certificate and public certificates into a single certificate file

Combine your server certificate and the public certificates, in that order, into a single file that is in privacy-enhanced mail (PEM) format. If the file is not in PEM format, Splunk Enterprise cannot read and verify the combined certificates as valid.

1. cat mySplunkWebCert.pem myCACertificate.pem > mySplunkWebCertificate.pem
   
   cat mySplunkWebCertificate.pem mySplunkWebPrivateKey.key > myFinalCert.pem
   

5. Set up multiple certificates (Optional)

If you have to use multiple server certificates to secure the connection between Splunk Enterprise and Splunk Log Observer Connect, append the intermediate certificate to the end of the server certificate file in the following order:

[ server certificate]
[ intermediate certificate]
[ root certificate (if required) ]

For example, the certificate chain might look like this:

-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the root certificate for the CA)...
-----END CERTIFICATE-----

After you have concatenated the certificates, distribute them and configure Splunk Web and the Splunk daemon to use them. See "Install and configure certificates on the management port" later in this topic for the procedure.

Obtain a certificate from a third party to secure the Splunk Enterprise management port

If you want to use a signed third-party certificate from a CA such as Let's Encrypt, Sectigo, or Symantec, you can acquire the certificate directly from those CAs, install them into your Splunk Enterprise instance, and then confirm that Splunk Observability Cloud can connect to the instance.

There are many ways to obtain third-party certificates, and the process to obtain them from each CA is different. When choosing the best CA for you, pick one that meets your security needs and budget.

After you get the certificates, you must combine the public certificate, server certificate, and public key into a single PEM format file. You can use the procedures "Combine the server certificate and public certificates into a single certificate file" and, if necessary, "Set up multiple certificates" in this topic to concatenate and combine certificates in the same way you would if you created your own certificates.

After you do this, you can distribute the certificates and configure Splunk Web and the Splunk daemon to use them. See "Install and configure certificates on the Splunk Enterprise management port" in this topic for the procedure.

Install and configure certificates on the Splunk Enterprise management port

After you obtain the necessary certificates, you can then secure Splunk Enterprise with them. Securing Splunk Enterprise involves making edits to configuration files on the instance to point to the certificate files and reloading the updated configurations so that Splunk Enterprise can use the certificates to secure the connection between it and Splunk Log Observer Connect.

Prerequisites for installing and configuring the certificates on the Splunk Enterprise management port

Before you can install and configure certificates into Splunk Enterprise, you must have the following:

• The certificates and keys you want to use to secure the Splunk Enterprise management port. The certificate files must be in PEM format.
• Administrative access to the Splunk Enterprise system whose management port you want to secure, You can configure Splunk Web to use certificates and keys using configuration files only if you have access to the file system, such as if you are a system administrator. If you don't have this access, obtain it first.

Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual to learn how to edit configuration files.

The following procedure uses $SPLUNK_HOME/etc/auth/sloccerts/ as the directory where the key and certificate are located: $SPLUNK_HOME/etc/auth/sloccerts/myFinalCert.pem

Configure Splunk Enterprise to use the key and certificate files

Perform the following steps to configure the Splunk management port to use the key and certificate files.

Never change or copy the configuration files in the $SPLUNK_HOME/etc/default directory. The files in the default directory must remain intact and in their original location. Instead, make changes to the files in the $SPLUNK_HOME/etc/system/local directory.

1. Using a text editor, open or create a server.conf configuration file in the $SPLUNK_HOME/etc/system/local directory.
2. Under the [general] stanza for this file, add the name of your Splunk Enterprise server to the serverName setting.
3. Under the [sslConfig] stanza, add the serverCert setting to configure the path to myFinalCert.pem certificate file.
   

   You can use an absolute path when you configure these settings by adding a forward slash ( / ) before the path. Non-absolute paths are relative to the Splunk Enterprise installation directory ($SPLUNK_HOME). If you use a non-absolute path, do not add $SPLUNK_HOME to the path.

   Following is an example of a server.conf configuration file that references the path to a certificate:
   

   [general]
   serverName = mysplunkenterpriseserver.mycompany.com
   
   [sslConfig] 
   serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem
   requireClientCert = false
   

4. Save the server.conf file and close it.
5. Open or create a web.conf configuration file in the $SPLUNK_HOME/etc/system/local directory.
6. Under the [settings] stanza for this file, add the serverCert setting to configure the path to myFinalCert.pem certificate file.
   Following is an example of a web.conf configuration file that references the path to a certificate:
   

   [settings]
   enableSplunkWebSSL = true
   serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem
   

7. Save the web.conf file and close it.
8. Restart the splunkd service on your Splunk Enterprise instance.
   

   $SPLUNK_HOME/bin/splunk restart splunkd