Get data into behavioral analytics service

On your heavy forwarder that is already forwarding events to Splunk Enterprise Security (ES) in Splunk Cloud Platform, configure an additional output to also forward events to Splunk Mission Control. From there, the events are ingested by behavioral analytics service.

When your heavy forwarder sends an event, it uses its client certificate to authenticate and authorize the client, determine which tenant the event belongs to, then route the event to the correct tenant.

Perform the following tasks to get data into behavioral analytics service. In the instructions, replace my_forwarder, my_organization, and email@example.com with your own information as appropriate.

1. If you don't already have a heavy forwarder set up in your environment, or if you want to set up a new heavy forwarder for dedicated use with behavioral analytics service, use Splunk Web to set up a Splunk Enterprise instance as a heavy forwarder. See Deploy a heavy forwarder in the Splunk Enterprise Forwarding Data manual for instructions.
2. Generate your client certificate on the heavy forwarder.
3. Upload your client certificate.
4. Configure the heavy forwarder to use SSL.
5. Set up a target Forwarders service.
6. Restart the heavy forwarder and verify the configuration

Generate your client certificate on the heavy forwarder

Perform the following steps to generate your client certificate:

1. On your heavy forwarder, run the following commands to generate your client certificate:

   openssl genrsa -out my_forwarder.key 2048
   openssl req -new -key "my_forwarder.key" -out "my_forwarder.csr" -subj "/C=US/ST=CA/O=my_organization/CN=my_forwarder/emailAddress=email@example.com"
   openssl x509 -req -days 730 -in "my_forwarder.csr" -signkey "my_forwarder.key" -out "my_forwarder.pem" -sha256
   

2. Concatenate your private and public keys into a single file:

   cat my_forwarder.pem my_forwarder.key > my_forwarder-keys.pem

3. (Optional) Remove the my_forwarder.csr file from your system.

Upload your client certificate

You can use the web interface in Splunk Mission Control or the SCloud command line tool to upload your new client certificate.

Use the web interface to upload your client certificate

Perform the following tasks in Splunk Mission Control to upload your client certificate:

1. In Splunk Mission Control, click the more icon ( ) and click Admin Settings.
2. Under Product Settings, click Ingestion.
3. Click Add Certificate.
4. Paste the client certificate that you generated in the Add client certificate text box.
5. Click Submit.

Use the SCloud command line tool to upload your client certificate

Perform the following tasks to use the SCloud command line tool to upload your client certificate:

1. Verify that you have the latest version of the SCloud command line tool. See Get started with SCloud in the Install and administer the Data Stream Processor manual.
2. From the home directory of your heavy forwarder, run the scloud command to log into Splunk Cloud Platform and upload your certificate. You can log in only if you are granted access to a tenant.

   scloud login
   scloud forwarders add-certificate --input-datafile my_forwarder.pem
   

   See Get started with SCloud in the Install and administer the Data Stream Processor manual if you are using an older version of the SCloud command lien tool, as the command might vary between releases. Depending on the version of the SCloud command line tool you are using, you may be prompted to specify a target tenant when uploading the certificate.

Configure the heavy forwarder to use SSL

Configure your heavy forwarder to trust the Splunk Forwarder Service certificate, which is signed by DigiCert.

1. Run the following command to download the DigiCert Global Root CA:

   wget https://cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem -O DigiCertGlobalRootCA.pem

   You can also download the DigiCert Global Root CA directly from the "DigiCert Trusted Root Authority Certificates" page on the DigiCert website.
2. Add the following property and value to the tcpout stanza in the $SPLUNK_FORWARDER_HOME/etc/system/local/outputs.conf file.

   sslVerifyServerCert=true

3. Add the following property and value to the sslConfig stanza in the $SPLUNK_FORWARDER_HOME/etc/system/local/server.conf file.

   sslRootCAPath = /path/to/DigiCertGlobalRootCA.pem

   Replace /path/to with the actual path to your certificate, such as in the following example:

   sslRootCAPath = $SPLUNK_HOME/etc/apps/splunk_ba_forwarding/auth/DigiCertGlobalRootCA.pem

Set up a target Forwarders service

Configure the $SPLUNK_FORWARDER_HOME/etc/system/local/outputs.conf file. The following example shows SSL enabled and a target Forwarders service of forwarders.scs.splunk.com:9997:

[tcpout:splunk_ba]
clientCert = $SPLUNK_HOME/etc/apps/splunk_ba_forwarding/auth/my_forwarder-keys.pem
disabled = False
dropClonedEventsOnQueueFull = 0s 
dropEventsOnQueueFull = 0s 
server = forwarders.scs.splunk.com:9997
sslCommonNameToCheck = *.forwarders.scs.splunk.com
sslRootCAPath = $SPLUNK_HOME/etc/apps/splunk_ba_forwarding/auth/DigiCertGlobalRootCA.pem
sslVerifyServerCert = True
useACK = False

Configure the optional dropClonedEventsOnQueueFull and dropEventsOnQueueFull properties to help prevent blocking downstream queues in case the heavy forwarder is unable to communicate with behavioral analytics service:

• Set dropClonedEventsOnQueueFull to the amount of time in milliseconds to wait before dropping new cloned events when the output queue becomes blocked. You only need to set this property if there are multiple tcpout queues configured on the heavy forwarder.
• Set dropEventsOnQueueFull to the amount of time in milliseconds to wait before dropping new events in case the output queue becomes blocked.

See outputs.conf in the Splunk Enterprise Admin Manual for more information about these properties.

Forward events from your HTTP Event Collector to behavioral analytics service

To forward events from your HTTP Event Collector (HEC) to behavioral analytics service, edit the inputs.conf file on the heavy forwarder and modify the outputgroup property in the HTTP stanza. The following example sends events from the HEC to both splunkcloud and splunkBA:

[http://default]
disabled = 0
outputgroup = splunkcloud,splunkBA

See http: (HTTP Event Collector) in the Splunk Enterprise Admin Manual for more information about HTTP stanza in the inputs.conf file.

Send data to Splunk Cloud Platform and behavioral analytics service

This example shows how to configure a heavy forwarder to send events to multiple tcpout groups. In this example, send events to Splunk Cloud Platform and also a copy of each event to behavioral analytics service.

Following is the stanza in the outputs.conf file on the heavy forwarder that sends events from the heavy forwarder to behavioral analytics service:

[tcpout:splunk_ba]
clientCert = $SPLUNK_HOME/etc/apps/splunk_ba_forwarding/auth/my_forwarder-keys.pem
disabled = False
dropClonedEventsOnQueueFull = 0s # Optional: Helps prevent blocking other output and downstream queues in case of lost connectivity to behavioral analytics service
server = forwarders.scs.splunk.com:9997
sslCommonNameToCheck = *.forwarders.scs.splunk.com
sslVerifyServerCert = True
sslRootCAPath = $SPLUNK_HOME/etc/apps/splunk_ba_forwarding/auth/DigiCertGlobalRootCA.pem
useACK = False

Following is the stanza in the outputs.conf file on the heavy forwarder that sends events from the heavy forwarder to Splunk Cloud Platform:

[tcpout:splunkcloud]
clientCert = $SPLUNK_HOME/etc/apps/100_buttercup_splunkcloud/default/buttercup_server.pem
compressed = false
server = inputs1.buttercup.splunkcloud.com:9997, inputs2.buttercup.splunkcloud.com:9997, inputs3.buttercup.splunkcloud.com:9997, inputs4.buttercup.splunkcloud.com:9997, inputs5.buttercup.splunkcloud.com:9997, inputs6.buttercup.splunkcloud.com:9997, inputs7.buttercup.splunkcloud.com:9997, inputs8.buttercup.splunkcloud.com:9997, inputs9.buttercup.splunkcloud.com:9997, inputs10.buttercup.splunkcloud.com:9997, inputs11.buttercup.splunkcloud.com:9997, inputs12.buttercup.splunkcloud.com:9997, inputs13.buttercup.splunkcloud.com:9997, inputs14.buttercup.splunkcloud.com:9997, inputs15.buttercup.splunkcloud.com:9997
sslCommonNameToCheck = *.buttercup.splunkcloud.com
sslPassword = ...
sslVerifyServerCert = true
useClientSSLCompression = true

Be sure that your sslRootCAPath points to your DigiCert certificate. Since forwarding to Splunk Cloud Platform is also configured, the sslRootCAPath value in the server.conf file takes precedence over the value of sslRootCAPath in outputs.conf. Perform the following steps to work around this issue:

1. On the heavy forwarder, run the following command to add both sslRootCAPath values to the /etc/auth/cacerts.pem file:

   cat etc/apps/100_buttercup_splunkcloud/default/buttercup_cacert.pem > etc/auth/cacert.pem && cat etc/apps/splunk_ba_forwarding/auth/DigiCertGlobalRootCA.pem etc/auth/cacert.pem

2. Update the server.conf file in any app or in the /system/local directory and edit the sslRootCAPath property as follows:

   [sslConfig]
   sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
   

3. On the heavy forwarder, configure the tcpout stanza in the outputs.conf file to send data to both tcpput groups. You can use the defaultGroup to do this, similar to how you configure data to be sent to a single output group:

   [tcpout]
   defaultGroup=splunkcloud,splunk_ba
   

Restart the heavy forwarder and verify the configuration

Run the following commands to restart the heavy forwarder so that your configuration changes take effect. Then, verify the active forwarders:

1. Restart the heavy forwarder:

   cd $SPLUNK_FORWARDER_HOME
   bin/splunk restart
   

2. Check and verify the list of active forwarders:

   cd $SPLUNK_FORWARDER_HOME
   bin/splunk list forward-server
   

   Following is some sample output from this command:

   Active forwards:
       forwarders.scs.splunk.com:9997
   Configured but inactive forwards:
       None