Send events to a DSP data pipeline using a Splunk forwarder

You can forward data to your data pipeline using a Splunk heavy forwarder or a universal forwarder. A forwarder is a Splunk Enterprise instance that forwards data to another Splunk Enterprise instance, such as an indexer or another forwarder, or to a third-party system. In this case, the forwarder is forwarding data to the DSP Forwarder service, which your data pipeline then ingests data from.

The DSP Forwarder service identifies incoming connections from forwarders by the client certificate on the forwarder. Your forwarder must be configured to use a self-signed client certificate. Before using the DSP Forwarder service, you must:

• Create and upload a self-signed certificate to the DSP Forwarder service.
• Configure your forwarder to trust the client certificate.

Download and install a universal forwarder

1. Confirm that your system meets the system requirements.
2. Download the correct forwarder for your operating system and application.
3. See the install the universal forwarder software in the Forwarder manual for instructions on how to install the universal forwarder.

Using a universal forwarder

Universal forwarder events arrive in DSP unparsed and in 64KB chunks. This means that your events could be truncated if they are too long or multiple events could be housed in one event. To create a stream of complete individual events, you must use the Group by and Merge Events functions. After following the steps listed on this topic to set up your universal forwarder, see Create a Splunk Universal Forwarder pipeline for instructions on how to build a DSP pipeline that successfully ingests universal forwarder data.

Generate a client certificate for the DSP Forwarders service

You must secure the connection between the Splunk Data Stream Processor and your Splunk instance.

Prerequisites:

1. Choose a name to identify your Splunk instance. Use this name for the generated key, certificate signing request (CSR), and PEM files.
2. Choose an email address to associate to the certificate. The email address is included in the name of the certificate.

Steps

1. From the command line, generate a private key to sign your certificates.

   openssl genrsa -out my_forwarder.key 2048

2. If you do not have a .rnd file in your home directory, you must use openSSL to generate a new file.

   openssl rand -writerand ~/.rnd

3. Use your private key to generate a CSR file.

   openssl req -new -key "my_forwarder.key" -out "my_forwarder.csr" -subj "/C=US/ST=CA/O=my_organization/CN=my_forwarder/emailAddress=email@example.com"

4. Sign your CSR file with your newly created private key and create a client certificate that expires in 2 years.

   openssl x509 -req -days 730 -in "my_forwarder.csr" -signkey "my_forwarder.key" -out "my_forwarder.pem" -sha256

5. Use the SCloud tool to format your certificate and upload your certificate to your tenant. For more information about SCloud, see SCloud.
   • If you are using SCloud 1.0.0:

     scloud forwarders create-certificate my_forwarder.pem 
     

   • If you are using SCloud 4.0.0:

     scloud forwarders add-certificate --input-datafile my_forwarder.pem

After the key is added to the tenant, the server response shows details about the certificate including the hash, subject, issuer, and validity dates.

By default, you can add up to 100 certificates to your tenant.

Configure your forwarder to use the client certificate

1. On the host that forwards the data that you want to collect, open a shell or command prompt or PowerShell window.
2. Concatenate your private and public keys into one file.

   cat my_forwarder.pem my_forwarder.key > my_forwarder-keys.pem
   

3. Navigate to the configuration directory for the forwarder.

   cd $SPLUNK_HOME/etc/system/local

4. Open outputs.conf in etc/system/local for editing and add the following stanzas. If outputs.conf does not exist, create it. See outputs.conf.spec in the Admin Manual for more information about these settings.

   [tcpout]
   defaultGroup = my_defaultGroup
   
   [tcpout:my_defaultGroup]
   server=<ip-address-of-master-node>:30001
   
   clientCert=/path/to/my_forwarder-keys.pem
   sslVerifyServerCert=false 
   

5. (Optional) If you want to use indexer acknowledgment, set useACK to true in outputs.conf.

   useACK=true
   

If you are sending data from a large number of forwarders and you are experiencing throughput issues, you should increase the number of replicas for the ingest-s2s deployment in your Kubernetes cluster. You can increase the number of replicas by adding more nodes to your cluster. See Install the Data Stream Processor for instructions on how to join a new node to your cluster.

The forwarder starts forwarding data to DSP. You can now use the forwarder as a data source by creating a pipeline that starts with the Forwarders Service source function.

How the Forwarders Service sends data to a DSP pipeline

Data flows from a Splunk forwarder into the Forwarders service which sends the data to DSP. Data from the Forwarders service has the following schema.

Field	Description
body	The event's payload. If you are using a Universal Forwarder, then you need to do additional processing in DSP to break and merge your events properly. See Create a Splunk DSP pipeline that processes universal forwarder data. The default type of body is a union type of all possible types. To pass body as an argument to a downstream function that requires a specific type, you should first cast body to the appropriate more specific type. See cast or ucast for an example.
attributes	Contains metadata fields about your forwarder.
host	Contains information about the host for your forwarder.
source	Contains information about the source for your forwarder.
source_type	Contains information about the source_type for your forwarder.
timestamp	Contains the timestamp of your event. When receiving data from a heavy forwarder, DSP uses the timestamp determined and extracted by the heavy forwarder. When receiving data from a universal forwarder, which does not do any processing of your events, DSP sets the timestamp to the time when the event was ingested.
nanos	Contains nanoseconds beyond the timestamp.
id	Contains a unique ID of the event. If it is not specified, the system generates an ID.

Add permissions to users to send data with the Forwarders Service

By default, tenant administrators have permissions to use the Forwarders service. Perform the following steps to grant permissions to another user in your tenant to send data using the Forwarders service.

1. Ask the user you want to grant permissions to for their token UUID.
   • Ask the user to login to SCloud:

     scloud login

   • Ask the user to run the following SCloud command and send you their UUID:

     scloud identity validate-token

2. Create a new group to give permissions to.

   scloud identity create-group <groupName>

3. Create a new role to give permissions to.

   scloud identity create-role <roleName>
   

4. Add write, read, and delete permissions to the role that you created or to the pre-existing role.
   • To add write permissions, type the following command:

     scloud identity add-role-permission <roleName> default:*:forwarders.certificate.write

   • To add read permissions, type the following command:

     scloud identity add-role-permission <roleName> default:*:forwarders.certificate.read

   • To add delete permissions, type the following command:

     scloud identity add-role-permission <roleName> default:*:forwarders.certificate.delete

5. Add the role to the group that you created.

   scloud identity add-group-role <groupName> <roleName>

6. Add the user to the group.

   scloud identity add-group-member <groupName> <user-UUID>