Splunk® Connect for Kafka

Install and Use Splunk Connect for Kafka

Download manual as PDF

Download topic as PDF

System requirements

To install Splunk Connect for Kafka, you must meet the following system requirements:

  • A Kafka Connect environment running Kafka version 1.0.0 or later.
  • Java 8 or later.
  • Splunk platform environment of version 6.5 or later.
  • Configured and valid HTTP Event Collector (HEC) tokens.

If you are using Splunk Cloud, use the Splunk Support Portal to request that Splunk Connect for Kafka be installed on your deployment. Splunk Support will set up and provide a URL for your HTTP Event Collector endpoint. If you are ingesting Kinesis Firehose events, you can reuse the HTTP Event Collector (HEC) endpoint setting you configured for the Splunk Add-on for Amazon Kinesis Firehose.

Architecture requirements

Splunk Connect for Kafka supports two types of architectures:

  • Directly inject data to a Splunk platform indexer cluster. For example:
A Kafka Connect Cluster (in containers or virtual machines or physical machines) -> Splunk Indexer Cluster (HEC)
  • Set up a heavy forwarder layer in front of a Splunk platform indexer cluster to offload the data injection load to your Splunk platform indexer cluster. Setting up a heavy forwarder layer can help distribute computational resources across your Splunk platform deployment. For example:
A Kafka Connect Cluster (in containers, virtual machines, or physical machines) -> Heavy Forwarders (HEC) -> Splunk Indexer Cluster

Optionally, the Splunk Connect for Kafka can use its internal load balancing to communicate to HEC ports on the indexers directly. See the parameter splunk.hec.uri in the Parameters topic of this manual to learn more.

See the Configuration examples topic of this manual to see examples of load balancing with a list of HEC enabled endpoints, and load balancing with a preconfigured load balancer.

HTTP Event Collector (HEC) requirements

  • HEC token settings must be the same on all Splunk platform data injection nodes in your environment, including indexers and heavy forwarders.
  • (Optional) When creating a HEC token, enable indexer acknowledgment in order to prevent potential data loss.
  • Enable HEC token acknowledgements in order to avoid data loss. This is a best practice.

If indexer acknowledgment is enabled, set ackIdleCleanup to true in inputs.conf

See Set up and use HTTP Event Collector in Splunk Web in the Splunk Enterprise manual for more information.
See the Indexer acknowledgment section of the Splunk developer portal for more information.

PREVIOUS
Plan your deployment
  NEXT
Install Splunk Connect for Kafka

This documentation applies to the following versions of Splunk® Connect for Kafka: 1.1.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters