Install Splunk Connect for Kafka

The below steps have been tested on both Apache Kafka and Confluent platform deployments.

To install Splunk Connect for Kafka, perform the following steps:

1. Download Splunk Connect for Kafka from Splunkbase.
2. Start your Apache Kafka cluster, and confirm it is running.
3. (Optional) Create a directory to store your Kafka Connect connectors.
4. Copy the downloaded Splunk Connect for Kafka file onto every host in the directory that contains your Kafka Connect connectors.
5. Navigate to your /$KAFKA_CONNECT_HOME/config/ directory, and create a properties file called connect-distributed.properties.
6. Modify the connect-distributed.properties file to include the below information:

   #These settings may already be configured if you have deployed a connector in your Kafka Connect Environment
   bootstrap.servers=<BOOTSTRAP_SERVERS>
   plugin.path=<PLUGIN_PATH>
   
   #Required configurations for Splunk Connect for Kafka
   key.converter=org.apache.kafka.connect.storage.StringConverter
   value.converter=org.apache.kafka.connect.storage.StringConverter
   key.converter.schemas.enable=false
   value.converter.schemas.enable=false
   internal.key.converter=org.apache.kafka.connect.json.JsonConverter
   internal.value.converter=org.apache.kafka.connect.json.JsonConverter
   internal.key.converter.schemas.enable=false
   internal.value.converter.schemas.enable=false
   offset.flush.interval.ms=10000
   
   #Recommended configurations for Splunk Connect for Kafka
   group.id=kafka-connect-splunk-hec-sink
   

7. Modify <BOOTSTRAP_SERVERS> to point to one of your Kafka brokers. For example, localhost:9092.
   If you have multiple brokers, you can comma separate them in the config. For example, broker1:9092,broker2.9092.

   bootstrap.servers=<BOOTSTRAP_SERVER1>,<BOOTSTRAP_SERVER2>

8. Modify <PLUGIN_PATH> to point to the top-level directory where you are storing your connectors. For example, /opt/connectors.

   If you are running Kafka version 0.10.x, PLUGIN_PATH is not a valid configuration property. To make the connector visible to Kafka Connect, the connectors directory must be added to the classpath. For example, export CLASSPATH=/opt/connectors/*.

9. Save your changes.
10. Run $KAFKA_CONNECT_HOME/bin/connect-distributed.sh $KAFKA_CONNECT_HOME/config/connect-distributed.properties to start Kafka Connect or restart Kafka Connect.

    $KAFKA_CONNECT_HOME is the home directory where your Kafka Connect deployment is located. This could be the same as your Kafka home directory if you are running on a shared system.

11. Validate your connector deployment by running the following command:
    curl http://<KAFKA_CONNECT_HOST>:8083/connector-plugins
    The response should contain an entry named {"class":"com.splunk.kafka.connect.SplunkSinkConnector","type":"sink","version":"v1.0.0”}.