Configuration examples
Depending on your deployment, use the following configuration examples to configure your Splunk Connect for Kafka deployment.
If raw events need to go through Splunk's index time extraction, you will need to use the HEC /raw event endpoint. When using the /raw HEC endpoint and when your raw data does not contain a timestamp or contains multiple timestamps or carriage returns, you must configure the splunk.hec.raw.line.breaker
and setup a corresponding props.conf
inside your Splunk platform to honor this line breaker setting. This will assist Splunk to do event breaking. For example, in Connection configuration, set "splunk.hec.raw.line.breaker":"####"
for sourcetype "s1".
In props.conf
, you can set up the line breaker as follows:
[s1] # sourcetype name LINE_BREAKER = (####) SHOULD_LINEMERGE = false
The auto-assigned timestamp will work for all deployments that use the /event HEC endpoint.
Splunk indexing with acknowledgment
Using HEC /raw endpoint
curl <KAFKA_CONNECT_HOST>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534", "splunk.hec.ack.enabled : "true", "splunk.hec.ack.poll.interval" : "20", "splunk.hec.ack.poll.threads" : "2", "splunk.hec.event.timeout" : "120", "splunk.hec.raw" : "true", "splunk.hec.raw.line.breaker" : "####" } }'
Using HEC /event endpoint
curl <KAFKA_CONNECT_REST_ENDPOINT>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534", "splunk.hec.ack.enabled : "true", "splunk.hec.ack.poll.interval" : "20", "splunk.hec.ack.poll.threads" : "2", "splunk.hec.event.timeout" : "120", "splunk.hec.raw" : "false", "splunk.hec.json.event.enrichment" : "org=fin,bu=south-east-us", "splunk.hec.track.data" : "true" } }'
Splunk indexing without acknowledgment
Using HEC /raw endpoint
curl <KAFKA_CONNECT_REST_ENDPOINT>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534", "splunk.hec.ack.enabled : "false", "splunk.hec.raw" : "true", "splunk.hec.raw.line.breaker" : "####" } }'
Using HEC /event endpoint
curl <KAFKA_CONNECT_REST_ENDPOINT>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534", "splunk.hec.ack.enabled : "false", "splunk.hec.raw" : "false", "splunk.hec.json.event.enrichment" : "org=fin,bu=south-east-us", "splunk.hec.track.data" : "true" } }'
Example of a connector to send collectd metrics to a Splunk metrics index
The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual.
curl <hostname>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.sourcetypes": "collectd_http", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534", "splunk.hec.ack.enabled : "true", "splunk.hec.ack.poll.interval" : "20", "splunk.hec.ack.poll.threads" : "2", "splunk.hec.event.timeout" : "120", "splunk.hec.raw" : "true", "splunk.hec.raw.line.breaker" : "####" } }'
Example of a connector with 10 topics and 10 parallelized tasks
Use the following command to create a connector called splunk-prod-financial
for 10 topics and 10 parallelized tasks. The connector will use the HEC /event endpoint with acknowledgments enabled. The data is injected into a three-server Splunk platform indexer cluster.
curl <KAFKA_CONNECT_REST_ENDPOINT>:8083/connectors -X POST -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "10", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534" } }'
Example of a connector with 20 parallelized tasks
Use the following command to update the connector to use 20 parallelized tasks.
curl <KAFKA_CONNECT_REST_ENDPOINT>:8083/connectors/splunk-prod-financial/config -X PUT -H "Content-Type: application/json" -d'{ "name": "splunk-prod-financial", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "20", "topics": "t1,t2,t3,t4,t5,t6,t7,t8,t9,t10", "splunk.hec.uri": "https://idx1:8088,https://idx2:8088,https://idx3:8088", "splunk.hec.token": "1B901D2B-576D-40CD-AF1E-98141B499534" } }'
Configure Splunk Connect for Kafka | Parameters |
This documentation applies to the following versions of Splunk® Connect for Kafka: 1.0.0
Feedback submitted, thanks!