Kafka metrics receiver ๐
The Kafka metrics receiver collects Kafka metrics (such as brokers, topics, partitions, and consumer groups) from a Kafka server and converts them to OTLP format. The supported pipeline type is metrics
. See Process your data with pipelines for more information.
Note
Out-of-the-box dashboards and navigators arenโt supported for the Kafka metrics receiver yet, but are planned for a future release.
Get started ๐
Follow these steps to configure and activate the component:
Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:
Configure the receiver as described in the next section.
Restart the Collector.
Sample configuration ๐
To activate the Kafka metrics receiver, add kafkametrics
to the receivers
section of your configuration file, as shown in the following example:
receivers:
kafkametrics:
protocol_version: 2.0.0
scrapers:
- brokers
- topics
- consumers
To complete the configuration, include the receiver in the metrics
pipeline of the service
section of your configuration file. For example:
service:
pipelines:
metrics:
receivers: [kafkametrics]
Configuration options ๐
These settings are required:
protocol_version
. No default. The Kafka protocol version, for example2.0.0
.scrapers
. No default. Any combination of the following scrapers can be enabled:topics
consumers
brokers
The following settings are optional:
brokers
.localhost:9092
by default. The list of brokers to read from.resolve_canonical_bootstrap_servers_only
.false
by default. Determines whether to resolve then reverse-lookup broker IPs during startup.topic_match
.^[^_].*$)
by default. Regex pattern of topics to filter on metrics collection. The default filter excludes internal topics, which start with_
.group_match
..*)
by default. Regex pattern of consumer groups to filter on for metrics.client_id
.otel-metrics-receiver
by default. Consumer client Id.collection_interval
.1m
by default. Frequency of metric collection/scraping.initial_delay
.1s
by default. Determines how long this receiver waits before starting.auth
. No default. Use one of the following:plain_text
. It has the following fields:username
. The username to use.password
. The password to use.
tls
. It has the following fields:ca_file
. Use only ifinsecure
is set tofalse
. Path to the CA cert. For a client it verifies the server certificate.cert_file
. Use only ifinsecure
is set tofalse
. Path to the TLS cert to use for TLS required connections.key_file
. Use only ifinsecure
is set tofalse
. Path to the TLS key to use for TLS required connections.insecure
.false
by default. Disables the verification of the serverโs certificate chain and host name,InsecureSkipVerify
in the tls configuration.server_name_override
. Indicates the name of the server requested by the client in order to support virtual hosting.
kerberos
. It has the following fields:service_name
. Kerberos service name.realm
. Kerberos realm.use_keytab
. Iftrue
, the keytab is used instead of the password.username
. The Kerberos username used to authenticate with KDC.password
. The Kerberos password used to authenticate with KDC.config_file
. Path to Kerberos configuration, for example/etc/krb5.conf
.keytab_file
. Path to the keytab file, for example/etc/security/kafka.keytab
.disable_fast_negotiation
.false
by default. Disables the PA-FX-FAST negotiation (Pre-Authentication Framework - Fast). Some common Kerberos implementations do not support PA-FX-FAST negotiation.
Configuration example: Set authentication and the collection interval to 5 seconds ๐
This example sets the collection interval for all scrapers to 5 seconds and configures TLS authentication:
receivers:
kafkametrics:
brokers: 10.10.10.10:9092
protocol_version: 2.0.0
scrapers:
- brokers
- topics
- consumers
auth:
tls:
ca_file: ca.pem
cert_file: cert.pem
key_file: key.pem
collection_interval: 5s
Settings ๐
The following table shows the configuration options for the Kafka metrics receiver:
Metrics ๐
The following metrics, resource attributes, and attributes are available.
Activate or deactivate specific metrics ๐
You can activate or deactivate specific metrics by setting the enabled
field in the metrics
section for each metric. For example:
receivers:
samplereceiver:
metrics:
metric-one:
enabled: true
metric-two:
enabled: false
The following is an example of host metrics receiver configuration with activated metrics:
receivers:
hostmetrics:
scrapers:
process:
metrics:
process.cpu.utilization:
enabled: true
Note
Deactivated metrics arenโt sent to Splunk Observability Cloud.
Billing ๐
If youโre in a MTS-based subscription, all metrics count towards metrics usage.
If youโre in a host-based plan, metrics listed as active (Active: Yes) on this document are considered default and are included free of charge.
Learn more at Infrastructure Monitoring subscription usage (Host and metric plans).
Troubleshooting ๐
If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.
Available to Splunk Observability Cloud customers
Submit a case in the Splunk Support Portal .
Contact Splunk Support .
Available to prospective customers and free trial users
Ask a question and get answers through community support at Splunk Answers .
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.