Docs » Configure application receivers » Configure application receivers for orchestration » Kubernetes events

Kubernetes events 🔗

Description 🔗

The Splunk Distribution of OpenTelemetry Collector provides the kubernetes-events monitor type by using the Splunk Observability Cloud Smart Agent Receiver.

This monitor type listens for Kubernetes events by calling the K8s API running on manager nodes, and sends Kubernetes events into Splunk Observability Cloud as Infrastructure Monitoring events.

Upon startup, the Kubernetes events monitor type sends all of the events that K8s has that are still persisted and then send any new events as they come in. The various agents perform leader election amongst themselves to decide which instance will send events, unless the alwaysClusterReporter config option is set to true.

When alwaysClusterReporter is set to true, every node, with the configuration, will emit the same metrics. There is no additional querying of the manager node. When enabled each agent on every node of the cluster fetches events from the k8s API. Which can bring down k8s api = manager nodes.

Note 🔗

Larger clusters might encounter instability when setting this configuration across a large number of nodes. Enable with caution.


This monitor type is available on Kubernetes, Linux, and Windows.

Benefits 🔗

After you’ve configured the integration, you can:

  • View metrics using the built-in dashboard. For information about dashboards, see View dashboards in Observability Cloud.

  • View a data-driven visualization of the physical servers, virtual machines, AWS instances, and other resources in your environment that are visible to Infrastructure Monitoring. For information about navigators, see Splunk Infrastructure Monitoring navigators.

  • Access Metric Finder and search for metrics sent by the monitor. For information about Metric Finder, see Use the Metric Finder.

Installation 🔗

Follow these steps to deploy the integration:

  1. Deploy the Splunk Distribution of OpenTelemetry Collector to your host or container platform:

  2. Configure the monitor, as described in the next section.

  3. Restart the Splunk Distribution of OpenTelemetry Collector.

Configuration 🔗

This monitor is available in the Smart Agent Receiver, which is part of the Splunk Distribution of OpenTelemetry Collector. The Smart Agent Receiver lets you use existing Smart Agent monitors as OpenTelemetry Collector metric receivers.

Using this monitor assumes that you have a configured environment with a functional Smart Agent release bundle on your system, which is already provided for x86_64/amd64 Splunk Distribution of OpenTelemetry Collector installation paths.

To activate this monitor in the Splunk Distribution of OpenTelemetry Collector, add the following to your configuration file:

receivers:
   smartagent/kubernetes-events:
   type: kubernetes-events
   ... # Additional config

To use this monitor type, configure which events to send. You can see the types of events happening in your cluster with the following command:

kubectl get events -o yaml --all-namespaces

From the output, you can select which events to send by the Reason (Started, Created, Scheduled) and Kind (Pod, ReplicaSet, Deployment…) combinations. These events need to be specified individually with a single reason and involveObjectKind for each event rule you want to allow and are placed in the whitelistedEvents configuration option as a list of events you want to send.

Note Event names will match the reason name

Example YAML configuration:

receivers:
   smartagent/kubernetes-events:
     type: kubernetes-events
     whitelistedEvents:
       - reason: Created
         involvedObjectKind: Pod
       - reason: SuccessfulCreate
         involvedObjectKind: ReplicaSet

To complete this monitor type activation, you must also include it in a metrics pipeline. To do this, add the monitor type to the service > pipelines > metrics > receivers section of your configuration file. For example:

service:
   pipelines:
     metrics:
       receivers: [smartagent/kubernetes-events]

Configuration settings 🔗

Config option Required Type Description
kubernetesAPI no object (see below) Configuration of the Kubernetes API client.
whitelistedEvents no list of objects (see below) A list of event types to send events for. Only events matching these items will be sent.
alwaysClusterReporter no bool Whether to always send events from this agent instance or to do leader election to only send from one agent instance. Default is false.

The nested kubernetesAPI config object has the following fields:

Config option Required Type Description
authType no string To authenticate to the K8s API server:
- none for no authentication.
- tls to use manually specified TLS client certs (not recommended).
- serviceAccount to use the standard service account token provided to the agent pod.
- kubeConfig to use credentials from ~/.kube/config.
- Default is serviceAccount.
skipVerify no bool Whether to skip verifying the TLS certificate from the API server. Almost never needed. Default is false
clientCertPath no string The path to the TLS client certificate on the pod's filesystem, if using tls authentication.
clientKeyPath no string The path to the TLS client key on the pod's filesystem, if using tls authentication.
caCertPath no string Path to a CA certificate to use when verifying the API server's TLS certificate. Generally this is provided by K8s alongside the service account token, which will be picked up automatically, so this should rarely be necessary to specify.

The nested whitelistedEvents configuration object has the following fields:

Config option Required Type Description
reason no string
involvedObjectKind no string

Troubleshooting 🔗

If you are not able to see your data in Splunk Observability Cloud:

  • Ask questions and get answers through community support at Splunk Answers.

  • If you have a support contract, file a case using the Splunk Support Portal. See Support and Services.

  • To get professional help with optimizing your Splunk software investment, see Splunk Services.