Docs » Supported integrations in Splunk Observability Cloud » Configure application receivers for orchestration » Kubernetes events

Kubernetes events 🔗

The Splunk Distribution of OpenTelemetry Collector uses the Smart Agent receiver with the kubernetes-events monitor type to listen for Kubernetes events. The integration calls the Kubernetes API running on manager nodes, and sends Kubernetes events into Splunk Observability Cloud as Infrastructure Monitoring events through the OpenTelemetry pipeline.

After it starts, the Kubernetes events monitor type sends all of the events that Kubernetes has that are still persisted, and any new events as they come in. The various agents decide which instance will lead and sends event. If alwaysClusterReporter is set to true, every node emits the same data, and there is no additional querying of the manager node.

This monitor type is available on Kubernetes, Linux, and Windows.

Benefits 🔗

Configure the integration to access these features:

Installation 🔗

Follow these steps to deploy this integration:

  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:

  2. Configure the integration, as described in the Configuration section.

  3. Restart the Splunk Distribution of the OpenTelemetry Collector.

Deploy with Helm 🔗

To activate this monitor with the Helm chart, include this argument with the helm install command:

-set splunkObservability.infrastructureMonitoringEventsEnabled='true'

Deploy without Helm 🔗

To deploy without Helm, include the following in the OTel configuration:

processors:
  resource/add_event_k8s:
    attributes:
      - action: insert
        key: kubernetes_cluster
        value: CHANGEME

receivers:
  smartagent/kubernetes-events:
   type: kubernetes-events
   alwaysClusterReporter: true

service:
  pipelines:
    logs/events:
      exporters:
        - signalfx
      processors:
        - memory_limiter
        - batch
        - resourcedetection
        - resource/add_event_k8s
      receivers:
        - smartagent/kubernetes-events

Configuration 🔗

To use this integration of a Smart Agent monitor with the Collector:

  1. Include the Smart Agent receiver in your configuration file.

  2. Add the monitor type to the Collector configuration, both in the receiver and pipelines sections.

Example 🔗

To activate this integration, add the following to your Collector configuration:

receivers:
   smartagent/kubernetes-events:
   type: kubernetes-events
   ... # Additional config

Next, add the monitor to the service.pipelines.metrics.receivers section of your configuration file:

services:
  logs/events:
    receivers:
      - smartagent/kubernetes-events

Lastly, configure which events to send. You can see the types of events happening in your cluster with the following command:

kubectl get events -o yaml --all-namespaces

To send all events, set the option _sendAllEvents to true in your values.yaml, and remove the whitelistedEvents option.

From the output, combine Reason (Started, Created, Scheduled) and Kind (Pod, ReplicaSet, Deployment…) to select which events to send. - Specify a single reason and involveObjectKind individually for each event rule you want to allow. - Events are placed in the whitelistedEvents configuration option as a list of events you want to send. - Event names will match the reason name.

Configuration settings 🔗

Option

Required

Type

Description

kubernetesAPI

no

object (see below)

Configuration of the Kubernetes API client.

whitelistedEvents

no

list of objects (see below)
A list of event types to send events for. Only events matching

these items will be sent.

alwaysClusterReporter

no

bool
Whether to always send events from this agent instance or to do

leader election to only send from one agent instance. Default is false.

The nested kubernetesAPI config object has the following fields:

Option

Required

Type

Description

authType

no

string
To authenticate to the K8s API server: - none for no

authentication. - tls to use manually specified TLS client certs (not recommended). - serviceAccount to use the standard service account token provided to the agent pod. - kubeConfig to use credentials from ~/.kube/config. - Default is serviceAccount.

skipVerify

no

bool
Whether to skip verifying the TLS certificate from the API

server. Almost never needed. Default is false

clientCertPath

no

string
The path to the TLS client certificate on the pod’s filesystem,

if using tls authentication.

clientKeyPath

no

string
The path to the TLS client key on the pod’s filesystem, if using

tls authentication.

caCertPath

no

string
Path to a CA certificate to use when verifying the API server

TLS certificate. Generally this is provided by K8s alongside the service account token, which will be picked up automatically, so this should rarely be necessary to specify.

The nested whitelistedEvents configuration object has the following fields:

Option

Required

Type

reason

no

string

involvedObjectKind

no

string

Example YAML configuration:

receivers:
   smartagent/kubernetes-events:
     type: kubernetes-events
     whitelistedEvents:
       - reason: Created
         involvedObjectKind: Pod
       - reason: SuccessfulCreate
         involvedObjectKind: ReplicaSet

Troubleshooting 🔗

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Available to prospective customers and free trial users

  • Ask a question and get answers through community support at Splunk Answers .

  • Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

This page was last updated on Sep 18, 2024.