Docs » Get started with the Splunk Distribution of the OpenTelemetry Collector » Splunk OpenTelemetry Zero Configuration Auto Instrumentation » Splunk OpenTelemetry Zero Configuration Auto Instrumentation for Java » Zero Configuration Automatic Instrumentation for Kubernetes Java applications

Zero Configuration Automatic Instrumentation for Kubernetes Java applications πŸ”—

Use the OTel Collector with the Operator in a Kubernetes environment to automatically instrument your Java applications. By using zero configuration automatic instrumentation, you can quickly send Java application data to Splunk APM without configuring the OpenTelemetry Collector or changing your application code.

To install zero configuration automatic instrumentation for Java, complete the following steps:

  1. Deploy the Helm Chart with the Kubernetes Operator

  2. Verify all the OpenTelemetry resources are deployed successfully

  3. Set annotations to instrument Java applications

  4. View results at Splunk Observability APM

Requirements πŸ”—

Zero Config Auto Instrumentation for Java requires the following components:

Deploy the Helm Chart with the Kubernetes Operator πŸ”—

To deploy the Helm Chart, create a file called values.yaml. In this file, you can define the settings to activate or deactivate when installing the OpenTelemetry Collector with the Helm Chart.

Populate values.yaml with the following fields and values:

clusterName: <your_cluster_name>

# Your Splunk Observability Cloud realm and access token
splunkObservability:
  realm: <splunk_realm>
  accessToken: <splunk_access_token>

# Activates the OpenTelemetry Kubernetes Operator
operator:
  enabled: true

You might need to populate the file with additional values depending on your environment. See Add certificates and Set the deployment environment for more information.

Add certificates πŸ”—

The Operator requires certain TLS certificates to work. Use the following command to check whether a certification manager is available:

# Check if cert-manager is already installed, don't deploy a second cert-manager.
kubectl get pods -l app=cert-manager --all-namespaces

If a certification manager isn’t available in the cluster, then you’ll need to add certmanager.enabled=true to your values.yaml file. For example:

clusterName: my-cluster

splunkObservability:
  realm: <splunk_realm>
  accessToken: <splunk_access_token>

certmanager:
  enabled: true
operator:
  enabled: true

Set the deployment environment πŸ”—

To properly ingest trace telemetry data, the attribute deployment.environment must be onboard the exported traces. The following table demonstrates the different methods for setting this attribute:

Method

Scope

Implementation

Through the values.yaml file environment configuration

Applies the attribute to all telemetry data (metrics, logs, traces) exported through the collector.

The chart will set an attribute processor to add deployment.environment=prd to all telemetry data processed by the collector.

Through the values.yaml file and operator.instrumentation.spec.env or operator.instrumentation.spec.{instrumentation_library}.env configuration

Allows you to set deployment.environment either for all auto-instrumented applications collectively or per auto-instrumentation language.

Add the OTEL_RESOURCE_ATTRIBUTES environment variable, setting its value to deployment.environment=prd.

Through your Kubernetes application deployment, daemonset, or pod specification

Allows you to set deployment.environment at the level of individual deployments, daemonsets, or pods.

Employ the OTEL_RESOURCE_ATTRIBUTES environment variable, assigning the value deployment.environment=prd.

The following examples show how to set the attribute using each method:

Set the environment option in the values.yaml file. This adds the deployment.environment attribute to all telemetry data the Collector receives, including data from automatically-instrumented pods.

  clusterName: my-cluster

  splunkObservability:
    realm: <splunk_realm>
    accessToken: <splunk_access_token>

  environment: prd

  certmanager:
    enabled: true
  operator:
    enabled: true

Deploy the Helm Chart πŸ”—

After configuring values.yaml, use the following command to deploy the Helm Chart:

helm install splunk-otel-collector -f ./values.yaml

Verify all the OpenTelemetry resources are deployed successfully πŸ”—

Resources include the Collector, the Operator, webhook, and instrumentation.

Run the following commands to verify the resources are deployed correctly:

kubectl  get pods -n monitoring
# NAME                                                          READY
# NAMESPACE     NAME                                                            READY   STATUS
# monitoring    splunk-otel-collector-agent-lfthw                               2/2     Running
# monitoring    splunk-otel-collector-cert-manager-6b9fb8b95f-2lmv4             1/1     Running
# monitoring    splunk-otel-collector-cert-manager-cainjector-6d65b6d4c-khcrc   1/1     Running
# monitoring    splunk-otel-collector-cert-manager-webhook-87b7ffffc-xp4sr      1/1     Running
# monitoring    splunk-otel-collector-k8s-cluster-receiver-856f5fbcf9-pqkwg     1/1     Running
# monitoring    splunk-otel-collector-opentelemetry-operator-56c4ddb4db-zcjgh   2/2     Running

kubectl get mutatingwebhookconfiguration.admissionregistration.k8s.io -n monitoring
# NAME                                      WEBHOOKS   AGE
# splunk-otel-collector-cert-manager-webhook              1          14m
# splunk-otel-collector-opentelemetry-operator-mutation   3          14m

kubectl get otelinst -n <target_application_namespace>
# NAME                          AGE   ENDPOINT
# splunk-instrumentation        3m   http://$(SPLUNK_OTEL_AGENT):4317

Set annotations to instrument Java applications πŸ”—

You can activate auto instrumentation for Java applications before runtime.

If the related Kubernetes object (deployment, daemonset, or pod) is not deployed, add the instrumentation.opentelemetry.io/inject-java annotation to the application object YAML.

For example, given the following deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-java-app
  namespace: monitoring
spec:
  template:
    spec:
      containers:
      - name: my-java-app
        image: my-java-app:latest

Activate auto instrumentation by adding instrumentation.opentelemetry.io/inject-java: "true" to the spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-java-app
  namespace: monitoring
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: my-java-app
        image: my-java-app:latest

To deactivate automatic instrumentation, remove the annotation. The following command removes the annotation for automatic instrumentation, deactivating it:

kubectl patch deployment <my-deployment> -n <my-namespace> --type=json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-java"}]'

Verify instrumentation πŸ”—

To verify that the instrumentation was successful, run the following command on an individual pod:

kubectl describe pod <application_pod_name> -n <namespace>

The instrumented pod contains an initContainer named opentelemetry-auto-instrumentation and the target application container should have several OTEL_* environment variables similar to those in the following demo output:

# Name:             opentelemetry-demo-frontend-57488c7b9c-4qbfb
# Namespace:        otel-demo
# Annotations:      instrumentation.opentelemetry.io/inject-java: true
# Status:           Running
# Init Containers:
#   opentelemetry-auto-instrumentation:
#     Command:
#       cp
#       -a
#       /autoinstrumentation/.
#       /otel-auto-instrumentation/
#     State:          Terminated
#       Reason:       Completed
#       Exit Code:    0
# Containers:
#   frontend:
#     State:          Running
#     Ready:          True
#     Environment:
#       FRONTEND_PORT:                              8080
#       FRONTEND_ADDR:                              :8080
#       AD_SERVICE_ADDR:                            opentelemetry-demo-adservice:8080
#       CART_SERVICE_ADDR:                          opentelemetry-demo-cartservice:8080
#       CHECKOUT_SERVICE_ADDR:                      opentelemetry-demo-checkoutservice:8080
#       CURRENCY_SERVICE_ADDR:                      opentelemetry-demo-currencyservice:8080
#       PRODUCT_CATALOG_SERVICE_ADDR:               opentelemetry-demo-productcatalogservice:8080
#       RECOMMENDATION_SERVICE_ADDR:                opentelemetry-demo-recommendationservice:8080
#       SHIPPING_SERVICE_ADDR:                      opentelemetry-demo-shippingservice:8080
#       WEB_OTEL_SERVICE_NAME:                      frontend-web
#       PUBLIC_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:  http://localhost:8080/otlp-http/v1/traces
#       NODE_OPTIONS:                                --require /otel-auto-instrumentation/autoinstrumentation.java
#       SPLUNK_OTEL_AGENT:                           (v1:status.hostIP)
#       OTEL_SERVICE_NAME:                          opentelemetry-demo-frontend
#       OTEL_EXPORTER_OTLP_ENDPOINT:                http://$(SPLUNK_OTEL_AGENT):4317
#       OTEL_RESOURCE_ATTRIBUTES_POD_NAME:          opentelemetry-demo-frontend-57488c7b9c-4qbfb (v1:metadata.name)
#       OTEL_RESOURCE_ATTRIBUTES_NODE_NAME:          (v1:spec.nodeName)
#       OTEL_PROPAGATORS:                           tracecontext,baggage,b3
#       OTEL_RESOURCE_ATTRIBUTES:                   splunk.zc.method=autoinstrumentation-java:0.41.1,k8s.container.name=frontend,k8s.deployment.name=opentelemetry-demo-frontend,k8s.namespace.name=otel-demo,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=opentelemetry-demo-frontend-57488c7b9c,service.version=1.5.0-frontend
#     Mounts:
#       /otel-auto-instrumentation from opentelemetry-auto-instrumentation (rw)
# Volumes:
#   opentelemetry-auto-instrumentation:
#     Type:        EmptyDir (a temporary directory that shares a pod's lifetime)

View results at Splunk Observability APM πŸ”—

Allow the Operator to do the work. The Operator intercepts and alters the Kubernetes API requests to create and update annotated pods, the internal pod application containers are instrumented, and trace and metrics data populates the APM dashboard.

(Optional) Configure the instrumentation πŸ”—

You can configure the Splunk Distribution of OpenTelemetry Java to suit your instrumentation needs. In most cases, modifying the basic configuration is enough to get started.

You can add advanced configuration like activating custom sampling and including custom data in the reported spans with environment variables and Java system properties. To do so, use the values.yaml file and operator.instrumentation.sampler configuration. For more information, see the documentation in GitHub and example in GitHub .

You can also use the methods shown in Set the deployment environment to configure your instrumentation with the OTEL_RESOURCE_ATTRIBUTES environment variable and other environment variables. For example, if you want every span to include the key-value pair build.id=feb2023_v2, set the OTEL_RESOURCE_ATTRIBUTES environment variable:

kubectl set env deployment/<my-deployment> OTEL_RESOURCE_ATTRIBUTES=build.id=feb2023_v2

See Configure the Java agent for Splunk Observability Cloud for the full list of supported environment variables.

Troubleshooting πŸ”—

If you’re having trouble setting up auto instrumentation, see the following troubleshooting guidelines.

Check the logs for failures πŸ”—

Examine logs to make sure that the operator and cert manager are working.

Application

kubectl command

Operator

kubectl logs -l app.kubernetes.io/name=operator

Cert manager

  • kubectl logs -l app=certmanager

  • kubectl logs -l app=cainjector

  • kubectl logs -l app=webhook

Resolve certificate manager issues πŸ”—

A hanging operator can indicate issues with the certificate manager.

  • Check the logs of your cert-manager pods.

  • Restart the cert-manager pods.

  • Ensure that your cluster has only one instance of cert-manager. This includes certmanager, certmanager-cainjector, and certmanager-webhook.

See the official cert manager troubleshooting guide for more information: https://cert-manager.io/docs/troubleshooting/.

Validate certificates πŸ”—

Ensure that certificates are available for use. Use the following command to search for certificates:

kubectl get certificates
# NAME                                          READY   SECRET                                                           AGE
# splunk-otel-collector-operator-serving-cert   True    splunk-otel-collector-operator-controller-manager-service-cert   5m

Learn more πŸ”—