Caution

Preview: Splunk Zero Config Auto Instrumentation

Preview features described in this document are provided by Splunk to you “as is” without any warranties, maintenance and support, or service-level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

Zero Configuration Auto Instrumentation for Java Applications on Kubernetes 🔗

Zero Configuration Auto Instrumentation for Java activates automatic instrumentation for Kubernetes Java applications. When you activate automatic instrumentation, you only have to restart any applications that are already running.

Prerequisites 🔗

Automatic instrumentation is only available for applications using supported Java libraries. See Supported libraries and frameworks. If your application isn’t supported, manually instrument your service to generate trace data. See Manually instrument Java applications for Splunk Observability Cloud .
Ensure you are using supported Java and JVM versions.
Your Splunk Observability Cloud realm and access token.
- To get an access token, see Retrieve and manage user API access tokens using Splunk Observability Cloud.
- To find the realm name of your account, open the navigation menu in Observability Cloud. Select Settings, and then select your username. The realm name appears in the Organizations section.

Install the Splunk OpenTelemetry Collector Kubernetes Operator on a compatible version of Kubernetes.

Activate automatic instrumentation of Java applications on Kubernetes 🔗

Before deployment, you can activate automatic instrumentation for a Kubernetes Deployment or pod by adding the otel.splunk.com/inject-java annotation.

When you activate instrumentation, the Collector operator injects the Splunk OTel Java agent into Java applications to capture telemetry data.

To activate automatic instrumentation, add this annotation to the spec for a deployment or pod: otel.splunk.com/inject-java: "true". If you add the annotation to a pod, restarting the pod removes the annotation.

You can also activate automatic instrumentation on a running workload.

Activate or deactivate automatic instrumentation before runtime 🔗

If the deployment is not deployed, add the otel.splunk.com/inject-java annotation to the application deployment YAML file.

For example, given the following deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-java-app
spec:
  template:
    spec:
      containers:
      - name: my-java-app
        image: my-java-app:latest

Activate auto instrumentation by adding otel.splunk.com/inject-java: "true" to the spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-java-app
spec:
  template:
    metadata:
      annotations:
        otel.splunk.com/inject-java: "true"
    spec:
      containers:
      - name: my-java-app
        image: my-java-app:latest

The Collector operator activates automatic instrumentation for any Java applications in the deployment.

To deactivate automatic instrumentation, remove the annotation or set its value to false.

Activate or deactivate automatic instrumentation on a running workload 🔗

If the application is already running, patch the deployment using kubectl patch to activate instrumentation.

Caution

Patching a deployment restarts the pods in the deployment.

Use the following snippet as an example. Replace <my-deployment> with your deployment’s name.

kubectl patch deployment <my-deployment> -p '{"spec": {"template":{"metadata":{"annotations":{"otel.splunk.com/inject-java":"true"}}}} }'

To deactivate automatic instrumentation, run the same command but change the value of the annotation to false:

kubectl patch deployment <my-deployment> -p '{"spec": {"template":{"metadata":{"annotations":{"otel.splunk.com/inject-java":"false"}}}} }'

Check the status of automatic instrumentation 🔗

When you successfully activate instrumentation for a deployment, the metadata for every pod in the deployment includes the annotation otel.splunk.com/injection-status:success.

Use the following command to check for the injection-status annotation. Replace <POD_NAME> with the name of your pod.

kubectl get pod  <POD_NAME> -o yaml | grep inject

The command result is similar to the following:

otel.splunk.com/inject-java: "true"
otel.splunk.com/injection-status: success

If the injection-status annotation is not present or is not set to success, auto instrumentation is not activated. See the troubleshooting section for next steps.

If the injection-status annotation is set to success, you have activated instrumentation correctly. You can Verify that your data is coming into Splunk APM or optionally configure instrumentation settings.

Optionally configure instrumentation 🔗

The default settings for auto instrumentation are sufficient for most cases. You can add advanced configuration like activating custom sampling and including custom data in the reported spans with environment variables and Java system properties.

For example, if you want every span to include the key-value pair build.id=feb2023_v2, set the OTEL_RESOURCE_ATTRIBUTES environment variable.

kubectl set env deployment/<my-deployment> OTEL_RESOURCE_ATTRIBUTES=build.id=feb2023_v2

See Configure the Java agent for Splunk Observability Cloud for the full list of supported environment variables.

Next steps 🔗

After activating automatic instrumentation, Verify that your data is coming into Splunk APM.

Troubleshooting 🔗

If you activate auto instrumentation and you do not see any telemetry data in Observability Cloud APM, try the following steps:

Check the Collector operator logs. Look for the pods in the splunk-otel-operator-system namespace, and then examine their logs:

kubectl get pods  --namespace=splunk-otel-operator-system

NAME                                                      READY   STATUS    RESTARTS   AGE
splunk-otel-agent-7cspj                                   1/1     Running   0          31h
splunk-otel-agent-gkmts                                   1/1     Running   0          31h
splunk-otel-agent-xbnpm                                   1/1     Running   0          31h
splunk-otel-cluster-receiver-8cd9874c8-6jlz6              1/1     Running   0          31h
splunk-otel-operator-controller-manager-8455c8bc7-m8f24   1/1     Running   0          31h

kubectl logs  --namespace=splunk-otel-operator-system  splunk-otel-operator-controller-manager-8455c8bc7-m8f24

Run this command to see the logs for one of the pods:

kubectl logs  --namespace=splunk-otel-operator-system  <pod-name>

You can also follow the steps to troubleshoot the Java agent.

Report an issue 🔗

Before you create an issue or open a support request, try gathering the following information:

What happened and the impact of the issue.
All the steps you’ve followed until the issue appeared.
What was the expected outcome.
Your attempts to solve the issue, including workarounds.
The operating system, runtime or compiler version, libraries, frameworks, and application servers of your environment, including your instrumentation settings.
Debug logs and other logs that might help troubleshoot the issue.

To get help, see Splunk Observability Cloud support.

Related Topics