Docs » Get started with the Splunk Distribution of the OpenTelemetry Collector » Automatic discovery of apps and services » Automatic discovery for Kubernetes » Advanced customization for automatic discovery and instrumtenation in Kubernetes

Advanced customization for automatic discovery and instrumtenation in Kubernetes 🔗

Learn how to customize Splunk automatic discovery and instrumentation for advanced scenarios.

Through advanced customization, you can achieve the following tasks:

Note

For advanced customization options in Linux, see Advanced customization for automatic discovery in Linux.

Change instrumentation version 🔗

By default, the Splunk Distribution of OpenTelemetry Collector uses the latest version of the related instrumentation library. To change the version of your instrumentation library, follow these steps:

  1. Open your values.yaml file.

  2. In the instrumentation language section, change the tag value to match your desired version. The following example changes the Java instrumentation version to v1.27.0.

    clusterName: myCluster
splunkObservability:
  realm: <splunk-realm>
  accessToken: <splunk-access-token>
environment: prd
certmanager:
  enabled: true
operator:
  enabled: true
instrumentation:
  spec:
    java:
      repository: ghcr.io/signalfx/splunk-otel-java/splunk-otel-java
      tag: v1.27.0

  3. Reinstall the Splunk OTel Collector chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.

    helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml

Note

If you don’t see the java field in your values.yaml file, then you have to add the field and the repository value before changing the version. The repository value is always ghcr.io/signalfx/splunk-otel-java/splunk-otel-java.

See the following pages for information about previous versions for each language:

Override default instrumentation settings 🔗

You can override default automatic discovery settings to use features such as AlwaysOn profiling and runtime metrics.

Configure AlwaysOn Profiling 🔗

You can configure AlwaysOn Profiling in Kubernetes by editing the values.yaml file for the Helm Chart.

Follow these steps to activate Profiling for a language:

  1. Open the values.yaml file.

  2. In the instrumentation.spec.<language>.env section, add the SPLUNK_PROFILER_ENABLED="true", SPLUNK_PROFILER_MEMORY_ENABLED="true", and SPLUNK_PROFILER_CALL_STACK_INTERVAL environment variables. For example, the following values.yaml file configures AlwaysOn Profiling to sample call stacks from a 5000 millisecond interval:

    operator:
  enabled: true
instrumentation:
  spec:
    nodejs:
      env:
      # Activates AlwaysOn Profiling for Node.js
      - name: SPLUNK_PROFILER_ENABLED
        value: "true"
      - name: SPLUNK_PROFILER_MEMORY_ENABLED
        value: "true"
      # Samples call stacks from a 5000 millisecond interval.
      # If excluded, samples from a 10000 millisecond interval by default.
      - name: SPLUNK_PROFILER_CALL_STACK_INTERVAL
        value: 5000

  3. Reinstall the Splunk OTel Collector chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.

    helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml

Activate runtime metrics collection (Java and Node.js only) 🔗

You can activate runtime metrics collection for Java and Node.js applications running in your Kubernetes environment.

Follow these steps to activate runtime metrics collection:

  1. Open the values.yaml file.

  2. In the instrumentation.spec.<language>.env section, add the SPLUNK_METRICS_ENABLED=true environment variable. For example, the following values.yaml file activates runtime metrics collection for Java applications:

    operator:
  enabled: true
instrumentation:
  spec:
    java:
      env:
      # Activates runtime metrics collection for Java
      - name: SPLUNK_METRICS_ENABLED
        value: "true"

  3. In the instrumentation.spec.env section, add the following environment variables and values to configure the endpoint to which the Collector sends runtime metrics:

    operator:
  enabled: true
instrumentation:
  spec:
    env:
    - name: SPLUNK_OTEL_AGENT
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: SPLUNK_METRICS_ENDPOINT
      value: http://$(SPLUNK_OTEL_AGENT):9943/v2/datapoint

  4. Reinstall the Splunk OTel Collector chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.

    helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml

Use automatic discovery with gateway mode 🔗

The Splunk OTel Collector chart uses the agent mode by default. Activating gateway mode deploys an instance of the OpenTelemetry Collector in a separate container, and this instance collects data from the entire cluster.

To learn more about the gateway mode, see Data forwarding (gateway) mode.

Follow these steps to send data to a gateway endpoint:

  1. Open the values.yaml file.

  2. Set the instrumentation.spec.exporter.endpoint value to the gateway endpoint. For example:

    clusterName: myCluster
splunkObservability:
  realm: <splunk-realm>
  accessToken: <splunk-access-token>
environment: prd
certmanager:
  enabled: true
operator:
  enabled: true
instrumentation:
  spec:
    exporter:
      endpoint: <gateway-endpoint>

  3. Reinstall the Splunk OTel Collector chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.

    helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml

You can also automatically send data to a deployed gateway endpoint by setting agent.enabled: false and gateway.enabled: true in your values.yaml file. For example:

clusterName: myCluster
splunkObservability:
  realm: <splunk-realm>
  accessToken: <splunk-access-token>
environment: prd
certmanager:
  enabled: true
operator:
  enabled: true

agent:
  enabled: false
gateway:
  enabled: true

Using this configuration, automatic discovery automatically sends data to a running gateway endpoint.

Additional settings 🔗

There are many other settings you can customize in automatic instrumentation.

For a list of settings that you can change for each language, see the following resources:

Language

Resource

Java

Configure the Java agent for Splunk Observability Cloud

Node.js

Configure the Splunk Distribution of OTel JS for Splunk Observability Cloud

For a complete list of settings that you can modify in values.yaml for Kubernetes, see https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml.

This page was last updated on Oct 03, 2024.