Advanced customization for automatic discovery in Kubernetes π
Learn how to customize Splunk automatic discovery and configuration for advanced scenarios.
Through advanced customization, you can achieve the following tasks:
Note
For advanced customization options in Linux, see Advanced customization for automatic discovery in Linux.
Change instrumentation version π
By default, the Splunk Distribution of OpenTelemetry Collector uses the latest version of the related instrumentation library. To change the version of your instrumentation library, follow these steps:
Open your
values.yaml
file.In the instrumentation language section, change the
tag
value to match your desired version. The following example changes the Java instrumentation version tov1.27.0
.clusterName: myCluster splunkObservability: realm: <splunk-realm> accessToken: <splunk-access-token> environment: prd certmanager: enabled: true operator: enabled: true instrumentation: spec: java: repository: ghcr.io/signalfx/splunk-otel-java/splunk-otel-java tag: v1.27.0
Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
Note
If you donβt see the java
field in your values.yaml
file, then you have to add the field and the repository
value before changing the version. The repository value is always ghcr.io/signalfx/splunk-otel-java/splunk-otel-java
.
See the following pages for information about previous versions for each language:
Override default instrumentation settings π
You can override default automatic discovery settings to use features such as AlwaysOn profiling and runtime metrics.
Configure AlwaysOn Profiling π
You can configure AlwaysOn Profiling in Kubernetes by editing the values.yaml file for the Helm Chart.
Follow these steps to activate Profiling for a language:
Open the values.yaml file.
In the
instrumentation.spec.<language>.env
section, add theSPLUNK_PROFILER_ENABLED="true"
,SPLUNK_PROFILER_MEMORY_ENABLED="true"
, andSPLUNK_PROFILER_CALL_STACK_INTERVAL
environment variables. For example, the following values.yaml file configures AlwaysOn Profiling to sample call stacks from a 5000 millisecond interval:operator: enabled: true instrumentation: spec: nodejs: env: # Activates AlwaysOn Profiling for Node.js - name: SPLUNK_PROFILER_ENABLED value: "true" - name: SPLUNK_PROFILER_MEMORY_ENABLED value: "true" # Samples call stacks from a 5000 millisecond interval. # If excluded, samples from a 10000 millisecond interval by default. - name: SPLUNK_PROFILER_CALL_STACK_INTERVAL value: 5000
Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
Activate runtime metrics collection (Java and Node.js only) π
You can activate runtime metrics collection for Java and Node.js applications running in your Kubernetes environment. To learn more about runtime metrics collection, see :
Follow these steps to activate runtime metrics collection:
Open the values.yaml file.
In the
instrumentation.spec.<language>.env
section, add theSPLUNK_METRICS_ENABLED=true
environment variable. For example, the following values.yaml file activates runtime metrics collection for Java applications:operator: enabled: true instrumentation: spec: java: env: # Activates runtime metrics collection for Java - name: SPLUNK_METRICS_ENABLED value: "true"
In the
instrumentation.spec.env
section, add the following environment variables and values to configure the endpoint to which the Collector sends runtime metrics:operator: enabled: true instrumentation: spec: env: - name: SPLUNK_OTEL_AGENT valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SPLUNK_METRICS_ENDPOINT value: http://$(SPLUNK_OTEL_AGENT):9943/v2/datapoint
Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
Use automatic discovery with gateway mode π
The Splunk OTel Collector Chart uses the agent mode by default. Activating gateway mode deploys an instance of the OpenTelemetry Collector in a separate container, and this instance collects data from the entire cluster.
To learn more about the gateway mode, see Data forwarding (gateway) mode.
Follow these steps to send data to a gateway endpoint:
Open the values.yaml file.
Set the
instrumentation.spec.exporter.endpoint
value to the gateway endpoint. For example:clusterName: myCluster splunkObservability: realm: <splunk-realm> accessToken: <splunk-access-token> environment: prd certmanager: enabled: true operator: enabled: true instrumentation: spec: exporter: endpoint: <gateway-endpoint>
Reinstall the Splunk OTel Collector Chart with the following command. Replace <CURRENT_VERSION> with the current version of your splunk-otel-collector-chart.
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --version <CURRENT_VERSION> -f values.yaml
You can also automatically send data to a deployed gateway endpoint by setting agent.enabled: false
and gateway.enabled: true
in your values.yaml file. For example:
clusterName: myCluster
splunkObservability:
realm: <splunk-realm>
accessToken: <splunk-access-token>
environment: prd
certmanager:
enabled: true
operator:
enabled: true
agent:
enabled: false
gateway:
enabled: true
Using this configuration, automatic discovery automatically sends data to a running gateway endpoint.
Additional settings π
There are many other settings you can customize in automatic discovery and configuration.
For a list of settings that you can change for each language, see the following resources:
Language |
Resource |
---|---|
Java |
|
Node.js |
Configure the Splunk Distribution of OTel JS for Splunk Observability Cloud |
For a complete list of settings that you can modify in values.yaml for Kubernetes, see https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml.