Advanced customization for automatic discovery in Linux π
Learn how to customize automatic discovery and configuration for advanced scenarios.
Through advanced customization, you can achieve the following tasks:
Note
For advanced customization options in Kubernetes, see Advanced customization for automatic discovery in Kubernetes.
Override default automatic discovery settings π
You can override default automatic discovery settings to use features for profiling and runtime metrics collection.
Activate AlwaysOn Profiling π
You can activate CPU and memory profiling by updating the environment variables for your instrumentation. AlwaysOn Profiling continuously collects stack traces so that you can discover which lines of code are slowing your processes down. To learn more about Profiling, see Introduction to AlwaysOn Profiling for Splunk APM.
To activate profiling globally, add the --enable-profiler
flag upon installation for CPU profiling, or --enable-profiler-memory
flag for memory profiling. For example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory
You can also activate profiling for individual languages. By using this approach, you can determine which languages profiling gathers call stacks from.
Follow these steps to activate AlwaysOn Profiling for an individual language:
Open the <language>.conf file located in the
/etc/splunk/zeroconfig
directory.Set the environment variable
SPLUNK_PROFILER_ENABLED=true
for CPU profiling, andSPLUNK_PROFILER_MEMORY_ENABLED=true
for memory profiling.Restart your applications.
Note
If youβre using systemd
, the environment variables are instead located in /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
.
To sample call stacks from a specific interval, change the SPLUNK_PROFILER_CALL_STACK_INTERVAL
setting to your desired interval in milliseconds. The default value is 10000
.
For example, SPLUNK_PROFILER_CALL_STACK_INTERVAL=5000
sets the call stack interval to 5000 milliseconds.
Activate runtime metrics collection π
You can activate runtime metrics collection by updating the environment variables for your instrumentation. This setting configures the agent to collect additional metrics from your application.
To activate runtime metrics globally, add the --enable-metrics
flag upon installation. For example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-metrics
You can also activate runtime metrics collection for individual languages. By using this approach, you can determine which runtime metrics the Collector sends at a language-level.
To activate runtime metrics collection for an individual language, follow these steps:
Open the <language>.conf file located in the
/etc/splunk/zeroconfig
directory.Set the environment variable
SPLUNK_METRICS_ENABLED=true
.Restart your applications.
Use automatic discovery with gateway mode π
The Splunk OTel Collector Chart uses the agent mode by default. Activating gateway mode deploys an instance of the OpenTelemetry Collector in a separate container, and this instance collects data from the entire cluster.
To learn more about the gateway mode, see Data forwarding (gateway) mode.
You canβt directly deploy automatic discovery in gateway mode for Linux, but you can send collected data from automatic discovery to an existing gateway deployment.
Follow these steps to send data to a gateway deployment of the OpenTelemetry Collector:
Open the <language>.conf file located in
/etc/splunk/zeroconfig
.Set the environment variable
OTEL_EXPORTER_OTLP_ENDPOINT=<gateway_endpoint>
where <gateway_endpoint> is the port of your gateway deployment.Restart your applications.
The Collector now sends data to your gateway deployment.
Customize discovery settings for third-party applications π
By default, automatic discovery reads the built-in configuration provided by the Collector executable. You can provide your own configuration to modify settings or adjust the existing configuration in case of a partial discovery status.
The priority order for discovery configuration values from lowest to highest is:
Default
bundle.d
component configuration files, built into the Collector executableconfig.d/<receivers or extensions>/*.discovery.yaml
component configuration filesconfig.d/properties.discovery.yaml
properties file content in mapped formconfig.d/properties.discovery.yaml
properties file content using--set
formSPLUNK_DISCOVERY_<xyz>
property environment variables available to the Collector process--set splunk.discovery.<xyz>
property command line options
Define properties through the properties file π
You can override or add properties by creating the etc/otel/collector/config.d/properties.discovery.yaml
file. Each mapped property in the file overrides existing discovery settings. For example:
splunk.discovery: receivers: postgresql: username: "${PG_USERNAME_ENVVAR}" password: "${PG_PASSWORD_ENVVAR}"
You can use the --discovery-properties=<filepath.yaml>
argument to load discovery mode properties that you donβt want to share with other Collectors. If you specify discovery properties using this argument, properties contained in config.d/properties.discovery.yaml
are ignored.
Create custom configurations π
To create custom discovery configurations, follow these steps:
Navigate to the
config.d
folder in/etc/otel/collector/config.d
on Linux.Create a
<name>.discovery.yaml
file and place it inside a subdirectory ofconfig.d
, for exampleextensions
orreceivers
where<name>
is the name of the component you want to use.Edit the
<name>.discovery.yaml
files to add the desired configuration. For example, if youβre adding a receiver, discovery mode loads the content inside thereceivers
object of the Collector configuration.
Custom configurations consist of the fields you want to override in the default configuration. For example:
# <some-receiver-type-with-optional-name.discovery.yaml>
<receiver_type>(/<receiver_name>):
enabled: <true | false> # true by default
rule:
<observer_type>(/<observer_name>): <receiver creator rule for this observer>
config:
default:
<default embedded receiver config>
<observer_type>(/<observer_name>):
<observer-specific config items, merged with `default`>
status:
metrics:
<discovery receiver metric status entries>
statements:
<discovery receiver statement status entries>
Use the --dry-run
option to check the resulting discovery configuration before using it with the Collector.
See the Discovery receiver README file for more information.
Define a custom configuration directory π
To define a custom directory for discovery settings, use the --config-dir
option as in the example:
otelcol --discovery --config-dir <custom_path>
Additional settings for language runtimes π
There are many other settings you can customize using automatic discovery and configuration.
For a list of settings that you can change for each language, see the following resources:
Language |
Resource |
---|---|
Java |
|
Node.js |
Configure the Splunk Distribution of OTel JS for Splunk Observability Cloud |