Zero-code instrumentation for back-end applications in Linux ๐
The Splunk Distribution of the OpenTelemetry Collector uses automatic discovery with zero-code instrumentation to automatically detect back-end applications running in your Linux environment. By deploying the Collector with zero-code instrumentation, you can monitor applications and send data to Splunk Observability Cloud without editing your applicationโs code or configuring files.
Zero-code instrumentation for Linux can detect and configure the following applications and language runtimes:
Java
Node.js
.NET
How zero-code instrumentation for Linux works ๐
Zero-code instrumentation for Linux operates as a mode of the Splunk Distribution of the OpenTelemetry Collector. You install and activate zero-code instrumentation for the Collector by using the Linux installer script or package manager. During installation, you can specify the types of language runtimes you want the Collector to detect.
After installation, the Collector runs in your Linux environment and listens for requests to your applications. When the Collector detects activity, it gathers telemetry data from your application runtime and sends this data to Splunk Application Performance Monitoring (APM).
Requirements ๐
You need the following components to use zero-code instrumentation for back-end Linux applications:
systemd
curl
sudo
Make sure youโve also installed the components specific to your language runtime:
Java version 8 or higher and supported libraries. See Java agent compatibility and requirements for more information.
Node.js version 14 or higher and supported libraries. See Splunk OTel JS compatibility and requirements for more information.
.NET version 6.0 or higher and supported libraries. See OpenTelemetry .NET instrumentation compatibility and requirements for more information.
Zero-code instrumentation for .NET is only supported for x86_64/AMD64 architectures.
Get started ๐
To install and use zero-code instrumentation for Linux, follow these steps:
Install the package ๐
Using the installer script, you can install and activate zero-code instrumentation for either all supported applications on the host via the system-wide method or only for applications running as systemd
services.
Using the installer script, you can install the zero-code instrumentation package for Java and activate zero-code instrumentation for Java for either all supported Java applications on the host via the system-wide method or for only Java applications running as systemd
services.
Note
By default, zero-code instrumentation is activated for all languages (Java, Node.js, and .NET) when using the installer script. To deactivate zero-code instrumentation for other languages, add the --without-instrumentation-sdk [language]
option in the installer script command.
Run the installer script with the --with-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the --with-fluentd
option.
The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/java.conf
.
To automatically define the optional deployment.environment
resource attribute at installation time, run the installer script with the --deployment-environment <env>
option. Replace <env>
with the desired attribute value, for example, prod
, as shown in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.
Run the installer script with the --with-systemd-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
The systemd
instrumentation automatically adds environment variables to /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
.
Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the --with-fluentd
option.
To automatically define the optional deployment.environment
resource attribute at installation time, run the installer script with the --deployment-environment <env>
option. Replace <env>
with the desired attribute value, for example, prod
, as shown in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.
Note
You must first install the Splunk OpenTelemetry Collector using the Install the Collector for Linux with downloaded packages.
After installing the Collector, follow these steps to install the package using the Debian or RPM repositories with root
privileges:
You can either download the
splunk-otel-auto-instrumentation
package directly from the GitHub Releases page or add the Splunk repository to the package repositories on your Linux host. See Debian or RPM package repositories for instructions on how to configure your package repository.Run the following commands to install the package. Replace
<path to splunk-otel-auto-instrumentation deb/rpm>
with the local path to the downloaded package.sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
sudo rpm -ivh <path to splunk-otel-auto-instrumentation rpm>
Edit the
/etc/otel/collector/splunk-otel-collector.conf
file to set theSPLUNK_ACCESS_TOKEN
andSPLUNK_REALM
variables to the values you got earlier. If the file does not exist, use the provided sample at/etc/otel/collector/splunk-otel-collector.conf.example
as a starting point.SPLUNK_ACCESS_TOKEN=<access_token> SPLUNK_REALM=<realm>
Start the collector service:
sudo systemctl start splunk-otel-collector
Using the installer script, you can install and activate zero-code instrumentation for Node.js for either all supported Node.js applications on the host via the system-wide method or for only Node.js applications running as systemd
services.
The installer script installs the Node.js package using the npm install
command. To specify a custom path to npm
for installation, use the --npm-path <path>
option as in the following example:
--npm-path /custom/path/to/npm
Note
By default, zero-code instrumentation is activated for all languages (Java, Node.js, and .NET) when using the installer script. To deactivate zero-code instrumentation for other languages, add the --without-instrumentation-sdk [language]
option in the installer script command.
To install the package, run the Collector installer script with the --with-instrumentation
option. The installer script will install the Collector and the Node.js agent from the Splunk Distribution of OpenTelemetry JS. The Node.js agent automatically loads when a Node.js application starts on the local machine.
Run the installer script with the --with-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the
--with-fluentd
option.
The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/node.conf
.
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the collector service is running and restart your Node.js application(s). See Ensure the collector service is running and Start your applications.
Run the installer script with the --with-systemd-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>The
systemd
zero-code instrumentation method automatically adds environment variables to/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
.Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the
--with-fluentd
option.
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the collector service is running and restart your Node.js application(s). See Ensure the collector service is running and Start your applications.
Using the installer script, you can install and activate zero-code instrumentation for .NET for either all supported .NET applications on the host via the system-wide method or for only .NET applications running as systemd
services.
Note
By default, zero-code instrumentation is activated for all languages (Java, Node.js, and .NET) when using the installer script. To deactivate zero-code instrumentation for other languages, add the --without-instrumentation-sdk [language]
option in the installer script command.
Run the installer script with the --with-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the --with-fluentd
option.
The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/dotnet.conf
.
To automatically define the optional deployment.environment
resource attribute at installation time, run the installer script with the --deployment-environment <env>
option. Replace <env>
with the desired attribute value, for example, prod
, as shown in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.
Run the installer script with the --with-systemd-instrumentation
option, as shown in the following example. Replace <SPLUNK_REALM>
and <SPLUNK_ACCESS_TOKEN>
with your Splunk Observability Cloud realm and token, respectively.
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
The systemd
instrumentation automatically adds environment variables to /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
.
Note
If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the --with-fluentd
option.
To automatically define the optional deployment.environment
resource attribute at installation time, run the installer script with the --deployment-environment <env>
option. Replace <env>
with the desired attribute value, for example, prod
, as shown in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics
Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.
Ensure the collector service is running ๐
After a successful installation, run the following command to ensure the splunk-otel-collector
service is running:
sudo systemctl status splunk-otel-collector
If the service is not running, start or restart it with the following command:
sudo systemctl restart splunk-otel-collector
If the service fails to start, check that the SPLUNK_REALM
and SPLUNK_ACCESS_TOKEN
in /etc/otel/collector/splunk-otel-collector.conf
are correct. You can also view the service logs with this command:
sudo journalctl -u splunk-otel-collector
Start your applications ๐
For zero-code instrumentation to take effect, you must either restart the host or manually start or restart any applications on the host where you installed the package. You must restart the host or applications after installing the zero-code instrumentation package for the first time and whenever you make any changes to the configuration file.
After your applications are running, you can verify your data. See View results in Splunk APM. You can also configure instrumentation settings. See (Optional) Configure the instrumentation.
(Optional) Configure the instrumentation ๐
You can configure the Splunk Distribution of OpenTelemetry Collector to suit your instrumentation needs. In most cases, modifying the basic configuration is enough to get started.
To learn more, see the following resources:
Update zero-code instrumentation ๐
You can upgrade the package by using the package repository or by using Debian or RPM packages.
If you installed the package using the installer script, or if you configured the Debian or RPM package repositories manually, run the following commands according to your platform. Upgrading the package requires root
privileges.
Run the following commands:
sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation
You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
For the RPM package management system, run the following commands:
yum:
sudo yum upgrade splunk-otel-auto-instrumentation
dnf:
sudo dnf upgrade splunk-otel-auto-instrumentation
zypper:
sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation
After youโve upgraded the packages, manually start or restart the Java applications on the host for the changes to take effect.
To manually upgrade the package:
Download the
splunk-auto-auto-instrumentation
Debian or RPM package for the target system from the GitHub Releases page .Run the following commands to install the package. Replace
<path to splunk-otel-auto-instrumentation deb/rpm>
with the local path to the downloaded package:
sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
sudo rpm -Uvh <path to splunk-otel-auto-instrumentation rpm>
After upgrading the Debian package, you might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
You can also upgrade using the same package repositories as the Collector. See Debian or RPM packages for more information.
To update the Node.js agent to the latest provided version, you must first update the splunk-otel-auto-instrumentation
package.
You can upgrade the package by using the package repository or by using Debian or RPM packages.
If you installed the package using the installer script, or if you configured the Debian or RPM package repositories manually, run the following commands according to your platform. Upgrading the package requires root
privileges.
Run the following commands:
sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation
You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
For the RPM package management system, run the following commands:
yum:
sudo yum upgrade splunk-otel-auto-instrumentation
dnf:
sudo dnf upgrade splunk-otel-auto-instrumentation
zypper:
sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation
After youโve upgraded the packages, manually start or restart the Node.js applications on the host for the changes to take effect.
To manually upgrade the package:
Download the
splunk-auto-auto-instrumentation
Debian or RPM package for the target system from the GitHub Releases page .Run the following commands to install the package. Replace
<path to splunk-otel-auto-instrumentation deb/rpm>
with the local path to the downloaded package:
sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
sudo rpm -Uvh <path to splunk-otel-auto-instrumentation rpm>
After upgrading the Debian package, you might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
You can also upgrade using the same package repositories as the Collector. See Debian or RPM packages for more information.
After updating the splunk-otel-auto-instrumentation
package, run the following command:
cd /usr/lib/splunk-instrumentation/splunk-otel-js && \
sudo npm install /usr/lib/splunk-instrumentation/splunk-otel-js.tgz
The default zero-code instrumentation configuration expects the Node.js agent to be installed under the /usr/lib/splunk-instrumentation/splunk-otel-js
path.
If the Node.js agent is installed under a different path, manually update the path for the NODE_OPTIONS
environment variable in either /etc/splunk/zeroconfig/node.conf
for system-wide services or /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
for systemd
services. For example:
NODE_OPTIONS=-r /custom/nodejs/install/path/@splunk/otel/instrument
You can upgrade the package by using the package repository or by using Debian or RPM packages.
If you installed the package using the installer script, or if you configured the Debian or RPM package repositories manually, run the following commands according to your platform. Upgrading the package requires root
privileges.
Run the following commands:
sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation
You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
For the RPM package management system, run the following commands:
yum:
sudo yum upgrade splunk-otel-auto-instrumentation
dnf:
sudo dnf upgrade splunk-otel-auto-instrumentation
zypper:
sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation
After youโve upgraded the packages, manually start or restart the .NET applications on the host for the changes to take effect.
To manually upgrade the package:
Download the
splunk-auto-auto-instrumentation
Debian or RPM package for the target system from the GitHub Releases page .Run the following commands to install the package. Replace
<path to splunk-otel-auto-instrumentation deb/rpm>
with the local path to the downloaded package:
sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
sudo rpm -Uvh <path to splunk-otel-auto-instrumentation rpm>
After upgrading the Debian package, you might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/
. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.
You can also upgrade using the same package repositories as the Collector. See Debian or RPM packages for more information.
Troubleshooting ๐
To troubleshoot common errors that occur when instrumenting applications, see the following troubleshooting guides:
View results in Splunk APM ๐
After activating zero-code instrumentation, ensure your data is flowing into Splunk Observability Cloud. See Verify that your data is coming into Splunk APM.