Docs » Supported integrations in Splunk Observability Cloud » Instrument back-end applications to send spans to Splunk APM » Instrument Java applications for Splunk Observability Cloud » Connect Java trace data with logs for Splunk Observability Cloud

Connect Java trace data with logs for Splunk Observability Cloud 🔗

The Splunk Distribution of OpenTelemetry Java agent automatically captures application logs and sends them to the OpenTelemetry Collector using the OTLP exporter, annotating the log events with trace ID, span ID, and trace flags.

If needed, you can configure your Java logging library to produce logs that include additional attributes provided automatically by the Splunk OTel Java agent, like the version of your service or the deployment environment.

Note

Logs export requires the Splunk Distribution of OpenTelemetry Collector.

Check compatibility and requirements 🔗

The Splunk OTel Java agent supports the following logging libraries:

  • Log4j 2 2.7 and higher

  • Log4j 1 1.2 and higher

  • Logback 1.0 and higher

  • JBoss LogManager 1.1.0 and higher

The java.util.logging library is fully supported in all JDK versions that are compatible with the Splunk Distribution of OpenTelemetry Java. See Java agent compatibility and requirements.

Trace metadata in log statements 🔗

The Splunk OTel Java agent automatically adds the following mapped diagnostic context (MDC) fields for logging libraries by default:

  • Trace information: trace_id and span_id

  • Trace flags

The default behavior is to send the MDC fields to the Collector but not write them to the application log files. To include this information in the log files, you must set your logging framework output format to include MDC fields. To configure your logging framework, see Configure your logging library.

The Collector sends the annotated logs using the OTLP exporter.

Deactivate logs export 🔗

To turn off logs export, set the OTEL_LOGS_EXPORTER environment variable or the otel.logs.exporter system property to none.

Inject resource attributes 🔗

While the Java agent automatically generates and sends logs to Splunk through the Collector, you can still produce annotated logs using a compatible log library, so that logs can be collected manually or go through the Universal Forwarder. See Introduction to Splunk Log Observer Connect.

For example, you can inject resource attributes in your log statements, such as service.name and deployment.environment. This requires defining the attributes you want to inject and configuring your logger manually.

Define the resource attributes 🔗

Before injecting attributes, you must make them available through the MDC by setting the mdc.resource-attributes property at runtime. For example:

-Dotel.instrumentation.common.mdc.resource-attributes=service.name,deployment.environment

Configure your logging library 🔗

The Splunk Distribution of OpenTelemetry Java exposes resource attributes as context properties, which you can use to configure logger libraries.

The following examples show how to include additional metadata in log statements produced by the logging library:

Edit your Log4j configuration, for example in the src/main/resources/log4j2.xml file. Depending on your environment, you might have to edit a different file or use a different configuration system.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
   <Appenders>
      <Console name="STDOUT" target="SYSTEM_OUT">
         <JsonLayout compact="true" eventEol="true">
            <KeyValuePair key="trace_id" value="${ctx:trace_id}"/>
            <KeyValuePair key="span_id" value="${ctx:span_id}"/>
            <KeyValuePair key="service.name" value="${ctx:service.name}"/>
            <KeyValuePair key="environment" value="${ctx:deployment.environment}"/>
            <KeyValuePair key="trace_sampled" value="${ctx:trace_flags}"/>
         </JsonLayout>
      </Console>
   </Appenders>
   <!-- More configuration -->
</Configuration>

For Spring Boot applications, you can also edit the application.properties file to add the following logging pattern:

logging.pattern.console = %d{yyyy-MM-dd HH:mm:ss} - %logger{36} - %msg trace_id=%X{trace_id} span_id=%X{span_id} service=%X{service.name}, env=%X{deployment.environment} trace_flags=%X{trace_flags} %n

If you’re instrumenting a serverless service or application, use environment variables instead. The deployment environment requires that you set an arbitrary environment variable, for example OTEL_ENV_NAME.

<PatternLayout>
   <pattern>
      service.name=${OTEL_SERVICE_NAME}, deployment.environment=${OTEL_ENV_NAME} %m%n
   </pattern>
</PatternLayout>

This page was last updated on Aug 19, 2024.