Docs » Supported integrations in Splunk Observability Cloud » Instrument back-end applications to send spans to Splunk APM » Instrument Java applications for Splunk Observability Cloud » Migration guide for OpenTelemetry Java 2.x metrics

Migration guide for OpenTelemetry Java 2.x metrics 🔗

OpenTelemetry Java Instrumentation 2.x contains a set of breaking changes, introduced as part of recent OpenTelemetry HTTP semantic convention updates. Versions 2.5.0 and higher of the Splunk Distribution of OpenTelemetry Java are fully compatible with the updated semantic conventions.

The following migration instructions assume the following:

  • You’re sending Java application metrics using version 1.x of the Splunk Distribution of OpenTelemetry Java.

  • You’ve created custom dashboards, detectors, or alerts based on Java application metrics.

Follow the steps in this guide to migrate to 2.x metrics and HTTP semantic conventions, and to convert your custom reporting elements to the new metrics format.

Note

Version 2.x of the Java agent collects metrics and logs by default. This might result in increased data ingest costs.

Caution

The default protocol changed in the Java 2.0 instrumentation from gRPC to http/protobuf. For custom configurations, verify that you’re sending data to http/protobuf endpoints. For troubleshooting guidance, see Telemetry export issues.

Prerequisites 🔗

To migrate from OpenTelemetry Java 1.x to OpenTelemetry Java 2.x, you need the following:

If you’re instrumenting your Java services using the Splunk Distribution of OpenTelemetry Java 1.x or the equivalent upstream instrumentation, you can already migrate to the version 2.5.0 and higher of the Java agent.

Note

AlwaysOn Profiling metrics are not impacted by this change.

Migration best practices 🔗

The following best practices can help you when initiating the migration process:

  1. Familiarize yourself with this documentation.

  2. Read the release notes. See Releases on GitHub.

  3. Use a development or test environment.

  4. Migrate production services gradually and grouped by type.

  5. Identify changes in your instrumentation settings.

  6. Validate the data in Splunk Observability Cloud.

  7. Verify the impact of HTTP semantic convention changes. See Migrate APM custom reporting to OpenTelemetry Java Agent 2.0.

Use the Data Migration tool 🔗

Due to the changes in metric names, upgrading to Java OTel 2.x might break existing dashboards, detectors, and other features. To prevent sudden loss of access to custom reporting elements, use the Data Migration tool, which transforms and duplicates metric data in 1.x to the 2.x formats for a limited period of time at no additional cost.

Data Migration tool running migration metrics rules for Java 2.x metrics

To access the Data Migration tool:

  1. Go to Settings, Data Configuration.

  2. Select Data Migration.

  3. Inside the Start migration card, select Start.

For each supported process, you can turn on and off the data migration, see the number of Metric Time Series (MTS) migrated, and view the grace period duration. The duplication and double-publishing of metrics follows a set of predefined rules that are activated when you decide to migrate.

Note

Metric rules are treated as system rules and can’t be edited.

Grace period 🔗

The grace period for receiving and processing duplicated metrics at no additional cost lasts 6 months, starting with the release of the Java agent version 2.5.0 on June 25, 2024 and ending on January 31, 2025.

Migration support is available for 12 months after the release of version 2.5.0 and will be deprecated after 18 months.

Note

After the grace period, duplicated metric data is billed as custom metric data. Make sure to turn off the Data Migration action after you’ve completed the migration to avoid surcharges.

Migrate to OTel Java 2.x 🔗

To migrate your instrumentation to the version 2.5.0 or higher of the Java agent, follow these steps:

  1. Turn on the migration of Java metrics in the Data Migration page:

    • Go to Settings, Data Configuration.

    • Select Data Migration.

    • Inside the Start migration card, select Start.

  2. Turn on OTLP histograms in the Splunk Distribution of OpenTelemetry Collector.

    To send histogram data to Splunk Observability Cloud, set the send_otlp_histograms option to true. For example:

    exporters:
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    sync_host_metadata: true
    correlation:
    send_otlp_histograms: true

  3. Make sure version 2.5.0 or higher of the Splunk Distribution of the Java agent is installed. See Upgrade the Splunk Distribution of OpenTelemetry Java.

  4. If you defined a custom Collector endpoint for metrics, make sure to update the port and use the correct property:

    # Legacy property and value: -Dsplunk.metrics.endpoint=http(s)://collector:9943
# You can also use the OTEL_EXPORTER_OTLP_METRICS_ENDPOINT environment variable
-Dotel.exporter.otlp.metrics.endpoint=http://localhost:4318/v1/metrics

    Review all others settings to check that they’re still applicable to version 2.5.0. See Configure the Java agent for Splunk Observability Cloud.

  5. Migrate your custom reporting elements:

  6. (Optional) Start using the new Java metrics 2.x built-in dashboards. Built-in dashboard versions are available for Java service metrics representing metrics from versions 1.x and 2.x.

  7. When ready, turn off the migration:

    • Go to Settings, Data Configuration.

    • Select Data Migration.

    • In the Stop migration card, select Stop.

Caution

If you don’t turn off the Data Migration stream for Java metrics after the grace period, the duplicated metrics are billed as custom metrics. See Use the Data Migration tool.

New metric names for version 2.x 🔗

The following table shows the metrics produced by default by OpenTelemetry Java 2.0 and higher, together with their legacy equivalent from version 1.x.

OTel Java 2.0 metric

Legacy metric (1.x)

db.client.connections.create_time (Histogram, ms)

db.pool.connections.create_time

db.client.connections.idle.max

db.pool.connections.idle.max

db.client.connections.idle.min

db.pool.connections.idle.min

db.client.connections.max

db.pool.connections.max

db.client.connections.pending_requests

db.pool.connections.pending_threads

db.client.connections.timeouts

db.pool.connections.timeouts

db.client.connections.usage[state=idle]

db.pool.connections.idle

db.client.connections.usage[state=used]

db.pool.connections.active

db.client.connections.use_time (Histogram, ms)

db.pool.connections.use_time

db.client.connections.wait_time (Histogram, ms)

db.pool.connections.wait_time

jvm.buffer.count

runtime.jvm.buffer.count

jvm.buffer.memory.limit

runtime.jvm.buffer.total.capacity

jvm.buffer.memory.usage

runtime.jvm.buffer.memory.used

jvm.class.count

runtime.jvm.classes.loaded

jvm.class.unloaded

runtime.jvm.classes.unloaded

jvm.gc.duration{jvm.gc.name=<concurrent gcs>} (Histogram)
jvm.gc.action

runtime.jvm.gc.concurrent.phase.time

jvm.gc.duration{jvm.gc.name!=<concurrent gcs>}
jvm.gc.action

runtime.jvm.gc.pause

jvm.memory.allocated*

runtime.jvm.gc.memory.allocated
process.runtime.jvm.memory.allocated

jvm.memory.committed

runtime.jvm.memory.committed

jvm.memory.limit

runtime.jvm.memory.max

jvm.memory.limit{jvm.memory.pool.name=<long lived pools>}

runtime.jvm.gc.max.data.size

jvm.memory.used

runtime.jvm.memory.used

jvm.memory.used_after_last_gc{jvm.memory.pool.name=<long lived pools>}

runtime.jvm.gc.live.data.size

jvm.thread.count

runtime.jvm.threads.daemon
runtime.jvm.threads.live

* This is a Splunk specific metric and it’s not present in the upstream semantic conventions.

Note

The previous table contains metrics generated by default. Additional metrics might be emitted by supported metrics instrumentation, for example when instrumenting application servers.

For more information on the HTTP semantic convention changes, see HTTP semantic convention stability migration guide on GitHub.

Metrics no longer reported 🔗

Due to changes in the metrics emitted by the Java instrumentation version 2.5.0 and higher, detectors or dashboards that use the following metrics might not work as before the migration:

  • db.pool.connections

  • executor.tasks.completed

  • executor.tasks.submitted

  • executor.threads

  • executor.threads.active

  • executor.threads.core

  • executor.threads.idle

  • executor.threads.max

  • runtime.jvm.memory.usage.after.gc

  • runtime.jvm.gc.memory.promoted

  • runtime.jvm.gc.overhead

  • runtime.jvm.threads.peak

  • runtime.jvm.threads.states

Built-in dashboards 🔗

Use the following links to access the built-in OTel Java 2.X APM services dashboards:

Optionally, you can navigate to the dashboards on your own:

  1. In the left navigation menu, select Dashboards.

  2. In the Built-in dashboard groups section, scroll down to the APM Java services (OTel 2.X) dashboard group, where the three dashboards are grouped.

Note

To view the built-in dashboards, the jvm.classes.loaded metric must be received by Splunk Observability Cloud.

Troubleshooting 🔗

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Available to prospective customers and free trial users

  • Ask a question and get answers through community support at Splunk Answers .

  • Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

This page was last updated on Sep 18, 2024.