Docs » Get AlwaysOn Profiling data into Splunk APM

Get AlwaysOn Profiling data into Splunk APM 🔗

Follow these instructions to get profiling data into Splunk APM using AlwaysOn Profiling:

  1. Set up the Splunk OpenTelemetry Collector.

  2. Instrument your application or service.

  3. Enable AlwaysOn Profiling.

  4. Check that Observability Cloud is receiving profiling data.

Set up the Splunk OpenTelemetry Collector 🔗

To get data into Splunk AlwaysOn Profiling, you must have a Splunk OpenTelemetry (OTel) collector set up and working. See Install and configure Splunk Distribution of OpenTelemetry Collector.

Configure a pipeline for profiling data 🔗

AlwaysOn Profiling requires the Splunk HTTP Event Collector (HEC) exporter to send profiling data to Splunk Observability Cloud. In the Splunk OpenTelemetry Collector configuration file, make sure that a profiling pipeline exists with an OTLP gRPC receiver and a Splunk HEC exporter. For more information, see Configure the Splunk OpenTelemetry Collector.

The following example shows you how to configure a pipeline in the agent-config.yaml file. Set the SPLUNK_ACCESS_TOKEN environment variable to a valid access token. See Create and manage organization access tokens using Splunk Observability Cloud for more information.

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  splunk_hec:
    token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v1/log"
  logging/info:
    loglevel: info

service:
  pipelines:
    logs/profiling:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging/info, splunk_hec]

Instrument your application or service 🔗

AlwaysOn Profiling requires APM tracing data to correlate stack traces to your application requests. To instrument your application for Splunk APM, follow the steps for the appropriate programming language:

Language

Available instrumentation

Documentation

Java

Splunk Distribution of OpenTelemetry Java

Instrument a Java application for Splunk Observability Cloud

Note

See Application Performance Monitoring (APM) for information on Profiling data retention periods.

Enable AlwaysOn Profiling 🔗

After you’ve instrumented your service for Observability Cloud and checked that APM data is getting into Splunk APM, enable the profiler.

To enable AlwaysOn Profiling, follow the steps for the appropriate programming language:

  • Enable the profiler by setting the splunk.profiler.enabled system property or the SPLUNK_PROFILER_ENABLED environment variable to true.

  • Make sure that the splunk.profiler.logs-endpoint system property or the SPLUNK_PROFILER_LOGS_ENDPOINT environment variable point to http://localhost:4317.

The following example shows how to enable the profiler using the system property:

java -javaagent:./splunk-otel-javaagent.jar \
-Dsplunk.profiler.enabled=true \
-Dsplunk.profiler.logs-endpoint=https://localhost:4317 \
-jar <your_application>.jar

For more configuration options, see Java settings for AlwaysOn Profiling.

Check that Observability Cloud is receiving profiling data 🔗

After you set up and enable AlwaysOn Profiling, check that profiling data is coming in:

  1. Log in to Splunk Observability Cloud.

  2. In the left navigation menu, select APM.

  3. In Splunk APM, select AlwaysOn Profiling.

  4. Check that your spans have call stacks available. See Browse stack traces linked to spans in Splunk APM to learn how to locate and browse call stacks.

You can also browse all stack traces coming from your application in the flame graph. See Understand and use the flame graph for more information about the flame graph.