Troubleshoot AlwaysOn Profiling 🔗

If you have instrumented an application but are not seeing profiling data in Splunk APM, use the following guidelines to troubleshoot AlwaysOn Profiling:

No profiling data in Splunk Observability Cloud
AlwaysOn Profiling is not accessible in Observability Cloud
Instrumentation-specific troubleshooting

Note

AlwaysOn Profiling requires the Splunk Distribution of OpenTelemetry Collector version 0.44 or higher.

No profiling data in Splunk Observability Cloud 🔗

If profiling data does not appear in Observability Cloud, do the following:

Check that you’ve instrumented your application 🔗

You can extract AlwaysOn Profiling data only if you’ve instrumented your application or service for Splunk APM. If the APM instrumentation is not loading or isn’t working, the profiler cannot work.

To solve this, check that you’ve instrumented your application and that the application is sending trace data to APM. See Instrument your application or service.

Check the OpenTelemetry Collector configuration 🔗

AlwaysOn Profiling requires the Splunk HTTP Event Collector (HEC) exporter to send profiling data to Splunk Observability Cloud. If the Splunk HEC exporter isn’t configured, the Collector drops profiling data.

To solve this issue, edit the configuration file of the Collector and make sure that a profiling pipeline exists with an OTLP gRPC receiver and a Splunk HEC exporter. See Splunk HEC exporter for more information.

The following example shows you how to configure a pipeline in the agent-config.yaml file. Set the SPLUNK_ACCESS_TOKEN environment variable to a valid access token. See Create and manage organization access tokens using Splunk Observability Cloud.

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  # Profiling
  splunk_hec/profiling:
    token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "${SPLUNK_INGEST_URL}/v1/log"
    log_data_enabled: false

processors:
  batch:
  memory_limiter:
    check_interval: 2s
    limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}

service:
  pipelines:
    logs/profiling:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [splunk_hec, splunk_hec/profiling]

The exporter is configured automatically for the Splunk OTel Collector version 0.44.0 and higher. If you’re using a version of the Collector lower than 0.44.0, you might have to edit the configuration manually.

Check that you’ve activated AlwaysOn Profiling 🔗

Depending on the programming language, you can activate AlwaysOn Profiling by setting a system property, a function argument, or an environment variable. System properties and function arguments always take precedence. If the profiler is not activated, Observability Cloud can’t receive profiling data.

To solve this issue, check that you’ve activated the profiler. See Activate AlwaysOn Profiling.

Check the Helm chart configuration 🔗

If you’ve deployed the Collector in a Kubernetes environment, make sure that the splunkObservability.profilingEnabled=true is present. See Helm chart deployments for more information.

Error exporting profiling data Error: 14 UNAVAILABLE: No connection established 🔗

Check the following configurations:

The profiling exporting endpoint is correctly set to the host and port where the OTEL collector is running.
- If the SPLUNK_PROFILER_LOGS_ENDPOINT environment variable or splunk.profiler.logs-endpoint system property is set, this value is used.
- Otherwise, if the OTEL_EXPORTER_OTLP_ENDPOINT environment variable or otel.exporter.otlp.endpoint system property is set, this value is used.
- Finally, if neither is set, the default value of http://localhost:4317
The OTel collector is running, the port is open and accessible from the host where the profiled application is running.
The receiver for OTLP/gRPC is turned on in the OTel collector configuration.
The profiling pipeline is turned on in OTel collector configuration.

No call stacks available for a span 🔗

Span might lack call stacks if the duration of the span is shorter than the snapshot interval for capturing call stacks. For example, the default snapshot interval for Java instrumentation is 10 seconds, so spans shorter than 10 seconds might not contain call stacks. To set a shorter interval, set the SPLUNK_PROFILER_CALL_STACK_INTERVAL environment variable to a value lower than 10000 milliseconds.

Another cause for call stacks not appearing is when HTTP requests follow an async/await pattern. When the processing thread was executing in the scope of a span from another trace when the snapshot was recorded.

AlwaysOn Profiling is not accessible in Observability Cloud 🔗

If you’re sending profiling data to Observability Cloud but can’t see AlwaysOn Profiling in Splunk APM, your organization might be lacking the profiler entitlement.

AlwaysOn Profiling is activated for all host-based subscriptions. For TAPM-based subscriptions, AlwaysOn Profiling might be deactivated depending on the contract.

To solve this issue, reach out to Splunk Support to request they activate the AlwaysOn Profiling feature.

Instrumentation-specific troubleshooting 🔗

Some profiler issues might be specific to the APM instrumentation. See the following instructions to troubleshoot instrumentation-specific issues:

Deactivate profiling log data for specific hosts 🔗

If you don’t need AlwaysOn Profiling data for a specific host or container, see Unwanted profiling logs appearing in Splunk Observability Cloud.

Related Topics