Manually instrument Python applications for Splunk Observability Cloud 🔗
Instrumenting applications automatically using the agent of the Splunk Distribution of OpenTelemetry Python covers most needs. Manually instrumenting your application is only necessary when, for example, you need to add custom attributes to spans or need to manually generate spans.
Start tracing using code 🔗
If you can’t use the splunk-py-trace
command to launch the application, you can instead import and configure start_tracing
by adding the following snippet to your application code:
from splunk_otel.tracing import start_tracing start_tracing() # Also accepts optional settings. For example: # # start_tracing( # service_name='<my-python-service>', # span_exporter_factories=[OTLPSpanExporter] # access_token='<access_token>', # max_attr_length=12000, # trace_response_header_enabled=True, # resource_attributes={ # 'service.version': '<your_version>', # 'deployment.environment': '<your_environment>', # })
Note
Don’t add this code to the application if you’re using the splunk-py-trace
command to launch the application.
Create custom traces 🔗
If you’re adding manual instrumentation on top of auto-instrumentation, you can capture additional spans as follows:
Import the OpenTelemetry SDK:
from opentelemetry import trace
Create a tracer for your spans:
tracer = trace.get_tracer("tracer.name")
Create a span as current span:
def reticulate_splines(): with tracer.start_as_current_span("span-name") as span: print("Reticulating splines...") # When the 'with' block goes out of scope, the 'span' is closed
Alternatively, if you’re not using auto-instrumentation, use the following steps instead:
Import the OpenTelemetry SDK:
from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( BatchSpanProcessor, ConsoleSpanExporter, )
Create a tracer for your spans:
provider = TracerProvider() processor = BatchSpanProcessor(ConsoleSpanExporter()) provider.add_span_processor(processor) trace.set_tracer_provider(provider) tracer = trace.get_tracer("tracer.name")
Create a span as current span:
def reticulate_splines(): with tracer.start_as_current_span("span-name") as span: print("Reticulating splines...") # When the 'with' block goes out of scope, the 'span' is closed
For more examples, see the OpenTelemetry official documentation .
Create custom metrics 🔗
The Splunk Distribution of OpenTelemetry Python supports the following instrumentations:
Counter (synchronous)
Counter (asynchronous)
Gauge (asynchronous)
UpDownCounter (synchronous)
UpDownCounter (asynchronous)
To create custom metrics, follow the steps depending on the type of metric instrumentation.
Synchronous instruments, like counters, are invoked inline with business logic. An example of synchronous instrument is a counter for the number of bytes sent to a server. They support context propagation.
Import the OpenTelemetry API:
from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import ( ConsoleMetricExporter, PeriodicExportingMetricReader, )
Create a meter provider:
meter := otel.Meter("ExampleService")
Create an instrument to take measurements:
metric_reader = PeriodicExportingMetricReader(ConsoleMetricExporter()) provider = MeterProvider(metric_readers=[metric_reader]) metrics.set_meter_provider(provider) meter = metrics.get_meter("my.meter.name")
Perform the measurements:
peanut_counter = meter.create_counter( "peanut.counter", unit="1", description="Counts the number of consumed peanuts" ) def do_stuff(work_item): peanut_counter.add(1, {"work.type": work_item.work_type}) print("Collecting peanuts...")
Asynchronous instruments, like asynchronous gauges, provide callback functions that the OTel SDK runs in the background. An example of asynchronous instrument is a humidity sensor that is polled every minute for new data. They don’t support context propagation.
Import the OpenTelemetry API:
from typing import Iterable from opentelemetry.metrics import CallbackOptions, Observation
Write a callback to request data:
def get_temp_data(options: CallbackOptions) -> Iterable[Temperature]: r = requests.get( "http://weather/data/city", timeout=options.timeout_millis / 10**3 ) for metadata in r.json(): yield Temperature( metadata["temperature"], {"city.name": metadata["temperature"]} )
Create an instrument to take asynchronous measurements:
meter.create_observable_gauge( "city.temperature", callbacks=[get_temp_data], description="Mean temperature of the city", )
For more examples, see the OpenTelemetry official documentation .
Frameworks that require manual instrumentation 🔗
Some Python frameworks only support manual instrumentation. For specific instructions, see:
Note
Manual OTel instrumentation is fully compatible with Splunk automatic Python instrumentation and is fully supported by Splunk.