Manually instrument Python applications for Splunk Observability Cloud π
Instrumenting applications automatically using the agent of the Splunk Distribution of OpenTelemetry Python covers most needs. Manually instrumenting your application is only necessary when, for example, you need to add custom attributes to spans or need to manually generate spans.
Create custom traces π
To create custom spans and traces, follow these steps:
If you canβt use the
splunk-py-trace
command, import and configurestart_tracing
:from splunk_otel.tracing import start_tracing start_tracing() # Also accepts optional settings. For example: # # start_tracing( # service_name='<my-python-service>', # span_exporter_factories=[OTLPSpanExporter] # access_token='<access_token>', # max_attr_length=12000, # trace_response_header_enabled=True, # resource_attributes={ # 'service.version': '<your_version>', # 'deployment.environment': '<your_environment>', # })
As an alternative, you can also import the OpenTelemetry SDK:
from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( BatchSpanProcessor, ConsoleSpanExporter, )
Create a tracer for your spans:
provider = TracerProvider() processor = BatchSpanProcessor(ConsoleSpanExporter()) provider.add_span_processor(processor) trace.set_tracer_provider(provider) tracer = trace.get_tracer("tracer.name")
Create a span as current span:
def reticulate_splines(): with tracer.start_as_current_span("span-name") as span: print("Reticulating splines...") # When the 'with' block goes out of scope, the 'span' is closed
For more examples, see the Manual instrumentation docs at https://opentelemetry.io/docs/instrumentation/python/manual/.
Create custom metrics π
The Splunk Distribution of OpenTelemetry Python supports the following instrumentations:
Counter (synchronous)
Counter (asynchronous)
Gauge (asynchronous)
UpDownCounter (synchronous)
UpDownCounter (asynchronous)
To create custom metrics, follow the steps depending on the type of metric instrumentation.
Synchronous instruments, like counters, are invoked inline with business logic. An example of synchronous instrument is a counter for the number of bytes sent to a server. They support context propagation.
Import the OpenTelemetry API:
from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import ( ConsoleMetricExporter, PeriodicExportingMetricReader, )
Create a meter provider:
meter := otel.Meter("ExampleService")
Create an instrument to take measurements:
metric_reader = PeriodicExportingMetricReader(ConsoleMetricExporter()) provider = MeterProvider(metric_readers=[metric_reader]) metrics.set_meter_provider(provider) meter = metrics.get_meter("my.meter.name")
Perform the measurements:
peanut_counter = meter.create_counter( "peanut.counter", unit="1", description="Counts the number of consumed peanuts" ) def do_stuff(work_item): peanut_counter.add(1, {"work.type": work_item.work_type}) print("Collecting peanuts...")
Asynchronous instruments, like asynchronous gauges, provide callback functions that the OTel SDK runs in the background. An example of asynchronous instrument is a humidity sensor that is polled every minute for new data. They donβt support context propagation.
Import the OpenTelemetry API:
from typing import Iterable from opentelemetry.metrics import CallbackOptions, Observation
Write a callback to request data:
def get_temp_data(options: CallbackOptions) -> Iterable[Temperature]: r = requests.get( "http://weather/data/city", timeout=options.timeout_millis / 10**3 ) for metadata in r.json(): yield Temperature( metadata["temperature"], {"city.name": metadata["temperature"]} )
Create an instrument to take asynchronous measurements:
meter.create_observable_gauge( "city.temperature", callbacks=[get_temp_data], description="Mean temperature of the city", )
For more examples, see the Manual instrumentation docs at https://opentelemetry.io/docs/instrumentation/python/manual/.
Frameworks that require manual instrumentation π
Some Python frameworks only support manual instrumentation. For specific instructions, see:
Note
Manual OTel instrumentation is fully compatible with Splunk automatic Python instrumentation and is fully supported by Splunk.