Scenario: Deepu accelerates troubleshooting using Business Workflows on Tag Spotlight 🔗
Deepu, the payment service owner at Buttercup Games, wants to understand how errors in the payment service impact business KPIs.
To understand the impact of errors on business KPIs, Deepu takes the following steps:
Deepu filters Tag Spotlight to find the subset of traces where most errors occur
Deepu switches to Splunk Log Observer for further troubleshooting
Deepu opens Tag Spotlight and selects the affected service 🔗
Deepu uses search to search for Tag Spotlight and selects the navigation result to go to Tag Spotlight. They select the paymentservice.
Deepu reviews the errors by workflow 🔗
Deepu reviews the errors by workflow and notices that the requests for the frontend:/cart/checkout workflow return a majority of root-cause errors.
Deepu filters Tag Spotlight to find the subset of traces where most errors occur 🔗
Because APM supports infinite cardinality, Deepu adds the frontend:/cart/checkout workflow to the filter and applies additional filters for version to v350.10
and Kind to SERVER
to examine the subset of traces where the most errors occur:
Deepu opens an example trace 🔗
Because Splunk APM stores all traces without sampling, Deepu selects the peak of errors on the chart to see example traces of those tag combinations at that timestamp. Deepu finds that the requests from checkoutservice to paymentservice are failing.
Deepu switches to Splunk Log Observer for further troubleshooting 🔗
Deepu remembers that version 350.10 of the payment service recently deployed a backward-incompatible change to its API. Because Deepu turned on Related Content in Splunk APM, Deepu can select Logs for payment service to switch to Splunk Log Observer to investigate whether the deployment causes the error.
Summary 🔗
Using Business Workflows and Tag Spotlight Deepu quickly isolated a trace with an error. Because Deepu had related content turned on, Deepu was able to quickly navigate from the trace to logs to further investigate causes of the errors.
Learn more 🔗
For details about business workflows, see Correlate traces to track Business Workflows.
For details about using Related Content, see Related Content in Splunk Observability Cloud.
For more information about using Splunk Log Observer Connect to detect the source of problems, see Introduction to Splunk Log Observer Connect.