Scenario: Deepu accelerates troubleshooting using Business Workflows on Tag Spotlight 🔗
Deepu, the payment service owner at Buttercup Games, wants to understand how errors in the payment service impact business KPIs.
To understand the impact of errors on business KPIs, Deepu takes the following steps:
Deepu selects the payment service on the service map and opens Tag Spotlight
Deepu filters Tag Spotlight to find the subset to the traces where most errors occur
Deepu switches to Splunk Log Observer for further troubleshooting
Deepu selects the payment service on the service map and opens Tag Spotlight 🔗
Deepu selects the paymentservice node on the service map and then selects Tag Spotlight to look at errors and latency for the Workflow tag.
Deepu reviews the errors by workflow 🔗
Deepu reviews the errors by workflow and notices that the requests for the frontend:/cart/checkout workflow return a majority of root-cause errors.
Deepu filters Tag Spotlight to find the subset to the traces where most errors occur 🔗
Because APM supports infinite cardinality, Deepu selects the frontend:/cart/checkout workflow and applies filters to show only data from traces. Deepu filters version to
v350.10 and Kind to
SERVER to view the tags to examine the subset of traces where the most errors occur:
Deepu opens an example trace 🔗
Because Splunk APM stores all traces without sampling, Deepu clicks the peak of errors on the chart to see example traces of those tag combinations at that timestamp. Deepu finds that the requests from checkoutservice to paymentservice are failing.
Deepu switches to Splunk Log Observer for further troubleshooting 🔗
Deepu remembers that version 350.10 of the payment service recently deployed a backward-incompatible change to its API. Because Deepu turned on Related Content in Splunk APM, Deepu can select Logs for payment service to switch to Splunk Log Observer to investigate whether the deployment causes the error.
Using Business Workflows and Tag Spotlight Deepu quickly isolated a trace with an error. Because Deepu had related content turned on, Deepu was able to quickly navigate from the trace to logs to further investigate causes of the errors.
Learn more 🔗
For details about business workflows, see Correlate traces to track Business Workflows.
For details about using Related Content, see Related Content in Splunk Observability Cloud.
For more information about using Splunk Log Observer to detect the source of problems, see Introduction to Splunk Log Observer.