Troubleshoot the Content Pack for Splunk Observability Cloud
The Content Pack for Splunk Observability Cloud relies on the Splunk Infrastructure Monitoring Add-on for input collection. When troubleshooting, determine whether the issue you are experiencing is relevant to the content pack or to the add-on. In general, if your data successfully reaches your Splunk indexes, the content pack requires troubleshooting. If data isn't reaching your Splunk indexes, then check for configuration problems with the accounts and inputs handled by the add-ons.
Here are some common issues in Content Pack for Splunk Observability Cloud and how to resolve them.
The Service Tree isn't lighting up or not all entities are displaying for KPIs
Problem
The Service Tree doesn't light up or not all entities are displaying for the KPIs.
Cause
KPIs are scheduled to run every 5 minutes with a 5-minute calculation window. If data is being sent to Splunk at an interval greater than 5 minutes, it might cause the entity to drop from the service tree.
Solution
In ITSI or IT Essentials Work, go to Configuration > KPI Base Searches. These are the KPI base searches used in this content pack:
RUM_App_Base
RUM_Browser_Base
RUM_Synthetics_Base
SIM_cloud_aws_ec2
SIM_cloud_aws_lambda
SIM_cloud_azure_functions
SIM_cloud_azure_vm
SIM_cloud_gcp_compute
SIM_cloud_gcp_functions
SIM_containers
SIM_data_center_hosts
SIM_kubernetes
SplunkAPM Rate Base Search
SSM_api_tests
SSM_browser_tests
SSM_http_tests
DA-ITSI-APM:BusinessWorkflows
DA-ITSI-APM:ServiceKPIs
DA-ITSI-APM:ServiceEndpointKPIs
Locate the KPI base search that corresponds to the impacted service and follow these steps:
Step 1: Determine the monitoring lag.
To determine your recommended lag, select Determine Recommended Lag under Monitoring Lag in a base search. This runs a search and displays the recommended monitoring lag, as well as the maximum, average and minimum indexing lag.
If the recommended lag is less than 300, copy over the recommended monitoring lag into the base search and save. You can skip step 2 in this case. If the recommended lag is greater than or equal to 300, keep the monitoring lag at 30 seconds in the base search and complete step 2.
Learn more about Monitoring Lag.
Step 2: Increase the Calculation Window
In the base search, increase the Calculation Window and KPI Search Schedule to Last 15 minutes and save.
Step 3: Repeats steps for all impacted services' KPI base searches. Repeat steps 1 and 2 for the remaining impacted services' KPI base searches.
Splunk APM Business Workflows are not showing in the Service Analyzer
Problem
After importing Splunk APM Business Workflows using the guided configuration, services are not available in the Service Analyzer.
Cause
There are two possible causes for this:
- The Saved Searches used for importing the Splunk APM Business Workflows are not enabled.
- The services were imported into an ITSI Service Sandbox
Solution
Confirm the Saved Searches used for importing Splunk APM Business Workflows are enabled:
- In Splunk Enterprise go to Settings > Searches, reports, and alerts.
- In the Type dropdown, select All.
- In the App dropdown, select Content Pack for Splunk Observability Cloud (DA-ITSI-CP-splunk-observability)
- In the Owner dropdown, select All.
- Select Edit > Enable for the following searches:
- ITSI Import Objects - Splunk APM Business Workflows
- ITSI Import Objects - Splunk APM Business Workflow Entities
- Use the Splunk Observability - Import Business Workflows dashboard to import the Splunk APM Business Workflow again.
If you are using a version of ITSI which includes ITSI Service Sandboxes (Version 4.17 or later), Splunk APM Business Workflows are imported into a Service Sandbox by default. Check to see if the services were imported into a Service Sandbox.
For ITSI 4.17 (If Service Sandboxes are enabled):
- Select Configuration > Service Sandboxes from the ITSI menu.
- Select the Default Service Sandbox .
- Confirm whether expected Splunk APM Business Workflow Services are available.
For ITSI 4.18 or later:
- Select Configuration > Service Monitoring > Service Sandboxes from the ITSI menu.
- Select Splunk APM Business Workflows: <Splunk Observability Cloud Organization> Service Sandbox.
- Confirm whether expected Splunk APM Business Workflow Services are available.
Splunk APM Business Workflows | KPI reference for the Content Pack for Splunk Observability Cloud |
This documentation applies to the following versions of Content Pack for Splunk Observability Cloud: 3.3.0
Feedback submitted, thanks!