Troubleshoot the Content Pack for Splunk Observability Cloud

The Content Pack for Splunk Observability Cloud relies on the Splunk Infrastructure Monitoring Add-on for input collection. When troubleshooting, determine whether the issue you are experiencing is relevant to the content pack or to the add-on. In general, if your data successfully reaches your Splunk indexes, the content pack requires troubleshooting. If data isn't reaching your Splunk indexes, then check for configuration problems with the accounts and inputs handled by the add-ons.

Here are some common issues in Content Pack for Observability Cloud and how to resolve them.

The Service Tree isn't lighting up or not all entities are displaying for KPIs

Problem

The Service Tree doesn't light up or not all entities are displaying for the KPIs.

Cause

KPIs are scheduled to run every 5 minutes with a 5-minute calculation window. If data is being sent to Splunk at an interval greater than 5 minutes, it might cause the entity to drop from the service tree.

Solution

In ITSI or IT Essentials Work, go to Configuration > KPI Base Searches. These are the KPI base searches used in this content pack:

• RUM_App_Base
• RUM_Browser_Base
• RUM_Synthetics_Base
• SIM_cloud_aws_ec2
• SIM_cloud_aws_lambda
• SIM_cloud_azure_functions
• SIM_cloud_azure_vm
• SIM_cloud_gcp_compute
• SIM_cloud_gcp_functions
• SIM_containers
• SIM_data_center_hosts
• SIM_kubernetes
• SplunkAPM Rate Base Search
• SSM_api_tests
• SSM_browser_tests
• SSM_http_tests

Locate the KPI base search that corresponds to the impacted service and follow these steps:

Step 1: Determine the monitoring lag.

To determine your recommended lag, select Determine Recommended Lag under Monitoring Lag in a base search. This runs a search and displays the recommended monitoring lag, as well as the maximum, average and minimum indexing lag.

If the recommended lag is less than 300, copy over the recommended monitoring lag into the base search and save. You can skip step 2 in this case. If the recommended lag is greater than or equal to 300, keep the monitoring lag at 30 seconds in the base search and complete step 2.

Learn more about Monitoring Lag.

Step 2: Increase the Calculation Window

In the base search, increase the Calculation Window and KPI Search Schedule to Last 15 minutes and save.

Step 3: Repeats steps for all impacted services' KPI base searches. Repeat steps 1 and 2 for the remaining impacted services' KPI base searches.