Troubleshoot the Content Pack for Splunk Observability Cloud

The Content Pack for Splunk Observability Cloud relies on the Splunk Infrastructure Monitoring Add-on for input collection. When troubleshooting, determine whether the issue you are experiencing is relevant to the content pack or to the add-on. In general, if your data successfully reaches your Splunk indexes, the content pack requires troubleshooting. If data isn't reaching your Splunk indexes, then check for configuration problems with the accounts and inputs handled by the add-ons.

Here are some common issues in Content Pack for Splunk Observability Cloud and how to resolve them.

The Service Tree isn't lighting up or not all entities are displaying for KPIs

Problem

The Service Tree doesn't light up or not all entities are displaying for the KPIs.

Cause

KPIs are scheduled to run every 5 minutes with a 5-minute calculation window. If data is being sent to Splunk at an interval greater than 5 minutes, it might cause the entity to drop from the service tree.

Solution

In ITSI or IT Essentials Work, go to Configuration > KPI Base Searches. These are the KPI base searches used in this content pack:

• RUM_App_Base
• RUM_Browser_Base
• RUM_Synthetics_Base
• SIM_cloud_aws_ec2
• SIM_cloud_aws_lambda
• SIM_cloud_azure_functions
• SIM_cloud_azure_vm
• SIM_cloud_gcp_compute
• SIM_cloud_gcp_functions
• SIM_containers
• SIM_data_center_hosts
• SIM_kubernetes
• SplunkAPM Rate Base Search
• SSM_api_tests
• SSM_browser_tests
• SSM_http_tests
• DA-ITSI-APM:BusinessWorkflows
• DA-ITSI-APM:ServiceKPIs
• DA-ITSI-APM:ServiceEndpointKPIs

Locate the KPI base search that corresponds to the impacted service and follow these steps:

Step 1: Determine the monitoring lag.

To determine your recommended lag, select Determine Recommended Lag under Monitoring Lag in a base search. This runs a search and displays the recommended monitoring lag, as well as the maximum, average and minimum indexing lag.

If the recommended lag is less than 300, copy over the recommended monitoring lag into the base search and save. You can skip step 2 in this case. If the recommended lag is greater than or equal to 300, keep the monitoring lag at 30 seconds in the base search and complete step 2.

Learn more about Monitoring Lag.

Step 2: Increase the Calculation Window

In the base search, increase the Calculation Window and KPI Search Schedule to Last 15 minutes and save.

Step 3: Repeats steps for all impacted services' KPI base searches. Repeat steps 1 and 2 for the remaining impacted services' KPI base searches.

Splunk APM Business Workflows are not showing in the Service Analyzer

Problem

After importing Splunk APM Business Workflows using the guided configuration, services are not available in the Service Analyzer.

Cause

There are two possible causes for this:

• The Saved Searches used for importing the Splunk APM Business Workflows are not enabled.
• The services were imported into an ITSI Service Sandbox

Solution

Confirm the Saved Searches used for importing Splunk APM Business Workflows are enabled:

1. In Splunk Enterprise go to Settings > Searches, reports, and alerts.
2. In the Type dropdown, select All.
3. In the App dropdown, select Content Pack for Splunk Observability Cloud (DA-ITSI-CP-splunk-observability)
4. In the Owner dropdown, select All.
5. Select Edit > Enable for the following searches:
1. ITSI Import Objects - Splunk APM Business Workflows
2. ITSI Import Objects - Splunk APM Business Workflow Entities
6. Use the Splunk Observability - Import Business Workflows dashboard to import the Splunk APM Business Workflow again.

If you are using a version of ITSI which includes ITSI Service Sandboxes (Version 4.17 or later), Splunk APM Business Workflows are imported into a Service Sandbox by default. Check to see if the services were imported into a Service Sandbox.

For ITSI 4.17 (If Service Sandboxes are enabled):

1. Select Configuration > Service Sandboxes from the ITSI menu.
2. Select the Default Service Sandbox .
3. Confirm whether expected Splunk APM Business Workflow Services are available.

For ITSI 4.18 or later:

1. Select Configuration > Service Monitoring > Service Sandboxes from the ITSI menu.
2. Select Splunk APM Business Workflows: <Splunk Observability Cloud Organization> Service Sandbox.
3. Confirm whether expected Splunk APM Business Workflow Services are available.