Troubleshoot cardinality in Monitoring MetricSets 🔗
Follow these steps to troubleshoot high cardinality issues and Monitoring MetricSet configurations.
You’ve reached your subscription limit or you’re unable to create new Monitoring MetricSets 🔗
Cause: Your cardinality exceeds your subscription limit 🔗
If the cardinality of your Monitoring MetricSets (MMS) is high, you might reach your subscription limit. If a new MMS you are creating exceeds your subscription limit you are unable to create the new Monitoring MetricSet. To see your MMS subscription limit, go to Settings then Subscription Usage. Depending on your org subscription model, this might be Settings then Billing and Usage. Select the APM tab and then select the Monitoring MetricSets panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see Monitor Splunk APM billing and subscription usage.
Solution: Configure your MMS to be more efficient 🔗
If you have high-cardinality metrics that exceed the system limit or the subscription limit for your contract, try these methods to reduce cardinality for your MMS.
Note
The following options for reducing cardinality are not available for inferred-service MMS.
Turn off an endpoint MetricSet by choosing Service MMS only.
Add a filter by tag value to reduce the number of results.
Remove or filter existing Monitoring MetricSets. The cardinality check in APM estimates your total cardinality of existing MMS in addition to your proposed changes.
MetricSets with zero cardinality are not valid. If you have zero cardinality, verify that your configuration details are correct or try again after you are ingesting the relevant tracing data.
Try to run the cardinality check again if the Monitoring MetricSet timed out.
Provide a list of tag values or a regular expression in the MMS.
For more information on troubleshooting high and low cardinality, see Guidelines for working with low and high cardinality data.
Solution: Make sure you chose the correct service or endpoint-level MetricSets 🔗
The following table outlines the difference between endpoint and service level MetricSets.
Note
The following options for reducing cardinality are not available for inferred-service MMS.
Endpoint-level MetricSet |
Service-level MetricSet |
|---|---|
|
|
The MMS at the service level are an aggregation of the metric sets at the endpoint-level. If you specify the endpoints that are of interest to you in your Monitoring MetricSet, then you can reduce the amount of data.
However, limiting the number of endpoints in your Monitoring MetricSet isn’t always a guaranteed way to reduce your cardinality.
Solution: Apply tag filters 🔗
Note
The following option for reducing cardinality are not available for inferred-service MMS.
In this example, the custom dimension is customer.id.
Suppose you want to track a checkout workflow on your application. In your environment, this workflow is called the checkout service. You create a Monitoring MetricSet with 2 endpoints, add-to-cart, and checkout. However, you find that your cardinality is high for the Monitoring MetricSet for these 2 endpoints because there are 10,000 unique customer IDs for the customer.id tag associated with these endpoints. To reduce the overall cardinality for this Monitoring MetricSet, you can filter by tag values to a smaller subset of customer IDs that are of interest to you instead of needlessly processing 10,000 unique tags.