Docs » Analyze services with span tags and MetricSets in Splunk APM » Index span tags to generate Troubleshooting MetricSets

Index span tags to generate Troubleshooting MetricSets ๐Ÿ”—

Use span tags or processes to break down services and inter-service calls along trace characteristics or attributes. To get additional value from a span tag or process, a Splunk APM administrator can run an action known as indexing, which activates additional analysis of the indexed span tag or process. One benefit of indexing is to get aggregated metrics, called MetricSets, across all spans that contain a specified indexed tag or process.

What are MetricSets? ๐Ÿ”—

MetricSets are metric time series (MTS) you can use to track the performance of specific metrics over time. To help you analyze your service performance, APM provides 2 categories of MetricSets: Troubleshooting MetricSets (TMS) and Monitoring MetricSets (MMS). Use Troubleshooting MetricSets for high-cardinality troubleshooting. Use Monitoring MetricSets for real-time monitoring and alerting.

APM provides a collection of MetricSets by default. You can also index additional span tags or processes to generate additional TMS and, as part of the indexing process, you can opt to create custom dimensionalized Monitoring MetricSets (MMS). See Learn about MetricSets in APM for an overview of the types of MetricSets in APM and a list of the MetricSets available by default.

Use this topic to learn how to index a span tag or process to generate Troubleshooting MetricSets. See Generate a Monitoring MetricSet with a custom dimension to learn how to add custom dimensionalized Monitoring MetricSets.

Prerequisites ๐Ÿ”—

Use Troubleshooting MetricSets to filter and breakdown metrics ๐Ÿ”—

Every Troubleshooting MetricSet (TMS) generates the following metrics, also known as Request, Error, and Duration (RED) metrics. The following metrics appear when you select a service from the service map in the Troubleshooting view:

  • Request rate

  • Error rate

  • Root cause error rate

  • p50, p90, and p99 latency

The measurement precision of Troubleshooting MetricSets is 10 seconds. Splunk APM reports quantiles from a distribution of metrics for each 10-second reporting window.

TMS appear on the service map and in Tag Spotlight. Use TMS to filter the service map and create breakdowns across the values of a given indexed span tag or process.

Default indexed span tags ๐Ÿ”—

Splunk APM indexes and generates Troubleshooting MetricSets for several tags by default. For more details about these tags, see Default indexed span tags in APM. You canโ€™t modify or stop APM from indexing these span tags, but you can index additional span tags. See Index a new span tag or process to learn how.

Index a new span tag or process ๐Ÿ”—

Follow these steps to index a span tag or process to create a Troubleshooting MetricSet.

  1. To get to the APM MetricSets page, do one of the following:

    1. In Splunk APM, select APM Configuration and select APM MetricSets from the menu. The APM MetricSets page opens.

    2. From anywhere in Splunk Observability Cloud, select Settings in the navigation bar and select APM MetricSets under Data Configuration.

  2. On the APM MetricSets page, select New MetricSet.

  3. Enter the Name of a span tag or process you want to index.

  4. The Scope determines how APM associates the span tag or process with services in a trace:

    • Select Service to associate the span tag or process with services. This means the value of the span tag or process might change across services in a given trace. Select All Services to index the span tag or process for every service. Select specific services to index the span tag or process for only those services.

      Note

      If you index a span tag or process at the service level, you can also use it as a custom dimension in Monitoring MetricSets. See Generate a Monitoring MetricSet with a custom dimension for instructions.

    • Select Global to associate the span tag or process with traces. This means the value of the span tag or process is the same for all services in a given trace.

    For more information about scope, see Types of span tags you can index.

  5. Select Start Analysis to submit the configuration. Splunk APM runs an analysis of the span tag or process to calculate the potential cardinality contribution of the MetricSet to determine whether it is within your entitlement limit.

  6. Your new metric set shows under Pending MetricSets section of the table with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.

    Caution

    Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.

  7. After the cardinality check has run, review the cardinality and activate your new MetricSet.

    • If the cardinality impact of indexing this new MetricSet is within your subscription limit, you see a green check mark and โ€œWithin Entitlement.โ€ Select the check mark under Actions to activate your Troubleshooting MetricSet.

    • If the cardinality check times out before you activate the MetricSet, run the analysis again.

    • If the cardinality check shows as failed, your cardinality is over your subscription limit for TMS. See Cardinality contribution of indexed span tags and processes.

    The following example shows a cardinality check.

    This image shows the cardinality check for a Monitoring MetricSet.

Cardinality contribution of indexed span tags and processes ๐Ÿ”—

When you index a new span tag or process to generate Troubleshooting MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span tag or process. This gives you control of what you index and helps you to account for organization subscription limits.

If you try to index a span tag or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes by modifying or removing indexed span tags or processes.

To see your TMS subscription limit, go to Settings then Subscription Usage. Depending on your organization subscription, you might need to go to Settings then Billing and Usage. Select the APM tab and then select the Troubleshooting MetricSets panel to view your subscription limit for TMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see Monitor Splunk APM subscription usage.

Manage existing Troubleshooting MetricSets ๐Ÿ”—

After youโ€™ve successfully indexed a span tag or process, Splunk APM saves the configuration in the APM MetricSets page in Data Configuration. Go to this page to view the index scope and status of the span tag or process.

You can modify the configuration for existing indexed tags and processes, including adding and removing services for specific indexed tags or processes and modifying the scope. You can also pause or stop indexing without deleting the configuration. This is useful when you want to temporarily stop indexing a span tag or process, but donโ€™t want to remove the configuration.

To review or modify existing indexed span tags or processes, do the following:

  1. Go to Settings then Data Configuration then APM MetricSets.

  2. Find the indexed span tag or process you want to view under the Custom MetricSets section of the configuration table.

  3. See Status of configured Troubleshooting MetricSets to interpret the status of each indexed span tag or process.

  4. Make any desired changes using the buttons in the Actions column:

    • Use the pencil button to edit the scope of an indexed span tag or process.

    • Use the pause button to pause generating MetricSets for a given span tag or process.

    • Use the trash button to delete a MetricSet configuration.

Status of configured Troubleshooting MetricSets ๐Ÿ”—

After youโ€™ve configured custom Troubleshooting MetricSets you can see their status. The following table lists all available statuses:

Status

Description

Active

The application is indexing the span tag or process to generate Troubleshooting MetricSets.

Paused

You or another administrator paused indexing for the span tag or process. The application isnโ€™t generating Troubleshooting MetricSets for the span tag or process, and you canโ€™t view any data you previously indexed for it.

Stopped

The application stopped indexing the span tag or process and is no longer generating Troubleshooting MetricSets for the span tag or process. You canโ€™t view any data you previously indexed.

Status of Database MetricSets ๐Ÿ”—

You can pause and resume indexing Database Query Performance data from the Standard MetricSets section of APM MetricSets. You can also control the indexing of SQL and NoSQL tags separately.

To pause the indexing of Database Query Performance data, select Pause indexing next to the database type. To resume, select Resume indexing.