Metrics, data points, and metric time series in Splunk Observability Cloud 🔗

In Splunk Observability Cloud, metric data consists of a numerical measurement called a metric, the metric type, and one or more dimensions. Each piece of data in this form is a data point. For example, a data point can be the CPU utilization of host server1 with metric type gauge, metric value 0.7, dimensions "hostname":"server1" and "host_location":"Tokyo", and the timestamp 1557225030000.

A metric time series (MTS) contains all the data points that have the same metric name, metric type, and set of dimensions. Observability Cloud automatically creates MTS from incoming data points. For example, the following data points for the cpu.utilization metric with the same "hostname":"server1" and "location":"Tokyo" dimensions, but with different values and timestamps, make up a single MTS.

%%{ init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#FFFFFF', 'primaryTextColor': '#000000', 'primaryBorderColor': '#000000', 'nodeBorder':'#000000', 'lineColor': '#000000', } } }%% flowchart LR accTitle: Metric time series diagram accDescr: This example metric time series consists of 3 data points. They all share the same metric name, cpu.utilization, the same metric type, gauge, and the same set of dimension key-value pairs, which are hostname:server1 and location:Tokyo. The first data point has a value of .85 and a timestamp of 1557225030000 in UNIX time. The second data point has a value of .9 and a timestamp of 1557225030100 in UNIX time. The third data point has a value of .7 and a timestamp of 1557225030200 in UNIX time. %% LR indicates the direction (left-to-right) %% You can define classes to style nodes and other elements classDef default fill:#FFFFFF,stroke:#000 classDef name fill:#BFFFFF classDef type fill:#ffdda6 classDef value fill:#fff7a1 classDef timestamp fill:#dedeff classDef host fill:#BFFFBF classDef location fill:#FFBFBF %% Each subgraph determines what's in each category subgraph Metric time series direction LR dp0~~~dp1~~~dp2 subgraph dp0[Data point 1] direction LR name0(name: cpu.utilization):::name type0(type: gauge):::type value0(value: .85):::value subgraph dimensions0[Dimensions] direction LR k0(hostname: server1):::host k1(location: Tokyo):::location end timestamp0(timestamp: 1557225030000):::timestamp end subgraph dp1[Data point 2] direction LR name1(name: cpu.utilization):::name type1(type: gauge):::type value1(value: .9):::value subgraph dimensions1[Dimensions] direction LR k2(hostname: server1):::host k3(location: Tokyo):::location end timestamp1(timestamp: 1557225030100):::timestamp end subgraph dp2[Data point 3] direction LR name2(name: cpu.utilization):::name type2(type: gauge):::type value2(value: .7):::value subgraph dimensions2[Dimensions] direction LR k4(hostname: server1):::host k5(location: Tokyo):::location end timestamp2(timestamp: 1557225030200):::timestamp end end

Metrics 🔗

A metric is a measurable number that varies over time. Multiple sources of the same general type, such as host machines, usually report the metric values for a single set of metric names. For example, a server cluster that has 100 host machines might report a single set of metrics named cpu.utilization, api.calls, and dropped.packets, although metric values might be different for each machine.

Note

All metrics and MTS generated by Splunk Observability Cloud start with the prefix sf. or sf_metric.

Metric type 🔗

There are three types of metrics: gauge, cumulative counter, and counter. See more in Metric types.

Metric type	Description	Example
Gauge	Value of a measurement at a specific point in time	CPU utilization percentage of a server
Cumulative counter	Total number of occurrences or items since the measurement began	Total number of Splunk Infrastructure Monitoring API calls served since starting the web server
Counter	Number of new occurrences or items since the last measurement	The number of packets that fail to reach their destinations over each 24-hour period
Histograms	Distribution of measurements across time. Splunk Observability Cloud supports explicit bucket histograms.	Response time (performance) or successful screen loads (availability)

Metric category 🔗

There are about 20 metric categories in Splunk Observability Cloud. Metric category, especially metrics categorized as custom, can impact billing.

Learn all metric categories and how to identify them in Metric categories.

Metric resolution 🔗

By default, Splunk Observability Cloud processes metrics at a 10-second resolution. If metrics have a native resolution courser than 10 seconds, then Splunk Observability Cloud processes the metrics at their native resolution.

Optionally, metrics can be ingested at a higher resolution of 1 second. High-resolution metrics enable exceptionally fine-grained and low-latency visibility and alerting for your infrastructure, applications, and business performance.

Note

To process a metric at high resolution, set the dimension sf_hires to 1 in any MTS.

Metric metadata 🔗

Metrics can have associated metadata such as dimensions, custom properties, or tags. Learn more in Metadata: Dimensions, custom properties, tags, and attributes.

To add or edit dimensions:

Search the Metric Finder and Metadata catalog.
Use the API. See how in our developer portal .

Data points 🔗

A data point contains a metric name and value, the type of the metric, and the dimensions of the metric. Dimensions are the key-value pairs that identify the source of the reported value. Infrastructure Monitoring assumes that incoming data points contain a metric as well as a dimension, or a unique key-value pair that describes some aspect of the metric source.

A data point consists of the following components:

Component	Description	Examples
Metric type	The specified metric type determines the way that Splunk Observability Cloud works with the metric. To learn more about metric types, see Metric types.	One of three metric types: `counter`, `cumulative counter`, or `gauge`.
Metric name	A metric name identifies the values that you send into Infrastructure Monitoring. For example, the AWS metric `4xxErrorRate` represents the percentage of all HTTP requests for which the HTTP status code is 4xx. The data source often determines the metric name, but application receivers and other integrations might map the data source metric name to another name used by Observability Cloud. To learn more about metrics naming constraints, see Naming conventions for metrics and dimensions.	`memory.free`, `CPUUtilization`, `page_visits`
Metric value	The measurement from your system, represented as a number. Metric values must be a signed integer, float, or numeric string in decimal or fixed-point notation. The system stores them as 64-bit integers. See more in the Send Traces, Metrics and Events API documentation.	`99.98751`, `0.7`, `"1.13"`
Dimensions	Key-value pairs that describe some aspect of the source of the metric. A data point can have one or more dimensions. The most common dimension is a source. For example, a dimension can be a host or instance for infrastructure metrics, or it can be an application component or service tier for application metrics. Dimensions are considered metric metadata. To learn more about dimensions, see Metadata: Dimensions, custom properties, tags, and attributes.	`"hostname":"server1"`, `"host_location":"Tokyo"`
Timestamp (Optional)	Either the time that data is sent by the software, or the time at which the data arrives in Observability Cloud. The timestamp is in *nix time in milliseconds.	1557225030000

Metric time series 🔗

A metric time series (MTS) is a collection of data points that have the same metric and the same set of dimensions.

For example, the following sets of data points are in three separate MTS:

MTS1: Gauge metric cpu.utilization, dimension "hostname": "host1"
MTS2: Gauge metric cpu.utilization, dimension "source_host": "host1"
MTS3: Gauge metric cpu.utilization, dimension "hostname": "host2"

MTS 2 has the same host value as MTS 1, but not the same dimension key. MTS 3 has the same host name as MTS 1, but not the same host name value.

Splunk Observability Cloud retains inactive MTS for 13 months.

Use unique dimensions to create independent MTS 🔗

It’s important to configure the Collector or ingest to provide at least one dimension that identifies a unique entity.

For example, when you report on the CPU utilization of 10 hosts in a cluster, the metric is the CPU utilization.

If each host in the cluster shares the exact same dimensions with all the other hosts, the cluster generates only one MTS. As a result, you might have difficultly in differentiating and monitoring the CPU utilization of each individual host in the cluster.

However, if each host in the cluster has at least one unique dimension (typically a unique hostname), the cluster generates 10 MTS, or one for each host. Each MTS represents the CPU utilization over time for a single host.

Related Topics