KPI reference for the Content Pack for Splunk Observability Cloud

The Content Pack for Splunk Observability Cloud contains Applications Performance Monitoring services, Infrastructure Monitoring services, Real User Monitoring, and Synthetic Checks services. For each service, there are KPIs used to help you monitor the health of your applications. All parent and child services report up to the overall Splunk Observability Cloud service at the top level.

Applications Performance Monitoring services and KPIs

There are 4 Application Performance Monitoring services. This includes a service for Application Error Rate, Application Rate (Throughput), and Application Duration. Each of these roll up to a top-level Application Performance Monitoring service. For each of these services, there are KPIs used to help you monitor the health of your applications.

Service	KPI	Description
Application Duration	SplunkAPM Duration	Shows the combined application response time (duration) metrics from all ingested APM vendor metrics. Thresholds: Low = 0-10 ms, Medium = 10-15 ms, High > 15 ms
Application Error Rate	Splunk APM Error Rate	Shows the combined percentage of requests with errors over total requests for the application from all Splunk APM Services reported metrics. Thresholds: Normal <= 0.01 < Low < 25 < Medium < 50 < High < 90 < Critical
Application Rate (Throughput)	SplunkAPM Rate (Throughput)	Shows the combined application requests per second (Rate) from all ingested Splunk APM service metrics. Thresholds: No predefined threshold. For steps to configure KPI thresholds, see Configure KPI thresholds in ITSI in the Service Insights manual.

Infrastructure Monitoring services and KPIs

There are 13 Infrastructure Monitoring services. This includes a service per entity type – OS Hosts, GCP Compute Engine, GCP Cloud Functions, Azure Functions. Each of these roll up to a corresponding parent service, which roll up to a top-level Infrastructure Monitoring service. For each of these services, there are KPIs used to help you monitor the health of your applications.

AWS services and KPIs

Service	KPI	Description
AWS EC2	AWS EC2 CPU Ultilization	The percentage of allocated EC2 compute units that are currently in use on the instance. Thresholds: High = 90, Medium = 80, Low = 50
	AWS EC2 Disk Read Bytes	The number of bytes read from all instance store volumes available to the instance.
	AWS EC2 Disk Read Operations	The completed read operations from all instance store volumes available to the instance in a specified period of time.
	AWS EC2 Disk Write Bytes	The completed write operations to all instance store volumes available to the instance in a specified period of time.
	AWS EC2 Disk Write Operations	The completed write operations to all instance store volumes available to the instance in a specified period of time.
	AWS EC2 Network In	The number of bytes received by the instance on all network interfaces. This metric identifies the volume of incoming network traffic to a single instance. Thresholds: High = 0, Normal = 10
	AWS EC2 Network Out	The number of bytes sent out by the instance on all network interfaces. This metric identifies the volume of outgoing network traffic from a single instance. Thresholds: High = 0, Normal = 10
	AWS EC2 Network Packets In	The number of packets received by the instance on all network interfaces. This metric identifies the volume of incoming traffic in terms of the number of packets on a single instance. Thresholds: High = 0, Normal = 10
	AWS EC2 Network Packets Out	The number of packets sent out by the instance on all network interfaces. This metric identifies the volume of outgoing traffic in terms of the number of packets on a single instance. Thresholds: High = 0, Normal = 10
AWS Lambda	AWS Lambda Duration	The amount of time that your function code spends processing an event. The billed duration for an invocation is the value of duration rounded up to the nearest millisecond. Thresholds: High = 800,000, Medium = 400,000, Low = 200,000
	AWS Lambda Errors	The number of invocations that result in a function error. Function errors include exceptions thrown by your code and exceptions thrown by the Lambda runtime. Thresholds: High = 1, Normal = 0
	AWS Lambda Invocations	The number of times your function code is executed, including successful executions and executions that result in a function error. Invocations aren't recorded if the invocation request is throttled or otherwise resulted in an invocation error.
	AWS Lambda Throttles	The number of invocation requests that are throttled. When all function instances are processing requests and no concurrency is available to scale up, Lambda rejects additional requests with TooManyRequestsException. Throttled requests and other invocation errors don't

Azure services and KPIs

Service	KPI	Description
Azure Functions	Azure Functions 5xx Errors	The count of 5xx errors from the Azure Function handler. Thresholds: High = 1, Normal = 0
	Azure Functions Bytes Received	The rate at which the function process is receiving bytes to I/O operations. Thresholds: High = 0, Normal = 100
	Azure Functions Bytes Sent	The rate at which the function process is sending bytes to I/O operations. Thresholds: High = 0, Normal = 100
	Azure Functions Executions	The number of times your function has executed. This value correlates to the number of times a function runs in your app.
	Azure Functions Memory	The current amount of memory used by an Azure function.
	Azure Functions Usage Cost	The function cost in terms of function execution units which are a combination of execution time and your memory usage.
Azure VM	Azure VM CPU Percentage	The percentage of allocated compute units that are currently in use by the virtual machine. Thresholds: High = 90, Medium = 80, Low = 50
	Azure VM Disk Read Bytes	The bytes read from disk during monitoring period.
	Azure VM Disk Write Bytes	The bytes written to disk during monitoring period.
	Azure VM Network In	The number of bytes received on all network interfaces by the virtual machine. Thresholds: High = 0, Normal = 10
	Azure VM Network Out	The number of bytes out on all network interfaces by the virtual machine. Thresholds: High = 0, Normal = 10

GCP services and KPIs

Service	KPI	Description
GCP Cloud Functions	GCP Cloud Functions Active	The number of active function instances.
	GCP Cloud Functions Execution Time	The distribution of functions execution times in nanoseconds. Thresholds: High = 800,000, Medium = 400,000, Low = 200,000
	GCP Cloud Functions Executions	The count of function executions.
	GCP Cloud Functions Memory	The distribution of maximum functions' memory usage during execution, in bytes.
	GCP Cloud Functions Network Egress	The outgoing network traffic of function, in bytes.
GCP Compute Engine	GCP Compute Engine CPU Utilization	The fractional utilization of allocated CPU on this instance. Thresholds: High = 90, Medium = 80, Low = 50
	GCP Compute Engine Network Bytes In	The count of bytes received from the network. Thresholds: High = 0, Normal = 10
	GCP Compute Engine Network Bytes Out	The count of bytes sent over the network. Thresholds: High = 0, Normal = 10
	GCP Compute Engine Disk Read Bytes	The count of bytes read on disk.
	GCP Compute Engine Disk Write Bytes	The count of bytes written on disk.

My Data Center Hosts services and KPIs

Service	KPI	Description
OS Hosts	CPU Utilization	CPU Utilization by % Thresholds: High = 90, Medium = 80, Low = 50
	Memory Free	Memory free in megabytes.
	Disk Write IOps	Number of write IOps by a given instance.
	Disk Read IOps	Number of read IOps by a given instance.
	Network Rx	Network octets received.
	Network Tx	Network octets sent.
Kubernetes Pods	CPU Utilization	Pod CPU Utilization by % Thresholds:High = 90, Medium = 80, Low = 50
	Memory Utilization	Pod Memory Utilization by %
	Network Rx Errors	Pod Network Receive Errors
	Network Tx Errors	Pod Network Transmit Errors
Docker Containers	CPU Utilization	Thresholds:High = 90, Medium = 80, Low = 50
	Disk Read	Disk Read
	Disk Write	Disk Write
	Memory Free	Memory free in bytes

Splunk RUM services and KPIs

There are 6 Splunk Real User Monitoring (RUM)-related services. This includes two services each for RUM Browser, RUM Mobile, and RUM Synthetics.

Service	KPI	Description
RUM Browser Metrics	Front-end requests	Front-end requests by browser.
	Front-end errors	Front-end errors by browser.
	Document load latency (P75)	Document load latency in ms by browser (P75).
	Endpoint Requests	Browser Endpoint Requests by browser.
	Endpoint Latency (P75)	Browser Endpoint Latency in ms by browser (P75).
RUM Browser Web Vitals	Largest Contentful Paint in ms (P75)	Largest Contentful Paint in ms by browser (P75).
	Cumulative Layout Shift (P75)	Cumulative Layout Shift by browser (P75).
	First Input Delay in ms (P75)	First Input Delay in ms by browser (P75).
RUM Mobile Metrics	Front-end requests	Front-end requests by app
	Document load latency (P75)	Document load latency in ms by app (P75).
	Endpoint Requests	Mobile Endpoint Requests by App.
	Endpoint Latency (P75)	Mobile Endpoint Latency in ms by App (P75).
RUM Mobile App Metrics	App crash count	Count of app crashes by app
	App error	Count of app errors by app
	App Cold Start in ms (P75)	App Cold Start times in ms (P75).
	App Cold Starts	Count of app cold starts.
RUM Synthetic Metrics	Front-end requests	Front-end requests by Splunk Synthetics.
	Front-end errors	Splunk Synthetics Front-end errors by browser.
	Document load latency (P75)	Splunk Synthetics Document load latency in ms by browser (P75).
	Endpoint Requests	Splunk Synthetics Endpoint Requests by browser.
	Endpoint Latency (P75)	Splunk Synthetics Endpoint Latency in ms by browser (P75).
RUM Synthetic Web Vitals	Largest Contentful Paint in ms (P75)	Splunk Synthetics Largest Contentful Paint in ms by browser (P75).
	Cumulative Layout Shift (P75)	Splunk Synthetics Cumulative Layout Shift by browser (P75).
	First Input Delay in ms (P75)	Splunk Synthetics First Input Delay in ms by browser (P75)

Synthetic Checks services and KPIs

There are 7 Synthetic Checks services. This includes a service per entity type or synthetic check, as well as a top-level synthetic check. For each of these services, there are KPIs used to help you monitor the health of your web applications. There is also an example service that you can use to create a custom configuration.

Note, these KPIs don't have predefined thresholds configured. For steps to configure KPI thresholds, see Configure KPI thresholds in ITSI in the Service Insights manual.

Service	KPI	Description
API Checks	Failure	Indicates the endpoint failed to respond.
	Response Code	The response code that is returned by the API endpoint.
	Response Time	The elapsed time for the response by the API endpoint.
Benchmark Checks	Number of Competitors	The number of competitors the page is checked against.
	Rank	The rank against all competitors.
	Score	A generated score from 0-100 that relates performance relative to the competition. It's calculated based on performance across various metrics. Rank is based on this score. Higher is better.
Content Checks	Broken Links	The count of links returning in HTTP error codes in the 400 range.
	Links Checked	The count of links checked on the webpage.
	Warning Links	The count of links returning HTTP codes in the 300 range.
HTTP Checks	Days Since Last Error	The count of days since the last error.
	Failure	Indicates the webpage failed to respond.
	Response Code	The response code that is returned by the web application.
	Response Time	The elapsed time for the response by the web application.
Browser Checks	Failure	Indicates the webpage failed to respond when accessed using the Chrome web browser.
	Response Code	The response code that is returned by the webpage when accessed using the Chrome web browser.
	Response Time	The elapsed time for the response by the webpage when accessed using the Chrome web browser.