You can monitor Kubernetes metrics with Splunk Observability Cloud. Splunk Observability Cloud uses the Splunk Distribution of OpenTelemetry Collector for Kubernetes to provide robust infrastructure monitoring capabilities.
You can also export and monitor data related to your Kubernetes clusters, as described in the following table.
Use the Kubernetes navigator
Note
Kubernetes version 1.21 and higher are compatible with the Kubernetes navigator. Using lower versions of Kubernetes might result in the navigator not displaying all clusters. See endoflife.date for more information.
View the health of entire Kubernetes clusters at a glance from the Infrastructure page. From the Infrastructure page, you can drill down into and analyze detailed metrics about these Kubernetes resources:
Nodes
Pods
Containers
Workloads
You can use the Kubernetes navigator to obtain a real-time, at-a-glance view of the overall health and performance of your Kubernetes environment. You also have visibility all the way through the stack as you drill down and across elements of your environment, reflecting the fact that the infrastructure, Kubernetes control plane, containers, applications, and services are all related layers, not just individual system components.
When you navigate to the Kubernetes navigator from the landing page of Infrastructure Monitoring, the default view is Cluster Map. You can switch to other views to see information about your clusters, nodes, pods, containers, and workloads by selecting a panel from the navigator selection bar:
Map: Visualize the entire cluster, and drill down into nodes, pods, and containers with the Map view. The Map view lets you explore a cluster and visualize the health of everything at a glance. Nodes, pods, and containers are colored by health and status, as reported by Kubernetes.
Nodes: Display a compact list of all the nodes in your Kubernetes cluster. Use the Nodes view to see the health and status of all nodes at once.
Workloads: Display a compact list of all the workloads running in a selected cluster. Use the Workloads view to see the health and status of all workloads at once.
Node Detail: Display detailed charts about a selected node is a cluster. Use the Node Detail view when investigating an incident to get specific details.
Workload Detail: Display detailed information about a selected workload in a cluster. Use the Workload Detail view when investigating an incident to get specific details.
Pod Detail: Display detailed information about a selected pod in a cluster. Use the Pod Detail view when investigating an incident to get specific details.
Container Detail: Display properties of a selected container in a cluster. Use the Container Detail view when investigating an incident to get specific details.
Follows these steps to monitor and analyze Kubernetes from the Infrastructure page:
Select Navigation menu > Infrastructure.
Select Kubernetes from the Platforms menu.
Specify the cluster you want to view by clicking on the map or selecting it by name from the filter bar.
Select a node to view more details about it in the Info panel.
Select a node, or hover and click the magnifying glass to visualize pods and containers in the node.
Select a pod or container to view more details about it in the Info panel.
By default, you see data from the last 3 hours. You can use the time picker to choose a new time range. When you select a new time range, the map and all pages update to show the status of nodes, pods, and containers present during that time. Streaming metrics charts also update to show the time range you selected.
Depending on your view selection, the page displayed might be divided into a main area on the left and a sidebar on the right.
The main area
The Kubernetes navigator automatically discovers the full hierarchy of elements–clusters, nodes, pods, containers–and their associated metadata, as well as the workloads running in them. As that information is streamed through Infrastructure Monitoring, the Kubernetes navigator dynamically produces interactive cluster maps, builds detailed node and workload lists, and populates built-in performance dashboards.
To explore the main area, you can take the following actions:
Hover: Hover over an element to display a border and to open a tooltip that displays information about the element, including the element’s condition or phase, if applicable.
Zoom: Hover a cluster or node to display a magnifying glass icon in the upper left corner of the element. Click the magnifying glass icon to apply the filter and change the zoom level of the map.
Select: Click an element on the map to drill down for more detail. Details about the element display in the Info panel of the sidebar on the right. Selecting an element on the map does not change the zoom level or filters on the map, but does add a border around the selected element.
Filter: Filter the map by any available metadata in your Kubernetes data, such as a namespace, a workload, or any other key-value pair. When you filter the map, you see nodes that match the filter. Nodes that do not match the filter are hidden. Pods and containers that match the filter are highlighted, and non-matching pods and containers are dimmed. You can still select the dimmed pods and containers to view details about them in the Info panel.
Drill down in the Kubernetes navigator
When you zoom into a single cluster, you can focus on just that cluster and the Analyzer (Cluster Map only) panel displays suggested filters for that specific cluster. When you zoom into a node element, the Analyzer panel displays suggested filters for that specific node and the Info panel displays charts related to that element.
Map
The Map view displays your Kubernetes infrastructure in an interactive cluster map. Select elements in the map to explore data about each of those elements, represented by various charts in the Info tab. The level of detail shown on the map is dynamic and depends on the number of elements shown, either specified through the filters or whether you zoom in on to drill down for more detail.
Note
The color or statistics for an element, such as a pod, might change as you drill down or click through your system, because the information, such as, the state of the pod or its memory consumption statistics, might refresh between the time you start navigation and the time a target element appears.
Explore the cluster map:
Nodes are colored by condition (ready | not ready | pressure | etc.)
Pods are colored by phase (running | pending | failed | etc.)
Containers are colored by status (ready | not ready)
Info
The Info panel in the sidebar displays details about a workload, a node, a pod, or a container that you selected from the main area. The top chart shows metadata about the selected element, similar to the results that the Kubernetes kubectl command returns. The other charts are a combination of table charts, which list related objects in the selected element, and area charts, which display infrastructure metrics. Click links to related objects in the Info panel to drill down through the sidebar content without losing context.
To view the data in the Info panel using the entire width of the screen, click the fullscreen icon at the top left of the Info panel to navigate to the corresponding Detail page.
Analyzer (Cluster Map only)
Available in the Enterprise Edition.
The Analyzer panel helps you troubleshoot Kubernetes problems at scale by highlighting Kubernetes objects that are in a bad state such as nodes that are not ready. Then, the Analyzer produces theories about what those objects might have in common. For example, that all of the objects are running the same workload or all objects are located in the same AWS region. Click on a finding in the Analyzer panel to filter the map.
The Analyzer panel in the sidebar displays suggested filters about the elements selected in the cluster map. Click links in the Analyzer panel to add filters to the cluster map and explore interesting conditions across your entire Kubernetes environment.
The Analyzer uses AI-driven insights to examine patterns that nodes, pods, or containers could have in common. Trouble indicators are:
pods that are in pending status
pods that are in failed status
pods with unknown condition
containers with high restart counts
nodes not ready
nodes with unknown condition
nodes experiencing high CPU
nodes experiencing high memory
The Analyzer displays overrepresented metrics properties for known conditions, such as pods in pending status, pods in failed status, and so on. You can use properties that are highly correlated with these conditions to filter the cluster map. You can explore data about each of those elements in the Info tab, allowing you to identify the underlying patterns noticeable on the filtered map that might be correlated with Kubernetes issues (for example, all failed pods only being in certain types of clusters) and provides suggested paths to follow for troubleshooting such issues.
List panels
The following list panels provide compact lists of node or workload elements in your Kubernetes environment.
Nodes
The Nodes panel displays a compact list of all the nodes in your cluster, along with the infrastructure metrics, basic configuration, and health indicators such as CPU used, memory used, disk used, and so forth, for each node in your selected Kubernetes clusters. This list makes it easy to see hot spots. You can sort or group this list by available keys.
Click the name of a node to open the sidebar and drill down to details without losing context.
Workloads
The Workloads panel displays a compact list of all the workloads running in a selected cluster, along with the metadata and infrastructure metrics for each workload. You can specify a different cluster, namespace, or workload type. You can sort or group this list by available keys.
If you are a Splunk Application Performance Monitoring customer, you can go from Kubernetes Navigator to Splunk APM to view, understand, and explore the relationship between various infrastructure objects and the services running on them. Click a service name to navigate to the APM built-in service dashboard.
Click the name of a workload to open the sidebar and drill down to details without losing context.
Detail panels
The following detail panels display metadata, infrastructure metrics, and events from the elements (node, workload, pod, or container) of your environment that are specified through the filters at the top of the page or that you selected from another page. You can learn properties about the element such as what is running on it, what related alerts have triggered, and what kind of trends exist.
Node Detail
The Node Detail panel displays detailed information about a selected node, including additional properties, workloads running on the node, containers on this node, and so on. The properties in the upper left are metadata about the node. If desired, you can specify a different cluster or node. The status of the workloads helps you understand the health of the workloads.
Click around on workloads and containers on the node to open the sidebar and drill down to details on these elements without losing context. You can search or group by workload or container for this node. Recent trigger and clear events appear in the Node Events chart. You can sort this list by available keys.
Workload Detail
The Workload Detail panel displays detailed information about a selected workload. The properties in the upper left are metadata about the workload. If desired, you can specify a different cluster, namespace, or workload type.
Click around on pods and nodes to open the sidebar and drill down to details on these elements without losing context. You can search or group by the pod list for this workload. Recent trigger and clear events appear in the Workload Events chart. You can sort these lists by available keys.
Pod Detail
The Pod Detail panel displays detailed information about a selected pod, including its containers. Use this view to track the activity on one pod or across all pods in your cluster. The properties in the upper left are metadata about the pod. If desired, you can specify a different cluster, node, or pod.
Click around on containers in the pod to open the sidebar and drill down to details without losing context. You can search or group by the container list for this pod. Recent trigger and clear events appear in the Pod Events chart. You can sort this list by available keys.
Container Detail
The Container Detail panel displays detailed information about a selected container. The properties in the upper left are metadata about the container. If desired, you can specify a different cluster or container. You can also specify the metric graphs you want to see by applying a filter to the metrics. Recent trigger and clear events appear in the Container Events chart.