Kubernetes cluster receiver 🔗
The Kubernetes cluster receiver collects cluster metrics using the Kubernetes API server. You can use a single instance of this receiver to monitor an entire Kubernetes cluster. The supported pipeline type is metrics. To filter in or out other Kubernetes elements, such as containers, pods, nodes, namespaces, or clusters, use the Filter processor instead. Learn more at Filter processor. See Process your data with pipelines for more information on the different types of pipelines.
Kubernetes version 1.21 and higher is compatible with the Kubernetes navigator. Using lower versions of Kubernetes is not supported for this receiver and might result in the navigator not displaying all clusters.
Note
This receiver replaces the kubernetes-cluster Smart Agent monitor.
Get started 🔗
By default, the Kubernetes cluster receiver is already activated in the Helm chart of the Splunk OpenTelemetry Collectors. See Configure the Collector for Kubernetes with Helm for more information, including the default Collected metrics and dimensions for Kubernetes.
To activate the Kubernetes cluster receiver manually in the Collector configuration, add k8s_cluster to the receivers section of your configuration file, as shown in the following example:
receivers:
k8s_cluster:
auth_type: kubeConfig
collection_interval: 30s
node_conditions_to_report: ["Ready", "MemoryPressure"]
allocatable_types_to_report: ["cpu","memory"]
metadata_exporters: [signalfx]
To complete the configuration, include the receiver in the metrics pipeline of the service section of your
configuration file. For example:
service:
pipelines:
metrics:
receivers: [k8s_cluster]
Sync metadata_exporters 🔗
Use metadata_exporters as a list of metadata exporters to sync with metadata collected by the Kubernetes cluster receiver. For example:
receivers:
k8s_cluster:
auth_type: serviceAccount
metadata_exporters:
- signalfx
Exporters specified in this list need to implement the following interface. If an exporter doesn’t implement the interface, startup fails.
type MetadataExporter interface {
ConsumeMetadata(metadata []*MetadataUpdate) error
}
type MetadataUpdate struct {
ResourceIDKey string
ResourceID ResourceID
MetadataDelta
}
type MetadataDelta struct {
MetadataToAdd map[string]string
MetadataToRemove map[string]string
MetadataToUpdate map[string]string
}
Set node_conditions_to_report 🔗
Use the following configuration to have the k8s_cluster receiver emit two metrics, k8s.node.condition_ready and k8s.node.condition_memory_pressure, one for each condition in the configuration:
# ...
k8s_cluster:
node_conditions_to_report:
- Ready
- MemoryPressure
# ...
The value is 1 if the ConditionStatus for the corresponding Condition is True, 0 if it’s False, and -1 if it’s Unknown. To learn more, search for “Conditions” in the Kubernetes documentation.
Settings 🔗
The following table shows the configuration options for the receiver:
| Name | Type | Default | Description |
|---|---|---|---|
auth_type | string | serviceAccount | How to authenticate to the K8s API server. This can be one of |
collection_interval | int64 | 10s | Collection interval for metrics. |
node_conditions_to_report | slice | Node condition types to report. See all condition types, see here: https://kubernetes.io/docs/concepts/architecture/nodes/#condition. | |
allocatable_types_to_report | slice | Allocate resource types to report. See all resource types, see here: https://kubernetes.io/docs/concepts/architecture/nodes/#capacity | |
metadata_exporters | slice | List of exporters to which metadata from this receiver should be forwarded to. | |
distribution | string | kubernetes | Whether OpenShift supprot should be enabled or not. |
Metrics 🔗
Caution
The SignalFx exporter excludes some available metrics by default. Learn more about default metric filters in List of metrics excluded by default. See Collected metrics and dimensions for Kubernetes to see how the Collector processes Kubernetes metrics.
The following metrics, resource attributes, and attributes are available. For more information, including which resource attributes are included by default, refer to Kubernetes Cluster receiver metadata in GitHub.
| Name | Type | Unit | Active | Description | Attributes |
|---|---|---|---|---|---|
k8s.container.cpu_request | Gauge | {cpu} | Yes | Resource requested for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.cpu_limit | Gauge | {cpu} | Yes | Maximum resource limit set for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.memory_request | Gauge | By | Yes | Resource requested for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.memory_limit | Gauge | By | Yes | Maximum resource limit set for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.storage_request | Gauge | By | Yes | Resource requested for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.storage_limit | Gauge | By | Yes | Maximum resource limit set for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.ephemeralstorage_request | Gauge | By | Yes | Resource requested for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.ephemeralstorage_limit | Gauge | By | Yes | Maximum resource limit set for the container. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core for details | |
k8s.container.restarts | Gauge | {restart} | Yes | How many times the container has restarted in the recent past. This value is pulled directly from the K8s API and the value can go indefinitely high and be reset to 0 at any time depending on how your kubelet is configured to prune dead containers. It is best to not depend too much on the exact value but rather look at it as either == 0, in which case you can conclude there were no restarts in the recent past, or > 0, in which case you can conclude there were restarts in the recent past, and not try and analyze the value beyond that. | |
k8s.container.ready | Gauge | Yes | Whether a container has passed its readiness probe (0 for no, 1 for yes) | ||
k8s.pod.phase | Gauge | Yes | Current phase of the pod (1 - Pending, 2 - Running, 3 - Succeeded, 4 - Failed, 5 - Unknown) | ||
k8s.pod.status_reason | Gauge | No | Current status reason of the pod (1 - Evicted, 2 - NodeAffinity, 3 - NodeLost, 4 - Shutdown, 5 - UnexpectedAdmissionError, 6 - Unknown) | ||
k8s.deployment.desired | Gauge | {pod} | Yes | Number of desired pods in this deployment | |
k8s.deployment.available | Gauge | {pod} | Yes | Total number of available pods (ready for at least minReadySeconds) targeted by this deployment | |
k8s.cronjob.active_jobs | Gauge | {job} | Yes | The number of actively running jobs for a cronjob | |
k8s.daemonset.current_scheduled_nodes | Gauge | {node} | Yes | Number of nodes that are running at least 1 daemon pod and are supposed to run the daemon pod | |
k8s.daemonset.desired_scheduled_nodes | Gauge | {node} | Yes | Number of nodes that should be running the daemon pod (including nodes currently running the daemon pod) | |
k8s.daemonset.misscheduled_nodes | Gauge | {node} | Yes | Number of nodes that are running the daemon pod, but are not supposed to run the daemon pod | |
k8s.daemonset.ready_nodes | Gauge | {node} | Yes | Number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready | |
k8s.hpa.max_replicas | Gauge | {pod} | Yes | Maximum number of replicas to which the autoscaler can scale up. | |
k8s.hpa.min_replicas | Gauge | {pod} | Yes | Minimum number of replicas to which the autoscaler can scale up. | |
k8s.hpa.current_replicas | Gauge | {pod} | Yes | Current number of pod replicas managed by this autoscaler. | |
k8s.hpa.desired_replicas | Gauge | {pod} | Yes | Desired number of pod replicas managed by this autoscaler. | |
k8s.job.active_pods | Gauge | {pod} | Yes | The number of actively running pods for a job | |
k8s.job.desired_successful_pods | Gauge | {pod} | Yes | The desired number of successfully finished pods the job should be run with | |
k8s.job.failed_pods | Gauge | {pod} | Yes | The number of pods which reached phase Failed for a job | |
k8s.job.max_parallel_pods | Gauge | {pod} | Yes | The max desired number of pods the job should run at any given time | |
k8s.job.successful_pods | Gauge | {pod} | Yes | The number of pods which reached phase Succeeded for a job | |
k8s.namespace.phase | Gauge | Yes | The current phase of namespaces (1 for active and 0 for terminating) | ||
k8s.replicaset.desired | Gauge | {pod} | Yes | Number of desired pods in this replicaset | |
k8s.replicaset.available | Gauge | {pod} | Yes | Total number of available pods (ready for at least minReadySeconds) targeted by this replicaset | |
k8s.replication_controller.desired | Gauge | {pod} | Yes | Number of desired pods in this replication_controller | |
k8s.replication_controller.available | Gauge | {pod} | Yes | Total number of available pods (ready for at least minReadySeconds) targeted by this replication_controller | |
k8s.resource_quota.hard_limit | Gauge | {resource} | Yes | The upper limit for a particular resource in a specific namespace. Will only be sent if a quota is specified. CPU requests/limits will be sent as millicores | |
k8s.resource_quota.used | Gauge | {resource} | Yes | The usage for a particular resource in a specific namespace. Will only be sent if a quota is specified. CPU requests/limits will be sent as millicores | |
k8s.statefulset.desired_pods | Gauge | {pod} | Yes | Number of desired pods in the stateful set (the | |
k8s.statefulset.ready_pods | Gauge | {pod} | Yes | Number of pods created by the stateful set that have the | |
k8s.statefulset.current_pods | Gauge | {pod} | Yes | The number of pods created by the StatefulSet controller from the StatefulSet version | |
k8s.statefulset.updated_pods | Gauge | {pod} | Yes | Number of pods created by the StatefulSet controller from the StatefulSet version | |
openshift.clusterquota.limit | Gauge | {resource} | Yes | The configured upper limit for a particular resource. | |
openshift.clusterquota.used | Gauge | {resource} | Yes | The usage for a particular resource with a configured limit. | |
openshift.appliedclusterquota.limit | Gauge | {resource} | Yes | The upper limit for a particular resource in a specific namespace. | |
openshift.appliedclusterquota.used | Gauge | {resource} | Yes | The usage for a particular resource in a specific namespace. | |
k8s.node.condition | Gauge | {condition} | No | The condition of a particular Node. |
Resource Attributes
| Name | Type | Description |
|---|---|---|
k8s.namespace.uid | string | The k8s namespace uid. |
k8s.namespace.name | string | The k8s namespace name. |
k8s.node.uid | string | The k8s node uid. |
k8s.node.name | string | The k8s node name. |
container.id | string | The container id. |
container.image.name | string | The container image name |
container.image.tag | string | The container image tag |
k8s.container.name | string | The k8s container name |
k8s.pod.name | string | The k8s pod name. |
k8s.pod.uid | string | The k8s pod uid. |
k8s.pod.qos_class | string | The k8s pod qos class name. One of Guaranteed, Burstable, BestEffort. |
k8s.replicaset.name | string | The k8s replicaset name |
k8s.replicaset.uid | string | The k8s replicaset uid |
k8s.replicationcontroller.name | string | The k8s replicationcontroller name. |
k8s.replicationcontroller.uid | string | The k8s replicationcontroller uid. |
k8s.resourcequota.uid | string | The k8s resourcequota uid. |
k8s.resourcequota.name | string | The k8s resourcequota name. |
k8s.statefulset.uid | string | The k8s statefulset uid. |
k8s.statefulset.name | string | The k8s statefulset name. |
k8s.deployment.uid | string | The UID of the Deployment. |
k8s.deployment.name | string | The name of the Deployment. |
k8s.cronjob.uid | string | The k8s CronJob uid. |
k8s.cronjob.name | string | The k8s CronJob name |
k8s.daemonset.name | string | The k8s daemonset name. |
k8s.daemonset.uid | string | The k8s daemonset uid. |
k8s.hpa.uid | string | The k8s hpa uid. |
k8s.hpa.name | string | The k8s hpa name. |
k8s.job.name | string | The k8s pod name. |
k8s.job.uid | string | The k8s job uid. |
k8s.kubelet.version | string | The version of Kubelet running on the node. |
container.runtime | string | The container runtime used by Kubernetes Node. |
container.runtime.version | string | The version of container runtime used by Kubernetes Node. |
os.description | string | The os description used by Kubernetes Node. |
os.type | string | The os type used by Kubernetes Node. |
openshift.clusterquota.uid | string | The k8s ClusterResourceQuota uid. |
openshift.clusterquota.name | string | The k8s ClusterResourceQuota name. |
k8s.container.status.last_terminated_reason | string | Last terminated reason of a container. |
Attributes
| Name | Type | Description | Values |
|---|---|---|---|
k8s.namespace.name | string | The k8s namespace name. | |
resource | string | the name of the resource on which the quota is applied | |
condition | string | the name of Kubernetes Node condition. Example: Ready, Memory, PID, DiskPressure | |
Activate or deactivate specific metrics 🔗
You can activate or deactivate specific metrics by setting the enabled field in the metrics section for each metric. For example:
receivers:
samplereceiver:
metrics:
metric-one:
enabled: true
metric-two:
enabled: false
The following is an example of host metrics receiver configuration with activated metrics:
receivers:
hostmetrics:
scrapers:
process:
metrics:
process.cpu.utilization:
enabled: true
Note
Deactivated metrics aren’t sent to Splunk Observability Cloud.
Billing 🔗
If you’re in a MTS-based subscription, all metrics count towards metrics usage.
If you’re in a host-based plan, metrics listed as active (Active: Yes) on this document are considered default and are included free of charge.
Learn more at Infrastructure Monitoring subscription usage (Host and metric plans).
Troubleshooting 🔗
If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.
Available to Splunk Observability Cloud customers
Submit a case in the Splunk Support Portal .
Contact Splunk Support .
Available to prospective customers and free trial users
Ask a question and get answers through community support at Splunk Answers .
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.
