Docs ยป Supported integrations in Splunk Observability Cloud ยป Configure application receivers for orchestration ยป Kubernetes scheduler

Kubernetes scheduler ๐Ÿ”—

The Splunk Distribution of the OpenTelemetry Collector uses the Smart Agent receiver with the kubernetes-scheduler monitor type to export Prometheus metrics from the kube-scheduler.

This monitor type is available on Kubernetes, Linux, and Windows.

Benefits ๐Ÿ”—

After you configure the integration, you can access these features:

Installation ๐Ÿ”—

Follow these steps to deploy this integration:

  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:

  2. Configure the integration, as described in the Configuration section.

  3. Restart the Splunk Distribution of the OpenTelemetry Collector.

Configuration ๐Ÿ”—

To use this integration of a Smart Agent monitor with the Collector:

  1. Include the Smart Agent receiver in your configuration file.

  2. Add the monitor type to the Collector configuration, both in the receiver and pipelines sections.

Example ๐Ÿ”—

To activate this integration, add the following to your Collector configuration:

receivers:
  smartagent/kubernetes-scheduler
    type: kubernetes-scheduler
    ... # Additional config

Next, add the monitor to the service.pipelines.metrics.receivers section of your configuration file:

service:
   pipelines:
     metrics:
       receivers: [smartagent/kubernetes-scheduler]

Configuration settings ๐Ÿ”—

Config option

Required

Type

Description

httpTimeout

no

int64

HTTP timeout duration for both read and writes. This should be a

duration string that is accepted by https://golang.org/pkg/time/#ParseDuration Default is 10s.

username

no

string

Basic authentication username to use on each request, if any.

password

no

string

Basic authentication password to use on each request, if any.

useHTTPS

no

bool

If true, the agent will connect to the server using HTTPS

instead of plain HTTP. Default is false.

httpHeaders

no

map of strings

A map of HTTP header names to values. Comma separated multiple

values for the same message-header is supported.

skipVerify

no

bool

If useHTTPS is true and this option is also true, the exporter

TLS cert will not be verified. Default is false

sniServerName

no

string

If useHTTPS is true and skipVerify is true, the sniServerName is

used to verify the hostname on the returned certificates. It is also included in the clientโ€™s handshake to support virtual hosting unless it is an IP address.

caCertPath

no

string

Path to the CA cert that has signed the TLS cert, unnecessary if

skipVerify is set to false.

clientCertPath

no

string

Path to the client TLS cert to use for TLS required connections.

clientKeyPath

no

string

Path to the client TLS key to use for TLS required connections.

host

yes

string

Host of the exporter.

port

yes

integer

Port of the exporter.

useServiceAccount

no

bool

Use pod service account to authenticate. Default is

false.

metricPath

no

string

Path to the metrics endpoint on the exporter server, usually

/metrics. Default is /metrics.

sendAllMetrics

no

bool

Send all the metrics that come out of the Prometheus exporter

without any filtering. This option has no effect when using the prometheus exporter monitor directly since there is no built-in filtering, only when embedding it in other monitors. Default is false.

Metrics ๐Ÿ”—

The following metrics are available for this integration:

NameDescriptionTypeCategory
apiserver_audit_event_total

Counter of audit events generated and sent to the audit backend.

cumulativeCustom
apiserver_audit_requests_rejected_total

Counter of apiserver requests rejected due to an error in audit logging backend.

cumulativeCustom
apiserver_client_certificate_expiration_seconds_count

Distribution of the remaining lifetime on the certificate used to authenticate a request. (count)

cumulativeCustom
apiserver_client_certificate_expiration_seconds

Distribution of the remaining lifetime on the certificate used to authenticate a request. (sum)

cumulativeCustom
apiserver_client_certificate_expiration_seconds_bucket

Distribution of the remaining lifetime on the certificate used to authenticate a request. (bucket)

cumulativeCustom
apiserver_storage_data_key_generation_duration_seconds_count

Latencies in seconds of data encryption key(DEK) generation operations. (count)

cumulativeCustom
apiserver_storage_data_key_generation_duration_seconds

Latencies in seconds of data encryption key(DEK) generation operations. (sum)

cumulativeCustom
apiserver_storage_data_key_generation_duration_seconds_bucket

Latencies in seconds of data encryption key(DEK) generation operations. (bucket)

cumulativeCustom
apiserver_storage_data_key_generation_failures_total

Total number of failed data encryption key(DEK) generation operations.

cumulativeCustom
apiserver_storage_data_key_generation_latencies_microseconds_count

(Deprecated) Latencies in microseconds of data encryption key(DEK) generation operations. (count)

cumulativeCustom
apiserver_storage_data_key_generation_latencies_microseconds

(Deprecated) Latencies in microseconds of data encryption key(DEK) generation operations. (sum)

cumulativeCustom
apiserver_storage_data_key_generation_latencies_microseconds_bucket

(Deprecated) Latencies in microseconds of data encryption key(DEK) generation operations. (bucket)

cumulativeCustom
apiserver_storage_envelope_transformation_cache_misses_total

Total number of cache misses while accessing key decryption key(KEK).

cumulativeCustom
go_gc_duration_seconds_count

A summary of the GC invocation durations. (count)

cumulativeCustom
go_gc_duration_seconds

A summary of the GC invocation durations. (sum)

cumulativeCustom
go_gc_duration_seconds_quantile

A summary of the GC invocation durations. (quantized)

gaugeCustom
go_goroutines

Number of goroutines that currently exist.

gaugeCustom
go_info

Information about the Go environment.

gaugeCustom
go_memstats_alloc_bytes

Number of bytes allocated and still in use.

gaugeCustom
go_memstats_alloc_bytes_total

Total number of bytes allocated, even if freed.

cumulativeCustom
go_memstats_buck_hash_sys_bytes

Number of bytes used by the profiling bucket hash table.

gaugeCustom
go_memstats_frees_total

Total number of frees.

cumulativeCustom
go_memstats_gc_cpu_fraction

The fraction of this program's available CPU time used by the GC since the program started.

gaugeCustom
go_memstats_gc_sys_bytes

Number of bytes used for garbage collection system metadata.

gaugeCustom
go_memstats_heap_alloc_bytes

Number of heap bytes allocated and still in use.

gaugeCustom
go_memstats_heap_idle_bytes

Number of heap bytes waiting to be used.

gaugeCustom
go_memstats_heap_inuse_bytes

Number of heap bytes that are in use.

gaugeCustom
go_memstats_heap_objects

Number of allocated objects.

gaugeCustom
go_memstats_heap_released_bytes

Number of heap bytes released to OS.

gaugeCustom
go_memstats_heap_sys_bytes

Number of heap bytes obtained from system.

gaugeCustom
go_memstats_last_gc_time_seconds

Number of seconds since 1970 of last garbage collection.

gaugeCustom
go_memstats_lookups_total

Total number of pointer lookups.

cumulativeCustom
go_memstats_mallocs_total

Total number of mallocs.

cumulativeCustom
go_memstats_mcache_inuse_bytes

Number of bytes in use by mcache structures.

gaugeCustom
go_memstats_mcache_sys_bytes

Number of bytes used for mcache structures obtained from system.

gaugeCustom
go_memstats_mspan_inuse_bytes

Number of bytes in use by mspan structures.

gaugeCustom
go_memstats_mspan_sys_bytes

Number of bytes used for mspan structures obtained from system.

gaugeCustom
go_memstats_next_gc_bytes

Number of heap bytes when next garbage collection will take place.

gaugeCustom
go_memstats_other_sys_bytes

Number of bytes used for other system allocations.

gaugeCustom
go_memstats_stack_inuse_bytes

Number of bytes in use by the stack allocator.

gaugeCustom
go_memstats_stack_sys_bytes

Number of bytes obtained from system for stack allocator.

gaugeCustom
go_memstats_sys_bytes

Number of bytes obtained from system.

gaugeCustom
go_threads

Number of OS threads created.

gaugeCustom
http_request_duration_microseconds_count

The HTTP request latencies in microseconds. (count)

cumulativeCustom
http_request_duration_microseconds

The HTTP request latencies in microseconds. (sum)

cumulativeCustom
http_request_duration_microseconds_quantile

The HTTP request latencies in microseconds. (quantized)

gaugeCustom
http_request_size_bytes_count

The HTTP request sizes in bytes. (count)

cumulativeCustom
http_request_size_bytes

The HTTP request sizes in bytes. (sum)

cumulativeCustom
http_request_size_bytes_quantile

The HTTP request sizes in bytes. (quantized)

gaugeCustom
http_requests_total

Total number of HTTP requests made.

cumulativeCustom
http_response_size_bytes_count

The HTTP response sizes in bytes. (count)

cumulativeCustom
http_response_size_bytes

The HTTP response sizes in bytes. (sum)

cumulativeCustom
http_response_size_bytes_quantile

The HTTP response sizes in bytes. (quantized)

gaugeCustom
kubernetes_build_info

A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.

gaugeCustom
leader_election_master_status

Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.

gaugeCustom
process_cpu_seconds_total

Total user and system CPU time spent in seconds.

cumulativeCustom
process_max_fds

Maximum number of open file descriptors.

gaugeCustom
process_open_fds

Number of open file descriptors.

gaugeCustom
process_resident_memory_bytes

Resident memory size in bytes.

gaugeCustom
process_start_time_seconds

Start time of the process since unix epoch in seconds.

gaugeCustom
process_virtual_memory_bytes

Virtual memory size in bytes.

gaugeCustom
process_virtual_memory_max_bytes

Maximum amount of virtual memory available in bytes.

gaugeCustom
rest_client_request_duration_seconds_count

Request latency in seconds. Broken down by verb and URL. (count)

cumulativeCustom
rest_client_request_duration_seconds

Request latency in seconds. Broken down by verb and URL. (sum)

cumulativeCustom
rest_client_request_duration_seconds_bucket

Request latency in seconds. Broken down by verb and URL. (bucket)

cumulativeCustom
rest_client_request_latency_seconds_count

(Deprecated) Request latency in seconds. Broken down by verb and URL. (count)

cumulativeCustom
rest_client_request_latency_seconds

(Deprecated) Request latency in seconds. Broken down by verb and URL. (sum)

cumulativeCustom
rest_client_request_latency_seconds_bucket

(Deprecated) Request latency in seconds. Broken down by verb and URL. (bucket)

cumulativeCustom
rest_client_requests_total

Number of HTTP requests, partitioned by status code, method, and host.

cumulativeDefault
scheduler_binding_duration_seconds_count

Binding latency in seconds (count)

cumulativeCustom
scheduler_binding_duration_seconds

Binding latency in seconds (sum)

cumulativeDefault
scheduler_binding_duration_seconds_bucket

Binding latency in seconds (bucket)

cumulativeCustom
scheduler_binding_latency_microseconds_count

(Deprecated) Binding latency in microseconds (count)

cumulativeCustom
scheduler_binding_latency_microseconds

(Deprecated) Binding latency in microseconds (sum)

cumulativeDefault
scheduler_binding_latency_microseconds_bucket

(Deprecated) Binding latency in microseconds (bucket)

cumulativeCustom
scheduler_e2e_scheduling_duration_seconds_count

E2e scheduling latency in seconds (scheduling algorithm + binding) (count)

cumulativeCustom
scheduler_e2e_scheduling_duration_seconds

E2e scheduling latency in seconds (scheduling algorithm + binding) (sum)

cumulativeDefault
scheduler_e2e_scheduling_duration_seconds_bucket

E2e scheduling latency in seconds (scheduling algorithm + binding) (bucket)

cumulativeCustom
scheduler_e2e_scheduling_latency_microseconds_count

(Deprecated) E2e scheduling latency in microseconds (scheduling algorithm + binding) (count)

cumulativeCustom
scheduler_e2e_scheduling_latency_microseconds

(Deprecated) E2e scheduling latency in microseconds (scheduling algorithm + binding) (sum)

cumulativeDefault
scheduler_e2e_scheduling_latency_microseconds_bucket

(Deprecated) E2e scheduling latency in microseconds (scheduling algorithm + binding) (bucket)

cumulativeCustom
scheduler_pending_pods

Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ.

gaugeCustom
scheduler_pod_preemption_victims

Number of selected preemption victims

gaugeCustom
scheduler_schedule_attempts_total

Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.

cumulativeCustom
scheduler_scheduling_algorithm_duration_seconds_count

Scheduling algorithm latency in seconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_duration_seconds

Scheduling algorithm latency in seconds (sum)

cumulativeDefault
scheduler_scheduling_algorithm_duration_seconds_bucket

Scheduling algorithm latency in seconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_latency_microseconds_count

(Deprecated) Scheduling algorithm latency in microseconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_latency_microseconds

(Deprecated) Scheduling algorithm latency in microseconds (sum)

cumulativeDefault
scheduler_scheduling_algorithm_latency_microseconds_bucket

(Deprecated) Scheduling algorithm latency in microseconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation_count

(Deprecated) Scheduling algorithm predicate evaluation duration in microseconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation

(Deprecated) Scheduling algorithm predicate evaluation duration in microseconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation_bucket

(Deprecated) Scheduling algorithm predicate evaluation duration in microseconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation_seconds_count

Scheduling algorithm predicate evaluation duration in seconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation_seconds

Scheduling algorithm predicate evaluation duration in seconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_predicate_evaluation_seconds_bucket

Scheduling algorithm predicate evaluation duration in seconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation_count

(Deprecated) Scheduling algorithm preemption evaluation duration in microseconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation

(Deprecated) Scheduling algorithm preemption evaluation duration in microseconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation_bucket

(Deprecated) Scheduling algorithm preemption evaluation duration in microseconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation_seconds_count

Scheduling algorithm preemption evaluation duration in seconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation_seconds

Scheduling algorithm preemption evaluation duration in seconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_preemption_evaluation_seconds_bucket

Scheduling algorithm preemption evaluation duration in seconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation_count

(Deprecated) Scheduling algorithm priority evaluation duration in microseconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation

(Deprecated) Scheduling algorithm priority evaluation duration in microseconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation_bucket

(Deprecated) Scheduling algorithm priority evaluation duration in microseconds (bucket)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation_seconds_count

Scheduling algorithm priority evaluation duration in seconds (count)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation_seconds

Scheduling algorithm priority evaluation duration in seconds (sum)

cumulativeCustom
scheduler_scheduling_algorithm_priority_evaluation_seconds_bucket

Scheduling algorithm priority evaluation duration in seconds (bucket)

cumulativeCustom
scheduler_scheduling_duration_seconds_count

Scheduling latency in seconds split by sub-parts of the scheduling operation (count)

cumulativeCustom
scheduler_scheduling_duration_seconds

Scheduling latency in seconds split by sub-parts of the scheduling operation (sum)

cumulativeCustom
scheduler_scheduling_duration_seconds_quantile

Scheduling latency in seconds split by sub-parts of the scheduling operation (quantized)

gaugeCustom
scheduler_scheduling_latency_seconds_count

(Deprecated) Scheduling latency in seconds split by sub-parts of the scheduling operation (count)

cumulativeCustom
scheduler_scheduling_latency_seconds

(Deprecated) Scheduling latency in seconds split by sub-parts of the scheduling operation (sum)

cumulativeCustom
scheduler_scheduling_latency_seconds_quantile

(Deprecated) Scheduling latency in seconds split by sub-parts of the scheduling operation (quantized)

gaugeCustom
scheduler_total_preemption_attempts

Total preemption attempts in the cluster till now

cumulativeCustom
scheduler_volume_scheduling_duration_seconds_count

Volume scheduling stage latency (count)

cumulativeCustom
scheduler_volume_scheduling_duration_seconds

Volume scheduling stage latency (sum)

cumulativeCustom
scheduler_volume_scheduling_duration_seconds_bucket

Volume scheduling stage latency (bucket)

cumulativeCustom

Notes ๐Ÿ”—

  • To learn more about the available in Splunk Observability Cloud see Metric types

  • In host-based subscription plans, default metrics are those metrics included in host-based subscriptions in Splunk Observability Cloud, such as host, container, or bundled metrics. Custom metrics are not provided by default and might be subject to charges. See Metric categories for more information.

  • In MTS-based subscription plans, all metrics are custom.

  • To add additional metrics, see how to configure extraMetrics in Add additional metrics

Non-default metrics (version 4.7.0+) ๐Ÿ”—

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.

Troubleshooting ๐Ÿ”—

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Available to prospective customers and free trial users

  • Ask a question and get answers through community support at Splunk Answers .

  • Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

This page was last updated on Feb 11, 2025.