Advanced configurations for Kubernetes π
The following sections describe available settings for configuring the Splunk Distribution of OpenTelemetry Collector for Kubernetes.
Configure the Helm chart π
The values.yaml lists all supported configurable parameters for the Helm chart, along with a detailed explanation of each parameter. Review values.yaml to understand how to configure this chart.
The Helm chart can also be configured to support different use cases, such as trace sampling and sending data through a proxy server. See Examples of chart configuration for more information.
Configure the following required values to send data to Splunk Enterprise or Splunk Cloud:
splunkPlatform:
# Required for Splunk Enterprise or Splunk Cloud.
# Add a URL for the Splunk instance to send data to.
# For example, "http://X.X.X.X:8088/services/collector".
# Setting this parameter enables Splunk Platform as a destination.
endpoint: ""
# Required for Splunk Enterprise or Splunk Cloud if `endpoint` is specified.
token: ""
# The Splunk HTTP Event Collector (HEC) token.
Configure the following required values to send data to Splunk Observability Cloud:
splunkObservability:
# Required for Observability Cloud.
# Add the Observability Cloud realm to send telemetry data to.
# Setting this parameter enables Observability Cloud as a destination.
accessToken: xxxxxx
# The Observability Cloud org access token.
realm: us0
# Required for Observability Cloud if `realm` is specified.
clusterName: my-k8s-cluster
See Create and manage organization access tokens using Splunk Observability Cloud for information on creating and managing your access token.
Configure a cloud provider π
If applicable, use the cloudProvider
parameter to provide information about the cloud provider. The following options are supported:
aws
for Amazon Web Servicesgcp
for Google Cloud Platformazure
for Microsoft Azure
For example:
splunkObservability:
accessToken: xxxxxx
realm: us0
clusterName: my-k8s-cluster
cloudProvider: aws
Configure the Kubernetes distribution π
If applicable, use the distribution
parameter to provide information about the underlying Kubernetes deployment. This parameter allows the connector to automatically scrape additional metadata. The following options are supported:
aks
for Azure AKSeks
for Amazon EKSeks/fargate
for Amazon EKS with Fargate profilesgke
for Google GKE or Standard modegke/autopilot
for Google GKE or Autopilot modeopenshift
for Red Hat OpenShift
For example:
splunkObservability:
accessToken: xxxxxx
realm: us0
clusterName: my-k8s-cluster
distribution: gke
Configure the deployment environment π
If applicable, use the environment
parameter to specify an additional deployment.environment
attribute to be added to all telemetry data. This attribute helps Observability Cloud users investigate data coming from different sources separately. Example values include development
, staging
, and production
.
splunkObservability:
accessToken: xxxxxx
realm: us0
environment: production
Disable particular types of telemetry π
By default, OpenTelemetry sends only metrics and traces to Observability Cloud and sends only logs to Splunk Platform. You can enable or disable any kind of telemetry data collection for a specific destination. For example, with the following configuration, the Collector sends all collected telemetry data to Observability Cloud and Splunk Platform, assuming youβve properly configured them.
splunkObservability:
metricsEnabled: true
tracesEnabled: true
logsEnabled: true
splunkPlatform:
metricsEnabled: true
logsEnabled: true
Configure Windows worker nodes π
The Splunk Distribution of OpenTelemetry Collector for Kubernetes supports collecting metrics, traces, and logs (using OpenTelemetry native logs collection only) from Windows nodes. All Windows images are available in the quay.io/signalfx/splunk-otel-collector-windows
repository.
Use the following values.yaml configuration to install the Helm chart on Windows worker nodes:
isWindows: true
image:
otelcol:
repository: quay.io/signalfx/splunk-otel-collector-windows
logsEngine: otel
readinessProbe:
initialDelaySeconds: 60
livenessProbe:
initialDelaySeconds: 60
If you have both Windows and Linux worker nodes in your Kubernetes cluster, you need to install the Helm chart twice. One of the installations with the default configuration set to isWindows: false
is applied on Linux nodes. The second installation with the values.yaml configuration (shown in the previous example) is applied on Windows nodes.
Disable the clusterReceiver
on one of the installations to avoid cluster-wide metrics duplication. To do this, add the following lines
to the values.yaml configuration of one of the installations:
clusterReceiver:
enabled: false
Configure Google Kubernetes Engine Autopilot π
To run the Collector in Google Kubernetes Engine Autopilot mode, set the distribution
option to gke/autopilot
, as shown in the following example:
distribution: gke/autopilot
Search for βAutopilot overviewβ on the Google Cloud documentation site for more information.
Note
Native OpenTelemetry logs collection is not yet supported in Google Kubernetes Engine Autopilot mode.
The Collector agent daemonset can have problems scheduling in Autopilot mode. If this happens, do the following to assign the daemonset a higher priority class to ensure that the daemonset pods are always present on each node:
Create a new priority class for the Collector agent:
cat <<EOF | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: splunk-otel-agent-priority
value: 1000000
globalDefault: false
description: "Higher priority class for the Splunk Distribution of OpenTelemetry Collector pods."
EOF
Use the created priority class in the helm install/upgrade command using the --set="priorityClassName=splunk-otel-agent-priority"
argument, or add the following line to your custom values.yaml:
priorityClassName: splunk-otel-agent-priority
Configure EKS Fargate π
To run the Collector in the Amazon Elastic Kubernetes Service with Fargate profiles, set the required distribution
value to eks/fargate
, as shown in the following example:
distribution: eks/fargate
Note
Fluentd and native OpenTelemetry logs collection are not yet automatically configured in EKS with Fargate profiles.
This distribution operates similarly to the eks
distribution, but with the following distinctions:
The Collector agent daemonset is not applied since Fargate does not support daemonsets. Any desired Collector instances running as agents must be configured manually as sidecar containers in your custom deployments. This includes any application logging services like Fluentd. Set
gateway.enabled
totrue
and configure your instrumented applications to report metrics, traces, and logs to the gatewayβs<installed-chart-name>-splunk-otel-collector
service address. Any desired agent instances that would run as a daemonset should instead run as sidecar containers in your pods.- Since Fargate nodes use a VM boundary to prevent access to host-based resources used by other pods, pods are not able to reach their own kubelet. The cluster receiver for the Fargate distribution has two primary differences between regular
eks
to work around this limitation: The configured cluster receiver is deployed as a two-replica StatefulSet instead of a Deployment, and uses a Kubernetes Observer extension that discovers the clusterβs nodes and, on the second replica, its pods for user-configurable receiver creator additions.Using this observer dynamically creates the Kubelet Stats receiver instances that report kubelet metrics for all observed Fargate nodes. The first replica monitors the cluster with a
k8s_cluster
receiver, and the second cluster monitors all kubelets except its own (due to an EKS/Fargate networking restriction).The first replicaβs Collector monitors the secondβs kubelet. This is made possible by a Fargate-specific
splunk-otel-eks-fargate-kubeletstats-receiver-node
node label. The Collectorβs ClusterRole foreks/fargate
allows thepatch
verb onnodes
resources for the default API groups to allow the cluster receiverβs init container to add this node label for designated self monitoring.
- Since Fargate nodes use a VM boundary to prevent access to host-based resources used by other pods, pods are not able to reach their own kubelet. The cluster receiver for the Fargate distribution has two primary differences between regular
Collect logs π
The Helm chart currently uses Fluentd for Kubernetes logs collection. Logs collected with Fluentd are sent through the Collector agent, which does all of the necessary metadata enrichment. The OpenTelemetry Collector also has native functionality for logs collection, which will soon be migrated from Fluentd to the OpenTelemetry logs collection.
Add the following line to your configuration to use OpenTelemetry logs collection instead of Fluentd:
logsEngine: otel
The following are known limitations of native OpenTelemetry logs collection:
The
service.name
attribute is not automatically constructed in an Istio environment, which means that correlation between logs and traces does not work in Splunk Observability Cloud. Use Fluentd for logs collection if you deploy the Helm chart withautodetect.istio=true
.Journald logs cannot be natively collected by the Collector at this time.
Logs collection is not supported in GKE Autopilot at this time.
See also other rules and limitations for metrics and dimensions. For instance, you can have up to 36 dimensions per MTS, otherwise the data point is dropped.
Add log files from Kubernetes host machines or volumes π
Add additional log files to be ingested from Kubernetes host machines and Kubernetes volumes by configuring agent.extraVolumes
, agent.extraVolumeMounts
, and logsCollection.extraFileLogs
in the values.yaml file used to deploy the Collector for Kubernetes.
The following example shows how to add logs from Kubernetes host machines:
logsCollection:
extraFileLogs:
filelog/audit-log:
include: [/var/log/kubernetes/apiserver/audit.log]
start_at: beginning
include_file_path: true
include_file_name: false
resource:
com.splunk.source: /var/log/kubernetes/apiserver/audit.log
host.name: 'EXPR(env("K8S_NODE_NAME"))'
com.splunk.sourcetype: kube:apiserver-audit
agent:
extraVolumeMounts:
- name: audit-log
mountPath: /var/log/kubernetes/apiserver
extraVolumes:
- name: audit-log
hostPath:
path: /var/log/kubernetes/apiserver
Process multi-line logs π
The Splunk Distribution of OpenTelemetry Collector for Kubernetes supports parsing of multi-line logs to help read, understand, and troubleshoot the multi-line logs in a better way. Process multi-line logs by adding the following section to your values.yaml configuration:
logsCollection:
containers:
multilineConfigs:
- namespaceName:
value: default
podName:
value: buttercup-app-.*
useRegexp: true
containerName:
value: server
firstEntryRegex: ^[^\s].*
Use regex101 to find a golang regex that works for your format and specify it in the config file for the config option firstEntryRegex
.
Collect journald events π
The Splunk Distribution of OpenTelemetry Collector for Kubernetes can collect journald events from Kubernetes environment. Process journald events by adding the following section to your values.yaml configuration:
logsCollection:
journald:
enabled: true
directory: /run/log/journal
# List of service units to collect and configuration for each. Update the list as needed.
units:
- name: kubelet
priority: info
- name: docker
priority: info
- name: containerd
priority: info
# Optional: Route journald logs to a seperate Splunk Index by specifying the index
# value. Make sure the index exists in Splunk and is configured to receive HEC
# traffic (not applicable to Observability Cloud).
index: ""
Review performance benchmarks π
Some configurations used with the OpenTelemetry Collector (as set using the Collector for Kubernetes Helm chart) can have an impact on overall performance of log ingestion. The more receivers, processors, exporters, and extensions that are added to any of the pipelines, the greater the performance impact.
The Splunk Distribution of OpenTelemetry Collector for Kubernetes can exceed the default throughput of the HTTP Event Collector (HEC). To address capacity needs, monitor the HEC throughput and back pressure on the Collector for Kubernetes deployments and be prepared to add additional nodes as needed.
The following table provides a summary of performance benchmarks run internally:
Log generator count |
Event size (byte) |
Agent CPU usage |
Agent EPS |
---|---|---|---|
1 |
256 |
1.8 |
30,000 |
1 |
516 |
1.8 |
28,000 |
1 |
1024 |
1.8 |
24,000 |
5 |
256 |
3.2 |
54,000 |
7 |
256 |
3 |
52,000 |
10 |
256 |
3.2 |
53,000 |
The data pipelines for these test runs involved reading container logs as they are being written, then parsing filename for metadata, enriching it with Kubernetes metadata, reformatting the data structure, and sending logs (without compression) to the Splunk HEC endpoint.
Run the container in non-root user mode π
Collecting logs often requires reading log files that are owned by the root user. By default, the container runs with securityContext.runAsUser = 0
, which gives the root user permission to read those files. To run the container in non-root user mode, set .agent.securityContext
to 20000
to cause the container to run the required file system operations as UID and GID 20000
(this can be any other UID and GUI).
Note
Setting the containerRuntime:
parameter to cri-o
did not work during internal testing for logs collection.
Override a control plane configuration π
If agent.controlPlaneEnabled=true
, the Helm chart sets up the Collector to collect metrics from the control plane.
To collect control plane metrics, the Helm chart uses the Collector on each node to use the receiver creator to represent control plane receivers at runtime. The receiver creator has a set of discovery rules that know which control plane receivers to create. The default discovery rules can vary depending on the Kubernetes distribution and version. See the receiver creator documentation in GitHub for more information.
If your control plane is using non-standard specifications, then you can provide a custom configuration to allow the Collector to successfully connect to it.
The Collector relies on pod-level network access to collect metrics from the control plane pods. Since most cloud Kubernetes as a service distributions donβt expose the control plane pods to the end user, collecting metrics from these distributions is not supported.
The following distributions are supported:
Kubernetes 1.22 (kops created)
OpenShift version 4.9
The following distributions are not supported:
AKS
EKS
EKS/Fargate
GKE
GKE/Autopilot
See the agent template for the default configurations for the control plane receivers.
Refer to the following documentation for information on the configuration options and supported metrics for each control plane receiver:
There is a known limitation when using the Kubernetes proxy control plane receiver. When using a kops created Kubernetes cluster, a network connectivity issue has been reported that prevents proxy metrics from being collected. The limitation can be addressed by updating the kubeProxy metric bind address in the kops cluster specification:
Set
kubeProxy.metricsBindAddress: 0.0.0.0
in the kops cluster specification.Run
kops update cluster {cluster_name}
andkops rolling-update cluster {cluster_name}
to deploy the change.
Using custom configurations for nonstandard control plane components π
You can override the default configuration values used to connect to the control plane. If your control plane uses nonstandard ports or custom TLS settings, you need to override the default configurations. The following example shows how to connect to a nonstandard API server that uses port 3443 for metrics and custom TLS certs stored in the /etc/myapiserver/ directory.
agent:
config:
receivers:
receiver_creator:
receivers:
# Template for overriding the discovery rule and configuration.
# smartagent/{control_plane_receiver}:
# rule: {rule_value}
# config:
# {config_value}
smartagent/kubernetes-apiserver:
rule: type == "port" && port == 3443 && pod.labels["k8s-app"] == "kube-apiserver"
config:
clientCertPath: /etc/myapiserver/clients-ca.crt
clientKeyPath: /etc/myapiserver/clients-ca.key
skipVerify: true
useHTTPS: true
useServiceAccount: false
Add additional telemetry sources π
Use the autodetect
configuration option to enable additional telemetry sources.
Set autodetect.prometheus=true
if you want the Collector to scrape Prometheus metrics from pods that have generic Prometheus-style annotations. Add the following annotations on pods to allow a fine control of the scraping process:
prometheus.io/scrape: true
: The default configuration scrapes all pods. If set tofalse
, this annotation excludes the pod from the scraping process.prometheus.io/path
: The path to scrape the metrics from. The default value is/metrics
.prometheus.io/port
: The port to scrape the metrics from. The default value is9090
.
If the Collector is running in an Istio environment, set autodetect.istio=true
to make sure that all traces, metrics, and logs reported by Istio are collected in a unified manner.
For example, use the following configuration to enable automatic detection of both Prometheus and Istio telemetry sources:
splunkObservability:
accessToken: xxxxxx
realm: us0
clusterName: my-k8s-cluster
autodetect:
istio: true
prometheus: true
Override the underlying OpenTelemetry agent configuration π
You can override the underlying OpenTelemetry agent configuration to use your own OpenTelemetry Agent configuration. To do this, include a custom configuration in the agent.config
parameter in the values.yaml configuration. This custom configuration is merged into the default agent configuration. Parts of the configuration (for example, service
, pipelines
, logs
, and processors
need to be fully re-defined after the files are merged.
The following example shows a values.yaml file with custom gateway values:
clusterName: my-cluster
splunkObservability:
realm: us0
accessToken: my-access-token
agent:
config:
exporters:
otlp:
endpoint: <custom-gateway-url>:4317
insecure: true
signalfx:
ingest_url: http://<custom-gateway-url>:9943
api_url: http://<custom-gateway-url>:6060
service:
pipelines:
traces:
exporters: [otlp, signalfx]
metrics:
exporters: [otlp]
logs:
exporters: [otlp]
clusterReceiver:
config:
exporters:
signalfx:
ingest_url: http://<custom-gateway-url>:9943
api_url: http://<custom-gateway-url>:6060