Set up Network Explorer π
You can install and configure Network Explorer as part of the Splunk Distribution of OpenTelemetry Collector Helm chart.
Prerequisites π
To use Network Explorer, you must meet the following requirements.
Prerequisite
Description
Environment
Network Explorer is only supported in Kubernetes-based environments on Linux hosts. Use Helm-based management.
Operating system
RedHat Linux versions 7.6 or higher, Ubuntu versions 16.04 or higher, Debian Stretch+, Amazon Linux 2, Google COS
Kubernetes version
Network Explorer is supported on all active releases of Kubernetes. For more information, see Releases in the Kubernetes documentation.
Network Explorer components π
Network Explorer consists of the following components:
Component
Description
Required?
Enabled by default?
The reducer
The reducer takes the data points collected by the collectors and reduces them to actual metric time series (MTS). The reducer also connects to the Splunk Distribution of OpenTelemetry Collector on the OTLP gRPC port.
Yes. Install and configure at least one instance of the reducer.
Yes
The kernel collector
The Extended Berkeley Packet Filter (eBPF) agent responsible for gathering data points from the kernel.
Yes. Install and configure the kernel collector on each of your hosts.
Yes
The Kubernetes collector
The Kubernetes collector further enriches collected data points with additional metadata.
No. If you want to get additional metadata, install and configure at least one instance of the Kubernetes collector on each Kubernetes cluster.
Yes. If you want to disable the Kubernetes collector, set
k8sCollector.enabled
tofalse
.The cloud collector
The cloud collector further enriches collected data points with additional metadata.
No. If your Kubernetes is hosted by, or installed within, AWS, and you want to get additional metadata, install and configure at least one instance of the cloud collector.
No. If you want to enable the cloud collector, set
cloudCollector.enabled
totrue
.
Install Network Explorer π
For the Splunk Distribution of OpenTelemetry Collector to work with Network Explorer, you must install it in Gateway mode, and perform the following steps:
Enable OTLP gRPC reception by configuring an OTLP gRPC metric receiver on the Gateway.
Enable SignalFx export by configuring a SignalFx exporter on the Gateway with the valid realm and access token.
The OTLP gRPC metric receiver and SignalFx exporter are already configured in the Helm chart for the Splunk Distribution of OpenTelemetry Collector, so if you use the Helm chart method to install the Splunk Distribution of OpenTelemetry Collector, you donβt need to configure these requirements separately.
The following table shows required parameters for this installation:
Parameter
Description
namespace
The Kubernetes namespace to install into. This value must match the value for the namespace of the Network Explorer.
splunkObservability.realm
Splunk realm to send telemetry data to. For example,
us0
.
splunkObservability.accessToken
The access token for your organization. An access token with ingest scope is sufficient. For more information, see Create and manage organization access tokens using Splunk Observability Cloud.
clusterName
An arbitrary value that identifies your Kubernetes cluster.
networkExplorer.enabled
Set this to
true
to enable Network Explorer.
agent.enabled
If you are adding Network Explorer to an existing Splunk Distribution of OpenTelemetry Collector configuration, leave
agent.enabled
as is.If you are installing a new instance of the Splunk Distribution of OpenTelemetry Collector and only want to collect telemetry from Network Explorer, set this to
false
to disable installing the Splunk Distribution of OpenTelemetry Collector in Agent mode on each Kubernetes node.If you are installing a new instance of the Splunk Distribution of OpenTelemetry Collector and want to collect telemetry from both Network Explorer and the individual OpenTelemetry Collector agents, set this to
true
.
clusterReceiver.enabled
If you are adding Network Explorer to an existing Splunk Distribution of OpenTelemetry Collector configuration, leave
clusterReceiver.enabled
as is.If you are installing a new instance of the Splunk Distribution of OpenTelemetry Collector and only want to collect telemetry from Network Explorer, set this to
false
since Network Explorer doesnβt usesplunk-otel-collector-k8s-cluster-receiver
.If you are installing a new instance of the Splunk Distribution of OpenTelemetry Collector and want to collect cluster-level metrics from the Kubernetes API server in addition to Network Explorer telemetry, set this to
true
.
gateway.replicaCount
Set this to
1
since Network Explorer doesnβt support communication to multiple gateway replicas.
Example π
In this example, the reducer, the kernel collector, and the Kubernetes collector are configured. The cloud collector isnβt enabled.
Follow these steps to install Network Explorer using the Helm chart method:
Run the following command to deploy the Helm chart.
helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
Run the following command to update the Helm chart.
helm repo update
Run the following command to install the Splunk Distribution of OpenTelemetry Collector. Replace the parameters with their appropriate values.
helm --namespace=<NAMESPACE> install my-splunk-otel-collector \ --set="splunkObservability.realm=<REALM>" \ --set="splunkObservability.accessToken=<ACCESS_TOKEN>" \ --set="clusterName=<CLUSTER_NAME>" \ --set="networkExplorer.enabled=true" \ --set="agent.enabled=false" \ --set="clusterReceiver.enabled=false" \ --set="gateway.replicaCount=1" \ splunk-otel-collector-chart/splunk-otel-collector
helm --namespace=<NAMESPACE> install splunk-otel-collector \ --set="splunkObservability.realm=<REALM>" \ --set="splunkObservability.accessToken=<ACCESS_TOKEN>" \ --set="clusterName=<CLUSTER_NAME>" \ --set="splunkObservability.logsEnabled=true" \ --set="splunkObservability.infrastructureMonitoringEventsEnabled=true" \ --set="networkExplorer.enabled=true" \ --set="networkExplorer.podSecurityPolicy.enabled=false" \ --set="agent.enabled=true" \ --set="clusterReceiver.enabled=true" \ --set="gateway.replicaCount=1" \ --set="environment=<APM_ENV>" \ --set="gateway.resources.limits.cpu=500m" \ --set="gateway.resources.limits.memory=1Gi" \ splunk-otel-collector-chart/splunk-otel-collector
(Optional) The Network Explorer kernel collector requires kernel headers to run the kernel in each Kubernetes node. The kernel collector installs the headers automatically unless your nodes donβt have access to the internet.
If you need to install the required packages manually, run the following command:
sudo apt-get install --yes linux-headers-$(uname -r)
sudo yum install -y kernel-devel-$(uname -r)
For additional Splunk Distribution of OpenTelemetry Collector configuration, see Install the Collector for Kubernetes.
Change the resource footprint of Splunk Distribution of OpenTelemetry Collector π
Each Kubernetes node has a Splunk Distribution of OpenTelemetry Collector, so you might want to adjust your resources depending on the number of Kubernetes nodes you have.
You can update the Splunk Distribution of OpenTelemetry Collector values file, or specify different values during installation.
These are the default resource configurations.
resources: limits: cpu: 4 memory: 8GiUse the following approximations to determine your resource needs.
Approximation
Resource needs
Up to 500 nodes/5,000 data points per second
CPU: 500m, memory: 1 Gi
Up to 1,000 nodes/10,000 data points per second
CPU: 1, memory: 2 Gi
Up to 2,000 nodes/20,000 data points per second
CPU: 2, memory: 4 Gi
Example π
In the following example, CPU is set to 500m, and memory is set to 1 Gi.
resources: limits: cpu: 500m memory: 1Gihelm --namespace=<NAMESPACE> install my-splunk-otel-collector --set="splunkObservability.realm=<REALM>,splunkObservability.accessToken=<ACCESS_TOKEN>,clusterName=<CLUSTER_NAME>,agent.enabled=false,clusterReceiver.enabled=false,networkExplorer.enabled=true,gateway.replicaCount=1,gateway.resources.limits.cpu=500m,gateway.resources.limits.memory=1Gi" splunk-otel-collector-chart/splunk-otel-collector
Resize your Network Explorer installation π
Depending on the number of Kubernetes nodes you have, your resource needs might vary. You can make the following adjustments to your installation.
Change the resource footprint of the reducer π
The reducer is a single pod per Kubernetes cluster. If your cluster contains a large number of pods, nodes, and services, you can increase the resources allocated to it.
The reducer processes telemetry in multiple stages, with each stage partitioned into one or more shards, where each shard is a separate thread. Increasing the number of shards in each stage expands the capacity of the reducer.
Change the following parameters in the Splunk Distribution of OpenTelemetry Collector values file to increase or decrease the number of shards per reducer stage. You can set between 1-32 shards.
The default configuration is 1 shard per reducer stage.
networkExplorer: reducer: ingestShards: 1 matchingShards: 1 aggregationShards: 1
Example π
The following example uses 4 shards per reducer stage.
networkExplorer: reducer: ingestShards: 4 matchingShards: 4 aggregationShards: 4
Estimate reducer CPU and memory usage π
To estimate the CPU and memory usage the reducer may require from a node, you can use these simple formulas:
Memory in Mebibytes (Mi) = 4 * Number of nodes in your cluster + 60
Fractional CPU in milliCPU (m) = Number of nodes in your cluster + 30
This gives you an appromixate expected usage. Multiply the final numbers by a factor of 1.5 or 2 to give headroom for growth and spikes in usage.
Customize network telemetry generated by Network Explorer π
If you want to collect fewer or more network telemetry metrics, you can update the Splunk Distribution of OpenTelemetry Collector values file.
The following sections show you how to disable or enable different metrics.
Enable all metrics, including metrics turned off by default π
networkExplorer: reducer: disableMetrics: - none
Disable entire metric categories π
networkExplorer: reducer: disableMetrics: - tcp.all - udp.all - dns.all - http.all
Disable an individual TCP metric π
networkExplorer: reducer: disableMetrics: - tcp.bytes - tcp.rtt.num_measurements - tcp.active - tcp.rtt.average - tcp.packets - tcp.retrans - tcp.syn_timeouts - tcp.new_sockets - tcp.resets
Disable an individual UDP metric π
networkExplorer: reducer: disableMetrics: - udp.bytes - udp.packets - udp.active - udp.drops
Disable an individual DNS metric π
networkExplorer: reducer: disableMetrics: - dns.client.duration.average - dns.server.duration.average - dns.active_sockets - dns.responses - dns.timeouts
Disable an individual HTTP metric π
networkExplorer: reducer: disableMetrics: - http.client.duration.average - http.server.duration.average - http.active_sockets - http.status_code
Disable an internal metric π
networkExplorer: reducer: disableMetrics: - ebpf_net.bpf_log - ebpf_net.otlp_grpc.bytes_sent - ebpf_net.otlp_grpc.failed_requests - ebpf_net.otlp_grpc.metrics_sent - ebpf_net.otlp_grpc.requests_sent - ebpf_net.otlp_grpc.successful_requests - ebpf_net.otlp_grpc.unknown_response_tags
Note
This list represents the set of internal metrics which are enabled by default.
Enable entire metric categories π
networkExplorer: reducer: enableMetrics: - tcp.all - udp.all - dns.all - http.all - ebpf_net.all
Enable an individual TCP metric π
networkExplorer: reducer: enableMetrics: - tcp.bytes - tcp.rtt.num_measurements - tcp.active - tcp.rtt.average - tcp.packets - tcp.retrans - tcp.syn_timeouts - tcp.new_sockets - tcp.resets
Enable an individual UDP metric π
networkExplorer: reducer: enableMetrics: - udp.bytes - udp.packets - udp.active - udp.drops
Enable an individual DNS metric π
networkExplorer: reducer: enableMetrics: - dns.client.duration.average - dns.server.duration.average - dns.active_sockets - dns.responses - dns.timeouts
Enable an individual HTTP metric π
networkExplorer: reducer: enableMetrics: - http.client.duration.average - http.server.duration.average - http.active_sockets - http.status_code
Enable an internal metric π
networkExplorer: reducer: enableMetrics: - ebpf_net.span_utilization_fraction - ebpf_net.pipeline_metric_bytes_discarded - ebpf_net.codetiming_min_ns - ebpf_net.entrypoint_info - ebpf_net.otlp_grpc.requests_sent
Note
This list does not include the entire set of internal metrics.
Example π
In the following example, all HTTP metrics along with certain individual TCP and UDP metrics are disabled. All DNS metrics are collected.
networkExplorer: reducer: disableMetrics: - http.all - tcp.syn_timeouts - tcp.new_sockets - tcp.resets - udp.bytes - udp.packets
In the following example, all HTTP metrics along with certain individual internal metrics are enabled.
Note
The
disableMetrics
flag is evaluated before theenableMetrics
flag.networkExplorer: reducer: enableMetrics: - http.all - ebpf_net.codetiming_min_ns - ebpf_net.entrypoint_info
Next steps π
Once you set up Network Explorer, you can start monitoring network telemetry metrics coming into your Splunk Infrastructure Monitoring platform using one or more of the following options:
Built-in Network Explorer navigators. To see the Network Explorer navigators, follow these steps:
From the Splunk Observability Cloud home page, select Infrastructure on the left navigator.
Select Network Explorer.
Select the card for the Network Explorer navigator you want to view.
For more information, see Use navigators in Splunk Infrastructure Monitoring.
Service map. For more information, see Monitoring service dependencies with network map.
Alerts and detectors. For more information, see Introduction to alerts and detectors in Splunk Observability Cloud.
For more information on metrics available to collect with Network Explorer, see Network telemetry metrics.