Deploy the Collector with Nomad 🔗
Use the Nomad deployment orchestrator to create a job that provides a unified way to receive, process, and export metric and trace data for Splunk Observability Cloud.
Note
Job files are provided as a reference only and are not intended for production use.
Get started 🔗
To run the job files, you need:
Access to a Nomad cluster
(Optional) Access to a Consul cluster
To start a local dev agent for Nomad and Consul:
Download the nomad binary file and the consul binary .
Run the following commands in two different terminals:
$ nomad agent -dev -network-interface='{{ GetPrivateInterfaces | attr "name" }}' $ consul agent -dev
To deploy the Collector job on the Nomad cluster, set the environment variable in the Nomad job configuration, as shown in the following example:
env {
SPLUNK_ACCESS_TOKEN = "<SPLUNK_ACCESS_TOKEN>"
SPLUNK_REALM = "<SPLUNK_REALM>"
SPLUNK_MEMORY_TOTAL_MIB = 2048
// You can specify more environment variables to override default values.
}
You can specify content in the template stanza if you want to use your own Collector configuration file, as shown in the following example:
template {
data = <<EOF
# The following example shows how to set up your Collector configuration file.
extensions:
health_check: null
zpages: null
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu: null
disk: null
filesystem: null
load: null
memory: null
network: null
paging: null
processes: null
processors:
batch: null
memory_limiter:
check_interval: 2s
limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
exporters:
signalfx:
access_token: ${SPLUNK_ACCESS_TOKEN}
api_url: https://api.${SPLUNK_REALM}.signalfx.com
correlation: null
ingest_url: https://ingest.${SPLUNK_REALM}.signalfx.com
sync_host_metadata: true
debug:
verbosity: detailed
service:
extensions:
- health_check
- zpages
pipelines:
metrics:
exporters:
- logging
- signalfx
processors:
- memory_limiter
- batch
receivers:
- hostmetrics
- signalfx
EOF
destination = "local/config/otel-agent-config.yaml"
}
Deployment modes 🔗
Run the Collector as a gateway or as an agent. See Collector deployment modes for more information.
Run the Collector as a gateway 🔗
Run the Collector as a gateway by registering a service job, as shown in the following example:
$ git clone https://github.com/signalfx/splunk-otel-collector.git
$ cd splunk-otel-collector/deployments/nomad
$ nomad run otel-gateway.nomad
Use the service scheduler to schedule long lived services that should never go down. As such, the service scheduler ranks a large portion of the nodes that meet the job’s constraints and selects the optimal node to place a task group on.
Service jobs are intended to run until explicitly stopped by an operator. If a service task exits, it is considered a failure and handled according to the job’s restart and reschedule stanzas.
Run the Collector as an agent 🔗
Run the Collector as an agent by registering a system job, as shown in the following example:
$ git clone https://github.com/signalfx/splunk-otel-collector.git
$ cd splunk-otel-collector/deployments/nomad
$ nomad run otel-agent.nomad
Use the system scheduler to register jobs that should be run on all clients that meet the job’s constraints. The system scheduler is also invoked when clients join the cluster or transition into the ready state. This means that all registered system jobs are re-evaluated and their tasks are placed on the newly available nodes if the constraints are met.
The system scheduler type is useful for deploying and managing tasks that should be present on every node in the cluster. Since these tasks are managed by Nomad, they can take advantage of job updating, service discovery, and more.
Since Nomad 0.9, the system scheduler preempts eligible lower priority tasks running on a node if there isn’t enough capacity to place a system job. See preemption for details on how tasks that get preempted are chosen.
Systems jobs are intended to run until explicitly stopped either by an operator or preemption. If a system task exits, it is considered a failure and handled according to the job’s restart stanza; system jobs do not have rescheduling.