Docs » Get started with the Splunk Distribution of the OpenTelemetry Collector » Other Collector deployment tools and options: ECS/EC2, Fargate, Nomad, PCF » Deploy the Collector with Amazon ECS Fargate

Deploy the Collector with Amazon ECS Fargate 🔗

Deploy the Splunk Distribution of the OpenTelemetry Collector as a Daemon service in an Amazon ECS with AWS Fargate. Unless stated otherwise, the Collector is deployed as a sidecar (additional container) to ECS tasks. AWS Fargate is classified as a container, and Fargate metrics are classified as container metrics.

For an example on how to use the Collector in an ECS Fargate environment to monitor a Java application, see Scenario: Monitor a Java service with OpenTelemetry in ECS Fargate.

Requirements 🔗

Before you proceed, you require previous knowledge of AWS Fargate (Fargate). See the official AWS Fargate user guide for more information.

This deployment requires Collector release v0.33.0 or higher, which corresponds to image tag 0.33.0 or above. See the image repository to download the latest image.

Use the guided set-up 🔗

Use the guided set-up to deploy the Collector as a sidecar to ECS tasks. Choose one of the following Collector configuration options:

  • Default: The /etc/otel/collector/fargate_config.yaml file in the Collector image is used for the Collector configuration.

  • File: Specify the file to use for the Collector configuration. See ecs_observer configuration.

  • AWS Parameter Store: Specify the AWS Parameter Store key or ARN to use for the Collector configuration. See ecs_observer configuration.

Access the guided set-up 🔗

Open the Amazon Fargate guided setup .

Optionally, you can navigate to the guided setup on your own:

  1. Log in to Splunk Observability Cloud.

  2. On the navigation menu, select Data Management.

  3. Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.

  4. Select the tile for Amazon Fargate.

  5. Follow the steps provided in the guided set-up.

Get started 🔗

Copy the default Collector container definition shown in the example and make the following changes:

  • Replace MY_SPLUNK_ACCESS_TOKEN and MY_SPLUNK_REALM with valid values.

  • Update the image tag to the newest version.

  • Add the configuration to the containerDefinitions section of your task definition.

{
"environment": [
  {
    "name": "SPLUNK_ACCESS_TOKEN",
    "value": "MY_SPLUNK_ACCESS_TOKEN"
  },
  {
    "name": "SPLUNK_REALM",
    "value": "MY_SPLUNK_REALM"
  },
  {
    "name": "SPLUNK_CONFIG",
    "value": "/etc/otel/collector/fargate_config.yaml"
  },
  {
    "name": "ECS_METADATA_EXCLUDED_IMAGES",
    "value": "[\"quay.io/signalfx/splunk-otel-collector*\"]"
  }
],
"image": "quay.io/signalfx/splunk-otel-collector:0.33.0",
"essential": true,
"name": "splunk_otel_collector"
}

In this example container definition, the Collector is configured to use the default configuration file /etc/otel/collector/fargate_config.yaml. The Collector image Dockerfile is available at Dockerfile and the contents of the default configuration file can be seen at Fargate configuration . Note that the smartagent/ecs-metadata receiver is activated by default.

In summary, the default Collector container definition does the following:

  • Specifies the Collector image.

  • Sets the access token using environment variable SPLUNK_ACCESS_TOKEN.

  • Sets the realm using environment variable SPLUNK_REALM.

  • Sets the default configuration file path using environment variable SPLUNK_CONFIG.

  • Excludes ecs-metadata metrics from the Collector image using environment variable ECS_METADATA_EXCLUDED_IMAGES.

Assign a stringified array of metrics you want excluded to environment variable METRICS_TO_EXCLUDE. You can set the memory limit for the memory_limiter processor using environment variable SPLUNK_MEMORY_LIMIT_MIB. The default memory limit is 512 MiB.

Use a custom configuration 🔗

The following example shows an excerpt of the container definition for the Collector configured to use custom configuration, where /path/to/custom/config/file is a placeholder value for the actual custom configuration file path and 0.33.0 is the latest image tag at present. The custom configuration file should be present in a volume attached to the task.

{
"environment": [
  {
    "name": "SPLUNK_CONFIG",
    "value": "/path/to/custom/config/file"
  }
],
"image": "quay.io/signalfx/splunk-otel-collector:0.33.0",
"essential": true,
"name": "splunk_otel_collector"
}

The custom Collector container definition:

  • Specifies the Collector image.

  • Sets environment variable SPLUNK_CONFIG with the custom configuration file path.

Alternatively, you can specify the custom configuration YAML directly using the SPLUNK_CONFIG_YAML environment variable, as described in Configure ecs_observer.

ecs_observer configuration 🔗

Use the extension Amazon Elastic Container Service Observer (ecs_observer) in your custom configuration to discover metrics targets in running tasks, filtered by service names, task definitions and container labels. ecs_observer is currently limited to Prometheus targets and requires the read-only permissions below. You can add the permissions to the task role by adding them to a customer-managed policy that is attached to the task role.

ecs:List*
ecs:Describe*

The following custom configuration examples shows the ecs_observer configured to find Prometheus targets in the lorem-ipsum-cluster cluster and us-west-2 region, where the task ARN pattern is ^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$.

The results are written to /etc/ecs_sd_targets.yaml. The prometheus receiver is configured to read targets from the results file. The values for access_token and realm are read from the SPLUNK_ACCESS_TOKEN and SPLUNK_REALM environment variables , which must be specified in your container definition.

extensions:
  ecs_observer:
    refresh_interval: 10s
    cluster_name: 'lorem-ipsum-cluster'
    cluster_region: 'us-west-2'
    result_file: '/etc/ecs_sd_targets.yaml'
    task_definitions:
      - arn_pattern: "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$"
       metrics_ports: [9113]
       metrics_path: /metrics
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'lorem-ipsum-nginx'
          scrape_interval: 10s
          file_sd_configs:
            - files:
                - '/etc/ecs_sd_targets.yaml'
processors:
  batch:
  resourcedetection:
    detectors: [ecs]
    override: false
exporters:
  signalfx:
    access_token: ${SPLUNK_ACCESS_TOKEN}
    realm: ${SPLUNK_REALM}
service:
  extensions: [ecs_observer]
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch, resourcedetection]
      exporters: [signalfx]

Set the ARN pattern in the ecs_observer configuration 🔗

Using this task ARN pattern causes the ecs_observer to discover targets in running revisions of task lorem-ipsum-task. This means that when multiple revisions of task lorem-ipsum-task are running, the ecs_observer discovers targets outside the task in which the Collector sidecar container is running. In a sidecar deployment, the Collector and the monitored containers are in the same task, so metric targets must be within the task.

To solve this problem, use the complete task ARN as shown below. Note that you’ll need to update the task ARN pattern to keep pace with task revisions.

...
- arn_pattern: "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:3$"
...

Use a direct configuration 🔗

Since the file system is not readily available in Fargate, you need to specify the configuration YAML directly using the SPLUNK_CONFIG_YAML environment variable.

For example, you can store the custom configuration YAML in a parameter called splunk-otel-collector-config in AWS Systems Manager Parameter Store. In your Collector container definition, assign the parameter to the SPLUNK_CONFIG_YAML environment variable using valueFrom. In the following example, MY_SPLUNK_ACCESS_TOKEN and MY_SPLUNK_REALM are placeholder values and 0.33.0 is the image tag.

{
"environment": [
  {
    "name": "SPLUNK_ACCESS_TOKEN",
    "value": "MY_SPLUNK_ACCESS_TOKEN"
  },
  {
    "name": "SPLUNK_REALM",
    "value": "MY_SPLUNK_REALM"
  }
],
"secrets": [
  {
    "valueFrom": "splunk-otel-collector-config",
    "name": "SPLUNK_CONFIG_YAML"
  }
],
"image": "quay.io/signalfx/splunk-otel-collector:0.33.0",
"essential": true,
"name": "splunk_otel_collector"
}

Note

If you want a task to have read access to the Parameter Store, add the AmazonSSMReadOnlyAccess policy to the task role.

Standalone task 🔗

The ecs_observer extension is capable of scanning for targets in the entire cluster. Scanning lets you collect telemetry data by deploying the Collector in a task that is separate from tasks containing monitored applications. This is in contrast to the sidecar deployment where the Collector container, and the monitored application containers are in the same task.

Do not configure the ECS resourcedetection processor for the standalone task, as it would detect resources in the standalone Collector task itself as opposed to resources in the tasks containing the monitored applications.

AWS Graviton2 🔗

AWS Graviton2 is supported with the default Fargate configuration. The Collector Docker image can run on both AMD64 and ARM64 architectures.

This page was last updated on May 08, 2024.