Deploy the Collector with Amazon ECS EC2 🔗

Use the guided setup to deploy the Collector as a sidecar in an Amazon ECS EC2 cluster.

To access the guided setup for AWS integration, perform the following steps:

Log in to Splunk Observability Cloud.
On the navigation menu, select Data Management.
Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.
Select the tile for Amazon ECS EC2.
Follow the steps provided in the guided setup, which provides a JSON task definition for the Collector.

Choose one of the following Collector configuration options:

Default: The file /etc/otel/collector/ecs_ec2_config.yaml in the Collector image is used for the Collector configuration.
File: Specify the file to use for the Collector configuration. See Deploy the Collector using a custom configuration.
AWS Parameter Store: Specify the AWS Parameter Store key or ARN to use for the Collector configuration. See Deploy the Collector using the AWS Parameter Store.

Add the Collector as a sidecar 🔗

Note

To use this option you need to be familiar with Amazon ECS EC2 launch. See Getting started with the classic console using Amazon EC2 for further reading.

Open the ECS task definition in which you’ll add the Collector sidecar:

Locate the task definition for the Collector from the repository .
Merge the definitions of the Collector with the existing ECS task definition.
Replace MY_SPLUNK_ACCESS_TOKEN and MY_SPLUNK_REALM with valid values. You can pin the image version to a specific version instead of latest if you want to avoid automatic upgrades.

The Collector is configured to use the default configuration file /etc/otel/collector/ecs_ec2_config.yaml. The Collector image Dockerfile is available at Dockerfile and the contents of the default configuration file can be seen at ECS EC2 configuration .

Notes:

You do not need the awsecscontainermetrics receiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in Deploy the Collector using a custom configuration.
To exclude metrics assign them as a stringified array to environment variable METRICS_TO_EXCLUDE.
You can set the memory limit for the memory_limiter processor using environment variable SPLUNK_MEMORY_LIMIT_MIB. The default memory limit is 512 MiB.

Deploy the Collector using a custom configuration 🔗

To use a custom configuration file, replace the value of the SPLUNK_CONFIG environment variable with the file path of the custom configuration file in the Collector task definition.

Alternatively, you can specify the custom configuration YAML directly using the SPLUNK_CONFIG_YAML environment variable, as described in Configure the Amazon Elastic Container Service Observer extension.

Configure the Amazon Elastic Container Service Observer extension 🔗

Use the Amazon Elastic Container Service Observer (ecs_observer) extension in your custom configuration to discover metric targets in running tasks, filtered by service names, task definitions, and container labels.

Prerequisites 🔗

The following applies:

The Collector must run as an ECS daemon. See Launch the Collector as a daemon from the ECS console.
The ECS Observer is currently limited to Prometheus targets.
The ECS Observer requires the read-only permissions below. Add them to the customer-managed policy that is attached to the task role.

ecs:List*
ecs:Describe*

Launch the Collector as a daemon from the ECS console 🔗

To launch the Collector from the Amazon ECS console:

Go to your cluster in the console and select Services.
Select Create and define the following options:

Launch Type: EC2

Task Definition (Family): splunk-otel-collector

Task Definition (Revision): 1 (or whatever the latest is in your case)

Service Name: splunk-otel-collector

Service type: DAEMON

Leave everything else as default and proceed to Next step until you’re required to create the service.
Select Create Service to deploy the Collector onto each node in the ECS cluster. You should see infrastructure and docker metrics flowing soon.

Configuration example 🔗

The following example configures the ecs_observer to find Prometheus targets in the lorem-ipsum-cluster cluster and us-west-2 region, where the task ARN pattern is ^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$. The results are written to /etc/ecs_sd_targets.yaml.

The prometheus receiver is configured to read targets from the results file. The values for access_token and realm are read from the SPLUNK_ACCESS_TOKEN and SPLUNK_REALM environment variables, which you must specify in your container definition.

extensions:
  ecs_observer:
    refresh_interval: 10s
    cluster_name: 'lorem-ipsum-cluster'
    cluster_region: 'us-west-2'
    result_file: '/etc/ecs_sd_targets.yaml'
    task_definitions:
      - arn_pattern: "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$"
       metrics_ports: [9113]
       metrics_path: /metrics
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'lorem-ipsum-nginx'
          scrape_interval: 10s
          file_sd_configs:
            - files:
                - '/etc/ecs_sd_targets.yaml'
processors:
  batch:
  resourcedetection:
    detectors: [ecs]
    override: false
exporters:
  signalfx:
    access_token: ${SPLUNK_ACCESS_TOKEN}
    realm: ${SPLUNK_REALM}
service:
  extensions: [ecs_observer]
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch, resourcedetection]
      exporters: [signalfx]

Deploy the Collector using the AWS Parameter Store 🔗

To use the AWS Parameter Store, specify the config YAML directly in the SPLUNK_CONFIG_YAML environment variable instead of using SPLUNK_CONFIG.

Note

To have read access to the Parameter Store add the policy AmazonSSMReadOnlyAccess to the task role. See Systems manager parameter store for more information.

Follow these steps:

Store the custom configuration for the Configure the Amazon Elastic Container Service Observer extension in the parameter splunk-otel-collector-config in the AWS Systems Manager Parameter Store.
Next, assign the splunk-otel-collector-config parameter to SPLUNK_CONFIG_YAML using the valueFrom option, as shown in the following example:

{
"name": "lorem-ipsum-cluster",
"valueFrom": "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$""
}

Caution

The AWS Parameter Store limits the size of the config file to 4096 bytes.

This page was last updated on Nov 26, 2024.

Related Topics