Deploy the Collector with Amazon ECS EC2 🔗
Use the guided setup to deploy the Collector as a Sidecar in an Amazon ECS EC2 cluster. The guided setup provides a JSON task definition for the Collector.
Choose one of the following Collector configuration options:
Default: The file /etc/otel/collector/ecs_ec2_config.yaml in the Collector image is used for the Collector configuration.
File: Specify the file to use for the Collector configuration. See Use a custom configuration.
AWS Parameter Store: Specify the AWS Parameter Store key or ARN to use for the Collector configuration. See Use a custom configuration.
To access the guided setup for AWS integration, perform the following steps:
Log in to Splunk Observability Cloud.
On the navigation menu, select Data Management.
Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.
Select the tile for Amazon ECS EC2.
Follow the steps provided in the guided setup.
Getting started 🔗
The following sections describe how to create a task definition and launch the Collector. A task definition is required to run Docker containers in Amazon ECS. After creating the task definition, you need to launch the Collector.
Add the Collector as a Sidecar 🔗
Note
To use this option you need to be familiar with Amazon ECS EC2 launch type. See Getting started with the classic console using Amazon EC2 for further reading.
Open the ECS task definition to which the Collector Sidecar is going to be added:
Locate the task definition for the Collector from the repository .
Merge the definitions of the Collector with the existing ECS task definition.
Replace
MY_SPLUNK_ACCESS_TOKENandMY_SPLUNK_REALMwith valid values. You can pin the image version to a specific version instead oflatestif you want to avoid automatic upgrades.
The Collector is configured to use the default configuration file /etc/otel/collector/ecs_ec2_config.yaml. The Collector image Dockerfile is available at Dockerfile and the contents of the default configuration file can be seen at ECS EC2 configuration .
Notes:
You do not need the
awsecscontainermetricsreceiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in Use a custom configuration.To exclude metrics assign them as a stringified array to environment variable
METRICS_TO_EXCLUDE.You can set the memory limit for the
memory_limiterprocessor using environment variableSPLUNK_MEMORY_LIMIT_MIB. The default memory limit is 512 MiB.
Use a custom configuration 🔗
To use a custom configuration file, replace the value of the SPLUNK_CONFIG environment variable with the file path of the custom configuration file in the Collector task definition.
Alternatively, you can specify the custom configuration YAML directly using the SPLUNK_CONFIG_YAML environment variable, as described in Configure ecs_observer.
Configure ecs_observer 🔗
Use extension Amazon Elastic Container Service Observer (ecs_observer) in your custom configuration to discover metrics targets in running tasks, filtered by service names, task definitions, and container labels. ecs_observer is currently limited to Prometheus targets and requires the read-only permissions below. The Collector should be configured to run as an ECS Daemon. You can add the permissions to the task role by adding them to a customer-managed policy that is attached to the task role.
ecs:List*
ecs:Describe*
The following custom configuration examples show the ecs_observer configured to find Prometheus targets in the lorem-ipsum-cluster cluster and us-west-2 region, where the task ARN pattern is ^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$.
The results are written to /etc/ecs_sd_targets.yaml. The prometheus receiver is configured to read targets from the results file. The values for access_token and realm are read from the SPLUNK_ACCESS_TOKEN and SPLUNK_REALM environment variables , which must be specified in your container definition.
extensions:
ecs_observer:
refresh_interval: 10s
cluster_name: 'lorem-ipsum-cluster'
cluster_region: 'us-west-2'
result_file: '/etc/ecs_sd_targets.yaml'
task_definitions:
- arn_pattern: "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$"
metrics_ports: [9113]
metrics_path: /metrics
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'lorem-ipsum-nginx'
scrape_interval: 10s
file_sd_configs:
- files:
- '/etc/ecs_sd_targets.yaml'
processors:
batch:
resourcedetection:
detectors: [ecs]
override: false
exporters:
signalfx:
access_token: ${SPLUNK_ACCESS_TOKEN}
realm: ${SPLUNK_REALM}
service:
extensions: [ecs_observer]
pipelines:
metrics:
receivers: [prometheus]
processors: [batch, resourcedetection]
exporters: [signalfx]
Launch the Collector as a Daemon 🔗
To launch the Collector from the Amazon ECS console:
Go to your cluster in the console.
Select Services.
Select Create.
Select the following options: #. Launch Type: EC2 #. Task Definition (Family): splunk-otel-collector #. Task Definition (Revision): 1 (or whatever the latest is in your case) #. Service Name: splunk-otel-collector #. Service type: DAEMON #. Leave everything else at default.
Select Next step.
Leave everything on this next page at their defaults and select Next step.
Leave everything on this next page at their defaults and select Next step.
Select Create Service to deploy the Collector onto each node in the ECS cluster. You should see infrastructure and docker metrics flowing soon.
Use the AWS Parameter Store 🔗
Use the SPLUNK_CONFIG_YAML environment variable to specify the configuration YAML directly. Use SPLUNK_CONFIG_YAML in place of SPLUNK_CONFIG.
For example, first, store the custom configuration for the Configure ecs_observer in a parameter called splunk-otel-collector-config in the AWS Systems Manager Parameter Store.Next, assign the parameter to SPLUNK_CONFIG_YAML using the valueFrom option, as shown in the following example:
{
"name": "lorem-ipsum-cluster",
"valueFrom": "^arn:aws:ecs:us-west-2:906383545488:task-definition/lorem-ipsum-task:[0-9]+$""
}
Note
You should add policy AmazonSSMReadOnlyAccess to the task role for the task to have read access to the Parameter Store. See Systems manager parameter store for more information.