Coralogix OpenTelemetry Agent for ECS-EC2. CloudFormation template.

This CloudFormation template deploys an ECS Service and Task Definition for running the OpenTelemetry Collector agent on an ECS cluster. This deployment is able to collect Logs, Metrics and Traces. The template will deploy a daemonset which runs an instance of OpenTelemetry Collector on each node in a cluster.

CloudFormation template to launch the Coralogix Distribution for OpenTelemetry ("CDOT") into an existing ECS cluster. This CDOT deployment is able to collect Logs, Metrics and Traces. CDOT is deployed in the OTEL Agent deployment pattern, as an ECS Daemon Service type, which runs an instance of the OpenTelemetry Collector agent on each node in a cluster.

Template

• template.yaml - Uses S3 to store the OpenTelemetry configuration file. The config is typically generated by the Coralogix UI when deploying from the platform, or can be obtained from the integration chart.

Container image

This solution uses the coralogixrepo/coralogix-otel-collector image which is a custom distribution of OpenTelemetry containing custom components developed by Coralogix. The image is available on Docker Hub. See the ECS components reference for details.

The OTEL Collector/agent/daemon image used is the Coralogix Distribution for OpenTelemetry Docker Hub image. It is deployed as a Daemon ECS Task, i.e. one OTEL Collector agent container on each EC2 instance (i.e. ECS container instance) across the cluster.

CDOT extends upon the main OpenTelemetry Collector Contrib project, adding features specifically to enhance integration with AWS ECS, among other improvements.

The OTEL agent is deployed as a Daemon ECS Task and connected using host network mode. OTEL-instrumented application containers that need to send telemetry to the local OTEL agent can lookup the IP address of the CDOT container using a number of methods, making it easier for Application Tasks using awsvpc and bridge network modes to connect with the OTEL agent. OTEL-instrumented application containers should also consider which resource attributes to use as telemetry identifiers.

The CDOT OTEL agent also features enhancements specific to ECS integration. These improvements are proprietary to the Coralogix Distribution for OpenTelemetry.

Logs

The OTEL agent uses a filelog receiver to read the docker logs of all containers on the EC2 host. OTLP is also accepted. Coralogix provides the awsecscontainermetricsd receiver which enables metrics collection of all tasks on the same host. The coralogix exporter forwards telemetry to your configured Coralogix endpoint.

Logs are collected from all containers that log to /var/lib/docker/containers/*/*.log. The container requires privileges to mount the read-only host filesystem path /var/lib/docker/.

Container metrics

Container metrics are collected from all containers running on the ECS cluster. The metrics are collected using the awsecscontainermetricsd receiver. If you do not wish to collect container metrics, comment out or delete the metrics/containermetrics pipeline from the configuration.

OpenTelemetry Collector metrics

The default configuration exposes OpenTelemetry Collector metrics on port 8888 via the path /metrics. The metrics are collected using a prometheus scrape job. These are performance metrics for the OpenTelemetry Collector containers, including records received and processed and submission faults.

Traces

A GRPC(4317) and HTTP(4318) endpoint is exposed for sending traces to the local OTLP endpoint.

By default, traces are sampled at 10% using head sampling — a probabilistic sampler applied at the collection point, before any processing. When enabled, it creates a separate pipeline for sampled traces, reducing trace volume while keeping a representative sample.

The sampling configuration can be adjusted using the following parameters: - EnableHeadSampler: Enable/disable head sampling - SamplerMode: Choose between proportional, equalizing, or hash_seed sampling modes - SamplingPercentage: Set the desired sampling rate (0-100%)

Span metrics

When enabled, the spanmetrics connector generates metrics from traces, providing insights into trace performance and patterns. This feature creates additional metrics pipelines that convert span data into metrics for monitoring and alerting purposes.

Database traces

When enabled, database operation traces are processed separately with dedicated metrics generation. This feature provides specialized monitoring for database operations with optimized bucket configurations and filtering.

Requires

• An existing ECS cluster
• AWS CLI (if deploying via CLI)

Parameters

Deploy the CloudFormation template

The template requires the OpenTelemetry configuration to be stored in S3. The config is typically generated by the Coralogix UI when deploying from the platform, or can be obtained from the integration chart.

aws cloudformation deploy --template-file template.yaml --stack-name <stack_name> \
    --region <region> \
    --parameter-overrides \
        S3ConfigBucket=<your-s3-bucket> \
        S3ConfigKey=<path/to/config.yaml> \
        ClusterName=<ecs cluster name> \
        CDOTImageVersion=<image tag> \
        CoralogixApiKey=<your-api-key> \
        CoralogixRegion=<coralogix-region> \
        HealthCheckEnabled=true

Note that these are just examples of how this could be deployed. You can also deploy this template using the AWS Console or any CloudFormation management tools.

OpenTelemetry configuration

The template loads the OpenTelemetry configuration from S3. The template passes these environment variables to the collector:

• CORALOGIX_DOMAIN – region-specific domain (from CoralogixRegion parameter)
• CORALOGIX_PRIVATE_KEY – your API key (from CoralogixApiKey parameter)

Configuration source: The canonical config is maintained in the telemetry-shippers integration chart: - otel-ecs-ec2/examples/otel-config.yaml - Raw config (for download)

Alternatively, use the config generated by the Coralogix UI when deploying from the platform.

The configuration monitors container logs, listens for logs, metrics and traces on port 4317/4318, and collects container metrics using the awsecscontainermetricsd receiver.

Health check

Requires OTEL collector image version v0.4.2 or later. The default config will expose a health check on port 13133 on localhost via the path /. The health check is exposed using the health_check extension.

The healthy response should look like this:

{
  "status": "Server available",
  "upSince": "2023-10-25T15:37:32.003837622Z",
  "uptime": "2m5.2610063s"
}

ECS container health check

You can customize the health check settings using the following parameters: - HealthCheckInterval (default: 30) - HealthCheckTimeout (default: 5) - HealthCheckRetries (default: 3) - HealthCheckStartPeriod (default: 10)

Example deployment with custom health check settings:

aws cloudformation deploy --template-file template.yaml --stack-name <stack_name> \
    --region <region> \
    --parameter-overrides \
        S3ConfigBucket=<your-s3-bucket> \
        S3ConfigKey=<path/to/config.yaml> \
        ClusterName=<ecs cluster name> \
        CDOTImageVersion=<image tag> \
        CoralogixApiKey=<your-api-key> \
        CoralogixRegion=<coralogix-region> \
        HealthCheckEnabled=true \
        HealthCheckInterval=60 \
        HealthCheckTimeout=10 \
        HealthCheckRetries=5 \
        HealthCheckStartPeriod=20

Further info

See documentation: AWS ECS-EC2 using OpenTelemetry.