This tutorial demonstrates how to deploy the OpenTelemetry Collector to AWS ECS on EC2 (**Linux** container instances) to facilitate the collection of logs, metrics, and traces generated by ECS Tasks. This integration is not intended for Fargate tasks.

For **Windows Server** EC2 container instances, use the [AWS ECS-EC2 (Windows) using OpenTelemetry](https://coralogix.com/docs/opentelemetry/configuration-options/aws-ecs-ec2-windows-using-opentelemetry/index.md) guide instead. It covers Windows-specific parameters, configuration sources, and Service Discovery.

Telemetry is sent to Coralogix via the Coralogix Exporter, which allows for the use of enrichments such as dynamic `application` or `subsystem` name, defined using `application_name_attributes` and `subsystem_name_attributes`, respectively. Find out more [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/coralogixexporter).

## Prerequisites

- [Coralogix account](https://signup.coralogix.com/#/)
- [AWS account](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html) with [AWS credentials configured](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html)
- [ecs-cli](https://github.com/aws/amazon-ecs-cli#installing)
- [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
- Coralogix distribution for OpenTelemetry ("CDOT") ECS Service daemon, deployed using [CloudFormation template](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/opentelemetry/ecs-ec2) or [Terraform module](https://github.com/coralogix/terraform-coralogix-aws/tree/master/modules/ecs-ec2)

## Image

This implementation utilises the **Coralogix Opentelemetry Collector** image [coralogixrepo/coralogix-otel-collector](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector/tags). This image is an enhanced version of the official OpenTelemetry Contrib distribution, featuring custom components and extended configuration loading options, including environment variable support.

Note the latest tag on this image is not updated, you must explicitly select a version when pulling. Tags can be found [here](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector/tags).

The image configuration utilizes the otlp receiver for both HTTP (on 4318) and GRPC (on 4317). Data can be sent using either endpoint.

## OpenTelemetry configuration

The collector loads its configuration from an S3 object at startup. Provide the bucket and key via the CloudFormation parameters `S3ConfigBucket` and `S3ConfigKey` (both required). The Coralogix UI deployment flow uploads a default configuration for you; to customize, edit the uploaded file in S3 and redeploy the task.

A reference configuration that works with the `coralogixrepo/coralogix-otel-collector` image is maintained in the [Terraform module](https://github.com/coralogix/terraform-coralogix-aws/tree/master/modules/ecs-ec2).

## Deploy a New ECS Cluster

If you already have an existing ECS cluster, skip this step.

Deploy a new cluster:

```bash
ecs-cli up --region <region> --keypair <your-key-pair> --cluster <cluster-name> --size <no. of instances> --capability-iam
```

**Notes**:

- The `--keypair` flag is not mandatory. However, if not supplied, you cannot connect to any of the instances in the cluster via SSH. Create a key pair using the command below:

```bash
aws ec2 create-key-pair --key-name MyKeyPair --query 'KeyMaterial' --output text > MyKeyPair.pem
```

- The `ecs-cli up` command will leverage CloudFormation to create an ECS cluster.
- Default values will be used to create and configure a VPC and Subnets. These and other values can be controlled from `ecs-cli` using the following command:

```bash
ecs-cli up --help
```

## Deploy OTEL agent ECS task definition & service

Once an ECS cluster has been deployed, deploy a task definition to be used by ECS to create an ECS service to run OpenTelemetry.

- An AWS **ECS** **task definition** serves as template defining a container configuration.
- An **ECS service** is a configuration item that defines and orchestrates how a task definition should be run.

**Option 1: CloudFormation template**

Deploy the CloudFormation template from [`coralogix/cloudformation-coralogix-aws`](https://github.com/coralogix/cloudformation-coralogix-aws/tree/main/opentelemetry/ecs-ec2). The required parameters are `S3ConfigBucket`, `S3ConfigKey`, `ClusterName`, `CDOTImageVersion`, `CoralogixRegion`, and `CoralogixApiKey`; profiling, health check, memory, and task role are configurable via optional parameters.

For the full parameter reference and a ready-to-use `aws cloudformation deploy` command, see the synced [AWS ECS-EC2 OpenTelemetry CloudFormation reference](https://coralogix.com/docs/external/cloudformation-coralogix-aws/opentelemetry/ecs-ec2/index.md).

Once the template is deployed successfully, verify that the container is running:

```bash
ecs-cli ps --region <region> -c <cluster name>
```

**Option 2: Terraform module**

- This [ECS EC2 Open Telemetry Agent Terraform module](https://github.com/coralogix/terraform-coralogix-aws/blob/master/modules/ecs-ec2/README.md) sets up the OTEL Agent as an ECS Daemon Service on a specified ECS cluster. See readme for usage instructions.

## Configure the Application Container with the Location of the OTEL Agent

The Coralogix OTEL Collector is typically deployed as a Daemon Service [Agent](https://opentelemetry.io/docs/collector/deployment/agent/) on each ECS container instance / EC2 instance. The OTEL Agent Task is connected using host networking, so it is accessible via the primary private IP address of the EC2 instance.

### Preliminary notes

#### OTEL configuration for ECS and Fargate tasks

For ECS Tasks running on EC2, Tasks that are OTEL-instrumented will need to discover the IP address of an OTEL Agent to connect with. The discovery approach will differ based on which [AWS ECS Task networking modes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html) your ECS Tasks utilize. In each case, set the [`OTEL_EXPORTER_OTLP_ENDPOINT` environment variable](https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/) to the address of the OTEL Agent. Your ECS Task should be configured to connect to the OTEL Collector daemon task listening at the primary IP of the EC2 host.

#### Metadata fetching from within a container on ECS

In some cases, when fetching metadata from the EC2 metadata endpoint from within a container on ECS, it may be useful to modify the instance metadata options. This is particularly relevant for ensuring reliable access to the metadata service when additional network hops are involved. The following command can be used to modify these options:

```bash
aws ec2 modify-instance-metadata-options \ --instance-id <instance_id> \ --http-put-response-hop-limit 2 \ --http-endpoint enabled \ --region <your-aws-region>
```

- *--http-put-response-hop-limit 2*: This sets the hop limit for PUT response headers to 2, allowing the metadata service to be accessed through one additional network hop, which is common in containerized environments.
- *--http-endpoint enabled*: This ensures the metadata service endpoint is enabled and accessible.

Adding this configuration helps ensure that the ECS task can reliably fetch the necessary metadata, such as the primary IP of the EC2 host, especially in complex network setups.

### Methods

- **Method 1: Using EC2 instance metadata v1.** The simplest and recommended approach is to retrieve the EC2 host's primary IP address from the [EC2 instance metadata v1](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) endpoint. This method uses HTTP protocol and gRPC protocol on port 4317, which can be adjusted as needed.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://"$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4 )":4317"
```

- **Method 2: from EC2 instance metadata v2.** As a variation, secure the same call with the [EC2 instance metadata v2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) security token.

```bash
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/local-ipv4
```

- **Method 3: from ECS container metadata file.** Alternately, if you have the [ECS container metadata file](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-metadata.html#metadata-file-format) enabled, the host private IP can be parsed from the file:

```bash
cat $ECS_CONTAINER_METADATA_FILE | jq -r .HostPrivateIPv4Address
```

**Set variables during the application container entryPoint**

Update the application container to set `OTEL_EXPORTER_OTLP_ENDPOINT` at startup.

One approach is to override the [ECS Task Definition's entryPoint](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/example_task_definitions.html#example_task_definition-ping). Prepend setting variables prior to calling the original application command. E.g.:

```json
"entryPoint": [
  "sh",
  "-c",
  "export OTEL_EXPORTER_OTLP_ENDPOINT='http://'$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4 )':4317'; <Actual Startup Command>"
]
```

Other approaches could include embedding commands in the Entrypoint script file itself, calling out to a helper script, calling an intermediate entry point wrapper script, etc. Given the wide variety of container configuration options, it is left to the customer to decide on the optimal ways to script and set environment variables.

## Configure the application container to send identifying resource attributes

Applications that are instrumented for OpenTelemetry, can mark their telemetry by specifying attribute name/value pairs in the [`OTEL_RESOURCE_ATTRIBUTES` environment variable](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_resource_attributes).

If you decide on Container ID as the resource attribute identifier for telemetry, obtain container ID from ECS Container Metadata endpoint.

```bash
#!/bin/bash

# Must run within an ECS Docker container
# Requires jq cli tool

# get container ID
containerID=$(curl ${ECS_CONTAINER_METADATA_URI_V4} | jq '.DockerId' -r)

# set env variable for resource attributes
export OTEL_RESOURCE_ATTRIBUTES="containerID=${containerID},$OTEL_RESOURCE_ATTRIBUTES"
```

Like `OTEL_EXPORTER_OTLP_ENDPOINT`, the `OTEL_RESOURCE_ATTRIBUTES` environment variable should be set upon container startup.

## Additional resources

|          |                                                                                                                                                                                                                   |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| External | [GitHub Repo](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2/template.yaml#L376)                                                                                     |
| Features | [Coralogix APM Features](https://coralogix.com/docs/user-guides/apm/getting-started/apm-onboarding-tutorial/index.md) [APM using Amazon EC2](https://coralogix.com/docs/integrations/aws/apm-amazon-ec2/index.md) |

ECS to Coralogix Using OpenTelemetry

## Support

**Need help?**

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Contact us **via our in-app chat** or by emailing [support@coralogix.com](mailto:support@coralogix.com).