This tutorial demonstrates how to deploy OpenTelemetry to AWS ECS-EC2 to facilitate the collection of logs, metrics, and traces.
Telemetry is sent to Coralogix via the Coralogix Exporter, which allows for the use of enrichments such as dynamic application
or subsystem
name, defined using application_name_attributes
and subsystem_name_attributes
, respectively. Find out more here.
This implementation utilises the Coralogix Opentelemetry Collector image coralogixrepo/coralogix-otel-collector. This image is an enhanced version of the official OpenTelemetry Contrib distribution, featuring custom components and extended configuration loading options, including environment variable support.
Note the latest tag on this image is not updated, you must explicitly select a version when pulling. Tags can be found here.
The image configuration utilizes the otlp receiver for both HTTP (on 4318) and GRPC (on 4317). Data can be sent using either endpoint.
The OpenTelemetry configuration for the agent is stored in a Base64 encoded environment variable and applied at runtime. This allows you to dynamically pass any configuration values you choose as a parameter to CloudFormation.
The following configuration files work directly with the coralogixrepo/coralogix-otel-collector docker image for ECS:
Create other configurations by combining logs, metrics and/or traces.
If you already have an existing ECS cluster, skip this step.
Deploy a new cluster:
ecs-cli up --region <region> --keypair <your-key-pair> --cluster <cluster-name> --size <no. of instances> --capability-iam
Notes:
--keypair
flag is not mandatory. However, if not supplied, you cannot connect to any of the instances in the cluster via SSH. Create a key pair using the command below:aws ec2 create-key-pair --key-name MyKeyPair --query 'KeyMaterial' --output text > MyKeyPair.pem
ecs-cli up
command will leverage CloudFormation to create an ECS cluster. ecs-cli
using the following command:ecs-cli up --help
Once an ECS cluster has been deployed, deploy a task definition to be used by ECS to create an ECS service to run OpenTelemetry.
Option 1: CloudFormation template
Once the template is deployed successfully, verify that the container is running:
ecs-cli ps --region <region> -c <cluster name>
Option 2: Terraform module
The Coralogix OTEL Collector is typically deployed as a Daemon Service Agent on each ECS container instance / EC2 instance. The OTEL Agent Task is connected using host networking, so it is accessible via the primary private IP address of the EC2 instance.
Note: The steps in this section will not apply to Fargate Tasks. These will connect to an OTEL Collector Gateway service deployed in a Gateway architecture, or an OTEL “sidecar” container. Daemon deployments are not supported in Fargate. Use a suitable OTEL service discovery approach.
For ECS Tasks running on EC2, Tasks that are OTEL-instrumented will need to discover the IP address of an OTEL Agent to connect with. The discovery approach will differ based on which AWS ECS Task networking modes your ECS Tasks utilize. In each case, set the [OTEL_EXPORTER_OTLP_ENDPOINT
environment variable](https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/) to the address of the OTEL Agent.
Your ECS Task should be configured to connect to the OTEL Collector daemon task listening at the primary IP of the EC2 host.
export OTEL_EXPORTER_OTLP_ENDPOINT="http://"$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4 )":4317"
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/local-ipv4
cat $ECS_CONTAINER_METADATA_FILE | jq -r .HostPrivateIPv4Address
Set variables during the application container entryPoint
Update the application container to set OTEL_EXPORTER_OTLP_ENDPOINT
at startup.
One approach is to override the ECS Task Definition’s entryPoint. Prepend setting variables prior to calling the original application command. E.g.:
"entryPoint": [ "sh", "-c", "export OTEL_EXPORTER_OTLP_ENDPOINT='http://'$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4 )':4317'; <Actual Startup Command>" ]
Other approaches could include embedding commands in the Entrypoint script file itself, calling out to a helper script, calling an intermediate entry point wrapper script, etc. Given the wide variety of container configuration options, it is left to the customer to decide on the optimal ways to script and set environment variables.
Applications that are instrumented for OpenTelemetry, can mark their telemetry by specifying attribute name/value pairs in the OTEL_RESOURCE_ATTRIBUTES
environment variable.
If you decide on Container ID as the resource attribute identifier for telemetry, obtain container ID from ECS Container Metadata endpoint.
#!/bin/bash # Must run within an ECS Docker container # Requires jq cli tool # get container ID containerID=$(curl ${ECS_CONTAINER_METADATA_URI_V4} | jq '.DockerId' -r) # set env variable for resource attributes export OTEL_RESOURCE_ATTRIBUTES="containerID=${containerID},$OTEL_RESOURCE_ATTRIBUTES"
Like OTEL_EXPORTER_OTLP_ENDPOINT
, the OTEL_RESOURCE_ATTRIBUTES
environment variable should be set upon container startup.
External | GitHub Repo |
Features | Coralogix APM Features APM using Amazon EC2 |
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Contact us via our in-app chat or by emailing [email protected].