Seamlessly stream logs, metrics, and traces generated by AWS ECS Fargate containers to Coralogix for optimal monitoring, analysis, and visualization.
Send Coralogix your ECS Fargate logs using AWS FireLens via Fluent Bit, a lightweight data shipper for your AWS ECS Fargate workloads.
This integration requires that you deploy the fluentbit log_router into an existing AWS ECS Fargate task definition. The example below uses an AWS customized Fluent Bit image called aws-for-fluent-bit [init version]. A CloudFormation template may also be used.
The aws-for-fluent-bit image, maintained by AWS here, enables loading the Fluent Bit configuration via S3 or local files, making it more convenient and dynamic than using a static configuration in your container image.
The base_filters.conf file includes a set of filters to ensure proper ingestion by the Coralogix backend. This should be included as the first configuration file for your instance deployment. Ensure you upload this to an S3 bucket in your AWS account.
You can load multiple configuration files from S3 to build your final configuration by setting custom environment variables within the task definition.
The following is an example container declaration.
"containerDefinitions": [ { <Existing Container Definitions> }, { "name": "log_router", "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.31.12", "cpu": 0, "portMappings": [], "essential": false, "environment": [ { "name": "aws_fluent_bit_init_s3_1", "value": "arn:aws:s3:::<Your S3 Bucket>/base_filters.conf" }, { "name": "aws_fluent_bit_init_s3_2", "value": "arn:aws:s3:::<Your S3 Bucket>/more_filters.conf" }, { "name": "aws_fluent_bit_init_s3_3", "value": "arn:aws:s3:::<Your S3 Bucket>/custom_parser.conf" } ], "mountPoints": [], "volumesFrom": [], "user": "0", "firelensConfiguration": { "type": "fluentbit", "options": {} } } ]
Notes:
aws_fluent_bit_init_s3_1
environment variable points to the base_filters.conf file hosted in your S3 bucket.aws_fluent_bit_init_file_1
instead. You can use S3 and local files in the same deployment.To allow container access to the S3 object, provide the s3:GetObject and s3:GetBucketLocation action permissions to the task, as in the following example.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation" ], "Resource": "<Your specific bucket ARN>" }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "<Your specific bucket ARN>/*" } ] }
Notes:
After adding the above container to your existing Task Definition, adjust the logConfiguration for the containers you wish to forward to Coralogix.
Add the following “logConfiguration” section to each of your application containers at the root.
"logConfiguration": { "logDriver": "awsfirelens", "options": { "Format": "json_lines", "Header": "authorization Bearer <<API_key>>", "Host": "ingress.<Coralogix Domain>", "Name": "http", "Port": "443", "Retry_Limit": "10", "TLS": "On", "URI": "/logs/v1/singles", "compress": "gzip" } }
Notes:
Send your ECS Fargate metrics and traces using OpenTelemetry (Otel) Collector, offering a vendor-agnostic implementation of how to receive, process and export telemetry data.
This integration requires that you add the Otel Collector as a sidecar agent to your ECS Task Definitions. The example below uses an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT). A CloudFormation template may also be used.
The ADOT image, maintained by AWS here, enables the loading of the OpenTelemetry configuration via Systems Manager Parameter Store, making configuration adjustment more convenient and dynamic when compared with a static configuration in your container image.
The config.yaml file includes a standard configuration, ensuring proper ingestion by the Coralogix backend. Create this Parameter Store in the same region as your ECS cluster, as in the CloudFormation template provided. Once the Parameter Store has been created, add the container to your existing Task Definition.
"containerDefinitions": [ { <Existing Container Definitions> }, { "name": "otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector", "cpu": 0, "portMappings": [ { "name": "otel-collector-4317-tcp", "containerPort": 4317, "hostPort": 4317, "protocol": "tcp", "appProtocol": "grpc" }, { "name": "otel-collector-4318-tcp", "containerPort": 4318, "hostPort": 4318, "protocol": "tcp", "appProtocol": "grpc" } ], "essential": false, "environment": [ { "name": "PRIVATE_KEY", "value": "<Coralogix PrivateKey>" }, { "name": "CORALOGIX_DOMAIN", "value": "<Coralogix Domain>" } ], "mountPoints": [], "volumesFrom": [], "secrets": [ { "name": "AOT_CONFIG_CONTENT", "valueFrom": "config.yaml" } ], "logConfiguration": { "logDriver": "awsfirelens", "options": { "Format": "json_lines", "Header": "authorization Bearer <API_key>", "Host": "ingress.<Coralogix Domain>", "Name": "http", "Port": "443", "Retry_Limit": "10", "TLS": "On", "URI": "/logs/v1/singles", "compress": "gzip" } } } ]
Notes:
API_key
."logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "true", "awslogs-group": "<Log Group Name>", "awslogs-region": "<Your Region>", "awslogs-stream-prefix": "<Stream Prefix>" } }
If you wish to store your Coralogix Privatekey
in Secrets Manager, remove the "Header"
from "options"
and create one under "secretOptions"
and reference the Secret’s ARN. Store the secret as plaintext with the same format as above. You will also need to add the secretsmanager:GetSecretValue
permission to your ECS Task Execution Role.
"secretOptions": [ { "name": "Header", "valueFrom": "arn:aws:secretsmanager:us-east-1:<redacted>:secret:<redacted>" } ]
To allow container access to the Systems Manager Parameter Store, provide the ssm:GetParameters
action permissions to the Task Execution Role, as in the following example.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:GetParameters" ], "Resource": [ "arn:aws:ssm:region:aws_account_id:parameter/parameter_name" ] } ] }
Notes:
Need help?
Contact us via our in-app chat or by emailing [email protected].