AWS ECS Fargate

Seamlessly stream logs, metrics, and traces generated by AWS ECS Fargate containers to Coralogix for optimal monitoring, analysis, and visualization.

Logs

Send Coralogix your ECS Fargate logs using Fluent Bit, a lightweight data shipper for your AWS ECS Fargate workloads.

Overview

This integration requires that you deploy the fluentbit log_router into an existing AWS ECS Fargate task definition. The example below uses an AWS customized Fluent Bit image called aws-for-fluent-bit [init version]. A CloudFormation template may also be used.

The aws-for-fluent-bit image, maintained by AWS here, enables loading the Fluent Bit configuration via S3 or local files, making it more convenient and dynamic than using a static configuration in your container image.

The base_filters.conf file includes a set of filters to ensure proper ingestion by the Coralogix backend. This should be included as the first configuration file for your instance deployment. Ensure you upload this to an S3 bucket in your AWS account.

You can load multiple configuration files from S3 to build your final configuration by setting custom environment variables within the task definition.

Container Declaration Within a Task Definition

The following is an example container declaration.

    "containerDefinitions": [
        {
            <Existing Container Definitions>
        },
        {
            "name": "log_router",
            "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.31.12",
            "cpu": 0,
            "portMappings": [],
            "essential": false,
            "environment": [
                {
                    "name": "aws_fluent_bit_init_s3_1",
                    "value": "arn:aws:s3:::<Your S3 Bucket>/base_filters.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_2",
                    "value": "arn:aws:s3:::<Your S3 Bucket>/more_filters.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_3",
                    "value": "arn:aws:s3:::<Your S3 Bucket>/custom_parser.conf"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {}
            }
        }
    ]

Notes:

• Our aws_fluent_bit_init_s3_1 environment variable points to the base_filters.conf file hosted in your S3 bucket.
• Add additional configuration files by increasing the _# suffix to reference additional files.
• Create your own custom image and include files locally if preferred. To do this, use environment variable aws_fluent_bit_init_file_1 instead. You can use S3 and local files in the same deployment.
• Full details can be found in the AWS documentation here.

Allow Container Access to Your S3 Bucket

In order to allow container access to the S3 object, provide the s3:GetObject and s3:GetBucketLocation action permissions to the task, as in the following example.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetObject",
				"s3:GetBucketLocation"
			],
			"Resource": "<Your specific bucket ARN>"
            #or
            "Resource": "*"
		}
	]
}

Notes:

• The permission must be added to the Task Role.

Container Adjustment

After adding the above container to your existing Task Definition, adjust the logConfiguration for the containers you wish to forward to Coralogix.

Add the following “logConfiguration” section to each of your application containers at the root.

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Format": "json_lines",
                    "Header": "private_key <Coralogix PrivateKey>",
                    "Host": "ingress.<Coralogix Domain>",
                    "Name": "http",
                    "Port": "443",
                    "Retry_Limit": "10",
                    "TLS": "On",
                    "URI": "/logs/rest/singles",
                    "compress": "gzip"
                }
            }

Notes:

• Input the “logConfiguration” section at the same level as “name”, “image”, etc.

Metrics & Traces

Send your ECS Fargate metrics and logs using OpenTelemetry (Otel) Collector, offering a vendor-agnostic implementation of how to receive, process and export telemetry data.

Overview

This integration requires that you add the Otel Collector as a sidecar agent to your ECS Task Definitions. The example below uses an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT). A CloudFormation template may also be used.

The ADOT image, maintained by AWS here, enables the loading of the OpenTelemetry configuration via Systems Manager Parameter Store, making configuration adjustment more convenient and dynamic when compared with a static configuration in your container image.

The config.yaml file includes a standard configuration, ensuring proper ingestion by the Coralogix backend. Create this Parameter Store in the same region as your ECS cluster, as in the CloudFormation template provided. Once the Parameter Store has been created, add the container to your existing Task Definition.

Container Declaration Within a Task Definition

    "containerDefinitions": [
        {
            <Existing Container Definitions>
        },
        {
            "name": "otel-collector",
            "image": "public.ecr.aws/aws-observability/aws-otel-collector",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "otel-collector-4317-tcp",
                    "containerPort": 4317,
                    "hostPort": 4317,
                    "protocol": "tcp",
                    "appProtocol": "grpc"
                },
                {
                    "name": "otel-collector-4318-tcp",
                    "containerPort": 4318,
                    "hostPort": 4318,
                    "protocol": "tcp",
                    "appProtocol": "grpc"
                }
            ],
            "essential": false,
            "environment": [
                {
                    "name": "PRIVATE_KEY",
                    "value": "<Coralogix PrivateKey>"
                },
                {
                    "name": "CORALOGIX_DOMAIN",
                    "value": "<Coralogix Domain>"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "secrets": [
                {
                    "name": "AOT_CONFIG_CONTENT",
                    "valueFrom": "config.yaml"
                }
            ],
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Format": "json_lines",
                    "Header": "private_key <Coralogix PrivateKey>",
                    "Host": "ingress.<Coralogix Domain>",
                    "Name": "http",
                    "Port": "443",
                    "Retry_Limit": "10",
                    "TLS": "On",
                    "URI": "/logs/rest/singles",
                    "compress": "gzip"
                }
            }
        }
    ]

Notes:

• The “logConfiguration” section included in the example will forward Otel logs to the Coralogix platform, as documented in our Fluent Bit log processing configuration instructions here.
• If you don’t want to have logs submitted to the Coralogix platform, replace the logConfiguration with the logDriver configuration of preference.
• To submit to Cloudwatch, use the following configuration:

"logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "<Log Group Name>",
                    "awslogs-region": "<Your Region>",
                    "awslogs-stream-prefix": "<Stream Prefix>"
                }
            }

Access Systems Manager Parameter Store

In order to allow container access to the Systems Manager Parameter Store, provide the ssm:GetParameters action permissions to the Task Execution Role, as in the following example.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": [
        "arn:aws:ssm:region:aws_account_id:parameter/parameter_name"
      ]
    }
  ]
}

Notes:

• The permission must be added to the Task Execution Role.
• After adding the above container to your existing Task Definition, your applications will submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. They will also collect container metrics from all containers in the Task Definition.

Additional Resources

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].