Amazon Web Services (AWS) ECS (EC2/Fargate) Logs

Coralogix provides integration to collect and send your ECS cluster logs straight to Coralogix.

General

Private Key – Your private key is a unique ID representing your company.

Application Name – The name of your main application, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.

SubSystem Name – Your application probably has multiple subsystems, for example Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

EC2

To start sending logs from your ecs ec2 containers we need to create a new ECS Task Definition.

Go to the ECS console and choose ‘Task Definitions’ -> ‘Create new task definition’.

Under ‘Select launch type compatibility:

• Choose ‘EC2’ and click on the next step

Scroll all the way down and click on the ‘Configure via JSON’ button.

• Paste there the predefined file

Replace inside the file these environment variables:

• ENDPOINT- your team endpoint depending on the geo location
• log_level- the log level in which the fluend is set to (default is “error”)
• PRIVATE_KEY- your team’s private key

You can add these environment variables, if they’re not added the default is a dynamic value:

• APP_NAME- The dynamic value will be taken from the “ecs.cluster”, can be a static value if added as an environment variable
• SUB_SYSTEM- The dynamic value will be taken from the “conainer-name”, can be a static value if added as an environment variable

Run AWS ECS Task on your cluster:

Choose Placement Template as One Task Per Host:

Note: change ‘Number of tasks’ to be your number of ECS instances in the cluster.

When the task is ready, logs will start shipping to Coralogix.

Fargate

To start sending logs from your ecs fargate containers we need to create a new ECS Task Definition.

Go to the ECS console and choose ‘Task Definitions’ -> ‘Create new task definition’.

Under ‘Select launch type compatibility:

• Choose ‘FARGATE’ and click on the next step

Scroll down and under ‘Log router integration’:

• Tick ‘Enable FireLens integration’
• For ‘Type’ choose ‘fluentd’
• For ‘Image’ write ‘docker.io/coralogixrepo/fluentd-coralogix-ecs:1.7.1’

It will add log_router container to the “Containers definitions” section.

If you haven’t added your application container yet then do it now to have two containers on the list.

Configure awsfirelens logging driver for the container which you want to send the logs from (in the example above it is testapp container):

• Under log configuration, the log driver must be: awsfirelens
• The @type should be: null

Next click on the ‘log_router’ container to edit it:

Under ‘Environment variables’ add these variables:

• APP_NAME- your application name
• ENDPOINT- your coralogix cluster URL.
• log_level- the log level in which the fluend is set to (use ‘error’)
• PRIVATE_KEY- your team’s private key
• SUB_SYSTEM- the subsystem name that will appear in Coralogix

In order to get dynamic values for APP_NAME and SUB_SYSTEM, delete the lines from the Environment variables. The values are taken from:

• APP_NAME- value is taken from the field: ecs_cluster
• SUB_SYSTEM- value is taken from the field: container_name

Example:

Save the changes to the container.

Scroll all the way down and click on the ‘Configure via JSON’ button.

Replace ‘fireLensConfiguration’:

      "firelensConfiguration": {
        "type": "fluentd"
      },

with

      "firelensConfiguration": {
        "type": "fluentd",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluentd/etc/firelens.conf"
        }
      },

Note: To manage multiline logs you will need to build the fluentd image from here while adding a concat filter in the firelens.conf that match your needs.

Endpoints