AWS EKS Fargate

Open in ChatGPTOpen in Claude

Collect logs, metrics, and traces from containerized workloads running on Amazon EKS on AWS Fargate.

The integration has two independent halves, and you can deploy either on its own:

• Metrics and traces through an OpenTelemetry Collector that queries the Kubelet stats API.
• Logs through the AWS log router built into the Fargate Kubelet, forwarded over Amazon Data Firehose.

For the upstream manifests, see the OTel EKS Fargate reference.

Note

Fargate hosts are managed by AWS, so node-level metrics are not available. The Collector reports pod and container metrics only.

What you need

• An EKS cluster with a cx-eks-fargate-otel namespace, linked to an AWS Fargate profile. Create a profile if none exists — workloads cannot run on Fargate without one.
• A Coralogix Send-Your-Data API key.
• kubectl and eksctl configured against the cluster.

Create the secret

Note

This integration reads the API key from PRIVATE_KEY. The OpenTelemetry ECS Fargate integration uses CORALOGIX_PRIVATE_KEY instead. Mixing them up is a common cause of a Collector that starts but never authenticates.

1.

Export the key and namespace

Set both in your shell:

export PRIVATE_KEY=<Send-Your-Data API key>
export NAMESPACE=cx-eks-fargate-otel

2.

Create the secret

Run:

kubectl create secret generic coralogix-keys -n $NAMESPACE --from-literal=PRIVATE_KEY=$PRIVATE_KEY

3.

Confirm it exists

Run:

kubectl get secret coralogix-keys -o yaml -n $NAMESPACE

Create the service account

The Collector binds to a service account to reach the Kubernetes API. Set CLUSTER_NAME and REGION; leave the rest as-is.

#!/bin/bash
CLUSTER_NAME=<EKS Cluster Name>
REGION=<EKS Cluster Region>
SERVICE_ACCOUNT_NAMESPACE=cx-eks-fargate-otel
SERVICE_ACCOUNT_NAME=cx-otel-collector
SERVICE_ACCOUNT_IAM_ROLE=EKS-Fargate-cx-OTEL-ServiceAccount-Role
SERVICE_ACCOUNT_IAM_POLICY=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

eksctl utils associate-iam-oidc-provider \
--cluster=$CLUSTER_NAME \
--approve

eksctl create iamserviceaccount \
--cluster=$CLUSTER_NAME \
--region=$REGION \
--name=$SERVICE_ACCOUNT_NAME \
--namespace=$SERVICE_ACCOUNT_NAMESPACE \
--role-name=$SERVICE_ACCOUNT_IAM_ROLE \
--attach-policy-arn=$SERVICE_ACCOUNT_IAM_POLICY \
--approve

Deploy the Collector

The manifest deploys an OpenTelemetry Collector, a ClusterIP service that receives application traces and metrics, and the cluster permissions needed to query the Kubernetes API.

1.

Set the environment variables

They are listed at the top of the manifest.

2.

Apply it

Run:

kubectl apply -f cx-eks-fargate-otel.yaml

This manifest is all you need to collect cluster metrics and to receive application metrics and traces from gRPC sources. Point your instrumentation at the OTLP gRPC endpoint:

http://cx-otel-collector-service.cx-eks-fargate-otel.svc.cluster.local:4317

Monitor the Collector itself

Fargate networking prevents a pod from reaching its own host, so the Collector cannot scrape its own pod metrics. A second Collector fills that gap.

Set the environment variables listed at the top of the self-monitoring manifest, then apply it:

kubectl apply -f cx-eks-fargate-otel-self-monitoring.yaml

Collect logs with the log router

The Fargate Kubelet ships a built-in log router that forwards container logs to a supported destination.

Note

The log router supports ElasticSearch, Amazon Data Firehose, and CloudWatch. Use Amazon Data Firehose — it submits directly to Coralogix and adds the least cost.

Grant the Fargate profile permission

The sidecar needs put permissions on the delivery stream. Add this policy to every Fargate profile pod execution role whose logs you want.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": [
                "<firehose_ARN>"
            ]
        }
    ]
}

Deploy the log router configuration

The router reads its configuration from a specific namespace with specific labels. You will not run workloads there, so it needs no Fargate profile.

The configuration resembles a standard Fluent Bit configuration, but only certain sections and modules are honored. Apply this manifest, which creates the namespace and the ConfigMap:

---
kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  filters.conf: |
    [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio

    [FILTER]
        Name             kubernetes
        Match            kube.*
        Merge_Log           On
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s
        Keep_Log Off
        Merge_Log_Key log_obj
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Annotations Off

  output.conf: |
    [OUTPUT]
      Name  kinesis_firehose
      Match *
      region <AWS Region> (no quotes)
      delivery_stream <Data Firehose Delivery Stream Name> (no quotes)

  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep true

Warning

The log router attaches only to workloads started after the manifest is applied. Restart your pods, or nothing is forwarded.

Two further limits are worth knowing:

• Additional filters are allowed, but only of these types: grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, and kubernetes.
• Adding a CloudWatch output requires extra permissions on the Fargate profile. See the AWS Fargate logging documentation.

Related resources

Send logs using Amazon Data FirehoseOpenTelemetry on ECS FargateSend-Your-Data API key