Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Docs

AWS Resource Enrichment AWS Resource Enrichment

Last Updated: Sep. 06, 2023

Coralogix now offers AWS Resource Enrichment, allowing you to enrich your logs with tags from Amazon Web Services (AWS) EC2 instances. Use this feature to connect your business and operation metadata from AWS and gain greater insight into your data.

Overview

The feature enriches every log that contains a particular ec2_instance_id, associated with the metadata of a chosen AWS EC2 instance.

Example:

Sample log before enrichment:

{
	"kubernetes": {
		"pod_name": "service-0",
		"namespace_name": "default",
		"pod_id": "123456-aefb-456e-b315-77b014de9d5b",
		"labels": {
			"app.kubernetes.io\\/managed-by": "Helm",
			"statefulset.kubernetes.io\\/pod-name": "service-0"
		},
		"host": "ip-1-2-3-4.eu-west-1.compute.internal",
		"container_name": "service-0",
		"docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2",
	},
	"hostname": "ip-1-2-3-4.eu-west-1.compute.internal",
	"account_id": "1234567",
	"stream": "stdout",
	"log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).",
	"ec2_instance_id": "i-050d53c29af57c74a",
	"ec2_instance_type": "r5dn.2xlarge",
	"az": "eu-west-1c",
	"time": "2022-11-15T12:55:36.768697532Z"
}

AWS enrichment:

aws resource enrichment

Sample log after enrichment:

{
	"kubernetes": {
		"pod_name": "service-0",
		"namespace_name": "default",
		"pod_id": "123456-aefb-456e-b315-77b014de9d5b",
		"labels": {
			"app.kubernetes.io\\/managed-by": "Helm",
			"statefulset.kubernetes.io\\/pod-name": "service-0"
		},
		"host": "ip-1-2-3-4.eu-west-1.compute.internal",
		"container_name": "service-0",
		"docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2",
	},
	"hostname": "ip-1-2-3-4.eu-west-1.compute.internal",
	"account_id": "1234567",
	"stream": "stdout",
	"log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).",
	"ec2_instance_id": "i-050d53c29af57c74a",
	"ec2_instance_type": "r5dn.2xlarge",
	"az": "eu-west-1c",
	"time": "2022-11-15T12:55:36.768697532Z",
  "tag_a": "a", 
  "tag_b": "b"
}

Prerequisites

  1. Sign up for a Coralogix account. Set up your account on the Coralogix domain corresponding to the region within which you would like your data stored.
  2. Access your Coralogix Send Your Data – API Key.

Setup

Installation & Configuration

STEP 1. Make sure your logs contain a key with an EC2 instance ID value:

  • Fluent Bit: ec2_instance_id
  • FluentD: instance_id (default key name) using Fluent-plugin-ec2-metadata plugin

STEP 2. We strongly recommend installing and configuring Fluent Bit as a best practice. You can also install and configure FluentD. The example below provides the configuration for Fluent Bit.

    STEP 3. Add the AWS filter with the desired additional labels. FluentD metadata labels are configured differently.

    [FILTER]
            Name aws
            Match *
            ec2_instance_type true
            account_id true
            hostname true
    

    Note: ec2_instance_id and az are configured by default.

    1. As part of the nest operation defining root fields, specify the desired AWS metadata labels under the nest filter.
    [FILTER]
            Name        nest
            Match       kube.*
            Operation   nest
            Wildcard    kubernetes
            Wildcard    account_id
            Wildcard    hostname
            Wildcard    az
            Wildcard    ec2_instance_type
            Wildcard    ec2_instance_id
            Wildcard    log
            Wildcard    log_obj
            Wildcard    stream
            Wildcard    time
            Nest_under  json
    

    AWS Lambda Deployment

    Deploy the AWS lambda function running in your AWS account. The platform collects data from a chosen EC2 instance in your AWS account and sends them to Coralogix, where they are enriched with tags associated with your instance.

    1. Fill in the Application settings fields.
      • Input the Coralogix domain within which your account has been created in CoralogixRegion.
      • Input your Coralogix Send Your Data – API Key in PrivateKey.
      • Do not change the default settings for the FunctionMemorySize and FunctionTimeout.
    aws application settings
    1. Deploy your application to collect tabs. It will collect tags every 10 minutes and send them to Coralogix.

    AWS Enrichment on Your Coralogix Dashboard

    Access AWS Enrichment on your Coralogix dashboard

    1. In your Coralogix dashboard, click on the Data Flow tab > Data Enrichment. Scroll down the page to view the AWS Enrichment option.
    aws enrichment in coralogix
    1. Select the chosen AWS resource type.
    2. Select the key with your EC2 instance ID.
    3. Add the enrichment.
    4. Logs associated with the chosen AWS resource should now appear enriched with AWS tags in your dashboard. To view them, click on the Explore tab > Logs.

    Limitations

    AWS Resource Enrichment feature grants the following:

    • Active resources (existing in the last 24h) / Coralogix team: 100,000
    • 50 tags / resource (equal to AWS limits)
    • Tag key length <= 127 (equal to AWS limits)
    • Tag value length <= 255 (equal to AWS limits)

    Additional Resources

    Github: Lambda documentation for Coralogix AWS Enrichment

    AWS: Coralogix-Resource-Tabs

    Support

    Need help?

    Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

    Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].

    On this page