Amazon Web Services (AWS) Resource Enrichment

Coralogix now offers AWS Resource Enrichment, allowing you to enrich your logs with tags from Amazon Web Services (AWS) EC2 instances. Use this feature to connect your business and operation metadata from AWS and gain greater insight into your data.

Overview

The feature enriches every log that contains a particular ec2_instance_id, associated with the metadata of a chosen AWS EC2 instance.

Example:

Sample log before enrichment:

{
	"kubernetes": {
		"pod_name": "service-0",
		"namespace_name": "default",
		"pod_id": "123456-aefb-456e-b315-77b014de9d5b",
		"labels": {
			"app.kubernetes.io\\/managed-by": "Helm",
			"statefulset.kubernetes.io\\/pod-name": "service-0"
		},
		"host": "ip-1-2-3-4.eu-west-1.compute.internal",
		"container_name": "service-0",
		"docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2",
	},
	"hostname": "ip-1-2-3-4.eu-west-1.compute.internal",
	"account_id": "1234567",
	"stream": "stdout",
	"log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).",
	"ec2_instance_id": "i-050d53c29af57c74a",
	"ec2_instance_type": "r5dn.2xlarge",
	"az": "eu-west-1c",
	"time": "2022-11-15T12:55:36.768697532Z"
}

AWS enrichment:

Sample log after enrichment:

{
	"kubernetes": {
		"pod_name": "service-0",
		"namespace_name": "default",
		"pod_id": "123456-aefb-456e-b315-77b014de9d5b",
		"labels": {
			"app.kubernetes.io\\/managed-by": "Helm",
			"statefulset.kubernetes.io\\/pod-name": "service-0"
		},
		"host": "ip-1-2-3-4.eu-west-1.compute.internal",
		"container_name": "service-0",
		"docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2",
	},
	"hostname": "ip-1-2-3-4.eu-west-1.compute.internal",
	"account_id": "1234567",
	"stream": "stdout",
	"log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).",
	"ec2_instance_id": "i-050d53c29af57c74a",
	"ec2_instance_type": "r5dn.2xlarge",
	"az": "eu-west-1c",
	"time": "2022-11-15T12:55:36.768697532Z",
  "tag_a": "a", 
  "tag_b": "b"
}

Prerequisites

1. Sign up for a Coralogix account. Set up your account on the Coralogix domain corresponding to the region within which you would like your data stored.
2. Access your Coralogix private key.

Setup

Installation & Configuration

1. Make sure your logs contain a key with an EC2 instance ID value:
   • Fluent Bit: ec2_instance_id
   • FluentD: instance_id (default key name) using Fluent-plugin-ec2-metadata plugin
   • Filebeat: cloud.instance.id

2. We strongly recommend installing and configuring Fluent Bit as a best practice. You can also install and configure FluentD or Filebeat. The example below provides the configuration for Fluent Bit.
3. Add the AWS filter with the desired additional labels. FluentD metadata labels and Filebeat metadata labels are configured differently.

[FILTER]
        Name aws
        Match *
        ec2_instance_type true
        account_id true
        hostname true

Note: ec2_instance_id and az are configured by default.

4. As part of the nest operation defining root fields, specify the desired AWS metadata labels under the nest filter.

[FILTER]
        Name        nest
        Match       kube.*
        Operation   nest
        Wildcard    kubernetes
        Wildcard    account_id
        Wildcard    hostname
        Wildcard    az
        Wildcard    ec2_instance_type
        Wildcard    ec2_instance_id
        Wildcard    log
        Wildcard    log_obj
        Wildcard    stream
        Wildcard    time
        Nest_under  json

AWS Lambda Deployment

Deploy the AWS lambda function running in your AWS account. The platform collects data from a chosen EC2 instance in your AWS account and sends them to Coralogix, where they are enriched with tags associated with your instance.

1. Fill in the Application settings fields.
   • Input the Coralogix domain within which your account has been created in CoralogixRegion.
   • Input your Coralogix private key in PrivateKey.
   • Do not change the default settings for the FunctionMemorySize and FunctionTimeout.

2. Deploy your application to collect tabs. It will collect tags every 10 minutes and send them to Coralogix.

AWS Enrichment on Your Coralogix Dashboard

Access AWS Enrichment on your Coralogix dashboard

1. In your Coralogix dashboard, click on the Data Flow tab > Data Enrichment. Scroll down the page to view the AWS Enrichment option.

2. Select the chosen AWS resource type.
3. Select the key with your EC2 instance ID.
4. Add the enrichment.
5. Logs associated with the chosen AWS resource should now appear enriched with AWS tags in your dashboard. To view them, click on the Explore tab > Logs.

Limitations

AWS Resource Enrichment feature grants the following:

• Active resources (existing in the last 24h) / Coralogix team: 100,000
• 50 tags / resource (equal to AWS limits)
• Tag key length <= 127 (equal to AWS limits)
• Tag value length <= 255 (equal to AWS limits)

Additional Resources

Github: Lambda documentation for Coralogix AWS Enrichment

AWS: Coralogix-Resource-Tabs

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].