Coralogix now offers AWS Resource Enrichment, allowing you to enrich your logs with tags from Amazon Web Services (AWS) EC2 instances. Use this feature to connect your business and operation metadata from AWS and gain greater insight into your data.
The feature enriches every log that contains a particular ec2_instance_id
, associated with the metadata of a chosen AWS EC2 instance.
Example:
Sample log before enrichment:
{ "kubernetes": { "pod_name": "service-0", "namespace_name": "default", "pod_id": "123456-aefb-456e-b315-77b014de9d5b", "labels": { "app.kubernetes.io\\/managed-by": "Helm", "statefulset.kubernetes.io\\/pod-name": "service-0" }, "host": "ip-1-2-3-4.eu-west-1.compute.internal", "container_name": "service-0", "docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2", }, "hostname": "ip-1-2-3-4.eu-west-1.compute.internal", "account_id": "1234567", "stream": "stdout", "log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).", "ec2_instance_id": "i-050d53c29af57c74a", "ec2_instance_type": "r5dn.2xlarge", "az": "eu-west-1c", "time": "2022-11-15T12:55:36.768697532Z" }
AWS enrichment:
Sample log after enrichment:
{ "kubernetes": { "pod_name": "service-0", "namespace_name": "default", "pod_id": "123456-aefb-456e-b315-77b014de9d5b", "labels": { "app.kubernetes.io\\/managed-by": "Helm", "statefulset.kubernetes.io\\/pod-name": "service-0" }, "host": "ip-1-2-3-4.eu-west-1.compute.internal", "container_name": "service-0", "docker_id": "12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2", }, "hostname": "ip-1-2-3-4.eu-west-1.compute.internal", "account_id": "1234567", "stream": "stdout", "log": "[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms).", "ec2_instance_id": "i-050d53c29af57c74a", "ec2_instance_type": "r5dn.2xlarge", "az": "eu-west-1c", "time": "2022-11-15T12:55:36.768697532Z", "tag_a": "a", "tag_b": "b" }
STEP 1. Make sure your logs contain a key with an EC2 instance ID value:
ec2_instance_id
instance_id
(default key name) using Fluent-plugin-ec2-metadata pluginSTEP 2. We strongly recommend installing and configuring Fluent Bit as a best practice. You can also install and configure FluentD. The example below provides the configuration for Fluent Bit.
STEP 3. Add the AWS filter with the desired additional labels. FluentD metadata labels are configured differently.
[FILTER] Name aws Match * ec2_instance_type true account_id true hostname true
Note: ec2_instance_id
and az
are configured by default.
[FILTER] Name nest Match kube.* Operation nest Wildcard kubernetes Wildcard account_id Wildcard hostname Wildcard az Wildcard ec2_instance_type Wildcard ec2_instance_id Wildcard log Wildcard log_obj Wildcard stream Wildcard time Nest_under json
Deploy the AWS lambda function running in your AWS account. The platform collects data from a chosen EC2 instance in your AWS account and sends them to Coralogix, where they are enriched with tags associated with your instance.
Access AWS Enrichment on your Coralogix dashboard
AWS Resource Enrichment feature grants the following:
Github: Lambda documentation for Coralogix AWS Enrichment
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].