Coralogix now offers AWS Resource Enrichment, allowing you to enrich your logs with tags from Amazon Web Services (AWS) EC2 instances. Use this feature to connect your business and operation metadata from AWS and gain greater insight into your data.
The feature enriches every log containing the following fields with metadata.
ec2_instance_id | OpenTelemetry |
instance_id | Fluentd |
ec2_instance_id | Fluent Bit |
Sample Otel Log Before Enrichment
{ resource:{ attributes:{ cloud.availability_zone:us-east-1b cloud.platform:aws_ec2 cloud.provider:aws cloud.region:us-east-1 ec2_instance_id:i-0c352ce4c433c58a5 host.image.id:ami-051f7e7f6c2f40dc1 host.name:ip-172-31-37-204.ec2.internal host.type:t2.micro } } scope:{ attributes:{} } resourceSchemaUrl:<https://opentelemetry.io/schemas/1.6.1> logRecord:{ attributes:{ log.file.name:log2.log log.file.path:log2.log } body:{ account_id:035955823196 application_name:my application test ec2_instance_type:t2.micro ec2_region:us-east-1 hosename:ip-172-31-37-204.ec2.internal image_id:ami-051f7e7f6c2f40dc1 kubernetes:{ container_name:service-0 docker_id:12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2 labels:{ app.kubernetes.io\\/managed-by:Helm statefulset.kubernetes.io\\/pod-name:service-0 } namespace_name:default pod_id:123456-aefb-456e-b315-77b014de9d5b pod_name:service-0 } log:[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms). platform:aws_ec2 stream:stdout time:2022-11-15T12:55:36.768697532Z } observedTimeUnixNano:1694069570457198600 } }
AWS Enrichment
Sample Otel Log After Enrichment
{ resource:{ attributes:{ ec2_instance_id_aws:{ ##enrichment part companyId:32757 resourceId:i-0c352ce4c433c58a5 resourceType:aws:ec2:instance tags:{ tag_a:a tag_b:b } } cloud.availability_zone:us-east-1b cloud.platform:aws_ec2 cloud.provider:aws cloud.region:us-east-1 ec2_instance_id:i-01s32s14p423a98a5 host.image.id:ami-051f7e7f6c2f40dc1 host.name:ip-172-62-93-414.ec2.internal host.type:t2.micro } } scope:{ attributes:{} } resourceSchemaUrl:<https://opentelemetry.io/schemas/1.6.1> logRecord:{ attributes:{ log.file.name:log.log log.file.path:log.log } body:{ kubernetes:{ container_name:service-0 docker_id:12345186e060681175942786de75a102c679f13c3f0fb330231bdd94c5570e2 labels:{ app.kubernetes.io\\/managed-by:Helm statefulset.kubernetes.io\\/pod-name:service-0 } namespace_name:default pod_id:123456-aefb-456e-b315-77b014de9d5b pod_name:service-0 } log:[INFO] 2022-11-15T12:55:36,768 kafka.streams.processor.internals.StateDirectory service-0 stream-thread [categorization-0-CleanupThread] Deleting obsolete state directory 0_2 for task 0_2 as 32827574178ms has elapsed (cleanup delay is 600000ms). platform:aws_ec2 stream:stdout time:2022-11-15T12:55:36.768697532Z } observedTimeUnixNano:1694069570457198600 } }
We strongly recommend installing and configuring with OpenTelemetry. The default example below uses the configuration for OpenTelemetry.
STEP 1. Ensure your logs contain a key with EC2 instance ID value:
ec2_instance_id | OpenTelemetry |
instance_id (default key name) using Fluent-plugin-ec2-metadata plugin | FluentD |
ec2_instance_id | Fluent Bit |
STEP 2. Add the resourcedetection and transform processors, enable the desired attributes, and add it to your config file. FluentD metadata labels and Fluent Bit metadata labels are configured differently.
You must enable host.id
for the enrichment to work.
processors: resourcedetection: detectors: [ec2] override: true ec2: resource_attributes: host.id: enabled: true host.name: enabled: true host.image.id: enabled: true host.type: enabled: true cloud.region: enabled: true cloud.account.id: enabled: true cloud.availability_zone: enabled: true transform: error_mode: ignore log_statements: - context: log statements: - set(resource.attributes["ec2_instance_id"],resource.attributes["host.id"]) - delete_key(resource.attributes, "host.id")
STEP 1. Add the AWS filter with the desired additional labels.
[FILTER] Name aws Match * ec2_instance_type true account_id true hostname true
STEP 2. Specify the desired AWS metadata labels under the nested filter as part of the nest operation defining root fields.
[FILTER] Name nest Match kube.* Operation nest Wildcard kubernetes Wildcard account_id Wildcard hostname Wildcard az Wildcard ec2_instance_type Wildcard ec2_instance_id Wildcard log Wildcard log_obj Wildcard stream Wildcard time Nest_under json
STEP 1. Deploy the AWS lambda function running in your AWS account. The platform collects logs from a chosen EC2 instance in your AWS account and sends them to Coralogix, enriching them with tags associated with your instance.
STEP 2. Fill in the Application settings fields.
STEP 3. Deploy your application to collect tags. It will collect tags every 10 minutes and send them to Coralogix.
STEP 1. From your Coralogix toolbar, navigate to Data Flow > Data Enrichment. Scroll down the page to view the AWS Enrichment Option.
STEP 2. Select the chosen AWS resource type.
STEP 3. Select the key with your EC2 instance ID.
resource.attributes.ec2_instance_id | OpenTelemetry |
ec2_instance_id | Fluent Bit |
instance_id | Fluentd |
Note: If you have modified the default option, you may have a different key.
STEP 4. Add the enrichment.
STEP 5. Logs associated with the chosen AWS resource should now appear enriched with AWS tags in your dashboard. To view them, navigate to Explore > Logs or to LiveTail.
AWS Resource Enrichment feature grants the following:
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by emailing [email protected].