AWS Resource Metadata Collection

Deploy the AWS Resource Metadata Collection AWS Lambda function in your AWS account. The function collects metadata of EC2 instances and AWS Lambda functions in the region of your AWS account and sends them to Coralogix.

What you'll find

This tutorial presents:

Configuration of the AWS Resource Metadata integration for standard users
Configuration of the AWS Resource Metadata (High Volume) integration for users with more than 5,000 Lambda functions

Overview

AWS resources can be vast and interconnected. To better understand log data and troubleshoot issues, it's important to have context about which AWS resources are involved. The AWS Resource Metadata Collection integration collects information about AWS resources that are associated with log events. This contextual information can include details about the AWS service, resource tags, AWS region, timestamps for resource creation or modification, and any relevant custom tags specific to the organization's AWS environment.

Benefits

The collection of EC2 instance and Lambda function metadata serves as a foundation for better AWS resource management, optimization, security, and efficient troubleshooting. It helps you make informed decisions and take actions based on a more comprehensive understanding of your AWS resources and their interactions.

Improve troubleshooting. The metadata can provide valuable operational insights into your AWS environment. You can analyze resource-specific patterns, performance trends, and utilization to ensure your applications run smoothly. Having resource context helps in identifying the source of issues more quickly and accurately.
Resource optimization. By collecting metadata, you can understand the relationships between different AWS resources, such as how Lambda functions interact with specific EC2 instances or other services. This can be essential for monitoring and managing complex AWS architectures, and making informed decisions about resource usage and optimization.
Security and compliance. Resource metadata can assist in security monitoring, compliance reporting, and auditing.
Cost management. Understanding resource attributes can be valuable for cost management and allocation.

Prerequisites

AWS account
Permissions to create Lambda functions
If you are using Secret Manager you should first deploy the SM Lambda layer. Note that you should only deploy one layer per region.

Standard configuration

STEP 1. In your navigation pane, click Data Flow > Integrations. View the list of available integrations.

STEP 2. Select AWS Resource Metadata.

STEP 3. Click + ADD NEW.

STEP 4. Input the integration details.

Input a name for your integration.
Select the authentication type, either APIKey or Existing Secret.
- If using an API key, input an existing Coralogix Send-Your-Data API Key or click CREATE NEW KEY.
- If using an existing secret, enter the AWS Secret Name.
Mark the Collect Aliases checkbox if you want to collect the aliases of the resources.
Select your AWS Region from the dropdown list.
If you want to use AWS PrivateLink, click Advanced Settings and mark the Use AWS PrivateLink checkbox. AWS PrivateLink is a service that facilitates secure and private connections between VPCs and AWS services, bypassing the need for the public internet. It is worth noting that the integration might not succeed if AWS PrivateLink is not properly set up.

STEP 5. Click NEXT.

STEP 6. View the instructions for your integration, then click CREATE CLOUDFORMATION.

STEP 7. You will be rerouted to the AWS website. Verify that all of the auto pre-populated values are correct, then click the acknowledgement checkboxes, and click Create Stack.

STEP 8. Go back to the Coralogix application and click COMPLETE to ensure your deployment is successful. This triggers a test to verify the deployment, the result of which can be seen on the next page as either Failed or Connected.

STEP 9. View your integration information.

STEP 10. Upon successful deployment, leverage the Coralogix APM Serverless Monitoring feature to access detailed insights into the Lambda functions operating within the deployed region.

High Volume configuration

For scenarios where you have more than 5,000 Lambda functions, you need to use the AWS Resource Metadata (High Volume) integration. It's a more advanced version of the Resource Metadata integration, designed to handle larger volumes of metadata from 5,000 up to 100,000 Lambda functions in the target AWS region.

For deployment, follow the same steps, but select the AWS Resource Metadata (High Volume) integration in STEP 2.

Event Mode

The High Volume integration supports the Event Mode feature. Event Mode allows you to create Lambda and EC2 resources in Coralogix on a near-real-time basis, starting metadata collection as soon as a new function or instance is created. It usually takes 3-5 seconds for the resource to appear in Coralogix after being created in AWS.

This feature is disabled by default. Enable it in the Integration Details section.

Event Mode

All options for Event Mode:

Disabled
EnabledWithExistingTrail – Skips the creation of a CloudTrail trail and S3 bucket. This option is used if there is already one trail running in the target AWS region.
EnabledCreateTrail – Creates all resources, including CloudTrail trail and S3 bucket.

Parameters and descriptions

Parameter	Description	Default Value	Required
CoralogixRegion	The Coralogix location region, possible options are [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom].In case that you want to use Custom domain, leave this as default and write the Custom doamin in the `CustomDomain` filed.	`Custom`	✔️
CustomDomain	The Coralogix custom domain, leave empty if you don't use Custom domain.
AplicationName	The stack name of this application created via AWS CloudFormation.		✔️
CreateSecret	Set to False In case you want to use secrets manager with a predefine secret that was already created and contains Coralogix Send Your Data API key.	`True`
ApiKey	Your Coralogix Send Your Data – API Key. If using a pre-created secret from AWS secret manager, input the name of the secret that contains the Coralogix Send-Your-Data key.		✔️
ResourceTtlMinutes	Once a resource is collected, how long should it remain valid. See "Notes" for more details.	`60`
LatestVersionsPerFunction	How many latest published versions of each Lambda function should be collected.	`0`
CollectAliases	[True/False]	`False`
LambdaFunctionIncludeRegexFilter	If specified, only lambda functions with ARNs matching the regex will be included in the collected metadata
LambdaFunctionExcludeRegexFilter	If specified, only lambda functions with ARNs NOT matching the regex will be included in the collected metadata
LambdaFunctionTagFilters	If specified, only lambda functions with tags matching the filters will be included in the collected metadata. Values should follow the JSON syntax for --tag-filters as documented here
ExcludedEC2ResourceType	Set to true to Excluded EC2 Resource Type	`False`
ExcludedLambdaResourceType	Set to true to Excluded Resource Type	`False`
Schedule	Collect metadata on a specific schedule. See "Notes" for more details.	`rate(30 minutes)`
LayerARN	In case you want to use Secret Manager This is the ARN of the Coralogix lambda layer. See "Notes" for more details.
NotificationEmail	If the lambda fails a notification email will be sent to this address via SNS (requires you have a working SNS, with a validated domain).
FunctionArchitecture	Lambda function architecture, possible options are [x86_64, arm64].	`x86_64`
FunctionMemorySize	The maximum allocated memory this lambda may consume. Default value is the minimum recommended setting please consult coralogix support before changing.	`256`
FunctionTimeout	The maximum time in seconds the function may be allowed to run. Default value is the minimum recommended setting please consult coralogix support before changing.	`300`

Additional parameters for high-volume mode

Parameter	Description	Default Value	Required
EventMode	Additionally to the regular schedule, enable real-time processing of CloudTrail events via EventBridge for immediate generation of new resources in Coralogix [Disabled, EnabledWithExistingTrail, EnabledCreateTrail].	`Disabled`
MaximumConcurrency	Maximum number of concurrent SQS messages to be processed by `generator` lambda after the collection has finished.	`5`

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

AWS VPC Flow Logs Terraform Module

Next AWS Resource Metadata Collection Terraform Module