AWS resource metadata collection
Deploy the AWS Resource Metadata Collection AWS Lambda function in your AWS account. The function collects metadata of EC2 instances and AWS Lambda functions in the region of your AWS account and sends them to Coralogix.
What you'll find
This tutorial presents:
Configuration of the AWS Resource Metadata integration for standard users
For users with more than 5,000 Lambda functions or multi-account and multi-region collection, see AWS Resource Metadata Collection (High Volume).
Overview
AWS resources can be vast and interconnected. To better understand log data and troubleshoot issues, it's important to have context about which AWS resources are involved. The AWS Resource Metadata Collection integration collects information about AWS resources that are associated with log events. This contextual information can include details about the AWS service, resource tags, AWS region, timestamps for resource creation or modification, and any relevant custom tags specific to the organization's AWS environment.
Benefits
The collection of EC2 instance and Lambda function metadata serves as a foundation for better AWS resource management, optimization, security, and efficient troubleshooting. It helps you make informed decisions and take actions based on a more comprehensive understanding of your AWS resources and their interactions.
Improve troubleshooting. The metadata can provide valuable operational insights into your AWS environment. You can analyze resource-specific patterns, performance trends, and utilization to ensure your applications run smoothly. Having resource context helps in identifying the source of issues more quickly and accurately.
Resource optimization. By collecting metadata, you can understand the relationships between different AWS resources, such as how Lambda functions interact with specific EC2 instances or other services. This can be essential for monitoring and managing complex AWS architectures, and making informed decisions about resource usage and optimization.
Security and compliance. Resource metadata can assist in security monitoring, compliance reporting, and auditing.
Cost management. Understanding resource attributes can be valuable for cost management and allocation.
Prerequisites
AWS account
Permissions to create Lambda functions
If you are using Secret Manager you should first deploy the SM Lambda layer. Note that you should only deploy one layer per region.
Standard configuration
STEP 1. Access Data Flow > Integrations. View the list of available integrations.
STEP 2. Select AWS Resource Metadata.
STEP 3. Click + ADD NEW.
STEP 4. Input the integration details.
Input a name for your integration.
Select the authentication type, either APIKey or Existing Secret.
If using an API key, input an existing Coralogix Send-Your-Data API Key or click CREATE NEW KEY.
If using an existing secret, enter the AWS Secret Name.
Mark the Collect Aliases checkbox if you want to collect the aliases of the resources.
Select your AWS Region from the dropdown list.
If you want to use AWS PrivateLink, click Advanced Settings and mark the Use AWS PrivateLink checkbox. AWS PrivateLink is a service that facilitates secure and private connections between VPCs and AWS services, bypassing the need for the public internet. It is worth noting that the integration might not succeed if AWS PrivateLink is not properly set up.
STEP 5. Click NEXT.
STEP 6. View the instructions for your integration, then click CREATE CLOUDFORMATION.
STEP 7. You will be rerouted to the AWS website. Verify that all of the auto pre-populated values are correct, then click the acknowledgement checkboxes, and click Create Stack.
STEP 8. Go back to the Coralogix application and click COMPLETE to ensure your deployment is successful. This triggers a test to verify the deployment, the result of which can be seen on the next page as either Failed or Connected.
STEP 9. View your integration information.
STEP 10. Upon successful deployment, leverage the Coralogix APM Serverless Monitoring feature to access detailed insights into the Lambda functions operating within the deployed region.
Parameters and descriptions
| Parameter | Description | Default Value | Required |
|---|---|---|---|
| CoralogixRegion | The Coralogix location region, possible options are [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom].In case that you want to use Custom domain, leave this as default and write the Custom doamin in the CustomDomain filed. | Custom | ✔️ |
| CustomDomain | The Coralogix custom domain, leave empty if you don't use Custom domain. | ||
| AplicationName | The stack name of this application created via AWS CloudFormation. | ✔️ | |
| CreateSecret | Set to False In case you want to use secrets manager with a predefine secret that was already created and contains Coralogix Send Your Data API key. | True | |
| ApiKey | Your Coralogix Send Your Data – API Key. If using a pre-created secret from AWS secret manager, input the name of the secret that contains the Coralogix Send-Your-Data key. | ✔️ | |
| ResourceTtlMinutes | Once a resource is collected, how long should it remain valid. See "Notes" for more details. | 60 | |
| LatestVersionsPerFunction | How many latest published versions of each Lambda function should be collected. | 0 | |
| CollectAliases | [True/False] | False | |
| LambdaFunctionIncludeRegexFilter | If specified, only lambda functions with ARNs matching the regex will be included in the collected metadata | ||
| LambdaFunctionExcludeRegexFilter | If specified, only lambda functions with ARNs NOT matching the regex will be included in the collected metadata | ||
| LambdaFunctionTagFilters | If specified, only lambda functions with tags matching the filters will be included in the collected metadata. Values should follow the JSON syntax for --tag-filters as documented here | ||
| ExcludedEC2ResourceType | Set to true to Excluded EC2 Resource Type | False | |
| ExcludedLambdaResourceType | Set to true to Excluded Resource Type | False | |
| Schedule | Collect metadata on a specific schedule. See "Notes" for more details. | rate(30 minutes) | |
| LayerARN | In case you want to use Secret Manager This is the ARN of the Coralogix lambda layer. See "Notes" for more details. | ||
| NotificationEmail | If the lambda fails a notification email will be sent to this address via SNS (requires you have a working SNS, with a validated domain). | ||
| FunctionArchitecture | Lambda function architecture, possible options are [x86_64, arm64]. | x86_64 | |
| FunctionMemorySize | The maximum allocated memory this lambda may consume. Default value is the minimum recommended setting please consult coralogix support before changing. | 256 | |
| FunctionTimeout | The maximum time in seconds the function may be allowed to run. Default value is the minimum recommended setting please consult coralogix support before changing. | 300 |
For high-volume mode parameters (Event Mode, cross-account, cross-region, MaximumConcurrency, EC2ChunkSize), see AWS Resource Metadata Collection (High Volume).
Support
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].