Coralogix Resource Metadata (SQS mode)

This application collects AWS resource metadata and sends it to your Coralogix account.

This is a specific version of the resource-metadata application, which is designed to:

1. Handle huge amount of Lambda functions in the target AWS Region (5000+).
2. Support cross-account and multi-region collection of metadata from multiple AWS accounts.

Prerequisites

• Permissions to create required AWS resources (Lambda, SQS, SNS, Cloudtrail etc.)
• An AWS account.
• A Coralogix account.
• If you use AWS Secrets Manager, first deploy the SM lambda layer. Deploy only one layer per region.
• If you want to collect metadata from multiple AWS accounts, see Cross-Account Collection for more details.

Cross-Region Collection

To enable cross-region collection within an account, simply add the SourceRegions parameter. You don't need to do anything else.

Cross-Account Collection

This function supports cross-account collection of metadata from multiple AWS accounts.

To enable this feature, you need to specify the CrossAccountMode parameter (Disabled by default).

There are two options to collect metadata across accounts:

1. IAM Cross-account roles (StaticIAM): loops over account IDs (AccountIds), assuming IAM role in each account. IAM role should have the same name in each account (CrossAccountIAMRoleName).
2. AWS Config Resource Aggregator (Config): runs a Query on AWS Config Aggregator (ConfigAggregatorName). If the aggregator is in a different account, set ConfigCrossAccountRole to the IAM role ARN in that account; the collector will assume it when querying Config. It also requires IAM cross-account roles setup for the generator.

There are many other ways to collect the metadata about Lambda functions and EC2 instances from different accounts. See the cross-account solutions doc for more details. The rest of those options can be implemented on demand.

Cross-account permissions

Here is the set of required IAM permissions that should be set on the target roles:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
            "ec2:DescribeInstances",
            "lambda:ListFunctions",
            "lambda:ListVersionsByFunction", 
            "lambda:GetFunctionConfiguration",
            "lambda:GetFunctionConcurrency",
            "lambda:ListTags",
            "lambda:ListAliases",
            "lambda:ListEventSourceMappings",
            "lambda:GetPolicy",
            "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}

As you will know the exact functions' role ARNs only after the template is deployed, you need to follow these steps to make it work:

1. Create the roles in the target accounts, setting necessary IAM permissions, but without setting the trust relationship to the source account, since we don't know Lambda functions role ARNs yet.
2. Deploy the template with the CrossAccountIAMRoleName parameter, mentioning the target roles' name.
3. After the template is deployed, set the trust relationship to the source account, using the Lambda functions' role ARNs. If you're using a StaticIAM mode, make sure to include both collector and generator functions' role ARNs. If you're using a Config mode, you need to include only generator role ARN.

Here is an example of a trust relationship policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789012:role/mystackname-GeneratorLambdaFunctionRole-randomid",
                    "arn:aws:iam::123456789012:role/mystackname-CollectorLambdaFunctionRole-randomid"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

After setting the trust relationship, the generator and collector functions will be able to assume the target roles and collect metadata from those accounts.

Event Mode

Apart from running a central collector, you can enable the EventMode in order to get the near-real-time events from CloudTrail to the SQS queue by using EventBridge. As soon as the event appears in the queue, it gets processed by generator function.

Events from the function account/region

You can enable it in two formats:

• EnabledCreateTrail – creates CloudTrail Trail and EventBridge Rule
• EnabledWithExistingTrail – creates EventBridge Rule only

Events from other accounts/regions

You can route CloudTrail events from other accounts and regions to the function's SQS queue.

To make it work:

1. cross-account mode needs to be enabled
2. OrganizationId parameter has to be set
3. the function needs to have access to the source account by using IAM Assume Role
4. the Eventbridge rules have to use the following event pattern:

Lambda + EC2:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ec2",
    "aws.lambda"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com",
      "lambda.amazonaws.com"
    ],
    "eventName": [
      "RunInstances",
      "CreateFunction20150331"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

Lambda only:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.lambda"
  ],
  "detail": {
    "eventSource": [
      "lambda.amazonaws.com"
    ],
    "eventName": [
      "CreateFunction20150331"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

EC2 only:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ec2"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "RunInstances"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

Fields

Coralogix Configuration Parameters

Parameter	Description	Default Value	Required
CoralogixRegion	The Coralogix location region, possible options are [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom].In case that you want to use Custom domain, leave this as default and write the Custom doamin in the CustomDomain filed.	Custom
ApiKey	Your Coralogix Send Your Data – API Key, or, if you use a pre-created secret (created in AWS Secrets Manager), the name of the secret that contains the Coralogix Send Your Data API key.
CustomDomain	The Coralogix custom domain, leave empty if you don't use Custom domain.

Event Mode Parameters

Cross-account and cross-region parameters

Integration configuration

Lambda Configuration

Security configuration

Notes

1. Both layers and lambda need to be in the same AWS Region.
2. The Schedule parameter needs to be longer than the time it takes to collect the metadata. For example, if it takes 10 minutes to collect the metadata from all lambda functions, the Schedule parameter should be set to rate(15 minutes) at least.
3. The ResourceTtlMinutes parameter needs to be longer than the Schedule parameter. For example, if the Schedule parameter is set to rate(15 minutes), the ResourceTtlMinutes parameter should be set to at least 20 minutes.

License

This project is licensed under the Apache-2.0 License.