Skip to content

AWS CloudWatch Metrics

Overview

The Coralogix AWS Metrics integration offers a simple and easy way to ingest AWS metrics into Coralogix. It’s a cost-efficient alternative to Amazon Firehose. In addition, it accepts S3 metrics with 24-hour frequency that can’t be collected via Firehose.

The metrics data is collected by Amazon CloudWatch across all your AWS resources. Integrate AWS metrics to retrieve and ingest the data into Coralogix using CloudWatch API.

Poll intervals

  • By default, metrics are pulled every 10 minutes. To increase savings, you can configure a higher number.

  • Metric definitions are polled every 5 minutes.

  • Tag poll interval is 5 minutes for both resource group tagging and specialized APIs.

Prerequisites

STEP 1. Make sure you have STS endpoint enabled for the region where the integration is deployed. To check that it is enabled or to enable it, follow the instructions in the AWS documentation.

STEP 2. Create IAM policy and with CloudFormation template.

STEP 3. Pick a CoralogixRegion based on the target Coralogix region (for example: US1, US2, etc. View updated list of regions on the Coralogix Domain page). Pay attention that the ExternalId value passed to --parameter-overrides must be in the Key=Value format.

aws cloudformation deploy --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
   --template-file template.yaml \
   --stack-name coralogix-cloudwatch-metrics-integration \
   --parameter-overrides CoralogixRegion=<coralogix_account_region> RoleName=coralogix-cloudwatch-metrics-integration ExternalId=my_super-secret-Id-1234

STEP 4. Run it in:

  • A terminal with installed AWS CLI and configured credentials, or:

  • AWS CloudShell.

STEP 5. Get an ARN of the role for the integration.

aws cloudformation describe-stacks --stack-name coralogix-cloudwatch-metrics-integration --query 'Stacks[0].Outputs[?OutputKey==`CoralogixAwsMetricsRoleArn`].OutputValue' --output text

AWS metrics integration deployment

Basic

STEP 1. From your Coralogix toolbar, go to Data Flow > Integrations.

STEP 2. From the Integrations section, select AWS Metrics.

STEP 3. Enter configuration parameters according to your application requirements (some of the fields are optional):

  • Integration name - A meaningful name assigned to the integration for easier identification of this integration instance.

  • AWS Role ARN - The role ARN created as described in the Prerequisites section.

  • AWS Region - Metrics collection region. To collect data from multiple regions, create multiple integrations.

  • ExternalId - An external ID allows the user that is assuming the role to assert the circumstances in which they are operating. It also provides a way for the account owner to permit the role to be assumed only under specific circumstances. The primary function of the external ID is to address and prevent the confused deputy problem.

Note

AWS does not treat the external ID as a secret. After you create a secret like an access key pair or a password in AWS, you cannot view them again. The external ID for a role can be seen by anyone with permission to view the role.

  • Select CloudWatch Metric Namespaces - Predefined list of all known AWS metric namespaces for metrics collection. If left blank, no namespaces wll be chosen.

  • Metric Selectors allows defining a list of metric selectors in JSON format.

    • This configuration helps limit metrics collected per namespace.
    • If you define a namespace in both "Select CloudWatch Metric Namespaces" and here, only the metric_names specified here will be collected.
    • We do not support duplicate metric selector configurations per namespace.
    • Example configuration: [{"namespace": "AWS/EC2", "metric_names": ["CPUUtilization", "NetworkIn"]}]
    • This option was added in 0.6.1.

  • If With Aggregations is selected, the _min, _max and _avg metrics will be collected too, if available. Note that you can get the average from the metric with a PromQL query.

  • Select Enrich With Tags to include resource tags of resource-related metrics. Note that some metrics are related to specific resources, while others are not. For example, the AWS/RDS metric CPUUtilization can be ingested with a set of multiple dimensions. The metric will be enriched with tags for DBInstanceIdentifier, but not for DatabaseClass or EngineName. If the name of the tag conflicts with the dimension name, it will get a cloud_tag prefix. In case of the CPUUtilization metric, DBInstanceIdentifier would be renamed to cloud_tag_DBInstanceIdentifier, but my_custom_tag would remain unchanged.

aws_metrics_cloudwatch.png

Advanced

  • Application Name and Subsystem Name fields define cx_application_name and cx_subsystem_name labels to be added to all imported metrics.

  • Resource Tags Filter allows to filter the resources on which to collect metrics. Note that some metrics are not related to specific resources (see Enrich With Tags above). This filter has no effect on these metrics. The filter is defined by a comma-separated list of key-value pairs. For example, team,eks:cluster-name=staging will select all resources that either have the tag team set with any value or have tag eks:cluster-name=staging.

Note

The AWS API does not return tags for AWS/AutoScaling, AWS/DMS, AWS/EC2Spot, AWS/Prometheus and AWS/StorageGateway. As a result, the resources from these namespaces are excluded from filtering.

  • Tags Key Filter allows selection of the tags to be added as labels to the metrics. If empty, all tags will be added, prefixed with cloud_tag.

  • Static Labels allows entering a list of key-value pairs to be added as labels to all metrics in the integration.

  • Poll Interval In Minutes defines how often the metrics are collected. Shorter intervals mean lower latency and higher costs, and vice versa.

  • Additional Statistics Configuration allows defining additional metric statistics in JSON format. For example, [{"metric_selector":{"namespace":"AWS/RDS","metric_names":["CPUUtilization"]},"extended_statistics":["p99"]}] .

    • Available percentile statistics include p99, p95, p90, p80, and p70. These statistics are currently based on the percentiles supported by AWS CloudWatch Metrics Streams.

Enhanced monitoring

STEP 4. Click Save to create the integration.

After the integration is created, metrics will start appearing in several minutes. The integration checks the list of available resources with a period of 10 minutes. As a result, it may take up to 10 minutes and above between the creation of a resource and the appearance of the metric. Resources that are created and removed between two scans may never show in the metrics.

Troubleshooting

If the provided account ARN cannot be assumed by our service, an error message will be displayed.

Error Solutions
Wrong IAM role and or external id combination provided - Check that your AWS role ARN is correct.
- Make sure that you are using the correct ExternalId in the configuration.
- Check your role trust relationship. Make sure that CoralogixRegion and AwsRegion match.
- Make sure that you have STS enabled, as specified in the AWS Guide.
MalformedPolicyDocumentException This should not happen if the policy is generated by cloudformation. If it is and you still get this error, please contact Coralogix support.
PackedPolicyTooLargeException Cloudformation generated policy should be small enough. If you still get this error, please contact Coralogix support.
RegionDisabledException Make sure that you have STS enabled, as specified in the AWS Guide.
ExpiredTokenException The security token included in the request is expired.

Additional resources
Documentation Get Started with Coralogix
AWS CloudWatch Metric Streams with Amazon Data Firehose
External links What is CloudWatch
View available metrics

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

Previous Previous AWS CloudWatch: Data Collection Options
Next ECS Enhanced Monitoring for CloudWatch Metrics Next
Home
User Guides User Guides
DataPrime DataPrime
Integrations Integrations
OpenTelemetry OpenTelemetry
Developer Portal Developer Portal