Collect CloudWatch Metrics with Telegraf
Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and applications you run on AWS in real time. Use the Coralogix destination to easily forward metrics for your AWS resources to Coralogix using Telegraf.
This guide demonstrates how to:
Use CloudWatch to collect and track metrics and send them to Coralogix using Telegraf
Visualize metrics in your Coralogix dashboard
Import pre-built dashboards and set up alerts in your Coralogix account
Metrics Collection
There are two ways of sending AWS service metrics to Coralogix:
Real-time CloudWatch metrics processing using CloudWatch Metric Stream and Firehose Delivery Stream. Opt for this if you’d like to use a native AWS solution without an agent by setting up a CloudWatch metric stream and Firehose delivery steam. Use the Amazon CloudWatch pricing calculator to estimate the costs of this option.
Poll CloudWatch metrics using Telegraf. Opt for this if you are already using this shipper or seeking to reduce costs. The instructions below guide you through this option.
Prerequisites
Install Telegraf.
Identify the AWS services that publish metrics to CloudWatch and their namespaces. Use them in your configuration to collect the metrics.
Note: Namespaces are case-sensitive.
Configuration
- Create a configuration file. The example below reads and sends RDS and ElastiCache metrics to Coralogix in 5-minute intervals.
# Global tags can be specified here in key="value" format.
[global_tags]
dc = "us-east-2"
user = "$USER"
env = "CoE"
# Configuration for telegraf agent
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
hostname = ""
omit_hostname = false
###############################################################################
# OUTPUT PLUGINS #
###############################################################################
# Configuration for sending metrics to Coralogix
[[outputs.opentelemetry]]
service_address = "coralogix-metrics.endpoint"
insecure_skip_verify = true
compression = "gzip"
[outputs.opentelemetry.coralogix]
private_key = "your-coralogix-api-key"
application = "prod"
subsystem = "cloudwatch"
###############################################################################
# INPUT PLUGINS #
###############################################################################
# # Pull Metric Statistics from Amazon CloudWatch
[[inputs.cloudwatch]]
region = "us-east-2"
## Amazon Credentials
## Credentials are loaded in the following order
## 1) Web identity provider credentials via STS if role_arn and
## web_identity_token_file are specified
## 2) Assumed credentials via STS if role_arn is specified
## 3) explicit credentials from 'access_key' and 'secret_key'
## 4) shared profile from 'profile'
## 5) environment variables
## 6) shared credentials file
## 7) EC2 Instance Profile
access_key = "your-aws-account-access-key"
secret_key = "your-aws-account-secret-key"
# # token = ""
# # role_arn = ""
# # web_identity_token_file = ""
# # role_session_name = ""
# # profile = ""
# # shared_credential_file = ""
#
# ## The minimum period for Cloudwatch metrics is 1 minute (60s). However not
# ## all metrics are made available to the 1 minute period. Some are collected
# ## at 3 minute, 5 minute, or larger intervals.
# ## See https://aws.amazon.com/cloudwatch/faqs/#monitoring.
# ## Note that if a period is configured that is smaller than the minimum for a
# ## particular metric, that metric will not be returned by the Cloudwatch API
# ## and will not be collected by Telegraf.
# #
# ## Requested CloudWatch aggregation Period (required)
# ## Must be a multiple of 60s.
period = "5m"
#
# ## Collection Delay (required)
# ## Must account for metrics availability via CloudWatch API
delay = "5m"
#
# ## Recommended: use metric 'interval' that is a multiple of 'period' to avoid
# ## gaps or overlap in pulled data
interval = "5m"
# Case sensitive check https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html
namespaces = ["AWS/ElastiCache","AWS/RDS"]
In order to send your data to Coralogix, you are required to declare the following variables in your configuration:
service_address
: In order to send metrics to Coralogix, you will need to include your account’s specific domain in the Coralogix endpoint:otel-metrics.<span class="domain-value"></span>:443
.- For example, if you are located in Region US1, your
service_address
should appear as:otel-metrics.coralogix.us:443
.
- For example, if you are located in Region US1, your
private_key
: Access your Coralogix Send-Your-Data API key. Your key is recorded in the override file as a secret in order to ensure that this sensitive information remains protected and unexposed.application
&subsystem
: Customize and organize your data in your Coralogix dashboard using application and subsystem names.
Note: Find out more about different attributes available for the CloudWatch statistics input plugin here.
- If you would like to add additional dimensions to your CloudWatch metrics, add them in the
global_tags
section of your configuration, as in the example below.
- Save the configuration file and restart Telegraf. You should now see your metrics on your Coralogix dashboard.
Dashboard
- Download and import the Telegraf version of your RDS dashboard.
- Download and import the Telegraf version of your ElastiCache dashboard.
Alerts
Use PromQL to create metrics alerts:
- RDS CPU utilization above 80%
avg(cloudwatch_aws_rds_cpu_utilization_average{db_instance_identifier!=""}) by (db_instance_identifier)
- RDS Writer CPU utilization above 90%
avg(cloudwatch_aws_rds_cpu_utilization_average{db_cluster_identifier!="",role="WRITER"}) by (db_cluster_identifier)
- ElastiCache miss above 50%
(sum(cloudwatch_aws_elasti_cache_cache_misses_sum{})/(sum(cloudwatch_aws_elasti_cache_cache_hits_sum{})+sum(cloudwatch_aws_elasti_cache_cache_misses_sum{})))*100
Validation
To validate your configuration, access your Coralogix - Grafana dashboard.
- On the right-hand corner of your dashboard, click on the Grafana drop-down tab.
- Once you’ve accessed your Coralogix-Grafana dashboard, click on Explore tab in the left-hand browser.
- Click on the drop-down arrow of the Metrics browser and input
rds
(RDS metrics) andelasti_cache
(ElastiCache metrics) in the "Select a Metric" column.
Additional Resources
Support
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].