Importing your Cloudwatch Metrics into Prometheus

Cloudwatch is the de facto method of consuming logs and metrics from your AWS infrastructure. The problem is, it is not the de facto method of capturing metrics for your applications. This creates two places where observability is stored, and can make it difficult to understand the true state of your system.

That’s why it has become common to unify all data into one place, and Prometheus offers an open-source, vendor-agnostic solution to that problem. But we need some way of integrating Cloudwatch and Prometheus together.  

Why is one Observability source better than two?

Tool Sprawl is a real issue in the DevOps world. If your infrastructure metrics are held in one silo, and your application metrics are held elsewhere, what is the best way of correlating the two measurements? The truth is, it’s challenging to truly aggregate across multiple data sources. This is why, at Coralogix, we strongly advise that you aim for a single location to hold your telemetry data. 

Fortunately, with Prometheus and Cloudwatch, there is the perfect tool in existence.

The Prometheus Cloudwatch Exporter

To ingest Cloudwatch metrics into Prometheus, you will need to use a tool called the Prometheus Cloudwatch Exporter. This tool is a Prometheus exporter that runs on Amazon Linux, and it allows you to scrape metrics from Cloudwatch and make them available to Prometheus.

This tool acts as the bridge between Cloudwatch and Prometheus. It will unify all your application and infrastructure metrics into a single repository, which can be correlated and analyzed.

Prerequisites

• You’ll need a running Amazon EC2 instance. This can be part of a Kubernetes cluster, but we’re keeping things simple for this tutorial.

Once your EC2 instance is running, you will need to install the Prometheus Cloudwatch Exporter. You can do this by running the following command from the terminal:

curl -s https://api.github.com/repos/prometheus/cloudwatch_exporter/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

After downloading the Prometheus Cloudwatch Exporter, you will need to unpack the binary and make it executable. You can do this by running the following commands:

tar -xf cloudwatch_exporter-*.tar.gz
cd cloudwatch_exporter-*
chmod +x cloudwatch_exporter

Once the Prometheus Cloudwatch Exporter is installed, you will need to configure it to scrape your Cloudwatch metrics. You can create a new configuration file and specify the metrics you want to scrape. Here is an example configuration file:

—

region: us-east-1
namespace: AWS/EC2

metrics:

  - name: CPUUtilization

    dimensions:

      - name: InstanceId

        value: i-12345678

After you have created your configuration file, you can start the Prometheus Cloudwatch Exporter by running the following command:

./cloudwatch_exporter -config.file=config.yml

Finally, you will need to configure Prometheus to scrape the metrics from the Prometheus Cloudwatch Exporter. You can do this by adding the following configuration to your Prometheus configuration file:

scrape_configs:
- job_name: cloudwatch

    scrape_interval: 15s

    static_configs:

      - targets: ['your-ec2-instance-ip:9106']

Once you have completed these steps, Prometheus should start scraping metrics from Cloudwatch and making them available for you to query.

Does this sound like a lot of work?

It is – managing all of your own observability data can be painful. Rather than do it all yourself, why not check out the Coralogix documentation, which shows various options for integrating Coralogix and Cloudwatch. With massive scalability, a uniquely powerful architecture, and some of the most sophisticated observability features on the market, Coralogix can meet and beat your demands!