Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Integrations

Amazon Kinesis Data Firehose – Metrics Amazon Kinesis Data Firehose – Metrics

Last Updated: Nov. 24, 2022

Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for getting data from multiple AWS services.

Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS  Competencies in DevOps. The platform enables you to easily explore and analyze logs, metrics, and traces to gain deeper insights into the state of your applications and AWS infrastructure. You can analyze all your AWS service metrics to uncover trends in your AWS services.

Using Coralogix with Amazon Kinesis Data Firehose offers a few significant benefits compared with other solutions:

  • It keeps monitoring simple.
  • It integrates flawlessly.
  • It’s flexible with minimum maintenance.
  • Scale, scale, scale.

Requirements

  •  A Private Key – You can find your key in your Coralogix account under ‘Data Flow’ -> ‘API Keys’ -> ‘Send your logs’.
data flow > api keys
api access
  • A metrics bucket configured – You can configure your metrics bucket in your Coralogix account under ‘Data Flow’ -> ‘Setup Archive’.

Configuration

Go to the Kinesis Data Firehose console and choose ‘Create delivery stream’.

Under ‘Choose source and destination’:

  • For ‘Source’ choose ‘Direct PUT’
  • For ‘Destination’ choose Coralogix 
  • For ‘Delivery stream name’ – fill in the desired stream name.

Scroll down to ‘Destination settings’:

  • For ‘HTTP endpoint URL’ choose based on your Region.
  • For ‘Private key’ enter your Coralogix Private Key.
  • For ‘Content encoding’ select GZIP.
  • For ‘Retry duration’ choose 300 seconds.

Scroll down to ‘Parameters’:

By default, your delivery stream arn and name will be used as ‘applicationName’ and ‘subsystemName’.

To override the associated ‘applicationName’ or ‘subsystemName’ add a new parameter with the wanted value.

  • Key – ‘applicationName’ , value – ‘new-app-name’
  • Key – ‘subsystemName’ , value – ‘new-subsystem-name’

The source of the data in firehose determines the ‘integrationType’ parameter value:

  • For CloudWatch Metrics use ‘CloudWatch_Metrics_OpenTelemetry070’.

Optional configuration

Scroll down to ‘Backup settings’:

  • Source record backup in Amazon S3 – We suggest selecting Failed data only.
  • S3 backup bucket – Choose an existing bucket or create a new one.
  • Buffer hints, compression, encryption – Leave these fields as is.

Review your settings and choose ‘Create delivery stream’.

Metrics subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.

Data source configuration

Cloudwatch metrics

To start sending your metrics to coralogix you first need to create a metric stream.

Go to the Cloudwatch console and choose ‘Streams’ under the ‘Metrics’ side menu.

Click on ‘Create metric stream’

Under ‘Metrics to be streamed’:

  • choose what metrics to send

Scroll down to ‘Configuration’:

  • For ‘Select configuration option’ choose ‘Select an existing Firehose owned by your account’
  • For ‘Select your Kinesis Data Firehose stream’ choose the delivery stream created above

Scroll down to ‘Change output format’

  • Make sure that ‘OpenTelemetry 0.7’ is selected

Scroll down to ‘Custom metric stream name’ and pick a name for the metrics stream.

Scroll down and click on ‘Create metric stream’.

After a few minutes, the metrics will start streaming to coralogix and you will see them on the Grafana dashboard.

Destination Errors

Common destinations errors that firehose might show and their possible solutions

MessageSolution
The delivery timed out before a response was received and will be retried. If this error persists, contact the AWS Firehose service team.None needed – no data loss
Delivery to the endpoint was unsuccessful. See Troubleshooting HTTP Endpoints in the Firehose documentation for more information. Response received with status code. 502…Coralogix returned HTTP 502 error code, firehose will resend the data. None needed – no data loss

To get the most out of the platform, be sure to check out the documentation which will help you get started with everything from parsing and enrichment to alerting and data clustering.

If you have questions about the configuration or want help diving into more advanced features, our support team is available 24/7 via our in-app chat!

On this page