We're launching a new cybersecurity venture! Learn more about Snowbit >

Amazon Kinesis Data Firehose Amazon Kinesis Data Firehose

Last Updated: Apr. 20, 2022

Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for getting data from multiple AWS services.

Kinesis Data Firehose provides built-in, fully managed error handling, transformation, conversion, aggregation, and compression functionality, so you won’t need to write applications to handle these complexities.

Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS  Competencies in DevOps. The platform enables you to easily explore and analyze logs to gain deeper insights into the state of your applications and AWS infrastructure. You can analyze all your AWS service logs while storing only the ones you need, and generate metrics from aggregated logs to uncover and alert on trends in your AWS services.

Using Coralogix with Amazon Kinesis Data Firehose offers a few significant benefits compared with other solutions:

  • It keeps monitoring simple.
  • It integrates flawlessly.
  • It’s flexible with minimum maintenance.
  • Scale, scale, scale.

Requirements

  •  A Private Key – You can find your key in your Coralogix account under ‘Data Flow’ -> ‘API Keys’ -> ‘Send your logs’.
data flow > api keys
api access

Configuration

Go to the Kinesis Data Firehose console and choose ‘Create delivery stream’.

Under ‘Choose source and destination’ choose these:

  • For ‘Source’ choose ‘Direct PUT’
  • For ‘Destination’ choose Coralogix 

Under ‘Delivery stream name’ fill desired stream name:

create delivery stream

Scroll down to ‘Destination settings’ to select your ‘HTTP endpoint URL’ based on your Region and Coralogix account configuration:

firehose destination settings
  • For ‘Private key’ enter your Coralogix Private Key.
  • For ‘Content encoding’ select GZIP.
  • For ‘Retry duration’ select 30 seconds.
firehose destination settings

In order to override the associated ‘applicationName’ or ‘subsystemName’, under the Parameters section, enter the parameter you want to change as the key and choose a new value.

It will then be overridden as a different value, the default is ‘firehose’.

app and subsystem name firehose

Use this same process for changing additional parameters such as ‘integrationType’.

In order to use Firehose to send CloudWatch Logs natively to Coralogix, set the value for ‘integrationType’ to ‘CloudWatch’.

integration type cloudwatch aws firehose

Scroll down to ‘Backup settings’ and fill it to your liking.

Ideally for ‘Source record backup in Amazon S3’` select ‘Failed data only’.

For ‘S3 backup bucket’ either choose an existing bucket or create a new one.
You can keep ‘Buffer hints, compression and encryption’ as is.

  • You can keep ‘Advanced settings’ as is.
  • Review your settings and choose ‘Create delivery stream’.

Logs subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.

Log Formatting

Logs sent to Coralogix should be structured in the following way:

Required?Property NameProperty TypeNotes
YestimestampnumberUTC milliseconds since 1970 (supports sub millisecond via a floating point)
Yesseveritynumber1 – Debug, 2 – Verbose, 3 – Info, 4 – Warn, 5 – Error, 6 – Critical
Yestextstring
categorystring
classNamestring
methodNamestring
threadIdstring

Find more information in our Coralogix Rest API documentation.

To get the most out of the platform, be sure to check out the documentation which will help you get started with everything from parsing and enrichment to alerting and data clustering.

If you have any questions about the configuration or want help to dive into more advanced features, our support team is available 24/7 via our in-app chat!

On this page