Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Integrations

Amazon Kinesis Data Firehose – Logs Amazon Kinesis Data Firehose – Logs

Last Updated: Sep. 13, 2022

Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for getting data from multiple AWS services.

Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS  Competencies in DevOps. The platform enables you to easily explore and analyze logs to gain deeper insights into the state of your applications and AWS infrastructure. You can analyze all your AWS service logs while storing only the ones you need, and generate metrics from aggregated logs to uncover and alert on trends in your AWS services.

Using Coralogix with Amazon Kinesis Data Firehose offers a few significant benefits compared with other solutions:

  • It keeps monitoring simple.
  • It integrates flawlessly.
  • It’s flexible with minimum maintenance.
  • Scale, scale, scale.

Requirements

  •  A Private Key – You can find your key in your Coralogix account under ‘Data Flow’ -> ‘API Keys’ -> ‘Send your logs’.
data flow > api keys
api access

Configuration

Go to the Kinesis Data Firehose console and choose ‘Create delivery stream’.

Under ‘Choose source and destination’:

  • For ‘Source’ choose ‘Direct PUT’
  • For ‘Destination’ choose Coralogix 
  • For ‘Delivery stream name’ – fill in the desired stream name.

Scroll down to ‘Destination settings’:

  • For ‘HTTP endpoint URL’ choose based on your Region.
  • For ‘Private key’ enter your Coralogix Private Key.
  • For ‘Content encoding’ select GZIP.
  • For ‘Retry duration’ choose 30 seconds.

Scroll down to ‘Parameters’:

By default, your delivery stream arn and name will be used as ‘applicationName’ and ‘subsystemName’.

To override the associated ‘applicationName’ or ‘subsystemName’ add a new parameter with the wanted value.

  • Key – ‘applicationName’ , value – ‘new-app-name’
  • Key – ‘subsystemName’ , value – ‘new-subsystem-name’

To use dynamic values for applicationName and subsystemName add parameter.

  • Key – ‘dynamicMetadata’ , value – ‘true’

Note: Dynamic values searches for specific fields in the logs to populate the fields, also, hard-coded parameter takes precedence on dynamic values.

Dynamic values table is shown below.

The source of the data in firehose determines the ‘integrationType’ parameter value:

  • For most CloudWatch logs use ‘CloudWatch_JSON’.
  • For CloudTrail logs in CloudWatch use ‘CloudWatch_CloudTrail’.
  • For logs coming from EKS Fargate using our guide use ‘EksFargate’
  • For all other data sources use ‘Default’.

Note: without adding the ‘integrationType’ parameter the ‘Default’ integration type is selected

For integration of type ‘Default’ the logs should be structured according to our REST API rules

RequiredProperty NameProperty TypeNotes
YestimestampnumberUTC milliseconds since 1970 (supports sub millisecond via a floating point)
Yesseveritynumber1 – Debug, 2 – Verbose, 3 – Info, 4 – Warn, 5 – Error, 6 – Critical
Yestextstring
hiResTimestampstringUTC nanoseconds since 1970
(supports millisecond, microsecond and nanosecond)
classNamestring
methodNamestring
threadIdstring
categorystring

Optional configuration

Scroll down to ‘Backup settings’:

  • Source record backup in Amazon S3 – We suggest selecting Failed data only.
  • S3 backup bucket – Choose an existing bucket or create a new one.
  • Buffer hints, compression, encryption – Leave these fields as is.

Review your settings and choose ‘Create delivery stream’.

Logs subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.

Dynamic values table

TypeDynamic applicationNameDynamic subsystemNameNotes
CloudWatch_JSONnonethe cloudwatch log groupsupplied by aws
CloudWatch_CloudTrailnonethe cloudwatch log groupsupplied by aws
Default‘applicationName’ field‘subsystemName’ fieldneed to be supplied in the log to be used
EksFargate‘kubernetes.namespace_name’ field‘kubernetes.container_name’ fieldsupplied by the default configuration

Data source configuration

Below are a couple of ways to connect your data source to firehose.

Cloudwatch logs

To start sending your logs to coralogix you first need to create a subscription filter inside your cloudwatch log group.

First, create a new role in IAM for your cloudwatch log group to allow sending data to firehose

Go to the IAM console and choose ‘Roles’ under ‘Access management’

Click on ‘Create role’ on the right

Under ‘Trusted entity type’ choose ‘Custom trust policy’ and insert this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.<region_code>.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Note: change ‘<region_code>’ into your AWS region. e.g ‘us-east-1’

Now on ‘Add permissions’ click on ‘Create policy’.

on the opened window click on the ‘JSON’ tab and insert this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": [
                "<firehose_ARN>"
            ]
        }
    ]
}

Note: change ‘<firehose_ARN>’ to your firehose amazon resource name (arn).

After creating the policy go back to the role creation page and click on the refresh button

Pick the newly created policy.

Finally, give your role a name and create it.

After creating the role, go to the Cloudwatch console and choose ‘Logs groups’ under the ‘Logs’ side menu.

Create a new subscription filter for the relevant log group – ‘<Your_log_group>’ -> ‘Subscription filters’ -> ‘Create Kinesis Firehose subscription filter’.

Under ‘Choose destination’:

  • For ‘Destination account’ choose ‘Current account’
  • For ‘Kinesis Firehose delivery stream’ choose the created firehose delivery stream 

Scroll down to ‘Grant permission’:

  • For ‘Select an existing role’ choose the role created above

After that scroll down and click on ‘Start streaming’.

That’s it logs coming to your cloudwatch log group will also be directed to firehose.

Note: be sure to use the correct integration type inside your firehose configuration.

Kinesis Data Stream

To start sending your Kinesis data stream logs to coralogix we need to connect the Data stream to Firehose.

Go to the Kinesis Data Stream console and choose ‘Create data stream’.

Under ‘Data stream configuration’:

  • For ‘Data stream name’ enter the name of the data stream

Scroll down to ‘Data stream capacity’:

  • For ‘Capacity mode’ choose ‘On-demand’

After that scroll down and click on ‘Create data stream’.

Note: to connect a kinesis data stream to a firehose delivery stream the delivery stream must use ‘Amazon kinesis data streams’ as its source instead of ‘Direct PUT’.

To get the most out of the platform, be sure to check out the documentation which will help you get started with everything from parsing and enrichment to alerting and data clustering.

If you have questions about the configuration or want help diving into more advanced features, our support team is available 24/7 via our in-app chat!

On this page