Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Integrations

AWS Kinesis Data Firehose – Logs AWS Kinesis Data Firehose – Logs

Last Updated: Apr. 30, 2023

Amazon Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for retrieving data from multiple AWS services.

Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS  Competencies in DevOps. The platform enables you to easily explore and analyze logs to gain deeper insights into the state of your applications and AWS infrastructure. Analyze all of your AWS service logs while storing only those you need. Generate metrics from aggregated logs to uncover and alert on trends in your AWS services.

Overview

Using Coralogix with Amazon Kinesis Data Firehose offers significant benefits when compared with other solutions.

  • It keeps monitoring simple.
  • It integrates flawlessly.
  • It’s flexible with minimum maintenance.
  • Scale, scale, scale.
  • Real-time push monitoring involves pushing events instead of pulling.

Prerequisites

1. Sign up for a Coralogix account. Set up your account on the Coralogix domain corresponding to the region within which you would like your data stored.

2. Access your Coralogix private key.

Configuration

STEP 1. Navigate to the Kinesis Data Firehose console and choose ‘Create delivery stream’.

STEP 2. Under ‘Choose source and destination’:

  • Source: Choose Direct PUT
  • Destination: Choose Coralogix 
  • Delivery stream name: Fill in the desired stream name

STEP 3. Scroll down to ‘Destination settings’:

  • HTTP endpoint URL: Choose a HTTP endpoint URL based on your Coralogix region.
  • Private key: Enter your Coralogix private key.
  • Content encoding: Select GZIP.
  • Retry duration: Choose 300 seconds.

STEP 4. Scroll down to ‘Parameters’. This section allows you to add and configure additional parameters surrounding the Coralogix platform.

The following parameters are available:

ParameterDescription
applicationNameThe desired applicationName within the Coralogix Platform
subsystemNameThe desired subsystemName within the Coralogix Platform
integrationTypeData structure:
– CloudWatch_JSON: data from cloudWatch log groups
– WAF
– CloudWatch_CloudTrail
– EksFargate
– Default
– RawText: use for VPC flow logs
dynamicMetadataWhen set to true, it fetches set the applicationName / subsystemName dynamically

Notes:

  • By default, your delivery stream name will be used as ‘applicationName’ and ARN as ‘subsystemName’.
  • To override the associated ‘applicationName’ or ‘subsystemName’, add a new parameter with the desired value.
    • Key: ‘applicationName’ , value – ‘new-app-name’
    • Key: ‘subsystemName’ , value – ‘new-subsystem-name’
  • The source of the data in Firehose determines the ‘integrationType’ parameter value:
    • For CloudWatch logs, use ‘CloudWatch_JSON’.
    • For CloudTrail logs in CloudWatch, use ‘CloudWatch_CloudTrail’.
    • For logs coming from EKS Fargate using our guide, use ‘EksFargate’.
    • For logs coming from AWS WAF, use ‘WAF’.
    • For data sources matching the Coralogix log ingestion format, use ‘Default’.
    • For all other data sources, use ‘RawText’. This moves all the text to text field of log, adds severity of Info, and generates a current timestamp. All further parsing of these logs should be done using parsing rules.
  • Without adding the ‘integrationType’ parameter, the ‘Default’ integration type is selected.
  • For integration of type ‘Default’, the logs should be structured according to our REST API rules.
  • Dynamic values search for specific fields in the logs to populate the fields. Hard-coded parameters take precedence over dynamic values. To use dynamic values for applicationName and subsystemName, add a parameter.
  • Dynamic values are displayed in the table below.
RequiredProperty NameProperty TypeNotes
YestimestampnumberUTC milliseconds since 1970 (supports sub millisecond via a floating point)
Yesseveritynumber1 – Debug, 2 – Verbose, 3 – Info, 4 – Warn, 5 – Error, 6 – Critical
Yestextstring
hiResTimestampstringUTC nanoseconds since 1970
(supports millisecond, microsecond and nanosecond)
classNamestring
methodNamestring
threadIdstring
categorystring

STEP 5. Set up a recovery bucket (recommended). Enabling source data backup ensures that the data can be recovered if record processing transformation does not produced the desired results.

STEP 6. Scroll down to ‘Backup settings’:

  • Source record backup in Amazon S3: We suggest selecting Failed data only.
  • S3 backup bucket: Choose an existing bucket or create a new one.
  • Buffer hints, compression, encryption: Leave these fields as is.

STEP 7. Review your settings and select Create delivery stream.

Logs subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.

Dynamic values table

TypeDynamic applicationNameDynamic subsystemNameNotes
CloudWatch_JSONthe cloudwatch log groupnonesupplied by aws
CloudWatch_CloudTrailthe cloudwatch log groupnonesupplied by aws
Default‘applicationName’ field‘subsystemName’ fieldneed to be supplied in the log to be used
EksFargate‘kubernetes.namespace_name’ field‘kubernetes.container_name’ fieldsupplied by the default configuration
WAFThe web acl namenonesupplied by aws

Data Source Configuration

Below are a couple of ways to connect your data source to firehose.

Cloudwatch logs

To start sending your logs to Coralogix you first need to create a subscription filter inside your Cloudwatch log group.

First, create a new role in IAM for your Cloudwatch log group to allow sending data to firehose

Go to the IAM console and choose ‘Roles’ under ‘Access management’

Click on ‘Create role’ on the right

Under ‘Trusted entity type’ choose ‘Custom trust policy’ and insert this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.<region_code>.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Note: change ‘<region_code>’ into your AWS region. e.g ‘us-east-1’

Now on ‘Add permissions’ click on ‘Create policy’.

on the opened window click on the ‘JSON’ tab and insert this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": [
                "<firehose_ARN>"
            ]
        }
    ]
}

Note: change ‘<firehose_ARN>’ to your firehose amazon resource name (arn).

After creating the policy go back to the role creation page and click on the refresh button

Pick the newly created policy.

Finally, give your role a name and create it.

After creating the role, go to the Cloudwatch console and choose ‘Logs groups’ under the ‘Logs’ side menu.

Create a new subscription filter for the relevant log group – ‘<Your_log_group>’ -> ‘Subscription filters’ -> ‘Create Kinesis Firehose subscription filter’.

Under ‘Choose destination’:

  • For ‘Destination account’ choose ‘Current account’
  • For ‘Kinesis Firehose delivery stream’ choose the created firehose delivery stream 

Scroll down to ‘Grant permission’:

  • For ‘Select an existing role’ choose the role created above

After that scroll down and click on ‘Start streaming’.

That’s it logs coming to your cloudwatch log group will also be directed to firehose.

Note: be sure to use the correct integration type inside your firehose configuration.

Kinesis Data Stream

To start sending your Kinesis data stream logs to coralogix we need to connect the Data stream to Firehose.

Go to the Kinesis Data Stream console and choose ‘Create data stream’.

Under ‘Data stream configuration’:

  • Data stream name: Enter the name of the data stream

Scroll down to ‘Data stream capacity’:

  • Capacity mode: Choose ‘On-demand’

After that scroll down and click on ‘Create data stream’.

Note: to connect a kinesis data stream to a firehose delivery stream the delivery stream must use ‘Amazon kinesis data streams’ as its source instead of ‘Direct PUT’.

To get the most out of the platform, be sure to check out the documentation which will help you get started with everything from parsing and enrichment to alerting and data clustering.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

On this page