Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

AWS Kinesis with Lambda function

Usage

Coralogix provides a predefined Lambda function to forward your Kinesis stream straight to Coralogix.

The preferred and easiest integration method will be to use our aws Serverless Application Repository. Search for ‘coralogix’. Don’t forget to check the ‘Show apps that create custom IAM roles or resource policies’ box located just under the search field to see all available applications. Select your application of choice and click on it. You will see detailed instructions in the readme section on the left.

Automation

You can include SAM (Serverless Application Model) in your automation frameworks. If you need access to the latest and greatest Lambda code go to https://github.com/coralogix/coralogix-aws-serverless/tree/master/src.

The rest of this document describes a manual configuration of this integration and the Lambda associated with it and should be used if there is a need for special customization.

This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

 .com.us.in
Elasticsearch-APIhttps://coralogix-esapi.coralogix.com:9443https://esapi.coralogix.us:9443https://es-api.app.coralogix.in:9443
SSL Certificateshttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crthttps://www.amazontrust.com/repository/AmazonRootCA1.pemhttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN
.pem
coralogix.comcoralogix.usapp.coralogix.in

1. Create an “author from scratch” Node.js 12.x runtime lambda with the following permissions(Create execution role tutorial):

2. At “Code entry type” choose “Edit code inline” and paste this function

3. Add the mandatory environment variables:

  • Private Key – A unique ID that represents your company. The private key can be found under ‘settings’->’ send your logs’. It is located in the upper left corner.
  • Application Name – Used to separate your environment, e.g. SuperApp-test/SuperApp-prod.
  • If your Coralogix account top level domain is not ‘.com’, add the following environment variable:
    CORALOGIX_URL=https://api.Cluster URL/api/v1/logs
  • SubSystem Name – Your application probably has multiple subsystems, for example, Backend servers, Middlewar, Frontend servers etc.Amazon kinesis integration coralogix environment variables

4.   Go to Add triggers and add Kinesis:

amazon kinesis integration coralogix designer

5.  Configure the trigger, select the desired “Kinesis stream” and “Consumer”, change “Batch size” equals to 10:
amazon kinesis integration coralogix configure triggers

6.  Increase Memory to 1024MB and Timeout to 1 min.
amazon kinesis integration coralogix basic settings

7.  Click “Save”.