Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

S3 Log Collection

Description

S3 to Coralogix lambda allows you to send your logs from your S3 bucket to Coralogix.

The preferred and easiest integration method will be to use our aws Serverless Application Repository. Search for ‘coralogix’. Don’t forget to check the ‘Show apps that create custom IAM roles or resource policies’ box located just under the search field to see all available applications. Select your application of choice and click on it. You will see detailed instructions in the readme section on the left.

Automation

You can include SAM (Serverless Application Model) in your automation frameworks. If you need access to the latest and greatest Lambda code go to https://github.com/coralogix/coralogix-aws-serverless/tree/master/src.

The rest of this document describes a manual configuration of this integration and the Lambda associated with it and should be used if there is a need for special customization.

Usage

This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

 .com.us.in
Elasticsearch-APIhttps://coralogix-esapi.coralogix.com:9443https://esapi.coralogix.us:9443https://es-api.app.coralogix.in:9443
SSL Certificateshttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crthttps://www.amazontrust.com/repository/AmazonRootCA1.pemhttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN
.pem
coralogix.comcoralogix.usapp.coralogix.in

You must add the following environment variables when using this lambda.

1. Create an “author from scratch” Node.js 12.x runtime lambda with an S3 read permissions 

2. At “Code entry type” choose “Upload a ZIP file” and upload “s3ToCoralogix.zip”

https://s3-eu-west-1.amazonaws.com/coralogix-public/tools/s3ToCoralogix.zip

3. Add the mandatory environment variables: private_key, app_name, sub_name

env vars for coralogix S3 lambda

If your Coralogix account top level domain is not ‘.com’, add the following variable:

CORALOGIX_LOG_URL=https://api.Cluster URL/api/v1/logs

4. Increase Memory to 1024MB and Timeout to 3 min

5. Choose the S3 bucket you want to get triggered by and change the event type from “PUT” to “Object Created(All)”

S3 bucket for coralogix

6. Multiline pattern: Coralogix supports multiline pattern by default, you can define a custom pattern with an environment variables, for example:

newline_pattern [\s(?={)|(?<=})\s,\s(?={)|(?<=})\s\]. 

custom multiline in S3

6. Click ‘Save’

Your all set! data would be streaming to Coralogix, need help? ping us on our in-app chat for support@coralogix.com.