We're launching a new cybersecurity venture! Learn more about Snowbit >

S3 Log Collection S3 Log Collection

Last Updated: Mar. 30, 2022

One of the easiest ways to collect logs is to send all of them to S3 buckets.
We aim to help and make the log collection a breeze so we have An application published in AWS’s Serverless Application Repository.

Requirements

  • You need to have a ready made S3 bucket.
  • The bucket should be clear of any Lambda triggers.
  • Your AWS user should have permissions to create lambdas and IAM roles.

Installation

  • Navigate to: Application Page
  • Fill in the required parameters
  • be sure to check the checkbox labeled as
    I acknowledge that this app creates custom IAM roles and resource policies.
  • Click Deploy

Once the deployment is done every new gzipped log file placed in the lambda will be sent to Coralogix

Parameters and Descriptions

VariableDescription
Application NameThe stack name of this application created via AWS CloudFormation.
If your log is JSON format, can use a Dynamic value from it.
Example:
$.level1.level2.value
NotificationEmailFailure notification email address
S3BucketNameThe name of the S3 bucket to watch
ApplicationNameApplication Name in Coralogix.
BufferSizeBuffer size for logs in the lambda function.
CoralogixRegionThe Coralogix location region [Europe, India, Singapore, US] Check your Coralogix account if url ends in .us its US. Region. Same for the others.
DebugCoralogix logger debug mode
FunctionArchitectureOur Function supports x86_64 or arm64
FunctionMemorySizeMax Memory for the function itself.
FunctionTimeoutFunction TimeOut
NewlinePatternPattern for multiLine example: [\s(?={)|(?<=})\s,\s(?={)|(?<=})\s]
PrivateKeyYour Coralogix account ‘send your logs’ key. Can be found in DataFlow -> API Keys
SamplingRateSets the sampling rate
SubsystemNameSubSystem name in Coralogix.
If your log is JSON format, can use a Dynamic value from it.
Example:
$.level1.level2.value
S3KeyPrefixThe S3 path prefix to watch, if you want to watch a particular subfolder within the bucket.
S3KeySuffixThe S3 path suffix to watch

Deploying this lambda with Terraform

We realize your infrastructure is rapidly growing.

You may wish to replicate this lambda or simply have multiple deployments managed as code.

We provide a terraform example just for that. You can find the Terraform module on GitHub

On this page