Our next-gen architecture is built to help you make sense of your ever-growing data. Watch a 4-min demo video!

Back to All Docs

Forward AWS Logs via Lambda Shipper Forward AWS Logs via Lambda Shipper

Last Updated: Feb. 10, 2024

Our newest AWS integration offers the most seamless way to link up with Coralogix. Using a predefined Lambda function, you can send your AWS logs and events to your Coralogix subscription for in-depth analysis, monitoring, and troubleshooting.

Overview

This integration guide focuses on connecting your AWS environment and Coralogix using AWS Lambda functions. To complete this integration, you may either use the Coralogix platform UI, CloudFormation templates from AWS, AWS SAM applications, or a dedicated Terraform module from our GitHub repository.

Our latest integration is based on the coralogix-aws-shipper, which is written in Rust. It is designed with a focus on optimizing memory safety and runtime performance. Rust improves control over system resources without sacrificing safety. As a result, your integration is more resource-efficient.

We will show you how to complete our predefined Lambda function template to simplify the integration. Your task will be to provide specific configuration parameters, based on the service that you wish to connect. The reference list for these parameters is provided below.

Note: As we improve coralogix-aws-shipper, we invite you to contribute, ask questions, and report issues in the repository.

Supported Services

Amazon S3, CloudTrail, VPC Flow Logs and more

This integration is based on S3. Your Amazon S3 bucket can receive log files from all kinds of services, such as CloudTrail, VPC Flow Logs, Redshift, Network Firewall, or different types of load balancers (ALB/NLB/ELB). This data is then sent to Coralogix for analysis.

You may also include SNS/SQS in the pipeline so that the integration triggers upon notification.

Amazon CloudWatch

Coralogix can be configured to directly receive data directly from your CloudWatch log group. In this case, the S3 is not used as an intermediary.

Amazon Kinesis

Coralogix can receive stream data from your AWS account. This option does not use S3 and lets you connect two services directly.

Amazon MSK & Kafka

Coralogix can be configured to receive data directly from your MSK or Kafka cluster.

Amazon ECR Image Security Scan

Coralogix can be configured to receive ECR Image Scanning.

Note: Although coralogix-aws-shipper handles all of the AWS product integrations, some of the parameters are product-specific. Consult the Configuration Parameters for product-specific requirements.

Deployment Options

⚠️ Important: Before you get started, ensure that your AWS user has the permissions to create Lambda functions and IAM roles.

Integrate using the Coralogix Platform (Recommended)

The fastest way to deploy your predefined Lambda function is from within the Coralogix platform. Fill out an integration form and confirm the integration from your AWS account. Product integrations can be by navigating to Data Flow > Integrations in your Coralogix toolbar. For detailed UI instructions, please read the Integration Packages tutorial.

Quick Create a CloudFormation Stack

You can always launch the CloudFormation stack by filling out a Quick Create template. This is done from within your AWS Management Console. Log into your AWS account and click the button below to deploy the CloudFormation stack.

If you use AWS CLI, you can use a CloudFormation template from our repository.

Deploy the AWS Serverless Application

Alternatively, you may use the SAM deployment link. The procedure is very similar to filling out the above Quick Create template.

Terraform Module

If you are using Terraform to launch your infrastructure, you can access coralogix-aws-shipper it via our Terraform Module. Use the parameters defined in the repository README, as they better reflect the configuration process.

Configuration Parameters

This document explains the basic config options for your template. You will need these values to launch your integration. For additional optional parameters, view our Advanced Configuration options.

Use the tables below as a guide to properly configure your deployment. The provided configuration variables are for the Serverless or CloudFormation deployment options. The variable requirements are slightly different if you wish to deploy with Terraform. Please refer to the Terraform Module for further details.

Universal Configuration

Use an existing Coralogix Send-Your-Data API key to make the connection or create one as you fill our pre-made template. Additionally, make sure your integration is region-specific.

Note: You should always deploy the AWS Lambda function in the same AWS Region as your resource (e.g. the S3 bucket).

ParameterDescriptionDefault ValueRequired
Application nameThis will also be the name of the CloudFormation stack that creates your integration. It can include letters (A–Z and a–z), numbers (0–9) and dashes (-).✔️
IntegrationTypeChoose the AWS service that you wish to integrate with Coralogix. Can be one of: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront’S3✔️
CoralogixRegionYour data source should be in the same region as the integration stack. You may choose from one of the default Coralogix regions: [Custom, EU1, EU2, AP1, AP2, US1, US2]. If this value is set to Custom you must specify the Custom Domain to use via the CustomDomain parameter.Custom✔️
CustomDomainIf you choose a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com).
ApplicationNameThe name of the application for which the integration is configured. Advanced Configuration specifies dynamic value retrieval options.✔️
SubsystemNameSpecify the name of your subsystem. For a dynamic value, refer to the Advanced Configuration section. For CloudWatch, leave this field empty to use the log group name.
ApiKeyThe Send-Your-Data API Key validates your authenticity. This value can be a direct Coralogix API Key or an AWS Secret Manager ARN containing the API Key.✔️
StoreAPIKeyInSecretsManagerEnable this to store your API Key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.True✔️

S3/CloudTrail/VpcFlow/S3Csv Configuration

This is the most flexible type of integration, as it is based on receiving log files to Amazon S3. First, your bucket can receive log files from all kinds of other services, such as CloudTrail, VPC Flow Logs, Redshift, Network Firewall or different types of load balancers (ALB/NLB/ELB). Once the data is in the bucket, a pre-made Lambda function will then transmit it to your Coralogix account.

Tip: The S3 integration supports generic data. You can ingest any generic text, JSON, and CSV data stored in your S3 bucket.

Figure 1: Sending data directly from an S3 bucket. Your applications will deposit their logs and events in a specified S3 Bucket. Each S3 Bucket event will trigger a Lambda invocation, thus sending the data to your Coralogix Account.

Adding SNS/SQS

If you don’t want to send data directly as it enters S3, you can also use SNS/SQS to maintain notifications before any data is sent from your bucket to Coralogix. For this, you need to set the SNSTopicArn or SQSTopicArn parameters.

Figure 2: First trigger via SNS, then send to an S3 bucket. Alternatively, you can configure a notification to invoke the Lambda. The SNS/SQS is triggered as your S3 Bucket event occurs.

Note: All resources, such as S3 or SNS/SQS, should be provisioned already. If you are using an S3 bucket as a resource, please make sure it is clear of any Lambda triggers located in the same AWS region as your new function.

ParameterDescriptionDefault ValueRequired
S3BucketNameSpecify the name of the AWS S3 bucket that you want to monitor.✔️
S3KeyPrefixSpecify the prefix of the log path within your S3 bucket. This value is ignored if you use the SNSTopicArn/SQSTopicArn parameter.CloudTrail/VpcFlow ‘AWSLogs/’
S3KeySuffixFilter for the suffix of the file path in your S3 bucket. This value is ignored if you use the SNSTopicArn/SQSTopicArn parameter.CloudTrail ‘.json.gz’, VpcFlow ‘.log.gz’
NewlinePatternEnter a regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).
SNSTopicArnThe ARN for the SNS topic that contains the SNS subscription responsible for retrieving logs from Amazon S3.
SQSTopicArnThe ARN for the SQS queue that contains the SQS subscription responsible for retrieving logs from Amazon S3
CSVDelimiterSpecify a single character to be used as a delimiter when ingesting a CSV file with a header line. This value is applicable when the S3Csv integration type is selected, for example, “,” or ” “.,

CloudWatch Configuration

Coralogix can be configured to receive data directly from your CloudWatch log group. This option does not use S3. You must provide the log group name as a parameter during setup.

Figure 3: CloudWatch logs are streamed directly to Coralogix via Lambda.

ParameterDescriptionDefault ValueRequired
CloudWatchLogGroupNameProvide a comma-separated list of CloudWatch log group names to monitor, for example, (log-group1, log-group2, log-group3).✔️

SNS Configuration

To receive SNS messages directly to Coralogix, use the SNSIntegrationTopicARN parameter. This differs from the above use of SNSTopicArn, which notifies based on S3 events.

ParameterDescriptionDefault ValueRequired
SNSIntegrationTopicARNProvide the ARN of the SNS topic to which you want to subscribe for retrieving messages.✔️

SQS Configuration

To receive SQS messages directly to Coralogix, use the SQSIntegrationTopicARN parameter. This differs from the above use of SQSTopicArn, which notifies based on S3 events.

ParameterDescriptionDefault ValueRequired
SQSIntegrationTopicARNProvide the ARN of the SQS queue to which you want to subscribe for retrieving messages.✔️

Kinesis Configuration

We can receive direct stream data from your AWS account. This option does not use S3. Your Kinesis stream ARN is a required parameter in this case.

Figure 4: Streaming Kinesis data from AWS Services directly to Coralogix

ParameterDescriptionDefault ValueRequired
KinesisStreamARNProvide the ARN of the Kinesis Stream to which you want to subscribe for retrieving messages.✔️

Kafka Configuration

ParameterDescriptionDefault ValueRequired
KafkaBrokersComma-delimited list of Kafka brokers to establish a connection with.✔️
KafkaTopicSubscribe to this Kafka topic for data consumption.✔️
KafkaBatchSizeSpecify the size of data batches to be read from Kafka during each retrieval.100
KafkaSecurityGroupsComma-delimited list of Kafka security groups for secure connection setup.✔️
KafkaSubnetsComma-delimited list of Kafka subnets to use when connecting to Kafka.✔️

MSK Configuration

Your Lambda function must be in a VPC that has access to the MSK cluster. You can configure your VPC via the provided VPC configuration parameters.

ParameterDescriptionDefault ValueRequired
MSKBrokersComma-delimited list of MSK brokers to connect to.✔️
KafkaTopicSubscribe to this Kafka topic for data consumption.✔️

Generic Configuration (Optional)

These are optional parameters if you wish to 1) receive notification emails, 2) exclude certain logs or 3) send messages to Coralogix at a particular rate.

ParameterDescriptionDefault ValueRequired
NotificationEmailA failure notification will be sent to this email address.
BlockingPatternEnter a regular expression to identify lines excluded from being sent to Coralogix. For example, use MainActivity.java:\d{3} to match log lines with MainActivity followed by exactly three digits.
SamplingRateSend messages at a specific rate, such as 1 out of every N logs. For example, if your value is 10, a message will be sent for every 10th log.1✔️

Lambda Configuration (Optional)

These are the default presets for Lambda. Read Troubleshooting for more information on changing these defaults.

ParameterDescriptionDefault ValueRequired
FunctionMemorySizeSpecify the memory size for the Lambda function in megabytes.1024✔️
FunctionTimeoutSet a timeout for the Lambda function in seconds.300✔️
LogLevelSpecify the log level for the Lambda function, choosing from the following options: INFO, WARN, ERROR, DEBUG.WARN✔️
LambdaLogRetentionSet the CloudWatch log retention period (in days) for logs generated by the Lambda function.5✔️

VPC Configuration (Optional)

Use the following options if you need to configure a private link with Coralogix.

ParameterDescriptionDefault ValueRequired
LambdaSubnetIDSpecify the ID of the subnet where the integration should be deployed.✔️
LambdaSecurityGroupIDSpecify the ID of the Security Group where the integration should be deployed.✔️
UsePrivateLinkSet this to true if you will be using AWS PrivateLink.false✔️

Advanced Configuration

AWS PrivateLink

If you want to bypass using the public internet, you can use AWS PrivateLink to facilitate secure connections between your VPCs and AWS Services. This option is available under the VPC Configuration tab. To turn it on, either check off the Use Private Link box in the Coralogix UI or set the parameter to true. For additional instructions on AWS PrivateLink, please follow our dedicated tutorial.

Dynamic Values

If you wish to use dynamic values for the Application and Subsystem Name parameters, consider the following:

  • JSON support: To reference dynamic values from the log, use $.my_log.field. For the CloudTrail source, use $.eventSource.
  • S3 folder: Use the following tag: {{s3_key.value}} where the value is the folder level. For example, if the file path that triggers the event is AWSLogs/112322232/ELB1/elb.log or AWSLogs/112322232/ELB2/elb.log and you want ELB1 and ELB2 to be the subsystem, your subsystemName should be {{s3_key.3}}

Troubleshooting

Timeout errors

If you see “Task timed out after”, you need to increase the Lambda Timeout value. You can do this from the AWS Lambda function settings under Configuration > General Configuration.

Not enough memory

If you see “Task out of memory”, you should increase the Lambda maximum Memory value. In the AWS Lambda function settings, go to Configuration > General Configuration.

Verbose logs

To add more verbosity to your function logs, set RUST_LOG to DEBUG.

⚠️ Warning: Remember to change it back to INFO after troubleshooting.

Changing defaults

  • Set the MAX_ELAPSED_TIME variable for default change (default = 250).
  • Set BATCHES_MAX_SIZE (in MB) sets batch max size before sending to Coralogix. This value is limited by the max payload accepted by the Coralogix endpoint (default = 4).
  • Set BATCHES_MAX_CONCURRENCY sets the maximum amount of concurrent batches that can be sent.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Contact us via our in-app chat or by emailing [email protected].

On this page