Our next-gen architecture is built to help you make sense of your ever-growing data. Watch a 4-min demo video!

Back to All Docs

AWS MSK & Kafka AWS MSK & Kafka

Last Updated: Dec. 16, 2023

Use Coralogix Kafka lambdas to seamlessly send Kafka Topics data to Coralogix.

Overview

Many organizations collect important operational and business data in Kafka topics and would like to aggregate, analyze, and correlate it with data collected from other sources, in order to enable a better view of the technology and business stack and improve its management.

Use Coralogix Kafka lambdas to seamlessly send Kafka Topics data to Coralogix with our AWS serverless application repository to gain these desired insights into your data.

Requirements

  • AWS account with permissions to create lambdas and IAM roles
  • A ready-made MSK cluster
  • A ready-made Kafka cluster

Installation

STEP 1. Navigate to this application page.

STEP 2. Fill in the required parameters

STEP 3. Check this checkbox: “I acknowledge that this app creates custom IAM roles.”

STEP 4. Click Deploy.

Parameters and Descriptions

VariableDescription
Application NameStack name of this application created via AWS CloudFormation
ApplicationNameApplication name as it appears in your Coralogix UI
If your log is JSON format, use its dynamic value, for example: $.level1.level2.value
CoralogixRegionRegion [Europe, Europe2, India, Singapore, or US] associated with your Coralogix account domain
In case that you want to use Custom domain, leave this as default and write the custom domain in the CustomDomain filed.
CustomDomainCoralogix custom domain. Leave empty if you do not use a custom domain.
FunctionArchitectureFunction supports x86_64 or arm64
FunctionMemorySizeMax memory for the function itself
FunctionTimeoutMaximum time in seconds the function may be allowed to run
MSKClusterArnARN of the Amazon MSK Kafka cluster
NotificationEmailFailure notification email address
PrivateKeyCoralogix Send-Your-Data API Key
SsmEnabledTrue, if you want to store your coralogix private_key as a secret
False, if you do not
SubsystemNameSubsystem name as it appears in your Coralogix UI
If your log is in JSON format, use its dynamic value, for example: $.level1.level2.value.
TopicName of the Kafka topic used to store records in your Kafka cluster
LayerARNCoralogix SSM Layer ARN

Notes:

  • You can dynamically set the application and subsystem names by setting the corresponding parameter above with a filter string with the following syntax: $.first_key.additional_key.
  • The example $.computedValues.functionName uses the functionName of a computedValues array as your dynamic value.

Automation

You can include SAM (Serverless Application Model) in your automation frameworks. If you need access to the latest and greatest Lambda code go to coralogix-aws-serverless/src at master · coralogix/coralogix-aws-serverless.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at support@coralogix.com.

On this page