Overview

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that allows the use of Apache Kafka for streaming data processing. This guide details how to deploy an AWS MSK and prepare it to receive parsed and enriched data from Coralogix. It’s a companion guide to the Forwarders documentation, which explains how to set up and configure Coralogix to send your telemetry data to AWS MSK.

Deployment

Use one of the following deployment methods:

• Terraform

• Manual

Terraform

module "msk-data-stream" {
  source = "coralogix/aws/coralogix//modules/msk-data-stream"

  cluster_name = <your cluster name>
}

Manual

Prerequisites

• VPC

• 3 public subnets

Configuration

Update cluster policy

After the cluster has been updated and became active, copy and paste this policy.

  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<coralogix_role_arn>"
        ]
      },
      "Action": "kafka-cluster:Connect",
      "Resource": "< your_msk_cluster_arn >"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "< coralogix_role_arn >"
        ]
      },
      "Action": [
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": "arn:aws:kafka:< aws_region >:< aws_account_id >:topic/< your_cluster_name >/*"
    }
  ]

Replace the following parameters with your values:

• your_aws_account_id - Your AWS account ID

• aws_region - The AWS region that you deployed the MSK in

• your_cluster_name - The cluster name

• your_msk_cluster_arn - Your MSK cluster ARN

• aws_account_id - Your AWS account ID

• coralogix_role_arn - Copy the role from the following map:

  • EU1 - arn:aws:iam::625240141681:role/coralogix-archive-eu1

  • EU2 - arn:aws:iam::625240141681:role/coralogix-archive-eu2

  • US1 - arn:aws:iam::625240141681:role/coralogix-archive-us1

  • US2 - arn:aws:iam::739076534691:role/coralogix-archive-us2

  • AP1 - arn:aws:iam::625240141681:role/coralogix-archive-ap1

  • AP2 - arn:aws:iam::625240141681:role/coralogix-archive-ap2

Additional settings

• To create a new topic, connect to a machine that is in the same VPC as the MSK. Alternatively, connect to a machine with the AWS configuration for the account with the MSK. Once you are in the machine, run the following commands:

  • sudo yum install -y java-1.8*

  • wget <https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-3.5.1.tgz> && tar -xzvf kafka_2.13-3.5.1.tgz && rm kafka_2.13-3.5.1.tgz

  • kafka_2.13-3.5.1/bin/kafka-topics.sh --create --bootstrap-server <public brokerst list > --topic <Topic-Name> --partitions 3 --replication-factor 3

• To restrict Coralogix access only to this topic, edit the cluster policy by changing the topic resource (Resource": "arn:aws:kafka:<region>:<Account-ID>:topic*") to "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:topic/<CLUSTER_NAME>/<TOPIC_NAME>"

• Extract public endpoint for brokers.

Additional resources

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].

Theme

Light

light

dark