# Overview

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that allows the use of Apache Kafka for streaming data processing. This guide details how to deploy an AWS MSK and prepare it to receive parsed and enriched data from Coralogix. It’s a companion guide to the [Forwarders](https://coralogix.com/docs/user-guides/data-flow/forwarders/index.md) documentation, which explains how to set up and configure Coralogix to send your telemetry data to AWS MSK.

# Deployment

Use one of the following deployment methods:

- Terraform
- Manual

## Terraform

1.

Run the following file to deploy an AWS MSK. If needed, refer to this [readme file](https://github.com/coralogix/terraform-coralogix-aws/blob/master/modules/provisioning/msk-data-stream/README.md) for more deployment options.

**Note**: Module creation might take between 40 and 90 minutes.

```tf
module "msk-data-stream" {
  source = "coralogix/aws/coralogix//modules/msk-data-stream"

  cluster_name = <your cluster name>
}
```

2.

After the module is created, copy the public endpoints of the cluster brokers. Use them in the **AWS MSK URLs** field of the **Forwarder Setup** dialog box.

## Manual

### Prerequisites

- VPC
- 3 public subnets

### Configuration

1.

In the Search bar of the AWS Management Console, search for **MSK**, then click **Create cluster**.

2.

Enter the following cluster parameters and click **Next**.

**Note:** you can increase the storage according to your requirements.

3.

In the **Networking** section, select an existing VPC.

4.

In the **Zones** section, select your zones and choose the public subnets.

5.

In the **Security** groups (Amazon EC2 section), click **Browse**, select your security group, and click **Choose**.

6.

Make sure that your security group allows Coralogix to access the MSK.

7.

In the **Security settings** section, enable the **IAM role-based authentication** option and click **Next**.

8.

In the **Monitoring** section, select **Basic monitoring**, then click **Next**.

**Note**: Optionally, you can enable the **Deliver to Amazon CloudWatch Logs**, which is useful for troubleshooting. If you do, choose your log group to use it.

9.

Review your configuration. If it’s OK, click **Create cluster**. Wait 15-45 minutes for your cluster to activate.

10.

After the cluster becomes active, select **Edit public access** and turn it on, then save the changes. The cluster will be updated. This might take up to 40 minutes.

### Update cluster policy

After the cluster has been updated and became active, copy and paste this policy.

```json
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<coralogix_role_arn>"
        ]
      },
      "Action": "kafka-cluster:Connect",
      "Resource": "< your_msk_cluster_arn >"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "< coralogix_role_arn >"
        ]
      },
      "Action": [
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": "arn:aws:kafka:< aws_region >:< aws_account_id >:topic/< your_cluster_name >/*"
    }
  ]
```

Replace the following parameters with your values:

- **your_aws_account_id** - Your AWS account ID

- **aws_region** - The AWS region that you deployed the MSK in

- **your_cluster_name** - The cluster name

- **your_msk_cluster_arn** - Your MSK cluster ARN

- **aws_account_id** - Your AWS account ID

- **coralogix_role_arn** - Copy the role from the following map:

  - EU1 - `arn:aws:iam::625240141681:role/coralogix-archive-eu1`
  - EU2 - `arn:aws:iam::625240141681:role/coralogix-archive-eu2`
  - US1 - `arn:aws:iam::625240141681:role/coralogix-archive-us1`
  - US2 - `arn:aws:iam::739076534691:role/coralogix-archive-us2`
  - AP1 - `arn:aws:iam::625240141681:role/coralogix-archive-ap1`
  - AP2 - `arn:aws:iam::625240141681:role/coralogix-archive-ap2`

### Additional settings

- To create a new topic, connect to a machine that is in the same VPC as the MSK. Alternatively, connect to a machine with the AWS configuration for the account with the MSK. Once you are in the machine, run the following commands:

  - `sudo yum install -y java-1.8*`
  - `wget <https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-3.5.1.tgz> && tar -xzvf kafka_2.13-3.5.1.tgz && rm kafka_2.13-3.5.1.tgz`
  - `kafka_2.13-3.5.1/bin/kafka-topics.sh --create --bootstrap-server <public brokerst list > --topic <Topic-Name> --partitions 3 --replication-factor 3`

- To restrict Coralogix access only to this topic, edit the cluster policy by changing the topic resource (`Resource": "arn:aws:kafka:<region>:<Account-ID>:topic*"`) to `"arn:aws:kafka:<REGION>:<ACCOUNT_ID>:topic/<CLUSTER_NAME>/<TOPIC_NAME>"`

- Extract public endpoint for brokers.

## Additional resources

|               |                                                                                                                |
| ------------- | -------------------------------------------------------------------------------------------------------------- |
| Documentation | [Forwarders](https://coralogix.com/docs/user-guides/data-flow/forwarders/index.md)                             |
| External      | [Getting started using Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html) |

## Support

**Need help?**

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us **via our in-app chat** or by sending us an email to [support@coralogix.com](mailto:support@coralogix.com).