AWS Shipper Terraform Module for Kafka Integration

Overview

Changed in v4.3.5

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

Kafka configuration

Name	Description	Type	Default	Required
kafka_brokers	Comma-delimited list of Kafka broker to connect to.	string	n/a	yes
kafka_topic	The Kafka topic to subscribe to.	string	n/a	yes
kafka_subnets_ids	List of Kafka subnets to use when connecting to Kafka.	list	n/a	yes
kafka_security_groups	List of Kafka security groups to use when connecting to Kafka.	list	n/a	yes

Configuration example

module "coralogix-shipper-kafka" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region  = "EU1"
  integration_type  = "Kafka"
  api_key           = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name  = "kafka-application"
  subsystem_name    = "kafka-subsystem"
  kafka_brokers     = "kafka-broker-1,kafka-broker-2"
  kafka_topic       = "kafka-topic-name"
  kafka_subnets_ids = ["kafka-subnet-1", "kafka-subnet-2"]
  kafka_security_groups = ["kafka-security-group-1", "kafka-security-group-2"]
  ]
}

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add ingress. to the domain.	string	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	string	S3	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key. Required for metrics, REST logs, and direct Coralogix OTLP; leave empty for Collector OTLP.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
log_export_protocol	Log delivery protocol when telemetry_mode is logs: coralogix_rest (default) or otlp_grpc. Ignored for metrics.	string	coralogix_rest	no
otlp_endpoint	Optional Collector http:// or https:// origin for otlp_grpc. Empty selects direct Coralogix OTLP; non-empty selects unauthenticated Collector delivery.	string	""	no
disable_log_severity_detection	Disable keyword-based severity detection for logs. When true, logs use protocol-level Info severity without modifying the original log body. Ignored for metrics.	bool	false	no
application_name	The name of your application. For a dynamic value, use $.my_log.field. This option is not supported since version 1.1.0 for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use $.my_log.field for CloudWatch log group leave empty. This option is not supported since version 1.1.0 for the source code	string	n\a	yes

When telemetry_mode is logs, three export routes are available (matching the shipper CloudFormation template):

1. Coralogix REST (default): log_export_protocol = "coralogix_rest".
2. Direct Coralogix OTLP/gRPC: log_export_protocol = "otlp_grpc" with empty otlp_endpoint (uses coralogix_region/custom_domain and api_key).
3. Collector OTLP/gRPC: log_export_protocol = "otlp_grpc" with a non-empty otlp_endpoint (no Coralogix API key).

OTLP routes require a shipper binary that supports them; pin source_code_version after that shipper release is published.

For additional configuration options, please refer to our main Terraform documentation.