AWS Shipper Terraform Module for MSK Integration

Overview

Changed in v4.3.5

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

MSK configuration

Name	Description	Type	Default	Required
msk_cluster_arn	The ARN of the MSK cluster to subscribe to retrieving messages.	string	n/a	yes
msk_topic_name	List of The Kafka topic names used to store records in your Kafka cluster ["topic-name1" ,"topic-name2"].	list of strings	n/a	yes

Configuration example

module "coralogix-shipper-msk" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region  = "EU1"
  integration_type  = "MSK"
  api_key           = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name  = "msk-application"
  subsystem_name    = "msk-subsystem"
  msk_cluster_arn   = "msk-cluster-arn"
  msk_topic_name    = "msk-topic-name"
}

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add ingress. to the domain.	string	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	string	S3	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key. Required for metrics, REST logs, and direct Coralogix OTLP; leave empty for Collector OTLP.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
log_export_protocol	Log delivery protocol when telemetry_mode is logs: coralogix_rest (default) or otlp_grpc. Ignored for metrics.	string	coralogix_rest	no
otlp_endpoint	Optional Collector http:// or https:// origin for otlp_grpc. Empty selects direct Coralogix OTLP; non-empty selects unauthenticated Collector delivery.	string	""	no
disable_log_severity_detection	Disable keyword-based severity detection for logs. When true, logs use protocol-level Info severity without modifying the original log body. Ignored for metrics.	bool	false	no
application_name	The name of your application. For a dynamic value, use $.my_log.field. This option is not supported since version 1.1.0 for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use $.my_log.field for CloudWatch log group leave empty. This option is not supported since version 1.1.0 for the source code	string	n\a	yes

When telemetry_mode is logs, three export routes are available (matching the shipper CloudFormation template):

1. Coralogix REST (default): log_export_protocol = "coralogix_rest".
2. Direct Coralogix OTLP/gRPC: log_export_protocol = "otlp_grpc" with empty otlp_endpoint (uses coralogix_region/custom_domain and api_key).
3. Collector OTLP/gRPC: log_export_protocol = "otlp_grpc" with a non-empty otlp_endpoint (no Coralogix API key).

OTLP routes require a shipper binary that supports them; pin source_code_version after that shipper release is published.

Traces

telemetry_mode = "traces" forwards AWS CloudWatch Transaction Search spans from the aws/spans log group to Coralogix as traces over OTLP/gRPC. It gives trace coverage for AWS-managed services that cannot be instrumented directly - Step Functions, API Gateway, AppSync - with no application code changes.

Requires shipper 1.4.16 or later - a pinned source_code_version below that is refused, since earlier binaries do not recognise the mode and ship spans as log lines. Enable Transaction Search and X-Ray tracing on the services you want traced, then:

telemetry_mode   = "traces"
integration_type = "CloudWatch"
log_groups       = ["aws/spans"]

As with logs, an empty otlp_endpoint sends direct to Coralogix and a non-empty one sends through a Collector, which is what makes traces work from a lambda in a private subnet.

Traces mode fails at plan time on a non-CloudWatch integration_type, log_groups other than ["aws/spans"], an S3, SNS, Kinesis, Kafka or MSK trigger, enable_dlq, or direct delivery without an api_key or without a custom_domain when coralogix_region = "Custom". integration_info is not supported - traces has a single log group, so configure it with the top-level variables.

Running in a VPC is supported. With an empty otlp_endpoint the lambda resolves the public ingress.<domain>, so the subnets need egress to it - a NAT gateway or an equivalent route. Otherwise set otlp_endpoint to a Collector reachable from the VPC.

Known limitation: AWS records service.name only on the root span of a trace, so child spans arrive unnamed. Traces render correctly - structure, timings, errors and stacktraces are intact - but service-level views are incomplete. See the shipper README.

For additional configuration options, please refer to our main Terraform documentation.