AWS Shipper Terraform Module for S3 Integration

Overview

Changed in v4.3.5

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

S3, CloudTrail, Vpc Flow, S3Csv configuration

Name	Description	Type	Default	Required
s3_bucket_name	Comma separated list of the names for the S3 bucket to watch.	string	n/a	yes
s3_key_prefix	The S3 path prefix to watch.	string	n/a	no
s3_key_suffix	The S3 path suffix to watch.	string	n/a	no
s3_bucket_kms_arn	The AWS ARN of the KMS key used to encrypt/decrypt objects in the specified S3 bucket. If provided, the Lambda policy will include permissions to decrypt using this key.	string	n/a	no
s3_notification	Controls whether an aws_s3_bucket_notification resource should be created to send S3 events to the application-specific Lambda functions. Set to false to disable the notification.	bool	true	no
csv_delimiter	A single character to be used as a delimiter when ingesting a CSV file with a header line. This value is applicable when the S3Csv integration type is selected, for example, “,” or ” “.	string	n/a	no
custom_csv_header	List separated by cs delimiter of a new headers for your csv, the variable must be with the same delimiter as the cs_delimiter. For example, if the cs_delimiter is ";", then the variable value should be name;country;age, resulting in the following new headers in Coralogix: name, country, age	string	n/a	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	string	n/a	no
integration_info	A map of integration information. Use this to deploy multiple integrations using the same s3 bucket. See the integration parameters.	mapping	n/a	no

Additional parameters for integration_info

Name	Description	Type	Default	Required
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, S3Csv, CloudFront.	string	n/a	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret mManager ARN containing the API key.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
application_name	The name of your application. For dynamic values, refer to Metadata	string	n\a	yes
subsystem_name	The name of your subsystem. For dynamic values, refer to Metadata	string	n\a	yes
lambda_log_retention	The CloudWatch log retention period (in days) for logs generated by the Lambda function.	number	5	no
lambda_name	Name of the Lambda function that you want to create.	string	n/a	no
s3_key_prefix	The S3 path prefix to watch.	string	n/a	no
s3_key_suffix	The S3 path suffix to watch.	string	n/a`	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	string	n/a	no

Configuration examples

S3 (default)

provider "aws" {}

module "coralogix-shipper-s3" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
}

CloudTrail-SNS

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail-sns"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "The name of your sns topic"
}

CloudTrail with Custom SNS Policy

module "coralogix-shipper-cloudtrail-custom-policy" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region        = "EU1"
  integration_type        = "CloudTrail"
  api_key                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name        = "cloudtrail-custom-policy"
  subsystem_name          = "logs"
  s3_bucket_name          = "test-bucket-name"
  sns_topic_name          = "your-existing-sns-topic"
  create_sns_topic_policy = false  # Preserve existing SNS topic policy
}

S3Csv

module "coralogix-shipper-S3Csv" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3Csv"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "S3Csv"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  cs_delimiter       = ","
}

S3-SNS

• In this example, you deploy the S3 integration via SNS and set the subsystem to the value of a log field. For instance, if we send this log:
• In this example, the value of the subsystem will be set to "Subsystem name.”

{
    timestamp: "2024-01-01T00:00:01Z"
    massage: "log massage"
    dynamic:
      field: "Subsystem name"
}

module "coralogix-shipper-sns" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3-sns"
  subsystem_name     = "$.dynamic.field"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "test-sns-topic-name"
}

CloudTrail-SNS with the dynamic subsystem name

• When you set the subsystem to $.eventSource, the subsystem value will be populated with the name of your Trail.

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail"
  subsystem_name     = "$.eventSource"
  s3_bucket_name     = "test-bucket-name"
}

Vpc Flow

module "coralogix-shipper-vpcflow" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "VpcFlow"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "vpcflow-application"
  subsystem_name     = "vpcflow-subsystem"
  s3_bucket_name     = "test-bucket-name"
}

Multiple simultaneous S3 integrations using the integration_info variable

This example illustrates creation of the following Lambda functions:

• CloudTrail integration
• VPC Flow integration
• S3 integration with a prefix

module "coralogix-shipper-multiple-s3-integrations" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  s3_bucket_name     = "bucket name"
  integration_info = {
    "CloudTrail_integration" = {
      integration_type = "CloudTrail"
      application_name = "CloudTrail_application"
      subsystem_name   = "logs_from_cloudtrail"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "VpcFlow_integration" = {
      integration_type = "VpcFlow"
      application_name = "VpcFlow_application"
      subsystem_name   = "logs_from_vpcflow"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "S3_integration" = {
      integration_type = "S3"
      application_name = "s3_application"
      subsystem_name   = "s3_vpcflow"
      s3_key_prefix    = "s3_prefix"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
  }
}

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add ingress. to the domain.	string	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	string	S3	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key. Required for metrics, REST logs, and direct Coralogix OTLP; leave empty for Collector OTLP.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
log_export_protocol	Log delivery protocol when telemetry_mode is logs: coralogix_rest (default) or otlp_grpc. Ignored for metrics.	string	coralogix_rest	no
otlp_endpoint	Optional Collector http:// or https:// origin for otlp_grpc. Empty selects direct Coralogix OTLP; non-empty selects unauthenticated Collector delivery.	string	""	no
otlp_headers	Optional semicolon-separated key=value pairs sent as gRPC metadata on every export to otlp_endpoint, e.g. x-api-key=value;tenant=name. Collector route only; the direct Coralogix routes ignore it. Requires shipper 1.5.0 or later. Sensitive.	string	""	no
disable_log_severity_detection	Disable keyword-based severity detection for logs. When true, logs use protocol-level Info severity without modifying the original log body. Ignored for metrics.	bool	false	no
application_name	The name of your application. For a dynamic value, use $.my_log.field. This option is not supported since version 1.1.0 for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use $.my_log.field for CloudWatch log group leave empty. This option is not supported since version 1.1.0 for the source code	string	n\a	yes

When telemetry_mode is logs, three export routes are available (matching the shipper CloudFormation template):

1. Coralogix REST (default): log_export_protocol = "coralogix_rest".
2. Direct Coralogix OTLP/gRPC: log_export_protocol = "otlp_grpc" with empty otlp_endpoint (uses coralogix_region/custom_domain and api_key).
3. Collector OTLP/gRPC: log_export_protocol = "otlp_grpc" with a non-empty otlp_endpoint (no Coralogix API key).

OTLP routes require a shipper binary that supports them; pin source_code_version after that shipper release is published.

For a gateway in front of the Collector that authenticates with its own header, set otlp_headers (shipper 1.5.0 or later). It applies to Collector logs and traces only, and plan fails if source_code_version is pinned below 1.5.0 while it is in use. Keys must be lowercase a-z, 0-9, _, ., - and values printable ASCII; the shipper rejects anything else at startup. The variable is sensitive, which hides it from plan output, but the value is still stored in plaintext in Terraform state and in the Lambda's environment variables, so protect state accordingly.

Traces

telemetry_mode = "traces" forwards AWS CloudWatch Transaction Search spans from the aws/spans log group to Coralogix as traces over OTLP/gRPC. It gives trace coverage for AWS-managed services that cannot be instrumented directly - Step Functions, API Gateway, AppSync - with no application code changes.

Requires shipper 1.4.16 or later - a pinned source_code_version below that is refused, since earlier binaries do not recognise the mode and ship spans as log lines. Enable Transaction Search and X-Ray tracing on the services you want traced, then:

telemetry_mode   = "traces"
integration_type = "CloudWatch"
log_groups       = ["aws/spans"]

As with logs, an empty otlp_endpoint sends direct to Coralogix and a non-empty one sends through a Collector, which is what makes traces work from a lambda in a private subnet.

Traces mode fails at plan time on a non-CloudWatch integration_type, log_groups other than ["aws/spans"], an S3, SNS, Kinesis, Kafka or MSK trigger, enable_dlq, or direct delivery without an api_key or without a custom_domain when coralogix_region = "Custom". integration_info is not supported - traces has a single log group, so configure it with the top-level variables.

Running in a VPC is supported. With an empty otlp_endpoint the lambda resolves the public ingress.<domain>, so the subnets need egress to it - a NAT gateway or an equivalent route. Otherwise set otlp_endpoint to a Collector reachable from the VPC.

Known limitation: AWS records service.name only on the root span of a trace, so child spans arrive unnamed. Traces render correctly - structure, timings, errors and stacktraces are intact - but service-level views are incomplete. See the shipper README.

For additional configuration options, please refer to our main Terraform documentation.