AWS Shipper Terraform Module for S3 Integration

Overview

Changed in v4.3.5

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

S3, CloudTrail, Vpc Flow, S3Csv configuration

Name	Description	Type	Default	Required
s3_bucket_name	Comma separated list of the names for the S3 bucket to watch.	string	n/a	yes
s3_key_prefix	The S3 path prefix to watch.	string	n/a	no
s3_key_suffix	The S3 path suffix to watch.	string	n/a	no
s3_bucket_kms_arn	The AWS ARN of the KMS key used to encrypt/decrypt objects in the specified S3 bucket. If provided, the Lambda policy will include permissions to decrypt using this key.	string	n/a	no
s3_notification	Controls whether an aws_s3_bucket_notification resource should be created to send S3 events to the application-specific Lambda functions. Set to false to disable the notification.	bool	true	no
csv_delimiter	A single character to be used as a delimiter when ingesting a CSV file with a header line. This value is applicable when the S3Csv integration type is selected, for example, “,” or ” “.	string	n/a	no
custom_csv_header	List separated by cs delimiter of a new headers for your csv, the variable must be with the same delimiter as the cs_delimiter. For example, if the cs_delimiter is ";", then the variable value should be name;country;age, resulting in the following new headers in Coralogix: name, country, age	string	n/a	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	string	n/a	no
integration_info	A map of integration information. Use this to deploy multiple integrations using the same s3 bucket. See the integration parameters.	mapping	n/a	no

Additional parameters for integration_info

Name	Description	Type	Default	Required
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, S3Csv, CloudFront.	string	n/a	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret mManager ARN containing the API key.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
application_name	The name of your application. For dynamic values, refer to Metadata	string	n\a	yes
subsystem_name	The name of your subsystem. For dynamic values, refer to Metadata	string	n\a	yes
lambda_log_retention	The CloudWatch log retention period (in days) for logs generated by the Lambda function.	number	5	no
lambda_name	Name of the Lambda function that you want to create.	string	n/a	no
s3_key_prefix	The S3 path prefix to watch.	string	n/a	no
s3_key_suffix	The S3 path suffix to watch.	string	n/a`	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	string	n/a	no

Configuration examples

S3 (default)

provider "aws" {}

module "coralogix-shipper-s3" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
}

CloudTrail-SNS

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail-sns"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "The name of your sns topic"
}

CloudTrail with Custom SNS Policy

module "coralogix-shipper-cloudtrail-custom-policy" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region        = "EU1"
  integration_type        = "CloudTrail"
  api_key                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name        = "cloudtrail-custom-policy"
  subsystem_name          = "logs"
  s3_bucket_name          = "test-bucket-name"
  sns_topic_name          = "your-existing-sns-topic"
  create_sns_topic_policy = false  # Preserve existing SNS topic policy
}

S3Csv

module "coralogix-shipper-S3Csv" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3Csv"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "S3Csv"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  cs_delimiter       = ","
}

S3-SNS

• In this example, you deploy the S3 integration via SNS and set the subsystem to the value of a log field. For instance, if we send this log:
• In this example, the value of the subsystem will be set to "Subsystem name.”

{
    timestamp: "2024-01-01T00:00:01Z"
    massage: "log massage"
    dynamic:
      field: "Subsystem name"
}

module "coralogix-shipper-sns" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3-sns"
  subsystem_name     = "$.dynamic.field"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "test-sns-topic-name"
}

CloudTrail-SNS with the dynamic subsystem name

• When you set the subsystem to $.eventSource, the subsystem value will be populated with the name of your Trail.

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail"
  subsystem_name     = "$.eventSource"
  s3_bucket_name     = "test-bucket-name"
}

Vpc Flow

module "coralogix-shipper-vpcflow" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "VpcFlow"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "vpcflow-application"
  subsystem_name     = "vpcflow-subsystem"
  s3_bucket_name     = "test-bucket-name"
}

Multiple simultaneous S3 integrations using the integration_info variable

This example illustrates creation of the following Lambda functions:

• CloudTrail integration
• VPC Flow integration
• S3 integration with a prefix

module "coralogix-shipper-multiple-s3-integrations" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  s3_bucket_name     = "bucket name"
  integration_info = {
    "CloudTrail_integration" = {
      integration_type = "CloudTrail"
      application_name = "CloudTrail_application"
      subsystem_name   = "logs_from_cloudtrail"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "VpcFlow_integration" = {
      integration_type = "VpcFlow"
      application_name = "VpcFlow_application"
      subsystem_name   = "logs_from_vpcflow"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "S3_integration" = {
      integration_type = "S3"
      application_name = "s3_application"
      subsystem_name   = "s3_vpcflow"
      s3_key_prefix    = "s3_prefix"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
  }
}

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add ingress. to the domain.	string	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	string	S3	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key. Required for metrics, REST logs, and direct Coralogix OTLP; leave empty for Collector OTLP.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
log_export_protocol	Log delivery protocol when telemetry_mode is logs: coralogix_rest (default) or otlp_grpc. Ignored for metrics.	string	coralogix_rest	no
otlp_endpoint	Optional Collector http:// or https:// origin for otlp_grpc. Empty selects direct Coralogix OTLP; non-empty selects unauthenticated Collector delivery.	string	""	no
disable_log_severity_detection	Disable keyword-based severity detection for logs. When true, logs use protocol-level Info severity without modifying the original log body. Ignored for metrics.	bool	false	no
application_name	The name of your application. For a dynamic value, use $.my_log.field. This option is not supported since version 1.1.0 for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use $.my_log.field for CloudWatch log group leave empty. This option is not supported since version 1.1.0 for the source code	string	n\a	yes

When telemetry_mode is logs, three export routes are available (matching the shipper CloudFormation template):

1. Coralogix REST (default): log_export_protocol = "coralogix_rest".
2. Direct Coralogix OTLP/gRPC: log_export_protocol = "otlp_grpc" with empty otlp_endpoint (uses coralogix_region/custom_domain and api_key).
3. Collector OTLP/gRPC: log_export_protocol = "otlp_grpc" with a non-empty otlp_endpoint (no Coralogix API key).

OTLP routes require a shipper binary that supports them; pin source_code_version after that shipper release is published.

For additional configuration options, please refer to our main Terraform documentation.