AWS Shipper Terraform Module for S3 Integration

Overview

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

S3, CloudTrail, Vpc Flow, S3Csv configuration

Name	Description	Type	Default	Required
s3_bucket_name	Comma separated list of the names for the S3 bucket to watch.	`string`	n/a	yes
s3_key_prefix	The S3 path prefix to watch.	`string`	n/a	no
s3_key_suffix	The S3 path suffix to watch.	`string`	n/a	no
s3_bucket_kms_arn	The AWS ARN of the KMS key used to encrypt/decrypt objects in the specified S3 bucket. If provided, the Lambda policy will include permissions to decrypt using this key.	`string`	n/a	no
csv_delimiter	A single character to be used as a delimiter when ingesting a CSV file with a header line. This value is applicable when the S3Csv integration type is selected, for example, “,” or ” “.	`string`	n/a	no
custom_csv_header	List seperated by cs delimiter of a new headers for your csv, the variable must be with the same delimiter as the cs_delimiter. For example, if the cs_delimiter is \";\", then the variable vakue should be name;country;age, resulting in the following new headers in Corlaogix: name, country, age	`string`	n/a	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	`string`	n/a	no
integration_info	A map of integration information. Use this to deploy multiple integrations using the same s3 bucket. See the integration parameters.	`mapping`	n/a	no

Additional parameters for `integration_info`

Name	Description	Type	Default	Required
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, S3Csv, CloudFront.	`string`	n/a	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret mManager ARN containing the API key.	`string`	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
application_name	The name of your application. For dynamic values, refer to Metadata	string	n\a	yes
subsystem_name	The name of your subsystem. For dynamic values, refer to Metadata	string	n\a	yes
lambda_log_retention	The CloudWatch log retention period (in days) for logs generated by the Lambda function.	`number`	5	no
lambda_name	Name of the Lambda function that you want to create.	`string`	n/a	no
s3_key_prefix	The S3 path prefix to watch.	`string`	n/a	no
s3_key_suffix	The S3 path suffix to watch.	`string`	n/a`	no
newline_pattern	A regular expression to detect a new log line for multiline logs, e.g., \n(?=\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}).	`string`	n/a	no

Configuration examples

S3 (default)

provider "aws" {}

module "coralogix-shipper-s3" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
}

CloudTrail-SNS

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail-sns"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "The name of your sns topic"
}

S3Csv

module "coralogix-shipper-S3Csv" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3Csv"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "S3Csv"
  subsystem_name     = "logs"
  s3_bucket_name     = "test-bucket-name"
  cs_delimiter       = ","
}

S3-SNS

In this example, you deploy the S3 integration via SNS and set the subsystem to the value of a log field. For instance, if we send this log:

In this example, the value of the subsystem will be set to "Subsystem name.”

{
    timestamp: "2024-01-01T00:00:01Z"
    massage: "log massage"
    dynamic:
      field: "Subsystem name"
}

module "coralogix-shipper-sns" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "S3"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "s3-sns"
  subsystem_name     = "$.dynamic.field"
  s3_bucket_name     = "test-bucket-name"
  sns_topic_name     = "test-sns-topic-name"
}

When you set the subsystem to $.eventSource, the subsystem value will be populated with the name of your Trail.

module "coralogix-shipper-cloudtrail" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "CloudTrail"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "cloudtrail"
  subsystem_name     = "$.eventSource"
  s3_bucket_name     = "test-bucket-name"
}

Vpc Flow

module "coralogix-shipper-vpcflow" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  integration_type   = "VpcFlow"
  api_key            = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
  application_name   = "vpcflow-application"
  subsystem_name     = "vpcflow-subsystem"
  s3_bucket_name     = "test-bucket-name"
}

Multiple simultaneous S3 integrations using the `integration_info` variable

This example illustrates creation of the following Lambda functions:

CloudTrail integration
VPC Flow integration
S3 integration with a prefix

module "coralogix-shipper-multiple-s3-integrations" {
  source = "coralogix/aws/coralogix//modules/coralogix-aws-shipper"

  coralogix_region   = "EU1"
  s3_bucket_name     = "bucket name"
  integration_info = {
    "CloudTrail_integration" = {
      integration_type = "CloudTrail"
      application_name = "CloudTrail_application"
      subsystem_name   = "logs_from_cloudtrail"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "VpcFlow_integration" = {
      integration_type = "VpcFlow"
      application_name = "VpcFlow_application"
      subsystem_name   = "logs_from_vpcflow"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
    "S3_integration" = {
      integration_type = "S3"
      application_name = "s3_application"
      subsystem_name   = "s3_vpcflow"
      s3_key_prefix    = "s3_prefix"
      api_key          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX"
    }
  }
}

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [`EU1`, `EU2`, `AP1`, `AP2`, `AP3`, `US1`, `US2`, `Custom`]	`string`	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add `ingress.` to the domain.	`string`	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	`string`	`S3`	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key.	`string`	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
application_name	The name of your application. For a dynamic value, use `$.my_log.field`. This option is not supported since version `1.1.0` for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use `$.my_log.field` for CloudWatch log group leave empty. This option is not supported since version `1.1.0` for the source code	string	n\a	yes

For additional configuration options, please refer to our main Terraform documentation.

MSK

Next SNS