Firehose Metrics Module

Firehose Metrics module is designed to support AWS Firehose Metrics integration with Coralogix. Leveraging AWS CloudWatch Metrics Stream.

Metrics - Usage

Changed in v4.3.5

Delivering all CloudWatch metrics

Provision a firehose delivery stream with CloudWatch metric stream to stream metrics to Coralogix.

module "cloudwatch_firehose_metrics_coralogix" {
  source           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream  = var.coralogix_firehose_stream_name
  api_key          = var.api_key
  coralogix_region = var.coralogix_region
}

Secrets Manager API key

Optionally pass api_key_secret_arn instead of api_key so Firehose fetches the key at runtime via SecretsManagerConfiguration. The secret value must be JSON: {"api_key": "..."}. Rotating the secret then takes effect without a Terraform apply. If the secret is encrypted with a customer managed KMS key, also set api_key_secret_kms_key_arn. When the module creates the Firehose IAM role, it grants secretsmanager:GetSecretValue (and kms:Decrypt when a CMK ARN is set). If you pass existing_firehose_iam, that role must already allow those actions.

module "cloudwatch_firehose_metrics_coralogix" {
  source             = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream    = var.coralogix_firehose_stream_name
  api_key_secret_arn = var.api_key_secret_arn
  coralogix_region   = var.coralogix_region
}

By default, the metric stream includes all namespaces [AWS/EC2, AWS/EBS, etc..] and metric names.

Delivering selected CloudWatch metrics by namespaces

Provision a firehose delivery stream with CloudWatch metric stream. The metric stream includes only selected namespaces and sends the metrics to Coralogix: When including specific namespaces, the variable 'include_metric_stream_namespaces' needs to include a list of the desired namespaces, which are case-sensitive. See the AWS namespaces list.

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key
  include_metric_stream_namespaces = var.include_metric_stream_namespaces
  coralogix_region                 = var.coralogix_region
}

Filtering selected metric names from namespaces

Provision a firehose delivery stream with CloudWatch metric stream. For more granular inclusive filters of metric names belonging to an included namespace:

The variable include_metric_stream_filter can be used to send only conditional metric names belonging to a selected metric namespace. For any selected namespace where the metric names list is empty or not specified, all metrics in that namespace are included.

Note: include_metric_stream_namespaces and include_metric_stream_filter are independent but related the same metric stream include filter and may conflict. If you see errors or metrics don't appear, verify your selections in the CloudWatch console under Metrics then Streams then Selected Metrics.

Metric namespaces are also case-sensitive. See the AWS namespaces list. For case-sensitive metric names belonging to a namespace, see the AWS View available metrics guide.

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key
  
  # If metric names is empty or not specified, the whole metric namespace is included
  include_metric_stream_filter     = [
    {
      namespace    = "AWS/EC2"
      metric_names = ["CPUUtilization", "NetworkOut"]
    },
    {
      namespace    = "AWS/S3"
      metric_names = ["BucketSizeBytes"]
    },
  ]
  coralogix_region                 = var.coralogix_region
}

Additional Statistics

Also, additional_metric_statistics provide a means to configure additional statistics to a given metric. This is done by specifying the metric_name and namespace and corresponding list of additional statistics. Read metric streams for more information. Set additional_metric_statistics_enable to true to enable this feature.

Depending on the output_format variable configured (default opentelemetry0.7). The json format would support streaming of statistics provided by CloudWatch and the opentelemetry0.7 (default) supports streaming percentile statistics (p99.).

If additional_metric_statistics is not configured but is enabled true, the module's default configuration of recommended metric and statistics is used which is configured to the p50, p75, p95 and p99 percentiles.

In the below example, additional_metric_statistics is enabled and the default configured metrics, namespaces and additional statistics percentiles are used. Note: as output_format of opentelemetry0.7 is configured, only percentile values are supported.

output_format = "opentelemetry0.7"

additional_metric_statistics_enable = true
additional_metric_statistics = [
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "VolumeTotalReadTime",
    namespace             = "AWS/EBS"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "VolumeTotalWriteTime",
    namespace             = "AWS/EBS"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "Latency",
    namespace             = "AWS/ELB"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "Duration",
    namespace             = "AWS/ELB"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "PostRuntimeExtensionsDuration",
    namespace             = "AWS/Lambda"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "FirstByteLatency",
    namespace             = "AWS/S3"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "TotalRequestLatency",
    namespace             = "AWS/S3"
  }
]

Cross-account Tag Enrichment (OAM)

When using OAM linked accounts, enable Lambda cross-account tag enrichment to fetch tags from source accounts:

# Enable cross-account tag enrichment
cross_account_enabled = true

# Map each source account ID to the role to assume in that account
cross_account_roles = {
  "123456789012" = "arn:aws:iam::123456789012:role/CoralogixMetricsReader"
}

Required permissions:

Monitoring account Lambda role must allow sts:AssumeRole on each role in cross_account_roles.
Each linked account role trust policy must allow the monitoring account Lambda processor role to assume it.
Linked account role policy should include at least tag:GetResources and service-specific read actions used for resource discovery.

Derived Labels

Add a label copied from existing resource tags. Each rule sets target from the first tag in sources that exists, isn't empty and isn't in exclude_values (case-insensitive). Requires lambda_processor_enable (default true) and Lambda processor v0.1.16 or later.

derived_labels = [
  { target = "cost_center", sources = ["CostCenter", "Department"], exclude_values = ["none"] },
]

Removal of CloudWatch Metric Streams Lambda transformation

By default, a Coralogix Lambda Transformation Function has been added to the Kinesis Firehose Data Transformation as a processing_configuration. This is done, to enrich the metrics from CloudWatch Metric Streams with AWS resource tags. The optional lambda function is deployed as part of the module, and can be removed by setting the variable lambda_processor_enable to false.

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  lambda_processor_enable          = false
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key
  coralogix_region                 = var.coralogix_region
}

Examples

Examples can be found under the firehose-metrics examples directory

Override Coralogix `applicationName` and `subsystemName`

The application name and subsystem name by default is the firehose delivery stream arn and name, but it can be overridden by setting the application_name and subsystem_name variables.

Coralogix account region

The Coralogix region variable accepts one of the following regions:

Coralogix regions and endpoints

Region	Domain	Endpoint
EU1	`eu1.coralogix.com`	`https://ingress.eu1.coralogix.com/aws/firehose`
EU2	`eu2.coralogix.com`	`https://ingress.eu2.coralogix.com/aws/firehose`
AP1	`ap1.coralogix.com`	`https://ingress.ap1.coralogix.com/aws/firehose`
AP2	`ap2.coralogix.com`	`https://ingress.ap2.coralogix.com/aws/firehose`
AP3	`ap3.coralogix.com`	`https://ingress.ap3.coralogix.com/aws/firehose`
US1	`us1.coralogix.com`	`https://ingress.us1.coralogix.com/aws/firehose`
US2	`us2.coralogix.com`	`https://ingress.us2.coralogix.com/aws/firehose`

Custom endpoints

To use a custom domain, set the custom_domain variable (e.g. cust.coralogix-123.net:8443). The endpoint becomes https://ingress.<custom_domain>/aws/firehose. This does not work for PrivateLink.

Metrics output format

The CloudWatch metric stream format is set with output_format (opentelemetry0.7, the default, or json). The Firehose integration type is set with integration_type_metrics: CloudWatch_Metrics_OpenTelemetry070_WithAggregations (default) or CloudWatch_Metrics_OpenTelemetry070. Both integration types expect OpenTelemetry 0.7, so keep output_format at opentelemetry0.7.

{/* BEGIN_TF_DOCS */}

Requirements

Name	Version
terraform	>= 1.9.0
aws	>= 6.0

Providers

Name	Version
aws	>= 6.0
null	n/a
random	n/a

Inputs

Name	Description	Type	Default	Required
coralogix_region	Coralogix account region: EU1, EU2, AP1, AP2, AP3, US1, US2	`string`	n/a	yes
api_key	Coralogix account API key. Ignored when api_key_secret_arn is set. Required unless api_key_secret_arn is provided.	`string`	`null`	no
api_key_secret_arn	ARN of a Secrets Manager secret holding the Coralogix API key as JSON, {"api_key": "..."}. When set, Firehose reads the key at runtime and api_key is ignored, so rotating the secret takes effect without a Terraform apply. This must be a separate secret from any plaintext value used with api_key. Must be in the same region as the delivery stream.	`string`	`null`	no
api_key_secret_kms_key_arn	Optional ARN of the KMS key used to encrypt the Secrets Manager secret. Required only if the secret uses a customer managed key (CMK).	`string`	`null`	no
firehose_stream	AWS Kinesis firehose delivery stream name	`string`	n/a	yes
application_name	The name of your application in Coralogix	`string`	`null`	no
subsystem_name	The subsystem name of your application in Coralogix	`string`	`null`	no
cloudwatch_retention_days	Days of retention in Cloudwatch retention days	`number`	`1`	no
custom_domain	Custom domain for Coralogix firehose integration endpoints, does not work for privatelink (e.g. cust.coralogix-123.net:8443 for https://ingress.cust.coralogix-123.net:8443/aws/firehose)	`string`	`null`	no
integration_type_metrics	The integration type of the firehose delivery stream: 'CloudWatch_Metrics_OpenTelemetry070' or 'CloudWatch_Metrics_OpenTelemetry070_WithAggregations'	`string`	`"CloudWatch_Metrics_OpenTelemetry070_WithAggregations"`	no
output_format	The output format of the cloudwatch metric stream: 'json' or 'opentelemetry0.7'	`string`	`"opentelemetry0.7"`	no
enable_cloudwatch_metricstream	Should be true if you want to create a new Cloud Watch metric stream and attach it to Firehose	`bool`	`true`	no
cloudwatch_metric_stream_custom_name	Set the name of the CloudWatch metric stream, otherwise variable '{firehose_stream}-cw' will be used	`string`	`null`	no
include_metric_stream_namespaces	List of specific namespaces to include in the CloudWatch metric stream, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html	`list(string)`	`[]`	no
include_metric_stream_filter	List of inclusive metric filters for namespace and metric_names. Specify this parameter, the stream sends only the conditional metric names from the metric namespaces that you specify here. If metric names is empty or not specified, the whole metric namespace is included	list(object({ namespace = string metric_names = list(string) }) )	`[]`	no
include_linked_accounts_metrics	include_linked_accounts_metrics (Optional) If you are creating a metric stream in a monitoring account, specify true to include metrics from source accounts that are linked to this monitoring account, in the metric stream. The default is false.	`bool`	`false`	no
additional_metric_statistics_enable	To enable the inclusion of additional statistics to the streaming metrics	`bool`	`false`	no
additional_metric_statistics	For each entry, specify one or more metrics (metric_name and namespace) and the list of additional statistics to stream for those metrics. Each configuration of metric name and namespace can have a list of additional_statistics included into the AWS CloudWatch Metric Stream	list(object({ additional_statistics = list(string) metric_name = string namespace = string }))	[ { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "VolumeTotalReadTime", "namespace": "AWS/EBS" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "VolumeTotalWriteTime", "namespace": "AWS/EBS" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "Latency", "namespace": "AWS/ELB" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "Duration", "namespace": "AWS/ELB" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "PostRuntimeExtensionsDuration", "namespace": "AWS/Lambda" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "FirstByteLatency", "namespace": "AWS/S3" }, { "additional_statistics": [ "p50", "p75", "p95", "p99" ], "metric_name": "TotalRequestLatency", "namespace": "AWS/S3" } ]	no
s3_backup_custom_name	Set the name of the S3 backup bucket, otherwise variable '{firehose_stream}-backup-metrics' will be used	`string`	`null`	no
existing_s3_backup	Use an existing S3 bucket to use as a backup bucket	`string`	`null`	no
govcloud_deployment	Enable if you deploy the integration in govcloud	`bool`	`false`	no
custom_s3_bucket	Custom S3 bucket to use for the lambda processor	`string`	`null`	no
server_side_encryption	Server side encryption configuration	object({ enabled = bool key_type = optional(string) key_arn = optional(string) })	{ "enabled": false, "key_type": "AWS_OWNED_CMK" }	no
static_labels	List of key-value pairs that will be added as labels to every metric in the integration.	`list(string)`	`[]`	no
derived_labels	Rules that add a label copied from the first resource tag in `sources` that exists, isn't empty and isn't in `exclude_values` (case-insensitive). Passed to the Lambda processor as DERIVED_LABELS. Requires lambda_processor_enable.	list(object({ target = string sources = list(string) exclude_values = optional(list(string), []) }))	`[]`	no
cross_account_enabled	Enable cross-account resource tag enrichment for OAM. When true, the Lambda will assume roles in linked accounts to fetch resource tags. Requires cross_account_roles to be set.	`bool`	`false`	no
cross_account_roles	Map of AWS account IDs to IAM role ARNs for cross-account tag enrichment. Example: {"123456789012" = "arn:aws:iam::123456789012:role/CoralogixMetricsReader"}	`map(string)`	`{}`	no
lambda_processor_enable	Enable lambda processor function, defaults to true	`bool`	`true`	no
lambda_processor_custom_name	Set the name of the lambda processor function, otherwise variable '{firehose_stream}-metrics-transform' will be used	`string`	`null`	no
lambda_processor_iam_custom_name	Set the name of the lambda processor IAM role & policy, otherwise variable '{firehose_stream}-lambda-processor-iam' will be used	`string`	`null`	no
existing_lambda_processor_iam	Use an existing lambda processor IAM role	`string`	`null`	no
firehose_iam_custom_name	Set the name of the firehose IAM role & policy, otherwise variable '{firehose_stream}-firehose-metrics-iam' will be used	`string`	`null`	no
existing_firehose_iam	Use an existing IAM role to use as a firehose role	`string`	`null`	no
metric_streams_iam_custom_name	Set the name of the cloudwatch metric streams IAM role & policy, otherwise variable '{firehose_stream}-cw-iam' will be used	`string`	`null`	no
existing_metric_streams_iam	Use an existing IAM role to use as a metric streams role	`string`	`null`	no
user_supplied_tags	Tags supplied by the user to populate to all generated resources	`map(string)`	`{}`	no
override_default_tags	Override and remove the default tags by setting to true	`bool`	`false`	no

Note

To deploy in GovCloud, set custom_s3_bucket to a bucket that exists in your account. The module downloads the Lambda processor and uploads it to that bucket.

Outputs

Name	Description
firehose_stream_arn	ARN of the Firehose Delivery Stream
firehose_stream_name	Name of the Firehose Delivery Stream
firehose_iam_role_arn	ARN of the Firehose IAM role
s3_backup_bucket_arn	ARN of the Firehose S3 Backup Bucket
lambda_processor_arn	ARN of the Lambda Processor
lambda_processor_iam_arn	ARN of the Lambda Processor IAM role
metric_stream_arn	ARN of the CloudWatch Metric Stream
metrics_stream_iam_role_arn	ARN of the CloudWatch Metric Stream IAM role

{/* END_TF_DOCS */}

Last updated on April 19, 2026

Metrics - Usage​

Delivering all CloudWatch metrics​

Secrets Manager API key​

Delivering selected CloudWatch metrics by namespaces​

Filtering selected metric names from namespaces​

Additional Statistics​

Cross-account Tag Enrichment (OAM)​

Derived Labels​

Removal of CloudWatch Metric Streams Lambda transformation​

Examples​

Override Coralogix applicationName and subsystemName​

Coralogix account region​

Coralogix regions and endpoints​

Custom endpoints​

Metrics output format​

Requirements​

Providers​

Inputs​

Outputs​