Firehose Metrics Module

Firehose Metrics module is designed to support AWS Firehose Metrics integration with Coralogix. Leveraging AWS CloudWatch Metrics Stream.

Metrics - Usage

Delivering all CloudWatch metrics

Provision a firehose delivery stream with CloudWatch metric stream to stream metrics to Coralogix.

module "cloudwatch_firehose_metrics_coralogix" {
  source           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream  = var.coralogix_firehose_stream_name
  api_key          = var.api_key
  coralogix_region = var.coralogix_region
}

By default, the metric stream includes all namespaces [AWS/EC2, AWS/EBS, etc..] and metric names.

Delivering selected CloudWatch metrics by namespaces

Provision a firehose delivery stream with CloudWatch metric stream. The metric stream includes only selected namespaces and sends the metrics to Coralogix: When including specific namespaces, the variable 'include_metric_stream_namespaces' needs to include a list of the desired namespaces, which are case-sensitive. please see the AWS namespaces list.

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key
  include_metric_stream_namespaces = var.include_metric_stream_namespaces
  coralogix_region                 = var.coralogix_region
}

Filtering selected metric names from namespaces

Provision a firehose delivery stream with CloudWatch metric stream. For more granular inclusive filters of metric names belonging to an included namespace:

The variable include_metric_stream_filter can be used to send only conditional metric names belonging to a selected metric namespace. For any selected namespace where the metric names list is empty or not specified, all metrics in that namespace is included.

Note: include_metric_stream_namespaces and include_metric_stream_filter are independent but related the same metric stream include filter and may conflict. If error or metrics do not show, check console CloudWatch -> Metrics -> Streams -> Selected Metrics table on result.

Metric namespaces are also case-sensitive, please see the AWS namespaces list. For case-sensitive metric names belonging to a namespace, please see the AWS View available metrics guide

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key

  # If metric names is empty or not specified, the whole metric namespace is included
  include_metric_stream_filter     = [
    {
      namespace    = "AWS/EC2"
      metric_names = ["CPUUtilization", "NetworkOut"]
    },
    {
      namespace    = "AWS/S3"
      metric_names = ["BucketSizeBytes"]
    },
  ]
  coralogix_region                 = var.coralogix_region
}

Additional Statistics

Also, additional_metric_statistics provide a means to configure additional statistics to a given metric. This is done by specifying the metric_name and namespace and corresponding list of additional statistics. Read metric streams for more infomation. Set additional_metric_statistics_enable to true to enable this featurem,

Depending on the output_format variable configured (default opentelemetry0.7). The json format would support streaming of statistics provided by CloudWatch and the opentelemetry0.7 (default) supports streaming percentile statistics (p99.).

If additional_metric_statistics is not configured but is enabled true, the module's default configuration of recommended metric and statistics is used which is configured to the p50, p75, p95 and p99 percentiles.

In the below example, additional_metric_statistics is enabled and the default configured metrics, namespaces and additional statistics percentiles are used. Note: as output_format of opentelemetry0.7 is configured, only percentile values are supported.

output_format = "opentelemetry0.7"

additional_metric_statistics_enable = true
additional_metric_statistics = [
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "VolumeTotalReadTime",
    namespace             = "AWS/EBS"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "VolumeTotalWriteTime",
    namespace             = "AWS/EBS"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "Latency",
    namespace             = "AWS/ELB"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "Duration",
    namespace             = "AWS/ELB"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "PostRuntimeExtensionsDuration",
    namespace             = "AWS/Lambda"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "FirstByteLatency",
    namespace             = "AWS/S3"
  },
  {
    additional_statistics = ["p50", "p75", "p95", "p99"],
    metric_name           = "TotalRequestLatency",
    namespace             = "AWS/S3"
  }
]

Cross-account Tag Enrichment (OAM)

When using OAM linked accounts, enable Lambda cross-account tag enrichment to fetch tags from source accounts:

## Enable cross-account tag enrichment
cross_account_enabled = true

## Map each source account ID to the role to assume in that account
cross_account_roles = {
  "123456789012" = "arn:aws:iam::123456789012:role/CoralogixMetricsReader"
}

Required permissions: - Monitoring account Lambda role must allow sts:AssumeRole on each role in cross_account_roles. - Each linked account role trust policy must allow the monitoring account Lambda processor role to assume it. - Linked account role policy should include at least tag:GetResources and service-specific read actions used for resource discovery.

Removal of CloudWatch Metric Streams Lambda transformation

By default, a Coralogix Lambda Transformation Function has been added to the Kinesis Firehose Data Transformation as a processing_configuration. This is done, to enrich the metrics from CloudWatch Metric Streams with AWS resource tags. The optional lambda function is deployed as part of the module, and can be removed by setting the variable lambda_processor_enable to false.

module "cloudwatch_firehose_metrics_coralogix" {
  source                           = "coralogix/aws/coralogix//modules/firehose-metrics"
  lambda_processor_enable          = false
  firehose_stream                  = var.coralogix_firehose_stream_name
  api_key                          = var.api_key
  coralogix_region                 = var.coralogix_region
}

Read more about the following:

• Statistics that can be streamed
• Metric streams output formats
• Statistical definitions

Examples

Examples can be found under the firehose-metrics examples directory

Override Coralogix applicationName and subsystemName

The application name and subsystem name by default is the firehose delivery stream arn and name, but it can be overriden by setting an environment variable called application_name and subsystem_name.

Coralogix account region

The coralogix region variable accepts one of the following regions: * EU1 * EU2 * AP1 * AP2 * AP3 * US1 * US2

Coralogix Regions & Description.

Custom endpoints

It is possible to pass a custom firehose ingress endpoint with by using the coralogix_firehose_custom_endpoint variable.

Metrics Output Format

Coralogix suppots both JSON format and OpenTelemtry format. The default format configured here is OpenTelemtry. if using Json in the firehose output format, which is configured via the integration_type_metrics variable, then the CloudWatch metric stream must be configured with the same format, configured via the output_format variable.

Requirements

Providers

Inputs

Coralgoix regions

Outputs