Skip to main content

AWS Kinesis Firehose Integration to Coralogix

This template can be used to deploy an AWS Kinesis Firehose Integration to send resource metrics to Coralogix.

For a more detailed description of the settigns and architecture of this AWS Kinesis Data Firehose setup, please refer to the Coralogix documentation on AWS Kinesis Data Firehose – Metrics.

Prerequisites​

  • AWS account.
  • Coralogix account.

Main Parameters​

ParameterDescriptionDefault ValueRequired
CoralogixRegionThe region of your Coralogix Account. If set to Custom, you must provide a CustomDomain otherwise url will be invalid.Allowed Values:
- Custom
- EU1
- EU2
- AP1
- AP2
- AP3
- US1
- US2
- US3
Default: Custom
✅
CustomDomainThe Custom Coralogix domain. If set, will be the domain to send telemetry.
ApiKeyYour Coralogix Private Key. Required unless ApiKeySecretArn is set.✅ (unless using ApiKeySecretArn)
ApiKeySecretArnARN of a Secrets Manager secret holding the Coralogix API key as JSON {"api_key": "..."}. When set, Firehose reads the key at runtime so rotating the secret takes effect without a stack update. Must be in the same region as the delivery stream. See AWS Firehose Secrets Manager.✅ (unless using ApiKey)
ApiKeyKmsKeyArnARN of the KMS key used to encrypt the Secrets Manager secret. Required only when the secret uses a customer-managed key (CMK).
ApplicationNameYour Coralogix Application name
SubsystemNameYour Coralogix Subsystem name

Metrics Stream Parameters​

ParameterDescriptionDefault ValueRequired
IntegrationTypeMetricsThe data structure of the Firehose delivery stream for metrics. For CloudWatch_Metrics_OpenTelemetry070_WithAggregations, additional aggregations here are _min, _max, _avg recorded as gauges. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-translation.htmlAllowed Values:
- CloudWatch_Metrics_OpenTelemetry070
- CloudWatch_Metrics_OpenTelemetry070_WithAggregations
Default: CloudWatch_Metrics_OpenTelemetry070_WithAggregations
OutputFormatThe output format of the cloudwatch metric streamAllowed Values:
- opentelemetry0.7
- json
Default: opentelemetry0.7
IncludeNamespacesA string comma-delimited list of namespaces to include to the metric stream
e.g. AWS/EC2,AWS/EKS,AWS/ELB,AWS/Logs,AWS/S3
IncludeNamespacesMetricNamesA string json list of namespaces and metric_names to include to the metric stream. JSON stringify the input to avoid format errors.
e.g. {"AWS/EC2":["CPUUtilization","NetworkOut"],"AWS/S3":["BucketSizeBytes"]}
AddtionalStatisticsConfigurationsA json list of additional statistics to include to the metric stream following MetricStream StatisticsConfiguration.
JSON stringify the input to avoid format errors.
"p50","p75","p95","p99" of the following
- AWS/EBS:[VolumeTotalReadTime,VolumeTotalWriteTime]
- AWS/ELB:[Latency,Duration],
- AWS/Lambda:[PostRuntimeExtensionsDuration]
- AWS/S3:[FirstByteLatency,TotalRequestLatency]
EnableMetricsTagsProcessorsEnable the lambda metrics tags processor function. Set to false to remove the processortrue
LambdaProcessorVersionVersion of the Lambda metrics tags processor to deploy (0.1.18 or later), from the write-once firehose-metrics-transformer-<version>.zip. Changing it updates the function code on stack update. See releases.0.1.18
IncludeLinkedAccountsMetricsEnable cross-account observability to include metrics from linked source accounts (requires CloudWatch OAM setup between monitoring and source accounts)false
CrossAccountEnabledEnable Lambda cross-account tag enrichment. When true, Lambda assumes per-account roles from CrossAccountRoles.false
CrossAccountRolesJSON map of source account IDs to role ARNs used for tag enrichment. Example: {"123456789123":"arn:aws:iam::123456789123:role/CoralogixMetricsReader"}{}
DerivedLabelsJSON array of rules (target, sources, exclude_values, drop_values) that add the target label, copied from the first resource tag in sources that exists, isn't empty and isn't in exclude_values. Example: [{"target":"cost_center","sources":["CostCenter","Department"],"exclude_values":["none"]}]. Optional drop_values drops data points where any final label with the target key matches (case-insensitive), for example [{"target":"tenant","sources":["Solution","Environment"],"exclude_values":["dev"],"drop_values":["tenant-a","tenant-b"]}]. See the notes below. Requires EnableMetricsTagsProcessors. Empty disables it. See the Lambda README.

Optional Parameters​

ParameterDescriptionDefault ValueRequired
CloudwatchRetentionDaysDays of retention in Cloudwatch retention days1

Notes:​

  • Provide either ApiKey or ApiKeySecretArn (or both; the secret wins). ApiKeySecretArn requires a JSON secret {"api_key": "..."} — a different format from plaintext secrets used with ApiKey or CloudFormation {{resolve:secretsmanager:...}} dynamic references.

  • Switching an existing stack from ApiKey to ApiKeySecretArn replaces the Firehose delivery stream (the stream name becomes auto-generated). Update any CloudWatch Metric Streams that target the previous stream ARN after the update.

  • When ApiKeySecretArn is set, the Coralogix integration status notifier is omitted (it cannot read the runtime secret).

  • For cross-account enrichment, each linked account role (for example CoralogixMetricsReader) must trust the monitoring account Lambda role and allow sts:AssumeRole:

    • Principal: arn:aws:iam::<monitoring-account-id>:role/<stack-name>-lambda
    • Action: sts:AssumeRole
  • drop_values in DerivedLabels requires processor v0.1.17 or later; the default LambdaProcessorVersion (0.1.18) supports it. If an existing stack runs an older processor, first update it to this template with your current DerivedLabels, wait for the update to complete, then add drop_values in a separate update: code and configuration aren't updated atomically, so the old code could receive drop_values and forward the data. Older processors forward data without filtering: v0.1.16 rejects the entire DERIVED_LABELS configuration when this field is present, which also disables derived labels, and v0.1.15 and earlier ignore DERIVED_LABELS. Filtering uses cached resource tags, so until the cache refreshes (FILE_CACHE_EXPIRATION Lambda environment variable, default one hour) new resources and tag changes are filtered using stale or missing labels, which can forward or drop data incorrectly. All Lambda environment variables together are limited to 4 KB (AWS limit), so a large drop_values list can fail the stack update. See Dropping data points for matching rules, S3 backup behaviour and limitations.

    To upgrade the processor, update the stack with a newer LambdaProcessorVersion. The function code is updated in place.

  • LambdaMetricsTagsProcessors lambda function code is deployed to the following s3 regions: [ us-east-1 us-east-2 us-west-1 us-west-2 ap-south-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1 ca-central-1 eu-central-1 eu-west-1 eu-west-2 eu-west-3 eu-north-1 eu-south-1 sa-east-1 ]. If you are using a different region, please contact Coralogix support.

Deploy the Cloudformation template using aws cli​

With the aws cli installed and configured, run the following command:

aws cloudformation create-stack --stack-name <stack_name> --template-body template.yaml --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameter-overrides ApiKey=<coralogix_api_key> CoralogixRegion=<region> ApplicationName=<application_name> SubsystemName=<subsystem_name> EnableLogsStream=<true/false> EnableMetricsStream=<true/false>

or with a parameters json file example:

aws cloudformation create-stack --stack-name <stack_name> --template-body template.yaml --parameters parameters.json --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM
Last updated on