AWS Kinesis Firehose Integration to Coralogix
This template can be used to deploy an AWS Kinesis Firehose Integration to send resource metrics to Coralogix.
For a more detailed description of the settigns and architecture of this AWS Kinesis Data Firehose setup, please refer to the Coralogix documentation on AWS Kinesis Data Firehose – Metrics.
Prerequisites
- AWS account.
- Coralogix account.
Main Parameters
| Parameter | Description | Default Value | Required |
|---|---|---|---|
| CoralogixRegion | The region of your Coralogix Account. If set to Custom, you must provide a CustomDomain otherwise url will be invalid. | Allowed Values: - Custom - EU1 - EU2 - AP1 - AP2 - AP3 - US1 - US2 - US3 Default: Custom | ✅ |
| CustomDomain | The Custom Coralogix domain. If set, will be the domain to send telemetry. | ||
| ApiKey | Your Coralogix Private Key. Required unless ApiKeySecretArn is set. | ✅ (unless using ApiKeySecretArn) | |
| ApiKeySecretArn | ARN of a Secrets Manager secret holding the Coralogix API key as JSON {"api_key": "..."}. When set, Firehose reads the key at runtime so rotating the secret takes effect without a stack update. Must be in the same region as the delivery stream. See AWS Firehose Secrets Manager. | ✅ (unless using ApiKey) | |
| ApiKeyKmsKeyArn | ARN of the KMS key used to encrypt the Secrets Manager secret. Required only when the secret uses a customer-managed key (CMK). | ||
| ApplicationName | Your Coralogix Application name | ||
| SubsystemName | Your Coralogix Subsystem name |
Metrics Stream Parameters
| Parameter | Description | Default Value | Required |
|---|---|---|---|
| IntegrationTypeMetrics | The data structure of the Firehose delivery stream for metrics. For CloudWatch_Metrics_OpenTelemetry070_WithAggregations, additional aggregations here are _min, _max, _avg recorded as gauges. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-translation.html | Allowed Values: - CloudWatch_Metrics_OpenTelemetry070 - CloudWatch_Metrics_OpenTelemetry070_WithAggregations Default: CloudWatch_Metrics_OpenTelemetry070_WithAggregations | |
| OutputFormat | The output format of the cloudwatch metric stream | Allowed Values: - opentelemetry0.7 - json Default: opentelemetry0.7 | |
| IncludeNamespaces | A string comma-delimited list of namespaces to include to the metric stream e.g. AWS/EC2,AWS/EKS,AWS/ELB,AWS/Logs,AWS/S3 | ||
| IncludeNamespacesMetricNames | A string json list of namespaces and metric_names to include to the metric stream. JSON stringify the input to avoid format errors. e.g. {"AWS/EC2":["CPUUtilization","NetworkOut"],"AWS/S3":["BucketSizeBytes"]} | ||
| AddtionalStatisticsConfigurations | A json list of additional statistics to include to the metric stream following MetricStream StatisticsConfiguration. JSON stringify the input to avoid format errors. | "p50","p75","p95","p99" of the following - AWS/EBS:[VolumeTotalReadTime,VolumeTotalWriteTime] - AWS/ELB:[Latency,Duration], - AWS/Lambda:[PostRuntimeExtensionsDuration] - AWS/S3:[FirstByteLatency,TotalRequestLatency] | |
| EnableMetricsTagsProcessors | Enable the lambda metrics tags processor function. Set to false to remove the processor | true | |
| LambdaProcessorVersion | Version of the Lambda metrics tags processor to deploy (0.1.18 or later), from the write-once firehose-metrics-transformer-<version>.zip. Changing it updates the function code on stack update. See releases. | 0.1.18 | |
| IncludeLinkedAccountsMetrics | Enable cross-account observability to include metrics from linked source accounts (requires CloudWatch OAM setup between monitoring and source accounts) | false | |
| CrossAccountEnabled | Enable Lambda cross-account tag enrichment. When true, Lambda assumes per-account roles from CrossAccountRoles. | false | |
| CrossAccountRoles | JSON map of source account IDs to role ARNs used for tag enrichment. Example: {"123456789123":"arn:aws:iam::123456789123:role/CoralogixMetricsReader"} | {} | |
| DerivedLabels | JSON array of rules (target, sources, exclude_values, drop_values) that add the target label, copied from the first resource tag in sources that exists, isn't empty and isn't in exclude_values. Example: [{"target":"cost_center","sources":["CostCenter","Department"],"exclude_values":["none"]}]. Optional drop_values drops data points where any final label with the target key matches (case-insensitive), for example [{"target":"tenant","sources":["Solution","Environment"],"exclude_values":["dev"],"drop_values":["tenant-a","tenant-b"]}]. See the notes below. Requires EnableMetricsTagsProcessors. Empty disables it. See the Lambda README. |
Optional Parameters
| Parameter | Description | Default Value | Required |
|---|---|---|---|
| CloudwatchRetentionDays | Days of retention in Cloudwatch retention days | 1 |
Notes:
-
Provide either
ApiKeyorApiKeySecretArn(or both; the secret wins).ApiKeySecretArnrequires a JSON secret{"api_key": "..."}— a different format from plaintext secrets used withApiKeyor CloudFormation{{resolve:secretsmanager:...}}dynamic references. -
Switching an existing stack from
ApiKeytoApiKeySecretArnreplaces the Firehose delivery stream (the stream name becomes auto-generated). Update any CloudWatch Metric Streams that target the previous stream ARN after the update. -
When
ApiKeySecretArnis set, the Coralogix integration status notifier is omitted (it cannot read the runtime secret). -
For cross-account enrichment, each linked account role (for example
CoralogixMetricsReader) must trust the monitoring account Lambda role and allowsts:AssumeRole:- Principal:
arn:aws:iam::<monitoring-account-id>:role/<stack-name>-lambda - Action:
sts:AssumeRole
- Principal:
-
drop_valuesinDerivedLabelsrequires processor v0.1.17 or later; the defaultLambdaProcessorVersion(0.1.18) supports it. If an existing stack runs an older processor, first update it to this template with your currentDerivedLabels, wait for the update to complete, then adddrop_valuesin a separate update: code and configuration aren't updated atomically, so the old code could receivedrop_valuesand forward the data. Older processors forward data without filtering: v0.1.16 rejects the entireDERIVED_LABELSconfiguration when this field is present, which also disables derived labels, and v0.1.15 and earlier ignoreDERIVED_LABELS. Filtering uses cached resource tags, so until the cache refreshes (FILE_CACHE_EXPIRATIONLambda environment variable, default one hour) new resources and tag changes are filtered using stale or missing labels, which can forward or drop data incorrectly. All Lambda environment variables together are limited to 4 KB (AWS limit), so a largedrop_valueslist can fail the stack update. See Dropping data points for matching rules, S3 backup behaviour and limitations.To upgrade the processor, update the stack with a newer
LambdaProcessorVersion. The function code is updated in place. -
LambdaMetricsTagsProcessorslambda function code is deployed to the following s3 regions: [ us-east-1 us-east-2 us-west-1 us-west-2 ap-south-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1 ca-central-1 eu-central-1 eu-west-1 eu-west-2 eu-west-3 eu-north-1 eu-south-1 sa-east-1 ]. If you are using a different region, please contact Coralogix support.
Deploy the Cloudformation template using aws cli
With the aws cli installed and configured, run the following command:
aws cloudformation create-stack --stack-name <stack_name> --template-body template.yaml --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameter-overrides ApiKey=<coralogix_api_key> CoralogixRegion=<region> ApplicationName=<application_name> SubsystemName=<subsystem_name> EnableLogsStream=<true/false> EnableMetricsStream=<true/false>
or with a parameters json file example:
aws cloudformation create-stack --stack-name <stack_name> --template-body template.yaml --parameters parameters.json --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM