Amazon Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for retrieving data from multiple AWS services.
Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in DevOps. The platform enables you to easily explore and analyze logs to gain deeper insights into the state of your applications and AWS infrastructure. Analyze all of your AWS service logs while storing only those you need. Generate metrics from aggregated logs to uncover and alert on trends in your AWS services.
Streamline your setup process using our automated integration depending on the integration type (e.g. CloudWatch, WAF). The selected package lets you preconfigure and deploy a template, replicating the manual setup
Choose one of the integration types based on the format of the logs that are being sent:
Default
and RawText
types. Its generic nature allows you to select the source of your logs flexibly. You can also define the application and subsystem names as dynamic extractions during the setup, accommodating various AWS resources and services.STEP 1. From your Coralogix toolbar, navigate to Data Flow > Integrations.
STEP 2. In the Integrations section, search “Firehose” and select one of the following:
STEP 3. Click ADD NEW.
STEP 4. Input your integration details.
integrationType
parameter value. For extracting the applicationName
and subsystemName
values dynamically, an explanation is in the Manual Setup: Step 4 and a reference table in the Dynamic Values Table section of this document.STEP 6. Click NEXT.
STEP 7. Review the instructions for your integration. Click CREATE CLOUDFORMATION.
STEP 8. You will be rerouted to the AWS website. Verify that all of the auto pre-populated values are correct. Click Create Stack.
STEP 9. Return to the Coralogix application, where you will find instructions on configuring the log delivery from the selected input source to AWS Kinesis Data Firehose (if relevant).
Notes:
If you provide a Kinesis Stream ARN, Coralogix assumes that the data is in the stream and does not provide any additional instructions. It is the user’s responsibility to deliver data to the stream. In place of the instructions, you will see a message that prompts the user to confirm the integration.
STEP 10. Click COMPLETE to close the module.
STEP 11. [Optional] Deploy the extension package of your choice to complement your integration needs. We offer the following extensions for data originating from CloudTrail and WAF:
STEP 12. View the logs by navigating to Explore > Logs in your Coralogix toolbar. Find out more here.
STEP 1. Navigate to the Kinesis Data Firehose console and choose ‘Create delivery stream’.
STEP 2. Under ‘Choose source and destination’:
STEP 3. Scroll down to ‘Destination settings’:
STEP 4. Scroll down to ‘Parameters’. This section allows you to add and configure additional parameters surrounding the Coralogix platform.
The following parameters are available:
Parameter | Description |
---|---|
applicationName | A comma-separated list of application name sources |
applicationNameDefault | Deprecated |
subsystemName | A comma-separated list of subsystem name sources |
subsystemNameDefault | Deprecated |
integrationType | Data structure: – CloudWatch_JSON: Data from CloudWatch log groups – WAF: CloudWatch_CloudTrail – EksFargate – Default – RawText: Use for VPC flow logs |
dynamicMetadata | Deprecated |
A name source can be a literal string (something), a quoted string (“something”), or a field reference (e.g. ${logGroup}
). Sources in a list are evaluated in order, variables without a value are skipped. For example, ${applicationName}, MyApp
for a Default integration will use applicationName
field if available, otherwise it will default to MyApp.
The Dynamic Values Table section in this document serves as a reference for setting the applicationName and subsystemName dynamically.
Notes:
CloudWatch_JSON
.CloudWatch_CloudTrail
.EksFargate
.Default
.RawText
. This moves all the text to text
field of log, adds severity of Info
, and generates a current timestamp. All further parsing of these logs should be done using parsing rules.WAF
. This requires configuration on WAF as follows:Default
integration type is selected.Default
, the logs should be structured according to our REST API rules.STEP 5. Set up a recovery bucket (recommended). Enabling source data backup ensures that the data can be recovered if the record processing transformation does not produce the desired results.
STEP 6. Scroll down to ‘Backup settings’:
STEP 7. Review your settings and select Create delivery stream.
Logs subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.
Below are a couple of ways to connect your data source to Firehose.
To send your logs to Coralogix, create a subscription filter inside your CloudWatch log group.
First, create a new role in IAM for your Cloudwatch log group to allow sending data to Firehose.
Go to the IAM console and choose ‘Roles’ under ‘Access management’.
Click on ‘Create role’ on the right.
Under ‘Trusted entity type’ choose ‘Custom trust policy’ and insert this policy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "logs.<region_code>.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
Note: Change <region_code>
into your AWS region. e.g us-east-1
In ‘Add permissions’, click ‘Create policy’.
On the opened window, click on the ‘JSON’ tab and insert this policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:PutRecordBatch" ], "Resource": [ "<firehose_ARN>" ] } ] }
Note: Change <firehose_ARN>
to your Firehose Amazon Resource Name (ARN).
After creating the policy, go back to the role creation page and click on the refresh button.
Pick the newly created policy.
Give your role a name and create it.
After creating the role, go to the Cloudwatch console and choose ‘Logs groups’ under the ‘Logs’ side menu.
Create a new subscription filter for the relevant log group – <Your_log_group>
-> ‘Subscription filters’ -> ‘Create Kinesis Firehose subscription filter’.
Under ‘Choose destination’:
Scroll down to ‘Grant permission’:
Scroll down and click on ‘Start streaming’.
Logs coming to your Cloudwatch log group will also be directed to Firehose.
Note: Use the correct integration type inside your Firehose configuration.
To start sending your Kinesis data stream logs to Coralogix, connect the Data stream to Firehose.
Go to the Kinesis Data Stream console and choose ‘Create data stream’.
Under ‘Data stream configuration’:
Scroll down to ‘Data stream capacity’:
After that scroll down and click on ‘Create data stream’.
Note: to connect a kinesis data stream to a firehose delivery stream the delivery stream must use Amazon kinesis data streams
as its source instead of Direct PUT
.
Using Coralogix Terraform modules, you can easily install and manage Coralogix integrations with AWS services as modules in your infrastructure code. Our open-source modules are available on our GitHub and in the Terraform Registry. Visit our full AWS Kinesis Firehose Terraform Module documentation for more info.
For logs, install AWS Kinesis Data Firehose by adding this declaration to your Terraform project:
module "cloudwatch_firehose_coralogix_logs" { source = "coralogix/aws/coralogix//modules/firehose-logs" private_key = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX" firehose_stream = "coralogix-firehose-logs" coralogix_region = "Europe" integration_type_logs = "Default" source_type_logs = "DirectPut" }
For application name and/or subsystem name to be set dynamically in relation to their integrationType
‘s resource fields (e.g. CloudWatch_JSON’s loggroup name, EksFargate’s k8s namespace). The source’s variable can be mapped as a string literal with pre-defined values:
Field | Source Var | Expected Parameter | Integration Type | Notes |
---|---|---|---|---|
applicationName field in logs | applicationName | ${applicationName} | Default | Needs to be supplied in the log to be used |
subsystemName field in logs | subsystemName | ${subsystemName} | Default | Needs to be supplied in the log to be used |
CloudWatch LogGroup name | logGroup | ${logGroup} | CloudWatch_JSONCloudWatch_CloudTrail | Supplied by AWS |
kubernetes.namespace_name field | kubernetesNamespaceName | ${kubernetesNamespaceName} | EksFargate | Supplied by the default configuration |
kubernetes.container_name field | kubernetesContainerName | ${kubernetesContainerName} | EksFargate | Supplied by the default configuration |
name part of the log.webaclId field | webAclName | ${webAclName} | WAF | Supplied by AWS |
The expected parameter for the field must be exact with no extra characters. Characters may, however, be exchanged – for instance, subsystemName
parameter with value ${applicationName}
or )
Note: RawText
integrationType does not support dynamic values.
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Contact us via our in-app chat or by emailing [email protected].