Amazon Kinesis Data Firehose delivers real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, or Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), and now supports delivering streaming data to Coralogix. There is no limit on the number of delivery streams, so it can be used for retrieving data from multiple AWS services.
Coralogix is an AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in DevOps. The platform enables you to easily explore and analyze logs to gain deeper insights into the state of your applications and AWS infrastructure. Analyze all of your AWS service logs while storing only those you need. Generate metrics from aggregated logs to uncover and alert on trends in your AWS services.
Using Coralogix with Amazon Kinesis Data Firehose offers significant benefits when compared with other solutions.
STEP 1. Navigate to the Kinesis Data Firehose console and choose ‘Create delivery stream’.
STEP 2. Under ‘Choose source and destination’:
STEP 3. Scroll down to ‘Destination settings’:
STEP 4. Scroll down to ‘Parameters’. This section allows you to add and configure additional parameters surrounding the Coralogix platform.
The following parameters are available:
Parameter | Description |
applicationName | Hard override of applicationName within the Coralogix Platform |
applicationNameDefault | Override of applicationName default |
subsystemName | Hard override of subsystemName within the Coralogix Platform |
subsystemNameDefault | Override of subsystemName default |
integrationType | Data structure: – CloudWatch_JSON: data from cloudWatch log groups – WAF – CloudWatch_CloudTrail – EksFargate – Default – RawText: use for VPC flow logs |
dynamicMetadata | When set to true, it fetches set the applicationName / subsystemName dynamically |
Notes:
text
field of log, adds severity of Info
, and generates a current timestamp. All further parsing of these logs should be done using parsing rules. Required | Property Name | Property Type | Notes |
---|---|---|---|
Yes | timestamp | number | UTC milliseconds since 1970 (supports sub millisecond via a floating point) |
Yes | severity | number | 1 – Debug, 2 – Verbose, 3 – Info, 4 – Warn, 5 – Error, 6 – Critical |
Yes | text | string | |
hiResTimestamp | string | UTC nanoseconds since 1970 (supports millisecond, microsecond and nanosecond) |
|
className | string | ||
methodName | string | ||
threadId | string | ||
category | string |
STEP 5. Set up a recovery bucket (recommended). Enabling source data backup ensures that the data can be recovered if record processing transformation does not produced the desired results.
STEP 6. Scroll down to ‘Backup settings’:
STEP 7. Review your settings and select Create delivery stream.
Logs subscribed to your delivery stream will be immediately sent and available for analysis within Coralogix.
For applicationName and subsystemName to be set dynamically, follow STEP 4 and set the ‘dynamicMetadata’ parameter to true
along with the ‘integrationType’ parameter (e.g. CloudWatch_JSON, EksFargate). Use the following dynamic values.
Type | Dynamic applicationName | Dynamic subsystemName | Notes |
CloudWatch_JSON | the cloudwatch log group | none | supplied by aws |
CloudWatch_CloudTrail | the cloudwatch log group | none | supplied by aws |
Default | ‘applicationName’ field | ‘subsystemName’ field | need to be supplied in the log to be used |
EksFargate | ‘kubernetes.namespace_name’ field | ‘kubernetes.container_name’ field | supplied by the default configuration |
WAF | The web acl name | none | supplied by aws |
Below are a couple of ways to connect your data source to firehose.
To start sending your logs to Coralogix you first need to create a subscription filter inside your Cloudwatch log group.
First, create a new role in IAM for your Cloudwatch log group to allow sending data to firehose.
Go to the IAM console and choose ‘Roles’ under ‘Access management’.
Click on ‘Create role’ on the right.
Under ‘Trusted entity type’ choose ‘Custom trust policy’ and insert this policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.<region_code>.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Note: change ‘<region_code>’ into your AWS region. e.g ‘us-east-1’
Now on ‘Add permissions’ click on ‘Create policy’.
on the opened window click on the ‘JSON’ tab and insert this policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": [
"<firehose_ARN>"
]
}
]
}
Note: change ‘<firehose_ARN>’ to your firehose amazon resource name (arn).
After creating the policy go back to the role creation page and click on the refresh button
Pick the newly created policy.
Finally, give your role a name and create it.
After creating the role, go to the Cloudwatch console and choose ‘Logs groups’ under the ‘Logs’ side menu.
Create a new subscription filter for the relevant log group – ‘<Your_log_group>’ -> ‘Subscription filters’ -> ‘Create Kinesis Firehose subscription filter’.
Under ‘Choose destination’:
Scroll down to ‘Grant permission’:
After that scroll down and click on ‘Start streaming’.
That’s it logs coming to your cloudwatch log group will also be directed to firehose.
Note: be sure to use the correct integration type inside your firehose configuration.
To start sending your Kinesis data stream logs to coralogix we need to connect the Data stream to Firehose.
Go to the Kinesis Data Stream console and choose ‘Create data stream’.
Under ‘Data stream configuration’:
Scroll down to ‘Data stream capacity’:
After that scroll down and click on ‘Create data stream’.
Note: to connect a kinesis data stream to a firehose delivery stream the delivery stream must use ‘Amazon kinesis data streams’ as its source instead of ‘Direct PUT’.
To get the most out of the platform, be sure to check out the documentation which will help you get started with everything from parsing and enrichment to alerting and data clustering.
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].