In addition to our TCO Optimizer, Coralogix now offers an extra layer of granularity for your data retention. Using our new ARCHIVE RETENTION feature, you can now control and modify the length of time your logs are archived.
When creating a new policy for your data retention in your TCO Quota Optimizer screen, you now have the option of defining a lifecycle policy – the length of archive retention for a specific group of logs, defined by application, subsystem and severity.
Users may choose between 4 default retention periods: default, short, intermediate, or long. The names and values for the latter three are subjective and determined by the tag names set by the user in his/her s3 bucket configuration. While one user may define a “short” retention period as 3 days, another may define this period as 15 days. Another user may choose to change the name of the “short” retention period to “minimum”.
Note:
STEP 1. Create an s3 bucket for configuration.
STEP 2. Configure GetObjectTagging
and PutObjectTagging
permissions.
{ "Version": "2012-10-17", "Id": "MyPolicyID", "Statement": [ { "Sid": "MyStatementSid", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::625240141681:root" }, "Action": [ "s3:GetObject", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", "s3:GetObjectTagging" ], "Resource": [ "arn:aws:s3:::<bucket name>", "arn:aws:s3:::<bucket name>/*" ] } ] }
STEP 3. Set up a Lifecycle configuration.
Note:
The following section demonstrates one method of creating a lifecycle policy using AWS CLI. The example below sets a policy to remove archive files with the retention “short” after 15 days.
STEP 1. Define a policy in a local lifecycle.json file.
{ "Rules": [ { "Filter": { "Tag": { "Key": "CORALOGIX_ARCHIVE_RETENTION", "Value": "Short" } }, "Status": "Enabled", "Expiration": { "Days": 15 }, "ID": "ExampleRuleShort" }, { "Filter": { "Tag": { "Key": "CORALOGIX_ARCHIVE_RETENTION", "Value": "Intermediate" } }, "Status": "Enabled", "Expiration": { "Days": 150 }, "ID": "ExampleRuleIntermediate" }, { "Filter": { "Tag": { "Key": "CORALOGIX_ARCHIVE_RETENTION", "Value": "Long" } }, "Status": "Enabled", "Expiration": { "Days": 365 }, "ID": "ExampleRuleLong" } ] }
STEP 2. Apply the policy.
aws s3api put-bucket-lifecycle-configuration --bucket <bucket-name> --lifecycle-configurationfile://lifecycle.json
Note:
get-bucket-lifecycle-configuration
STEP 3. Verify that the policy has been applied.
aws s3api get-bucket-lifecycle-configuration --bucket <bucket-name>
Once you configure your cx-data bucket in S3, set your Archive Retention settings.
STEP 1. In your Coralogix navigation bar, click Data Flow > select Setup Archive.
STEP 2. In the Archive Retention section, name Retention Periods 2, 3, and 4. You may opt for names ‘Short’, ‘Intermediate’, and ‘Long’ as in the example below, or you may choose otherwise. The value for each period – the length of time data will be retained in a specific Retention Period – is managed by the s3 storage lifecycle defined in your AWS account.
STEP 3. Click ACTIVATE. You will receive a popup message that reads: “An archive retention policy has been added to the <bucket name> Bucket successfully.”
Notes:
STEP 4. View your changes by navigating to Data Flow > TCO Quota Optimizer.
Once you create your Archive Retention settings, create a new data retention policy.
STEP 1. In your Coralogix navigation bar, click Data Flow > select TCO Quota Optimizer.
STEP 2. Click +ADD NEW POLICY.
STEP 3. Input POLICY NAME.
STEP 4. Define APPLICATIONS, SUBSYSTEMS and SEVERITY level.
STEP 5. Define PRIORITY.
Notes:
STEP 6. Define ARCHIVE RETENTION settings.
Notes:
Files created before the new lifecycle policy have no retention tag and are not changed retroactively when you establish a new retention policy. The data retention policy that applied to them beforehand – whether default or defined – will continue to apply even after your new settings are put in place.
{ "Rules": [ { "Status": "Enabled", "Expiration": { "Days": 365 }, "ID": "ExampleRule" } ] }
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].