Archive Retention Policy
In addition to our TCO Optimizer, Coralogix now offers an extra layer of granularity for your data retention. Using our new ARCHIVE RETENTION feature, you can now control and modify the length of time your logs are archived.
Overview
When creating a new policy for your data retention in your TCO Quota Optimizer screen, you now have the option of defining a lifecycle policy - the length of archive retention for a specific group of logs, defined by application, subsystem and severity.
Users may choose between 4 default retention periods: default, short, intermediate, or long. The names and values for the latter three are subjective and determined by the tag names set by the user in his/her s3 bucket configuration. While one user may define a "short" retention period as 3 days, another may define this period as 15 days. Another user may choose to change the name of the "short" retention period to "minimum".
Note:
Only newly archived files are affected by these new settings.
Files created before the new lifecycle policy have no retention tag and are not changed retroactively. The data retention policy that applied to them before the new lifecycle policy - default or defined - will continue to apply even after the new settings are put in place.
Configuration
Configure GetObjectTagging
and PutObjectTagging
permissions.
Search S3 in your AWS search bar and select this service.
Find and click the bucket of choice for storing the archive.
Navigate to the Permissions tab. Edit the Bucket policy.
- Paste the following code and update the name of your bucket:
{
"Version": "2012-10-17",
"Id": "MyPolicyID",
"Statement": [
{
"Sid": "MyStatementSid",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::625240141681:root"
},
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObjectTagging"
],
"Resource": [
"arn:aws:s3:::<bucket name>",
"arn:aws:s3:::<bucket name>/*"
]
}
]
}
- Click Save.
Set up a Lifecycle configuration.
- Navigate to Lifecycle configuration.
- Define Lifecycle rule name and at least one Lifecycle rule action.
- Select Limit the scope of this rule using one or more filters.
- Add Object tags by defining a Key/Value pair.
- Click Save.
Note:
- The value of your object tag must be identical in content and form (case-sensitivity) to the archive retention name in your Coralogix Archive Retention Settings.
Create a Lifecycle Policy
The following section demonstrates one method of creating a lifecycle policy using AWS CLI. The example below sets a policy to remove archive files with the retention "short" after 15 days.
Define a policy in a local lifecycle.json file.
{
"Rules": [
{
"Filter": {
"Tag": {
"Key": "CORALOGIX_ARCHIVE_RETENTION",
"Value": "Short"
}
},
"Status": "Enabled",
"Expiration": {
"Days": 15
},
"ID": "ExampleRuleShort"
},
{
"Filter": {
"Tag": {
"Key": "CORALOGIX_ARCHIVE_RETENTION",
"Value": "Intermediate"
}
},
"Status": "Enabled",
"Expiration": {
"Days": 150
},
"ID": "ExampleRuleIntermediate"
},
{
"Filter": {
"Tag": {
"Key": "CORALOGIX_ARCHIVE_RETENTION",
"Value": "Long"
}
},
"Status": "Enabled",
"Expiration": {
"Days": 365
},
"ID": "ExampleRuleLong"
}
]
}
Apply the policy.
aws s3api put-bucket-lifecycle-configuration --bucket <bucket-name> --lifecycle-configurationfile://lifecycle.json
Note:
- This command will completely override the current policy of the bucket, if one exists.
- If you wish to append your existing policy, proceed with this command first and only then update lifecycle.json accordingly (STEP 2 above).
Set Archive Retention Settings
Once you configure your cx-data bucket in S3, set your Archive Retention settings.
In your Coralogix navigation bar, click Data Flow > select Setup Archive.
In the Archive Retention section, name Retention Periods 2, 3, and 4. You may opt for names ‘Short’, ‘Intermediate’, and ‘Long’ as in the example below, or you may choose otherwise. The value for each period - the length of time data will be retained in a specific Retention Period - is managed by the s3 storage lifecycle defined in your AWS account.
Click ACTIVATE. You will receive a popup message that reads: "An archive retention policy has been added to the bucket-name
Bucket successfully."
Notes:
Only once you have activated your archive retention settings will they appear in your TCO Quota Optimizer.
When modifying existing archive retention settings, the ACTIVATE button will be replaced with a SAVE button.
View your changes by navigating to Data Flow > TCO Quota Optimizer.
Create a New Policy
Once you create your Archive Retention settings, create a new data retention policy.
In your Coralogix navigation bar, click Data Flow > select TCO Quota Optimizer.
Click +ADD NEW POLICY.
Input POLICY NAME.
Define APPLICATIONS, SUBSYSTEMS and SEVERITY level.
Define PRIORITY. Notes:
- If data is marked as ‘blocked’, it will not be archived.
- Only data marked as priority ‘High’, ‘Medium’, or ‘Low’ is archived. Only this data is eligible for additional archive retention settings.
Define ARCHIVE RETENTION settings. Notes:
- When you add an archive retention policy, logs meeting the established criteria (application, subsystem, and severity level) will be retained in your archive for the period of time associated with the policy.
- If you do not specify your retention policy, logs meeting the established criteria will be retained in your archive for the default period of time.
Previously Archived Files
Files created before the new lifecycle policy have no retention tag and are not changed retroactively when you establish a new retention policy. The data retention policy that applied to them beforehand - whether default or defined - will continue to apply even after your new settings are put in place.
Additional Resources
Support
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].