Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Docs

Connect S3 Archive Connect S3 Archive

Last Updated: Sep. 18, 2022

This tutorial will guide you through the Creation of your S3 bucket and where to update it in the Coralogix system.
The archive bucket can store all your data as long as you need, on your own S3 bucket, for infinite retention. Logs archive bucket can be stored in 2 formats: CSV and/or CX-Data.

Setting up in Coralogix Platform

  1. Go to the Data Flow tab and click on Setup Archive.
  2. There are three bucket options – two for logs and one for metrics*. Add the name of the bucket and choose the bucket’s region.
  3. Click Save.
  4. Click on Validate Settings to make sure you have succeeded setting your archive.
  • To avoid additional data transfer cost the region of the buckets need to match your account region (mandatory for Metric bucket):
     Coralogix Team URLAWS region
    EU.coralogix.comeu-west-1 (Ireland)
    USA.app.coralogix.usus-east-2 (Ohio)
    IN.app.coralogix.inap-south-1 (Mumbai)
    EU2.app.eu2.coralogix.comeu-north-1 (Stockholm)
    SG.app.coralogixsg.comap-southeast-1 (Singapore)
  • Metric bucket must be different than the other buckets. You cannot use the same bucket for metrics and logs together.

S3 Bucket Configuration

Upon creation of the bucket there is no need to perform any change. Once you created the S3 bucket, please follow the below steps to configure the bucket:

  1. Under your AWS account in the search type S3.
    Choose S3 - aws consule
  2. Find and click on the bucket you want to use for storing the archive.
  3. Navigate to the Permissions tab, next Bucket policy and click Edit.
    S3 - Edit bucket policy
  4. Paste the following code and update the name of your bucket:
{
    "Version": "2012-10-17",
    "Id": "MyPolicyID",
    "Statement": [
        {
            "Sid": "MyStatementSid",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::625240141681:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>",
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

For Metric bucket please note that on top of the actions above, we also need "s3:DeleteObject"

  1. Click Save changes button.
  2. Copy the name of the archive to Coralogix (see above – “Setting up in Coralogix Platform”) and click on Validate Settings to make sure you have succeeded setting your archive.

For Metrics you can query your Metrics S3 bucket through our hosted Grafana pointing to metric index, or using our plugin https://coralogix.com/docs/grafana-plugin/ and adding metrics_index.

(Optional) KMS Encrypted bucket

In case data on your bucket need to be encrypted and you are using KMS, we will need also permission to use the specific key to encrypt and decrypt the data when we store it on your bucket.

Note: you can only change key policy of keys that are ‘Customer Managed’.

Please go to KMS and choose the key used for encryption. Edit the Key Policy and add the highlighted section below to your KMS key policy OR copy everything to also give your root user full access. This gives us permissions to use the key.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable Coralogix Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::625240141681:root"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Enable root Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<your-user>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
     ]
}

Note: on key policy, an asterisk “*” on resource field means the specific KMS Key and NOT all resources.

S3 Bucket Configuration Using ACLs – Deprecated

  1. Under your AWS account in the search type S3.
  2. Navigate to the Permissions and click Edit.
  1. In Block public access (bucket settings) click ‘Edit’. Uncheck Block all public access and then check all bottom three options, leaving only Block public access to buckets and objects granted through new access control lists (ACLs) unchecked. Click Save changes.
  1. Scroll back down to the Object Ownership section and click the Edit button. Make sure ACLs enabled, and Object writer are selected, then click Save changes:
  1. Still Under Permissions scroll down to Access control list (ACL) and click Edit.
  1. Click on Add grantee and Insert Coralogix canonical id: fa35ef450b07d311b09810445df9c1c4a316118d1899e4cd3db935414e4ba62d. Check the Write Objects checkbox to give Coralogix write objects permissions and click Save changes.
  1. In order to use Coralogix’s advanced Archive query feature, check the List Objects and Read Bucket ACL options. Click Save changes.

Read/write permissions on the bucket are mandatory for direct Archive Query to work.

  1. After we already added the bucket details in Coralogix (see step 1), all we need to do next is to validate our archive. Click on Validate Settings to make sure you have Succedded setting your archive.

Note:
If encryption with AWS Key Management Service key (SSE-KMS) is enabled on your S3 bucket, then it is required to add the bold statement to your key policy. Example:

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable Coralogix Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::625240141681:root"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Enable root Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<your-user>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
     ]
}

Note: on key policy, an asterisk “*” on resource field means the specific KMS Key and NOT all resources.

Metrics note:

You can query your Metrics S3 through our hosted Grafana pointing to metric index, or using our plugin https://coralogix.com/docs/grafana-plugin/ and adding metrics_index.

On this page