Configure an S3 Bucket

This tutorial demonstrates configuring an S3 bucket to send your telemetry data to Coralogix.

Overview

Use your archive buckets to store all of your data as long as you need, on your own S3 bucket, for infinite retention. Your logs & traces archive bucket can be stored in CX data format. Metrics archive bucket stores Prometheus index blocks.

There are three configuration options:

• CloudFormation

• Terraform module

• Manual

S3 bucket usage across teams

Coralogix supports configuring a shared S3 bucket for multiple teams to store logs or metrics, with internal mechanisms to ensure data partitioning.

Metrics

• Multiple teams can upload metrics to the same S3 bucket.
• Data is partitioned by companyId, ensuring isolation.
• The companyId is specified in the metadata (meta.json) file accompanying each folder.

Logs & traces

• Logs from multiple teams can also be stored in the same S3 bucket.
• Each team writes to a separate directory path within the bucket, ensuring data separation.

A directory path is built as follows:

s3://BUCKET_NAME/PREFIX/OBJECT.suffix

Team "0" example:

s3://my-bucket-name/cx/parquet/v1/team_id=0/dt=2025-06-29/hr=07/my-object-name.parquet

Team "1" example:

s3://my-bucket-name/cx/parquet/v1/team_id=1/dt=2025-06-29/hr=07/my-object-name.parquet

Configuration with CloudFormation

Terraform

terraform init

terraform apply

Manual configuration

{
    "Version": "2012-10-17",
    "Id": "MyPolicyID",
    "Statement": [
        {
            "Sid": "MyStatementSid",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/coralogix-archive-<REGION>"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObjectTagging",
                "s3:DeleteObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>",
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

KMS encrypted bucket (optional)

If your bucket needs to be encrypted and you are using KMS, you will need to grant Coralogix permission to use the specific key to encrypt and decrypt the data when we store it on your bucket. Only those keys that are ‘Customer Managed’ may have their policy modified.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable Coralogix Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/coralogix-archive-<REGION>"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Enable root Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<your-user>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
     ]
}

Coralogix platform setup

Changing your S3 bucket destination

To update your S3 bucket destination, modify the bucket name in your platform setup.

When you update the S3 bucket destination used for storing your data, there is no disruption to data availability or query functionality. The system automatically tracks the change and ensures continuity:

• Historical data remains accessible even after the bucket switch.

• New data is written to the new bucket, starting from the moment of the change.

• When querying across a time range that spans both buckets, the system automatically pulls data from both sources as needed—transparently. It has no effect on the query you write.

Avoiding excess S3 costs

Avoid excess S3 costs when using Coralogix direct S3 queries.

Overview of Amazon S3 infrequent access storage class

Amazon S3 Infrequent Access storage class is ideally suited for data that isn’t accessed frequently but requires quick accessibility when needed. This storage class offers a cost-effective solution for storing such data. Coralogix customers must understand the specific cost implications of accessing data within this tier.

Cost considerations for Coralogix and Amazon S3 Infrequent Access

For applications utilizing Coralogix in conjunction with Amazon S3's Infrequent Access storage, it’s critical to be aware that accessing data stored in this tier might lead to additional costs, especially for querying the data from the Coralogix UI. This is due to the unique pricing structure Amazon S3 applies to data in the infrequent access category.

Recommended action

To ensure optimal data storage management and avoid unexpected costs, we advise all of our customers to review AWS S3 pricing and familiarize themselves with the details of AWS S3 pricing, especially the sections on the Infrequent Access storage class. Using any tier other than S3 Standard may incur additional AWS costs.

Previous Forwarders

Next Archive Retention Policy

Theme

Light

light

dark