Breaking News from AWS re:Invent
Coralogix receives AWS Rising Star award!
This tutorial demonstrates how to configure an S3 bucket to send your telemetry data to Coralogix.
Use your archive buckets to store all of your data as long as you need, on your own S3 bucket, for infinite retention. Your logs & traces archive bucket can be stored in CX data format. Metrics archive bucket stores Prometheus index blocks.
Users have three configuration options:
STEP 1. Create a new stack.
STEP 2. Specify your stack details.
STEP 3. Run the template to create the S3 archives.
STEP 1. Access the S3-archive module in the Coralogix Terraform Registry.
STEP 2. Create the Terraform file with the following variables:
Variable | Description |
aws_region | The AWS region within which you want to create the S3 bucket * Must be identical to the AWS region where your Coralogix account is set |
logs_bucket_name | The name of the S3 bucket to create for the logs archive * Leave empty if not needed |
metrics_bucket_name | The name of the S3 bucket to create for the metrics archive * Leave empty if not needed |
Notes:
STEP 3. [Optional] Add KMS keys to the buckets with logs_kms_arn
and metrics_kms_arn
variables.
STEP 4. Create the archive buckets by saving the file and running the following commands:
terraform init terraform apply
STEP 1. Create an S3 bucket for configuration with default parameters.
STEP 2. Search S3 in your AWS search bar and select this service.
STEP 3. Locate and select the bucket of choice for storing the archive.
STEP 4. Navigate to the Permissions tab. Edit the Bucket policy.
STEP 5. Paste the following code and update the name of your bucket. The following provides both a logs and metrics archive bucket configuration snippet. Note that the configuration snippet for metrics contains an additional delete permission.
Notes:
REGION | ACCOUNT_ID | ACCOUNT_URL_Example |
---|---|---|
US1 (Ohio) | 625240141681 | <team>.app.coralogix.us |
US2 (Oregon) | 739076534691 | <team>.app.cx498.coralogix.com |
EU1 (Ireland) | 625240141681 | <team>.coralogix.com |
EU2 (Stockholm) | 625240141681 | <team>.app.eu2.coralogix.com |
API (Mumbai) | 625240141681 | <team>.app.coralogix.in |
AP2 (Singapore) | 625240141681 | <team>.app.coralogixsg.com |
{ "Version": "2012-10-17", "Id": "MyPolicyID", "Statement": [ { "Sid": "MyStatementSid", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<Account_ID>:root" }, "Action": [ "s3:GetObject", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", "s3:GetObjectTagging" ], "Resource": [ "arn:aws:s3:::<bucket name>", "arn:aws:s3:::<bucket name>/*" ] } ] }
Note! For metric buckets, you are required to add the following value to the code above:”s3:DeleteObject”.
{ "Version": "2012-10-17", "Id": "MyPolicyID", "Statement": [ { "Sid": "MyStatementSid", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<Account_ID>:root" }, "Action": [ "s3:GetObject", "s3:ListBucket", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::<bucket name>", "arn:aws:s3:::<bucket name>/*" ] } ] }
STEP 6. Click Save changes.
STEP 7. Copy and save the name of the archive for your Coralogix platform setup.
If your bucket needs to be encrypted and you are using KMS, you will need to grant Coralogix permission to use the specific key to encrypt and decrypt the data when we store it on your bucket. Only those keys that are ‘Customer Managed’ may have their policy modified.
STEP 1. Access KMS and choose the key used for encryption. Note: The key must be in the same region as your archive bucket.
STEP 2. Edit the Key Policy and add the highlighted section below to your KMS key policy or copy the code in full to also give your root user full access.
{ "Version": "2012-10-17", "Id": "key-consolepolicy-3", "Statement": [ { "Sid": "Enable Coralogix Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<Account_ID>:root" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" }, { "Sid": "Enable root Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<your-user>:root" }, "Action": "kms:*", "Resource": "*" } ] }
STEP 3. Add the key to the S3 bucket. Navigate go to your archive bucket. Under Properties, edit the Default encryption.
STEP 4. Select Server-side encryption with AWS Key Management Service keys (SSE-KMS) and insert your KMS ARN. Enable the Bucket Key. Click Save changes.
Notes:
STEP 1. Select the Data Flow tab and click on Setup Archive.
STEP 2. Choose from two bucket options: one for logs & traces (CX data) and one for metrics. You cannot use the same bucket for metrics and logs together.
STEP 3. Click Save.
Notes:
STEP 4. Query your Metrics S3 bucket through our hosted Grafana metric index or using our plugin and inputting metrics_index.
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].