Coralogix provides a predefined function to forward your logs from Google Cloud Storage straight to Coralogix.
This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.
Elasticsearch-API | SSL Certificates | Cluster URL | |
---|---|---|---|
.com | https://coralogix-esapi.coralogix.com:9443 | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crt | coralogix.com |
.us | https://esapi.coralogix.us:9443 | https://www.amazontrust.com/repository/AmazonRootCA1.pem | coralogix.us |
.in | https://es-api.app.coralogix.in:9443 | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN.pem | app.coralogix.in |
.eu2. | https://es-api.eu2.coralogix.com:9443 | https://www.amazontrust.com/repository/AmazonRootCA1.pem | app.eu2.coralogix.com |
sg.com | https://es-api.coralogixsg.com:9443 | https://www.amazontrust.com/repository/AmazonRootCA1.pem | app.coralogixsg.com |
If your account top level domain is different than ‘.com’ make sure to set the following environment variables:
CORALOGIX_LOG_URL=https://api.Cluster URL/api/v1/logs
CORALOGIX_TIME_DELTA_URL=https://api.Cluster URL/sdk/v1/time
Create a Cloud Function in your Google Cloud Console with the following settings:
#!/usr/bin/python
# -*- coding: utf-8 -*-
"""
Coralogix GCP function for GCS
Author: Coralogix Ltd.
Email: [email protected]
"""
import os
import re
import sys
import gzip
import logging
from google.cloud import storage
from coralogix.handlers import CoralogixLogger
__name__ = 'gcsToCoralogix'
__author__ = 'Coralogix Ltd.'
__email__ = '[email protected]'
__copyright__ = 'Copyright 2021, Coralogix Ltd.'
__credits__ = ['Ariel Assaraf', 'Amnon Shahar', 'Eldar Aliiev']
__license__ = 'Apache Version 2.0'
__version__ = '1.0.0'
__maintainer__ = 'Coralogix Ltd.'
__date__ = '8 February 2021'
__status__ = 'Stable'
# Get function parameters
PRIVATE_KEY = os.environ.get('private_key')
APP_NAME = os.environ.get('app_name', 'NO_APPLICATION')
SUB_SYSTEM = os.environ.get('sub_name', 'NO_SUBSYSTEM')
NEWLINE_PATTERN = os.environ.get('newline_pattern', '(?:\r\n|\r|\n)')
# Function entrypoint
def to_coralogix(event, context):
"""
Function entrypoint
:param event: event metadata
:type event: dict
:param context: event context
:type context: dict
"""
def get_severity(message: str) -> int:
"""
Extract severity from message text
:param message: log record text
:type message: str
:return: severity value
:rtype: int
"""
severity = 3
if 'Warning' in message or 'warn' in message:
severity = 4
if 'Error' in message or 'Exception' in message:
severity = 5
return severity
# Initialize GCS client
client = storage.Client()
# Initialize Coralogix logger
logger = CoralogixLogger(
PRIVATE_KEY,
APP_NAME,
SUB_SYSTEM,
'GCS'
)
logging.info(f"Processing file {event['name']}")
# Get file content
bucket = client.get_bucket(event['bucket'])
blob = bucket.get_blob(event['name'])
content = blob.download_as_string()
# Check if file is compressed
if event['contentType'] == 'application/gzip' or \
event['name'].endswith('.gz'):
logging.info(f"Uncompress file {event['name']}")
try:
# Decompress file
content = gzip.decompress(content)
except Exception as exc:
logging.fatal(f"Cannot uncompress file {event['name']}: ", exc)
sys.exit(1)
# Split file into line and remove empty lines
logs = list(filter(None, re.split(NEWLINE_PATTERN, content.decode('utf-8'))))
logging.info(f"Number of logs: {len(logs)}")
# Send logs to Coralogix
for log in logs:
logger.log(
get_severity(log),
log,
thread_id=f"{event['bucket']}/{event['name']}"
)
7. Paste the following packages to requirements.txt:
coralogix_logger>=2.0.4
8. Increase “Timeout“ to “60 seconds“.
9. Add the mandatory environment variables: private_key, app_name, sub_name:
10. Multiline pattern: Coralogix supports multiline pattern by default, you can define a custom pattern with environment variables, for example:
newline_pattern [\s(?={)|(?<=})\s,\s(?={)|(?<=})\s\].
11. Click “Create”.
To set up the function, execute this:
$ curl -sSL -o gcsToCoralogix.zip https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/gcp/gcs/lambda/gcsToCoralogix.zip
$ unzip gcsToCoralogix.zip -d gcsToCoralogix/
$ gcloud functions deploy gcsToCoralogix \
--project=YOUR_GCP_PROJECT_ID \
--region=us-central1 \
--runtime=python38 \
--memory=1024MB \
--timeout=60s \
--entry-point=to_coralogix \
--source=gcsToCoralogix \
--trigger-resource=YOUR_BUCKET_NAME \
--trigger-event=google.storage.object.finalize \
--set-env-vars="private_key=YOUR_PRIVATE_KEY,app_name=APP_NAME,sub_name=SUB_NAME"
Here is the Terraform module to deploy the Cloud Function.
Add this module to your manifest and change its options:
provider "google" {
project = "YOUR_GCP_PROJECT_ID"
region = "us-central1"
}
module "gcs_to_coralogix" {
source = "git::https://github.com/coralogix/integrations-docs.git//integrations/gcp/gcs/terraform"
private_key = "YOUR_PRIVATE_KEY"
app_name = "APP_NAME"
sub_name = "SUB_NAME"
bucket_name = "YOUR_BUCKET_NAME"
}
Download the module and apply these changes:
$ terraform init
$ terraform apply