Google Cloud Storage

Coralogix provides a predefined function to forward your logs from Google Cloud Storage straight to Coralogix.

Setup

This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

If your account top level domain is different than ‘.com’ make sure to set the following environment variables:

CORALOGIX_LOG_URL=https://api.Cluster URL/api/v1/logs

CORALOGIX_TIME_DELTA_URL=https://api.Cluster URL/sdk/v1/time

Manually

Create a Cloud Function in your Google Cloud Console with the following settings:

1. Increase “Memory allocated“ to “1 GB“.
2. Change “Trigger“ to “Cloud Storage“.
3. Change “Event Type“ to “Finalize/Create“.
4. Select your GCS Bucket.
5. Change “Runtime“ to “Python 3.7“.
6. Paste the following code to main.py:

#!/usr/bin/python
# -*- coding: utf-8 -*-

"""
Coralogix GCP function
Author: Coralogix Ltd.
Email: info@coralogix.com
"""

import os
import re
import sys
import gzip
import logging
from google.cloud import storage
from coralogix.handlers import CoralogixLogger

__name__ = 'gcsToCoralogix'
__author__ = 'Coralogix Ltd.'
__email__ = 'info@coralogix.com'
__copyright__ = 'Copyright 2019, Coralogix Ltd.'
__credits__ = ['Ariel Assaraf', 'Amnon Shahar', 'Eldar Aliiev']
__license__ = 'Apache Version 2.0'
__version__ = '1.0.0'
__maintainer__ = 'Coralogix Ltd.'
__date__ = '1 April 2019'
__status__ = 'Stable'

# Get function parameters
PRIVATE_KEY = os.environ.get('private_key')
APP_NAME = os.environ.get('app_name', 'NO_APPLICATION')
SUB_SYSTEM = os.environ.get('sub_name', 'NO_SUBSYSTEM')
NEWLINE_PATTERN = os.environ.get('newline_pattern', '(?:\r\n|\r|\n)')


# Function entrypoint
def to_coralogix(event, context):
    """
    Function entrypoint
    :param event: event metadata
    :type event: dict
    :param context: event context
    :type context: dict
    """

    def get_severity(message: str) -> int:
        """
        Extract severity from message text
        :param message: log record text
        :type message: str
        :return: severity value
        :rtype: int
        """
        severity = 3
        if 'Warning' in message or 'warn' in message:
            severity = 4
        if 'Error' in message or 'Exception' in message:
            severity = 5
        return severity

    # Initialize GCS client
    client = storage.Client()

    # Initialize Coralogix logger
    logger = CoralogixLogger(
        PRIVATE_KEY,
        APP_NAME,
        SUB_SYSTEM,
        'GCP'
    )

    logging.info(f"Processing file {event['name']}")

    # Get file content
    bucket = client.get_bucket(event['bucket'])
    blob = bucket.get_blob(event['name'])
    content = blob.download_as_string()

    # Check if file is compressed
    if event['contentType'] == 'application/gzip' or \
       event['name'].endswith('.gz'):
        logging.info(f"Uncompress file {event['name']}")
        try:
            # Decompress file
            content = gzip.decompress(content)
        except Exception as exc:
            logging.fatal(f"Cannot uncompress file {event['name']}: ", exc)
            sys.exit(1)

    # Split file into line and remove empty lines
    logs = list(filter(None, re.split(NEWLINE_PATTERN, content.decode('utf-8'))))
    logging.info(f"Number of logs: {len(logs)}")

    # Send logs to Coralogix
    for log in logs:
        logger.log(
            get_severity(log),
            log,
            thread_id=f"{event['bucket']}/{event['name']}"
        )

7. Paste the following packages to requirements.txt:

google-cloud-storage>=1.14.0
coralogix_logger>=2.0.1.post7

8. Increase “Timeout“ to “60 seconds“.

9. Add the mandatory environment variables: private_key, app_name, sub_name:

• Private Key – A unique ID that represents your company, this Id will be sent to your mail once you register to Coralogix.
• Application Name – Used to separate your environment, e.g. SuperApp-test/SuperApp-prod.
• SubSystem Name – Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers etc.

10. Multiline pattern: Coralogix supports multiline pattern by default, you can define a custom pattern with environment variables, for example:

newline_pattern [\s(?={)|(?<=})\s,\s(?={)|(?<=})\s\].

11. Click “Create”.

gcloud CLI

To set up the function, execute this:

$ curl -sSL -o gcsToCoralogix.zip https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/gcp/gcs/lambda/gcsToCoralogix.zip
$ unzip gcsToCoralogix.zip -d gcsToCoralogix/
$ gcloud functions deploy gcsToCoralogix \
--project=YOUR_GCP_PROJECT_ID \
--region=us-central1 \
--runtime=python37 \
--memory=1024MB \
--timeout=60s \
--entry-point=to_coralogix \
--source=gcsToCoralogix \
--trigger-resource=YOUR_BUCKET_NAME \
--trigger-event=google.storage.object.finalize \
--set-env-vars="private_key=YOUR_PRIVATE_KEY,app_name=APP_NAME,sub_name=SUB_NAME"

Terraform

Here is the Terraform module to deploy the Cloud Function.

Add this module to your manifest and change its options:

provider "google" {
project = "YOUR_GCP_PROJECT_ID"
region = "us-central1"
}

module "gcs_to_coralogix" {
source = "git::https://github.com/coralogix/integrations-docs.git//integrations/gcp/gcs/terraform"

private_key = "YOUR_PRIVATE_KEY"
app_name = "APP_NAME"
sub_name = "SUB_NAME"
bucket_name = "YOUR_BUCKET_NAME"
}

Download the module and apply these changes:

$ terraform init
$ terraform apply