Google Workspace Data Ingestion – Google Cloud Platform (GCP)

The following tutorial demonstrates how to successfully integrate Google Workspace with Coralogix and send us your logs using Filebeat and Google report API.

This document includes cluster dependent URLs. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

Set up a GCP Service Account

Follow the official Google tutorial for setting up a service account.

Grant access to the Admin SDK API

Follow the official Google Workspace tutorial for granting access to the Admin API.

Note: to gather Google Alert Center, enable the Google Workspace Alert Center API on the GCP project.

Delegate domain-wide authority to your service account

• Go to your Google Workspace domain’s Admin console.
• Go to Main menu > Security > Access and data control >  API controls.
• Under  the Domain-wide delegation pane, select Manage Domain Wide Delegation.
• Click Add new, and fill in the details:
  • Client ID – Enter the service account’s Client ID – you can find it in the service account’s details under Unique ID. It is also found in the client_id field of the credentials file that was auto-downloaded when you created a new key for your service account.
  • OAuth Scopes – Enter https://www.googleapis.com/auth/admin.reports.audit.readonly
  • Click Authorize to confirm your changes.

Once done with the configuration of Google Workspace, now we should start with filebeat.

Filebeat Setup

Please follow the link for Coralogix filebeat setup (Make sure to use the latest version)

Please make sure to have at least version 7.12 and up for filebeat.

Filebeat Google Workspace Configuration

ignore_older: 3h
filebeat.modules:
- module: google_workspace
  saml:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"
  user_accounts:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"
  login:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"
  admin:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"
  drive:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"
  groups:
    enabled: true
    var.jwt_file: "path to the service accounts credentials file"
    var.delegated_account: "email of the primary admin Google Workspace user"

fields_under_root: true
fields:
  PRIVATE_KEY: "your_company_private_key"
  COMPANY_ID: your_company_Id
  APP_NAME: "App_name for example google"
  SUB_SYSTEM: "Sub_system_name"

processors:
- drop_fields:
    fields:
    - event.original
    ignore_missing: true

logging:
  level: debug
  to_files: true
  files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 10
  permissions: 0644

output.logstash:
  enabled: true
  hosts: ["logstashserver.Cluster URL:5015"] 
  tls.certificate_authorities: ["/etc/filebeat/ssl/coralogix.crt"]
  ssl.certificate_authorities: ["/etc/filebeat/ssl/coralogix.crt"] 

After applying the changes. Start your filebeat service.

If you want to learn more about the Google Workspace plugin please see the link.

Please note that Google Workspace defaults to a 2-hour polling interval because Google reports can go from some minutes up to 3 days of delay.
For more details on this, you can read more here.