Log Normalization

Make data analysis easier with log normalization.

Overview

Managing and analyzing logs across various products presents a significant challenge due to the diverse structures and formats in which logs are stored. This diversity complicates searching for specific information across product logs, as each product may use different keys to store the same information. For example, the source IP address might be recorded under sourceIPAddress, source.ip, srcaddr, or other keys, depending on the reporting product. This discrepancy makes data comparison and aggregation difficult, resulting in complex and inefficient data analysis processes.

Log normalization makes data analysis simpler by giving standard names to common values in logs, using parsing rules to organize them.

How It Works

Deploy one of many Coralogix extension packages to add the cx_security key to your product logs. This will normalize email addresses, IP addresses, and URLs across your logs into a single key name even if the same data comes from different key names.

Keys created under cx_security include source_ip, destination_ip, email, url, domain, username, userAgent, event_name, service_name, result. Additional keys will be added from time to time.

Here are some examples:

{
   cx_security: {
      email: "[email protected]"
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
      url: "www.google.com"
   }
   sourceIPAddress: "1.1.1.1"
   destinationIPAddress: "2.2.2.2"
   website: "www.google.com"
   user: {
      id: "[email protected]"
   }
}


{
   cx_security: {
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
   }
   network: {
      src: "1.1.1.1"
      dst: "2.2.2.2"
   }
}

Getting Started

Get started with log normalization by deploying a Coralogix extension package. Once deployed, the log normalization parsing rules are automatically activated using parsing rules.

STEP 1. Navigate to Data Flow > Extensions from your Coralogix toolbar.

STEP 2. Choose an extension package bound to your integration. Select from any of these 18+ extension packages, which include log normalization parsing rules:

• CloudTrail
• Okta
• GCP
• Google Workspaces
• Google Audit
• DUO Security
• Auth0
• JumpCloud
• FortiGate
• Trend Micro AV
• AWS WAF
• Cloudflare WAF
• Cloudflare Audit
• UpGuard
• Zscaler VPN
• SentinelOne
• Crowdstrike Falcon
• Route 53 Query Logging
• VPC Flow Logs

STEP 3. Define the parameters of the extension.

• Select applications and subsystems for which alerts and parsing rules will be applied. These should be identical to those in the associated integration package.
• If you have reached your parsing rules limit, please contact Customer Support to increase it.

STEP 4. Deploy.

Using Log Normalization

Explore Screen

Utilize the Explore screen to execute queries based on cx_security keys to retrieve all logs sharing the same value. For example, run the following to retrieve all activity logs associated with the user’s email [email protected]:

cx_security.email:”[email protected]”

Custom Dashboards

In your Custom Dashboard, create widgets grouped by normalized keys under cx_security.

For instance, create a horizontal bar chart grouping by cx_security.source_ip to visualize the top source IPs across product logs.

Alerts

Set up Flow Alerts to inform you know when a series of alerts with the same details happen in a row within a certain time. Grouping normalized keys under cx_security allows you to configure your Flow Alert with base alerts sharing the same value for these normalized keys.

For example, to detect a suspicious sequence of actions taken by the same user, where each action is reported by a different integration or product, create a Flow Alert where every alert has the same user’s email in cx_security.email.

Enrichment

Select the normalized keys to be enriched by Unified Threat Intelligence and Geo enrichments, rather than selecting each of the original keys per product log.

For instance, select the cx_security.url key for the Unified Threat Intelligence enrichment and define an alert to detect the presence of the enriched cx_security.url_suspected key across all normalized product logs containing a URL.ֿ

Best Practices

We recommend log normalization in parallel to deploying Google Workspace Users, which adds user context to the normalized log keys.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to contact us via our in-app chat or by emailing [email protected].