Our next-gen architecture is built to help you make sense of your ever-growing data. Watch a 4-min demo video!

Back to All Docs

Log Normalization

Last Updated: Jul. 04, 2024

Make data analysis easier with log normalization.

Overview

Managing and analyzing logs across various products presents a significant challenge due to the diverse structures and formats in which logs are stored. This diversity complicates searching for specific information across product logs, as each product may use different keys to store the same information. For example, the source IP address might be recorded under sourceIPAddress, source.ip, srcaddr, or other keys, depending on the reporting product. This discrepancy makes data comparison and aggregation difficult, resulting in complex and inefficient data analysis processes.

Log normalization simplifies data analysis by giving standard names to common values in logs and organizing them using parsing rules.

How It Works

Deploy one of many Coralogix extension packages to add the cx_security Add a key to your product logs. This will normalize email addresses, IP addresses, and URLs across your logs into a single key name, even if the same data comes from different key names.

Keys created under cx_security include source_ip, destination_ip, email, url, domain, username, userAgent, event_name, service_name, result, source_hostname, destination_hostname, resource, event_result, event_name, target_email, target_username. Additional keys will be added from time to time.

Here are some examples:

{
   cx_security: {
      email: "john@bank.com"
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
      url: "www.google.com"
   }
   sourceIPAddress: "1.1.1.1"
   destinationIPAddress: "2.2.2.2"
   website: "www.google.com"
   user: {
      id: "john@bank.com"
   }
}


{
   cx_security: {
      source_ip: "1.1.1.1"
      destination_ip: "2.2.2.2"
   }
   network: {
      src: "1.1.1.1"
      dst: "2.2.2.2"
   }
}

Getting Started

Get started with log normalization by deploying a Coralogix extension package. Once deployed, the log normalization parsing rules are automatically activated using parsing rules.

STEP 1. Navigate to Data Flow > Extensions from your Coralogix toolbar.

STEP 2. Choose an extension package bound to your integration. Select from any of our extension packages that include log normalization parsing rules:

  • 1Password
  • Amazon Route53 DNS Query Resolver
  • Amazon VPC Flow Logs
  • Auth0
  • AWS ALB
  • AWS CloudTrail
  • AWS VPN
  • AWS WAF
  • Azure Audit
  • Cloudflare
  • CrowdStrike
  • CyberArk PAM
  • DUO Security
  • Fortigate
  • Fortigate Firewall
  • GCP Audit
  • GCP Security Command Center
  • GitHub Enterprise
  • Google Workspace
  • Google Workspace Alert Center
  • InfoBlox DNS
  • JumpCloud
  • Okta
  • Palo Alto
  • Prisma SASE
  • SentinelOne
  • Trend Micro AV
  • TrendMicro
  • UpGuard
  • Zscaler ZIA
  • Zscaler ZPA

Note that this list may not reflect the most updated list of extensions.

STEP 3. Define the parameters of the extension.

  • Select applications and subsystems for which alerts and parsing rules will be applied. These should be identical to those in the associated integration package.
  • If you have reached your parsing rules limit, please contact Customer Support to increase it.

STEP 4. Deploy.

Using Log Normalization

Explore Screen

Utilize the Explore screen to execute queries based on cx_security keys to retrieve all logs sharing the same value. For example, run the following to retrieve all activity logs associated with the user’s email john@bank.com:

cx_security.email:”john@bank.com”

Custom Dashboards

In your Custom Dashboard, create widgets grouped by normalized keys under cx_security.

For instance, create a horizontal bar chart grouping by cx_security.source_ip to visualize the top source IPs across product logs.

Alerts

Set up Flow Alerts to inform you know when a series of alerts with the same details happen in a row within a certain time. Grouping normalized keys under cx_security allows you to configure your Flow Alert with base alerts sharing the same value for these normalized keys.

For example, to detect a suspicious sequence of actions taken by the same user, where each action is reported by a different integration or product, create a Flow Alert where every alert has the same user’s email in cx_security.email.

Enrichment

Select the normalized keys to be enriched by Unified Threat Intelligence and Geo enrichments, rather than selecting each of the original keys per product log.

For instance, select the cx_security.url key for the Unified Threat Intelligence enrichment and define an alert to detect the presence of the enriched cx_security.url_suspected key across all normalized product logs containing a URL.ֿ

Best Practices

We recommend log normalization in parallel to deploying Google Workspace Users, which adds user context to the normalized log keys.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to contact us via our in-app chat or by emailing support@coralogix.com.

On this page