Getting Started with Coralogix

This tutorial provides you with your first steps with Coralogix.

Sending data

Coralogix also supports many  integrations which we divide into 6 main categories: 

• File collectors 
• Metric data 
• Security & Audit logs 
• Cloud infrastructure & services 
• Contextual data 
• Code libraries 

Coralogix provides support for all its integrations and an integration session with our engineers can be booked directly here. 

Once you’ve started sending data, you can use one of our extension packages to easily set up predefined Coralogix alerts, parsing rules, and Kibana dashboards that are tailored for each particular integration.

Some available extension packages include Cloudflare, Cloudtrail, Fastly, Jenkins, and more!

Data Transformation

Parsing 

Coralogix allows data parsing using Regex to perform the following operations: 

• Parse – parse unstructured logs into JSON format using named Regex groups.
• Extract – use a named RegEx group to extract specific values you need as JSON keys without having to parse the entire log.
• JSON Extract – name a JSON field to extract its value directly into a Coralogix metadata field.
• Replace – replace rules are used to strings in order to fix log structure, change log severity, or obscure information.
• Block – block irrelevant information from entering the system to reduce costs, blocked data may still be routed through live tail and S3.
• Timestamp extract – easily extract any common timestamp format to the Coralogix metadata timestamp. 
• Remove Fields – allows to select fields that will not be indexed.

Enrichment

Coralogix enables data enrichment on the fly to make logs smarter/more readable. There are 3 types of enrichments: 

• Security – enrich IP fields with automatically updated IP blacklists to uncover suspicious activity from IPs accessing your system/application. 
• GEO – Enrich any IP with its GEO location and GEO point for easy map visualizations in Kibana/Grafana/Tableau. 
• Custom enrichment – create any enrichment logic in CSV and load it manually or via API

Monitoring & Insights

Coralogix Log screen

• Use the Logs screen and enter your query in the search bar.
• To filter your results by application or subsystem, Use the filter on the left side of the logs screen.

Basic Queries

Both unstructured and JSON-structured log queries are supported:

1. Querying unstructured logs

• Google-styled search query – Queries the entire log payload for the words entered in the search bar. Matches any log with the combination of words.

Example:

login unauthorized – Returns logs containing your login was unauthorized OR wrong password.

• Query an exact string

Example:

text:”your exact match string” – Returns logs containing the phrase your exact match string.

2. JSON-structured logs

• Google-styled search query as described above.
• Elastic simple query – with word tokenization according to word delimiters

Example:

url:”some url” – Returns logs that match url:/some/url.php or url:/some/url.html

• Keyword search – add the ‘.keyword’ suffix to the field name to query data without tokenization

Example: 

url.keyword:”/some/url.php” – Returns logs that exactly match  url:/some/url.php

• Numeric search – add the ‘.numeric’ suffix to the field name containing a number. Use this to query a search for a range of numbers.

Example:

statusCode.numeric:[200 TO 399] – Returns logs where value of statusCode key is between 200 and 399

Logs2Metrics

Coralogix Logs2Metrics enables you to generate metrics on-the-fly from your log data to optimize storage without sacrificing important data. Simply define a query and Coralogix will execute it every minute and store different data aggregations in a long-term index for a full year at no additional cost.

Metrics start to gather from the point in time in which they were defined. The available query time range for your Logs2Metrics indices is 30 days. Activating Logs2Metrics allows you to create up to 30 metrics with a 12 months retention period.

Learn more about how to create metrics from your log data.

Loggregation Templates

In order to make data investigations simpler and help you find that needle in the haystack, Coralogix created a proprietary real-time clustering algorithm that automatically identifies logs of the same type/origin and clusters them into a log template.

This enables turning hours of data and millions of records into a short list of data templates with easy visualization options. It also provides added value such as template normal behavior learning and the ability to zoom into specific templates.

Loggregation does not require any preconfiguration and works on all data types. To make Loggregation most accurate, have your main log message as a root key and not nested (typically “log”, “message”, “msg”, “text” etc). No need to do anything for unstructured logs. Learn more here. 

Alerting & Visualization

Coralogix Alerts

Alerts in Coralogix can be defined directly from your query by clicking on the “Create Alert” button or from the “Alert” interface at the top bar of the Coralogix screen. Coralogix has 6 main and 12 secondary alert types: 

• Standard alert
  • Immediate alert – triggers on each event
  • More than alert – triggers when More than X matches are met in Y time, allows 2 levels of “Group By” 
  • Less than alert – triggers when Less than X matches are met in Y time 
  • More than usual – triggers on more than usual matches for a specific query, allows 1 level of “Group By” 
• Ratio alert – Alert on the ratio between 2 different queries for SLA tracking 
• New Value alert – Alert on new value detected within a JSON key for a specific query match  
• Unique count – Alert on the unique count per specific JSON key, matching a specific query, allows one level of “Group By” 
• Time Relative Alert – Alert if a specific query is matched more than a relative timeframe:
  • Previous hour 
  • Previous day
  • Same hour yesterday 
  • Same day last week 
  • Same day last month
• Metric – Alert if a specific Log2Metric / Prometheus metric is Over/Under a certain (Max/Min/AVG/Sum/Count/Percentile) for a percentage of a defined timeframe – matching a specific label query.

Visualizations

Coralogix is all about making your life simple. View your Coralogix data and insights in any dashboard including:

• Coralogix UI
• Kibana
• Grafana
• Tableau
• & more!

Advanced Usage

Tags – Automatic Version Benchmarks 

Coralogix harnesses all its features, alerts, queries, anomalies, Loggregation templates, normal behavior learning, new & suspected error detection, and custom widgets to enable a next-generation experience for CICD acceleration. By using the Coralogix “Tags” feature, you can plug your CICD platform into Coralogix, and send your build logs, metrics, and most of all – Version Tags. Coralogix will then compare versions uploaded to the same service in 2 different points in time, and provide an automated benchmark of the key quality metrics for new version release, enabling you to add your own widgets for version over version comparison of any trend or SLA you would like to visualize. Learn more about Version Tags here.

Live Tail

The first place to see your logs after they have been parsed and enriched is the Coralogix Live Tail. Live tail is a low latency, pre-index/storage stream of logs, it sends logs directly from the Coralogix queue to your client and allows data filtering by app/sub, or any “grep” command or sequence. It also allows you to choose which specific log fields will be displayed and enables you to “prettify” JSON data or view it as raw text. Live tail is available in the Coralogix interface, or via CLI.

TCO Optimization 

Coralogix “Streama” engine allows it to analyze all data on the fly without the usage of storage, the TCO optimizer enables users to define the use case per app/sub/severity and define policies for optimized data routing. We typically see 70% cost reduction by this feature, and our support team is available 24/7 to assist in defining the correct policies and also policy exceptions.

This guide is the very basic getting started guide to get you up to speed and help you extract the initial value from the product. For additional guidance, feel free to reach out via our in-app chat, and we’ll walk you through step by step.