Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Getting Started with Coralogix

This tutorial will guide you on your first steps with Coralogix and help you start enjoying reduced time to detect and solve production problems.

Sending data: 

Coralogix supports many  integrations which we divide into 6 main categories: 

  • File collectors 
  • Metric data 
  • Security & Audit logs 
  • Cloud infrastructure & services 
  • Contextual data 
  • Code libraries 

Coralogix provides support for all its integrations and an integration session with our engineers can be booked directly here

 

Parsing: 

Coralogix allows data parsing using Regex to perform the following operations: 

  • Parse – turn unstructured data into JSON format 
  • Extract – pull specific text into a JSON key without parsing the entire log 
  • Replace – Replace any text string 
  • Block – Block irrelevant information from entering the system to reduce costs, blocked data may still be routed through live tail and S3. 
  • Drop keys – Drop specific JSON keys from logs to reduce costs, clutter. 
  • Timestamp extract – Easily extract any common timestamp format to the Coralogix metadata timestamp. 

TCO optimization: 

Coralogix “Streama” engine allows it to analyze all data on the fly without the usage of storage, the TCO optimizer enables users to define the use case per app/sub/severity and define policies for optimized data routing. We typically see 70% cost reduction by this feature, and our support team is available 24/7 to assist in defining the correct policies and also policy exceptions.

 

Enrichment:

Coralogix enables data enrichment on the fly to make logs smarter/more readable. There are 3 types of enrichments: 

  • Security – enrich IP fields with automatically updated IP blacklists to uncover suspicious activity from IPs accessing your system/application. 
  • GEO – Enrich any IP with its GEO location and GEO point for easy map visualizations in Kibana/Grafana/Tableau. 
  • Custom enrichment – create any enrichment logic in CSV and load it manually or via API

Live tail: 

The first place to see your logs after they have been parsed and enriched, is the Coralogix live tail. Live tail is a low latency, pre index/storage stream of logs, it sends logs directly from the Coralogix queue to your client and allows data filtering by app/sub, or any “grep” command or sequence. It also allows you to choose which specific log fields will be displayed and enables you to “prettify” JSON data or view it as raw text. Live tail is available in the Coralogix interface, or via CLI.

 

Coralogix Log screen:

  • Use the Logs screen and enter your query in the search bar.
  • To filter your results by application or subsystem, Use the filter on the left side of the logs screen.

 

Basic queries

Both unstructured and JSON-structured log queries are supported:

  1. Querying unstructured logs:
    1. Google-styled search query – Queries the entire log payload for the words entered in the search bar. Matches any log with the combination of words.

Example

  • login unauthorized

Returns logs containing Your login was unauthorized OR Wrong password. Login unauthorized. 

  1. Query an exact string.

Example

  • text:”your exact match string”

Returns logs containing the phrase your exact match string

  1. JSON-structured logs:
    1. Google-styled search query as described above.
    2. Elastic simple query – with word tokenization according to word delimiters

Example:

  • url:”some url”

Returns logs that match url:/some/url.php or url:/some/url.html

  1. Keyword search – add the ‘.keyword’ suffix to the field name to query data without tokenization

Example: 

  • url.keyword:”/some/url.php”

Returns logs that exactly match  url:/some/url.php

  1. Numeric search – add the ‘.numeric’ suffix to the field name containing a number. Use this to query a search for a range of numbers. 

Example:

  • statusCode.numeric:[200 TO 399]

Returns logs where value of statusCode key is between 200 and 399

Loggregation Templates

In order to make data investigations simpler, and help you find that needle in the haystack, Coralogix created a proprietary real-time clustering algorithm that automatically identifies logs of the same type/origin and clusters them into a log template. This enables turning hours of data, and millions of records into a short list of data templates with easy visualization options and added value such as template normal behavior learning or the ability to zoom into specific templates. Loggregation does not require any pre-defining and works on all data types. To make Loggregation most accurate, have your main log message as a root key and not nested (typically “log”, “message”, “msg”, “text” etc) No need to do anything for unstructured logs. Learn more here

Coralogix Alerts:

Alerts in Coralogix can be defined directly from your query by clicking on the “Create Alert” button or from the “Alert” interface at the top bar of the Coralogix screen. Coralogix has 6 main and 12 secondary alert types: 

  • Standard alert
    • Immediate alert – triggers on each event
    • More than alert – triggers when More than X matches are met in Y time, allows 2 levels of “Group By” 
    • Less than alert – triggers when Less than X matches are met in Y time 
    • More than usual – triggers on more than usual matches for a specific query, allows 1 level of “Group By” 
  • Ratio alert – Alert on the ratio between 2 different queries for SLA tracking 
  • New Value alert – Alert on new value detected within a JSON key for a specific query match  
  • Unique count – Alert on the unique count per specific JSON key, matching a specific query, allows one level of “Group By” 
  • Time Relative Alert – Alert if a specific query is matched more than a relative timeframe:
    • Previous hour 
    • Previous day
    • Same hour yesterday 
    • Same day last week 
    • Same day last month
  • Metric – Alert if a specific Log2Metric / Prometheus metric is Over/Under a certain (Max/Min/AVG/Sum/Count/Percentile) for a percentage of a defined timeframe – matching a specific label query.

Tags – Automatic Version Benchmarks: 

Coralogix harnesses all its features, alerts, queries, anomalies, Loggregation templates, Normal behavior learning, New & Suspected error detection, and custom widgets, to enable a next generation experience for CICD acceleration. By using the Coralogix “Tags” feature, you can plug your CICD platform into Coralogix, and send your build logs, metrics, and most of all – Version tags. Coralogix will then compare versions uploaded to the same service in 2 different points in time, and provide an automated benchmark of the key quality metrics for new version release, enabling you to add your own widgets for version over version comparison of any trend or SLA you would like to visualize. Learn more about Version Tags here.  

Kibana:

To enter Kibana you need to login to Coralogix first (see How to login chapter) and then click on Kibana button in the top right corner.

You will see the Kibana main screen:

On the left side you can see shortcuts to:

  • Recently viewed – useful when you want to go back to the recent search/visualization/dashboard
  • Discover – search/review logs in Kibana
  • Visualize – open or create new visualization
  • Dashboard – open or create new dashboard
  • Management – review/refresh index patterns, manage saved objects, Kibana settings

Discover

Discover enables you to quickly search and filter your data, and get information about the structure of the fields.

As you can see in the attached screenshot:

  • At the top left there is a menu:
    New – create new search
    Save – save the current search for later use
    Open – open earlier saved query
    Share – Share a link to your search
  • Below the menu there is a search bar.
    – Use the Lucene query language to easily copy queries from/to Kibana to/from Coralogix if needed
  • Next to the search bar, there is a time filter. Use it if you need to specify a timeframe

Below the search bar there is a “+Add filter” button. Use it for easier filtering. Details about the feature: https://www.elastic.co/guide/en/kibana/6.8/field-filter.html

 

As you can see as an example coralogix.metadata.applicationName and log fields were added as display filters. To add a new display filter click on the add button next to the field which is on the left at Available fields column. To remove a display filter click on the remove button next to the field which is on the left of in the Selected fields.

 

How to query different applications in Kibana:

Even though Kibana doesn’t have the Coralogix left filter for Applications/Subsystems, you may limit your search to logs from a specific application and subsystem by applying the following filters to your search on kibana search bar:

  • coralogix.metadata.applicationName:application
  • coralogix.metadata.subystemName:subsystem

 

This guide is the very basic getting started guide to get you up to speed and help you extract the initial value from the product. To learn more about Coralogix, visit our Help Center or the full Tutorials page  .You are always welcome to Schedule a 1:1 demo and we’ll walk you through step by step.