Loggregation: Making Big Data Small

In this post, you will learn how to make the most of Coralogix’s very own automatic log clustering feature: Loggregation©.

Loggregation is an algorithm that condenses millions of log entries into a narrow set of patterns. It does so by automatically analyzing each log record sent to Coralogix, then separating the log constants from its variables.

Let’s have a look.

1) Below you can see, ~12.6M log entries from a certain query:

coralogix loggregation all logs

2) Within seconds of clicking Loggregation, the 12.6M logs become grouped into their original patterns, thereby dramatically reducing the number of entries.

coralogix loggregation clicking loggregation

As you can see below, we went from 12.6M entries to only 710 templates. There is also a display of how many times each pattern arrived, the ratio of each pattern, and a full visualization of all parameters. This serves as a major timesaver in terms of log analysis.

3) Have JSON format log data? no problem! Coralogix clusters your JSON’s into their unique appearances while allowing you to graphically view the different values including an automatic text clustering. 

coralogix JSON loggregation / clustering

For further analysis, you can click on any of the variables, which are conveniently color-coded. Doing so will immediately display a graph with relevant information.

Here’s a closer look at the types of graphs for each (colored) variable.

Red variables are categorical (text) parameters. Clicking on them will display a categorical bar graph that describes the number of log entries for each value of the selected parameter (displaying up to 50 different values). For example:

coralogix loggregation red parameters

coralogix loggregation red parameter graph

Blue variables are categorical (text) parameters. Clicking on them will display a variable free distribution graph that describes the total number of log entries of the selected log pattern within the query time window. For example:

coralogix loggregation blue button

coralogix loggregation blue button graph

Green variables are numerical parameters. Clicking on them will display a line graph that describes the average value of the selected parameter per time-bin. Note that the size of each time-bin matches the size of the time-bins of the occurrences graph in ‘Logs” tab (e.g. for a 24-hour query time window the time-bin scale will be in hours; for a 1-hour time window it will be in minutes). For example:

coralogix loggregation green button

coralogix loggregation green button graph

4) For further analysis of a specific log pattern just mark the log, hoover it and click the three dots icon, which will appear, or click the ‘space’ button, this will open the ‘Info-Panel’. Click the ‘Query log template’ button, exit ‘Info-Panel’ by clicking the ‘space’ button once more, and press ‘GO’. Switch from Loggregarion back to Logs to see all logs within the selected group.

coralogix loggregation mark a log

coralogix loggregation query by template

coralogix loggregation template group

coralogix loggregation all logs in a group view

In addition, clicking the pin icon at the top right corner of each visualization allows you to send that graph directly to your dashboard in seconds.

That is it! Now you know how you can a full day’s worth of data in a matter of minutes, including deep analyses of distributions and variables. Save time and resources with Coralogix!

Signup to Coralogix

WordPress Lightbox