Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Flow Anomaly

When we examined the Log Analytics market, we saw quite a few companies with great products for indexing and visualizing logs. However, the competition between all these tools was narrowed to who makes the most flexible query language or who is the fastest in indexing log data. In other words: Who applies the most brute force to big data.

The problem with this approach was that Log Analytics users didn’t really know how to make the best out of the valuable log data they collect since they had to know what to search for and in what timeframe. Moreover, they reacted to their production problems instead of proactively tackling them.

Our goal at Coralogix is to disrupt this market with a whole new approach: get the data you need by push, and not by pull.

Coralogix automatically learns the system’s log sequences in order to detect production software problems in real-time. The algorithm identifies which logs arrive together and in what arrival ratio and alerts the user in case this ratio was broken.

An example from one of our customers was a pattern that consisted of 3 logs that always arrived together with a ratio of 33% for each log within the sequence:
1. About to send data to customer ID XXXXX in X seconds
2. Sending data to customer ID XXXXX
3. Total data sent to customer ID XXXXX is X KB

In this case, Coralogix detected a production bug in which data wasn’t sent to customers, this bug was reflected by the absence of log #2 describing the sending process. What Coralogix found was that log 1# arrived and then log #3 arrived with the value 0 for the amount of data sent in KB. Our user was notified in real-time and the problem was solved (one web server was badly configured).

On your main Dashboard timeline, you can see the anomalies that Coralogix automatically detected, each anomaly is represented with a circle shape the color of its severity. Coralogix will detect anomalies after 4 days of learning the system’s flows.

coralogix dashboard insights

By clicking an anomaly, you open the insights center with the selected anomaly displayed. The anomaly view contains the logs which usually arrive together with their current (anomalous) ratio Vs. their normal behavior. Note that there can be more than 1 template that behaves in an anomalous way. 

coralogix anomaly 2017

Below the main anomaly display you can see the automatic anomaly forensics: 

  1. Suspected Errors: High severity logs which arrived more than normal in the anomaly timeframe
  2. Top Errors: The top errors in the anomaly timeframe sorted by number of occurrences
  3. Newly introduced templates: Templates which have arrived for the first time in your application during the anomaly timeframe.
coralogix anomaly forensics
coralogix top errors
log flow anomaly coralogix

The ‘Logs’ tab presents all logs that have arrived in the anomaly timeframe with the logs participating in the anomaly highlighted.

anomaly log display

The Loggregation tab shows an aggregated view of all the logs from the anomaly timeframe with the logs participating in the anomaly highlighted:

anomaly loggregation view

By clicking the ‘Edit anomaly’ button to the right-hand of the anomaly name, you can change the anomaly name and severity, or mute the anomaly to have it hidden from your dashboard:

coralogix tune anomaly

Start now and enjoy Coralogix’s automatic anomaly detection capabilities.