Flow anomaly – automatic problem detection
- August , 2016,08
2 years ago when we examined the log analytics market, we saw quite a few companies with great products for indexing and visualizing logs. However, the competition between all these tools was narrowed to who makes the most flexible query language or who is the fastest in indexing log data. In other words: Who applies the most brute force on big data.
The problem with this approach was that Log Analytics users didn’t really know how to make the best out of the valuable log data they collect since they had to know what to search for and in what timeframe. Moreover, they reacted to their production problems instead of proactively tackling them.
Our goal at Coralogix is to disrupt this market with a whole new approach: get the data you need by push, and not by pull.
Coralogix automatically learns the system’s log sequences in order to detect production software problems in real time. The algorithm identifies which logs arrive together and in what arrival ratio and alerts the user in case this ratio was broken.
An example from one of our customers was a pattern which consisted of 3 logs that always arrived together with a ratio of 33% for each log within the sequence:
1. About to send data to customer ID XXXXX in X seconds
2. Sending data to customer ID XXXXX
3. Total data sent to customer ID XXXXX is X KB
In this case, Coralogix detected a production bug in which data wasn’t sent to customers, this bug was reflected by the absence of log #2 describing the sending process. What Coralogix found was that log 1# arrived and then log #3 arrived with the value 0 for the amount of data sent in KB. Our user was notified in real time and the problem was solved (one web server was badly configured).
On your main Dashboard timeline, you can see the anomalies that Coralogix automatically detected, each anomaly is represented with a circle shape the color of its severity. Coralogix will detect anomalies after 4 days of learning the system’s flows.
By clicking an anomaly, you open the insights center with the selected anomaly displayed. The anomaly view contains a column for the normal ratio as learned by Coralogix, the current (anomalous) ratio, and the number of occurrences per log in the presented timeframe.
The ‘Logs’ tab presents all logs that have arrived in the anomaly timeframe with the logs participating the anomaly highlighted.
The Loggregation tab shows an aggregated view of all the logs from the anomaly timeframe with the logs participating the anomaly highlighted:
By clicking the orange ‘Edit anomaly’ button to the right-hand of the anomaly name, you can change the anomaly name and severity, or mute the anomaly to have it hidden from your dashboard:
Start now and enjoy Coralogix’s automatic anomaly detection capabilities.
Signup to Coralogix