One Click Visibility: Coralogix expands APM Capabilities to Kubernetes

There is a common painful workflow with many observability solutions. Each data type is separated into its own user interface, creating a disjointed workflow that increases cognitive load and slows down Mean Time to Diagnose (MTTD).

At Coralogix, we aim to give our customers the maximum possible insights for the minimum possible effort. We’ve expanded our APM features (see documentation) to provide deep, contextual insights into applications – but we’ve done something different.

Why is APM so important?

Application Performance Monitoring (APM) is one of the most sophisticated capabilities in the observability industry. It allows engineers and operators to inspect detailed application and infrastructure performance metrics. This can include everything from correlated host and application metrics to the time taken for a specific subsystem call. 

APM has become essential due to two major factors:

  • Engineers are reusing more and more code. Open-source libraries provide vast portions of our applications. Engineers don’t always have visibility of most of our application(s).
  • As the application stack grows more extensive, with more and more components performing increasingly sophisticated calculations, the internal behavior of our applications contains more and more useful information.

What is missing in other providers?

Typically, most providers fall victim to the data silo. A siloed mentality encourages engineers to separate their interface and features from the data, not the user journey. This means that in most observability providers, APM data is held in its own place, hidden away from logs, metrics, traces, and security data.

This makes sense from a data perspective. They are entirely different datasets typically used, with varying data demands. This is the basis for the argument to separate this data. We saw this across our competitors and realized that this was slowing down engineers, prolonging outages, and making it more difficult for users to convert their data into actionable insights.

How is Coralogix approaching APM differently?

Coralogix is a full-stack observability platform, and the features across our application exemplify this. For example, our home dashboard covers logs, metrics, traces, and security data:

The expansion of our APM capability (see documentation) is no different. Rather than segregating our data, we want our customers to journey through the stack naturally rather than leaping between different data types to try and piece together the whole picture. With this in mind, It all begins with traces.

Enter the tracing UI and view traces. The filter UI allows users to slice data in several ways, for example, filtering by the 95th Percentile of latency.

Select a span within a trace. This opens up a wealth of incredibly detailed metrics related to the source application. Users can view the logs that were written during the course of this span. This workflow is typically achieved by noting the time of a span and querying them in the logging UI. At Coralogix, this is simply one click.

Track Application Pod Metrics

However, the UI now has the Pod and Host metric for a more detailed insight into application health at the time that the span was generated. These metrics will provide detailed insights into the health of the application pod itself within the Kubernetes cluster. It shows metrics from a few minutes before and after the span so that users can clearly see the sequence of events leading to their span. This level of detail allows users to diagnose even the most complex application issues immediately.

Track Infrastructure Host Metrics

In addition to tracking the application’s behavior, users can also take a wider view of the host machine. Now, it’s possible to detect when the root cause isn’t driven by the application but by a “noisy neighbor.” All this information is available, alongside the tracing information, with only one click between these detailed insights.

Tackle Novel Problems Instantly

If a span took longer than expected, inspect the memory and CPU to understand if the application was experiencing a high load. If an application throws an error, inspect the logs and metrics automatically attached to the trace to better understand why. This connection, between application level data and infrastructure data, is the essence of cutting-edge APM. 

Combined with a user-focused journey, with Coralogix, a 30-minute investigation becomes a 30-second discovery. 

Be GDPR Ready: Prepare Your Log Data

Organizations both small and large that deal with personal data must be compliant with GDPR rules. At Coralogix, we’ve been working hard to be prepared for GDPR logging monitoring. Preparing your data for GDPR log management can be a daunting task, so we thought we’d shed some light on the issue.

What’s GDPR?

The European Union’s (EU) General Data Protection Regulation, or GDPR, is a set of regulations designed to protect the privacy of EU citizens, particularly as the volume and pace of data generated is exploding. With privacy laws and data breaches being in the news of late, GDPR log retention is seen as a way for individuals to protect their data, access their data, and understand what is being done with that data.

The regulations come into effect on May 25, 2018, and in general, terms affect any EU organization, organizations with an EU presence, and organizations with EU customers. The penalties for those who should comply with the regulations are severe: companies could face penalties of up to €20 million or 4% of their annual worldwide revenue, whichever is higher.

GDPR aims to ensure certain key rights for individuals. These rights include the right to be informed, the right of access, the right to rectification, the right to erasure, the right to restrict processing, the right to data portability, the right to object, and the right not to be subject to automated decision-making including profiling.

Coralogix is a GDPR-ready data processor

Assisting users is one of the fundamental principles of Coralogix’s business, and so protecting users’ data and ensuring GDPR compliance is a natural extension of this and a priority for the company. To this end, we’re proud to declare that Coralogix is GDPR-ready.

Coralogix now runs its servers in Europe so EU citizens data doesn’t leave the continent, its infrastructure is SOC2 and PCI compliant as well as being GDPR ready. In addition to this, Coralogix’s application is certified by BDO to be SOC2 type2 compliant (security, availability, data Integrity) for 2018. A full report can be provided upon request. In order to make things easier for our customers facing GDPR regulations we have added a few features that are aimed at meeting the EU new standards:
Flexible data retention policies – change your retention plan upon request within 24H.
Data deletion – by date, or even by a specific key (e.g a specific customer requests to delete all his logs by email)

Available on the Coralogix website are our terms and conditions that spell out exactly what information is collected and why. Coralogix does not collect any personal information besides account username. It is the client’s responsibility to not send sensitive information to Coralogix’s servers.
In general, saving log data (specifically web servers which may contain PII) is allowed for a defined period (i.e retention period) for maintaining the customers’ systems availability and security.

Technical Aspect of GDPR-Ready

Besides our data security policies and data encryption throughout the sending and storing chain, Coralogix offers ways to make sure PII isn’t saved and helps you clean up PII in case it did reach your log data. 

  • Coralogix offers a centralized interface for masking or blocking, logs containing Personally Identifiable Information/sensitive data, in case they are accidentally sent even before they are indexed or stored anywhere.
  • In terms of the removal of data, Coralogix allows the deletion of data by day or key upon request and within 120 hours. Data is stored in different indexes for different teams/companies so that it is completely separated using Elastic Shield.

Preparing Your Log Data for GDPR

Part of being GDPR compliant is ensuring that your log data is prepared for GDPR, including understanding the types of data that shouldn’t exist in logs. The points we offer here are general in nature, and we recommend obtaining legal advice to ensure full compliance.

First off, logs can contain information classified as “personal data” by default under GDPR regulations. In general, the GDPR regulations encourage organizations not to collect any information about users (let it be email addresses, phone numbers, or even IPs), unless there is documented and informed consent for this collection. It also aims to achieve, through regulation, for collected information not to be used for anything but the specific purposes that consent was given for.

This is a far cry from what has been happening up until now with log data, where the focus was on collecting and storing as much data as possible and storing this for as long as possible.

For instance, web server logs, access logs, and security audit logs all contain personal information by default as defined in the GDPR regulations, and IP addresses specifically are defined as personal data.

As a general rule, if there is not a legitimate need to store these logs, you should disable logging for these components. This type of information is not even allowed to store without having direct consent from the user, outlining the purposes you intend to store the information for.

In fact, it’s best to be on the side of caution and ensure that as little customer information is stored as possible, and even then only when necessary and only with consent – along with compliance with all the other GDPR requirements.

There is an important exception to this general rule, however: collecting and storing personal data as part of your ability to maintain the security and availability of your system and prevent fraud and/or unauthorized access, is allowed for a limited (declared) period of time. 

In addition, for your application logs, make sure you don’t log any PII in your code, and define a clear retention policy so that data is periodically deleted and not stored forever. It is also important to have an easy way to track PII in your logs and delete it upon request, entirely or by a specific key/query. 

Bring It On

Coralogix, as your GDPR-ready partner, is already on your side when it comes to this complex issue, and by ensuring that we are compliant you can rest assured that you have a strong partner with which to achieve your business goals.