Tag: Log Management

Microservices on Kubernetes: 12 Expert Tips for Success

Posted on by eugene evdokimov

In recent years, microservices have emerged as a popular architectural pattern. Although these self-contained services offer greater flexibility, scalability, and maintainability compared to monolithic applications, they can be difficult to manage without dedicated tools. 

Kubernetes, a scalable platform for orchestrating containerized applications, can help navigate your microservices. In this article, we will explore the relationship between Kubernetes and microservices, key components and benefits of Kubernetes and best practices for deploying microservices on the platform.

Before we dive in, let’s take a moment to understand the concept of microservices and examine some of the challenges they present, such as log management.

What are microservices?

Microservices are an architectural style in software development where an application is built as a collection of small, loosely coupled, and independently deployable services. 

Each service represents a specific business capability and operates as a separate unit, communicating with other services through well-defined APIs. These services are designed to perform a single task or function, following a single responsibility principle.

In contrast to traditional monolithic architectures, where the entire application is tightly integrated and deployed as a single unit, microservices break down the application into smaller, more manageable pieces.

Source: https://aws.amazon.com/compare/the-difference-between-monolithic-and-microservices-architecture/

Benefits of microservices 

Adopting a microservice architecture has several benefits. The decentralized nature of microservices enables them to operate independently, allowing separate development, deployment, and scalability. This autonomy leads to decentralized decision-making, fostering an environment where teams can work autonomously. 

Additionally, it allows developers to use different technologies and frameworks across microservices, as long as they adhere to standardized APIs and communication protocols.

The modular structure of microservices brings flexibility and agility to development, facilitating easy modifications and updates without disrupting the entire application.

This flexibility enables development teams to swiftly respond to changing requirements, accelerating time-to-market. It also means that a failure in one service does not cascade to affect others, resulting in a more robust overall system. 

Lastly, microservices support horizontal scaling. Each service can replicate itself to handle varying workloads, ensuring optimal resource utilization and scalability as the application grows. 

Challenges of microservices 

While microservices offer many advantages, they also introduce complexities in certain areas, such as observability. In a monolithic application, it is relatively easy to understand the system’s behavior and identify issues since everything is tightly coupled. As an application is divided into independent microservices, the complexity naturally rises, requiring a shift in how observability is employed within the system. This is especially true for log observability for microservices, since we now have independent services that generate an important amount of logs when interacting with each other and handling requests. 

Other challenges of microservices include managing inter-service communication, data consistency, and orchestrating deployments across multiple services. Thus Kubernetes can help you by offering a robust and efficient solution to handle these challenges and streamline the management of microservices.

Components of Kubernetes

Before delving into the advantages of using Kubernetes for microservices, let’s take a brief look at its key components. 

A Kubernetes cluster is composed of a Control Plane and Worker Nodes. Each worker node is like a stage where your applications perform. Inside these nodes, you have small units called pods, which are like mini-containers for your applications.

These pods contain your application’s code and everything it needs to run. The control plane is like the mastermind, managing the entire show and keeping track of all the worker nodes and pods, making sure they work together harmoniously. The pods will also orchestrate the deployment, scaling, and health of your applications.

Source: https://kubernetes.io/docs/concepts/overview/components/

Kubernetes also provides other valuable features, including: 

  1. Deployments

With Deployments, you can specify the desired state for pods, ensuring that the correct number of replicas is always running. It simplifies the process of managing updates and rollbacks, making application deployment a smooth process..

  1. Services 

Kubernetes Services facilitate seamless communication and load balancing between pods. They abstract away the complexity of managing individual pod IP addresses and enable stable access to your application services.

  1. ConfigMaps and Secrets

ConfigMaps and Secrets offer a neat way to separate configuration data from container images. This decoupling allows you to modify configurations without altering the container itself and enables secure management of sensitive data.

  1. Horizontal Pod Autoscaling (HPA)

HPA is a powerful feature that automatically adjusts the number of pods based on resource utilization. It ensures that your applications can handle varying workloads efficiently, scaling up or down as needed.

Benefits of using Kubernetes for microservices

Kubernetes provides several advantages when it comes to managing microservices effectively.

  1. Scalability

Kubernetes excels at horizontal scaling, allowing you to scale individual microservices based on demand. This ensures that your applications can handle varying workloads effectively without over-provisioning resources.

  1. High availability

Kubernetes provides built-in self-healing capabilities. If a microservice or a node fails, Kubernetes automatically restarts the failed components or replaces them with new ones, ensuring high availability and minimizing downtime.

  1. Resource management

Kubernetes enables efficient resource allocation and utilization. You can define resource limits and requests for each microservice, ensuring fair distribution of resources and preventing resource starvation.

  1. Rolling updates and rollbacks

With Kubernetes Deployments, you can seamlessly perform rolling updates for your microservices, enabling you to release new versions without service disruption. In case of issues, you can quickly roll back to the previous stable version.

  1. Service discovery and load balancing

Kubernetes provides a built-in service discovery mechanism that allows microservices to find and communicate with each other. Additionally, Kubernetes automatically load-balances incoming traffic across multiple replicas of a service.

  1. Automated deployment

Kubernetes enables the automation of microservices deployment. By integrating CI/CD pipelines with Kubernetes, you can automate the entire deployment process, reducing the risk of human errors and speeding up the delivery cycle.

  1. Declarative configuration

Kubernetes follows a declarative approach, where you specify the desired state of your microservices in YAML manifests. Kubernetes then ensures that the actual state matches the desired state, handling the complexities of deployment and orchestration.

  1. Version compatibility

Kubernetes supports various container runtimes, such as Docker and containerd, allowing you to run containers built with different versions of the runtime. This makes it easier to migrate and manage microservices developed with diverse technology stacks.

  1. Community and ecosystem

Kubernetes has a vibrant and active open-source community, leading to continuous development, innovation, and support. Additionally, an extensive ecosystem of tools, plugins, and add-ons complements Kubernetes, enriching the overall user experience.

  1. Observability and monitoring

Kubernetes integrates well with various monitoring and observability tools, providing insights into the performance and health of microservices.

12 tips for using microservices on Kubernetes

Creating and deploying microservices on Kubernetes involves several steps, from containerizing your microservices to defining Kubernetes resources for their deployment. Here’s a step-by-step guide, featuring our Kubernetes tips, to help you get started:

1. Containerize your microservices

Containerize each microservice and Include all dependencies and configurations required for the service to run.

2. Set up Kubernetes cluster

Install and set up Kubernetes. Depending on your requirements, you can use a managed Kubernetes service (e.g., GKE, AKS, EKS) or set up your own Kubernetes cluster using tools like kubeadm, kops, or k3s.

3. Create Kubernetes deployment manifest

Write a Kubernetes Deployment YAML manifest for each microservice: Define the desired state of the microservice, including the container image, resource limits, number of replicas, and any environment variables or ConfigMaps needed.

4. Create Kubernetes service manifest 

If your microservices require external access or communication between services, define a Service resource to expose the microservice internally or externally with a Kubernetes Service YAML manifest. 

5. Apply the manifests

Use the kubectl apply command to apply the Deployment and Service manifests to your Kubernetes cluster. This will create the necessary resources and start the microservices.

6. Monitor and scale

Observability is especially important in microservices due to the challenges posed by the distributed and decentralized nature of microservices architecture. To ensure the best user experience, it is essential to have robust tools and observability practices in place. .

Once your observability tools are up and running, consider setting up Horizontal Pod Autoscaler (HPA) to automatically scale the number of replicas based on the metrics you gather on resource utilization.

7. Continuous integration and continuous deployment

Integrate your Kubernetes deployments into your CI/CD pipeline to enable automated testing, building, and deployment of microservices.

8. Service discovery and load balancing

Leverage Kubernetes’ built-in service discovery and load balancing mechanisms to allow communication between microservices. Services abstract the underlying Pods and provide a stable IP address and DNS name for accessing them.

9. Configure ingress controllers

If you need to expose your microservices to the external world, set up an Ingress Controller. This will manage external access and enable features like SSL termination and URL-based routing.

10. Manage configurations and secrets

Use ConfigMaps and Secrets to manage configurations and sensitive data separately from your container images. This allows you to change settings without redeploying the microservices.

11. Rolling updates and rollbacks

Utilize Kubernetes Deployments to perform rolling updates and rollbacks seamlessly. This allows you to release new versions of microservices without service disruption and easily revert to a previous stable version if needed.

12. Security best practices

Implement Kubernetes security best practices, such as Role-Based Access Control (RBAC), Network Policies, and Pod Security Policies, to protect your microservices and the cluster from potential threats.

What to find out more? Check out our introduction to Kubernetes observability for best observability practices with Kubernetes.

Everything You Need to Know About Log Management Challenges

Posted on by eugene evdokimov

Distributed microservices and cloud computing have been game changers for developers and enterprises. These services have helped enterprises develop complex systems easily and deploy apps faster.

That being said, these new system architectures have also introduced some modern challenges. For example, monitoring data logs generated across various distributed systems can be problematic.

With strong log monitoring tools and strategies in your developer’s toolkit, you’ll be able to centralize, monitor and analyze any wealth of data. In this article, we’ll first go over different log management issues you could potentially face down the line, and how to effectively overcome each one along the way.

Common log management problems

Monitoring vast data logs across a distributed system poses multiple challenges. When talking about a full-stack observability guide, here are some of the most common log management issues, and ways to fix them.

1. Your log management system is too complex

Overcomplexity is one of the primary causes of inefficient log systems. Traditional log monitoring tools are designed to handle data in a single monolithic system. Therefore, cross-platform interactions and integrations require the aid of third-party integration apps.

In the worst-case scenario, you might have to implement different integration procedures for different platforms to understand disparate outputs. This complicates your log monitoring system and drives up maintenance costs. 

Coralogix resolves this with a simple, centralized, and actionable log dashboard built for maximum efficiency. With a clear and simple graphical representation of your logs, you can easily drill down and identify issues. 

2. Dealing with an overwhelming amount of data 

Traditional legacy and modern cloud computing systems often produce vast amounts of unstructured data. Not just that, these different data formats are often incompatible with each other, resulting in data silos and hindered data integration efforts. The incompatibility between various data formats poses significant challenges for businesses in terms of data management, analysis, and decision-making processes.

Data volume also drives up the cost of traditional monitoring strategies. As your system produces more data, you will have to upgrade your monitoring stack to handle the increased volume. Having a modern log observability and monitoring tool can help you manage this data effectively.

You need an automated real-time log-parsing tool that converts data logs into structured events.  These structured events can help you extract useful insights into your system’s health and operating conditions. 

3. Taking too long to fix system bugs, leading to downtime

Log data is extremely useful for monitoring potential threats, containing time-stamped data of system conditions when incidents occur. However, the lack of visibility in distributed systems can make systems logs with bugs difficult to pinpoint. 

Therefore, you often have to spend a lot of time shifting through large amounts of data to system bugs. The longer it takes to find the bugs, the higher the likelihood that your system might face downtime. Modern distributed systems make this even harder, since system elements are scattered across many platforms. 

Coralogix’s real-time log monitoring dashboard helps you streamline this by providing a centralized view of the layers of connections between your distributed systems. This makes it possible to monitor and trace the path of individual requests and incidents without combing through tons of data logs. 

With this, you can greatly improve the accuracy of your log monitoring efforts, identify and resolve bugs faster and reduce the frequency of downtimes in your system.

4. Be proactive to prevent problems

Threat hunting and incident management is another common log monitoring problem. Traditional log monitoring software makes detecting threats in real time and deflecting them nearly impossible. 

In some situations, you only become aware of a threat after the system experiences downtime. Downtime has massive detrimental effects on a business, leading to loss of productivity, revenue and customer trust. Real-time log monitoring helps you resolve this by actively parsing through your data logs in real time and identifying unusual events and sequences. 

With a tool like Coralogix’s automated alerting system and AI prevention mechanism for log management, you can set up active alerts that are triggered by thresholds. The AI sets off alerts when your system encounters a previously unknown threshold. Thus, you can prevent threats before they affect your system.

Simplifying your log management system for better efficiency

Log monitoring is an essential task for forward-facing enterprises and developers. The simpler your log monitoring system, the faster you can find useful information from your data logs.

However, the data size involved in log management might make it challenging to eliminate problems manually. There are different log monitoring dashboards that can streamline your entire log monitoring journey. Choose the right one for your business. 

Analyzing Log Data: Why It’s Important

Posted on by eugene evdokimov

From production monitoring to security concerns, businesses need to know how to analyze logs on a daily basis to make sure their system is up to par. Here are the reasons why analyzing your log data is so important. According to Security Metrics, by performing log analysis and daily logging monitoring, you’ll be able to “demonstrate your willingness to comply with PCI DSS and HIPAA requirements, (and) it will also help you defend against insider and outsider threats.”

If you landed here, chances are you probably know what logs are, but we’ll start off with a short explanation of what it is. Typically, application logs capture timestamped data related to actions serviced by applications, decisions taken by applications, actions initiated by applications, and runtime characteristics of applications.

The analysis of log data, also known as data logging, is a process of making sense of computer-generated records (logs). This process helps businesses comply with security policies, audits or regulations, comprehend system troubleshoots as well as understand online user behavior. Businesses must review their logs daily to search for errors, anomalies, or suspicious activity that deviates from the norm. In fact, log analysis needs to interpret messages within the context of an application or system and map varying terminologies from log sources. It then turns them into a uniform terminology that sees to it that reports and statistics are clear.

So why is data logging analysis necessary? Well, here are some examples that will prove to you that it is not just important, but it is actually vital for any business that is looking to succeed, no matter the industry.

For Production Monitoring and Debugging:

Apps and systems are constantly growing in both size and complexity, and the use of logging platforms is now becoming a must for any growing business. By analyzing your key trends across your different systems, debugging as well as troubleshooting you’ll be able to create opportunities for improved operations on a smaller budget as well as new revenues. This is what Coralogix is all about: shortening the time a business needs to detect and solve production problems. The data, after all, is all there. It all depends on the ways in which organizations decide to utilize them to their advantage.

For Resource Usage:

When it comes to your system performance, often your software is not at fault, but rather your requests of the server are those that cause an overload your system has trouble dealing with. Tracking your resource usage will enable you to understand when the system is close to overload, so you’ll be able to prevent it from happening by adding additional capacity when needed.

For HTTP Errors:

A common use of log analysis is searching for HTTP errors. Through your analysis, you’ll be able to understand your HTTP errors, and on what pages they occurred so you can fix the problem and essentially prevent yourself from losing potential clients.

For Slow Queries:

By analyzing your log data, you’ll be able to detect when users are not getting the information they need or if this data is taking too long to load. By tracking slow queries, you’ll be able to see how your DB queries are performing and guarantee your user’s experience is up to par.

For Rogue Automated Robots:

If you’re under a DNS attack in which someone hammers your site to break your servers, your log data analysis will reveal a lot of useful information regarding your attackers. Your analysis will even be able to assist you in blocking them from accessing your site by their IP address. Search engine spiders are able to discover many errors that may not be noticed by your users but need to be promptly addressed.

For Security:

Log analysis may be the most under-appreciated, unsexy aspect of infosecurity. However, it is also one of the most important. From a security point of view, the purpose of a log is to act as a red flag when something bad is happening. As SANS Institute puts it,

“Logging can be a security administrator’s best friend. It’s like an administrative partner that is always at work, never complains, never gets tired, and is always on top of things. If properly instructed, this partner can provide the time and place of every event that has occurred in your network or system.”

Analyzing your logs regularly will allow your business a quicker response time to security threats and better security program effectiveness.

For Tracking Your Site’s/Platform’s Visitors:

Log data analysis will help you understand not only how many visitors have entered your site.platform, but on what pages they spent the most time, what they were doing on your site.platform, why there are changes in the number of visitors, etc. Trends and patterns like this will help you identify opportunities. As stated by Dave Talks, “examples include when to release a new version or product, when to send out a mailing or announcement, when to take down your website to test your new shopping cart, when to offer discounts, and much more.”

In short, analyzing your log data means you’ll be able to catch errors before your users have discovered them. Since your business is dealing with a vast amount of log data generated by your systems, using an ML-powered log analytics software is the best solution you could make if you don’t want to spend your time reviewing logs manually.

Coralogix’s Streama Technology: The Ultimate Party Bouncer

Posted on by eugene evdokimov

Coralogix is not just another monitoring or observability platform. We’re using our unique Streama technology to analyze data without needing to index it so teams can get deeper insights and long-term trend analysis without relying on expensive storage. 

So you’re thinking to yourself, “that’s great, but what does that mean, and how does it help me?” To better understand how Streama improves monitoring and troubleshooting capabilities, let’s have some fun and explore it through an analogy that includes a party, the police, and a murder!

Grab your notebook and pen, and get ready to take notes. 

Not just another party 

Imagine that your event and metric data are people, and the system you use to store that data is a party. To ensure that everyone is happy and stays safe, you need a system to monitor who’s going in, help you investigate, and remediate any dangerous situations that may come up. 

For your event data, that would be some kind of log monitoring platform. For the party, that would be our bouncer.

Now, most bouncers (and observability tools) are concerned primarily with volume. They’re doing simple ticket checks at the door, counting people as they come in, and blocking anyone under age from entering. 

As the party gets more lively, people continue coming in and out, and everyone’s having a great time. But imagine what happens if, all of a sudden, the police show up and announce there’s been a murder. Well, shit, there goes your night! Don’t worry, stay calm – the bouncer is here to help investigate. 

They’ve seen every person who has entered the room and can help the police, right?

Why can’t typical bouncers keep up?

Nothing ever goes as it should, this much we know. Crimes are committed, and applications have bugs. The key, then, is how we respond when something goes wrong and what information we have at our disposal to investigate.

Suppose a typical bouncer is monitoring our party, and they’re just counting people as they come in and doing a simple ID check to make sure they’re old enough to enter. In that case, the investigation process starts only once the police show up. At this point, readily-available information is sparse. You have all of these people inside, but you don’t have a good idea of who they are.

This is the biggest downfall of traditional monitoring tools. All data is collected in the same way, as though it carries the same potential value, and then investigating anything within the data set is expensive. 

The police may know that the suspect is wearing a black hat, but they still need to go in and start manually searching for anyone matching that description. It takes a lot of time and can only be done using the people (i.e., data) still in the party (i.e., data store). 

Without a good way to analyze the characteristics of people as they’re going in and out, our everyday bouncer will have to go inside and count everyone wearing a black hat one by one. As we can all guess, this will take an immense amount of time and resources to get the job done. Plus, if the suspect has already left, it’s almost like they were never there.

What if the police come back to the bouncer with more information about the suspect? It turns out that in addition to the black hat, they’re also wearing green shoes. With this new information, this bouncer has to go back into the party and count all the people with black hats AND green shoes. It will take him just as long, if not longer, to count all of those people again.

What makes Streama the ultimate bouncer?

Luckily, Streama is the ultimate bouncer and uses some cool tech to solve this problem.

Basically, Streama technology differentiates Coralogix from the rest of the bunch because it’s a bouncer that can comprehensively analyze the people as they go into the party. For the sake of our analogy, let’s say this bouncer has Streama “glasses,” which allow him to analyze and store details about each person as they come in.

Then, when the police approach the bouncer and ask for help, he can already provide some information about the people at the party without needing to physically go inside and start looking around.

If the police tell the bouncer they know the murderer had on a black hat, he can already tell them that X number of people wearing a black hat went into the party. Even better, he can tell them that without those people needing to be inside still! If the police come again with more information, the bouncer can again give them the information they need quite easily.  

In some cases, the bouncer won’t have the exact information needed by the police. That’s fine, they can still go inside to investigate further if required. By monitoring the people as they go in, though, the bouncer and the police can save a significant amount of time, money, and resources in most situations.

Additional benefits of Streama

Since you are getting the information about the data as it’s ingested, it doesn’t have to be kept in expensive hot storage just in case it’s needed someday. With Coralogix, you can choose to only send critical data to hot storage (and with a shorter retention period) since you get the insights you need in real-time and can always query data directly from your archive.

There are many more benefits to monitoring data in-stream aside from the incredible cost savings. However, that is a big one.

Data enrichment, dynamic alerting, metric generation from log data, data clustering, and anomaly detection occur without depending on hot storage. This gives better insights at a fraction of the cost and enables better performance and scaling capabilities. 

Whether you’re monitoring an application or throwing a huge party, you definitely want to make sure Coralogix is on your list!

5 Cybersecurity Tools to Safeguard Your Business

Posted on by eugene evdokimov

With the exponential rise in cybercrimes in the last decade, cybersecurity for businesses is no longer an option — it’s a necessity. Fuelled by the forced shift to remote working due to the pandemic, US businesses saw an alarming 50% rise in reported cyber attacks per week from 2020 to 2021. Many companies still use outdated technologies, unclear policies, and understaffed cybersecurity teams to target digital attacks.

So, if you’re a business looking to upgrade its cybersecurity measures, here are five powerful tools that can protect your business from breaches.

1. Access Protection

Designed to monitor outgoing and incoming network traffic, firewalls are the first layer of defense from unauthorized access in private networks. They are easy to implement, adopt, and configure based on security parameters set by the organization.

Among the different types of firewalls, one of the popular choices among businesses is a next-generation firewall. A next-generation firewall can help protect your network from threats through integrated intrusion prevention, cloud security, and application control. A proxy firewall can work well for companies looking for a budget option.

Even though firewalls block a significant portion of malicious traffic, expecting a firewall to suffice as a security solution would be a mistake. Advanced attackers can build attacks that can bypass even the most complex firewalls, and your organization’s defenses should catch up to these sophisticated attacks. Thus, instead of relying on the functionality of a single firewall, your business needs to adopt a multi-layer defense system. And one of the first vulnerabilities you should address is having unsecured endpoints.

2. Endpoint Protection

Endpoint Protection essentially refers to securing devices that connect to a company’s private network beyond the corporate firewall. Typically, these range from laptops, mobile phones, and USB drives to printers and servers. Without a proper endpoint protection program, the organization stands to lose control over sensitive data if it’s copied to an external device from an unsecured endpoint.

Softwares like antivirus and anti-malware are the essential elements of an endpoint protection program, but the current cybersecurity threats demand much more. Thus, next-generation antiviruses with integrated AI/ML threat detection, threat hunting, and VPNs are essential to your business.

If your organization has shifted to being primarily remote, implementing a protocol like Zero Trust Network Access (ZTNA) can strengthen your cybersecurity measures. Secure firewalls and VPNs, though necessary, can create an attack surface for hackers to exploit since the user is immediately granted complete application access. In contrast, ZTNA isolates application access from network access, giving partial access incrementally and on a need-to-know basis. 

Combining ZTNA with a strong antivirus creates multi-layer access protection that drastically reduces your cyber risk exposure. However, as we discussed earlier, bad network actors who can bypass this security will always be present. Thus, it’s essential to have a robust monitoring system across your applications, which brings us to the next point…

3. Log Management & Observability

Log management is a fundamental security control for your applications. Drawing information from event logs can be instrumental to identifying network risks early, mitigating bad actors, and quickly mitigating vulnerabilities during breaches or event reconstruction.

However, many organizations still struggle with deriving valuable insights from log data due to complex, distributed systems, inconsistency in log data, and format differences. In such cases, a log management system like Coralogix can help. It creates a centralized, secure dashboard to make sense of raw log data, clustering millions of similar logs to help you investigate faster. Our AI-driven analysis software can help establish security baselines and alerting systems to identify critical issues and anomalies. 

A strong log monitoring and observability system also protects you from DDoS attacks. A DDoS attack floods the bandwidth and resources of a particular server or application through unauthorized traffic, typically causing a major outage. 

With observability platforms, you can get ahead of this. Coralogix’s native Cloudflare integrations combined with load balancers give you the ability to cross-analyze attack and application metrics and enable your team to mitigate such attacks. Thus, you can effectively build a DDOS warning system to detect attacks early.

Along with logs, another critical business data that you should monitor regularly are emails. With over 36% of data breaches in 2022 attributed to phishing scams, businesses cannot be too careful.

4. Email Gateway Security

As most companies primarily share sensitive data through email, hacking email gateways is a prime target for cybercriminals. Thus, a top priority should be robust filtering systems to identify spam and phishing emails, embedded code, and fraudulent websites. 

Email gateways act as a firewall for all email communications at the network level — scanning and auto-archiving malicious email content. They also protect against business data loss by monitoring outgoing emails, allowing admins to manage email policies through a central dashboard. Additionally, they help businesses meet compliance by safely securing data and storing copies for legal purposes. 

However, the issue here is that sophisticated attacks can still bypass these security measures, especially if social engineering is involved. One wrong click by an employee can give hackers access to an otherwise robust system. That’s why the most critical security tool of them all is a strong cybersecurity training program.

5. Cybersecurity Training

Even though you might think that cybersecurity training is not a ‘tool,’ a company’s security measures are only as strong as the awareness among employees who use them. In 2021, over 85% of data breaches were associated with some level of human error. IBM’s study even found out that the breach would not have occurred if the human element was not present in 19 out of 20 cases that they analyzed.

Cybersecurity starts with the people, not just the tools. Thus, you need to implement a strong security culture about security threats like phishing and social engineering in your organization. All resources related to cybersecurity should be simplified and made mandatory during onboarding. These policies should be further reviewed, updated, and re-taught semi-annually in line with new threats. 

Apart from training, the execution of these policies can mean the difference between a hackable and a secure network. To ensure this, regular workshops and phishing tests should also be conducted to identify potential employee targets. Another way to increase the effectiveness of these training is to send out cybersecurity newsletters every week. 

Some companies like Dell have even adopted a gamified cybersecurity training program to encourage high engagement from employees. The addition of screen locks, multi-factor authentication, and encryption would also help add another layer of security. 

Upgrade Your Cybersecurity Measures Today!

Implementing these five cybersecurity tools lays a critical foundation for the security of your business. However, the key here is to understand that, with cyberattacks, it sometimes just takes one point of failure. Therefore, preparing for a breach is just as important as preventing it. Having comprehensive data backups at regular intervals and encryption for susceptible data is crucial. This will ensure your organization is as secure as your customers need it to be —  with or without a breach!

Python JSON Log Limits: What Are They and How Can You Avoid Them?

Posted on by eugene evdokimov

Python JSON logging has become the standard for generating readable structured data from logs. While monitoring logs in JSON is definitely much better than using the standard logging module, it comes with its own set of challenges. 

As your server or application grows, the number of logs also increases exponentially. It’s difficult to go through JSON log files, even if it’s structured, due to the sheer size of logs generated. These Python JSON log limits will become a real engineering problem for you.

Let’s dive into how log management solutions help with these issues and how they can help streamline and centralize your log management, so you can surpass your Python JSON log limits and tackle the real problems you’re looking to solve.

Python Log File Sizes

Based on the server you’re using, you’ll encounter server-specific log file restrictions due to the database constraints. 

For instance, AWS Cloudwatch skips the log event if the file size is larger than 256 KB. In such cases, especially with longer log files like JSON generates, retaining specific logs on the server is complex. 

The good news is, this is one of the easier Python JSON log limits to overcome. In some cases, you can avoid this by increasing the python log size limit configurations on the server level. However, the ideal log size limit for the server varies depending on the amount of data that your application generates. 

So how do you avoid this Python JSON Log limit on your files?

The solution here is to implement logging analytics via Coralogix. Through this platform, you can integrate and transform your logging data with any webhook and record vital data without needing to manage it actively. Since it is directly integrated with Python, your JSON logs can be easily parsed and converted.

Servers like Elasticsearch also roll logs after 256 MB based on timestamps. However, when you have multiple deployments, filtering them just based on the timestamp or on a file limit size becomes difficult. More log files can also lead to confusion and disk space issues.

To help tackle this issue, Coralogix cuts down on your overall development time by providing version benchmarks on logs and an intuitive visual dashboard.

Python JSON Log Formatting

Currently, programs use Python’s native JSON library or external libraries to implement JSON logging. Filtering these types of outputs needs additional development. For instance, you can only have name-based filtering natively, but if you want to filter logs based on time, severity, and so on, you’ll have to program those filters in. 

By using log management platforms, you can easily track custom attributes in the JSON log and implement specialized filters without having to do additional coding. You can also have alert mechanisms for failures or prioritized attributes. This significantly cuts down the time to troubleshoot via logs in case of critical failures. Correlating these attributes to application performance also helps you understand the bigger picture through the health and compliance metrics of your application.

Wrapping Up

Python JSON logging combined with a log management solution is the best way to streamline your logs and visualize them centrally. Additionally, you should also check out python logging practices to ensure that you format and collect the most relevant data. Your Python JSON logger limits will potentially distract you from adding value, and it’s important to get ahead of them.

If you want to make the most out of your Python JSON logs, our python integration should help!

10 Ways to Implement Effective IoT Log Management

Posted on by eugene evdokimov

The Internet of Things (IoT) has quickly become a huge part of how people live, communicate and do business. All kinds of everyday things make up this network – fridges, kettles, light switches – you name it. If it’s connected to WiFi, it’s part of the Internet of Things.

IoT raises significant challenges that could stand in your way of fully realizing its potential benefits. The matter of widespread adoption to a secure, functioning global device network still needs to be addressed. Plus, other concerns related to the hacking of Internet-connected devices and privacy fears have captured public attention.

Many of the challenges related to IoT are wide reaching and may be outside the scope of whatever you’re working on. That said, with effective IoT log management, you’ll be able to manage and troubleshoot these challenges and allow stakeholders to derive insights from data embedded in log events.

Key Challenges Facing IoT and How Your Logs Can Help You Handle Them

Here are some of the key IoT logging challenges along with some potential solutions that you can use to overcome them.

1. Log Management

In general, log management is especially important for IoT applications because of their dynamic, distributed, and fleeting nature. In a short period of time, IoT devices can generate millions of logged events. This in itself is a challenge. You’ll need to ensure that the data captured is accurate, that the common data types are standardized across all logs, and that the logs are protected.

Logs provide value to both troubleshooting and business insights. You can extract interesting metadata for optimization such as improving the onboarding process and making it more secure for connectivity purposes. In order to derive such insights, you’ll need to centralize your logs.

As IoT becomes more and more complex, so does the task of managing it. The goal is to get ahead of problems, and logging lets you do that. Rather than reacting to issues, proactively cut them off and fix them immediately.

IoT Log management has key functions that, if followed, will ensure your logging and monitoring go smoothly. This includes:

  • Log aggregation to centralized log storage. This means collectIng only the required logs from the needed source or endpoints and having dedicated servers that does buffering, parsing and enriching
  • Log search and analysis. Stored and indexed, your aggregated log files are now searchable
  • Log monitoring and alerting. Log management helps keep you on your toes, constantly providing data about how your IoT applications are performing

A log management policy for IoT will provide guidelines as to what types of actions need to be logged in order to trace inappropriate use (privacy), performance issues, and security breaches. 

2. Communication Protocols

Message Queuing Telemetry Transport (MQTT) is a very common example of a communication protocol widely used in IoT. A challenge with MQTT is exposed endpoints and the potential deployment of thousands of unsecure MQTT hosts. This results from a lack of secure configurations and the likelihood of misconfigurations in devices that use MQTT.

The use of any communication protocol of this nature has to ensure secure endpoints. Unsecure endpoints can expose records and leak information, some of which can be related to critical sectors, for any casual attacker to see. Then, of course, remains the risk of vulnerabilities that enable denial of service, or worse.

As MQTT does not check the data or payload that they transport, the information they carry can be really anything, posing data validation issues on the connected systems. Organizations should pay adequate attention to IoT security.

As an example, AWS IoT, part of Amazon Web Services (AWS), is essentially a managed MQTT service with strong and enforced security policies. It monitors AWS IoT using CloudWatch Logs to monitor, store, and access your log files. It can send progress events about each message, as it passes from your devices through the message broker and rules engine.  

Security teams with the right analytics tools, can use these captured logs for cyber forensic analysis. This can help to understand how to design secure IoT and ensure users do not connect an IoT device in an unsecure way. Otherwise cyber attackers will continue to take advantage of any exposed data that includes personal or potentially company sensitive information.

3. Application of Security in IoT

Each new IoT device provides a potential entry point for hackers to target your IoT network. Rather than allowing any device onto the network, new devices should be “provisioned”. This means you’ll need a robust, predictable process.

Data transmitted over IoT networks is at risk of being intercepted by criminal parties, so organizations should use only secure, password-protected wireless networks to ensure data is encrypted.

To guard against potential threats, organizations should build their networks with the assumption that any device connected to it is ’zero trust network’. Even if someone makes it into your network, they should still need authentication in order to access anything.

4. Connectivity Bottlenecks

The growth of IoT devices has already placed strain on many networks. Without the right ‘edge’ computing framework in place, company networks can become bogged down by latency and sub par bandwidth.

Device connectivity can be unreliable. 4G connections regularly disconnect and reconnect and don’t offer the same stability available to a typical broadband connection. Take a jogger with their smartwatch going out for a run, for example. They’re going from areas with strong connectivity to areas with poor connectivity and back again. Prolonged disconnections can result in the device running out of buffer memory to store its logs.

The biggest part of logging in IoT in these situations, is to understand where to store the generated data. Having a centralized log management system and a requirement that devices are connected to the Internet when they are updating, will ensure greater stability and reduce these types of bottlenecks.

It is important for companies developing IoT technology, to carefully examine their IoT connectivity providers and choose one with a strong record of service and innovation. If you want to take it to the next level, you can intelligently switch between networks, based on their relative strength at any given time.

5. Power Management

With a growing number of IoT devices comes growing power management requirements. Some IoT devices, like kitchen appliances, are located in accessible locations and draw on stable power sources. As we know, this isn’t always the case. Many devices rely solely on a battery for power.

Power consumption is not just a hardware issue. Greedy software can consume more resources than it needs and drain the limited power available to the device.

Power consumption is best captured using device log management and having a centralized location for those logs to be analyzed.  

Modern device data capture techniques integrating with cloud platform services, will help with power problems in IoT devices. Techniques captured from hardware-based power measurements, software-based power measurements embedded in devices and power tracking with anomaly detection, improve efficiency in power management requirements.  It will ensure the storage, RAM and CPU capacities of IoT devices are more effective and efficient in their use.       

The analysis of this data using forensics, security auditing, network tracing, or data analytics, enables the deep dive into power consumption details. This also gives context to historical power consumption.

6. Data Management

IoT networks generate huge amounts of data. Keeping track of all this data is a challenge in and of itself.

Edge computing can help here. Edge computing is an architectural decision to process data at or near the source. This pushes processing overhead down to the client, lowering the burden on some central system to keep track of everything. We do this instinctively in normal software, with fluentbit and fluentd transformations that format logs on the box, before sending them to a log collection server like Elasticsearch. 

Edge computing, data governance policies, and metadata management help organizations deal with issues of scalability and agility, security, and usability. This further assists them to decide whether to manage data on the edge or only after sending it to the cloud.

Organizations need to ensure they are collecting the specific data logging they are looking to isolate. They must then find the right software to keep track of this data and analyze it effectively. Whether in a centralized location or processed near the data source, the right storage is needed. Cloud storage is a solution but others options can rely on the local IoT device itself.

7. Device Management

From an organizational perspective, the advent of the IoT has made the range of devices IT needs to administer limitlessly. Devices need to be regularly patched and inspected to ensure they are at the highest possible level of performance and reliability. Remember, in an IoT system, someone can spill a glass of water and fry one of your devices. The hardware matters just as much as the software.

With the introduction of IoT device management software, this enables an onboarding process of device provisioning and provides a capability to monitor usage and performance metrics. The metrics captured locally and stored in a centralized data storage location for analytics purposes. 

This software provides secure on-boarding, organizing, monitoring, troubleshooting, and sending of firmware updates ‘over the air’ (OTA). It will assign them to devices and makes connected devices ready for service quickly. Device management software allows you to quickly zone in on one specific device, in a network of thousands.

8. Complexity of Data Captured

A major challenge of capturing IoT data is due to its complex nature. Often, organizations must not only prepare timestamp or geotag data, but combine it with more structured sources. Today an organization must figure out a way to leverage the resources they have in order to prepare the increasingly complex IoT data.

Organizations must equip their teams with data preparation platforms that can handle the volume and complexity of IoT data, as well as understand how this data can and will be joined with other sources across the organization. By adopting intelligent data preparation solutions and integrating them with a centralized logging repository, the universe of IoT and big data no longer overwhelms. This can be provided from IoT cloud services and ensures organizations are only collecting data that is useful for analytics, forensics, and intelligence purposes.

9. Threat of Cyber Attacks

One of the biggest security challenges is the creation of Distributed Destruction of Service (DDoS) attacks that employ swarms of poorly protected IoT devices, to attack public infrastructure through coordinated misuse of communication channels. An example is the use of IoT botnets that can direct enormous swarms of connected sensors, to cause damaging and unpredictable spikes in infrastructure use, leading to things like power surges, destructive water hammer attacks, or reduced availability of critical infrastructure on a wide scale.

A very large percentage of traffic from IoT devices to our honeypots is automated. A honeypot being the computer security mechanism set to detect, deflect, or counteract attempts at unauthorized use of information. This is a dangerous scenario, given that most modern bot armies and malware are scripted to attack at scale.

Centralizing all access logs will allow organizations to maintain all vulnerable devices under their control. The captured logs can be used for cyber forensic work and allows us to connect the dots and find correlations between events that may otherwise look unrelated.  

10. Compatibility and Updates

New waves of technology often feature a large stable of competitors jockeying for market share, and IoT is certainly no exception. When it comes to home automation using mesh networking, several competitors have sprung up to challenge Bluetooth’s mesh network offerings. Continued compatibility for IoT devices also depends upon users keeping their devices updated and patched. Unpatched IoT devices present serious security vulnerabilities and increase consumer risk.

Wrap-Up

IoT is one of the most exciting engineering developments of the past decade. It opens up a whole world of new capabilities and tooling, that can bring convenience and support to many consumers. With all of these new features, however, comes risk.

Without focusing on our observability responsibilities, a thousand disparate devices is a maintenance and security nightmare. Check out how Coralogix can make your life easier and consume all of those logs for you, in real-time. 

A Practical Guide to Logstash: Syslog Deep Dive

Posted on by eugene evdokimov

Syslog is a popular standard for centralizing and formatting log data generated by network devices. It provides a standardized way of generating and collecting log information, such as program errors, notices, warnings, status messages, and so on. Almost all Unix-like operating systems, such as those based on Linux or BSD kernels, use a Syslog daemon that is responsible for collecting log information and storing it. 

They’re usually stored locally, but they can also be streamed to a central server if the administrator wants to be able to access all logs from a single location. By default, port 514 and UDP are used for the transmission of Syslogs. 

Note: It’s recommended to avoid UDP whenever possible, as it doesn’t guarantee that all logs will be sent and received; when the network is unreliable or congested, some messages could get lost in transit.

For more security and reliability, port 6514 is often used with TCP connections and TLS encryption.

In this post, we’ll learn how to collect Syslog messages from our servers and devices with Logstash and send it to Elasticsearch. This will allow us to take advantage of its super-awesome powers of ingesting large volumes of data and then allowing us to quickly and efficiently search for what we need.

We’ll explore two methods. One involves using the Syslog daemon to send logs through a TCP connection to a central server running Logstash. The other method uses Logstash to monitor log files on each server/device and automatically index messages to Elasticsearch.

Getting Started

Let’s take a look at how typical syslog events look like. These are usually collected locally in a file named /var/log/syslog.

To display the first 10 lines, we’ll type:

sudo head -10 /var/log/syslog

Original image link

Let’s analyze how a syslog line is structured.

Original image link

We can see the line starts with a timestamp, including the month name, day of month, hour, minute and second at which the event was recorded. The next entry is the hostname of the device generating the log. Next is the name of the process that created the log entry, its process ID number, and, finally, the log message itself.

Logs are very useful when we want to monitor the health of our systems or debug errors. But when we have to deal with tens, hundreds, or even thousands of such systems, it’s obviously too complicated to log into each machine and manually look at syslogs. By centralizing all of them into Elasticsearch, it makes it easier to get a birds-eye view over all of the logged events, filter only what we need and quickly spot when a system is misbehaving.

Collecting syslog Data with Logstash

In this post, we’ll explore two methods with which we can get our data into Logstash logs, and ultimately into an Elasticsearch index:

  1. Using the syslog service itself to forward logs to Logstash, via TCP connections.
  2. Configuring Logstash to monitor log files and collect their contents as soon as they appear within those files.

Forwarding Syslog Messages to Logstash via TCP Connections

The syslog daemon has the ability to send all the log events it captures to another device, through a TCP connection. Logstash, on the other hand, has the ability to open up a TCP port and listen for incoming connections, looking for syslog data. Sounds like a perfect match! Let’s see how to make them work together.

For simplicity, we will obviously use the same virtual machine to send the logs and also collect them. But in a real-world scenario, we would configure a separate server with Logstash to listen for incoming connections on a TCP port. Then, we would configure the syslog daemons on all of the other servers to send their logs to the Logstash instance.

Important: In this exercise, we’re configuring the syslog daemon first, and Logstash last, since we want the first captured logged events to be the ones we intentionally generate. But in a real scenario, configure Logstash listening on the TCP port first. This is to ensure that when you later configure the syslog daemons to send their messages, Logstash is ready to ingest them. If Logstash isn’t ready, the log entries sent while you configure it, won’t make it into Elasticsearch.

We will forward our syslogs to TCP port 10514 of the virtual machine. Logstash will listen to port 10514 and collect all messages.

Let’s edit the configuration file of the syslog daemon.

sudo nano /etc/rsyslog.d/50-default.conf

Above the line “#First some standard log files. Log by facility” we’ll add the following:

*.*                         @@127.0.0.1:10514

Original image link here

*.* indicates to forward all messages. @@  instructs the rsyslog utility to transmit data through TCP connections.

To save the config file, we press CTRL+X, after which we type Y and finally press ENTER.

We’ll need to restart the syslog daemon (called “rsyslogd”) so that it picks up on our desired changes.

sudo systemctl restart rsyslog.service

If you don’t have a git tool available on your test system, you can install it with:

sudo apt update && sudo apt install git

Now let’s clone the repo which contains the configuration files we’ll use with Logstash.

sudo git clone https://github.com/coralogix-resources/logstash-syslog.git /etc/logstash/conf.d/logstash-syslog

Let’s take a look at the log entries generated by the “systemd” processes.

sudo grep "systemd" /var/log/syslog

Original image link here

We’ll copy one of these lines and paste it to the https://grokdebug.herokuapp.com/ website, in the first field, the input section.

Original image link here

Now, in a new web browser tab, let’s take a look at the following Logstash configuration: https://raw.githubusercontent.com/coralogix-resources/logstash-syslog/master/syslog-tcp-forward.conf.

Original image link

We can see in the highlighted “input” section how we instruct Logstash to listen for incoming connections on TCP port 10514 and look for syslog data.

To test how the Grok pattern we use in this config file matches our syslog lines, let’s copy it

%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}

and then paste it to the https://grokdebug.herokuapp.com/ website, in the second field, the pattern section.

Original image link

We can see every field is perfectly extracted.

Now, let’s run Logstash with this configuration file.

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-syslog/syslog-tcp-forward.conf

Since logs are continuously generated and collected, we won’t stop Logstash this time with CTRL+C. We’ll just leave it running until we see this:

Original image link

Specifically, we’re looking for the “Successfully started Logstash” message.

Let’s leave Logstash running in the background, collecting data. Leave its terminal window open (so you can see it catching syslog events) and open up a second terminal window to enter the next commands.

It’s very likely that at this point no syslog events have been collected yet, since we just started Logstash. Let’s make sure to generate some log entries first. A simple command such as

sudo ls

will ensure we’ll generate a few log messages. We’ll be able to see in the window where Logstash is running that sudo generated some log entries and these have been added to the Elasticsearch index.

Let’s take a look at an indexed log entry.

curl -XGET "https://localhost:9200/syslog-received-on-tcp/_search?pretty" -H 'Content-Type: application/json' -d'{"size": 1}'

The output we’ll get will contain something similar to this:

        {
        "_index" : "syslog-received-on-tcp",
        "_type" : "_doc",
        "_id" : "fWJ7QXMB9gZX17ukIc6D",
        "_score" : 1.0,
        "_source" : {
          "received_at" : "2020-07-12T05:24:14.990Z",
          "syslog_message" : " student : TTY=pts/1 ; PWD=/home/student ; USER=root ; COMMAND=/bin/ls",
          "syslog_timestamp" : "2020-07-12T05:24:14.000Z",
          "message" : "<85>Jul 12 08:24:14 coralogix sudo:  student : TTY=pts/1 ; PWD=/home/student ; USER=root ; COMMAND=/bin/ls",
          "syslog_hostname" : "coralogix",
          "port" : 51432,
          "type" : "syslog",
          "@timestamp" : "2020-07-12T05:24:14.990Z",
          "host" : "localhost",
          "@version" : "1",
          "received_from" : "localhost",
          "syslog_program" : "sudo"
        }

Awesome! Everything worked perfectly. Now let’s test out the other scenario.

Monitoring syslog Files with Logstash

We’ll first need to stop the Logstash process we launched in the previous section. Switch to the terminal where it is running and press CTRL+C to stop it.

Let’s open up this link in a browser and take a look at the Logstash config we’ll use this time: https://raw.githubusercontent.com/coralogix-resources/logstash-syslog/master/logstash-monitoring-syslog.conf.

Original image link

We can see that the important part here is that we tell it to monitor the “/var/log/syslog” file.

Let’s run Logstash with this config.

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-syslog/logstash-monitoring-syslog.conf

As usual, we’ll wait until it finishes its job and then press CTRL+C to exit the process.

Let’s see the data that has been parsed.

curl -XGET "https://localhost:9200/syslog-monitor/_search?pretty" -H 'Content-Type: application/json' -d'{"size": 1}'

We will get an output similar to this:

        {
        "_index" : "syslog-monitor",
        "_type" : "_doc",
        "_id" : "kmKYQXMB9gZX17ukC878",
        "_score" : 1.0,
        "_source" : {
          "type" : "syslog",
          "@version" : "1",
          "syslog_message" : " [origin software="rsyslogd" swVersion="8.32.0" x-pid="448" x-info="https://www.rsyslog.com"] rsyslogd was HUPed",
          "syslog_hostname" : "coralogix",
          "message" : "Jul 12 05:52:46 coralogix rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="448" x-info="https://www.rsyslog.com"] rsyslogd was HUPed",
          "received_at" : "2020-07-12T05:55:49.644Z",
          "received_from" : "coralogix",
          "host" : "coralogix",
          "syslog_program" : "rsyslogd",
          "syslog_timestamp" : "2020-07-12T02:52:46.000Z",
          "path" : "/var/log/syslog",
          "@timestamp" : "2020-07-12T05:55:49.644Z"
        }

Clean-Up Steps

To clean up what we created in this exercise, we just need to delete the two new indexes that we added

curl -XDELETE "https://localhost:9200/syslog-received-on-tcp/"

curl -XDELETE "https://localhost:9200/syslog-monitor/"

and also delete the directory where we placed our Logstash config files.

sudo rm -r /etc/logstash/conf.d/logstash-syslog

Conclusion

As you can see, it’s fairly easy to gather all of your logs in a single location, and the advantages are invaluable. For example, besides making everything more accessible and easier to search, think about servers failing. It happens a little bit more often than we like. If logs are kept on the server, once it fails, you lose the logs. Or, another common scenario, is that hackers delete logs once they compromise a machine. By collecting everything into Elasticsearch, though, you’ll have the original logs, untouched and ready to review to see what happened before the machine experienced problems.

What You Need to Know About IoT Logging

Posted on by eugene evdokimov

The Internet of Things (or, IoT) is an umbrella term for multiple connected devices sharing real-time data, and IoT logging is an important part of this. Troubleshooting bug fixes, connection problems, and general malfunctions rely heavily on logs, making them an invaluable asset not only in designing systems but also in system maintenance. 

To maximize system potential, this plethora of generated data needs to be managed efficiently. In this post, we’ll look at the different types of logs involved in IoT logging, different storage options and some common issues you may face.  

IoT Logging

Types of Logs 

IoT logging has many different flavors. Some are asynchronous and need to be stored only periodically whereas others need to be synchronous to ensure device uptime. Below are some of the many types of logs involved in IoT logging. 

Status Log

Status logs show the state of the device and whether it is online, offline, transmitting, or in an error state. They are important to give the user a holistic picture of the general state of the device(s). They’re usually stored and sent in frequent and regular intervals.

Error Log

Error logs are more specific than the status log and should generally trigger an alert for monitoring purposes. Errors mean downtime and that should be avoided. A good error log should provide contextual information such as what caused the error and where it occurred (a particular line of code, for instance). Error logs are usually asynchronous and sent whenever there is an error (provided internet connectivity has not been hindered). 

Authentication Log

Authentication logs enable you to see if a registered user(s) is logged in or not. It may be unfeasible to store each login attempt (as end-users might log in multiple times a day), but unsuccessful login attempts can be monitored to determine who is trying to gain access to the system/device.

Configuration Log 

Device attributes are pertinent to keep track of in case of future updates and bug fixes. A configuration log helps track all the different attributes for various IoT devices. This may not be useful for the end-user but it could be of vital importance for developers. If the configuration only really changes with a software update then it is worth storing and retrieving configuration logs asynchronously (i.e., with each update or downgrade). 

Memory Dump 

If you have a software crash, a memory dump or crash dump is particularly useful to determine what went wrong and where. In Microsoft Windows terminology, a memory dump file contains a small amount of information such as the stop message and its data and parameters, a list of loaded drivers, the processor context for the processor which stopped, and so on.

IoT Logging Storage

Given that many of these IoT logging types are needed retroactively, the next question is about where the logs will be stored. You have two options here, local (on-device) storage or cloud storage. Both have their own merits and may be more or less suitable depending on the situation.

On-Device Storage

On-device storage of logs is a highly scalable approach, only in as far as the number of devices is concerned. It is not affected by the number of devices as each device saves its own logs on local storage. This also means that each device will need manual intervention if there is downtime or if it runs out of memory for log storage.

Furthermore, storing logs locally requires a physical connection to a remote computer or bridge for download/upload of data. This may impact user perception of the device and may not be possible if devices cannot be accessed easily or if there are many devices. 

Cloud Storage

Cloud storage is the preferred option if you want immediate feedback and timely information about device status and performance. This approach is more scalable but relies on the existence of a fully functional log management system. 

The log management system should be able to aggregate data from many heterogeneous devices transmitting in real-time and process, index, and store them in a database that facilitates visualization through charts, dashboards, or other means.

Common Problems with IoT Logging

With many devices transmitting data over potentially unstable connections, guaranteeing a certain level of Quality of Service (QoS) becomes a real challenge. If you cannot get vital information about device downtime promptly, then the QoS rapidly declines. Below are some commonly encountered logging issues that arise with IoT devices.

Network Dropping

Lack of internet connectivity is among the most commonly encountered IoT logging issues. There could be many reasons for this including network congestion, lack of bandwidth, poor connection with wireless devices, and firewall issues. Moving the device to an area with better Wi-Fi strength, an antenna upgrade, and limiting the simultaneous number of connections (MAC address filtering) can help solve some of these issues.

Log Buffering

Log buffering for IoT devices is important, especially in instances when the network drops. Determining the right size for your log buffer is just as important, as it can have serious implications when issues arise. A smaller log buffer saves storage, but will contain fewer log messages which can impact your ability to troubleshoot network issues.

Latency

Latency can have far-reaching consequences, especially when it comes to system maintenance. In cases where a cyclic status message is received a few hours late, it can impact your ability to correctly troubleshoot an issue. To get around this, the device latency can be calculated by subtracting the server latency from end-to-end latency. This can help illustrate if the problem is with the device or with the server.   

Conclusion

IoT logging is a vital part of any system. Its function in system development and debugging cannot be understated. Using a centrally managed logging system for IoT devices has many advantages and can go a long way towards ensuring device downtime is kept to a minimum. 

Coralogix provides a fully managed log analytics solution for all of your IoT logging requirements. Tools like Loggregation for log clustering, benchmark reporting for build quality, and advanced anomaly detection alerts are all features to help you run an efficient and stable IoT system.

Minimal downtime is one of the hallmarks of a great product/service and a functioning and Coralogix can help achieve it.

5 Essential MySQL Database Logs To Keep an Eye On

Posted on by eugene evdokimov

MySQL is the de facto choice for open-source relational databases, and you should learn how to use MySQL database logs to improve efficiency and security. As an open-source product, it is free to use and has a large and active developer community. It is crucial to understand how to diagnose and monitor the performance of a MySQL instance in the long run.

Logs to the rescue!

Why is logging important?

A highly available and performant database is essential for an application’s performance. While using a MySQL instance in production, you will come across issues like slow queries, deadlocks, and aborted connections. Logging is essential to diagnosing these issues. A good understanding of your MySQL database logs will help you improve operations by reducing the mean time to recovery and the mean time between failures.

Log monitoring is also key to detecting and diagnosing security issues within your MySQL instance. In the event of a compromise, logs track the details of an attack and the actions taken by the attackers. This information provides context to your data and helps you take remedial action.

Is monitoring logs a complex affair?

Logging is often ignored because analyzing logs is considered a complex activity. However, monitoring logs from a MySQL instance isn’t a complex task, provided you know which variables to watch and where to find them.

If your MySQL instance is generating a large amount of log data each day, it might not be feasible to review all of them manually. You can automate the review process by using log monitoring software that can pinpoint problematic events. Some monitoring software can even be configured to send out email alerts when something suspicious is detected.

In this post, we’ll discuss five important logs and the specific ways in which they can help you monitor your MySQL instance.

  1. General Query Log
  2. Slow Query Log
  3. Error Log
  4. Binary Log
  5. Relay Log

Enable Logging on MySQL

Before moving ahead, it’s important to note that logging is disabled by default on MySQL except for the error log. Let’s take a quick look at how to enable the general and slow query logs.

To start using MySQL commands, open your command prompt, and log in to your MySQL instance with the following command.

mysql -u root -p

First, check the current state of the system variables by using the command

mysql> show variables;

If variables like general_log and slow_query_log are OFF, we need to switch them on.

general database logs

slow query database logs
You can enable the general query log with the following command. The default name of the log file is host_name.log but you can change the name and path as required.

mysql>SET GLOBAL general_log = ‘ON’;
mysql>SET GLOBAL general_log_file = ‘path_on_your_system’;

The slow query log can be enabled with the commands below.

mysql>SET GLOBAL slow_query_log = ‘ON’;
mysql>SET GLOBAL slow_query_log_file = ‘path_on_your_system’;

You can also control the destination of logs by setting the value of the log_output variable to FILE TABLE or FILE,TABLE . FILE selects logging to log files while TABLE selects logging to the MySQL system schema.

Now, let’s see which 5 logs you’ll want to keep an eye out for in your MySQL instance!

1. General Query Log

As the name implies, the general query log is a general record of what MySQL is doing. Information is written to this log when clients connect or disconnect to the server. The server also logs each SQL statement it receives from clients. If you suspect an error in a client, you can know exactly what the client sent to the MySQL instance by looking at the general query log.

You should be aware that MySQL writes statements to the general query log in the order in which it receives them. The order might differ from the order in which the queries are executed because, unlike other log formats, the query is written to this log file before MySQL even attempts to execute the query. MySQL database logs are therefore perfect for debugging MySQL crashes.

Since the general query log is a record of every query received by the server, it can grow large quite quickly. If you only want a record of queries that change data, it might be better to use the binary log instead (more on that later).

Impact on performance

In terms of performance, enabling the general query log does not have a noticeable impact on performance in most cases. However, it has been observed that writing logs to a file is faster than writing them to a table. If you want a detailed analysis of the performance impact of the general query, you can go through this article which goes into greater depth on this.

Viewing MySQL Database Logs on the workbench

To view logs on the MySQL workbench, go to the ‘Server’ navigation menu and then choose ‘Server Logs’. The following picture shows an example of entries in a general log file.

Viewing MySQL database logs on the workbench

2. Slow Query Log

As applications scale in size, queries that were once extremely fast can become quite slow. When you’re debugging a MySQL instance for performance issues, the slow query log is a good starting place to see which queries are the slowest and how often they are slow.

The slow query log is the MySQL database log queries that exceed a given threshold of execution time. By default, all queries taking longer than 10 seconds are logged. 

Configuration options

You can change the threshold query execution time by setting the value of the long_query_time system variable. It uses a unit of seconds, with an optional milliseconds component.

SET GLOBAL long_query_time = 5.0;

To verify if the slow query log is working properly, you can execute the following query with a time greater than the value of the long_query_time .

SELECT SLEEP(7);

Queries not using indexes are often good candidates for optimization. The log_queries_not_using_indexes system variable can be switched on to make MySQL log all queries that do not use an index to limit the number of rows scanned. In this case, logging occurs regardless of execution time of the query.

Parsing slow query logs

For large applications, the slow query log can become difficult to investigate. Fortunately, MySQL has a tool called mysqldumpslow which parses the slow query log files and prints a summary result with similar queries grouped. Normally, mysqldumpslow groups queries that are similar except for the particular values of number and string data values.

Finally, you should understand that not every query logged on the slow query log needs to be optimized. A query that takes long to run but is only run once a month is probably not a source of concern. On the other hand, a query that has a smaller execution time but runs thousands of times an hour may be a good candidate for optimization.

The following picture shows an example of entries in a slow query log file.

Entries for slow query logs

3. Error Log

MySQL uses the error log to record diagnostic messages, warnings, and notes that occur during server startup and shutdown, and while the server is running. The error log also records MySQL startup and shutdown times.

Error logging is always enabled. On Linux, if the destination is not specified, the server writes the error log to the console and sets the log_error system variable to stderr. On Windows, by default, the server writes the error log to the host_name.err file in the data directory. You can customize the path and file name of the error log by setting the value of the log_error system variable.

Commonly logged errors

Some common errors that MySQL logs to the error log are as follows:

  • Permission errors
  • Configuration errors
  • Out of memory errors
  • Errors with initiation or shutdown of plugins and InnoDB

Filtering the error log

MySQL database logs have the option to filter error logs, should you want to focus on critical errors. You can set the verbosity of the error log by changing the value of the log_error_verbosity system variable. Permitted values are 1 (errors only), 2 (errors and warnings), 3 (errors, warnings, and notes), with a default of 3.

MySQL 8.0 also provides error filtering based on user-defined rules using the log_filter_dragnet system variable. You can read more about how to enable this filtering on the official MySQL documentation.

The picture below shows a snippet of the error log.

MySQL error logs

4. Binary Log

The binary log is used by MySQL to record events that change the data within the tables or change the table schema itself. For example, binary logs record INSERT, DELETE and UPDATE statements but not SELECT or SHOW statements that do not modify data. Binary logs also contain information about how long each statement took to execute.

The logging order of a binary login is in contrast with that of the general query log. Events are logged only after the transaction is committed by the server.

MySQL writes binary log files in binary format. To read their contents in text format, you need to use the mysqlbinlog utility. For example, you can use the code below to convert the contents of the binary log file named binlog.000001 to text.

mysql> mysqlbinlog binlog.0000001

Purpose of the binary log

The primary purpose of the binary log is to keep a track of changes to the server’s global state during its operation. Thus binary log events can be used to reproduce the changes which have happened on a server earlier. The binary log has two important applications.

  • Replication: The binary log is used on the main server to record all of the events that modify database structure or content. Each replica that connects to the main server requests a copy of the binary log. The replica then executes the events from the binary log in order to reproduce the changes just as they were made on the main server.
  • Data Recovery: Data recovery operations also make use of binary logs. When a database is restored from a backup file, the events in the binary log that were recorded after the backup was made are re-executed. This makes the restored database up to date with the original.

Binary logging formats

MySQL database logs offer three formats for binary logging.

  • Statement-based logging: In this format, MySQL records the SQL statements that produce data changes. Statement-based logging is useful when many rows are affected by an event because it is more efficient to log a few statements than many rows.
  • Row-based logging: In this format, changes to individual rows are recorded instead of the SQL statements. This is useful for queries that require a lot of execution time on the source but result in just a few rows being modified.
  • Mixed logging: This is the recommended logging format. It uses statement-based logging by default but switches to row-based logging when required.

The binary logging format can be changed using the code below. However, you should note that it is not recommended to do so at runtime or while replication is ongoing.

SET GLOBAL binlog_format = 'STATEMENT';
SET GLOBAL binlog_format = 'ROW';
SET GLOBAL binlog_format = 'MIXED';

Enabling binary logging on your MySQL instance will lower the performance slightly. However, the advantages discussed above generally outweigh this minor dip in performance.

5. Relay Log

Relay logs are a set of numbered log files created by a replica during replication from the main server. Relay logs also consist of an index file that contains the names of all used relay log files.

During replication, the replica reads events from the main server’s binary log and writes it onto its relay logs. It then performs all the events in order to be in sync with the main server. After all events in the file have been executed, the replication SQL thread automatically deletes relay log files that are no longer needed.

The format of the relay log is the same as that of the binary log, so the mysqlbinlog utility can be used to display its contents.

Configuration options

By default, relay log names are of the form host_name-relay-bin.nnnnnn where host-name is the name of the replica server and #nnnnnn is the sequence number. The default filename for the relay log index file is host_name-relay-bin.index. Both relay log files and the relay log index file are stored in the data directory.

The filename and path of relay logs and the relay log index file can be changed by setting the relay_log and relay_log_index system variables respectively. This is useful if you anticipate that the replica’s hostname might change from time to time.

Managing Log Files

Over time, MySQL database logs become large and cumbersome. It is necessary to manage log files for two important reasons. 

First, you need to restrict the volume of log data to prevent old logs from taking up too much of your disk space. Second, breaking up your log files into smaller organized files makes troubleshooting and analyzing them much simpler. 

Log rotation is a simple way to achieve this.

Log Rotation

Log rotation is the process in which the current log file is renamed, usually by appending a “1” to the name, and a new log file is set up to record new log entries. Each time a new log file is started, the numbers in the file names of old log files are increased by one. Based on the threshold of files to be retained, old log files are then compressed, deleted, or archived separately to save space.

Depending on your requirements, you can decide on the maximum size of a log file, the frequency of rotations, and how many old log files you want to retain.

On Linux, the logrotate utility can be used to automate rotation compression, removal, and mailing of log files. Logrotate can be run as a scheduled job daily or weekly or when the log file gets to a certain size. On other systems, you can run a similar script using a scheduler.

Conclusion

Logging is an indispensable tool for managing your MySQL instances. Gaining a deeper understanding of the five logs mentioned above will help you preempt, diagnose, and monitor issues as your application scales. 

Although it may seem daunting at first, you will get better at troubleshooting and debugging logs over time. Start using them today!

Is your logging ready for the future?

Posted on by eugene evdokimov

Log scaling is something that should be top of mind for organizations seeking to future-proof their log monitoring solutions. Logging requirements will grow through use, particularly if not maintained or utilized effectively. There are barriers to successful log scaling, and in this post we’ll be discussing storage volume problems; increased load on the ELK stack, the amount of ‘noise’ generated by a growing ELK stack, and the pains of managing burgeoning clusters of nodes.

Growing Storage Requirements

Capacity is one of the biggest considerations when looking at scaling your ELK stack. Regardless of whether you are looking at expanding your logging solution, or just seeking to get more out of your existing solution, storage requirements are something to keep in mind.

If you want to scale your logging solution, you will increasingly lean on your infrastructure engineers to grow your storage, better availability and provisioning backups – all of which are vital to support your ambitions for growth. This is only going to draw attention away from their primary functions, which include supporting the systems and products which are generating these logs! You may find yourself needing to hire or train staff specifically to support your logging aspirations – this is something to avoid. Log output volumes can be unpredictable, and your storage solution should be elastic enough to support ELK stack scaling.

If you’re reading this then it is likely you are looking at log scaling to support some advanced analytics, potentially using machine learning (ML). For this to be effective, you need the log files to be in a highly available storage class. To optimize your storage provisioning for future ELK stack requirements, look at Coralogix’s ELK stack SaaS offering. Coralogix’s expertise when it comes to logging means that you can focus on what you want to do with your advanced logging, and we will take care of the future proofing.

Increased noise in the logs

So you want to scale your log solution? Well you need to prepare for the vast volume of logs that a scaled ELK stack is going to bring with it. If you aren’t sure exactly how you plan to glean additional insights from your bigger and better ELK stack deployment, then it most certainly isn’t of much use. 

In order to run effective machine learning algorithms over your log outputs, you need to be able to define what is useful and what is not. This becomes more problematic if you have yet to define what is a “useful” log output, and what is not. The costs associated with getting your unwieldy mass of logs into a workable database will quickly escalate. Coralogix’s TCO calculator (https://coralogixstg.wpengine.com/tutorials/optimize-log-management-costs/) will give you the ability to take stock over what is useful now, and help you to understand what outputs will be useful in the future, making sure that your scaled log solution gives you the insights you need.

Load on the ELK Stack

Optimizing the performance of each of the constituent parts of the ELK stack is a great way of fulfilling future logging-related goals. It isn’t quite as easy as just “turning up” these functions – you need to make sure your ELK stack can handle the load first.

The Load on Logstash

You can adjust Logstash to increase parsing, but this increases the ingestion load. You need to ensure that you have allocated sufficient CPU capacity for the ingestion boxes, but a common Logstash problem is that it will hoover up most of your processing capacity unless properly configured. Factor in your ingestion queue and any back pressure caused by persistent queues. Both of these issues are not only complex in resolution, but will hamper your log scaling endeavours. 

The Load on Elasticsearch

The load on Elasticsearch when scaled can vary greatly depending on how you choose to host it. With Elasticsearch scaling on-prem, failure to properly address and configure the I/O queue depth will grind your cluster to a total standstill. Of bigger importance is the compressed object pointer – if you approach its 32GB heap limit (although Elastic still isn’t certain that the limit isn’t even lower, elastic.co/blog/a-heap-of-trouble) then performance will deteriorate. Both of these Elasticsearch concerns compound with the volume or extent of ELK stack scaling you are attempting, so perhaps you should delegate this out to Coralogix’s fully managed ELK stack solution.

The Configurational Nuances of Log Scaling

Whilst forewarned is forearmed when it comes to these log scaling problems, you’re only well prepared if you have fully defined how you plan to scale your ELK stack solution. The differences and considerations for a small architecture logging solution scaling upward are numerous and nuanced: discrete parsing tier or multiple parsing tier; queue mediated ingestion or multiple event stores represent just a few of the decisions you have to make when deciding to scale. Coralogix have experience in bringing logging scalability to some of the industry’s biggest and most complex architectures. This means that whatever future proofing issues may present themselves, Coralogix will have seen them before.

Increased cluster size

The last tenet of log scaling to be mindful of is the implication of having larger clusters with bigger or more nodes. This brings with it a litany of issues guaranteed to cause you headaches galore if you lack some serious expertise.

Every time you add a node to a cluster, you need to ensure that your network settings are (and remain) correct, particularly when running Logstash, Beats or Filebeats on separate ELK and client servers.. You need to ensure that both firewalls are correctly configured, and this becomes an enterprise-sized headache with significant log scaling and cluster augmentation. An additional networking trap is the maintenance of Elasticsearch’s central configuration file. The larger the ELK stack deployment, the greater the potential for mishap in the config of this file, where the untrained or unaware will get lost in a mess of ports and hosts. At best, you have the possibility of networking errors and a malfunctioning cluster, but at worst, you will have an unprotected entrypoint into your network. 

Scaling isn’t as easy as it sounds

Adding more nodes, if done correctly, will fundamentally make your ELK stack more powerful. This isn’t as simple as it sounds as every node needs to be balanced to run at peak performance. Whilst Elasticsearch will try and balance shard allocation for you, this is a “one size fits all” configuration, and may not get the most out of your cluster. Shard allocation can be manually defined, but this is an ongoing process that changes with new indices, new clusters and any network changes. 

Where does Coralogix come in?

Coralogix are experts when it comes to log scaling and future proofing ELK stack deployments, dealing with shards, nodes, clusters and all of the associated headaches on a daily basis. The entire Coralogix product suite is designed to make log scaling, both up and down, headache free and fully managed. 

The Pivotal Role of Log Analytics in Modern IT Infrastructures

Posted on by eugene evdokimov

In this survey of over 200 CIOs in the US, the IDC analyses the critical role played by Log Analytics in any modern infrastructure.

IDC Opinion

To effectively manage modern IT environments, organizations are reliant on their ability to gain actionable insight from machine-generated data. Log monitoring, collection and analysis of log data has long been a common practice for achieving operational visibility. However, traditional log analytics solutions were not designed to handle the huge volume and variety of log data generated by enterprises today, driven by the massive adoption of cloud, mobile, social media, IoT devices, AI, machine learning, and other data-intensive technologies. Thus, organizations are faced with challenges such as mounting costs of log data storage, integration difficulties, multiple false positives, the need for customizations, and manual querying and correlation.

According to a recent IDC survey, 50% of all organizations process more than 100GB of log data of different formats per day, and the numbers continue to rise. Given that, IDC believes that the next generation of log analytics should offer advanced analytics and correlation capabilities as well as improved performance to keep up with scaling demands, leveraging the cloud to provide access to innovative technologies in a cost-effective manner. Based on these capabilities, log management can have a pivotal role in streamlining modern software development practices such as DevOps and CI/CD.

Methodology

This study examines the role of log management and analytics in modern IT environments. To gain insight into log management requirements, preferences, priorities, and challenges, IDC conducted a survey with 200 IT decision makers in enterprises of all sizes in key industries in the U.S.

Situation Overview

Since the very early days of computing, organizations have used log data generated from their IT environments for various operational needs such as monitoring and troubleshooting application performance and network problems, investigating security incidents, improving customer experience, and more. While log data plays an increasingly important role across a range of use cases, capturing and acting upon it has become a significant challenge.

Traditionally, the collection of log data was done using homegrown scripts, which involved time- and labor-intensive manual processes. With the transition from monolithic applications to distributed multi-tiered architectures, this approach became practically infeasible.

In response, the first generation of log management tools emerged to automate the processes of collecting, storing and managing logs. Still, making sense out of the data remained an issue. Users had to know exactly what to look for and make their own correlations to get the results they wanted. As IT environments continued to grow in scale and complexity, log management solutions offered new functionalities such as the ability to run queries to retrieve events based on keywords, specific values or values within a specific range, visualization of data to facilitate search, and more. Although early log management solutions required specialized skills such as knowledge of vendor-specific search languages and techniques for framing log-specific inquiries, they were very useful in times when IT environments used to consist of systems developed by a relatively small number of suppliers.

Today, to keep up with the ongoing transformation of the IT industry, log management should take another leap forward.

Over the years, log management achieved mainstream adoption. According to a recent IDC survey, 90% of organizations of all sizes are either using or planning to use a log management solution. Adoption is particularly strong among software vendors (around 98% are currently using or planning to use) and financial services companies (90%).

Figure 1

Log Management and Analytics Adoption

  1. Does your organization use, or is it planning to use, any tool or solution to manage log data?
percentage of companies using logs
Note: N = 180
Source: IDC, 2019

Log Management and Analytics Challenges

Despite the extensive use of log management, IDC’s survey revealed that many organizations are still struggling with different issues that compromise their ability to convert log data into actionable insights. In particular, the participants in the survey highlighted cost of data storage (mentioned by 53.33% of respondents), customization (51.67%) and integration (44.44%) as top concerns.

Figure 2

Log Management Challenges

  1. What are the biggest challenges related to effective use of log management across your organization?
log management challenges
Note: N = 180
Source: IDC, 2019

The challenges experienced by the survey participants reflect the difficulties of operating modern IT infrastructures that continue to grow in scale and complexity. Moreover, the emergence of containers, microservices and serverless computing architectures that are based on autonomous interactions between independent single-function modules via APIs, is drastically increasing the number of moving parts that must be constantly monitored.

Traditional log management solutions were not designed to handle the huge amounts of log data flowing in from these chaotic IT environments, which can explain the above mentioned difficulties. For example, lack of effective scaling leads to mounting storage costs; and manual-intensive efforts are made to compensate for the lack of automation and integration capabilities. Other challenges mentioned by a relatively high proportion of respondents – including high rate of false positives, the need for manual querying and correlation, and slow processing – provide further evidence of the technical shortcomings of traditional solutions.

How Much is Too Much Data?

The amounts of log data collected and managed by organizations are overwhelming. According to IDC’s survey, nearly 50% of all organizations process more than 100GB of log data per day. In some industries, and especially in highly regulated sectors where data must be retained and protected for compliance purposes, data volumes are significantly higher. For example, 15.15% of financial services companies process more than 1,000GB (1TB) of log data per day.

Figure 3

Amount of Log Data Processed per Day

  1. How much log data does your log management solution process per day?
log data average volume

Note: N = 180
Source: IDC, 2019

Log Diversity in the Workplace

Unlike structured data – such as data kept in relational databases or contained within enterprise applications like ERP and CRM – log data is mostly unstructured. As each system or application generates logs in its own format, and often uses its own time format, data must be transformed into a common structured format before it can be queried and analyzed. Driven by the proliferation of intelligent, sensor-based systems and IoT devices, new types of log data are constantly being added to this mix.

The diversity of log data often leads to operational issues. For example, due to the difficulty of achieving a unified view of logs collected from multiple different sources using traditional solutions, organizations are often required to manually search and correlate disparate data sets – an error-prone and labor-intensive process.

Log Analysis Paralysis:

While collecting and searching through logs is more difficult due to the increased size and diversity of datasets, organizations have become dependent on their ability to analyze operational data.

IDC’s survey provided a clear indication of the growing importance of log analytics. While IT operations troubleshooting and security remain the most common use case for log management, business analytics was the primary use case for companies that plan to use a log management solution in the next 12 months. The growing reliance on log analytics puts more emphasis on the need to automatically integrate and correlate different types of data from different sources to achieve comprehensive and accurate visibility, and the ability to detect and resolve problems quickly.

Figure 4

Log Management Use Cases

  1. What are the use cases for your organization’s log management solution?
use cases for log management
Note: N = 180
Source: IDC, 2019

Next-Generation Log Management and Analytics

IDC’s survey demonstrated the importance of gaining real-time operational intelligence from machine-generated data. To address this need, log management solutions should evolve to tackle the above mentioned challenges of scalability, format diversity, lack of automation, and perhaps most importantly – offer advanced analytics capabilities.

In this regard, Artificial intelligence (AI) machine learning (ML) analytics are key technologies that can be used to identify anomalies, help teams focus on what matters most, optimize processes such as log clustering (the grouping together of related logs to facilitate search), correlation and alerting, and more. As depicted above, integration with other IT management systems is also essential for log management solutions in order to achieve comprehensive visibility across different environments and uses cases.

Log management is seeing rising demand from organizations of all sizes and industries. Cloud deployment options are therefore required to provide organizations – and particularly ones that lack the expertise or resources to implement and maintain complex in-house systems – with access to advanced functionalities. At the same time, given the amounts of log data delivered over the cloud, next-generation solutions should be designed to scale on-demand to avoid network congestion and data access latency.

Log Analytics Across the CI/CD Pipeline

Another important requirement of next-generation solutions is the incorporation of log management and analytics into continuous integration/continuous deployment (CI/CD) processes. CI/CD is now a common method for bridging once-siloed development and operations teams and processes, creating streamlined delivery pipelines that support the rapid pace of changes. In accordance, 52% of the participants in IDC’s survey are already using CI/CD methodologies, while additional 36% are planning to use CI/CD in the near future.

Figure 5

CI/CD Adoption

  1. Does your organization use, or is it planning to use CI/CD methodology?
logging for CICD
Note: N = 200
Source: IDC, 2019

While having a CI/CD pipeline is a growing necessity in today’s dynamic, software-driven IT environments, it also introduces new challenges. As things get more complex, gaining insight into every stage of the delivery pipeline – for every version release – is more difficult. To measure the quality of software versions in a more accurate and timely manner, organizations often combine log data analysis with other methods including, test automation and metrics.

Metrics allow for measuring the functionality and performance of applications or services, typically through the establishment of thresholds for operational indicators such as CPU and memory utilization, transaction throughput, application response time, errors, etc.; as well as various business indicators. Continuously collected at regular intervals, metrics can be used for identifying trends and potential issues over time, as well as events, i.e. deviations from thresholds that point to unusual occurrences at specific time that need further investigation or analysis.

The combination of open source log analysis with different monitoring methods can help development, application support, site reliability engineering (SRE), DevOps and operations teams address various issues across all stages of the CI/CD pipeline, from design to release. Next-generation log management and analytics solutions can play a key role here, enabling organizations to make sense of huge amounts of operational data in various formats and obtain a unified and comprehensive view of the IT environment.

Coralogix Company Overview

Coralogix is a full-stack observability platform provider that aims to facilitate log data management and accelerate problem resolution with automation tools and machine learning algorithms. The solution, which launched the first version in 2015, is based on the open source ELK stack coupled with proprietary algorithms and advanced enterprise features.

To reduce the time wasted searching through logs to solve issues, Coralogix automatically clusters multiple logs are into a handful of shared templates without the need for configurations. The company has developed ML capabilities in order to model the normal behavior of environments and services and the flows in between, to automatically alert users to unnatural deviations.

For example, new errors are automatically identified as soon as they appear for the first time, or when a combination of logs that are expected to arrive in a particular sequence deviates from the expected flow. These capabilities should result in less time spent creating and managing alerts manually. The ML-assisted approach also has the effect of surfacing otherwise unknown issues before they drastically affect the customer experience.

To support the increased velocity of releases in CI/CD practices, Coralogix automatically generates version benchmark reports upon every release or environmental change. These reports let teams know how the software quality compares to previous versions in terms of new errors, higher error ratios, or new broken flows.

Coralogix is available as software-as-a-service with licensing based on the size of daily logs ingested and the retention period chosen. It includes quick integrations with the majority of clouds, servers, applications, frameworks, log files, platforms, containers, systems and databases available.

Coralogix has introduced several new features over the past 12 months, including threat intelligence by enriching logs based on reputational IP blacklists, quota usage management tools to save on cost, and volume anomaly detection which triggers an alert when a trend of errors or bad API responses is detected in one of the system components.

Coralogix is a good option to consider for midsize and enterprise businesses requiring a scaled and secure centralized logging solution for log management and analytics, compliance, and monitoring, as well as white glove support services which comes handy when facing more complicated integrations and setups.

Essential Guidance

The results from IDC’s survey demonstrate that despite the mass-scale adoption of log management solutions, organizations struggle to extract insight from the ever-increasing amount and diversity of machine-generated data. The survey highlighted customer concerns over cost of data storage, the need for customizations and integrations and other challenges that point to the limitations of traditional solutions. To tackle these issues and obtain continuous and comprehensive monitoring of modern IT environments, organizations should seek to leverage advanced log analytics across various use cases and as part of their CI/CD and DevOps pipelines.

About IDC research 

International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications, and consumer technology markets. IDC helps IT professionals, business executives, and the investment community make fact-based decisions on technology purchases and business strategy. More than 1,100 IDC analysts provide global, regional, and local expertise on technology and industry opportunities and trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world’s leading technology media, research, and events company.