Tag: CDN

A Complete Guide to Tracking CDN Logs

Posted on by eugene evdokimov

The Content Delivery Network (CDN) market is projected to grow from 17.70 billion USD to 81.86 billion USD by 2026, according to a recent study. As more businesses adopt CDNs for their content distribution, CDN log tracking is becoming essential to achieve full-stack observability. 

That being said, the widespread distribution of the CDN servers can also make it challenging when you want visibility into your visitors’ behavior, optimize performance, and identify distribution issues. Thankfully, CDN logging provides a solution to these issues. 

In this article, we’ll dive into what CDN logs are, how to track them and the kind of data you should be extracting from them.

Understanding CDN logs

CDNs speed up content delivery and improve user experience by caching assets like images and videos on edge servers distributed across the globe. And CDN logs curate data on all requests accessing your website or app through the CDN.

These include data like request URLs, response codes and time, client IP addresses, request times, caching information and the geolocation of requests. CDN log data can also help you determine who is accessing your app or website, where they are accessing it from and the type of data they access.

CDN logs typically follow a common log format, with some variation between providers. Here is a general outline of the information you’ll typically find in a CDN log:

  • Client IP address: The IP address of the user or device accessing your content through the CDN. 
  • Request time: The date and time the request was made, in UTC. 
  • Request URL: The full URL that was requested. 
  • Response code: The HTTP response code returned for the request, such as 200 (OK), 404 (Not Found), etc.
  • Cache status: Whether the requested resource was served from cache or fetched from the origin. 
  • Request method: The HTTP method used, typically GET but sometimes POST.
  • Server IP: The IP address of the specific CDN server or POP that served the request.
  • Object size: The size of the requested object, in bytes. 
  • Geolocation: The location of the requesting client, such as country and sometimes city or region. 

Most major CDN providers offer logging and analytics dashboards as part of their service, often for an additional fee. You can configure some CDNs to send their logs to a third-party analytics or logging system for further CDN log analysis.

What is CDN tracking exactly?

CDN tracking typically involves CDN monitoring tools and software that collect data on user interactions with CDN-delivered content. This data can include the user’s geographic location, device type, browser type, and the specific content items accessed. Analyzing this data helps you to identify patterns and trends in user behavior, and make informed decisions about how to optimize content delivery for different user segments.

CDN monitoring tools integrate with the CDN provider’s APIs to access real-time metrics and log data those APIs expose. They also allow you to analyze and report on that data within a central dashboard or interface.

CDN monitoring tools simplify the process of tracking your CDN logs by providing;

  • centralized monitoring of multiple CDNs in one place
  • standardized reporting and dashboards across CDNs and
  • advanced analytics capabilities not available in the CDN provider’s native tools

With the advanced tracking capabilities provided by CDN monitoring tools, you can extract all kinds of useful data from your CDN logs with ease. Let’s look at some data types you can extract from your CDN logs and how to use them.

What to extract from CDN logs?

Collecting CDN log data allows you to monitor the usage and performance of your CDN. By analyzing this data with a tool like Coralogix, you can identify performance bottlenecks, troubleshoot errors, and optimize CDN configurations to ensure fast and reliable content delivery.

CDN logs can be categorized based on the type of data they provide in 6 different types, as outlined below: 

1. Access logs

These logs provide information about every request made to the CDN, including the time of the request, the IP address of the requester, the HTTP status code, the requested URL, and the size of the response.

Use access logs to identify resources that are being accessed most frequently, which helps you with optimizing caching, CDN configurations, and resource allocation. Access logs also allow you to  identify and troubleshoot issues with specific URLs or origin servers. For example, if a particular URL is returning a 404 or 500 error code, access logs can be used to identify and investigate the root cause of the error.

2. Error logs 

Error logs only capture information about errors that occur while processing requests, such as 404 errors or 500 errors. They help you find and troubleshoot issues with specific requests, such as missing resources or incorrect server configurations.

Error logs can also be used to monitor and investigate errors in real-time, and take corrective actions to ensure uninterrupted service.

3. Performance logs

Performance logs contain real-time information about the performance of the CDN, including the response time, the number of requests served, and the number of requests that were cached.

They also let you optimize the CDN configuration by adjusting cache expiration times or configuring load balancing. Use performance logs to monitor and analyze the performance of the CDN infrastructure, and identify performance bottlenecks that could impact user experience.

4. Security logs

Security logs detail information about security-related events, such as failed login attempts or attacks on the CDN infrastructure. These logs can be used to monitor and detect suspicious activity, such as brute-force attacks or DDoS attacks.

You can also use security logs to identify and mitigate security threats, such as by blocking IP addresses or implementing rate limiting.

5. Analytics logs 

Analytics logs provide information about user behavior, such as the geographic location of users, the devices they are using, and the pages they are accessing. These types of logs help you understand user behavior and optimize the user experience, such as by optimizing page load times or improving content delivery.

They can be used to monitor and analyze user behavior, and identify patterns and trends that could impact the performance of the CDN infrastructure.

6. Real-time logs 

Real-time logs are generated in real-time and they provide information about the requests being processed by the CDN, enabling administrators to monitor and respond to issues as they occur.

These logs can be used to troubleshoot issues in real-time and ensure uninterrupted service. For instance, when you make changes to your CDN configuration, use real-time log analysis to validate the new settings are working as expected and alert you to any unintended consequences.

CDN log monitoring with Coralogix

Coralogix’s full-stack observability platform comes equipped with in-stream data analytics that allows you to collect and centralize CDN logs from various CDN services. Analyze these massive CDN logs in real-time without indexing using their Streama© streaming analytics engine.

The platform lets you set up real-time alerts on CDN logs to detect issues, outliers, and anomalies, while their in-stream alerting feature correlates events across log, metric, and trace data. CDN tracking with Coralogix is a seamless process. 

First, use Coralogix’s integrations with CDNs, such as Akamai or Amazon CloudFront, to collect and stream CDN logs directly to the dashboard. Once logs are collected, Coralogix’s search and filtering capabilities help you query and analyze the logs, create real-time alerts and dashboards to monitor CDN performance metrics and detect issues.Then, try integrating CDN log data with other contextual data, such as application logs, metrics, and traces, to gain a comprehensive view of issues impacting CDN performance.

Furthermore, with the help of CDN monitoring tools, gain valuable insights into how your CDN is performing and identify any issues affecting your content distribution. Investing in a robust CDN monitoring solution will improve the efficiency of your content delivery networks. CDN monitoring tools will work effectively to serve users the content they want, when and where they want it.

How to Choose the Best CDN Monitoring Tool for Your Needs

Posted on by eugene evdokimov

Rich content like videos and graphics used to cause network congestion and long load times when all the content was stored on a centrally located server. Fortunately, Content Delivery Networks (CDNs) monitoring tools came to the rescue in the late 1990s, letting users load rich content from a location geographically closer to them and reducing load times by distributing a cached version of content across servers worldwide. 

And since rich content is so widely used, CDNs have also become a critical component of IT architecture. Provided by third-party companies like Fastly, CloudFlare, and Akamai, CDNs allow logs to be sent to and monitored, with the help of strong full-stack observability platforms like Coralgoix, ensuring high performance standards and minimizing outages. 

This article will discuss what to look for in a CDN monitoring tool based on attributes users need to be monitoring.

What your CDN monitoring tool needs to measure

Before choosing a CDN monitoring tool, users should determine what they need to measure. Find a monitoring tool to both analyze your CDN logs and follow your specific performance indicators. Consider tracking the following metrics:

  1. CDN latency monitoring

Latency measures how long website pages take to load. Page speed is a critical measurement since latency directly impacts performance metrics like conversion rate. The higher the latency, the more likely a website with low performance could see a drop in conversion rate. 

CDNs are meant to reduce latency but should also meet service level agreements (SLAs). Further, monitoring both your source and edge servers for latency will help isolate any problems and help you identify whether the issue is with your CDN or software?

You can measure latency across your website by graphing latency metrics. For example, Coralogix can ingest CDN logs from various providers and convert logs to metrics for visualization. Coralogix also ingests logs from various sources and immediately analyzes them using its proprietary Streama technology. So IT teams can know at a glance when the latency is higher than usual and also spot where the cause is.

  1. Security monitoring

CDN logs come complete with entries for every request issued to your website. Log observability tools can either analyze these logs directly or convert these logs to metrics that can be analyzed for security issues. These logs help IT teams find what nefarious actors did and where they are located.

Due to the high volume of logs used to monitor CDNs, most monitoring tools take significant time to index and assess the data. Analysis of multiple log events is needed to detect anomalies like security breaches. Time is of the essence when detecting and handling threats. Reducing the time it takes to produce the analysis and signal an alert is crucial for limiting the scope of security breaches. With Coralogix’s Streama technology, contextual alerting in real-time occurs in the stream without indexing latency or mapping dependencies. This allows your IT teams to neutralize threats faster.

  1. Performance monitoring

CDN monitoring tools should determine performance changes quickly. The right monitoring tool allows you to look back at archived logs and analyze improvements in performance over time. Coralogix’s Archive Query feature allows you to query your logs directly from your S3 archive seamlessly, helping you store information on performance issues and more. CDNs will also export logs to third-party observability services to be converted to metrics for analysis.

Providing your own performance monitoring, independently of the CDN itself, allows IT teams to hold CDN providers accountable, including when there’s a service-level agreement (SLA) breach. Performance metrics can be leveraged to ensure your website gets the best possible service.

Furthermore, most full-stack observability tools are cost-prohibitive since they charge for the amount of stored data. Storing the logs necessary for performance monitoring would greatly increase the cost. With Coralogix, the pricing model is based on analysis, not size. We are able to provide performance monitoring within your budget and business needs.

How to identify the problem

If a website becomes unavailable or faces a security breach, IT teams should be notified immediately in order to effectively handle the issue. Tools used for CDN monitoring should include an alerting system when metrics do not meet standards. 

Alerts are typically split into two categories: static and dynamic. Static alerts are helpful if a threshold is known and unchanging. For example, you may use a static alert to notify IT teams if the latency of a webpage is higher than some number of seconds. Dynamic alerts are helpful when alerts need to be set up comparing changing values. For example, when users want to alert when the latency is higher than usual, a dynamic alert should be used. 

Your monitoring solution will ideally have both types of alerting so IT teams can make the most of CDN logs and quickly respond to errors and user experience changes. Coralogix provides both dynamic and static alerting that are customizable for your needs. Choose from a variety of dynamic alerts that are built in and easy to set up. These include time-relative alerts that are especially useful for detecting abnormal behaviors like an increase in errors from your CDN.

Beware of hidden CDN costs

CDN logs are notoriously large since every request generates a log. Most observability solutions will charge based on the volume of logs ingested. Since CDN logs tend to have large volumes, choose a full stack observability solution that is unique in their pricing model. Coralogix will not charge you for the amount of logs you store, allowing for a complete observability solution for your CDN logs.

Wrapping up

CDNs allow high-performance websites to deliver rich content to users by placing cached content across multiple servers worldwide. Monitoring these servers is critical to understanding whether your website is performing as it should. 

Choose a monitoring tool that identifies specific issues such as latency, load-balancing and availability and security, as well as analyze archived logs, alert IT when an issue arises, and keep costs low despite extensive volume log data.

4 CDN Monitoring Tools to Look At

Posted on by eugene evdokimov

Beyond their primary function of bringing internet content closer to client servers, CDN performance monitoring tools also play a vital role in network security. For instance, CDN helps you absorb traffic overloads from DDoS attacks by distributing traffic across many servers.

However, the volume of servers under your CDNs control and their geographically distributed nature presents its own set of risks, operational and security. Choosing the best CDN monitoring tool is critical to the end-user experience. 

What is CDN monitoring?

CDN monitoring uses the critical performance metrics and output from your network to determine the performance impact of the CDN on your network. It also allows you to identify performance issues at the network edge or the origin and fix them quickly. A CDN monitoring tool’s key metrics to identify anomalies include logs, latency data, bandwidth usage data, origin server data, etc.

What happens if you don’t have CDN monitoring in place?

Although their primary function is to simplify the delivery of content to your customers in different geographical locations, CDNs themselves take a complicated path to achieve it.

Various dependency layers integrate to make your CDN function efficiently. 

Further, some organizations adopt a multi-CDN approach, with each CDN handling hundreds, if not thousands, of servers. There are simply too many moving parts to leave unwatched, so CDN monitoring is critical.

Without monitoring your CDN, every performance issue has to be manually analyzed to pinpoint the cause. This will significantly increase your Mean Time To Resolution (MTTR), and some of your customers won’t stick around that long. 

Also, you won’t be able to identify these issues before they cause downtime. The downtime would be your first hint that something was wrong, and it’s usually too late. 

Most importantly, CDN monitoring helps you identify user experience optimization areas. Without CDN monitoring, you cannot improve your user experience, and yes, your customers will move on.

Benefits of CDN Monitoring

CDN monitoring directly benefits you in one crucial way: improved user experience. CDN monitoring helps you extract the most optimal performance from your CDN, ensuring that your users get the content they want, when, and where they want it. CDN monitoring helps you collect vital datasets that tell you the conditions in which your entire network, CDN included, is fulfilling user requests. 

For example, CDNs cache content at the network edge to reduce the bandwidth usage on the origin server. Instead of forwarding requests to the origin, the edge server simply fulfills requests with the cached content. 

Collecting data on the percentage of requests fulfilled by cached content (known as cache hit ratio) allows you to determine bandwidth usage on the origin. And take steps to reduce it where needed. A high cache hit ratio indicates that your CDN edge servers are functioning well, which means your users enjoy speedy content delivery.

Best CDN Monitoring Tools For Your Stack

1. An Observability tool: Coralogix

Observability tools provide you with real-time insight into the internal health of your network by monitoring output data like logs, metrics, and traces. A tool like Coralogix provides this data in a single, full-stack CDN monitoring dashboard.

For example, while the cache hit ratio might tell you the percentage of requests fulfilled at the edge, it does not show each request’s path. Distributed tracing features from the Coralogix dashboard, however, can track the path of each request, noting the servers it passes through and its final point of fulfillment. You can easily use distributed tracing to determine if each request is being fulfilled by the nearest edge server and map your system efficiently.

With Coralogix’s CDN monitoring dashboard, you can convert all your vast and noisy CDN log data into trackable metrics using the Logs2Metrics solution. You no longer need to index the log files manually. 

The Coralogix CDN monitoring tool also provides a one-click function called Archive Query that allows you to access all your archived CDN monitoring data in a single location.  

Real-time alerting and proactive security measures like the OOB (Out-of-the-Box) security dashboard complete Coralogix’s full-stack CDN monitoring dashboard. This easy-to-deploy security solution helps you track threats at the micro level with a rate-limiting function that protects against DDoS attacks and other malicious hacks.

2. A Synthetic monitoring tool: Catchpoint

Synthetic monitoring tools like Catchpoint’s IPM platform help you simulate real user traffic to test the performance of your CDN. They have servers distributed worldwide, and from there, they simulate user traffic and send it to your network to catch performance issues before it goes operational. 

For example, some tests test the latency difference between cache hits and cache misses (requests that go to the origin). By overriding the DNS, requests are sent directly to the origin, and then you can compare the latency with requests fulfilled from cached data.

3. Real User Monitoring (RUM) tool: KeyCDN

Real user monitoring tools provide data on transactions between your users and your CDN. KeyCDN’s monitoring tool is a good example of this, providing data on every piece of content it delivers to users and the conditions surrounding each request. These types of tools tell you exactly how your end users interact with the application and the effect of your CDN in smoothening the process.

4. CDN performance testing tools: Thousand Eyes

Performance testing tools like ThousandEyes work similarly to synthetic monitoring tools. They test each network component in isolation and together to identify performance issues. CDN performance monitoring tools can be deployed before or after your system goes operational. 

Coralogix’s Observability Dashboard is Your CDN Monitoring Superpower

Coralogix’s full-stack observability platform helps you optimize both your monitoring cost and ease of accessibility with innovative features such as OOB security dashboards and the logs module. With our Logs2Metrics module, you can directly convert your logs to useful data. Pull valuable insights from your log data without breaking the bank.

Not just that, your data is accessible as well. Coralogix pulls your CDN monitoring data into a single location where you can get an overview of your overall network health. Based on this data, OOB security dashboards can provide real-time alerting for abnormal behavior and help you monitor your CDN server effectively.

Choose better visibility into your CDN’s efficiency with Coralogix.

CDN Log Analysis

Posted on by eugene evdokimov

As user tolerance for slow page loads and refresh rates continues to diminish, poor website performance can directly impact visitor counts, click-throughs, and conversions. This is where CDN monitoring tools come in.

Since the beginning of the Internet, the speed of delivering content has been an issue. While processor enhancements, network acceleration, and web frameworks have brought drastic improvements to performance, the goalposts have continued to shift further away; devices operate on wireless connections with limited bandwidth, and the Internet is accessed from every corner of the globe.

Content Delivery Networks (also known as Content Distribution Networks) play a vital role in providing fast, responsive web pages, making it possible to live stream events around the globe and enabling real-time interactivity for remote, multi-player online games. Using a CDN to speed up content delivery to your users can help your business remain competitive and gain access to markets around the world.

What is a CDN?

A CDN consists of a network of edge servers in various geographic locations, each of which caches content supplied by the server that hosts your website (known as the origin web server). When a user’s device requests your website, it is first routed to the nearest CDN server, which tries to service the request with the cached content.

If the requested content is unavailable or out of date, the CDN server proxies the request to the origin web server as the source of truth. A CDN can significantly reduce latency and improve performance by reducing the physical distance that most requests and responses need to travel.

CDN providers, such as Akamai, Fastly, CloudFlare, and AWS, maintain a network of edge servers around the globe, which you can use to cache your web content and improve the experience for worldwide audiences. Storing content across a network of edge servers improves your site’s resilience by providing redundancy and protecting against DDoS attacks. Reducing the number of requests to your origin server can also reduce bandwidth costs while enabling you to handle more traffic.

Working with CDN logs

Your web server’s access logs can provide you with a wealth of information about the traffic hitting your site, such as user journeys and web crawler patterns, alerting you to errors, and helping diagnose issues. Collating and analyzing web access log data in real time is essential for proactively monitoring your website health and addressing issues.

When you augment your services with a CDN, many of the requests to your site are served directly by the local caches and are never seen by your origin web server. To maintain visibility of your web traffic, it’s essential to extend your web log analysis to include CDN server logs.

While most CDN providers offer an API to forward logs for storage and analysis, not all of them will store them for retrieval later if you don’t use that option. Let’s look at the most popular CDN providers:

●   Akamai – With Akamai DataStream, you can send raw logs from multiple endpoints to the destination of your choice every 30 or 60 seconds. If you don’t want to collect all logs, you can configure DataStream to sample the data for particular delivery properties. Akamai also provides a Log Delivery Service API that can be used to deliver edge server logs to a given destination on a schedule.

●   Fastly – Fastly offers several protocols to stream CDN logs in real-time, including both syslog and HTTPS, to multiple destinations. You can also change the log format and encrypt logs before sending them. These features help greatly with CDN log analysis. 

●   Cloudflare – Cloudflare monitoring allows you to push logs to specified cloud storage locations, including Amazon S3, or pull logs to any destination using their REST API every few minutes.

●       Amazon CloudFront – Two options are available for log streaming with CloudFront. With standard logs, all log entries are forwarded to an S3 bucket from which you can analyze or export them. Alternatively, you can use real-time logs and specify a sampling rate to send logs to Amazon Kinesis Data Streams within seconds of generation.

By collating log entries from your CDN’s multiple edge servers and combining them with your origin web server logs for analysis, you can derive several benefits that help with CDN log analysis.

Monitoring CDN performance through CDN log analysis

If you’re using a CDN, you’ll want to ensure it’s delivering the performance you require – both generally and when you’re expecting a peak in demand due to live streaming an event, launching a new product, or hosting an online game.

Using the data available in your web access logs, you can monitor the number of requests you’re receiving to each page and break those requests down by edge server location or region (using the requesting IP address). You can compare how users are experiencing your site worldwide and identify any performance reduction for particular pages or locations by looking at response times.

Monitoring the number of requests proxied by CDN servers to your origin web server will allow you to identify any unusual increase in cache misses, resulting in slower load times and increased demand on bandwidth. This might be an issue with the page itself or from misconfigured settings. When you need to make changes to your CDN configuration, you can use real-time log analysis to validate the new settings are working as expected and alert you to any unintended consequences. 

Understanding your site traffic

Just as with your origin web server logs, CDN logs can give you insights into who is visiting your site, the journey they’re taking through it, and the pages from which they exit. With more users blocking website tracking code, web server logs can help you better understand your users. Armed with that information, you can work out which pages you should invest your time in and which markets may require more targeted efforts.

However, users are not the only ones making requests to your site; bots will crawl your web pages, triggering requests to their nearest CDN server. By collating and analyzing requests from both edge and origin servers, you can understand where your search engine crawl budget is being spent and identify opportunities to optimize your site for SEO.

Log data can also help you work out if you’re being crawled by content scrapers or attacked by malicious form fillers, in which case you might want to investigate options for blocking such traffic.

Managing failures

Web log analysis plays a vital role in detecting and investigating failures. If a CDN serves most of your website traffic, you will need to collate and monitor edge server requests and responses to spot problems early.

While you should expect to serve a certain number of 4xx or 5xx error codes – due to mistyped URLs, broken referral links, or unauthenticated users – any significant increase in error responses warrants further investigation. Being able to immediately drill into the individual log entries will allow you to get to the bottom of the issue – whether that’s simply a bad link or something more serious like an upstream service failing – and fix it much more quickly than if you first have to retrieve the log files from the different edge servers.

Likewise, log data can track usage trends and key business metrics, such as transaction completion rates. Any deviation from the average could be a sign of an issue with your site, at which point tapping into the log files to see where users have dropped off will help you zero in on the cause of the problem fast.

Conclusion

CDN logs play a vital role in helping you understand how your site is being used, identify issues, and improve performance. When working with large volumes of log data from multiple sources, having a central platform to collate, parse, and analyze log entries will save considerable time and effort and enable you to derive insights fast to address issues as they emerge.

With Coralogix, you can collate log data from multiple sources, parse entries programmatically, and conduct real-time analysis to identify trends and detect anomalies automatically. Coralogix provides integrations with both Akamai Datastream and Fastly to enable real-time CDN log analysis. 

DDOS Attacks: How to Protect Yourself from the Political Cyber Attack

Posted on by eugene evdokimov

In the past 24 hours, funding website GiveSendGo has reported that they’ve been the victim of a DDOS attack, in response to the politically charged debate about funding for vaccine skeptics. The GiveSendGo DDOS is the latest in a long line of political cyberattacks that have relied on the DDOS mechanism as a form of political activism. There were millions of these attacks in 2021 alone. 

But wait, what is a DDOS attack?

Most attacks rely on some new vulnerability being released into the wild, like the Log4Shell vulnerability that appeared in December 2021. DDOS attacks are slightly different. They sometimes exploit known vulnerabilities, but DDOS attacks have another element at their disposal: raw power.

DDOS stands for Distributed Denial of Service attack. They have a single motive – to prevent the target from being able to deliver their service. This means that when you’re the victim of a DDOS attack, without adequate preparation, your entire system can be brought to a complete halt without any notice. This is the exact thing that the GiveSendGo DDOS attack has done. 

A DDOS attack usually consists of a network of attackers that collaborate together to form a botnet. A botnet is a network of machines willing to donate their processing power in service of an attack. These machines then collaborate to send a vast amount of traffic to a single target, like a digital siege, preventing other legitimate traffic in or out of the website.

What makes DDOS attacks so dangerous?

When a single user is scanning your system for vulnerabilities, a basic intrusion detection system will pick up on some patterns. They usually operate from a single location and can be blacklisted in seconds. DDOS attacks originate from thousands of different points in the botnet and often attempt to mimic legitimate traffic. Detecting the patterns requires a sophisticated observability system that many organizations do not invest in until it’s too late. 

But that’s not all…

It is widespread for DDOS attacks to attract more skilled hackers to the situation who are able to discover and exploit more serious vulnerabilities. DDOS attacks create a tremendous amount of chaos and noise. Monitoring stops working, servers crash, alerts trigger. All of this makes it difficult for your security engineers to defend your infrastructure actively. This may expose weaknesses that are difficult to combat.

Why are these attacks so common in political situations?

With enough volunteers, a DDOS attack can begin without the need for skilled cybersecurity specialists. They don’t rely on new vulnerabilities that require specialized software to be exploited. To make things worse, the people who take part in a DDOS don’t need to be technical experts either. They could be “script kiddies” who can make use of existing software, they could be technical experts or, most commonly, they could be people who can navigate to a website and follow some basic instructions. 

While we don’t know the details of the GiveSendGo DDOS attack yet, we can assume that this attack, like most other DDOS attacks, is the workings of a small group of tech-savvy instigators and a much larger group of contributors. This means that if a situation has enough people around it, a DDOS attack can rapidly form out of nothing and escalate a situation from a disagreement to a commercial disaster.

So what can you do about it?

There are several common steps that companies take to protect themselves from a DDOS attack. Each of these are crucial defensive mechanisms to ensure that if you do find yourself on the receiving end of a DDOS, you’re able to stay in service long enough to defend yourself.

Making use of a CDN and it was crucial in the GiveSendGo DDOS attack

Content Distribution Networks (CDN) provide a layer between you and the wider Internet. Rather than directly exposing your services to the public, use a CDN to distribute your content globally. CDNs have several great benefits, such as speeding up page load times and offering great reliability for your site. 

In the case of a DDOS attack, your CDN can act as a perimeter around your system and take the brunt of the attack. This buys you time to defend against the incoming storm proactively. The CloudFlare CDN has been one of the reasons why GiveSendGo hasn’t completely crashed during the attack. 

Route everything through a Web Application Firewall

A Web Application Firewall (WAF) is a specialized tool to process and analyze incoming traffic. It will automatically detect malicious attacks and prevent them from reaching your system. This step should come after your CDN. The CDN will provide resilience against sudden spikes in traffic. Still, you need this second layer of defense to ensure that anything that makes it through is scrutinized before it is permitted to communicate with your servers.

Invest in your Observability

Automated solutions that sit in front of your system will make your task easier, but they will never fully eliminate the problem. Your challenge is to create an observability stack that can help you filter out the noise of a DDOS attack and focus on the problems you’re trying to solve. 

Coralogix is a battle-tested, enterprise-grade SaaS observability solution that can do just that. That includes everything from machine learning to driven anomaly detection to SIEM/SOAR integrations and some of the most ubiquitous tools in the cybersecurity industry. Coralogix can give you operational insights on a range of typical challenges.

An investment in your observability stack is one of the fundamental steps in achieving a robust security posture in your organization. With the flexibility, performance, and efficiency of Coralogix, you can gain actionable insights into the threats that face your company as you innovate and achieve your goals.

CDN Logs – The 101 Guide

Posted on by eugene evdokimov

A Content Delivery Network (CDN) is a distributed set of servers that are designed to get your web-based content into the hands of your users as fast as possible. CDN monitoring produces CDN logs that can be analyzed, and this information is invaluable. Why? CDNs host servers all over the world and are designed to help you scale your traffic without maxing out your load balancers. A CDN also gives you added protection against many of the most common cyber attacks. This activity needs to be closely monitored.

A CDN, such as Akamai or Fastly, does all of this brilliant work, but we so often ignore the need to monitor it. CDN log analysis is the missing piece in your observability goal and it is a mistake to ignore it. A CDN is a fundamental part of your infrastructure and, as such, needs to be a first-class citizen in your alerting and monitoring conversations. 

Working with CDN Logs

Accessing the logs for your CDN will differ, depending on which provider you decide to go with. For example: 

Whichever the mechanism, you’ll need to create some method of extracting the logs directly from your provider. Once you have the logs, you need to understand what you’re looking at.  

A Typical Web Access Log

The following is a very common format of a web access log. Be mindful that on modern CDN monitoring solutions, you can change the format of your logs to use something more suited to CDN log analysis, like JSON, but this example will let you see the type of information that is typically available in your CDN logs and explain how to analyze your CDN logs:

127.0.0.1 username [10/Oct/2021:13:55:36 +0000] “GET /my_image.gif HTTP/2.0” 200 150 1289

Let’s break this line down into its constituent parts.

IP Address (127.0.0.1)

This is the source IP address from which the user has requested their data. This is useful because you’ll be able to see a high number of requests coming from the same IP address, which may give you an indication that someone is misusing your site.

Username (username)

Some providers will decode the Authorization header in the incoming request and attempt to find out the username. For example, a Basic authentication request contains the username and password encoded. If you detect any malicious activity, you may be able to trace it back to an account that you can close down.

Timestamp (10/Oct/2021:13:55:36 +0000)

As the name suggests, this portion of the log indicates when the request was sent. This is usually one of the key values when you’re looking to render this data out on a graph. For example, detecting sudden spikes in traffic.

Request Line (“GET /my_image.gif HTTP/2.0”)

The request line indicates the type of request and what was requested. For example, we can see that an HTTP GET request was issued. This means that the user was most likely requesting something from the server. Another example might be POST where the user is sending something to the server. You can also see which resource was requested and which version of the HTTP protocol was used.

HTTP Status (200)

The HTTP status lets you know whether your server was able to fulfill the request. As a general rule of thumb, if your HTTP status code begins with 2, it was most likely successful. Anything else indicates a different state. For example, 4XX status codes indicate that the request could not be fulfilled for some reason. For example, lack of authentication or a missing resource, as in the common error 404.

Latency (150)

Latency is a killer metric to track. Spikes in latency mean slow down for your users and can be the first indication that something is going wrong. Latency is the time taken between the request arriving at your CDN and the response being sent back to the user.

Response size (1289)

The response body size is an often ignored value, but it is incredibly important. If an endpoint that delivers a large response body is being used excessively, this can translate into much more work for the server. Understanding the response size gives you an idea of the true load that your application is under.

Monitoring Performance with CDN Log Analysis

So now you know what you can expect from your CDN logs, what kind of things should you be looking for? 

Slowdowns in response times (timestamp + latency)

If you monitor how much traffic you’re getting, you can immediately detect when something has gone wrong. If you include the latency property in this, you can quickly track when a slow-down is occurring.

Be careful of average latency

Averages are useful but they hide important information, such as variance. For example, if 9 of your requests respond in 100 ms but 1 of your requests take 10 seconds, your average latency will be about 1 second. From this, we can see that averages can hide information, so you need something different – percentiles.

It is best to take the median, 95th, and 99th percentile of your data. Using the same example again, the median from our data set would be 100ms (which reflects our most common data), the 95th would be 5545ms, and our 99th would be 9109ms. This shows us that while our data is around the 100ms mark, we’ve got a variance to investigate.

Managing live events

If you’re hosting a live event on your site, or perhaps hosting a webinar or live talk and you’re directing people to your site, that sudden influx of users is going to add a strain to your system and the CDN you’re using. You can check how much traffic you’re getting (by grouping requests into 1-second buckets and counting), or you can monitor latency to check for slowdowns. You could also look for errors in your logs to see if the users have uncovered a bug.

Understanding your site traffic with CDN Logs

It’s tempting to view your CDN logs as an operational measurement and nothing else, however, CDN logs are much more valuable. 

The marketing potential of CDN Logs

By monitoring the specific resources that users are requesting, you can identify your high-traffic pages. These high-traffic pages will make great locations for advertisements or product promotions. In addition, you can find where users drop off from your site and work to fix those pages.

Information Security

CDN logs help you to detect suspicious traffic. For example, web scraping software will work through your web pages. If you notice someone rapidly moving through every page in your site from the same IP address, you can be sure this is a scraper and you may wish to block this IP. 

But what do you do with these logs?

Coralogix offers a centralized, mature, scalable platform that can store and analyze your logs, using some of the most advanced observability techniques in the world. Correlating your logs across a centralized platform like Coralogix will enable you to combine your CDN insights with your application logs, your security scans, and much more, giving you complete observability and total insight into the state of your system. With integrations to Akamai, Fastly, Cloudflare, and AWS, you’re probably already in a position to get the best possible value out of Coralogix.

So is it worth it?

Whenever you use a CDN, these logs are a goldmine of useful information that will enable you to better understand the behavior of your users, the performance of your service, and the frequency of malicious requests that arrive at your website. These insights are fundamental for learning and growing your service, so you can safely scale and achieve your goals. While you grow, consider a full-stack observability platform like Coralogix, so you can skip the engineering headaches and get straight to the value.