Tag: remote query

Datadog Flex Logs vs Coralogix Remote Query

Posted on by eugene evdokimov

While Coralogix Remote Query is a solution to constant re-ingestion of logs, there are few other options today that also offer customers the ability to query unindexed log data. For instance, Datadog has recently introduced Flex Logs to enable their customers to store logs in a lower cost storage tier. 

Let’s go over the differences between Coralogix Remote Query vs Flex Logs and see how Datadog pricing compares.

Get a strong full-stack observability platform to scale your organization now.

Overall cost – ingestion and retention

To get a true picture of costs, we need to calculate the unified cost per GB on both platforms. Unified cost is the joined cost of ingestion and retention. The calculation is based on the initial ingestion cost, with the monthly cost of retention, that’s spread out over a year and alongside any additional costs.

DataDog Unified Cost per GB = $0.10 ingestion + $0.60 retention + unknown compute costs = $0.70 per GB per year (+ unknown compute costs).

Coralogix Unified Cost per GB = $0.17 ingestion + $0.05 retention = $0.22 per GB per year

This demonstrates that Coralogix is 70% less than the cost of DataDog, even without considering the unknown compute costs.

A real world example

Let’s say a customer produces ten million logs per day, and retains them for one year. For simplicity, we can also make the following assumptions.

  • Assume they’ve already got a year’s worth of data in their long term storage.
  • The average log size is 1KB, meaning that one million logs is about 1GB of data. 
  • The data is never queried. We can assume this because DataDog has not published their compute pricing. It has no effect on Coralogix pricing, because Coralogix does not charge anything per query. 

Coralogix Ingestion Cost: $0.17 per GB x 365 = $620

DataDog Ingestion Cost: $0.10 per GB x 365 = $365

Now, we must consider retention costs over the year. DataDog’s costs are recurring. Every month, customers are being charged for 3650GB of data, at $0.05 per GB. 

DataDog Retention Cost: $0.05 per GB per Month x 12 = $2190

Coralogix, on the other hand, relies on cloud object storage, because of its scalable performance and extremely low cost. Let’s be generous, and assume there is no intelligent tiering activated in S3 at all. Instead, let’s factor in our standard 5x compression, meaning any retention costs should be divided by 5.

Coralogix Retention Cost: $0.02 per GB per Month x 12 = $876 / 5 = $175.20

So when we combine these figures, there is a clear winner in overall cost.

DataDog costs over the year: $2555 (+ Unknown Compute costs) ❌

Coralogix costs over the year (ignoring any cost optimization): $795.20 ✅ (70% Lower)

A 70% difference in cost is a big deal. Coralogix is clearly the most cost effective solution in any real world scenario, even before we consider cost optimization available in S3, such as intelligent tiering, which drives down retention costs even further. If the goal is to retain data, without losing access, then Coralogix is the obvious choice.

Vendor lock-in: Flex Logs vs Coralogix Remote Query

DataDog stores Flex Logs within their system. For customers, this means that their data is never truly held in their own hands. Instead, DataDog keeps their data on their behalf. The data is held in a proprietary format, making vendor lock-in an unavoidable and serious concern.

Coralogix customers, on the other hand, hold their data in object storage in their own cloud account. The format, CX-Data, is based on Parquet. It’s so open source, customers can even directly index and query their logs using Amazon Athena. Data is never held hostage by Coralogix.

Analysis

DataDog Flex Logs are only available for querying. Any other activities require them to be rehydrated back into expensive, index based storage. On the contrary, Coralogix enables customers to view their data in LiveTail, parse and format their logs and train anomaly detection models, all at no extra cost. 

Summary: DataDog Flex Logs vs Coralogix Remote Query

Coralogix logs are less expensive to store and retain, they contain more features and they can be queried at no extra cost. Coralogix makes it much more predictable, scalable and usable solution, offering at least a 69% reduction in cost, when compared in a real world scenario to Flex Logs. In reality, after intelligent tiering is factored in, Coralogix is a fraction of the cost, with access to more features, standing out as the obvious choice for cost optimization and data problems in observability.

Remote Query Solves the Observability Data Problem

Posted on by eugene evdokimov

We are caught in a whirlwind of rapid data observability. As more engineers, services and sophisticated practices are helping generate an astronomical amount of digital information, there’s a growing challenge of the data explosion. 

Coralogix offers a completely unique solution to the data problem. Using Coralogix Remote Query, the platform can drive cost savings of 40-70% without sacrificing insights or functionality. 

Scale breeds complexity and cost

The propagation of microservices shows no sign of slowing down. In fact, 85% of respondents from large companies report using microservices, according to a 2021 global marketing survey.

In tandem, modern engineering practices will become more ubiquitous and accelerate the rate of change, influencing ephemeral infrastructure. As a result, the natural chaos of solution and demand for data will increase.

Learn More

Conventional wisdom is out of date

According to traditional advice for your data, you should adhere to a simple data life cycle. First, ingest and index everything, then archive your data after a certain period, and finally reingest data when you need it. 

However, that same life cycle drives a series of consequences, including:

  • Incurring maximum cost because all data becomes indexed by default.
  • Incurring costs in multiple places through ingestion, storage, reindexing, and more. 
  • Customers won’t know how much data they should reindex, resulting in more consumption that wastes money and time.

Coralogix Remote Query: An elegant solution to a complex problem

Coralogix Remote Query solution

Rather than ingest and compress data, Coralogix Remote Query transforms the traditional and stiff archive into a living, breathing part of the customer’s observability stack. Coralogix Remote Query is a solution to constant reingestion of logs and enables customers to get the best of both worlds:

  • Telemetry is stored in an S3 bucket in a customer’s account so they pay the bare minimum for storage costs.
  • Customers may query their bucket from the Coralogix UI, which is free of charge, and only pay Coralogix for the initial ingestion.

Coralogix’s offering can bring down costs and increase the dataset customers can access, accelerating insights and overcoming the data explosion. These queries also run faster than most OpenSearch solutions today. For example, a 10TB query takes roughly 10-seconds to complete, with millions of logs loaded from S3 in second. Gain speed and usability, while reducing costs. 

Unparalleled efficiency with the TCO optimizer

Coralogix TCO Optimizer allows customers to route their data to three different use cases intelligently. Data can be classified as frequent search (indexed), monitoring (converted into metrics before archiving) or compliance (enriched, transformed and archived). This drives huge cost savings because data can be analyzed and understood without the need to index. And data can be retrieved in seconds, thanks to Coralogix Remote Query. 

What does this mean for your organization?

TCO Optimizer and Remote Query have some clear and obvious outcomes for any organization struggling to balance costs with insights:

  • Access more data than ever, while maintaining unrivaled performance.
  • Drive down costs only indexing the data that you need, and nothing more. 
  • Transform and investigate data in new and exciting ways, using Coralogix’s DataPrime query language.

Set up your archive in less time

Creating a Coralogix archive is as simple as defining an S3 bucket and adding some simple IAM permissions to give Coralogix access to your bucket. Coralogix handles data formatting, enrichment and transformation. To begin, ship your company data to Coralogix using one of over 200 integrations. Then you’re ready to access the data however you need. 

Learn More

Four Things That Make Coralogix Unique

Posted on by eugene evdokimov

SaaS Observability is a busy, competitive marketplace. Alas, it is also a very homogeneous industry. Vendors implement the features that have worked well for their competition, and genuine innovation is rare. At Coralogix, we have no shortage of innovation, so here are four features of Coralogix that nobody else in the observability world has.

1. Customer Support Is VERY Fast

Customer support is the difference between some interesting features and an amazing product. Every observability vendor has some form of customer support, but none of them are even close to our response time.

At Coralogix, we respond to customer queries in 19 seconds (median) and achieve <40 minute resolution times on average.

This is fine for now – but how will Coralogix scale this support?

We already have! Coralogix has over 2,000 customers, all of whom are getting the same level of customer support because we don’t tier our service. 1 gigabyte or 100 terabytes – everyone gets the same fantastic standard of service. Don’t believe me? Sign up for a trial account and test our service!

2. Coralogix is Built Differently

The typical flow for data ingestion follows a set of steps:

  1. Data is initially stored and indexed. 
  2. Indexed data then triggers a series of events downstream, such as dashboard updates and triggering alarms.
  3. Finally, cost optimization decisions and data transformations are made.

This flow adds latency and overhead, which slows down alarms, log ingestion, dashboard updates, and more. It limits the decision-making capabilities of the platform. It’s impossible to skip indexing and go straight to archiving because every process depends on indexed data. At Coralogix, we saw that this wouldn’t work and endeavored to build our platform differently. 

So how does Coralogix do it?

Coralogix leverages the Streama© architecture. Streama focuses on processing the data first and delaying storage and indexing until all the important decisions have been made. 

It is a side-effect-free architecture, meaning it is entirely horizontally scalable, and adapts  beautifully to meet huge daily demands. This means the Coralogix platform is exponentially more efficient. 

3. Archiving and Remote Query

Many observability providers allow customers to archive their data in low-cost storage. In most providers, data is compressed and stored in low-cost storage, like Amazon S3. Customers need to rehydrate their data if they wish to access their data.

There are some key issues with this approach:

  • Archived data is far less discoverable.
  • Historical data may be held hostage by a SaaS provider using proprietary compression.
  • Customers now have to pay again for a massive volume of data in hot storage.

So how does it work at Coralogix?

At Coralogix, we do not demand that data must be rehydrated before it can be queried. Instead, archives can be queried directly. Our remote query engine is fast. Up to 5x faster than Athena and capable of processing terabytes of data in seconds. 

With support for schema on read and schema-on-write, Coralogix Remote Query is much more than a simple archiving solution. It’s an entire data analytics platform capable of processing Lucene, SQL, and DataPrime queries.

Does Remote Query save customers money?

In summary, yes. Customers are migrating to Coralogix daily, and they constantly report cost savings. One of the most interesting behaviors in new customers is their willingness to hold less data in “frequent search.” This means customers are paying for less data in hot storage because that data is still easily and instantly accessible in the archive.

This behavior shift and our TCO Optimizer regularly drive cost savings of between 40% and 70%. Speaking of…

4. The Most Advanced Cost Optimization on the Market

Most observability providers have a tiered solution, especially regarding cost optimization. Spending enough money unlocks certain features, like tiered storage. Our competitors need to gatekeep their cost optimization features because they are not architected for this type of early decision-making in the data process. This means they can only afford to optimize their biggest customers. 

Coralogix is Perfect for the Cost Optimization Challenge

Our Streama© architecture means we can make very early decisions, long before storage and indexing. This allows us to make cost-optimization decisions for all of our customers. Whether it’s our Frequent Search, Monitoring, or Compliance use case, Coralogix and our unique architecture regularly drive down customer costs.

More than this, we also have features that allow our customers to transform their data on the fly. This allows them to keep only the necessary information and drop everything they don’t. For example, Logs2Metrics, allows our customers to transform their expensive logs into optimized metrics that can be retained for far longer at a fraction of the cost.  

Coralogix is Different in All the Best Ways

Coralogix is more than just a full-stack observability platform with some interesting tools. It’s a revolutionary product that will scale to meet customer demands—every time. Our features, coupled with unprecedented customer support and incredible cost optimization make us one of the few observability providers that will help you to grow, help you to optimize, and, at the same time, save you money in the process.

Long-Term Storage: Coralogix vs. DataDog

Posted on by eugene evdokimov

Long-term storage, especially for logs, is essential to any modern observability provider. Each vendor has their own method for handling this problem. While there are numerous available solutions, let’s explore just one – Coralogix vs DataDog – and see the benefits and limitations.

Coralogix already outpaces Datadog in support, with 30-second response times, and cost, where customers have experienced 40-70% cost reductions.

Flex Logs

DataDog has released its feature, named Flex Logs, that enables users to store their logs in their own cloud storage, for example, S3. This allows them to archive logs in very low-cost storage, enabling them to retain more logs without hugely increasing costs. This is especially useful for teams with a large volume of logs and need to retain them for compliance issues.

So what’s missing? 

There are some limitations. While we believe that querying archived logs is possible in DataDog, there is a compute cost associated with this feature.

How does Coralogix Remote Query Compare?

Coralogix Remote Query has an initial similarity with DataDog. They both utilize cloud storage in the user’s cloud account for cost-effective log storage. However, Flex Logs is ostensibly a log storage mechanism, but Coralogix Remote Query is a data analytics solution that easily competes with DataDog Flex Logs, providing a wealth of insights out of the box.

Remote Query Data Does Not Need Rehydration

While DataDog does appear to offer direct archive query, it also appears that it charges monthly compute costs, which is likely to increase the total spend for the customer to access their own data. This is the fundamental difference when comparing Coralogix vs DataDog. Coralogix enables Remote Query, which allows users to directly query their archived data, without re-indexing. 

Additionally, Coralogix Remote Query supports both schema on read and schema on write, allowing users to define their schema upfront for maximum query precision and optimization, or discover their unstructured data as they go.

Why is Remote Query so Important For An Archive?

The need to reindex data raises some serious questions:

  • How much data is necessary to reindex?
  • How expensive will it be to hold all of this data in indexed storage?
  • How long will it take to reindex this data, especially if there is a lot? 

At Coralogix, we don’t force users down a data reindexing strategy. Coralogix does have a reindexing capability, which users can use as they wish. Still, with our unique architecture and the power of Remote Query, we have found that this is required far less than with other SaaS vendors.

Remote Query means users can easily discover their data at no extra cost. The only cost that the user incurs is their S3 hosting fees. Coralogix customers can issue as many queries as they like to their archive using Remote Query. This enables three key capabilities – discovering Coralogix Archive Data, holding less data in Frequent Search, and generating new insights via DataPrime.

Coralogix Archive Data is Discoverable

With Coralogix, data can be explored, new insights can be garnered, and new intelligence can be gathered without the need to reindex. This means that teams can freely (literally) explore their data without worrying about overages or charges for indexing a huge volume of data from the archive. DataDog, with its requirement to reindex, does not have this capability. 

Users Can Hold Far Less Data in Frequent Search

Coralogix Remote Query is extremely fast, up to 5x faster than AWS Athena in certain scenarios. Many Coralogix users have realized that the significantly reduced cost and remarkable performance of Remote Query means they keep fewer indexed logs. This further reduces their costs while allowing them to explore more of their data than ever before.

Users Can Generate Entirely new Insights with DataPrime

Coralogix Remote Query doesn’t just allow users to pull the data from remote storage. They can also perform aggregations on their data using the DataPrime syntax (SQL and Lucene are also supported). This enables true data discovery and the ability to directly generate reports on archived data, instantly. 

Coralogix vs Datadog: Remote Query is the Next Generation of Archiving

Whether it’s the feature-set, the price point, the analytics capabilities, or the performance, by our overall estimations Coralogix wins with long-term storage of observability data. The capabilities available as part of the Remote Query feature are entirely unparalleled in the industry. Because our unique and fundamentally different architecture drives them, many of our competitors are years away from meeting the standard that we have set.

If you don’t believe us, sign up for a free trial and try it for yourself.