Tag: cost

Fundamentally, there are logs that will be of intrinsic value to you, and others that are less business-critical. Are you aware of IT cost optimization? Are you aware of the logging cost to handle, analyze and store these different types of logs? Should you really have the same approach for mission-critical logs as you do for info or telemetry logs? Differentiating your approach for different logs is challenging. 

If no two logs are truly the same then why should you treat them the same?

Slow Query Times can Increase Costs

In a multi-application system architecture, you’re going to have a variety of different applications with very different behaviors. It’s commonplace to have some applications which generate infrequent, important logs, whilst one application may be spewing out literally millions of low priority and noisy debug logs. If you’re storing all of your logs on one disk or a database, and it nears capacity, then performance will dip.

This becomes dangerous if you’re running machine learning algorithms on the “useful logs”, which rely on a fully functioning database. Unfortunately, seeing what is useful, what might be in the future, and what never will be, is tough. One application may generate so many different logs that assessing their severity is challenging in itself. It goes without saying that this difficulty worsens with multi-application architecture. Logs are unruly, and visualizing outputs to help decide what should be done with them is a tricky job in itself. 

Where do I keep my logs?

Provisioning for log storage isn’t easy and the wrong type can seriously impact your logging cost. You may only read a few log outputs, ignoring the rest. If you’re running advanced visualizations on your logs, then you should enable frequent reads. If you’re never accessing a certain type of log, then you might consider cold storage. This requires engineering time and effort. You should spend that time, focused on your product. Your storage bill is going to shoot up exponentially without clear visibility and control of which logs you index. This means that even for those logs that you are not interested in, you’re still paying the same amount.

The white noise of logs

Figuring out which logs are critical is a challenge with numerous complexities. This challenge pales in comparison to having all of these logs and asking someone or a service to make sense of them. With a wilderness of jumbled and incoherent log files, lacking context or recency, you’re likely to invest considerable time, energy, and money into getting something useful out of them. If you’re planning on using a machine learning solution to help differentiate between the critical and less critical logs, it’s likely to be a more long-winded process of trawling through gigabytes or more of defunct data before finding something of use. 

How Coralogix can Help with your Logging Cost

Coralogix can allow you to take complete control of your logging solution. With the TCO Optimizer, you’ll be able to promote and demote your logs. This results in savings of up to 2/3rds of your logging bill. In a single view, you can see which of your outputs consume the most, broken down by the severity of the outputs themselves. You’ll be able to route those logs to different processing levels. For example, less important logs can simply be parsed and stored, but the most important logs can be processed as normal. This breaks the dichotomy of “drop” or “process” and allows you to take real control of your logging cost, with Coralogix.

The cost of logging is one of the big problems of a scaled software system. Logging solutions now need to support far more than they ever have. You need to make a real investment in a log monitoring software that can support these initiatives. However, the up-front costs of a custom-built logging solution are prohibitive for many organizations. No business wants its bottom line affected by logging costs. That’s where Coralogix comes in. Let’s look at the different IT cost optimizations associated with logging.

Operational Costs

We are all familiar with the cost-saving opportunities around using cloud infrastructure. Logging solutions have greatly benefited from the advent of cloud computing, but with scale comes cost. As you process a greater volume of logs, your system will need to grow in sophistication. More servers, more storage, more network traffic. This amounts to an expensive cloud bill that quickly endangers your ROI.

Staff Costs

If your company doesn’t build world-leading and competitive logging solutions, then training staff for such an endeavor is not cheap. Taking staff out of their day-to-day to learn and maintain the relevant skills will impact your company’s development process. Even more costly is finding an engineer who can do that and contribute to the other facets of your business. This is a dangerous gamble for a small or organization and a source of potential waste for a large company.

Go-live Time

On top of the time spent hiring a skilled group of engineers, building your logging solution will take time. Even if you simplify your logging solution, you have serious engineering challenges to tackle. In order to build a truly future-proof solution, the requirements gathering and architecture before development even begins will create a huge resource drain.

For example, Elastic Stack has no off-the-shelf security functionality. Security is not something that can be sidestepped safely and will need time, investment, and testing. This time will cost you and every vulnerability is a potential delay on your ROI.

Outage Costs

Downtime is one of the biggest frustrations with any service, and logging is no exception. This is particularly true if you are trying to build functions that use your logs as an input. Downtime in that scenario will have a knock-on effect on other business processes. You are likely to rely on your logging the most when there’s an issue elsewhere. If your logging solution isn’t fault-tolerant, then you are running a risk.

How can you use logs to assess what caused an outage, if you have an outage on your logging solution? The implications of this on your logging solution’s infrastructure is significant. Without logging expertise and IT cost optimization, you run the risk of frequent and protracted downtime.

The missed opportunity cost – what can I do with my logs?

A company’s logs represent a treasure trove of data that can feed into every aspect of your organization. Working out how to glean these nuggets from reams and reams of logs is a costly process. Leveraging machine learning is certainly one answer, but what if you don’t have in-house ML capabilities? The cost of hiring a data scientist is an expensive endeavor, and the time it takes to find the right person will further compound your missed opportunities. The time spent finding the right person could be far better spent growing your product, clients, or leads. 

So what can we do?

The core operational issue here is scale. The more logs you need to process, the more storage you need. You need larger servers to cope with demand. You need more sophisticated analytics to make sense of all of that data. The only easy way to stop this issue is to block certain logs, and allow them to disappear into the ether. The potential opportunity cost associated with this strategy is profound.  What we need is more fine-grained control of how we process those logs.

Handling logs in different ways poses a complex engineering challenge. How do we decide whether logs go into cold storage, or into rapid access servers? Building this type of capability in house can be a complex and risky undertaking.

How Coralogix can help

The TCO Optimizer helps you regain control of your logs, and provides savings of up to 2/3rds of your logging costs. Rather than process or block your logs, you’ll be able to tune the processing of each logging level. This can even be implemented retroactively. You can make a decision and change your mind a week later. Introducing new pipelines into your logging solution enables you to zoom in on the logs that really matter. No expensive upfront effort, or risky engineering projects. Just a simple, easy to use service that