How much does the free ELK stack cost you?

The free ELK stack (Elasticsearch, Logstash, Kibana) is not as free as it is cracked up to be.

This post will focus on the costs of maintaining your own ELK stack and the alternatives. 

Allow me to explain: Have you ever heard of The Weber-Fechner law?

Strangely enough, the Weber-Fechner theory is responsible for one of the most common mistakes companies make when choosing their log analytics solution.

Generally speaking, this law describes how people perceive change as a percentage of its baseline. By applying this theory to economic decision making, cognitive psychologists Amos Tversky and Daniel Kahneman discovered that people evaluate prices relative to a reference point, thereby making them more sensitive to a new expense rather than adding the same amount to an existing expense (see chart below).

But wait, how the hell is this related to Log Analytics?!

Well, remember those “free” ELK instances you have on your cloud? Their existence may prove to be the best example of the Weber-Fechner theory. These instances end up costing more than they initially appear at face value, however, most people tend to consider them free or cheaper than they are, as the price is added to the total amount that is paid to AWS.

That is why just like the chart below, you perceive their price lower than it actually is.

So what are the costs of deploying your own ELK stack?

Of course, the answer to this question varies and depends on several aspects like:

• How much log data is generated by your system(s).
• How long you want to retain that data.
• How accessible your data has to be.

We went for the classic case of a mid-size company:

• 50GB of log data per day.
• Retention period of 14 days.
• High data availability.

Price for building your own ELK stack on AWS:

1) 1 Master instance (c4.large, West US, no HA):

$0.124/hour * 720H/month = $89/month

2) 2 data instances (r4.xlarge) according to ES recommendation + with necessary redundancy:

$0.296/hour * 2 * 720 = $426/month

3) Disk, general purpose SSD (gp2)

$0.12/hour * 50GB/day * 14/days retention * 2 (data redundancy) * 1.2 (recommended extra disk for ES) = $201/month

Total HW expenses per month: $89 + $426 + $201 = $716

And now for the cost, most companies tend to ignore, despite it being what keeps the company running.

People Cost

It has been our experience that setting up the entire stack including the ES servers, mapping, Kibana and collectors will take the average engineer which is familiar with the ELK stack about 5 working days which costs $530/day according to the average daily salary of an engineer ($140K/year). Calculated monthly on a 2 years basis: $110/month.

• Monthly maintenance, about 3 days per month is the very least for this scale and it does not include crises (which do occur) and change requests from within the company: $1,590/month.

Total estimated price for building your own ELK stack on AWS: $716 + $110 + $1,590 = $2,416/month

Price for using AWS managed ES:

1) 1 Master instance (c4.large, west US, no HA):

$0.183/hour * 720H/month = $131/month

2) 2 ES machines (r4.xlarge.elasticsearch)

2 * $0.437/hour * 720H/month = $629/month

3) Hard Disk, EBS Standard volumes:

$0.162/hour * 50GB/day * 14/days retention * 2 (data redundancy) * 1.2 (recommended extra disk for ES) = $272/month

Total HW expenses per month: $131 + $629 + $272 = $1,032

• Setting up your ES stack when using AWS managed ES would take less than half the time it’ll take you to set-up everything on your own, so about 2 days which costs $530/day according to the average daily salary of an engineer ($140K/year).

Calculated monthly on a 2 years basis: $44/month.

• Monthly maintenance, about 1 day per month is the very least for this scale and it does not include crises (which do occur) and change requests from within the company: $530/month.

Total estimated price for a simple managed ES on AWS with Kibana and Logstash: $1,032 + $574 = $1,606/month

When you compare these numbers to services which cost about $2,500/month for 50GB/day 14 days retention and offer a fully managed cluster, alerting capabilities, higher availability, better redundancy, auto-scaling, and not to mention machine learning capabilities and anomaly detection, it is hard to understand why would anyone choose to set-up his own cluster. 

Coralogix offers a machine learning-powered logs, metrics and security solution, supporting the ELK experience, syntax, and API’s, without the hassle of maintenance and licensing risks. You are welcome to give it a spin and experience the difference between log management.

RELATED ARTICLES

Elasticsearch Mapping Exceptions

Elasticsearch Storage Optimization

Elasticsearch Update Index Settings