DevOps Security: Challenges and Best Practices
With the shift from traditional monolithic applications to the distributed microservices of DevOps, there is a need for a similar change in operational security policies. For…
Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!
Formats: PNG, PDF, and SVG
Files size: 2.8 MB
For brand guidelines, please click here
The free ELK stack (Elasticsearch, Logstash, Kibana) is not as free as it is cracked up to be.
This post will focus on the costs of maintaining your own ELK stack and the alternatives.
Allow me to explain: Have you ever heard of The Weber-Fechner law?
Strangely enough, the Weber-Fechner theory is responsible for one of the most common mistakes companies make when choosing their log analytics solution.
Generally speaking, this law describes how people perceive change as a percentage of its baseline. By applying this theory to economic decision making, cognitive psychologists Amos Tversky and Daniel Kahneman discovered that people evaluate prices relative to a reference point, thereby making them more sensitive to a new expense rather than adding the same amount to an existing expense (see chart below).
Well, remember those “free” ELK instances you have on your cloud? Their existence may prove to be the best example of the Weber-Fechner theory. These instances end up costing more than they initially appear at face value, however, most people tend to consider them free or cheaper than they are, as the price is added to the total amount that is paid to AWS.
That is why just like the chart below, you perceive their price lower than it actually is.
Of course, the answer to this question varies and depends on several aspects like:
We went for the classic case of a mid-size company:
1) 1 Master instance (c4.large, West US, no HA):
$0.124/hour * 720H/month = $89/month
2) 2 data instances (r4.xlarge) according to ES recommendation + with necessary redundancy:
$0.296/hour * 2 * 720 = $426/month
3) Disk, general purpose SSD (gp2)
$0.12/hour * 50GB/day * 14/days retention * 2 (data redundancy) * 1.2 (recommended extra disk for ES) = $201/month
Total HW expenses per month: $89 + $426 + $201 = $716
And now for the cost, most companies tend to ignore, despite it being what keeps the company running.
It has been our experience that setting up the entire stack including the ES servers, mapping, Kibana and collectors will take the average engineer which is familiar with the ELK stack about 5 working days which costs $530/day according to the average daily salary of an engineer ($140K/year). Calculated monthly on a 2 years basis: $110/month.
Total estimated price for building your own ELK stack on AWS: $716 + $110 + $1,590 = $2,416/month
1) 1 Master instance (c4.large, west US, no HA):
$0.183/hour * 720H/month = $131/month
2) 2 ES machines (r4.xlarge.elasticsearch)
2 * $0.437/hour * 720H/month = $629/month
3) Hard Disk, EBS Standard volumes:
$0.162/hour * 50GB/day * 14/days retention * 2 (data redundancy) * 1.2 (recommended extra disk for ES) = $272/month
Total HW expenses per month: $131 + $629 + $272 = $1,032
Calculated monthly on a 2 years basis: $44/month.
Total estimated price for a simple managed ES on AWS with Kibana and Logstash: $1,032 + $574 = $1,606/month
When you compare these numbers to services which cost about $2,500/month for 50GB/day 14 days retention and offer a fully managed cluster, alerting capabilities, higher availability, better redundancy, auto-scaling, and not to mention machine learning capabilities and anomaly detection, it is hard to understand why would anyone choose to set-up his own cluster.
Coralogix offers a machine learning-powered logs, metrics and security solution, supporting the ELK experience, syntax, and API’s, without the hassle of maintenance and licensing risks. You are welcome to give it a spin and experience the difference between log management.
Elasticsearch Mapping Exceptions
Elasticsearch Storage Optimization