At Payoneer, we use Coralogix to collect logs from all our environments from QA to PROD.
Each environment has its own account in Coralogix and thus its own limit. Coralogix price modules are calculated per account.
We as a company have our budget per account and we know how much we pay per each one.
In case you exceed the number of logs assigned per account you will pay for the “extra” logs. You can see the exact calculation in this link.
You can see our flow below showing each ENV disconnected from the other but all of them under our account in Coralogix.
In each environment (Except PROD) we allow our developers to decide what will be the log level they want to write, and it can cause somewhat of an issue if you constantly writing in DEBUG or VERBOSE. You can reach your Coralogix quota quite fast if you are not careful.
We needed a way ( without chasing the developer teams each day with a slap on the wrist ) to limit the number of logs with no human interactions.
We had a few options to consider on how to do it:
We wanted a solution that will not be part of our stack and will always run against Coralogix API.
We chose to use Azure Functions
Accelerate and simplify serverless application development with serverless compute
Azure function along with AWS Lambda and Google Cloud Functions was our main focus and we chose Azure function as we are already working with Azure and they provide 1 million executions on a free tier so the choice was easy.
The functions were written in python and you can see the flow below:
As you might have seen in the above diagram we use Coralogix rules to stop logs from being parsed and thus save money on ingested logs every single day. What are the rules:
Rules help you to process, parse, and restructure log data to prepare for monitoring and analysis
Coralogix offers many different types of log parsing rules like:
You can see the full list at the Coralogix site.
In our case, we used the Block option. Block rules allow you to filter out your incoming logs using RegEx.
The rules are part of log groups that can contain multiple rules.
See the example here:
Per request from our developers we added a way to unblock an application for a predefined period of time OR give them an added XXX lines of logs to be parsed and displayed in the UI before they are blocked again:
We had a necessity to lower our log collector SaaS cost and using azure functions we were able to moderate it to a manageable flow.
Most importantly we now have visibility to what application is costing us the most and we can work closely with the Dev team to reduce the number of logs they write.