TCO: Optimize your Total Logging Cost of Ownership

Coralogix TCO not only reduces your logging costs by up to two-thirds but also improves your ability to query, monitor, and manage your data, by allowing you to define the data pipeline your logs will go through, based on the importance of data to your business.

This enables you to get all of the benefits of an ML-powered logging solution at only a third of the cost and with more real-time analysis and alerting capabilities than before.

The TCO Optimizer allows you to assign different logging pipelines for each application/subsystem pair and log severity, giving you maximum control over your data. The priorities are described as follows:

Low – Compliance pipeline
Log data that needs to be kept for
compliance/post-processing reasons will go
straight to your archive.
LiveTail

Log Parsing

Archive

Reindex

Direct Archive queries

Medium – Monitoring pipeline

Logs that are used for monitoring or statistics will be
fully available for those use cases by allowing you to
define alerts, build dashboards, view statistics, query
the live data stream, and receive proactive anomalies.

LiveTail

Dashboard Visualizations

Loggregation

Logs2Metrics

Log Parsing

Alerts

Anomalies

Archive

Reindex

Direct Archive queries

High – Frequent search pipeline

These logs need to be individually queried and analyzed.
Typically high severity, or
business-critical data will be stored on highly available
SSDs replicated, and ready to be queried within seconds.

Lightning Queries

LiveTail

Dashboard Visualizations

Loggregation

Logs2Metric

Log Parsing

Alerts

Anomalies

Archive

Reindex

Direct Archive queries

The TCO optimizer screen includes three sections:

  • Log’s distribution that shows the % of logs for each TCO pipeline, ‘Frequent search’, ‘Monitoring’, and ‘Compliance’ (compliance includes low and blocked logs).
  • Policy criteria includes current policies and allows for the creation of new ones. Policies will be applied on combinations of applications/subsystems/severities as the logs are ingested and will assign them the appropriate TCO pipeline based on the policy content. The default policy for all logs is high. Policies simplify assigning TCO pipelines and will capture any applicable future logs on ingestion.

    Each policy creates new default values for the logs the policy is applicable for. If Policies conflict, the first policy will have precedence (in the order they appear on the screen).
    To create a new policy click the “ADD NEW POLICY” button

    choose the appropriate filters for applications/subsystems/severities combinations and set the desired TCO pipeline.

    These will become the default TCO pipelines assignments for all matching logs that will be ingested into Coralogix.
  • ‘Application and policy overrides’ section displays the usage of all applications and subsystems, sorted by the top consumers. Click on any row to view the application/subsystem usage broken down by level of severity and its default TCO pipelines assignments. You may use the top filters to easily locate a specific component.

With the drop-down menu on the right, you are assigning for each component its priority to determine the data pipeline the logs arriving from that subsystem will go through.

Also, you can assign different TCO pipelines to different severities of a specific application-subsystem pair. Click on the arrow sign on the right of each line to see a breakdown of your logs by severity. This allows you to decide on your priority for this severity. The main drop-down will automatically show Multiple to respect your multiple selections.

Note: If you would like to start overriding from the beginning, you can remove all overrides by clicking on the “Reset All Overrides” button.

If you wish to change the TCO pipeline of some of your logs, you can always amend your original selections. This allows you to try out lots of different configurations to find the perfect optimization for your log analytics. This unlocks greater visibility of problems, shorter troubleshooting times, and finely tuned logging costs that will scale with your needs.