What Is an Observability Pipeline?
Learn how observability pipelines collect, process, and route logs, metrics, and traces, and how they help control cost.
Coralogix Team10 min read

Observability pipelines give you a control point where each log line, metric, and span can be reshaped, reduced, or redirected before a backend indexes it and bills for it. The budget stakes are rising, with 44 percent of analyst clients now spending more than $1 million a year on observability, up from 36 percent. A processing layer between sources and tools changes how you decide cost, retention, and data quality, and it adds a component you have to run.
This guide covers the three stages telemetry moves through, the pitfalls that turn a pipeline into a liability, and a four-step way to start from your own volume numbers.
What Does an Observability Pipeline Do With Your Telemetry?
An observability pipeline is the layer between your telemetry sources and your destinations that collects, processes, and routes logs, metrics, traces, and events. "Telemetry pipeline" names the same thing, and the analyst market category is Telemetry Pipelines (Transitioning to Observability Pipelines). Per-record rules decide whether to keep, reshape, sample, or drop data, and where each copy goes. Each type of telemetry data has its own volume profile, so rules are written per signal.
The Three Stages of an Observability Pipeline
Telemetry moves through collection, in-flight processing, and routing. The path from source to sink looks like this:
apps | hosts | Kubernetes | cloud services
|
v
agents and collectors (OTel Collector, Fluent Bit)
|
v
processors: parse -> filter -> enrich -> redact -> sample -> aggregate
|
v
sinks: observability backend | SIEM | object storage | data lake
1. Collect from Sources and Agents
Collection starts wherever telemetry is produced, such as application code, host agents, Kubernetes nodes, and cloud service log streams. You keep existing agents such as Fluent Bit or Prometheus exporters and point them at the pipeline instead of a backend. OpenTelemetry Collector receivers then accept those formats and convert them into one internal model.
2. Process Telemetry in Flight
You control cost and quality through processing rules applied record by record, before storage. The rules cover seven operations:
- Parse: Extract fields such as status code and latency from an access log line.
- Filter: Drop records that match a condition, such as debug logs or health-check spans.
- Enrich: Add context such as Kubernetes pod labels or team ownership.
- Normalize: Map
statusandhttp_statusonto one attribute. - Redact: Mask, hash, or delete sensitive values before they leave the pipeline.
- Sample: Keep a representative fraction of high-volume data, by trace or by record.
- Aggregate: Roll many events into a metric, such as requests per service per minute.
The Collector runs processors in sequence, so put drops and redaction ahead of enrichment and never pay to enrich a record you were going to discard.
3. Route to One or More Destinations
Routing sends each record, or a copy of it, to the destination that suits its use. A backend serves debugging, a security information and event management (SIEM) tool serves detection, and object storage serves retention. In the OpenTelemetry Collector, every exporter in a pipeline receives a copy of each data element, so one stream reaches several tools without re-instrumenting a service.
When to Use an Observability Pipeline
An observability pipeline fits when you need to control telemetry costs, switch backends, or remove sensitive data. Each need has to outweigh the work of running another component.
1. Telemetry Volume and Cost Outgrow the Budget
A pipeline cuts telemetry cost by filtering records nobody queries, sampling high-volume traffic, and aggregating event floods into metrics. Coralogix, the telemetry lake for observability, moves that cost decision into the ingestion path. The TCO Optimizer in its Data Engine routes logs, traces, and real user monitoring events to Frequent Search, Monitoring, Compliance, or Blocked based on policies for application, subsystem, severity, or any event field.
Only Frequent Search data is indexed in Coralogix hot storage. Monitoring and Compliance data lands in your own archive bucket, and Blocked logs are dropped at ingestion. Policies are evaluated at ingestion and aren't retroactive, so set them before volume arrives.
2. Teams Want to Add or Switch Backends
A pipeline separates collection from destination, so adding or switching a backend stops being a reinstrumentation project. You can dual-ship to current and candidate backends, compare results, and cut over signal by signal. Routing moves data and leaves dashboards behind, so alerts and saved queries still need translation.
3. Sensitive Data Must Leave Before Storage
Redacting in flight keeps personal and cardholder data out of every downstream copy. Requirement 3.5.1 of the Payment Card Industry Data Security Standard (PCI DSS) requires the primary account number to be unreadable wherever it is stored, through keyed hashing, truncation, or tokenization. Hashing has limits under the EU's General Data Protection Regulation, since pseudonymization guidelines treat pseudonymized data as personal data when someone holds the means to re-link it. Dropping the field leaves nothing to re-identify.
Where OpenTelemetry and the Collector Fit
The OpenTelemetry Collector can serve as the pipeline, because OpenTelemetry builds it as one. Its architecture documentation defines a pipeline as the path data follows from receivers, through processors, to exporters, with connectors joining pipelines. Unlike traditional log shippers such as Fluentd, the Collector types every pipeline by signal and speaks the OpenTelemetry Protocol (OTLP) natively. The OpenTelemetry Collector is also designed as a single vendor-agnostic agent in place of several per-vendor agents, supporting observability without lock-in.
What Can Go Wrong With an Observability Pipeline?
A pipeline becomes a liability when it drops data you later need, loses data during its own outages, delays alerts, or runs on rules that drift between teams.
1. Dropping Data You Later Need
A filter or sampler removes data for good, and the loss surfaces during the incident nobody planned for. Head sampling decides at the root span, so it can't ensure all error traces are kept, and tail sampling can only do so at the cost of holding spans in memory. In Coralogix, data routed to the Monitoring or Compliance tier stays queryable in your object storage through DataPrime, without rehydration.
2. Another Component to Run and Scale
A pipeline is a distributed system with its own outages. The Collector's in-memory queue loses data on crash, retries stop after five minutes by default, and a full queue drops new data. With memory-only buffering, Fluent Bit backpressure can cause inputs to lose data, so use persistent queues and buffers sized for the longest backend outage you intend to survive.
3. Added Latency Before Alerts Fire
Every hop adds batching and queueing delay, and backends that index before evaluating alert conditions add more. Coralogix's Streama© engine evaluates alert conditions on telemetry still in the stream, before indexing or storage.
4. Rules That Drift Across Teams
Routing rules edited by hand across clusters diverge, and one misplaced filter can silently empty a SIEM feed. You can keep Collector configs in Git and run otelcol validate in continuous integration. Coralogix Fleet Management builds on the open-source OpAMP protocol to push, diff, and monitor Collector configurations from one place, though the OpAMP specification is still in beta.
How to Compare Observability Pipeline Options
Deployment model and pricing unit separate observability pipeline options more than processor lists do, since most options cover parsing, filtering, and routing. The options fall into three categories:
| Category | Best fit | What drives cost |
|---|---|---|
| Open-source collectors and agents | Teams standardizing on OTLP who can operate their own fleet | The compute and engineering time you run them on |
| Standalone pipeline products | Teams that want one neutral pipeline in front of many tools | A separate pipeline fee on top of backend fees |
| Pipelines built into observability platforms | Teams that want processing and tiering in one ingestion path | Included in the platform's ingestion pricing |
Open-source collectors carry no license fee but have mixed component stability, and standalone products are worth pricing against the volume they remove downstream. Built-in pipelines depend on the forwarding options the platform supports. Coralogix sits in that category and forwards filtered data to SIEMs, cloud storage, and Kafka, with usage-based pricing and no per-host, per-user, per-query, or per-feature charges.
How to Start an Observability Pipeline in Four Steps
Setting up an observability pipeline takes four steps, which are measuring volume per source, setting routing rules, filtering and sampling, and monitoring the pipeline itself. Measurement comes first because every later rule depends on knowing where volume comes from.
1. Measure Volume per Source
The baseline is GB per day per service and per signal, measured for at least two weeks and ranked by volume, with a record of how often each source is queried. A large source nobody queries is your first target.
2. Set Routing Rules per Destination
Map each source to a destination by query frequency. Incident-critical data goes to the hot backend, detection inputs go to the SIEM, and compliance data goes to archive, with each rule written as source, condition, and destination so reviewers can read intent.
3. Filter and Sample Carefully
Apply cuts in order of safety, first the drops your baseline shows nobody queries, then sampling, then moving rarely queried data to a cheaper tier. The worked example below models one application log stream of 200 GB per day at an assumed $0.50 per GB ingested, with a lower-cost tier at an assumed $0.25 per GB.
| Step | Volume per day | Ingestion cost per day |
|---|---|---|
| Before: all logs in the standard tier | 200 GB | $100.00 |
| Drop 40 GB of debug logs | 160 GB | $80.00 |
| Sample 30 GB of successful health checks at 10 percent | 133 GB | $66.50 |
| Route 100 GB of rarely queried logs to the lower-cost tier at $0.25 per GB | 133 GB | $41.50 |
At those rates, ingestion cost over 30 days falls from $3,000 to $1,245 before storage charges, a 58.5 percent cut.
4. Monitor the Pipeline Itself
Watch the Collector's internal metrics, especially otelcol_receiver_refused_* and otelcol_exporter_send_failed_*, and compare otelcol_exporter_queue_size against otelcol_exporter_queue_capacity for queue pressure. Sustained refused or failed rates warrant alerts, and OpenTelemetry's guidance is to scale out when queues reach 60 to 70 percent capacity.
Balancing Volume Control Against the Risk of Losing Signal
Cutting data lowers the bill, and dropped data can leave questions unanswerable later. Coralogix handles that trade-off by routing low-value data to lower-cost tiers instead of discarding it. Data in the Monitoring tier stays alertable and queryable, so the source you demoted is still there during the next incident.
Route your noisiest source to the Monitoring tier on the Coralogix platform and keep it queryable for your next incident at a lower cost.
Frequently Asked Questions About Observability Pipelines
How does an observability pipeline reduce SIEM costs?
The pipeline sends only detection-relevant events to the SIEM and routes the rest to object storage, where investigations can still reach them. Where SIEM pricing is tied to ingested volume, you then pay SIEM ingestion only for the events your detections use.
How does a pipeline redact PII before storage?
Processors match keys or value patterns and mask, hash, or delete them in flight. OpenTelemetry's sensitive data guidance warns that hashes of small, predictable inputs are reversible in practice. A keyed hash or a deleted field avoids that risk.
Should you build or buy an observability pipeline?
Building fits when your team can run collectors, buffers, and upgrades and your volume grows predictably. Buying fits when pipeline operations would compete with platform work, or when you want policy, storage tiering, and alerting from one vendor.