What Is Prometheus Monitoring? Architecture, Use Cases, and Limitations

Prometheus puts a query language on top of every metric your services expose. A latency spike on one pod becomes a question you can answer, and you can query dimensional metrics on demand across dozens of microservices.

This guide covers how Prometheus monitoring works, its architecture and metric types, and the practices and limits site reliability engineers (SREs) hit as clusters grow.

What Is Prometheus?

Prometheus is a monitoring system that collects numeric metrics from your services and stores them as time series. SoundCloud built Prometheus in 2012, and the Cloud Native Computing Foundation (CNCF) accepted it in May 2016 and graduated the project in August 2018, after Kubernetes. A single Prometheus server scrapes, stores, and alerts on its own, with no dependency on network storage or an external database.

Metrics arrive via a pull model over HTTP, and each series is identified by a metric name plus key-value labels. Samples go into a built-in time series database (TSDB) on local disk, and Prometheus Query Language (PromQL) queries them. Labels attach at instrumentation time and let you slice one metric by any of them later. Prometheus’s data model holds numeric values only.

How Prometheus Works

The server collects data by scraping. At fixed intervals, it requests each target’s metrics endpoint over HTTP and records the returned samples. The global scrape_interval defaults to one minute with a 10-second scrape_timeout, though the sample prometheus.yml sets 15 seconds. Pulling also gives you target health for free, because Prometheus generates an up metric on every scrape.

Each sample belongs to a series such as http_requests_total{job="apiserver", handler="/api/comments"}. The server writes samples to its on-disk TSDB in two-hour blocks, compacts them into larger blocks in the background, and deletes data once it passes the retention period. That period, along with scrape targets, intervals, and rule files, comes from your prometheus.yml file.

Prometheus Architecture

A Prometheus deployment is a set of single-purpose components, some of them optional. Client libraries and exporters expose metrics, and service discovery tells the server where to find them. The server scrapes and stores the data, while Alertmanager turns firing rules into notifications.

Prometheus Server

The server scrapes targets, evaluates rules, and stores results in the local TSDB. Recording rules precompute expensive expressions as new series, and alerting rules fire when a PromQL condition holds. Both run in groups at the evaluation_interval, which shares the default scrape interval.

Exporters

Exporters translate metrics from systems you can’t instrument directly. Node Exporter exposes hardware and operating system metrics from *NIX kernels on port 9100 by default. The official MySQL server exporter exposes MySQL server metrics, and official exporters also exist for HAProxy, Consul, and Memcached.

Pushgateway

The Pushgateway caches metrics from short-lived batch jobs that may exit before a scrape. Each job pushes its results on completion, and Prometheus scrapes the gateway afterward. The project’s guidance limits it to service-level batch jobs, because a shared gateway becomes a single point of failure, loses the up health metric, and keeps pushed series until someone deletes them manually.

Alertmanager

Alertmanager receives alerts from Prometheus and handles deduplication, grouping, silencing, and inhibition before routing notifications to receivers such as email, PagerDuty, or OpsGenie. A routing tree in its configuration file matches alert labels to receivers. During an outage, hundreds of alerts can fire at once, and grouping by labels such as cluster and alertname sends a single compact notification instead.

Client Libraries

Client libraries instrument your own code and implement the four metric types. The project lists libraries for Go, Java or Scala, Node.js, Python, Ruby, and Rust, with third-party options for other languages. Your application exposes its current metric state over an HTTP endpoint, conventionally /metrics, which Prometheus scrapes.

Service Discovery

Service discovery replaces static target lists with live lookups. Options include kubernetes_sd_configs, ec2_sd_configs, consul_sd_configs, dns_sd_configs, and file_sd_configs. When pods reschedule, or instances scale out, the server finds the new targets without anyone editing prometheus.yml.

Core Metric Types in Prometheus

Client libraries offer four core metric types, and the wrong choice produces queries that run cleanly but return meaningless numbers:

• Counter: A counter is a cumulative value that only increases or resets to zero on restart, such as http_requests_total, and you query it with rate() to get the per-second increase.
• Gauge: A gauge moves up or down, such as memory in use or queue depth, so taking a rate() of one produces nonsense.
• Histogram: A histogram counts observations like request durations into configurable buckets, exposed as _bucket, _sum, and _count, and the server computes percentiles at query time with histogram_quantile().
• Summary: A summary calculates configurable quantiles over a sliding window inside the application, alongside _sum and _count.

Histograms are the safer default for latency, because you can’t combine summary quantiles from separate replicas into one fleet-wide percentile.

Querying Metrics with PromQL

PromQL lets you select, aggregate, and calculate time series for graphs and alerts. Every expression evaluates to an instant vector (one sample per series at one timestamp), a range vector (a window of samples per series), a scalar, or a string. Label matchers such as =, =~, and !~ narrow the selection, and a duration like [5m] turns a selector into a range vector. The standard starting query applies rate() to a counter over five minutes:

rate(http_requests_total[5m])

The query returns the per-second rate for every http_requests_total series over the last five minutes. Counter resets don’t distort the result because rate() adjusts for them automatically. Wrapping it as sum by (job) (rate(http_requests_total[5m])) collapses instances, but keeps the job dimension.

Prometheus Use Cases

Prometheus fits environments where services can expose numeric metrics over HTTP. Common production setups include:

• Kubernetes clusters: Service discovery finds pods and nodes as they reschedule, a common base for broader Kubernetes monitoring.
• Microservices: Teams instrument services with counters for requests and errors and histograms for durations.
• Containers: The docker_sd_configs option discovers Docker containers as scrape targets, a starting point for container monitoring.

All three environments churn, because pods, containers, and service instances come and go faster than a static target list can track.

Why Teams Choose Prometheus Monitoring

Much of Prometheus’s appeal comes from its dimensional data model, where one metric with a code label replaces separate metrics for each status. Among graduated CNCF projects, Prometheus sits at 77 percent production use in a 2025 survey, with another 12 percent evaluating it. Grafana is the standard dashboard pairing and reads Prometheus series through a built-in data source without storing them, so Prometheus’s own retention limit decides how far back a dashboard can look. Coralogix, a cross-stack observability platform, lands telemetry in your own cloud bucket in open Parquet format, so a dashboard can look back past whatever a single Prometheus server keeps.

Limitations and Challenges

Local storage isn’t clustered or replicated, so it’s only as durable as the node it runs on. Without retention time or size limits, Prometheus keeps 15 days of data, well short of a quarter-over-quarter comparison. Every unique label combination creates a new series, so cardinality is the bigger risk. The naming guidance warns against labels that store user identifiers (IDs), email addresses, or other unbounded sets of values.

A single Prometheus server doesn’t scale horizontally on its own, and Prometheus exposes remote write as the way to send data elsewhere. Thanos and Cortex accept those writes for long-term, horizontally scaled storage, and other remote-write backends, including managed services, do the same. Each clustered option adds components to operate, and sharding Prometheus itself means planning shard counts as ingestion grows.

How to Keep Prometheus Stable Through a Cardinality Explosion

Unbounded label values cause a cardinality explosion, and budgeting every new label keeps Prometheus stable. A single deploy that puts request IDs into a label creates a new series for every value, and the resulting series count can overwhelm a server until it crashes and takes your production observability with it. Reverting the deploy is the short-term fix, but nothing blocks the next unbounded label.

Production setups that survive an unbounded-label incident share five decisions:

• Cap cardinality per metric: Instrumentation guidance recommends keeping metric cardinality below 10 where possible and avoiding anything that could exceed 100.
• Precompute dashboard queries: Recording rules store expensive aggregations ahead of time, and dashboards read one precomputed series on each refresh.
• Set scrape intervals per target type: You can override the global scrape_interval per job under scrape_configs to suit each target type.
• Page on symptoms: Alerts should fire on user-visible latency and errors, and Alertmanager grouping, inhibition, and silences limit alert fatigue.
• Plan storage on day one: Retention size should leave headroom on the allocated disk, and remote write should be in place before anyone needs older history.

A long-term backend belongs in the plan because no single server fully solves storage.

What to Add When Prometheus Hits Its Limits

Prometheus stays the collector, and a long-term backend picks up where its local storage and single-server design stop. PromQL support in Coralogix lets Grafana keep reading the same data. Coralogix accepts Prometheus Remote Write from your existing servers.

Start a free 14-day Coralogix trial with an eight-unit quota and no credit card. The trial lets you send remote write from one server and compare retention and queries against your current long-term storage.

Frequently Asked Questions About Prometheus Monitoring

How do I install Prometheus?

You download the latest release for your operating system, extract it, and define scrape targets and intervals in prometheus.yml. Running ./prometheus --config.file=prometheus.yml starts the server on port 9090. On Kubernetes, the Prometheus Operator manages that configuration as custom resources instead.

What is the difference between a Grafana alert and an Alertmanager alert?

An Alertmanager alert starts as a Prometheus alerting rule that the Prometheus server evaluates and then hands to Alertmanager for grouping, silencing, and delivery. Grafana Alerting evaluates Grafana-managed rules against data sources like Prometheus and routes them through a built-in Alertmanager that handles only Grafana-managed alerts. You can also forward Grafana-managed alerts to an external Alertmanager such as the Prometheus Alertmanager.

Can Prometheus handle logs and traces?

No. Prometheus records numeric time series, so logs and traces need separate tools. Coralogix accepts metrics through Prometheus Remote Write and takes logs and traces through OpenTelemetry-native ingestion, so all three signals sit in one place.

Which database does Prometheus use?

Prometheus uses its own time-series database on local disk, storing samples in time-based blocks and compacting them in the background. The default retention is 15 days, to keep longer history, raise the retention setting or send data to a long-term backend via remote write.

How do you scale Prometheus?

Prometheus has no built-in horizontal scaling, so teams send data elsewhere through remote write. Thanos and Cortex accept those writes for long-term, horizontally scaled storage, while sharding Prometheus itself means planning shard counts as ingestion grows. Each clustered option adds components to operate.