Kafka Monitoring: Key Metrics, Tools, and Best Practices

Apache Kafka monitoring tracks cluster health, throughput, latency, and consumer lag to keep your pipelines stable and prevent messages from aging out of retention before your consumers read them. Kafka’s event streaming model decouples your producers and consumers through brokers, so neither side waits on the other.

This guide covers the Kafka metrics worth paging on, the thresholds that trigger action, the tools that collect them, and the alerting practices behind a monitoring system you can trust.

What Is Kafka Monitoring?

Kafka monitoring continuously collects predefined metrics from brokers, controllers, producers, and consumers to surface cluster problems before a downstream team reports them. Cluster health has a short list of expected values:

• exactly one controller node reports an ActiveControllerCount of 1
• under-replicated and offline partitions sit at zero
• unclean leader elections stay at zero
• partition and leader counts spread mostly evenly across brokers

Kafka observability adds logs and traces to the metrics, which lets you debug failures that a metric alone can’t explain. Mid-incident, a lag graph tells you the payments consumer group is falling behind, while a trace through that consumer shows every record waiting on a slow call to a fraud-scoring service.

Which Kafka Metrics Should Page You

Kafka exposes its metrics as Java Management Extensions (JMX) managed beans (MBeans). Replication health turns on the in-sync replica (ISR) set and the under-replicated partition (URP) count, while resource health turns on Java Virtual Machine (JVM) heap and garbage collection (GC).

Expected values don’t always translate into universal alert thresholds, so the numeric guidance below should inform rather than replace your baselines. The table summarizes the signals, affected layers, and alert conditions that deserve attention:

Metric	Layer	Why You Track It	Alert Threshold Guidance
Consumer lag (records-lag-max)	Consumer	Consumers can’t keep up or have stopped committing; records-lag-max is client-side, while Admin API tools compute lag from committed offsets	Sustained growth over a rolling window, not a non-zero reading alone
ActiveControllerCount	Controller	Missing controller or conflicting active controllers	Any cluster-wide sum other than 1 across controller nodes
UnderReplicatedPartitions	Replication	ISR smaller than the replica set	Any sustained under-replication
OfflinePartitionsCount	Controller	Partitions with no leader, unreadable and unwritable	Any offline partition, page immediately
IsrShrinksPerSec	Replication	A follower dropping out of sync	Unexpected activity outside restarts; alert if no expand follows
TotalTimeMs (Produce, FetchConsumer)	Broker	Request latency; RemoteTimeMs isolates slow replicas under acks=all	No universal threshold; alert on deviation from baseline
record-error-rate	Producer	Sends failing, often NotEnoughReplicas under acks=all	Any sustained error rate
Disk, JVM heap, and GC time	Resource	Retention outrunning disk; pauses causing ISR shrinks	Alert before capacity or memory pressure threatens cluster health

Consumer Lag

Kafka consumer lag is the number of offsets between the latest message produced to a partition and the last offset a consumer group committed. It’s the clearest sign of whether consumers keep up with the data brokers store.

The client exposes it as records-lag-max and per-partition records-lag under kafka.consumer:type=consumer-fetch-manager-metrics, while commit-rate and commit-latency-avg show whether the group is committing at all.

Broker Metrics

Kafka broker metrics start with kafka.controller:type=KafkaController,name=ActiveControllerCount, where each controller node reports 0 or 1 and you alert whenever the sum across controller nodes isn’t 1.

In KRaft mode brokers don’t expose it, so check kafka.server:type=MetadataLoader,name=CurrentControllerId on brokers, where -1 means no known controller. Throughput lives under kafka.server:type=BrokerTopicMetrics as MessagesInPerSec, BytesInPerSec, and BytesOutPerSec.

Request latency is kafka.network:type=RequestMetrics,name=TotalTimeMs for the Produce, FetchConsumer, and FetchFollower requests, and its RemoteTimeMs component is where a slow in-sync replica shows up when producers run acks=all.

Replication Health

ISR shrinks come first in the causal chain. The leader removes a follower that hasn’t caught up within replica.lag.time.max.ms, the partition becomes under-replicated, and if the remaining replicas also fail, the partition goes offline.

A broker crash shrinks the ISR without any lag, since the controller removes a broker whose session it has lost. Activity in IsrShrinksPerSec and IsrExpandsPerSec should remain flat outside a broker restart, which makes a shrink with no matching expand shortly after your earliest warning.

Producer Metrics

Kafka producer metrics live under kafka.producer:type=producer-metrics,client-id=”{client-id}”: record-send-rate, request-latency-avg, record-error-rate, and batch-size-avg.

A rising record-error-rate with acks=all can mean the ISR has dropped below min.insync.replicas, at which point brokers return NotEnoughReplicas and the producer retries until delivery.timeout.ms expires (two minutes by default).

The per-topic variant under producer-topic-metrics adds record-retry-rate, which catches retries piling into tail latency before they become errors.

Resource Utilization

Resource alerts should separate storage, memory, and processor saturation:

• Disk capacity: Leave enough capacity for retention growth, slow-consumer backlogs, and unbounded retention, which are common root causes of storage pressure.
• Heap and GC: Preserve headroom, track GC time, and account for CPU bursts rather than relying on one fixed threshold. A long GC pause can surface as an ISR shrink because the leader drops a follower that hasn’t fetched within replica.lag.time.max.ms (30 seconds by default).
• Processor saturation: NetworkProcessorAvgIdlePercent and RequestHandlerAvgIdlePercent provide direct saturation signals that should ideally stay above 0.3, and sustained declines toward exhaustion warrant investigation.

Together, these signals show whether capacity or runtime pressure is starting to threaten cluster health.

How Kafka Monitoring Tools Split Broker and Cluster Views

Kafka signals split across collectors: JMX-based tools see request latency and the JVM on each broker, while tools that query the Kafka Admin application programming interface (API) see cluster-wide replication and consumer-group state without an agent on every node.

Teams commonly pair one of each and pick the pair that covers both views with the least operational weight.

Prometheus with JMX Exporter and Kafka Exporter

The Prometheus JMX Exporter runs as a Java agent inside the broker JVM, the mode its documentation recommends, and gives you per-broker request latency, GC, and heap metrics from a configuration file that selects MBeans.

Kafka Exporter runs as a separate process against the bootstrap servers and exports what JMX doesn’t: kafka_consumergroup_lag, kafka_consumergroup_members, and kafka_topic_partition_under_replicated_partition, provided it holds Describe permission on all topics and groups.

Grafana is the usual dashboard front end. It adds another service to operate, and your logs and traces still sit in separate tools.

Burrow

Burrow, LinkedIn’s open-source lag monitor, stores the offset, commit timestamp, and lag for the last 10 commits per partition by default and applies its evaluation rules to grade each partition OK, WARNING, STALLED, or STOPPED, with ERROR reported at the group level.

Any zero lag in the window means OK; offsets that stop moving while lag holds or grows mean STALLED; a gap since the last commit longer than the whole window means STOPPED.

That design tolerates rebalance spikes, though the window scales with commit frequency. Frequent commits give a short window, while a low-throughput topic can take hours to flag a stop. Burrow collects no broker JMX or health metrics, which makes it a complement to Kafka Exporter rather than a replacement.

OpenTelemetry-Based Collection

The OpenTelemetry Collector’s kafka_metrics receiver scrapes brokers, topics, and consumer groups and exports them through the OpenTelemetry Protocol (OTLP). It emits metrics such as kafka.consumer_group.lag, kafka.partition.replicas_in_sync, and kafka.topic.partitions.

It supports Kafka 2.x to 4.x, including KRaft, the metadata mode that replaced ZooKeeper. The receiver has no request latency or JVM metrics, which means broker JMX signals still need a JMX-side collector running next to it.

Kafka metrics travel the same pipeline as application traces carrying messaging.consumer.group.name, messaging.destination.name, and messaging.destination.partition.id attributes. Those attributes match the group, topic, and partition labels on the lag metric, which lets one backend line a lag spike up against the spans behind it.

Managed and Commercial Platforms

Confluent Control Center is versioned separately from Confluent Platform, uses Prometheus as its metrics store, and receives broker metrics through the Confluent Telemetry Reporter into Prometheus’s OTLP endpoint.

Dynatrace’s Apache Kafka extension reads JMX through OneAgent and ships topology, dashboards, and alerts for brokers, topics, producers, and consumers.

OTLP-native observability platforms such as Coralogix accept traces, logs, and metrics through a Collector exporter.

How to Alert on Kafka Before Consumers Fall Behind

Durable Kafka alerting uses trend windows and group-level rules informed by historical baselines. Trend windows distinguish brief rebalances from persistent lag, while group-level rules catch failures that broker dashboards miss. Historical baselines then account for expected traffic cycles before an alert reaches your on-call rotation.

Alert on Trends Instead of Snapshots

Consumer lag spikes during rebalances. The spikes are sharp on the classic protocol, where consumers pause while partitions are revoked and reassigned, and smaller on Kafka 4.0’s group.protocol=consumer. Alerts should fire when lag grows continuously for a sustained period rather than on an absolute value.

A useful Prometheus rule evaluates whether the lag derivative remains positive across multiple evaluation windows. Trend evaluation cuts alert fatigue compared with raw offset distance because bursts that self-correct within seconds never page anyone.

Monitor Consumer Groups Alongside Brokers

Broker metrics can read healthy while a consumer group falls quietly behind, which is why each group needs its own dashboard and dead-consumer rule. A Prometheus dead-consumer rule alerts when lag remains unchanged across a sustained window while still above zero.

Offset lag also reads poorly to humans. A large backlog may clear quickly for one group, while a small backlog can represent a long delay for another. Teams handle this by converting lag to time, dividing it by the group’s recent consumption rate.

A consumer group is a coordination label with no lifecycle of its own and is often shared across deployments. Map each group name to a service, a team, and a deploy before routing its alerts.

What Changes in Kafka Monitoring After the Move to KRaft

Kafka 4.0, released March 18, 2025, removed ZooKeeper entirely. KRaft is the only mode on current releases, and a different set of MBeans applies.

The ZooKeeper session metrics and ZooKeeperRequestLatencyMs have no KRaft replacement, and ControlPlaneNetworkProcessorAvgIdlePercent gives way to NetworkProcessorAvgIdlePercent. The version 4.3 ZooKeeper-to-KRaft page lists LeaderElectionRateAndTimeMs, ActiveControllerCount, and OfflinePartitionsCount as removed, although the KRaft controller table on the monitoring page and KIP-1001 show the last two still exist on controller nodes.

Quorum health comes from kafka.server:type=raft-metrics, where current-leader of -1 means no known leader and commit-latency-avg and election-latency-max tell you whether controllers are committing metadata. ActiveControllerCount should still sum to 1 across controller nodes, and broker-side last-applied-record-lag-ms shows how far each broker trails the controller’s metadata log.

Baseline Before You Alert

Static byte-rate or lag thresholds fire during legitimate traffic growth and miss degradation on quiet topics. Thresholds work better when built from historical data and seasonality. A baseline-aware rule alerts when lag climbs materially above baseline and keeps growing.

Lag that spikes during a nightly batch job and then recovers is healthy. Lag that climbs steadily over hours is a consumer that can’t keep up, and a baseline that has seen both weekday and weekend traffic is what tells them apart.

Coralogix anomaly detection alerts build this baseline from the previous seven days with a “more than usual” condition per consumer group. Because they learn daily patterns rather than weekly cycles, a recurring suppression rule covers predictable weekend dips.

Pulling Kafka Signals Into One Investigation Surface

A JMX collector on each broker, Prometheus or OpenTelemetry pulling cluster state, and Burrow judging consumer health will page you on controller loss, URPs, and sustained lag growth. When the orders consumer group lags, the evidence that distinguishes a slow downstream API, a bad deploy, and a GC storm lives in distributed tracing spans and broker logs for teams running Kafka in microservices architectures.

Coralogix’s OpenTelemetry-native ingestion accepts metrics, logs, and traces from the same Collector pipeline, which lets lag sit next to the spans that explain it. Its anomaly detection alerts cover the learned baselines, and unlimited retention in your own cloud storage keeps the history those baselines and investigations draw on.

If you want to see your own consumer lag lined up against the traces behind it, start a free 14-day Coralogix trial and point your Collector’s exporter at it. You’ll see Kafka metrics alongside your logs and traces in the same place.

Frequently Asked Questions About Kafka Monitoring

What Is Consumer Lag and Why Does It Matter in Kafka?

Consumer lag is the gap between the latest offset produced to a partition and the offset a consumer group last committed. Growing lag means consumers are falling behind producers, and if the oldest unread offset ages past the topic’s retention window, messages can age out before the group reads them.

What Is the Difference Between Kafka Monitoring and Kafka Observability?

Monitoring collects predefined metrics such as lag, under-replicated partitions, and broker throughput and fires alerts when they cross a line. Observability correlates those metrics with broker logs, application logs, and distributed traces to explain why a metric moved, which is the full-stack observability model Kafka-backed microservice teams end up needing. Monitoring tells you the orders group is lagging; observability tells you which downstream call made it lag.

Is Kafka Better Than Redis for Message Streaming?

The two solve different problems. Kafka keeps a committed message safe while at least one in-sync replica is alive, lets consumers rewind offsets to replay history, and rebalances consumer groups automatically. 

Redis Streams is a lighter append-only log with explicit MAXLEN trimming, and depending on its persistence configuration, it can lose recent writes on a crash (up to one second with appendfsync everysec), and its consumer groups recover a failed consumer’s pending messages only through explicit XCLAIM or XAUTOCLAIM calls. 

For pipelines where durability, replay, and consumer group management are priorities, Kafka is the stronger fit.