You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
backfill-earliest exports inflated lag while the backfill is still running #874
in-memory storage, Prometheus exporter scraped every 10 s; __consumer_offsets with 50 partitions, a few GB of data, a few hundred consumer groups.
What happens
Right after start Burrow serves /metrics while the backfill consumers are still replaying __consumer_offsets from the oldest offset. For every partition whose group has not committed since Burrow started, the offset ring contains only the historical commits replayed so far, so CurrentLag = current broker offset − old committed offset is far above the real lag until the replay reaches the tail of __consumer_offsets.
Observed: at the first scrape after start the total lag over all groups was tens of times higher than the real value and needed several minutes (the backfill duration, CPU-bound) to converge back. Individual groups were reported with a lag one to two orders of magnitude above their real one. Prometheus alerts with a short for fired on these values and resolved by themselves once the backfill finished.
Why
handlePrometheusMetrics exports partition.CurrentLag for every partition unconditionally; only current_offset (and topic_partition_status since 1.7) are gated on partition.Complete == 1.0.
AppReady (used by /burrow/admin/ready) is set as soon as the consumer modules have started (consumer/coordinator.go), i.e. before the backfill goroutines finish, so the readiness endpoint cannot be used to hold scrapes back either.
Expected
Either of:
/burrow/admin/ready returns READY only after all backfill partition consumers have reached their end offset (they are already tracked in module.running), so a Kubernetes readiness probe can keep the pod out of the scrape targets until the state is complete; or
the Prometheus exporter skips (or marks) partitions whose ring is still being backfilled, e.g. exports lag only when partition.Complete == 1.0, the same way it already does for current_offset.
Activity