Skip to content

backfill-earliest exports inflated lag while the backfill is still running #874

Open

Description

@MilkyWay-core

Version: v1.6.0 (same code in v1.9.6 / master: core/internal/consumer/kafka_client.go, core/internal/httpserver/prometheus.go)

Setup

[consumer.kafka-messaging]
class-name="kafka"
start-latest="true"
backfill-earliest="true"

in-memory storage, Prometheus exporter scraped every 10 s; __consumer_offsets with 50 partitions, a few GB of data, a few hundred consumer groups.

What happens

Right after start Burrow serves /metrics while the backfill consumers are still replaying __consumer_offsets from the oldest offset. For every partition whose group has not committed since Burrow started, the offset ring contains only the historical commits replayed so far, so CurrentLag = current broker offset − old committed offset is far above the real lag until the replay reaches the tail of __consumer_offsets.

Observed: at the first scrape after start the total lag over all groups was tens of times higher than the real value and needed several minutes (the backfill duration, CPU-bound) to converge back. Individual groups were reported with a lag one to two orders of magnitude above their real one. Prometheus alerts with a short for fired on these values and resolved by themselves once the backfill finished.

Why

  • handlePrometheusMetrics exports partition.CurrentLag for every partition unconditionally; only current_offset (and topic_partition_status since 1.7) are gated on partition.Complete == 1.0.
  • AppReady (used by /burrow/admin/ready) is set as soon as the consumer modules have started (consumer/coordinator.go), i.e. before the backfill goroutines finish, so the readiness endpoint cannot be used to hold scrapes back either.

Expected

Either of:

  1. /burrow/admin/ready returns READY only after all backfill partition consumers have reached their end offset (they are already tracked in module.running), so a Kubernetes readiness probe can keep the pod out of the scrape targets until the state is complete; or
  2. the Prometheus exporter skips (or marks) partitions whose ring is still being backfilled, e.g. exports lag only when partition.Complete == 1.0, the same way it already does for current_offset.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址