promqle2e

package module
v0.0.0-...-c731bef Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

promqle2e

This directory contains the promqle2e Go module offering a docker based Go framework for running fast, tailored acceptance tests for Prometheus compatibility, integrated into go test.

Compared to ../promql, that takes ~hours to run, it's focused on a quick feedback loop setup that runs within a ~minute to tell you if your "backend" are serving, on PromQL endpoint, certain metric types or cases as expected. Suitable for CI runs and quick debugging sessions.

Backend means any collection and storage pipeline e.g.:

  • Prometheus
  • Prometheus + Cortex
  • Prometheus + Thanos
  • Prometheus + vendor
  • otel-col + vendor
  • etc.

Collection can be either scrape bases on remote write based. More ingestion APIs might be added in the future.

Using

  1. Ensure prerequisites are installed: Go 1.23+ and Docker.
  2. Get promqle2e module go get github.com/prometheus/compliance/promqle2e@latest
  3. Create a new Go test in _test.go test file, similar to normal unit test.
  4. Write a scrape based tests e.g.:
func TestPrometheus_Counter_e2e(t *testing.T) {
	const interval = 30 * time.Second
  
	// Use prometheus backend, create your own backend OR use multiple ones for reference.
	prom1 := promqle2e.PrometheusBackend{Name: "prom1", Image: "quay.io/prometheus/prometheus:v3.2.0"}
	prom2 := promqle2e.PrometheusBackend{Name: "prom2", Image: "quay.io/prometheus/prometheus:v2.55.0"}

	// Create a scrape style test
	pt := promqle2e.NewScrapeStyleTest(t)

	//nolint:promlinter // Create a test metric.
	counter := promauto.With(pt.Registerer()).NewCounterVec(prometheus.CounterOpts{
		Name: "promqle2e_test_counter_total",
		Help: "Test counter used by promqle2e test framework for acceptance tests.",
	}, []string{"foo"})
	var c prometheus.Counter

	// Now record fake scrapes and its expected samples for a given metric and backend.
	
	// No metric expected, counterVec empty.
	pt.RecordScrape(interval)

	c = counter.WithLabelValues("bar")
	c.Add(200)
	pt.RecordScrape(interval).
		Expect(c, 200, prom1).
		Expect(c, 200, prom2)

	c.Add(10)
	pt.RecordScrape(interval).
		Expect(c, 210, prom1).
		Expect(c, 210, prom2)

	// so on... as much as you need to validate.

	// Setup resonable timeout e.g. default Prometheus backend performs real scrapes 
	// with 1s interval so each "RecordScrape" requires at least 1s.
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	t.Cleanup(cancel)
	
	// Actual test.
	pt.Run(ctx)
}

Documentation

Index

Constants

View Source
const (
	// PrometheusBackendModeOM that uses OpenMetrics recorded scrapes to ingest data.
	PrometheusBackendModeOM = PrometheusBackendMode("om")
	// PrometheusBackendModeRW2 that uses Remote Write 2.0 receiver to ingest data.
	PrometheusBackendModeRW2 = PrometheusBackendMode("prw2")
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Backend

type Backend interface {
	// Ref returns unique representation of the backend.
	Ref() string

	// StartAndWaitReady starts all pieces that has to be started (until ready) to
	// obtain the running backend e.g. docker containers.
	StartAndWaitReady(t testing.TB, env e2e.Environment) RunningBackend
}

Backend represents a metric backend pipeline that can be started.

type ExpectationsRecorder

type ExpectationsRecorder interface {
	// Expect sets an expectation for metric to have a float value of val for backend b.
	// NOTE(bwplotka): For multiple backends on the same test that reuse the same PromQL storage, CollectionLabels
	// has to be unique.
	Expect(metric prometheus.Metric, val float64, b Backend) ExpectationsRecorder
}

ExpectationsRecorder allows recording results for a scrape. TODO(bwplotka): Add histogram support.

type IngestByScrapeReplayer

type IngestByScrapeReplayer struct {
	// contains filtered or unexported fields
}

IngestByScrapeReplayer is a goroutine that expose scrape endpoint that will replay recorded samples on scrape. Only one scrape can touch this endpoint for an accurate recording.

It implements RunningBackend.IngestSamples which injects data to scrape. Before or after the replay the scrape target exposes no metrics.

This is essential when testing scraping part of the collectors like Prometheus or OpenTelemetry Collector.

func StartIngestByScrapeReplayer

func StartIngestByScrapeReplayer(t testing.TB, env e2e.Environment) *IngestByScrapeReplayer

StartIngestByScrapeReplayer starts and returns the replayer. It finishes when env is closing. See IngestByScrapeReplayer for details.

func (*IngestByScrapeReplayer) Endpoint

func (r *IngestByScrapeReplayer) Endpoint(env e2e.Environment) string

Endpoint returns host:port address where the server listens for scrapes.

func (*IngestByScrapeReplayer) IngestSamples

func (r *IngestByScrapeReplayer) IngestSamples(ctx context.Context, t testing.TB, recorded [][]*dto.MetricFamily)

type PrometheusBackend

type PrometheusBackend struct {
	Image string
	Name  string
	Mode  PrometheusBackendMode
}

PrometheusBackend See defaultPrometheusBackend for defaults.

func (PrometheusBackend) Ref

func (opts PrometheusBackend) Ref() string

func (PrometheusBackend) StartAndWaitReady

func (opts PrometheusBackend) StartAndWaitReady(t testing.TB, env e2e.Environment) RunningBackend

type PrometheusBackendMode

type PrometheusBackendMode = string

type RunningBackend

type RunningBackend interface {
	// API returns PromQL client connected to the backend.
	API() v1.API
	// CollectionLabels returns any extra labels that are expected to be added
	// by the backend's collection (e.g. job label for Prometheus).
	// This will be added to expected labels when comparing query result with the
	// expectations.
	CollectionLabels() map[string]string

	// IngestSamples ingests the recorded data (series with ordered samples)
	// to the backend so it can be queried later. It's up to the backend how
	// it's ingested e.g.
	// *  NewPrometheusBackend offers both scrape based (via RecordedGatherer
	// with explicit timestamps and OpenMetrics) and Remote Write based flow.
	IngestSamples(ctx context.Context, t testing.TB, recorded [][]*dto.MetricFamily)
}

RunningBackend represents a running metric backend, so any collection and storage pipeline that serves PromQL e.g.: * Prometheus * Prometheus + Cortex * Prometheus + Thanos * Prometheus + vendor * otel-col + vendor , etc.

func NewRunningScrapeReplayBasedBackend

func NewRunningScrapeReplayBasedBackend(
	replayer *IngestByScrapeReplayer,
	collectionLabels map[string]string,
	api v1.API,
) RunningBackend

NewRunningScrapeReplayBasedBackend is a helper function for crafting RunningBackend from scrape replayer, expected collection labels and Prometheus API client.

type ScrapeStyleTest

type ScrapeStyleTest struct {
	// contains filtered or unexported fields
}

func NewScrapeStyleTest

func NewScrapeStyleTest(t *testing.T, opts ...e2e.EnvironmentOption) *ScrapeStyleTest

NewScrapeStyleTest returns test instance that allows scrape style test. It allows registering metrics and recording fake scrapes with the expected values. That data will be then injected to expected backends and compared with the backend QueryInstant `<metric>[10h]` PromQL query.

All test functionality fails tests on error. TODO(bwplotka): Consider wrapping e2e.EnvironmentOptions with our own for stability.

func (*ScrapeStyleTest) Expect

func (*ScrapeStyleTest) RecordScrape

func (t *ScrapeStyleTest) RecordScrape(after time.Duration) ExpectationsRecorder

RecordScrape records a scrape with the chained expectations for a metric and backend. The after parameter controls the sample explicit timestamp to use in this scrape. See also SetCurrentTime. This allows a limited backfilling without waiting.

Run will replay those scraped record in real scrape interval e.g. 1s for Prometheus but explicit timestamp can shape the samples for test purposes.

A good "after" time would be a stable fake "scrape" interval e.g., 15-30 seconds.

0 or negative after durations (or variations of SetCurrentTime use) can cause backends to return "out of order" or "too old samples" errors.

func (*ScrapeStyleTest) RegisterBackends

func (t *ScrapeStyleTest) RegisterBackends(bs ...Backend)

RegisterBackends registers new backends for the test. This is optional, every backend used in the RecordScrape(...).Expect(...) method will be automatically registered.

func (*ScrapeStyleTest) Registerer

func (t *ScrapeStyleTest) Registerer() prometheus.Registerer

func (*ScrapeStyleTest) Run

func (t *ScrapeStyleTest) Run(ctx context.Context)

Run performs the test, by actually injecting the recorded data and expecting the PromQL output based on RecordScrape(...).Expect(...) methods. Only once and one of this can be run per NewScrapeStyleTest.

func (*ScrapeStyleTest) SetCurrentTime

func (t *ScrapeStyleTest) SetCurrentTime(currTime time.Time)

SetCurrentTime allows setting the current timestamp to a new value, which will be used on explicit timestamp for the record samples. By default, it starts with time.Now()-1h.

func (*ScrapeStyleTest) Transform

func (t *ScrapeStyleTest) Transform(transformFn func([][]*dto.MetricFamily) [][]*dto.MetricFamily)

Transform allows performing any transformations on top of the scrape recorded data e.g. adding/removing CT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址