querytee

package
v4.0.0-rc.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: AGPL-3.0 Imports: 46 Imported by: 0

README

query-tee

query-tee is a proxy that sits between a client and several Loki backends. It sends every write to every backend. It sends every read to every backend, except that the secondary backend receives only the reads that the selection strategy picks. It returns the response of the primary backend to the client. It can compare the responses of two backends.

Backends

-backend.endpoints is a comma-separated list of the backend endpoints. Each backend has one role:

Role How to set it Required Count
Primary -primary-backend.hostname=<hostname> Yes Exactly 1
Secondary -secondary-backend.hostname=<hostname> No At most 1
Other Listed in -backend.endpoints only No Any

To stop sending traffic to a backend, remove it from -backend.endpoints.

Which backends receive the traffic

Every backend receives every write on a registered write route. Every backend except the secondary receives every read on a registered read route. The secondary receives the reads that the selection strategy picks. The role of a backend otherwise only decides which response the client gets and which responses the proxy compares.

Traffic Backends that receive it Response the client gets
Read routes All, but the secondary only for the selected reads The primary's response, even when it is an error
Write routes (push) All The primary's response, even when it is an error
Other routes The primary only The primary's response

The proxy forwards other routes only when you set -proxy.passthrough-non-registered-routes=true.

The proxy returns the primary's response as soon as it arrives. It never falls back to another backend. The proxy collects the responses of the other backends in the background.

Which reads the secondary backend receives

-secondary-backend.selection-strategy=<strategy> picks the reads that the secondary backend receives:

Strategy Reads that the secondary backend receives
all Every read. This is the default.
dataobj The range and instant queries that the secondary backend reads from data objects.

The dataobj strategy applies the eligibility check of the query-frontend with the limits of the secondary backend's Loki cluster. Set -config.file=<path> to the Loki config file of that cluster, and -runtime-config.file=<path> for per-tenant limits. The proxy reads only the limits_config block of the config file. Set -dataobj.enabled and -dataobj.storage-lag to the values of that cluster. They have the same names and defaults as the Loki flags. When -dataobj.enabled is false, the strategy selects no read. The strategy selects a query when the query-frontend marks it eligible for data objects, whatever share of the time range data objects hold.

The loki_querytee_secondary_backend_decisions_total metric counts the reads by the decision to send them to the secondary backend, or the reason not to.

Which backends get compared

When you set -proxy.compare-responses=true, the proxy compares the secondary's response with the primary's response. This flag requires you to also set -secondary-backend.hostname=<hostname>. The proxy never compares other backends. The proxy compares only the reads that the secondary backend receives.

The proxy compares only the following routes:

  • /loki/api/v1/query_range
  • /loki/api/v1/query
  • /api/prom/query

The proxy compares in the background after the backends have answered, so the comparison adds no latency for the client.

The loki_querytee_responses_compared_total metric counts the comparisons by result and cause:

Result Meaning
match The responses are equal.
mismatch The responses differ.
fail The secondary request failed, the secondary did not answer with status 200, or the comparator could not compare the bodies.
skipped The primary failed or did not answer with status 200, or no samples remain after the proxy skips samples.

The cause label says how the bodies differ for a mismatch. For a fail or a skipped pair, it says why the proxy could not compare the bodies. When a backend failed, the cause describes the secondary for a fail, and the primary for a skipped pair:

Cause Result Meaning
timeout fail, skipped The backend did not answer before its timeout.
transport fail, skipped The request to the backend failed before a response arrived, for a reason other than a timeout.
http_4xx fail, skipped The backend answered with a 4xx status.
http_5xx fail, skipped The backend answered with a 5xx status.
http_other fail, skipped The backend answered with another status that is not 200.
compare_error fail The comparator could not compare the bodies.
status_mismatch mismatch The status fields of the response bodies differ, for example success and error.
result_type_mismatch mismatch The resultType fields of the response bodies differ.
metric_count_mismatch mismatch The responses have a different number of series.
metric_missing mismatch A series of the primary's response is missing from the secondary's response.
sample_count_mismatch mismatch A series has a different number of samples.
sample_value_mismatch mismatch A sample has a different value.
sample_timestamp_mismatch mismatch A sample has a different timestamp.
stream_count_mismatch mismatch The responses have a different number of streams.
stream_missing mismatch A stream of the primary's response is missing from the secondary's response.
stream_entry_count_mismatch mismatch A stream has a different number of entries.
stream_timestamp_mismatch mismatch An entry has a different timestamp.
stream_line_mismatch mismatch An entry has a different line.
structured_metadata_count_mismatch mismatch An entry has a different number of structured metadata pairs.
structured_metadata_mismatch mismatch An entry has different structured metadata.
parsed_labels_count_mismatch mismatch An entry has a different number of parsed label pairs.
parsed_labels_mismatch mismatch An entry has different parsed labels.

The cause is empty for a match, and for a skipped pair when no samples remain.

The proxy skips the samples from the last -proxy.compare-skip-recent-samples (default 60s) and the samples before -proxy.compare-skip-samples-before.

When several result rows apply, the proxy checks the cases in this order and uses the first that matches: the primary request failed, the secondary request failed, the primary status is not 200, the secondary status is not 200.

The loki_querytee_secondary_backend_stats_ratio metric observes the ratio of a query stat of the secondary over the same stat of the primary. The proxy observes it for each compared pair when both backends return a decoded response. It skips a stat when the primary value is zero.

Label Values
route The route of the compared request.
stat bytes_processed, lines_processed, exec_time, queue_time, splits, shards
queriers_load_status both_idle, primary_busy, secondary_busy, both_busy
result_cache_status both_miss, primary_hit, secondary_hit, both_hit

A backend is busy when its query waited more than 100ms in the queue. A backend hits the cache when the range query, instant metric or log results cache of its query-frontend found at least one entry.

To also report the number of missing series in the loki_querytee_missing_metrics_series metric, set -proxy.compare-report-missing-series=true. This flag requires -proxy.compare-responses=true.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BackendResponse

type BackendResponse struct {
	// contains filtered or unexported fields
}

type BackendRole

type BackendRole int

BackendRole tells how the proxy uses a backend.

const (
	// RoleOther is a backend that is neither the primary nor the secondary. It receives every
	// request on a registered route. The proxy never returns its response and never compares it.
	RoleOther BackendRole = iota
	// RolePrimary is the backend whose response the proxy returns to the client, even when the
	// response is an error. The proxy does not fall back to another backend.
	RolePrimary
	// RoleSecondary is the backend whose responses the proxy compares with the primary's. It
	// receives every write on a registered route. It receives a read only when the selection
	// strategy picks it. The proxy never returns its response. When -proxy.compare-responses is
	// set, the proxy compares its response with the primary's on routes that have a comparator.
	RoleSecondary
)

func (BackendRole) String

func (r BackendRole) String() string

String returns the name of the role. It is the value of the backend_alias metric label.

type DataObjConfig

type DataObjConfig struct {
	Enabled    bool
	StorageLag time.Duration
}

DataObjConfig holds the data object settings of the secondary backend's Loki cluster. The flags have the names, defaults and descriptions of the Loki flags. querytee does not read these settings from the Loki config file.

func (*DataObjConfig) RegisterFlags

func (cfg *DataObjConfig) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers the flags of the data object config.

type FanOutHandler

type FanOutHandler struct {
	// contains filtered or unexported fields
}

FanOutHandler implements queryrangebase.Handler and fans out requests to multiple backends. It sends each request to every backend, except that the secondary backend strategy decides whether the secondary backend gets it. It returns the primary backend's response as soon as it is ready. When it has a comparator, it compares the secondary backend's response with the primary's in the background. It never returns the secondary backend's response.

func NewFanOutHandler

func NewFanOutHandler(cfg FanOutHandlerConfig) *FanOutHandler

NewFanOutHandler creates a new FanOutHandler.

func (*FanOutHandler) Do

Do implements queryrangebase.Handler. It sends the request to every backend, except that the secondary backend strategy decides whether the secondary backend gets it. It returns the primary backend's response. In the background, when it has a comparator, it compares the secondary backend's response with the primary's.

func (*FanOutHandler) WithComparator

func (h *FanOutHandler) WithComparator(comparator comparator.ResponsesComparator) *FanOutHandler

WithComparator sets the response comparator.

type FanOutHandlerConfig

type FanOutHandlerConfig struct {
	Backends            []*ProxyBackend
	Codec               queryrangebase.Codec
	Logger              log.Logger
	Metrics             *ProxyMetrics
	RouteName           string
	Comparator          comparator.ResponsesComparator
	ReportMissingSeries bool

	// SecondaryBackendStrategy picks the requests that the secondary backend receives. Without a
	// strategy, the secondary backend receives every request.
	SecondaryBackendStrategy SelectionStrategy
}

FanOutHandlerConfig holds configuration for creating a FanOutHandler.

type HandlerFactory

type HandlerFactory struct {
	// contains filtered or unexported fields
}

HandlerFactory creates the query handler of a route.

func NewHandlerFactory

func NewHandlerFactory(cfg HandlerFactoryConfig) *HandlerFactory

NewHandlerFactory creates a new HandlerFactory.

func (*HandlerFactory) CreateHandler

func (f *HandlerFactory) CreateHandler(routeName string, comp comparator.ResponsesComparator) http.Handler

CreateHandler returns the query handler of a route. It sends each request to every backend, except that the strategy decides whether the secondary backend receives it. Without a secondary backend or a strategy, it sends every request to every backend.

type HandlerFactoryConfig

type HandlerFactoryConfig struct {
	Backends            []*ProxyBackend
	Codec               queryrangebase.Codec
	ReportMissingSeries bool
	Logger              log.Logger
	Metrics             *ProxyMetrics

	// SecondaryBackendStrategy picks the requests that the secondary backend receives. Without a
	// strategy, the secondary backend receives every request.
	SecondaryBackendStrategy SelectionStrategy
}

HandlerFactoryConfig holds configuration for creating a HandlerFactory.

type InstrumentationServer

type InstrumentationServer struct {
	// contains filtered or unexported fields
}

func NewInstrumentationServer

func NewInstrumentationServer(port int, registry *prometheus.Registry, logger log.Logger) *InstrumentationServer

NewInstrumentationServer returns a server exposing Prometheus metrics.

func (*InstrumentationServer) Start

func (s *InstrumentationServer) Start() error

Start the instrumentation server.

func (*InstrumentationServer) Stop

func (s *InstrumentationServer) Stop()

Stop closes the instrumentation server.

type LimitsConfig

type LimitsConfig struct {
	ConfigFile    string
	ExpandEnv     bool
	RuntimeConfig runtimeconfig.Config
}

LimitsConfig configures how querytee loads the limits of a Loki cluster. The flag names match the ones in Loki. The config file is read for the limits_config block only. The runtime config is read for the overrides only. When neither file is set, querytee loads no limits.

func (*LimitsConfig) RegisterFlags

func (cfg *LimitsConfig) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers the flags of the limits config.

type LokiLimits

type LokiLimits struct {
	// Overrides resolves the limits of a tenant: the per-tenant overrides of the runtime config on
	// top of the defaults of limits_config.
	Overrides *validation.Overrides
	// contains filtered or unexported fields
}

LokiLimits holds the limits that querytee loaded from the config files of a Loki cluster.

The runtime config reloads while querytee runs, so Overrides sees changed per-tenant limits. The limits_config defaults stay as loaded at start-up.

func LoadLokiLimits

func LoadLokiLimits(ctx context.Context, cfg LimitsConfig, logger log.Logger, registerer prometheus.Registerer) (*LokiLimits, error)

LoadLokiLimits loads the limits from the Loki config and runtime config files that the config names. It waits for the first load of the runtime config. It returns nil when the config names no file.

func (*LokiLimits) Stop

func (l *LokiLimits) Stop(ctx context.Context) error

Stop stops the reload of the runtime config. It is safe to call on a nil LokiLimits.

type NonDecodableResponse

type NonDecodableResponse struct {
	StatusCode int
	Body       []byte
}

NonDecodableResponse is a minimal response type used when returning errors. It satisfies the queryrangebase.Response interface, and allows the querytee to capture responses that would otherwise be lost.

func (*NonDecodableResponse) GetHeaders

GetHeaders implements queryrangebase.Response

func (*NonDecodableResponse) ProtoMessage

func (e *NonDecodableResponse) ProtoMessage()

ProtoMessage implements proto.Message

func (*NonDecodableResponse) Reset

func (e *NonDecodableResponse) Reset()

Reset implements proto.Message

func (*NonDecodableResponse) SetHeader

func (e *NonDecodableResponse) SetHeader(_, _ string)

SetHeader implements queryrangebase.Response

func (*NonDecodableResponse) String

func (e *NonDecodableResponse) String() string

String implements proto.Message

func (*NonDecodableResponse) WithHeaders

WithHeaders implements queryrangebase.Response

type PrimaryBackendConfig

type PrimaryBackendConfig struct {
	Hostname string
}

PrimaryBackendConfig configures the backend whose response the proxy returns to the client.

func (*PrimaryBackendConfig) RegisterFlags

func (cfg *PrimaryBackendConfig) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers the flags of the primary backend.

type Proxy

type Proxy struct {
	// contains filtered or unexported fields
}

func NewProxy

func NewProxy(
	cfg ProxyConfig,
	logger log.Logger,
	readRoutes, writeRoutes []Route,
	registerer prometheus.Registerer,
	lokiLimits *LokiLimits,
) (*Proxy, error)

NewProxy returns a Proxy for the backends in cfg.BackendEndpoints.

Exactly one backend must match the primary hostname. When the secondary hostname is set, exactly one other backend must match it. A backend that matches neither is an "other" backend. The proxy sends it every request and never compares its responses.

func (*Proxy) Await

func (p *Proxy) Await()

func (*Proxy) Endpoint

func (p *Proxy) Endpoint() string

func (*Proxy) Start

func (p *Proxy) Start() error

func (*Proxy) Stop

func (p *Proxy) Stop() error

type ProxyBackend

type ProxyBackend struct {
	// contains filtered or unexported fields
}

ProxyBackend holds the information of a single backend.

func NewProxyBackend

func NewProxyBackend(name string, endpoint *url.URL, timeout time.Duration, role BackendRole) (*ProxyBackend, error)

NewProxyBackend makes a new ProxyBackend.

func (*ProxyBackend) Alias

func (b *ProxyBackend) Alias() string

Alias returns the backend alias label value for metrics. It is "primary", "secondary" or "other".

func (*ProxyBackend) ForwardRequest

func (b *ProxyBackend) ForwardRequest(orig *http.Request, body io.ReadCloser) *BackendResponse

type ProxyBackends

type ProxyBackends []*ProxyBackend

ProxyBackends is a list of backends.

func (ProxyBackends) CountFunc

func (bs ProxyBackends) CountFunc(match func(*ProxyBackend) bool) int

CountFunc returns the number of backends for which match returns true.

type ProxyConfig

type ProxyConfig struct {
	ServerServicePort              int
	BackendEndpoints               string
	BackendReadTimeout             time.Duration
	CompareResponses               bool
	ValueComparisonTolerance       float64
	UseRelativeError               bool
	PassThroughNonRegisteredRoutes bool
	SkipRecentSamples              time.Duration
	SkipSamplesBefore              flagext.Time
	ReportMissingSeries            bool

	PrimaryBackend   PrimaryBackendConfig
	SecondaryBackend SecondaryBackendConfig
	DataObj          DataObjConfig
}

func (*ProxyConfig) RegisterFlags

func (cfg *ProxyConfig) RegisterFlags(f *flag.FlagSet)

type ProxyEndpoint

type ProxyEndpoint struct {
	// contains filtered or unexported fields
}

func NewProxyEndpoint

func NewProxyEndpoint(
	backends []*ProxyBackend,
	routeName string,
	metrics *ProxyMetrics,
	logger log.Logger,
) *ProxyEndpoint

func (*ProxyEndpoint) ServeHTTP

func (p *ProxyEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*ProxyEndpoint) WithQueryHandler

func (p *ProxyEndpoint) WithQueryHandler(handler http.Handler) *ProxyEndpoint

WithQueryHandler sets the middleware-based query handlers (logs and metrics) for the endpoint. When set, ServeHTTP uses this handler instead of the legacy executeBackendRequests.

type ProxyMetrics

type ProxyMetrics struct {
	// contains filtered or unexported fields
}

func NewProxyMetrics

func NewProxyMetrics(registerer prometheus.Registerer) *ProxyMetrics

type QueryHandler

type QueryHandler struct {
	// contains filtered or unexported fields
}

QueryHandler decodes a query request, sends it to the downstream handler and writes the response.

func NewQueryHandler

func NewQueryHandler(codec queryrangebase.Codec, fanOutHandler queryrangebase.Handler, logger log.Logger) *QueryHandler

NewQueryHandler returns a handler that serves query requests with the given handler. It requires the org ID of the tenant on every request.

func (*QueryHandler) ServeHTTP

func (h *QueryHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler. It answers with its own error when it cannot read the tenant or decode the request.

type Route

type Route struct {
	Path               string
	RouteName          string
	Methods            []string
	ResponseComparator comparator.ResponsesComparator
}

type SecondaryBackendConfig

type SecondaryBackendConfig struct {
	Hostname          string
	SelectionStrategy string
}

SecondaryBackendConfig configures the backend whose responses the proxy compares with the primary backend's responses when -proxy.compare-responses is set. The proxy never returns its responses to the client.

func (*SecondaryBackendConfig) RegisterFlags

func (cfg *SecondaryBackendConfig) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers the flags of the secondary backend.

func (*SecondaryBackendConfig) Validate

func (cfg *SecondaryBackendConfig) Validate() error

Validate checks that the selection strategy is a known one, and that a strategy other than the default has a secondary backend.

type Selection

type Selection struct {
	Selected bool
	// Reason is a value of the decision label. It comes from a small fixed set per strategy. A
	// strategy sets it only when Selected is false. The router counts a selected request as "sent".
	Reason string
}

Selection is the answer of a SelectionStrategy for one request.

type SelectionStrategy

type SelectionStrategy interface {
	// Select reports whether the secondary backend gets a copy of the request. The decision covers
	// the whole request. The proxy does not split its time range. The request can be of any type
	// that the codec decodes, not only a range or instant query.
	Select(tenants []string, req queryrangebase.Request) Selection
}

SelectionStrategy decides which requests the secondary backend receives. The proxy calls it from many request goroutines, before it calls the primary backend. An implementation must be safe for concurrent use, cheap, and free of I/O.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址