querypage

package
v1.11.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 11, 2026 License: GPL-3.0 Imports: 17 Imported by: 0

Documentation

Overview

Package querypage is the one production Query → Page engine for the v2 store: every typed-table and catalog serve path executes here (maintained stores queried in place, per-Build stores, and the catalog engine store), and this file's value-keyed Cursor is the ONE cursor codec on the wire.

The historical bespoke typed-table executor and its legacy cursor codec now live only as the byte-identity test oracle (backend/refresh/snapshot/typed_table_query_oracle_test.go); the per-domain parity gates compare this engine against that brute recompute. Cursor degrade semantics (undecodable/mismatched tokens, backward dead ends) are owned by Store.Query and surface on Page.CursorInvalid so executors cannot diverge; anchored jumps and offset pages are served by Store.QueryAround / Store.QueryAt on the same contract.

Index

Constants

This section is empty.

Variables

View Source
var ErrCursorMismatch = errors.New("querypage: cursor does not match the current query")

ErrCursorMismatch means a cursor was issued for a different query than the one now executing (different cluster, query shape, sort, direction, or page size). The caller must restart from the first page rather than risk silently mispaging.

Functions

This section is empty.

Types

type AnchorOutcome added in v1.10.3

type AnchorOutcome struct {
	// Found: the anchor row exists AND matches the query's filters/search;
	// Rank is then its 0-based position among matching rows in display order.
	Found bool
	// Filtered: the anchor row exists in the store but the query's
	// filters/search exclude it. Mutually exclusive with Found.
	Filtered bool
	// Rank is valid only when Found; -1 otherwise.
	Rank int
}

AnchorOutcome reports how QueryAround resolved its anchor row under the query's filters — the engine-level truth the serve layer maps to the contract's user-visible "filtered" / "not-found" reasons.

type Cursor

type Cursor struct {
	ClusterID string    `json:"c"`
	Signature string    `json:"q"` // hash of the query shape (filters, scope, …)
	Sort      string    `json:"s"`
	Direction Direction `json:"d"`
	Limit     int       `json:"l"`

	// Position holds the single comparable sort value of the boundary row this
	// cursor pins (the store's one-comparable-per-row invariant). UID is the final,
	// always-unique tiebreak. Together they are the keyset seek position. Empty
	// Position + empty UID = first page.
	Position string `json:"p,omitempty"`
	UID      string `json:"u,omitempty"`

	// Backward marks a prev-page cursor: the engine walks the sort order DOWNWARD from
	// Position (collecting the rows immediately before it) instead of upward. It is a
	// navigation property, not part of the query shape, so Validate ignores it.
	Backward bool `json:"b,omitempty"`
}

Cursor is the value-keyed keyset position for one page of a query. It is opaque to clients (encoded as base64(json)) and self-describing: it pins the query it belongs to so a stale cursor from a different query is rejected, not mispaged.

func Decode

func Decode(token string) (Cursor, error)

Decode parses an opaque token back into a Cursor. An empty token decodes to the zero Cursor (first page).

func FirstPage

func FirstPage(clusterID, signature, sort string, direction Direction, limit int) Cursor

FirstPage builds the start cursor for a query (no position yet).

func (Cursor) Encode

func (c Cursor) Encode() string

Encode renders the cursor as an opaque base64(json) token. An empty token is returned for a first-page cursor so callers can treat "" as "start".

func (Cursor) IsFirstPage

func (c Cursor) IsFirstPage() bool

IsFirstPage reports whether the cursor is at the start position (no keyset yet).

func (Cursor) Validate

func (c Cursor) Validate(clusterID, signature, sort string, direction Direction, limit int) error

Validate checks that the cursor belongs to the query identified by the given fields. A first-page (zero) cursor always validates; any mismatch on the pinned query identity returns ErrCursorMismatch.

type Direction

type Direction string

Direction is the page traversal direction over the sort order.

const (
	Ascending  Direction = "asc"
	Descending Direction = "desc"
)

type Page

type Page[R any] struct {
	Rows       []R
	NextCursor string
	PrevCursor string
	Facets     map[string]map[string]int
	Total      int

	// CursorInvalid reports that the request's cursor was rejected — undecodable,
	// pinned to a different query shape, or an un-navigable backward dead end (a
	// valid prev-page cursor whose predecessors were all deleted). The served page
	// is a first-page restart for the first two, and the empty dead-end page for
	// the last; callers forward the flag so the client resets its pagination
	// state. Owning this in the engine keeps every executor's degrade behavior
	// identical — callers need no cursor pre-validation of their own.
	CursorInvalid bool

	// PageStartRank is the 0-based rank of the served page's first row among the
	// query's matching rows — exact position honesty for the footer. Counted
	// serves (QueryAround/QueryAt) fill it; the cursor-based Query path returns
	// -1 (not computed — the O(rank) walk it would cost is gated behind the P9
	// benchmark). The wire layer maps ≥0 to a pointer so rank 0 survives
	// omitempty.
	PageStartRank int

	// SelfCursor addresses THIS page: a later plain Query with it reproduces the
	// window (the page-stability primitive live refetches use after an anchored
	// or offset landing — cursor-addressed pages already hold their own request
	// token). Counted serves mint it from the entry preceding the window; "" for
	// a first-page window, whose address is the empty token.
	SelfCursor string
}

Page is one page of results plus the (unfiltered) facet counts, the total number of rows matching filters+search, and the cursors for the next/prev page ("" when there is no page in that direction).

type Query

type Query struct {
	ClusterID string
	Signature string // identifies the query shape (filters+scope); pins the cursor
	Sort      string // a key in Schema.SortKeys
	Direction Direction
	Limit     int
	Search    string              // case-insensitive substring filter over SearchText
	Filters   map[string][]string // facet filters: facet name -> allowed values (set membership; OR within, AND across facets)
	MatchNone bool                // explicit empty multiselect: no row can match
	Cursor    string              // opaque continueToken from a previous Page
}

Query is one page request against a kind's store.

type Schema

type Schema[R any] struct {
	UID              func(R) string
	SortKeys         map[string]func(R) string
	Facets           map[string]func(R) string
	MultiFacets      map[string]func(R) []string
	FacetNormalizers map[string]func(string) string
	SearchText       func(R) string
}

Schema describes how the engine reads a stored row of type R: how to identify it, which sort orders and facets it exposes, and what text is searchable. Sort and facet values are strings; numeric sorts must be encoded order-preservingly by the extractor (e.g. zero-padded), so lexical order matches numeric order.

UID is the store's unique row key AND the tie order for equal sort values (ascLess/descLess tie-break by it, ascending in both directions), so its shape is user-visible whenever sort values collide: give it a human-meaningful form — the typed tables use the lowercased kind/namespace/name adapter key, the catalog its identity chain — never an arbitrary identifier like the Kubernetes object UID. Pinned by TestTiedSortValuesOrderByHumanKey.

type Store

type Store[R any] struct {
	// contains filtered or unexported fields
}

Store is the generic Query → Page engine for one kind. It holds rows in an interned columnar store keyed by UID, a b-tree keyset index per sort order, and exact facet counters, all maintained incrementally on Upsert/Delete. It is generic over R via the Schema extractors — the engine carries no per-kind logic.

Row storage is a columnStore (see columnar.go): a structure-of-arrays with dictionary-interned string columns and a recycled rowId arena, replacing a plain map[string]R. To avoid an O(N)-reconstruction-per-query regression, the Schema's match inputs (facet values + lowercased SearchText) are extracted ONCE per row at Upsert time and cached by rowId in `matchByRowID`, so Query filters/searches and the filtered-Total scan read those cached values instead of reconstructing rows. Full rows are reconstructed via the codec only for the page actually returned, for Snapshot, and for the deindex of a replaced/deleted row.

func NewStore

func NewStore[R any](schema Schema[R]) *Store[R]

NewStore builds an empty store for a schema, creating a per-direction index per sort key and the columnar row store (codec built by reflecting over R's zero value).

func OpenInternedColumnStore

func OpenInternedColumnStore[R any](path string, schema Schema[R]) (*Store[R], func() error, error)

OpenInternedColumnStore reopens a file written by SpillInternedColumns as a read-only Store whose scalar columns and dictionary strings alias the mapping (off-heap), rebuilding the uid→rowID map, sort indexes, and match cache from the aliased columns. Rows decoded from the store clone string fields before returning, so Page/Snapshot rows can outlive the mapping closer while the bulk columns stay off-heap. A bad magic / field-shape mismatch / truncation returns an error so the caller falls back to the heap path.

func RestoreColumnsFromFile

func RestoreColumnsFromFile[R any](path string, schema Schema[R]) (*Store[R], error)

RestoreColumnsFromFile reads a columnar file written by SpillColumns and rebuilds a NEW equivalent Store (reconstruct each row + Upsert — rebuilding columns, dicts, indexes, match cache). A bad magic, a field-shape mismatch against the schema's codec, or a truncated file returns an error so the caller can fall back to the gob path.

func RestoreStore

func RestoreStore[R any](r io.Reader, schema Schema[R]) (*Store[R], error)

RestoreStore gob-decodes a []R written by Spill, builds a fresh Store for the given schema, and Upserts every row. Each Upsert rebuilds that row's index entries, facet counts, and match-cache entry incrementally, so the returned store is fully equivalent to the one that was spilled.

func RestoreStoreFromFile

func RestoreStoreFromFile[R any](path string, schema Schema[R]) (*Store[R], error)

RestoreStoreFromFile opens path and rebuilds a Store from the spilled rows with a buffered reader. This is what the governor's re-warm path calls.

func (*Store[R]) Delete

func (s *Store[R]) Delete(uid string)

Delete removes a row by UID, maintaining every index + facet incrementally. A read-only (mmap-aliased) store ignores deletes — see Store.readOnly.

func (*Store[R]) Len

func (s *Store[R]) Len() int

Len reports the number of stored rows.

func (*Store[R]) Query

func (s *Store[R]) Query(q Query) (Page[R], error)

Query executes a page request: it seeks to the cursor position in the chosen sort index (O(log N)), walks in the requested direction collecting rows that pass the filters + search until the page is full, and returns the next cursor plus the facet counts and total match count.

func (*Store[R]) QueryAround added in v1.10.3

func (s *Store[R]) QueryAround(q Query, anchorKey string) (Page[R], AnchorOutcome, error)

QueryAround serves the PAGE-ALIGNED window containing the row whose schema UID equals anchorKey, under q's sort, direction, filters, and search — the engine primitive behind "jump to this object in the list". One counted O(rank + limit) walk (plus skipped non-matching entries) yields the exact 0-based rank, the aligned window (pageStart = rank - rank%limit), and ordinary keyset prev/next cursors minted from the window boundaries — so pagination after a jump is indistinguishable from pagination that arrived from page 1, and rank is derived per request, never stored. A missing or filtered-out anchor serves the FIRST page instead (one round trip, sane landing) with the outcome saying why. q.Cursor is ignored: anchor and continue are mutually exclusive at the contract layer.

func (*Store[R]) QueryAt added in v1.10.3

func (s *Store[R]) QueryAt(q Query, startRank int) (Page[R], error)

QueryAt serves the page starting at startRank (0-based) among matching rows — the bounded offset contract behind numbered page jumps. startRank is clamped to the last page-aligned start (negatives to 0) so a stale page number lands on the nearest real page instead of an empty one; the served start is reported on Page.PageStartRank. q.Cursor is ignored.

func (*Store[R]) ReopenInternedColumnsInPlace

func (s *Store[R]) ReopenInternedColumnsInPlace(path string) (func() error, error)

ReopenInternedColumnsInPlace spills this store's interned columns to path, then swaps the store's internals (columns, indexes, match cache) to a read-only, mmap-aliased view of that file — so the SAME *Store pointer keeps serving queries (callers holding it via a maintained store need no rewiring) but with its column data off-heap. Returned Page/Snapshot rows clone decoded strings, so the closer can run after the store is discarded or re-warmed without invalidating already-returned rows. On any error the store is left unchanged (safe-degrade: the caller keeps serving from heap). This is the Tier 2.6 Cold-serving transition at the store level.

func (*Store[R]) ReplaceWhere

func (s *Store[R]) ReplaceWhere(rows []R, owns func(R) bool)

ReplaceWhere replaces the rows owned by one source with rows from that source, preserving rows owned by other sources. Passing owns=nil replaces the whole store. It is the batch counterpart to Upsert/Delete for reflector relists: rebuild derived indexes/facets once from the post-relist row set instead of issuing N incremental updates through downstream sinks.

func (*Store[R]) RestoreColumnsFromFileInto

func (s *Store[R]) RestoreColumnsFromFileInto(path string) error

RestoreColumnsFromFileInto reads a columnar file written by SpillColumns and Upserts its rows INTO this store (using the store's own schema/codec), for restoring into a freshly-built maintained store in place. Same validation + error contract as RestoreColumnsFromFile; the store is left partially populated only if a per-row Upsert follows a successful parse (parse errors happen before any Upsert).

func (*Store[R]) RestoreFrom

func (s *Store[R]) RestoreFrom(r io.Reader) error

RestoreFrom gob-decodes a []R written by Spill and Upserts every row INTO THIS store, rebuilding each row's index/facet/match-cache entry incrementally — the same per-row path Upsert always takes. Unlike RestoreStore it loads into an existing store, preserving its schema and any wiring that already references it (a maintained store's informer handlers / ingest sink), which is what the governor's warm-restore on re-warm needs: the store is built+wired first, then pre-painted from disk before the fresh informers feed. Rows already present are overwritten by UID; rows absent from the spill are left untouched, so a later reconciling sync removes any that no longer exist upstream.

func (*Store[R]) RestoreFromFile

func (s *Store[R]) RestoreFromFile(path string) error

RestoreFromFile opens path and loads its spilled rows into this store with a buffered reader. It is the file counterpart of RestoreFrom — what the governor's re-warm path calls to pre-paint a freshly-built maintained store from its on-disk spill.

func (*Store[R]) Scope

func (s *Store[R]) Scope(base map[string][]string, search string) (map[string]map[string]int, int)

Scope returns, over the rows matching `base` filters + `search`, the per-facet value→count map and the total — computed from the by-rowId match cache WITHOUT reconstructing any row. Same filter/search semantics as Query (OR within a facet, AND across facets; case-insensitive substring search). The facet values are the raw values the schema's facet extractor produced (e.g. lowercased), matching the stored match cache. This is the cheap O(N)-no-reconstruction count a direct serve uses for facets + UnfilteredTotal.

func (*Store[R]) Snapshot

func (s *Store[R]) Snapshot() []R

Snapshot returns a copy of every stored row (unordered) under a read lock. It lets a caller hand the store's current rows to another consumer (e.g. a scope-filtered re-query) without exposing the internal store.

func (*Store[R]) Spill

func (s *Store[R]) Spill(w io.Writer) error

Spill gob-encodes the store's rows ([]R from Snapshot) to w. Snapshot takes the store's read lock and returns an independent copy, so the encode runs without holding the lock (avoiding a recursive RLock) yet still sees a consistent set of rows. The derived indexes/facets/match-cache are intentionally not written; Restore rebuilds them from the rows.

func (*Store[R]) SpillColumns

func (s *Store[R]) SpillColumns(path string) error

SpillColumns writes the store's rows to path in the columnar format.

func (*Store[R]) SpillInternedColumns

func (s *Store[R]) SpillInternedColumns(path string) error

SpillInternedColumns writes the store's interned columns to path in the mmap-aliasable format.

func (*Store[R]) SpillToFile

func (s *Store[R]) SpillToFile(path string) error

SpillToFile creates/truncates path and writes the store's spilled rows to it with a buffered writer. This is what the governor's Cold action calls.

func (*Store[R]) Upsert

func (s *Store[R]) Upsert(row R)

Upsert inserts or replaces a row, maintaining every index + facet incrementally. A read-only (mmap-aliased) store ignores writes — see Store.readOnly.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址