Documentation
¶
Overview ¶
Package querypage is the one production Query → Page engine for the v2 store: every typed-table and catalog serve path executes here (maintained stores queried in place, per-Build stores, and the catalog engine store), and this file's value-keyed Cursor is the ONE cursor codec on the wire.
The historical bespoke typed-table executor and its legacy cursor codec now live only as the byte-identity test oracle (backend/refresh/snapshot/typed_table_query_oracle_test.go); the per-domain parity gates compare this engine against that brute recompute. Cursor degrade semantics (undecodable/mismatched tokens, backward dead ends) are owned by Store.Query and surface on Page.CursorInvalid so executors cannot diverge; anchored jumps and offset pages are served by Store.QueryAround / Store.QueryAt on the same contract.
Index ¶
- Variables
- type AnchorOutcome
- type Cursor
- type Direction
- type Page
- type Query
- type Schema
- type Store
- func NewStore[R any](schema Schema[R]) *Store[R]
- func OpenInternedColumnStore[R any](path string, schema Schema[R]) (*Store[R], func() error, error)
- func RestoreColumnsFromFile[R any](path string, schema Schema[R]) (*Store[R], error)
- func RestoreStore[R any](r io.Reader, schema Schema[R]) (*Store[R], error)
- func RestoreStoreFromFile[R any](path string, schema Schema[R]) (*Store[R], error)
- func (s *Store[R]) Delete(uid string)
- func (s *Store[R]) Len() int
- func (s *Store[R]) Query(q Query) (Page[R], error)
- func (s *Store[R]) QueryAround(q Query, anchorKey string) (Page[R], AnchorOutcome, error)
- func (s *Store[R]) QueryAt(q Query, startRank int) (Page[R], error)
- func (s *Store[R]) ReopenInternedColumnsInPlace(path string) (func() error, error)
- func (s *Store[R]) ReplaceWhere(rows []R, owns func(R) bool)
- func (s *Store[R]) RestoreColumnsFromFileInto(path string) error
- func (s *Store[R]) RestoreFrom(r io.Reader) error
- func (s *Store[R]) RestoreFromFile(path string) error
- func (s *Store[R]) Scope(base map[string][]string, search string) (map[string]map[string]int, int)
- func (s *Store[R]) Snapshot() []R
- func (s *Store[R]) Spill(w io.Writer) error
- func (s *Store[R]) SpillColumns(path string) error
- func (s *Store[R]) SpillInternedColumns(path string) error
- func (s *Store[R]) SpillToFile(path string) error
- func (s *Store[R]) Upsert(row R)
Constants ¶
This section is empty.
Variables ¶
var ErrCursorMismatch = errors.New("querypage: cursor does not match the current query")
ErrCursorMismatch means a cursor was issued for a different query than the one now executing (different cluster, query shape, sort, direction, or page size). The caller must restart from the first page rather than risk silently mispaging.
Functions ¶
This section is empty.
Types ¶
type AnchorOutcome ¶ added in v1.10.3
type AnchorOutcome struct {
// Found: the anchor row exists AND matches the query's filters/search;
// Rank is then its 0-based position among matching rows in display order.
Found bool
// Filtered: the anchor row exists in the store but the query's
// filters/search exclude it. Mutually exclusive with Found.
Filtered bool
// Rank is valid only when Found; -1 otherwise.
Rank int
}
AnchorOutcome reports how QueryAround resolved its anchor row under the query's filters — the engine-level truth the serve layer maps to the contract's user-visible "filtered" / "not-found" reasons.
type Cursor ¶
type Cursor struct {
ClusterID string `json:"c"`
Signature string `json:"q"` // hash of the query shape (filters, scope, …)
Sort string `json:"s"`
Direction Direction `json:"d"`
Limit int `json:"l"`
// Position holds the single comparable sort value of the boundary row this
// cursor pins (the store's one-comparable-per-row invariant). UID is the final,
// always-unique tiebreak. Together they are the keyset seek position. Empty
// Position + empty UID = first page.
Position string `json:"p,omitempty"`
UID string `json:"u,omitempty"`
// Backward marks a prev-page cursor: the engine walks the sort order DOWNWARD from
// Position (collecting the rows immediately before it) instead of upward. It is a
// navigation property, not part of the query shape, so Validate ignores it.
Backward bool `json:"b,omitempty"`
}
Cursor is the value-keyed keyset position for one page of a query. It is opaque to clients (encoded as base64(json)) and self-describing: it pins the query it belongs to so a stale cursor from a different query is rejected, not mispaged.
func Decode ¶
Decode parses an opaque token back into a Cursor. An empty token decodes to the zero Cursor (first page).
func (Cursor) Encode ¶
Encode renders the cursor as an opaque base64(json) token. An empty token is returned for a first-page cursor so callers can treat "" as "start".
func (Cursor) IsFirstPage ¶
IsFirstPage reports whether the cursor is at the start position (no keyset yet).
type Direction ¶
type Direction string
Direction is the page traversal direction over the sort order.
type Page ¶
type Page[R any] struct { Rows []R NextCursor string PrevCursor string Facets map[string]map[string]int Total int // CursorInvalid reports that the request's cursor was rejected — undecodable, // pinned to a different query shape, or an un-navigable backward dead end (a // valid prev-page cursor whose predecessors were all deleted). The served page // is a first-page restart for the first two, and the empty dead-end page for // the last; callers forward the flag so the client resets its pagination // state. Owning this in the engine keeps every executor's degrade behavior // identical — callers need no cursor pre-validation of their own. CursorInvalid bool // PageStartRank is the 0-based rank of the served page's first row among the // query's matching rows — exact position honesty for the footer. Counted // serves (QueryAround/QueryAt) fill it; the cursor-based Query path returns // -1 (not computed — the O(rank) walk it would cost is gated behind the P9 // benchmark). The wire layer maps ≥0 to a pointer so rank 0 survives // omitempty. PageStartRank int // SelfCursor addresses THIS page: a later plain Query with it reproduces the // window (the page-stability primitive live refetches use after an anchored // or offset landing — cursor-addressed pages already hold their own request // token). Counted serves mint it from the entry preceding the window; "" for // a first-page window, whose address is the empty token. SelfCursor string }
Page is one page of results plus the (unfiltered) facet counts, the total number of rows matching filters+search, and the cursors for the next/prev page ("" when there is no page in that direction).
type Query ¶
type Query struct {
ClusterID string
Signature string // identifies the query shape (filters+scope); pins the cursor
Sort string // a key in Schema.SortKeys
Direction Direction
Limit int
Search string // case-insensitive substring filter over SearchText
Filters map[string][]string // facet filters: facet name -> allowed values (set membership; OR within, AND across facets)
MatchNone bool // explicit empty multiselect: no row can match
Cursor string // opaque continueToken from a previous Page
}
Query is one page request against a kind's store.
type Schema ¶
type Schema[R any] struct { UID func(R) string SortKeys map[string]func(R) string Facets map[string]func(R) string MultiFacets map[string]func(R) []string FacetNormalizers map[string]func(string) string SearchText func(R) string }
Schema describes how the engine reads a stored row of type R: how to identify it, which sort orders and facets it exposes, and what text is searchable. Sort and facet values are strings; numeric sorts must be encoded order-preservingly by the extractor (e.g. zero-padded), so lexical order matches numeric order.
UID is the store's unique row key AND the tie order for equal sort values (ascLess/descLess tie-break by it, ascending in both directions), so its shape is user-visible whenever sort values collide: give it a human-meaningful form — the typed tables use the lowercased kind/namespace/name adapter key, the catalog its identity chain — never an arbitrary identifier like the Kubernetes object UID. Pinned by TestTiedSortValuesOrderByHumanKey.
type Store ¶
type Store[R any] struct { // contains filtered or unexported fields }
Store is the generic Query → Page engine for one kind. It holds rows in an interned columnar store keyed by UID, a b-tree keyset index per sort order, and exact facet counters, all maintained incrementally on Upsert/Delete. It is generic over R via the Schema extractors — the engine carries no per-kind logic.
Row storage is a columnStore (see columnar.go): a structure-of-arrays with dictionary-interned string columns and a recycled rowId arena, replacing a plain map[string]R. To avoid an O(N)-reconstruction-per-query regression, the Schema's match inputs (facet values + lowercased SearchText) are extracted ONCE per row at Upsert time and cached by rowId in `matchByRowID`, so Query filters/searches and the filtered-Total scan read those cached values instead of reconstructing rows. Full rows are reconstructed via the codec only for the page actually returned, for Snapshot, and for the deindex of a replaced/deleted row.
func NewStore ¶
NewStore builds an empty store for a schema, creating a per-direction index per sort key and the columnar row store (codec built by reflecting over R's zero value).
func OpenInternedColumnStore ¶
OpenInternedColumnStore reopens a file written by SpillInternedColumns as a read-only Store whose scalar columns and dictionary strings alias the mapping (off-heap), rebuilding the uid→rowID map, sort indexes, and match cache from the aliased columns. Rows decoded from the store clone string fields before returning, so Page/Snapshot rows can outlive the mapping closer while the bulk columns stay off-heap. A bad magic / field-shape mismatch / truncation returns an error so the caller falls back to the heap path.
func RestoreColumnsFromFile ¶
RestoreColumnsFromFile reads a columnar file written by SpillColumns and rebuilds a NEW equivalent Store (reconstruct each row + Upsert — rebuilding columns, dicts, indexes, match cache). A bad magic, a field-shape mismatch against the schema's codec, or a truncated file returns an error so the caller can fall back to the gob path.
func RestoreStore ¶
RestoreStore gob-decodes a []R written by Spill, builds a fresh Store for the given schema, and Upserts every row. Each Upsert rebuilds that row's index entries, facet counts, and match-cache entry incrementally, so the returned store is fully equivalent to the one that was spilled.
func RestoreStoreFromFile ¶
RestoreStoreFromFile opens path and rebuilds a Store from the spilled rows with a buffered reader. This is what the governor's re-warm path calls.
func (*Store[R]) Delete ¶
Delete removes a row by UID, maintaining every index + facet incrementally. A read-only (mmap-aliased) store ignores deletes — see Store.readOnly.
func (*Store[R]) Query ¶
Query executes a page request: it seeks to the cursor position in the chosen sort index (O(log N)), walks in the requested direction collecting rows that pass the filters + search until the page is full, and returns the next cursor plus the facet counts and total match count.
func (*Store[R]) QueryAround ¶ added in v1.10.3
QueryAround serves the PAGE-ALIGNED window containing the row whose schema UID equals anchorKey, under q's sort, direction, filters, and search — the engine primitive behind "jump to this object in the list". One counted O(rank + limit) walk (plus skipped non-matching entries) yields the exact 0-based rank, the aligned window (pageStart = rank - rank%limit), and ordinary keyset prev/next cursors minted from the window boundaries — so pagination after a jump is indistinguishable from pagination that arrived from page 1, and rank is derived per request, never stored. A missing or filtered-out anchor serves the FIRST page instead (one round trip, sane landing) with the outcome saying why. q.Cursor is ignored: anchor and continue are mutually exclusive at the contract layer.
func (*Store[R]) QueryAt ¶ added in v1.10.3
QueryAt serves the page starting at startRank (0-based) among matching rows — the bounded offset contract behind numbered page jumps. startRank is clamped to the last page-aligned start (negatives to 0) so a stale page number lands on the nearest real page instead of an empty one; the served start is reported on Page.PageStartRank. q.Cursor is ignored.
func (*Store[R]) ReopenInternedColumnsInPlace ¶
ReopenInternedColumnsInPlace spills this store's interned columns to path, then swaps the store's internals (columns, indexes, match cache) to a read-only, mmap-aliased view of that file — so the SAME *Store pointer keeps serving queries (callers holding it via a maintained store need no rewiring) but with its column data off-heap. Returned Page/Snapshot rows clone decoded strings, so the closer can run after the store is discarded or re-warmed without invalidating already-returned rows. On any error the store is left unchanged (safe-degrade: the caller keeps serving from heap). This is the Tier 2.6 Cold-serving transition at the store level.
func (*Store[R]) ReplaceWhere ¶
ReplaceWhere replaces the rows owned by one source with rows from that source, preserving rows owned by other sources. Passing owns=nil replaces the whole store. It is the batch counterpart to Upsert/Delete for reflector relists: rebuild derived indexes/facets once from the post-relist row set instead of issuing N incremental updates through downstream sinks.
func (*Store[R]) RestoreColumnsFromFileInto ¶
RestoreColumnsFromFileInto reads a columnar file written by SpillColumns and Upserts its rows INTO this store (using the store's own schema/codec), for restoring into a freshly-built maintained store in place. Same validation + error contract as RestoreColumnsFromFile; the store is left partially populated only if a per-row Upsert follows a successful parse (parse errors happen before any Upsert).
func (*Store[R]) RestoreFrom ¶
RestoreFrom gob-decodes a []R written by Spill and Upserts every row INTO THIS store, rebuilding each row's index/facet/match-cache entry incrementally — the same per-row path Upsert always takes. Unlike RestoreStore it loads into an existing store, preserving its schema and any wiring that already references it (a maintained store's informer handlers / ingest sink), which is what the governor's warm-restore on re-warm needs: the store is built+wired first, then pre-painted from disk before the fresh informers feed. Rows already present are overwritten by UID; rows absent from the spill are left untouched, so a later reconciling sync removes any that no longer exist upstream.
func (*Store[R]) RestoreFromFile ¶
RestoreFromFile opens path and loads its spilled rows into this store with a buffered reader. It is the file counterpart of RestoreFrom — what the governor's re-warm path calls to pre-paint a freshly-built maintained store from its on-disk spill.
func (*Store[R]) Scope ¶
Scope returns, over the rows matching `base` filters + `search`, the per-facet value→count map and the total — computed from the by-rowId match cache WITHOUT reconstructing any row. Same filter/search semantics as Query (OR within a facet, AND across facets; case-insensitive substring search). The facet values are the raw values the schema's facet extractor produced (e.g. lowercased), matching the stored match cache. This is the cheap O(N)-no-reconstruction count a direct serve uses for facets + UnfilteredTotal.
func (*Store[R]) Snapshot ¶
func (s *Store[R]) Snapshot() []R
Snapshot returns a copy of every stored row (unordered) under a read lock. It lets a caller hand the store's current rows to another consumer (e.g. a scope-filtered re-query) without exposing the internal store.
func (*Store[R]) Spill ¶
Spill gob-encodes the store's rows ([]R from Snapshot) to w. Snapshot takes the store's read lock and returns an independent copy, so the encode runs without holding the lock (avoiding a recursive RLock) yet still sees a consistent set of rows. The derived indexes/facets/match-cache are intentionally not written; Restore rebuilds them from the rows.
func (*Store[R]) SpillColumns ¶
SpillColumns writes the store's rows to path in the columnar format.
func (*Store[R]) SpillInternedColumns ¶
SpillInternedColumns writes the store's interned columns to path in the mmap-aliasable format.
func (*Store[R]) SpillToFile ¶
SpillToFile creates/truncates path and writes the store's spilled rows to it with a buffered writer. This is what the governor's Cold action calls.