storage

package
v0.16.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: Apache-2.0 Imports: 35 Imported by: 0

Documentation

Index

Constants

View Source
const (
	OpAdd    uint8 = 0 // Sorts first - wins ties
	OpRemove uint8 = 1
)

Op represents the operation type for set entries. Constants are ordered so Add < Remove. With key order [Value][Op][Tx↓], this means:

  • For same Value: all Adds sort before all Removes
  • At same (Value, Tx): Add sorts first (add-wins semantics)
View Source
const BenchmarkDatabasePath = "datalog/storage/testdata/ohlc_benchmark.db"

BenchmarkDatabasePath names the pre-built OHLC benchmark database, from the module root. Where it lives is fixed by its readers — the benchmarks in this package open it — not chosen by whoever builds it.

Every test-data path is named from the module root and anchored by resolveTestDataPath, because the builder and the readers run from different directories: cmd/build-testdb from the module root, the package's tests from the package. A path relative to the current directory names two different files to those two callers, so a builder can fill one location while every reader — and the Makefile's own guard — looks at another.

View Source
const ElementIDSize = datalog.ElementIDSize

ElementIDSize is the size in bytes of an encoded ElementID

Variables

View Source
var (
	// ErrSnapshotExists is returned by Snapshot when the name is already taken.
	ErrSnapshotExists = errors.New("snapshot already exists")
	// ErrSnapshotNotFound is returned when a named snapshot does not exist.
	ErrSnapshotNotFound = errors.New("snapshot not found")
	// ErrRollbackInProgress is returned by a transaction begun while a TruncateTo rollback
	// is running on the same handle: by its writes (Add/Retract/Commit) and by every read
	// of its view. The transaction is dropped, not queued; retry on a fresh transaction
	// once the rollback completes.
	ErrRollbackInProgress = errors.New("write rejected: a rollback (TruncateTo) is in progress")
)
View Source
var DiscardIntake *scanReport

DiscardIntake is the report for a read no pattern asked for — schema reconstruction at open, the write path, export, cache warming. Passing it says nothing accounts for this scan.

View Source
var ErrBranchedAfterSnapshot = errors.New("a branch was forked after the snapshot")

ErrBranchedAfterSnapshot is returned by TruncateTo when the database forked a branch from a snapshot after the one it is asked to truncate to.

View Source
var ErrStoreLocked = errors.New("store directory is locked by another process")

ErrStoreLocked reports that a persistent backend's directory is held by another process. Badger reports the state by message and flattens the EWOULDBLOCK beneath it, so the message is matched where the backend is opened, in the one package that pins the Badger version. Declared here, outside any build tag, so the name exists on every target.

HEAD is the zero value representing the HEAD sentinel for RGA vectors. All vector elements reference HEAD or another element's ID as their "after" reference.

View Source
var Indices = []IndexType{EAVT, EATV, AEVT, AETV, ATEV, AVET, VAET, TAEV}

Indices lists all indices used for queries

Functions

func DatomFromKey

func DatomFromKey(index IndexType, key []byte, encoder *BinaryKeyEncoder, blobs BlobReader) (datalog.Datom, error)

DatomFromKey reconstructs a datom from an index key. Pass nil when Tier-3 values are not expected.

func DefaultPlannerOptions

func DefaultPlannerOptions() planner.PlannerOptions

DefaultPlannerOptions returns the default planner and executor options for the database

func ElementIDNode added in v0.7.1

func ElementIDNode(eid datalog.ElementID) edn.Node

ElementIDNode builds an EDN [lamport replica] vector node.

func EncodeElementIDForKey added in v0.6.0

func EncodeElementIDForKey(a ElementID) []byte

EncodeElementIDForKey writes 16 bytes for storage keys. Uses bitwise NOT so highest ElementID sorts first in forward scans. This enables O(1) "current value" lookups: first entry = highest Tx = current.

Example:

EncodeElementIDForKey(ElementID{Lamport: 100, ReplicaID: 5})
→ bytes where 100 becomes ^100 (0xFFFFFFFFFFFFFF9B)
→ higher Lamports produce smaller byte values, sorting first

func EncodeElementIDInto added in v0.6.0

func EncodeElementIDInto(a ElementID, buf []byte, offset int)

EncodeElementIDInto writes 16 bytes into the provided buffer starting at offset. The buffer must have at least offset+16 bytes available. Uses bitwise NOT for descending sort order (same as EncodeElementIDForKey).

func EncodeRGAElement added in v0.6.0

func EncodeRGAElement(elem RGAElement) []byte

EncodeRGAElement serializes an RGAElement for storage. The ID is NOT included - it's stored as the datom's Tx field.

func EncodeSetEntry added in v0.6.0

func EncodeSetEntry(entry SetEntry) []byte

EncodeSetEntry encodes a set entry for storage. Format: [type:1][Value:var][Op:1]

CRITICAL: Value comes BEFORE Op in encoding. This ensures that entries for the same logical value sort together in EAVT index, enabling efficient add-wins resolution by grouping all ops for the same value.

Example EAVT scan for [E][:tags]:

[E][:tags][str]["veteran"][Op:Add][Tx_high]    ← Same value grouped
[E][:tags][str]["veteran"][Op:Remove][Tx_low]  ← Same value grouped
[E][:tags][str]["warrior"][Op:Add][Tx_high]    ← Different value

func FindMaxElementID added in v0.6.0

func FindMaxElementID(elements []RGAElement) datalog.ElementID

FindMaxElementID returns the highest ElementID among the given elements. Used for cache version tracking.

func FormatCRDTOpEDN added in v0.7.1

func FormatCRDTOpEDN(op datalog.CRDTOp) string

FormatCRDTOpEDN formats a CRDTOp as an EDN keyword string.

func FormatDatomEDN added in v0.5.6

func FormatDatomEDN(d *datalog.Datom) string

FormatDatomEDN formats a single datom as an EDN vector string. The format is: [#identity "L85" :attribute value tx-id op [after-ref]]

func FormatDatomEDNCompressed added in v0.11.0

func FormatDatomEDNCompressed(d *datalog.Datom) string

FormatDatomEDNCompressed formats a datom as EDN, using #lzj for compressible values.

func FormatIdentityEDN added in v0.5.6

func FormatIdentityEDN(id datalog.Identity) string

FormatIdentityEDN formats an entity Identity as EDN string.

func FormatValueEDN added in v0.5.6

func FormatValueEDN(v interface{}) string

FormatValueEDN formats a value as EDN string.

func IdentityNode added in v0.7.1

func IdentityNode(id datalog.Identity) edn.Node

IdentityNode builds an EDN node for an entity Identity.

func IsDumpPath added in v0.15.2

func IsDumpPath(path string) bool

IsDumpPath reports whether path names an exported dump rather than a store directory. The extension answers only that question: which of the two dump formats it holds is read from the dump itself, so a name and its contents cannot disagree.

func ParseCRDTOpNode added in v0.7.1

func ParseCRDTOpNode(node *edn.Node) (datalog.CRDTOp, error)

ParseCRDTOpNode parses a CRDTOp from an EDN keyword node.

func ParseDatomEDN added in v0.5.6

func ParseDatomEDN(s string) (*datalog.Datom, error)

ParseDatomEDN parses a single EDN vector into a Datom. Expected format: [#identity "L85" :attribute value tx-id]

func ParseIdentityNode added in v0.5.6

func ParseIdentityNode(node *edn.Node) (datalog.Identity, error)

ParseIdentityNode parses an identity from an EDN node. Expects #identity "L85string" format.

func ParseValueNode added in v0.5.6

func ParseValueNode(node *edn.Node) (interface{}, error)

ParseValueNode parses a value from an EDN node.

func ReconstructRGA added in v0.6.0

func ReconstructRGA(elements []RGAElement) []any

ReconstructRGA builds an ordered list from RGA elements.

The algorithm:

  1. Build a map from AfterRef -> []elements (ALL elements, including tombstoned)
  2. Sort each child list by ElementID for deterministic order
  3. DFS from HEAD, emitting values only for non-tombstoned elements

Tombstoned elements are kept in the tree structure so their children remain reachable. For example, if elem2 is tombstoned but elem3 has AfterRef=elem2.ID, elem3 must still appear in the output.

Time complexity: O(n log n) where n = number of elements Space complexity: O(n) for the children map and result slice

CONCURRENT WRITE BEHAVIOR:

When two replicas concurrently insert elements at the same position (same AfterRef), BOTH elements are preserved. The order is determined by ElementID comparison: lower (Lamport, ReplicaID) comes first.

Example: Two replicas append to an empty vector concurrently:

Replica A: Add("stealth") → AfterRef=HEAD, ID=(L=5, R=100)
Replica B: Add("magic")   → AfterRef=HEAD, ID=(L=5, R=200)

After merge, children[HEAD] = [{stealth, (5,100)}, {magic, (5,200)}] Sorted by ElementID: stealth first (100 < 200) Result: ["stealth", "magic"]

This is NOT last-writer-wins. All concurrent writes are preserved and deterministically ordered. The order is based on ElementID, not wall-clock time.

For same-position updates via Set(), both new values appear adjacent in the result because both have the same AfterRef (the element before the changed position).

func ResolveAddWinsFromDatoms added in v0.6.0

func ResolveAddWinsFromDatoms(datoms []datalog.Datom) (map[string]any, map[string]datalog.ElementID, datalog.ElementID)

ResolveAddWinsFromDatoms resolves cardinality-many using add-wins CRDT semantics. Input: datoms for a single (E, A) pair, any order. Datoms include Op field. Output: set of current members, each member's newest assertion's Tx under the member's key (the datom the member resolves to, which a pattern's transaction position binds), max ElementID seen.

Add-wins semantics: - For each unique value, track highest add tx and highest remove tx - If add has higher Lamport: in set - If remove has higher Lamport: not in set - Same Lamport (concurrent): add wins

func ResolveLWWFromDatoms added in v0.6.0

func ResolveLWWFromDatoms(datoms []datalog.Datom) (any, datalog.ElementID)

ResolveLWWFromDatoms resolves cardinality-one using Last-Writer-Wins semantics. Input: datoms for a single (E, A) pair, any order. Output: current value (highest ElementID wins), max ElementID seen.

Returns (nil, zero ElementID) if no datoms provided. Returns (nil, maxID) if the highest-Tx datom is a Remove — the attribute has been tombstoned and does not currently exist. Mirrors the check in PatternMatcher.ResolveLWW (cache_resolver.go). Callers use maxID for cache freshness tracking regardless of whether V is nil.

func ResolveRGAFromDatoms added in v0.6.0

func ResolveRGAFromDatoms(datoms []datalog.Datom) ([]any, []datalog.ElementID, datalog.ElementID)

ResolveRGAFromDatoms resolves cardinality-vector using RGA reconstruction. Input: datoms for a single (E, A) pair, any order. Datoms must include Op and AfterRef. Output: ordered values, position index (element IDs), max ElementID seen.

RGA semantics: - OpRGAInsert: element inserted after AfterRef position - OpRGATombstone: element deleted (AfterRef is the deleted element's ID) - Final order determined by tree reconstruction

func TestDatabaseStats

func TestDatabaseStats(db *Database) error

TestDatabaseStats prints statistics about the test database

func ValueNode added in v0.7.1

func ValueNode(v interface{}) edn.Node

ValueNode builds an EDN node for a value.

func ValueNodeCompressed added in v0.11.0

func ValueNodeCompressed(v interface{}) edn.Node

ValueNodeCompressed builds an EDN node for a value, using #lzj for large string values that compress well. []byte values use #bytes (uncompressed) to avoid type ambiguity on import. Small strings use plain EDN.

Types

type AnalyzeResult added in v0.2.3

type AnalyzeResult struct {
	Plan      *planner.RealizedPlan // The query plan
	Result    executor.Relation     // Query result (not materialized)
	Events    []annotations.Event   // All annotation events from execution
	TotalTime time.Duration         // Total execution time
}

AnalyzeResult contains the query plan and execution statistics from Analyze().

func (*AnalyzeResult) String added in v0.2.3

func (ar *AnalyzeResult) String() string

String returns a formatted analysis report showing the query plan and execution statistics.

type Attribute

type Attribute [32]byte

Attribute represents an attribute name, stored directly as the keyword's UTF-8 bytes. Names longer than the array are rejected at write and schema time (see datalog.MaxAttributeBytes); they are never truncated.

func (Attribute) String

func (a Attribute) String() string

String returns the attribute as a string (if stored directly)

type AttributeSchema added in v0.16.2

type AttributeSchema[T any] map[datalog.Keyword]func(T) any

AttributeSchema maps keywords to accessor functions for NewSliceSource. Each entry defines how to extract a value from a slice item for a given attribute.

type Backend added in v0.15.2

type Backend struct {
	Name string

	// Persistent reports whether the backend keeps its data at Path. A caller
	// that reopens a path expecting its data back needs this; the in-process
	// stores ignore Path entirely.
	Persistent bool

	Open func(path string, encoder *BinaryKeyEncoder) (Store, error)
}

Backend is a store this build can open. Which ones exist is a property of the build: Badger needs a filesystem and does not compile under js/wasm, so a caller choosing a store — a command-line flag, a JS host, a diagnostic — asks AvailableBackends rather than re-deriving it from build tags.

func AvailableBackends added in v0.15.2

func AvailableBackends() []Backend

AvailableBackends lists every store this build can open. The order is in-process first, then persistent; DefaultBackend names the default rather than the position doing it.

func BackendNamed added in v0.15.2

func BackendNamed(name string) (Backend, error)

BackendNamed resolves a backend by name. The error lists what this build has, because the answer differs between a native binary and a wasm one.

func DefaultBackend added in v0.15.2

func DefaultBackend() Backend

DefaultBackend is what a Database opened by path uses. Native builds persist to Badger; wasm has no filesystem and holds the data in memory.

type BadgerEmulatorStore added in v0.16.0

type BadgerEmulatorStore struct {
	// contains filtered or unexported fields
}

BadgerEmulatorStore is the Badger emulator: an ordered, transactional implementation of Store that holds, in memory, the same binary keys BadgerStore writes to disk, including all eight indices, metadata, and content-addressed blobs. It exists so the backend contract runs against Badger's key layout in a process that has no disk. The in-process backend the engine opens by default is MemoryTreeStore, which holds typed datoms.

Commits apply Assert/Retract in call order against the live map under the write lock, journaling prior values so a failed apply can restore them. This matches BadgerTx immediacy for interleaving and avoids cloning the whole store on every commit (required for Import and other batched writes).

keys is a B-tree of entries map keys, kept in lockstep with puts and deletes so retract and scan can Seek/prefix-iterate in O(log N + range) instead of scanning the whole map (Badger's Seek/ValidForPrefix analogue). Insert and delete are O(log N) per key, so Import/batched Assert stay O(M log N) in the key index.

func NewBadgerEmulatorStore added in v0.16.0

func NewBadgerEmulatorStore(encoder *BinaryKeyEncoder) *BadgerEmulatorStore

func (*BadgerEmulatorStore) Assert added in v0.16.0

func (s *BadgerEmulatorStore) Assert(datoms []datalog.Datom) error

func (*BadgerEmulatorStore) AssertEach added in v0.16.0

func (s *BadgerEmulatorStore) AssertEach(produce func(add func(*datalog.Datom) error) error) error

AssertEach gathers what produce yields and asserts it. Gathering is the whole of it here: entries are a map keyed by encoded key, so there is no incremental structure to build into and a per-datom path would save nothing.

func (*BadgerEmulatorStore) BeginTx added in v0.16.0

func (s *BadgerEmulatorStore) BeginTx() (StoreTx, error)

func (*BadgerEmulatorStore) Close added in v0.16.0

func (s *BadgerEmulatorStore) Close() error

func (*BadgerEmulatorStore) DatomsAfter added in v0.16.0

func (s *BadgerEmulatorStore) DatomsAfter(eid datalog.ElementID) (after []datalog.Datom, afterErr error)

func (*BadgerEmulatorStore) DeleteDatoms added in v0.16.0

func (s *BadgerEmulatorStore) DeleteDatoms(datoms []datalog.Datom) (int, error)

DeleteDatoms removes each datom's keys from every index, and then any blob those datoms were the last to refer to.

func (*BadgerEmulatorStore) Encoder added in v0.16.0

func (s *BadgerEmulatorStore) Encoder() *BinaryKeyEncoder

func (*BadgerEmulatorStore) FinishBatch added in v0.16.0

func (s *BadgerEmulatorStore) FinishBatch() error

FinishBatch has nothing to complete: AssertEach writes its datoms before returning.

func (*BadgerEmulatorStore) GetMetadataUint64 added in v0.16.0

func (s *BadgerEmulatorStore) GetMetadataUint64(key string) (uint64, bool, error)

func (*BadgerEmulatorStore) MaxElementID added in v0.16.0

func (s *BadgerEmulatorStore) MaxElementID() (datalog.ElementID, error)

func (*BadgerEmulatorStore) MaxTxForEntity added in v0.16.0

func (s *BadgerEmulatorStore) MaxTxForEntity(e datalog.Identity) (datalog.ElementID, bool, error)

func (*BadgerEmulatorStore) NewReadSession added in v0.16.0

func (s *BadgerEmulatorStore) NewReadSession() (ReadSession, error)

NewReadSession opens a consistent read view at the store's current state. Clone mutates the source tree's copy-on-write context, so it runs under the store's write lock, exclusive with committers and other sessions.

func (*BadgerEmulatorStore) Retract added in v0.16.0

func (s *BadgerEmulatorStore) Retract(datoms []datalog.Datom) error

func (*BadgerEmulatorStore) Scan added in v0.16.0

func (s *BadgerEmulatorStore) Scan(bound ScanBound) (Iterator, error)

func (*BadgerEmulatorStore) ScanKeysOnly added in v0.16.0

func (s *BadgerEmulatorStore) ScanKeysOnly(bound ScanBound) (Iterator, error)

func (*BadgerEmulatorStore) SetMetadataUint64 added in v0.16.0

func (s *BadgerEmulatorStore) SetMetadataUint64(key string, value uint64) error

type BadgerIterator

type BadgerIterator struct {
	// contains filtered or unexported fields
}

BadgerIterator implements Iterator for BadgerDB

func (*BadgerIterator) Close

func (i *BadgerIterator) Close() error

Close closes the iterator and releases the underlying BadgerDB transaction. Session-owned iterators return themselves to the session instead — the shared transaction is discarded by the session, not by any one scan. Safe to call multiple times.

func (*BadgerIterator) Datom

func (i *BadgerIterator) Datom() (*datalog.Datom, error)

Datom returns the current datom decoded from the key Values are not stored - all datom information is in the key

func (*BadgerIterator) ElementID added in v0.6.0

func (i *BadgerIterator) ElementID() datalog.ElementID

ElementID extracts the transaction ElementID from the current key. This is more efficient than Datom() when only the ElementID is needed.

func (*BadgerIterator) Error added in v0.11.2

func (i *BadgerIterator) Error() error

Error returns nil — BadgerIterator.Next() does not perform any fallible operations; errors surface exclusively through Datom() on the current item. Returning nil satisfies the Iterator interface and tells wrapping iterators there is no deferred error to collect.

func (*BadgerIterator) Key added in v0.15.0

func (i *BadgerIterator) Key() []byte

Key returns the current index key without decoding the datom or resolving blobs. Valid after Next returns true until the next Next, Seek, or Close.

func (*BadgerIterator) Next

func (i *BadgerIterator) Next() bool

Next advances the iterator

func (*BadgerIterator) Scanned added in v0.15.2

func (i *BadgerIterator) Scanned() int

Scanned reports keys taken from the index inside this iterator's range.

func (*BadgerIterator) Seek

func (i *BadgerIterator) Seek(key []byte)

Seek positions the iterator at or after the given key

type BadgerStore

type BadgerStore struct {
	// contains filtered or unexported fields
}

BadgerStore implements Store using BadgerDB

func NewBadgerStore

func NewBadgerStore(path string, encoder *BinaryKeyEncoder) (*BadgerStore, error)

NewBadgerStore creates a new BadgerDB-backed store.

func (*BadgerStore) Assert

func (s *BadgerStore) Assert(datoms []datalog.Datom) error

Assert adds datoms to the store.

func (*BadgerStore) AssertEach added in v0.15.2

func (s *BadgerStore) AssertEach(produce func(add func(*datalog.Datom) error) error) error

AssertEach writes through a WriteBatch rather than one transaction. A producer yields arbitrarily many datoms, and at eight index keys each plus a blob per out-of-line value, Badger's per-transaction ceiling arrives long before a caller has reason to suspect a limit exists — a single entity's history is enough. The batch splits at that ceiling itself, so the arithmetic stays Badger's and cannot drift from it.

The cost is that this is not atomic: a mid-way failure leaves the datoms already committed in place. Re-asserting is safe, because an index key is derived wholly from its datom and a repeated write reproduces it exactly. A caller that needs a boundary uses BeginTx, whose StoreTx.Assert still writes into the one transaction that caller owns.

func (*BadgerStore) BeginTx

func (s *BadgerStore) BeginTx() (StoreTx, error)

BeginTx starts a new transaction

func (*BadgerStore) Close

func (s *BadgerStore) Close() error

Close closes the store

func (*BadgerStore) CountKeys

func (s *BadgerStore) CountKeys(index IndexType, start, end []byte) (int64, error)

CountKeys counts keys in a range without fetching values (fast counting)

func (*BadgerStore) DatomsAfter added in v0.13.3

func (s *BadgerStore) DatomsAfter(eid datalog.ElementID) ([]datalog.Datom, error)

DatomsAfter returns every datom whose Tx is strictly greater than eid. It scans TAEV, whose Tx-descending order puts the affected datoms first, and stops at the first datom with Tx <= eid. Read-only; the caller removes them with DeleteDatoms, doing whatever cache/clock coordination it needs in between.

func (*BadgerStore) DeleteDatoms added in v0.13.3

func (s *BadgerStore) DeleteDatoms(datoms []datalog.Datom) (int, error)

DeleteDatoms physically removes each given datom from all eight indices, returning the count handed over. Unlike Retract (which appends a tombstone at a higher Tx), this is a true rewind: the keys are gone, so the datoms vanish from History() too.

The deletes go through a WriteBatch for the same reason AssertEach's writes do, and it is the delete side that needs it more: a rewind's extent is the whole post-snapshot tail, and at eight index keys per datom the pending-write set is eight times a count the caller has no reason to think is bounded. Badger's per-transaction ceiling arrives around 26,000 datoms. The batch splits at that ceiling itself, so the arithmetic stays Badger's and cannot drift from it.

The cost is that this is not atomic: a mid-way failure leaves the datoms already deleted gone, and History() shows a torn tail until the rewind is re-run. Re-running completes it — TruncateTo restores the clock only after every delete succeeds, so a second pass collects the datoms that survived, and sweepUnreferencedBlobs below reaches the blobs of those that did not, which no pass could name from the datoms it can still see. A reader whose MVCC snapshot lands between chunks can observe a partially deleted tail for the duration.

When the call returns, no blob without a referring datom remains — including blobs this call never touched. The tier is this store's own answer to values too wide for a key, so nothing above the Store seam hears about it.

func (*BadgerStore) Encoder added in v0.15.0

func (s *BadgerStore) Encoder() *BinaryKeyEncoder

func (*BadgerStore) FinishBatch added in v0.15.2

func (s *BadgerStore) FinishBatch() error

FinishBatch has nothing to complete: AssertEach flushes its write batch before returning.

func (*BadgerStore) GetMetadataUint64 added in v0.6.0

func (s *BadgerStore) GetMetadataUint64(key string) (uint64, bool, error)

GetMetadataUint64 retrieves a uint64 metadata value by key. Returns (value, true) if found, (0, false) if not found.

func (*BadgerStore) MaxElementID added in v0.6.0

func (s *BadgerStore) MaxElementID() (datalog.ElementID, error)

MaxElementID returns the highest ElementID in the store by scanning TAEV index. With bitwise NOT encoding for Tx, forward scan gives highest Tx first (O(1)). Returns zero ElementID if store is empty.

func (*BadgerStore) MaxTxForEntity added in v0.13.3

func (s *BadgerStore) MaxTxForEntity(e datalog.Identity) (datalog.ElementID, bool, error)

MaxTxForEntity returns the highest Tx among a single entity's datoms. The bool is false when the entity has no datoms. TruncateTo uses it to find the floor below which a snapshot marker must be preserved.

func (*BadgerStore) NewReadSession added in v0.15.1

func (s *BadgerStore) NewReadSession() (ReadSession, error)

NewReadSession opens a consistent read view at the store's current state.

func (*BadgerStore) Retract

func (s *BadgerStore) Retract(datoms []datalog.Datom) error

Retract removes datoms from the store

func (*BadgerStore) Scan

func (s *BadgerStore) Scan(bound ScanBound) (Iterator, error)

Scan returns a workspace-decoded iterator for a range of keys. Scan and ScanKeysOnly share the same KeyOnlyIterator contract: Datom() returns the iterator's current workspace until Next, Seek, or Close.

func (*BadgerStore) ScanKeysOnly

func (s *BadgerStore) ScanKeysOnly(bound ScanBound) (Iterator, error)

ScanKeysOnly returns an iterator that decodes datoms from keys without fetching values This is much faster than regular scanning as it avoids the redundant value fetch

func (*BadgerStore) SetMetadataUint64 added in v0.6.0

func (s *BadgerStore) SetMetadataUint64(key string, value uint64) error

SetMetadataUint64 stores a uint64 metadata value by key.

type BadgerTx

type BadgerTx struct {
	// contains filtered or unexported fields
}

BadgerTx implements Tx for BadgerDB

func (*BadgerTx) Assert

func (t *BadgerTx) Assert(datoms []datalog.Datom) error

Assert adds datoms within a transaction. Unlike BadgerStore.Assert this does not split: the caller owns the transaction boundary, so an oversized commit surfaces Badger's ErrTxnTooBig for the caller to handle.

func (*BadgerTx) Commit

func (t *BadgerTx) Commit() error

Commit reclaims any blob this transaction's retracts left unreferenced, then commits. Reclaiming here rather than inside Retract is what lets the count see asserts the same transaction made afterwards.

func (*BadgerTx) Retract

func (t *BadgerTx) Retract(datoms []datalog.Datom) error

Retract removes datoms within a transaction

func (*BadgerTx) Rollback

func (t *BadgerTx) Rollback() error

Rollback rolls back the transaction

type BinaryExportOptions added in v0.15.0

type BinaryExportOptions struct {
	// SoftBudget is the advisory uncompressed chunk size in bytes. When the
	// open chunk reaches this size, export closes at the next entity boundary.
	// Zero selects the default (256KiB).
	SoftBudget int
	// SkipEntity, when non-nil, omits datoms whose entity returns true.
	SkipEntity func(datalog.Identity) bool
}

BinaryExportOptions configures JDZL binary export.

type BinaryImportOptions added in v0.15.0

type BinaryImportOptions struct {
	// Workers is the maximum number of chunks decoded and asserted concurrently.
	// Zero selects one: chunks then arrive in file order — the EAVT order the
	// tree store's bulk build takes as given — and the sweep in
	// BenchmarkDumpImportWorkers measured no throughput return on more.
	Workers int

	// Finalize runs once after the last chunk is written and the store's own
	// batch is complete, and not at all if any chunk failed. It is a hook for a
	// caller with its own work to close out; the store's batch is finished
	// whether or not one is given.
	Finalize func() error
}

BinaryImportOptions configures JDZL binary import.

type BinaryKeyEncoder

type BinaryKeyEncoder struct {
	CompressionThreshold int // 0 = disabled; values >= this size are compressed
}

BinaryKeyEncoder builds and parses the binary keys used by every physical index.

func (*BinaryKeyEncoder) DecodeKey

func (e *BinaryKeyEncoder) DecodeKey(index IndexType, key []byte) (entity [20]byte, attr [32]byte, value []byte, tx [16]byte, op byte, afterRef [16]byte, err error)

DecodeKey extracts components from a binary index key. Returns fixed-size arrays for entity, attr, tx, and op to avoid heap escape. tx is 16 bytes: Lamport (8) + ReplicaID (8) = ElementID Tx is stored with bitwise NOT for descending sort, reversed on decode. Op is 1 byte: 0-4 (see CRDTOp constants). Always the LAST byte of every key. AfterRef is optionally present for Op ∈ {OpRGAInsert(3), OpRGATombstone(4)}.

Key formats (Op always last — see docs/reference/OP_POSITION_PROOF.md):

EAVT: [prefix][E][A][type][value][Tx↓][AfterRef?][Op]
EATV: [prefix][E][A][Tx↓][type][value][AfterRef?][Op]
AEVT: [prefix][A][E][type][value][Tx↓][AfterRef?][Op]
AETV: [prefix][A][E][Tx↓][type][value][AfterRef?][Op]
ATEV: [prefix][A][Tx↓][E][type][value][AfterRef?][Op]
AVET: [prefix][A][type][value][E][Tx↓][AfterRef?][Op]
VAET: [prefix][type][value][A][E][Tx↓][AfterRef?][Op]
TAEV: [prefix][Tx↓][A][E][type][value][AfterRef?][Op]

Decoding strategy: Op = key[len-1]. If Op.HasAfterRef(), AfterRef = key[len-17:len-1]. No heuristic needed. No ambiguity.

func (*BinaryKeyEncoder) EncodeKey

func (e *BinaryKeyEncoder) EncodeKey(index IndexType, d *datalog.Datom) []byte

func (*BinaryKeyEncoder) EncodeKeyWithValueBytes added in v0.11.0

func (e *BinaryKeyEncoder) EncodeKeyWithValueBytes(index IndexType, d *datalog.Datom, vBytes []byte) []byte

EncodeKeyWithValueBytes builds an index key using pre-encoded value bytes. Use this when encoding the same datom into multiple indexes to avoid recomputing compression 7 times.

func (*BinaryKeyEncoder) EncodePrefix

func (e *BinaryKeyEncoder) EncodePrefix(index IndexType, parts ...[]byte) []byte

EncodePrefix creates a binary prefix key for range scans

func (*BinaryKeyEncoder) EncodePrefixRange

func (e *BinaryKeyEncoder) EncodePrefixRange(index IndexType, parts ...[]byte) (start, end []byte)

EncodePrefixRange creates start and end keys for a prefix scan

func (*BinaryKeyEncoder) EncodeScanBound added in v0.15.2

func (e *BinaryKeyEncoder) EncodeScanBound(b ScanBound) (EncodedRun, error)

EncodeScanBound renders a typed ScanBound as the run a binary-key scan walks: the byte range, and the membership test that narrows a prefix range to the value the bound names. This is the Badger-side projection of the bound; a backend that compares typed components directly never calls it.

func (*BinaryKeyEncoder) EncodeTxForPrefix added in v0.6.0

func (e *BinaryKeyEncoder) EncodeTxForPrefix(tx Tx) []byte

EncodeTxForPrefix encodes a Tx with bitwise NOT — the form Tx takes inside a key. Higher Tx values encode to lower byte values, so an index that orders Tx yields newest first. encodeBoundComponent renders a bound's Tx component through here, so a bound and the keys it addresses agree on the encoding.

A ScanBound binds Tx to a value; it has no second endpoint. A caller wanting a Tx *range* seeks to one end and walks with its own stop condition, as pull_batch.go does.

func (*BinaryKeyEncoder) EncodeValueBytes added in v0.11.0

func (e *BinaryKeyEncoder) EncodeValueBytes(v interface{}) (vBytes []byte, blobData *datalog.BlobData)

EncodeKey creates a binary index key from a datom Tx is encoded with bitwise NOT for descending sort order (highest Tx first). Op is always the LAST byte of every key. This enables deterministic decoding: op = key[len(key)-1]. If Op.HasAfterRef(), AfterRef is the 16 bytes before Op.

Key formats (Op always last — see docs/reference/OP_POSITION_PROOF.md):

EAVT: [prefix][E][A][type][value][Tx↓][AfterRef?][Op]  - groups by value for add-wins
EATV: [prefix][E][A][Tx↓][type][value][AfterRef?][Op]  - first entry is current (E-primary CRDT)
AEVT: [prefix][A][E][type][value][Tx↓][AfterRef?][Op]  - by attribute (V before Tx)
AETV: [prefix][A][E][Tx↓][type][value][AfterRef?][Op]  - first entry is current (A-primary CRDT)
ATEV: [prefix][A][Tx↓][E][type][value][AfterRef?][Op]  - Tx↓ ahead of E: AsOf-by-attribute seeks straight to the transaction
AVET: [prefix][A][type][value][E][Tx↓][AfterRef?][Op]  - value lookup
VAET: [prefix][type][value][A][E][Tx↓][AfterRef?][Op]  - reverse refs
TAEV: [prefix][Tx↓][A][E][type][value][AfterRef?][Op]  - transaction log

AfterRef? = 16 bytes present only if Op ∈ {OpRGAInsert(3), OpRGATombstone(4)} EncodeValueBytes computes the V run a key holds for a value: the type tag followed by the payload. Applies compression if enabled. Returns the bytes for the key and optional BlobData for Tier 3 values. Call this once per datom, then pass the result to EncodeKeyWithValueBytes for each index.

The payload carries no length and no terminator. It does not need one: every component behind V in every index layout is fixed width, and Op — the last byte of every key — says whether AfterRef is present, so a key's own length determines where its V ended. DecodeKey recovers V from that arithmetic, and a scan narrows a V-bound prefix range by the same arithmetic (EncodedRun's membership rule). A delimiter would buy exact byte ranges at the cost of the on-disk format.

This is the only producer of a V run. A scan bound's V component goes through it as well, so a bound and the keys it addresses cannot drift apart.

func (*BinaryKeyEncoder) VAETHashPrefix added in v0.15.3

func (e *BinaryKeyEncoder) VAETHashPrefix(vType datalog.ValueType, hash [20]byte) []byte

VAETHashPrefix returns the VAET key prefix naming every datom whose value is the blob at hash, under one hashed type tag.

VAET leads with the value payload, and a tier-3 payload is the fixed-width [type]hash, so this prefix is exact: every key carrying it references that blob, and no key referencing it under this tag lacks it. That is what makes a reference probe one seek rather than a scan. A caller asking whether a blob is referenced at all probes once per datalog.HashedValueTypes entry, because one blob is reachable under either tag.

type BlobReader added in v0.15.0

type BlobReader interface {
	ReadBlob(hash [20]byte, read func([]byte) error) error
}

BlobReader exposes content-addressed Tier-3 values to the key decoder.

type CRDTResolvingIterator added in v0.7.0

type CRDTResolvingIterator struct {
	// contains filtered or unexported fields
}

CRDTResolvingIterator wraps a storage iterator and applies CRDT resolution per (E, A) group using streaming.

Resolution strategies by cardinality:

  • CardinalityOne: First entry wins (Tx descending). Emit if not tombstoned.
  • CardinalityMany: Emit qualifying ADDs immediately. Minimal state.
  • CardinalityVector: RGA tree reconstruction.

func NewCRDTResolvingIterator added in v0.7.0

func NewCRDTResolvingIterator(source Iterator, schema schema.SchemaProvider, matcher *PatternMatcher, report *scanReport) *CRDTResolvingIterator

NewCRDTResolvingIterator creates a new CRDT-resolving iterator wrapper.

matcher carries the view the scan reads, so a datom after an as-of target is skipped, and resolves a CardinalityOne group whose attribute the schema declares Unique by the walk-based (A, V)-LWW rule: the first non-superseded Set entry in the group is emitted, with supersession decided by an AVET sub-scan for max-other-Tx per candidate V.

report is the arm's, and is where the unique walk's own scans accrue. It is nil when nothing is listening, and nil is also correct for a caller whose scan no report accounts for.

func (*CRDTResolvingIterator) Close added in v0.7.0

func (it *CRDTResolvingIterator) Close() error

Close releases resources held by the iterator.

func (*CRDTResolvingIterator) Datom added in v0.7.0

func (it *CRDTResolvingIterator) Datom() (*datalog.Datom, error)

Datom returns the current CRDT-resolved datom.

func (*CRDTResolvingIterator) ElementID added in v0.7.0

func (it *CRDTResolvingIterator) ElementID() datalog.ElementID

ElementID returns the transaction ElementID of the current entry.

func (*CRDTResolvingIterator) Error added in v0.11.2

func (it *CRDTResolvingIterator) Error() error

Error returns the first error encountered during unique-walk resolution, or nil if iteration proceeded cleanly. Callers that expect walk-based resolution should check this after iterator exhaustion.

func (*CRDTResolvingIterator) Next added in v0.7.0

func (it *CRDTResolvingIterator) Next() bool

Next advances to the next CRDT-resolved datom.

func (*CRDTResolvingIterator) Scanned added in v0.15.2

func (it *CRDTResolvingIterator) Scanned() int

Scanned reports the source's intake, and only that.

Resolution reads no index of its own on most paths, and the gap between this and what resolution emitted is the history depth the index made the query pay for. Unique CardinalityOne is the exception — processUniqueEntry opens an AVET supersession scan per Set entry — and those reads accrue into the arm's report at the point they happen, not here.

They cannot be added here. The scan this iterator wraps was acquired through the arm's report, so the source already accrues into it; folding the walk's reads into this total as well would either double-count them or, for an arm reading the report rather than this method, lose them entirely.

func (*CRDTResolvingIterator) Seek added in v0.7.0

func (it *CRDTResolvingIterator) Seek(bound ScanBound)

Seek positions the iterator at or after the bound's start, discarding the group state accumulated for the run it is leaving. The many state's maps stay for the next group start to empty, so their capacity survives the seek.

type Cache added in v0.6.0

type Cache struct {
	// contains filtered or unexported fields
}

Cache provides O(1) access to resolved CRDT views

The cache is the primary query resolution mechanism for latest and concrete AsOf queries. It stores resolved views (LWW value, add-wins set, RGA vector) for each cache key.

Datoms of a key can arrive below the highest ElementID among those already present: a transaction that drew earlier commits later. So a commit's arrival is an event of its own: it takes an ElementID from the database's clock once its datoms are readable, and raises its keys' versions to that ElementID. A reader takes its read point from the clock before it reads, and the cache stores a resolution only when no arrival on the key lies above that read point.

A view as of a target keeps its resolutions in a cache of its own (ForView), freed with the view. That cache records no arrival: it reads the database cache's record of each key, and serves an entry while no arrival on the key, and no drop of its slot, lies above the read point its resolution began at or above the reader's.

func NewCache added in v0.6.0

func NewCache(clock *LamportClock) *Cache

NewCache creates a cache for a database whose clock is clock. Its floor starts at the clock's read point: the datoms already in the store arrived before the cache was made, so a read that began before then stores nothing into a key with no slot.

func (*Cache) BeginInFlight added in v0.13.0

func (c *Cache) BeginInFlight(keys []CacheKey)

BeginInFlight marks the given keys as committing by storing the in-flight sentinel for each, overwriting any cached entry. Called by Transaction.Commit BEFORE the storage commit; the matching clear is Invalidate (called after the commit, or after a failed commit), which deletes the entries. While a key's sentinel stands, GetOrResolve resolves it from storage and storeIfNotInFlight refuses to cache it. Another commit's Invalidate or a drop of the key's slot can clear the sentinel before this commit's UpdateMaxVersion records the arrival.

func (*Cache) Clear added in v0.6.0

func (c *Cache) Clear()

Clear removes all entries from the cache. It raises the floor first, so a resolution that read before the clear is not stored afterward.

func (*Cache) ForView added in v0.16.2

func (c *Cache) ForView() *Cache

ForView creates the cache of a view as of a target over c's database. It holds the view's resolutions alone, so they are freed with the view, and it reads c's record of the arrivals on each key.

func (*Cache) GetCachedAttrs added in v0.6.0

func (c *Cache) GetCachedAttrs(e datalog.Identity) map[datalog.Keyword]bool

GetCachedAttrs returns the set of attributes we've cached for an entity Returns nil if no attributes are cached for this entity

func (*Cache) GetOrResolve added in v0.6.0

func (c *Cache) GetOrResolve(key CacheKey, def *schema.AttributeDefinition, resolver CacheResolver, readPoint datalog.ElementID, handler annotations.Handler, report *scanReport) (*CacheEntry, error)

GetOrResolve returns the cached entry if fresh, rebuilding if stale. In a view's cache it is getOrResolveForView.

def is the attribute's definition as the caller holds it, nil for an attribute its schema does not declare. Its cardinality and uniqueness are the rule the caller resolves the attribute under. An entry resolved under another rule is not a hit: a set's view serves no reader under cardinality one, and a value the unique walk settled serves no reader under the plain rule, so the entry is rebuilt under def's.

readPoint is the read point the caller took before it read: before its session opened, or now for a reader of the latest store. An entry serves the caller when no datom arrived on its key after the read point, which makes it the value the caller's session holds. Otherwise the caller resolves through its resolver, and the cache stores that resolution only when no datom arrived on the key after the read point.

An arrival clears its key's entry, and a resolution is stored only when it read every arrival, so an entry present and not in flight holds every datom of its key. False negatives (returning stale data) are NOT acceptable. The report accrues the index intake *this call* spends: the entry's build cost when this call built it, and nothing when the entry came from cache. It is not entry.scanned, which is the entry's build cost forever. The distinction is the difference between "what this read cost" and "what this (E, A) cost to resolve once, some time ago", and datoms.scanned means the first everywhere else in the engine — a trace of a thousand hits must not report a thousand index reads that did not happen.

Accrual happens inside the resolvers, so a rebuild that fails keeps what it read; the callers that report nothing pass DiscardIntake and say so.

func (*Cache) Invalidate added in v0.6.0

func (c *Cache) Invalidate(touched []CacheKey)

Invalidate removes cached entries for the given keys, another commit's in-flight sentinel included, and leaves each slot's version where it stands. Called by Database on tx.Commit() with all touched (E,A) pairs.

func (*Cache) InvalidateAttribute added in v0.11.2

func (c *Cache) InvalidateAttribute(a datalog.Keyword)

InvalidateAttribute removes all cached entries for the given attribute, regardless of entity. Used when a write to a unique attribute may have silently staled other entities' cached values under the walk-based (A, V)-LWW resolution.

Conservative strategy (CRDT_UNIQUE_SEMANTICS.md D3): one unique-attr write invalidates every cached entry for that attribute. A future optimization could maintain an (A, V) → [E] reverse index to invalidate only the entities whose walks could actually change; the current approach is simpler and correct.

It raises the floor before it drops the attribute's slots, so a resolution of another entity's key that read before the write is not stored, whether or not the key had a slot.

func (*Cache) InvalidateRewind added in v0.13.3

func (c *Cache) InvalidateRewind(touched []CacheKey)

InvalidateRewind drops the slots of keys whose datoms a rollback physically removed, once the removal is readable. It raises the floor first, so a resolution that read the removed datoms is not stored for them. Pairs with BeginInFlight: that opens the uncached window before the delete, this closes it after.

func (*Cache) PopulateFromDatoms added in v0.10.0

func (c *Cache) PopulateFromDatoms(key CacheKey, card datalog.Keyword, datoms []datalog.Datom, readPoint datalog.ElementID)

PopulateFromDatoms resolves a group of datoms for a single (E, A) pair and stores the result in the cache. Uses the canonical Resolve*FromDatoms functions from crdt_resolve.go — no CRDT logic duplication.

This is the dispatch target for PrefetchEntities: as the EATV iterator crosses an attribute boundary, the accumulated datoms are resolved and cached here in a single call.

readPoint is the read point the caller took before it began reading the datoms.

func (*Cache) ReadPoint added in v0.16.2

func (c *Cache) ReadPoint() datalog.ElementID

ReadPoint is the read point of a reader of the latest store that begins now.

func (*Cache) TrackEntityAttr added in v0.6.0

func (c *Cache) TrackEntityAttr(e datalog.Identity, a datalog.Keyword)

TrackEntityAttr records that we have cached an (E, A) entry Called when storing cache entries to track what's cached per entity

func (*Cache) UpdateMaxVersion added in v0.6.0

func (c *Cache) UpdateMaxVersion(key CacheKey, arrival datalog.ElementID)

UpdateMaxVersion records an arrival of datoms on key. arrival is an ElementID drawn from the database's clock once the datoms are readable. The slot's version rises to it, and the slot's entry, which may lack the datoms, is dropped; an in-flight sentinel stays for its commit to clear.

type CacheEntry added in v0.6.0

type CacheEntry struct {
	// contains filtered or unexported fields
}

CacheEntry holds the resolved CRDT view for an (E, A) pair

func ResolveEntry added in v0.7.0

func ResolveEntry(key CacheKey, def *schema.AttributeDefinition, resolver CacheResolver, report *scanReport) (*CacheEntry, error)

ResolveEntry resolves the current value for (E, A) under the rule def declares, def being the caller's definition of the attribute, nil for an attribute its schema does not declare.

It returns three outcomes, and they are three: an error, a non-nil entry, or a nil entry with a nil error meaning **this (E, A) has no datoms**. Absence is a resolved answer, not a failure and not an empty value, and it is never cached — so downstream, the existence of an entry *is* the existence of the attribute, and nothing has to reconstruct that from a count or a nil slice.

It takes no receiver: resolution reads storage through the resolver and touches no cache state. GetOrResolve is this plus the freshness check and the store; the cache-disabled read path calls it directly.

func (*CacheEntry) Cardinality added in v0.6.0

func (e *CacheEntry) Cardinality() datalog.Keyword

Cardinality returns the cardinality of this cache entry

func (*CacheEntry) ManySet added in v0.6.0

func (e *CacheEntry) ManySet() map[string]any

ManySet returns the set for cardinality-many attributes, each member under its memberKey.

func (*CacheEntry) ManyTx added in v0.16.3

func (e *CacheEntry) ManyTx() map[string]datalog.ElementID

ManyTx returns, under each member's memberKey, the Tx of the member's newest assertion for cardinality-many attributes.

func (*CacheEntry) OneTx added in v0.16.3

func (e *CacheEntry) OneTx() datalog.ElementID

OneTx returns the Tx of the datom a cardinality-one entry resolves to.

func (*CacheEntry) OneValue added in v0.6.0

func (e *CacheEntry) OneValue() any

OneValue returns the value for cardinality-one attributes

func (*CacheEntry) VectorIndex added in v0.6.0

func (e *CacheEntry) VectorIndex() []datalog.ElementID

VectorIndex returns the position index for cardinality-vector attributes

func (*CacheEntry) VectorList added in v0.6.0

func (e *CacheEntry) VectorList() []any

VectorList returns the ordered list for cardinality-vector attributes

func (*CacheEntry) Version added in v0.6.0

func (e *CacheEntry) Version() datalog.ElementID

Version returns the ElementID when this entry was computed

type CacheKey added in v0.6.0

type CacheKey struct {
	E datalog.Identity
	A datalog.Keyword
}

CacheKey identifies an (entity, attribute) pair for caching, by the interned identity and keyword a reader holds, so the key compares by pointer. The rule an entry was resolved under is the entry's own, its cardinality and unique fields, compared on a hit with the definition the reader passes (GetOrResolve).

The key names no view. A database's cache holds the latest resolutions, and a view as of a target holds its own in a cache of its own (Cache.ForView).

type CacheResolver added in v0.6.0

type CacheResolver interface {

	// ResolveLWW returns the current value for cardinality-one (highest ElementID wins),
	// under the unique rule when def declares a unique and the resolver reads
	// resolved state: the entity's first assertion no other entity's later
	// claim supersedes.
	// Returns (value, tx, version, present, error): tx is the Tx of the datom
	// the pair resolves to, the assertion the value was read from or the
	// withdrawal that left it nil, and version is the highest ElementID the
	// resolution read, other entities' claims included under the unique rule.
	// Intake accrues into the report.
	ResolveLWW(e datalog.Identity, a datalog.Keyword, def *schema.AttributeDefinition, report *scanReport) (any, datalog.ElementID, datalog.ElementID, bool, error)

	// ResolveAddWins returns the current set members for cardinality-many,
	// each under its memberKey, and under the same key the Tx of each
	// member's newest assertion, the datom the member resolves to.
	// Returns (members, memberTx, maxElementID, present, error); intake accrues
	// into the report.
	ResolveAddWins(e datalog.Identity, a datalog.Keyword, report *scanReport) (map[string]any, map[string]datalog.ElementID, datalog.ElementID, bool, error)

	// ResolveRGA returns the ordered vector for cardinality-vector
	// Returns (elements, positionIndex, maxElementID, present, error); intake
	// accrues into the report.
	ResolveRGA(e datalog.Identity, a datalog.Keyword, report *scanReport) ([]any, []datalog.ElementID, datalog.ElementID, bool, error)
}

CacheResolver provides methods to resolve CRDT values from storage This interface decouples the cache from specific storage implementation

Every resolve method accrues the index intake it spends into the report it is handed. Resolution reads the index on a pattern's behalf and emits no annotation of its own — one event per nested read would bury the query the stream exists to describe — so the count reaches its reporter through the report, and the cache stores the entry's own share on the CacheEntry, which is what the read bought.

type Database

type Database struct {
	// contains filtered or unexported fields
}

Database provides the main API for reading and writing datoms

func BuildTestDatabase

func BuildTestDatabase(config TestDataConfig) (*Database, error)

BuildTestDatabase creates a pre-populated BadgerDB for benchmarking. config.OutputPath is named from the module root (see BenchmarkDatabasePath), so the database lands where its readers look no matter which directory the builder was started in.

func NewDatabase

func NewDatabase(path string) (*Database, error)

NewDatabase creates a new database with BadgerDB storage

func NewDatabaseWithOptions added in v0.2.0

func NewDatabaseWithOptions(opts DatabaseOptions) (*Database, error)

NewDatabaseWithOptions creates a database with the specified options. This is the most flexible constructor, supporting all configuration options.

The store comes from exactly one of two places. Either this constructor opens one — BackendName (or the build default) at Path, with an encoder built from CompressionThreshold — or Store supplies one the caller opened already, in which case those three fields are errors rather than ignored inputs. On success Database.Close closes the store either way; on constructor failure only a store this constructor opened is Closed.

Options:

  • Schema: Optional schema for validation.
  • ReplicaID: For CRDT mode. 0 = auto-generate random; non-zero = use specified.
  • DisableCache: Disable EA cache; queries resolve directly from storage.

Example:

db, err := storage.NewDatabaseWithOptions(storage.DatabaseOptions{
    Path: "/path/to/db",
})

func NewDatabaseWithSchema added in v0.2.0

func NewDatabaseWithSchema(path string, s schema.SchemaProvider) (*Database, error)

NewDatabaseWithSchema creates a database with schema validation

func NewSliceSource added in v0.16.2

func NewSliceSource[T any](items []T, schema AttributeSchema[T]) (*Database, error)

NewSliceSource opens a memory database holding a Go slice's facts under an attribute schema. The item at index i is the entity "slice:i", holding at each attribute the value its accessor answers, or every element of the slice or array it answers. The facts open through OpenFacts, so an attribute at which one item holds two elements is cardinality-many.

Example:

type Rule struct {
    Key       string
    DependsOn []string
}

source, err := NewSliceSource(rules, AttributeSchema[Rule]{
    datalog.NewKeyword(":rule/key"):        func(r Rule) any { return r.Key },
    datalog.NewKeyword(":rule/depends-on"): func(r Rule) any { return r.DependsOn },
})
if err != nil {
    return err
}
defer source.Close()

func OpenFacts added in v0.16.2

func OpenFacts(facts []datalog.Datom, opts DatabaseOptions) (*Database, error)

OpenFacts opens a database on the memory-trees backend, under opts, holding facts: each datom's value asserted at its entity and attribute. Every fact holds. So an attribute opts.Schema leaves undefined is cardinality-many where one entity holds two values of it, and cardinality-vector where its value is a vector, each entity's vector written whole. Two different vectors at one entity cannot both hold, nor can a vector and a scalar at one attribute, and OpenFacts refuses each. A value is held as a transaction's Add holds it, and refused with Add's error where Add refuses it (writtenValue); a vector's elements each so.

OpenFacts writes the facts as an import writes a dump: each datom under the op its attribute's cardinality writes, asserted into the store, and then the schema installed from what the store states. The store infers the cardinality of each attribute no definition covers from those ops, so OpenFacts declares nothing beyond a given schema.

A datom's Tx is the database's clock's to assign and is not read. A datom carrying an op is a log's record, not a fact, and is refused: a log opens from its dump.

func OpenTestDatabase

func OpenTestDatabase(path string) (*Database, error)

OpenTestDatabase opens a pre-built test database. The path is named from the module root (see BenchmarkDatabasePath), so a reader names the same file the builder wrote regardless of which directory either was started in.

func (*Database) Analyze added in v0.2.3

func (d *Database) Analyze(queryInput interface{}, inputs ...interface{}) (*AnalyzeResult, error)

Analyze executes a query and returns detailed execution statistics. The query can be either an EDN string or a *query.Query from the query builder. This is like EXPLAIN ANALYZE in PostgreSQL - it actually runs the query and captures timing information at each step.

Example:

result, err := db.Analyze(`[:find ?e :in $ ?scenario :where [?e :task/scenario ?scenario]]`, scenarioID)
if err != nil { ... }
fmt.Println(result.String())  // Shows plan + execution trace

The result includes:

  • Query plan with index selection
  • Query result as a Relation (call .Size() or iterate to access)
  • Event trace with timing for each operation
  • Summary statistics

func (*Database) AsOf

func (d *Database) AsOf(target datalog.ElementID) *Database

AsOf returns a read-only Database handle that queries state as of the given transaction. The returned handle uses CRDT resolution filtered to that point in causal time.

The temporal handle shares the parent's store and schema. It keeps its resolutions in a cache of its own over the parent's (Cache.ForView), so they are freed with the handle and every arrival the parent's cache records reaches them. With the parent's cache disabled the handle has none. It supports reads (Query, NewExecutor, NewExecutorWithOptions, Matcher, Pull) but not writes: NewTransaction will panic. Close is a no-op — the parent owns the store lifetime.

func (*Database) AsOfSnapshot added in v0.13.3

func (d *Database) AsOfSnapshot(name string) (*Database, error)

AsOfSnapshot returns a read-only handle viewing the database as of the named snapshot. The returned handle rejects writes (NewTransaction panics on a temporal handle).

It reads at the marker's own high-water (snapshotMarkerMax), which is domain-equivalent to the captured point At — the marker datoms it adds are reserved-namespace and filtered from domain queries. An as-of handle carries its target beside its mode, so a handle at the zero ElementID, an empty-database snapshot's At, reads nothing: every datom lies after the target (shouldFilterTx).

func (*Database) AsOfTx added in v0.16.1

func (d *Database) AsOfTx() (txID datalog.ElementID, ok bool)

AsOfTx reports the transaction an AsOf handle reads as of, and whether this handle is one. The live database reads the store as it stands, and a History handle reads every datom ever written, so neither reads as of a transaction and ok is false for both.

func (*Database) Assert added in v0.8.0

func (d *Database) Assert(datoms []datalog.Datom) error

Assert adds datoms in a single transaction.

func (*Database) AttributeLookup added in v0.16.3

func (d *Database) AttributeLookup(attr datalog.Keyword) (query.AttributeLookup, error)

AttributeLookup implements executor.EntityLookupMatcher for a database passed as a named source, answering through its matcher.

func (*Database) Cache added in v0.6.0

func (d *Database) Cache() *Cache

Cache returns the CRDT resolution cache. Used by the matcher for O(1) access to resolved CRDT views.

func (*Database) CanFuseAttributeFetch added in v0.16.2

func (d *Database) CanFuseAttributeFetch(source query.Symbol, attr datalog.Keyword) bool

CanFuseAttributeFetch implements executor.AttributeFetchFusable for a database passed as a named source, answering through its matcher: whether a match of attr on the database answers, per entity, the one value its lookup answers, under the schema it runs with and in its view.

func (*Database) ClearPlanCache

func (d *Database) ClearPlanCache()

ClearPlanCache clears the query plan cache

func (*Database) Clock added in v0.6.0

func (d *Database) Clock() *LamportClock

Clock returns the Lamport clock for CRDT operations. Returns nil if the database was created without CRDT support.

func (*Database) Close

func (d *Database) Close() error

Close closes the database.

On a temporal handle (AsOf/History), Close is a no-op — the parent owns the store lifetime. Closing a temporal handle must not close the shared underlying store.

func (*Database) DeleteSnapshot added in v0.13.3

func (d *Database) DeleteSnapshot(name string) (deleteErr error)

DeleteSnapshot removes the named snapshot from the registry by retracting its marker. The rewindable timeline is untouched; only the registry entry goes away.

func (*Database) Explain added in v0.2.3

func (d *Database) Explain(queryInput interface{}, inputs ...interface{}) (*planner.RealizedPlan, error)

Explain returns the query plan for a query without executing it. The query can be either an EDN string or a *query.Query from the query builder. This is useful for understanding index selection, phase structure, and how input parameters affect symbol binding.

Example:

plan, err := db.Explain(`[:find ?e :in $ ?scenario :where [?e :task/scenario ?scenario]]`, scenarioID)
if err != nil { ... }
fmt.Println(plan.String())  // Human-readable plan output

The plan shows:

  • Index selection (EAVT, AEVT, AVET, VAET, TAEV)
  • Selectivity scores for patterns
  • Symbol availability and binding through phases
  • Predicate and expression assignment

func (*Database) ExplainAlgebra added in v0.15.1

func (d *Database) ExplainAlgebra(queryInput interface{}, inputs ...interface{}) (*planner.AlgebraExplanation, error)

ExplainAlgebra returns, without executing, the full record of what planning does to a query: the relational algebra it compiles to, every rewrite decision the optimization passes made (applied, or declined with the failed precondition), the optimized tree, the Datalog it decompiles back to, and the physical plan — all as values. The query can be either an EDN string or a *query.Query from the query builder.

Example:

expl, err := db.ExplainAlgebra(`[:find ?s (max ?h) :in $ :where ...]`)
if err != nil { ... }
fmt.Println(expl.String())          // full rendering
for _, r := range expl.Rewrites {   // typed transform provenance
    fmt.Println(r.Pass, r.Action, r.Reason)
}

When this database's planner options disable the algebra optimizer, the explanation carries the compiled algebra view only (the algebra of a query is a fact about the query), no rewrites, and the physical plan exactly as this database would produce it.

func (*Database) Export added in v0.5.6

func (d *Database) Export(w io.Writer, opts ...ExportOptions) error

Export writes all datoms in the database to EDN format. With no options, values are serialized as plain EDN strings. With Compressed: true, large strings use #lzj tagged literals.

func (*Database) ExportBinary added in v0.15.0

func (d *Database) ExportBinary(w io.WriteSeeker, opts ...BinaryExportOptions) (exportErr error)

ExportBinary writes the complete EAVT datom log as a seekable JDZL file. w must be an io.WriteSeeker so the trailer index offset can be patched.

func (*Database) ExportCompressed added in v0.11.0

func (d *Database) ExportCompressed(w io.Writer) error

ExportCompressed is a convenience alias for Export with Compressed: true. To filter entities during compressed export, use Export directly with ExportOptions.

func (*Database) Fork added in v0.15.4

func (d *Database) Fork(name string) (branch *Database, err error)

Fork starts a branch from the snapshot named name.

The branch holds what this database held as of the snapshot, the state AsOfSnapshot reads, and neither sees what the other commits afterward. It opens as any database does, with this database's given schema declared into it where its store lacks it, and runs with the schema its store states; it plans with this database's planner options and carries its own EA cache. It writes as a replica of its own: a fresh ReplicaID, and a clock started past every ElementID it holds, so every write it makes orders after everything it inherited.

Fork records the fork in this database: the snapshot's name, the point the snapshot captured, the point the branch holds this database's state as of, and the branch's ReplicaID. While a snapshot of that name holds the captured point, TruncateTo refuses to pass the branch's point; deleting the snapshot, or a later take of its name that captures another point, releases the branch. Fork opens the transaction that records the branch before it looks up the snapshot, so a TruncateTo that starts holding writers while the Fork runs waits for the record. A Fork begun after a TruncateTo starts holding writers is refused with ErrRollbackInProgress. An AsOf or History handle is read-only and cannot fork, and neither can a store with no way to start a branch.

func (*Database) GetBool added in v0.8.0

func (d *Database) GetBool(e datalog.Identity, a datalog.Keyword) (bool, bool, error)

GetBool returns a bool attribute value.

func (*Database) GetFloat added in v0.8.0

func (d *Database) GetFloat(e datalog.Identity, a datalog.Keyword) (float64, bool, error)

GetFloat returns a float64 attribute value.

func (*Database) GetInt added in v0.8.0

func (d *Database) GetInt(e datalog.Identity, a datalog.Keyword) (int64, bool, error)

GetInt returns an int64 attribute value.

func (*Database) GetRef added in v0.8.0

GetRef returns an entity reference attribute value.

func (*Database) GetString added in v0.8.0

func (d *Database) GetString(e datalog.Identity, a datalog.Keyword) (string, bool, error)

GetString returns a string attribute value.

func (*Database) GetStrings added in v0.8.0

func (d *Database) GetStrings(e datalog.Identity, a datalog.Keyword) ([]string, error)

GetStrings returns all string values for a cardinality-many attribute.

func (*Database) GetVectorLength added in v0.6.0

func (d *Database) GetVectorLength(e datalog.Identity, a datalog.Keyword) (int64, error)

GetVectorLength returns the length of a vector attribute as the handle reads it, through the handle's own matcher, so an as-of handle answers as of its target. Uses the cache for O(1) access when the cache is fresh. Returns 0 if the attribute holds no vector, and an error if the attribute is not a vector. An attribute wider than storage holds is refused before any read, and a database that refuses unknown attributes refuses an attribute its schema does not declare, as every typed accessor does.

func (*Database) GetVectorNth added in v0.6.0

func (d *Database) GetVectorNth(e datalog.Identity, a datalog.Keyword, n int64) (any, error)

GetVectorNth returns the nth element of a vector attribute as the handle reads it, through the handle's own matcher, so an as-of handle answers as of its target. Uses the cache for O(1) access when the cache is fresh. Returns nil if the index is out of bounds or the attribute holds no vector, and an error if the attribute is not a vector. An attribute wider than storage holds is refused before any read, and a database that refuses unknown attributes refuses an attribute its schema does not declare, as every typed accessor does.

func (*Database) History added in v0.2.0

func (d *Database) History() *Database

History returns a read-only Database handle that returns all raw datoms without CRDT resolution. Every write is visible, including superseded values.

The temporal handle shares the parent's underlying store, cache, and schema. It supports reads (Query, NewExecutor, NewExecutorWithOptions, Matcher, Pull) but not writes: NewTransaction will panic. Close is a no-op — the parent owns the store lifetime.

func (*Database) Import added in v0.5.6

func (d *Database) Import(r io.Reader) error

Import loads an EDN dump into the database (replayEDN) and then composes the schema the database runs with from the definitions the dump carried, under a given schema if one was declared, whose facts take ElementIDs past the restored clock and so supersede the imported ones for the idents it names.

func (*Database) ImportBinary added in v0.15.0

func (d *Database) ImportBinary(r io.ReadSeeker, opts ...BinaryImportOptions) error

ImportBinary reads a JDZL file and asserts its datoms. r must be an io.ReadSeeker. Chunks are decoded and asserted in parallel up to Workers.

The first worker error cancels remaining launches and in-flight work between steps, then that error is returned. Import is not transactional across chunks: workers that already Asserted leave a nondeterministic partial subset of the file in the store. Retrying into the same database is not a safe recovery; use a fresh database (or discard the target) after failure.

func (*Database) ImportDump added in v0.15.2

func (d *Database) ImportDump(r io.ReadSeeker) error

ImportDump loads an exported dump (replayDump) and then composes the schema the database runs with from the definitions the dump carried, under a given schema if one was declared.

func (*Database) LookupAttribute added in v0.16.2

func (d *Database) LookupAttribute(entity datalog.Identity, attr datalog.Keyword) (interface{}, bool, error)

LookupAttribute implements executor.EntityLookupMatcher for a database passed as a named source, answering through its matcher.

func (*Database) LookupByUnique added in v0.11.2

func (d *Database) LookupByUnique(attr datalog.Keyword, value interface{}) (datalog.Identity, error)

Pull retrieves entity data according to a pull pattern This provides Datomic-style entity attribute retrieval with nested reference following.

The pattern syntax:

  • Simple attributes: [:entity/name :entity/code]
  • Wildcard: [*] - all attributes
  • Nested references: [{:entity/region [:region/code :region/name]}]
  • Default values: [(default :entity/status "unknown")]
  • Limits (for cardinality-many): [(limit :entity/tags 10)]

Examples:

// Get name and code for an entity
result, err := db.Pull(entityID, `[:entity/name :entity/code]`)

// Get all attributes
result, err := db.Pull(entityID, `[*]`)

LookupByUnique returns the entity currently owning (attr, value) under (A, V)-LWW resolution. Returns a nil Identity with a nil error if no entity currently claims the value.

The attribute must be declared unique in the schema (UniqueValue or UniqueIdentity). Calling LookupByUnique on a non-unique attribute, an attribute not present in the schema, or a database without a schema returns an error.

LookupByUnique is the primitive for natural-key lookup in the CRDT model: application-layer upsert ("find-or-create by email") is built on top of it. See docs/proposals/CRDT_UNIQUE_SEMANTICS.md for the design rationale.

Concurrent writers may change the canonical owner between calls; treat the result as a snapshot. If an application needs to verify a write "won," it should call LookupByUnique after the commit and compare the returned Identity to its own.

func (*Database) LookupSource added in v0.16.2

func (d *Database) LookupSource(source query.Symbol) (executor.EntityLookupMatcher, error)

LookupSource implements executor.EntityLookupMatcher for a database passed as a named source: it is one source, so its matcher is the lookup of the source under any name, and get-else, missing? and get-some on that source read it, a get-else default typed by the schema it runs with.

func (*Database) Match added in v0.5.6

func (d *Database) Match(q *query.Query, bindings executor.Relations) (executor.Relation, error)

Match implements executor.PatternMatcher — the Database itself can answer pattern queries. This delegates to the Database's Matcher(), which uses the full BadgerDB index infrastructure.

func (*Database) Matcher

func (d *Database) Matcher() executor.PatternMatcher

Matcher returns the default-source pattern matcher, configured from the database's effective planner options (override or defaults) so the relations it produces carry the same executor options the query runs with — not a hardcoded default set.

func (*Database) NewExecutor

func (d *Database) NewExecutor() *executor.Executor

NewExecutor creates a new query executor under the database's effective options, with its plan cache and its registered annotation handler.

func (*Database) NewExecutorWithOptions

func (d *Database) NewExecutorWithOptions(opts planner.PlannerOptions) *executor.Executor

NewExecutorWithOptions creates a new query executor with custom planner options and the database's plan cache.

Builds the pattern matcher from the same custom options via matcherWithExecOptions, so the executor and the matcher's relations agree on the effective options — including opts.Handler, which reaches every relation either constructs — and schema, cache, and temporal mode (AsOf/History) are all applied. The matcher carries whether the database refuses unknown attributes, and the executor has its plans carry their rosters when it does, for the check at its entry.

func (*Database) NewTransaction

func (d *Database) NewTransaction() *Transaction

NewTransaction starts a new write transaction. It opens a read session, which holds the value the transaction's view reads until Commit or Rollback. A transaction begun while a TruncateTo holds writers is dropped: it opens no session, and its writes and every read of its view answer ErrRollbackInProgress.

Panics if called on a read-only temporal handle (AsOf/History). Temporal views share the parent's store and have no write-side state; use the parent handle for writes.

func (*Database) NewTransactionAt

func (d *Database) NewTransactionAt(t time.Time) *Transaction

NewTransactionAt starts a new write transaction with a specific time This is a convenience method for financial/temporal data

func (*Database) ParseCache added in v0.9.0

func (d *Database) ParseCache() *ParseCache

ParseCache returns the database's query parse cache

func (*Database) PlanCache

func (d *Database) PlanCache() *planner.PlanCache

PlanCache returns the database's query plan cache

func (*Database) Pull added in v0.2.0

func (d *Database) Pull(entityID datalog.Identity, patternStr string) (map[string]interface{}, error)
// Get attributes with nested reference
result, err := db.Pull(entityID, `[:entity/name {:entity/region [:region/code]}]`)

Returns nil if the entity does not exist.

func (*Database) PullInto added in v0.2.0

func (d *Database) PullInto(entityID datalog.Identity, v interface{}) error

PullInto retrieves entity data and populates the provided struct. The struct fields are mapped to attributes based on datalog struct tags. A pull pattern is automatically generated from the struct definition. Schema is used to properly handle cardinality-many attributes.

Example:

type Person struct {
    ID      datalog.Identity `datalog:"-,id"`
    Name    string           `datalog:"name"`
    Age     int64            `datalog:"age"`
    Friends []*Person        `datalog:"friends"`
}
var person Person
err := db.PullInto(entityID, &person)

func (*Database) PullIntoMany added in v0.2.0

func (d *Database) PullIntoMany(entityIDs []datalog.Identity, v interface{}) error

PullIntoMany retrieves data for multiple entities and populates the provided slice. The slice must be a pointer to a slice of structs (e.g., *[]Person or *[]*Person). Schema is used to properly handle cardinality-many attributes.

Example:

var people []Person
err := db.PullIntoMany(entityIDs, &people)

func (*Database) PullMany added in v0.2.0

func (d *Database) PullMany(entityIDs []datalog.Identity, patternStr string) ([]map[string]interface{}, error)

PullMany retrieves data for multiple entities using the same pull pattern This is more efficient than calling Pull multiple times.

Example:

results, err := db.PullMany(
    []datalog.Identity{entity1, entity2, entity3},
    `[:entity/name :entity/code]`,
)

Returns a slice of maps, one per entity. Nil entries indicate non-existent entities.

func (*Database) Query added in v0.8.0

func (d *Database) Query(queryInput interface{}, inputs ...interface{}) (executor.Relation, error)

Query executes a Datalog query and returns a streaming Relation. The caller should iterate via rel.Iterator() and must call iter.Close() when done. A registered handler sees the query invoked once and completed once: refused or failed before the result exists, or, for a query that runs, when the result's iterator is exhausted or closed.

func (*Database) QueryInto added in v0.2.0

func (d *Database) QueryInto(dest interface{}, queryInput interface{}, inputs ...interface{}) (err error)

QueryInto executes a Datalog query and populates a slice of structs with the results. The query can be either an EDN string or a *query.Query from the query builder. Fields are mapped by `datalog` tags (e.g., `datalog:"?name"`) or by position if untagged.

For aggregates, use the full expression as the tag (e.g., `datalog:"(sum ?salary)"`).

Example:

type PersonResult struct {
    Name string `datalog:"?name"`
    Age  int64  `datalog:"?age"`
}

var results []PersonResult
err := db.QueryInto(&results, `[:find ?name ?age :where [?e :person/name ?name] [?e :person/age ?age]]`)

Example with query builder:

e, name, age := qb.NewVar("e"), qb.NewVar("name"), qb.NewVar("age")
q := qb.Query().Find(name, age).Where(qb.Pat(e, PersonName, name), qb.Pat(e, PersonAge, age)).MustBuild()
err := db.QueryInto(&results, q)

Example with aggregates:

type DeptStats struct {
    Dept     string  `datalog:"?dept"`
    TotalPay float64 `datalog:"(sum ?salary)"`
}

var stats []DeptStats
err := db.QueryInto(&stats, `[:find ?dept (sum ?salary) :where [?e :employee/dept ?dept] [?e :employee/salary ?salary]]`)

func (*Database) QueryOneInto added in v0.2.0

func (d *Database) QueryOneInto(dest interface{}, queryInput interface{}, inputs ...interface{}) (found bool, err error)

QueryOneInto executes a Datalog query expecting at most one result and populates a value. The query can be either an EDN string or a *query.Query from the query builder. Supports both struct destinations (multi-symbol) and scalar destinations (single-symbol). Returns (true, nil) if a result was found and mapped successfully. Returns (false, nil) if the query returns no results (empty result is valid, not an error). Returns (false, ErrMultipleResults) if more than one result exists. Returns (false, err) for other errors (parse errors, type errors, etc.).

Example (struct):

var result PersonResult
found, err := db.QueryOneInto(&result, `[:find ?name ?age :where ...]`)

Example (scalar):

var name string
found, err := db.QueryOneInto(&name, `[:find ?name :where [?e :person/name ?name] [(= ?e eid)]]`)

func (*Database) RefusesUnknownAttributes added in v0.16.0

func (d *Database) RefusesUnknownAttributes(_ query.Symbol) bool

RefusesUnknownAttributes implements executor.AttributeVocabulary for a database passed as a named source: it refuses when opened with RefuseUnknownAttributes.

func (*Database) ReplicaID added in v0.6.0

func (d *Database) ReplicaID() uint64

ReplicaID returns this database's replica identifier for CRDT operations. Each database instance should have a unique ReplicaID for proper multi-replica merge semantics. The ID is auto-generated if not specified in DatabaseOptions.

func (*Database) ResolveAllAttributes added in v0.6.0

func (d *Database) ResolveAllAttributes(entity datalog.Identity) (resolved map[datalog.Keyword]interface{}, resolveErr error)

ResolveAllAttributes retrieves all CRDT-resolved attributes for an entity. This is used by wildcard pulls to get all attributes with proper CRDT resolution.

The attributes answered are the entity's own, kept to the set wildcardAttributes builds when the store declares any. In history mode an EAVT scan discovers them and ResolveEntityAttributes resolves each; otherwise one walk of the entity's EATV range discovers and resolves them, the batch wildcard walk applied to one entity. Per the codebase principle that the cache is an optimization, not a correctness requirement, this method works correctly under DisableCache: true.

Returns a map of keyword -> resolved value. The value type depends on cardinality:

  • CardinalityOne: single value (LWW)
  • CardinalityMany: []interface{} (add-wins set)
  • CardinalityVector: []interface{} (RGA ordered list)

func (*Database) ResolveAllAttributesMany added in v0.14.2

func (d *Database) ResolveAllAttributesMany(
	entities []datalog.Identity,
) ([]map[datalog.Keyword]interface{}, error)

ResolveAllAttributesMany resolves wildcard pulls for a complete entity set. The dominant non-unique EATV traversal shares one store scan session and seeks forward per entity; unique-ownership walks and blob dereferences retain their specialized reads. Results preserve input order. Duplicate entities are scanned once.

func (*Database) ResolveEntityAttributes added in v0.6.0

func (d *Database) ResolveEntityAttributes(entity datalog.Identity, attrs []datalog.Keyword) (map[datalog.Keyword]interface{}, error)

ResolveEntityAttributes resolves CRDT values for the requested attributes of an entity. This is the core resolution method that properly applies CRDT semantics: - CardinalityOne: LWW (Last-Writer-Wins) - CardinalityMany: Add-wins set - CardinalityVector: RGA ordered list

With the cache on, it partitions the attributes by what the cache has tracked for the entity, and resolves each attribute of both parts through GetOrResolve, which serves a fresh entry and resolves from storage otherwise.

Returns a map of keyword -> resolved value. Missing attributes are not included. An attribute asked that is wider than storage holds is refused before any read, and a database that refuses unknown attributes refuses every attribute asked that its schema does not declare, joined in order, as every typed accessor does.

func (*Database) Schema added in v0.2.0

func (d *Database) Schema() schema.SchemaProvider

Schema returns the current schema (may be nil)

func (*Database) SetParseCache added in v0.9.0

func (d *Database) SetParseCache(cache *ParseCache)

SetParseCache sets a custom parse cache or disables caching (if nil)

func (*Database) SetPlanCache

func (d *Database) SetPlanCache(cache *planner.PlanCache)

SetPlanCache sets a custom plan cache or disables caching (if nil)

func (*Database) SetSchema added in v0.2.0

func (d *Database) SetSchema(s schema.SchemaProvider) error

SetSchema declares s as the database's schema: its definitions are asserted into the store, superseding the stored ones for the idents it names, and the schema the database runs with is composed from both. A nil s withdraws the given schema, and the database runs with the store's definitions and what the ops of the other attributes infer; a database that refuses unknown attributes keeps a schema to check against and refuses nil, and a schema declaring an engine word is refused, each before anything changes, so a refused call leaves the given schema as it was. Schema changes take effect for new transactions. A read-only temporal handle has no store of its own to assert into and is refused.

func (*Database) Snapshot added in v0.13.3

func (d *Database) Snapshot(name string) (snap SnapshotInfo, snapErr error)

Snapshot names the current state of this handle as a :db.snapshot/* entity. It captures store.MaxElementID() as the snapshot point and writes the marker in a single transaction. Returns ErrSnapshotExists if the name is already in use.

func (*Database) Snapshots added in v0.13.3

func (d *Database) Snapshots() ([]SnapshotInfo, error)

Snapshots lists every snapshot, in causal order (by captured point, then name).

func (*Database) Stats

func (d *Database) Stats() (map[string]interface{}, error)

Stats returns database statistics

func (*Database) Store

func (d *Database) Store() Store

Store returns the underlying store for direct access (debugging/testing)

func (*Database) TruncateTo added in v0.13.3

func (d *Database) TruncateTo(name string) error

TruncateTo destructively rewinds this handle's writable timeline to the named snapshot. Every datom written after the snapshot's own marker is PHYSICALLY removed (not tombstoned), so it disappears from History() as well as from current reads; the clock resumes from the snapshot point so the next write does not collide. Snapshots taken after the target are pruned along with the timeline they indexed; the target snapshot and earlier ones survive. The record of a fork from a surviving snapshot survives with it, since a TruncateTo past that snapshot reads the record. See docs/proposals/BRANCHING_AND_SNAPSHOTS.md §12.1 (Slice B).

Concurrency (§12.1, "Rollback safety"): TruncateTo serializes against other rollbacks (rollbackMu), drains in-flight write transactions, and drops writes started while it runs (they return ErrRollbackInProgress). An in-flight write opened before the rollback is allowed to commit; its post-snapshot datoms are then erased like any other Tx > markerMax. Reads are never locked — BadgerDB MVCC keeps them consistent, and the cache in-flight window (BeginInFlight → delete → InvalidateRewind) keeps them from caching a soon-to-be- stale value while the rewind is visible mid-flight.

func (*Database) UnknownAttributes added in v0.16.0

func (d *Database) UnknownAttributes(_ query.Symbol, attrs []datalog.Keyword) error

UnknownAttributes implements executor.AttributeVocabulary for a database passed as a named source. Opened with RefuseUnknownAttributes, it refuses every attribute of the roster that its store does not declare and the engine does not reserve, joined in order; otherwise it refuses nothing, and the roster is not read.

func (*Database) WarmCache added in v0.6.0

func (d *Database) WarmCache(attributes []datalog.Keyword) error

WarmCache pre-populates cache entries for the specified attributes. Use this after process restart for attributes with high query frequency to avoid cold-start latency on first access.

For each attribute, this scans all entities and populates per-(E,A) cache entries with resolved CRDT values, each resolved through the handle's own matcher, so an as-of handle's entries hold the values as of its target.

This is O(n) where n = total datoms for the specified attributes. Call during application startup, not on the hot path.

A History handle resolves nothing, so its matcher refuses every cache key (cacheKey) and the call stores nothing.

With DisableCache there is no cache to warm and the call does nothing. Every other reader of d.cache in this file resolves from storage instead, because the cache is an optimization rather than a correctness requirement; warming has no such fallback, since warming is the whole of what it does.

type DatabaseOptions added in v0.2.0

type DatabaseOptions struct {
	// BackendName selects the store to open, from AvailableBackends. Empty means
	// DefaultBackend for this build. A named persistent backend requires Path; a
	// named in-process one rejects it, having nowhere to put it.
	BackendName string
	// Path is the directory holding a persistent backend's data. The build
	// default is the one case where a path can go unhonored: under js/wasm
	// DefaultBackend is MemoryTreeStore, and portable code that opens by path
	// gets an in-process database rather than an error, because the caller asked
	// for a location and the build chose what answers.
	Path string
	// Store injects an already-open ordered backend. The caller retains ownership:
	// construction error paths do not Close it. Mutually exclusive with
	// BackendName, Path and CompressionThreshold.
	Store  Store
	Schema schema.SchemaProvider // Optional schema for validation
	// RefuseUnknownAttributes makes a query or pull that names an attribute the
	// schema does not declare an error, where the database otherwise answers it
	// with no tuple. It requires Schema: with none to check against, the
	// constructor returns an error.
	RefuseUnknownAttributes bool
	AnnotationHandler       annotations.Handler     // Handler for query tracing; the database hands every event to it and to the Handler PlannerOptions carries
	ReplicaID               uint64                  // For CRDT mode: 0 = auto-generate random; non-zero = use specified. Ignored for existing DBs.
	DisableCache            bool                    // Disable EA cache; queries resolve directly from storage
	PlannerOptions          *planner.PlannerOptions // Optional override for default planner options
	CompressionThreshold    int                     // Compress string/[]byte values >= this size (0 = default 512; -1 to disable). Configures the encoder this constructor builds.
	// RemovePathOnClose hands Path to the database to own: Close removes the
	// directory once the store has released its files. For a caller that made
	// scratch space to hold data that came from somewhere else — a dump loaded
	// into a persistent backend — rather than a location the data lives at.
	// On a constructor error no database holds the path, and the caller that
	// made the directory removes it, as db.Open does at its error return.
	RemovePathOnClose bool
	// Dump, when set, is an exported dump the constructor loads into the store
	// it opened before it declares Schema, so the declarations the dump
	// carries are read back and never declared a second time, and a
	// load-and-export round trip under the writer's schema is byte-stable.
	// The format is read from the dump's own bytes (replayDump).
	Dump io.ReadSeeker
}

DatabaseOptions configures database creation.

Where the data lives is one question with one answer: either BackendName and Path name a backend for this constructor to open, or Store supplies one the caller opened already. Setting both is an error rather than a precedence rule — an injected store carries its own backend, location and encoder, so a precedence rule would silently discard whichever of those fields the caller meant something by.

type ElementID added in v0.6.0

type ElementID = datalog.ElementID

ElementID is a type alias for datalog.ElementID. The core type is defined in the datalog package so it can be used as a value type. This package adds key-encoding methods for storage.

func DecodeElementID added in v0.6.0

func DecodeElementID(buf []byte) ElementID

DecodeElementID reads 16 bytes from storage key format. Reverses the bitwise NOT applied during EncodeElementIDForKey().

func Max added in v0.6.0

func Max(a, b ElementID) ElementID

Max returns the larger of two ElementIDs according to the total order.

func Min added in v0.6.0

func Min(a, b ElementID) ElementID

Min returns the smaller of two ElementIDs according to the total order.

type EncodedRun added in v0.15.2

type EncodedRun struct {
	Start, End []byte
	Membership runMembership
	Fixed      fixedComponents
}

EncodedRun is a ScanBound projected onto binary keys: the range a scan walks, the rule deciding which keys inside it the bound actually names, and the components the bound's prefix fixes in every key the run yields.

type Entity

type Entity [20]byte

Entity represents a unique identifier for an entity (20-byte SHA1 hash)

func NewEntity

func NewEntity(id string) Entity

NewEntity creates an entity ID from a string identifier

func (Entity) String

func (e Entity) String() string

String returns the entity as hex string (first 8 bytes)

type ExportOptions added in v0.11.0

type ExportOptions struct {
	Compressed bool // Use #lzj tagged literals for large string values
	// SkipEntity, when non-nil, is called for each datom's entity Identity.
	// If it returns true, the datom is omitted from the export.
	// This enables filtering out entire entity classes (e.g., task metadata)
	// without a separate pass over the database.
	SkipEntity func(datalog.Identity) bool
}

ExportOptions configures EDN export behavior.

type IndexType

type IndexType uint8

IndexType represents different index orderings

const (
	EAVT IndexType = iota // Entity-Attribute-Value-Tx (for cardinality-many: group by V)
	EATV                  // Entity-Attribute-Tx-Value (for cardinality-one: first = current)
	AEVT                  // Attribute-Entity-Value-Tx
	AETV                  // Attribute-Entity-Tx-Value (for A-primary CRDT: first = current)
	ATEV                  // Attribute-Tx-Entity-Value (for AsOf-by-attribute: A-bound + Tx-bound scans seek straight to the transaction)
	AVET                  // Attribute-Value-Entity-Tx
	VAET                  // Value-Attribute-Entity-Tx
	TAEV                  // Tx-Attribute-Entity-Value (for clock recovery, audit log)
)

func (IndexType) String added in v0.15.2

func (i IndexType) String() string

String names the index. It is the one rendering of an IndexType, so %v in an error or an annotation reads "AVET" rather than "5" — the numbering is an implementation detail of the key prefix and means nothing to a reader.

type Iterator

type Iterator interface {
	Next() bool
	Datom() (*datalog.Datom, error)
	Close() error

	// Seek narrows iteration to the run the bound names, and to all of it: the
	// bound repositions the cursor at or after its start, and the same bound
	// supplies where the run ends and the membership rule governing what lies
	// between. An implementation adopting only the start walks past the sought
	// bound into whatever the scan's wider range still holds, and leaves the
	// caller to work out where its own run ended from the encoded key — which is
	// how key-layout arithmetic gets back above this seam.
	//
	// A bound that cannot be encoded is recorded as the iterator's sticky error
	// rather than dropped: Seek has no return, so Error() is where the failure
	// surfaces.
	Seek(bound ScanBound)

	// ElementID returns the transaction ElementID of the current entry.
	// This is more efficient than Datom() when only the ElementID is needed
	// (e.g., for cache freshness checks, CRDT version comparisons).
	// Returns zero ElementID if iterator is not positioned on a valid entry.
	ElementID() datalog.ElementID

	// Error returns the first error encountered during iteration.
	// Implementations that cannot error return nil. Wrapping iterators
	// propagate from their inner iterator (return the first non-nil
	// between the outer's own error and inner.Error()).
	Error() error

	// Scanned reports how many datoms this iterator has taken in from the
	// index so far — intake, before any narrowing this iterator performs.
	// A key the bound's membership rule rejects is counted: the range covered
	// it and the scan paid to look at it.
	//
	// This is what makes narrowing auditable. Against the consumer's own count
	// of what survived, the ratio is the amplification the index charged, and
	// it is the only way to see whether a bound narrowed anything. Required
	// rather than optional for that reason: an absent count is indistinguishable
	// from a scan that read nothing and from instrumentation that was never
	// wired, which is the failure this seam's annotations exist to prevent.
	//
	// Wrapping iterators delegate — a wrapper reads no index of its own.
	Scanned() int
}

Iterator provides sequential access to datoms.

Iteration pattern:

for iter.Next() {
    d, err := iter.Datom()
    if err != nil { handle }
    // use d
}
if err := iter.Error(); err != nil {
    // iteration was aborted by an internal failure
}

Error() must be checked after Next() returns false. A nil result indicates normal exhaustion; non-nil indicates that iteration aborted partway through (storage scan failure, sub-scan failure inside a wrapping iterator, or a decode failure retained as a sticky Error after Next() returned false).

Workspace contract: Datom() returns the iterator's current datom workspace until Next, Seek, or Close. Callers that retain values past those calls must copy. Scan and ScanKeysOnly share this contract.

func NewKeyOnlyIterator

func NewKeyOnlyIterator(store *BadgerStore, index IndexType, run EncodedRun) (Iterator, error)

func OpenKeyScan added in v0.15.2

func OpenKeyScan(reader StoreReader, report *scanReport, bound ScanBound) (Iterator, error)

OpenKeyScan is OpenScan for reads that need no stored values.

func OpenScan added in v0.15.2

func OpenScan(reader StoreReader, report *scanReport, bound ScanBound) (Iterator, error)

OpenScan opens a scan and attaches it to the report that accounts for it. Callers use this rather than the reader directly, which is what makes reporting a property of acquiring a scan: one way in, and it carries the accounting.

It takes the reader rather than hanging off the matcher because the matcher is not the only thing that holds one — a pull batch, a prefetch and the write path each reach the store by their own field, and an opener only the matcher could call would leave every one of them with the reader as its shortest route. TestScanAcquisitionGoesThroughAReport enforces that there is no such route, which is only honest if this is reachable from wherever a scan is.

The scan is unconditional because it is the work the arm asked for. Only the recording is tested, immediately before it, so a nil report costs the comparison and nothing else — no wrapper, no allocation.

type JoinStrategy

type JoinStrategy int

JoinStrategy represents different join strategies for pattern matching with bindings

const (
	// HashJoinScan builds a hash set from the bindings and probes it against
	// the keys' datoms: sought one by one for a binding at or below
	// executor.SmallBindingBoundary, or read by one scan of the pattern's
	// extent above it (matchWithHashJoin).
	HashJoinScan JoinStrategy = iota

	// MergeJoin merges sorted streams (good for large sets >50% selectivity)
	MergeJoin
)

func (JoinStrategy) String

func (js JoinStrategy) String() string

type KeyOnlyIterator

type KeyOnlyIterator struct {
	*BadgerIterator
	// contains filtered or unexported fields
}

KeyOnlyIterator wraps a BadgerIterator to decode datoms from keys.

func (*KeyOnlyIterator) Close added in v0.15.0

func (i *KeyOnlyIterator) Close() error

func (*KeyOnlyIterator) Datom

func (i *KeyOnlyIterator) Datom() (*datalog.Datom, error)

func (*KeyOnlyIterator) Error added in v0.11.2

func (i *KeyOnlyIterator) Error() error

func (*KeyOnlyIterator) Key added in v0.15.0

func (i *KeyOnlyIterator) Key() []byte

func (*KeyOnlyIterator) Next

func (i *KeyOnlyIterator) Next() bool

Next advances to the next key the run holds, stepping over the keys its byte range over-covers.

func (*KeyOnlyIterator) Seek added in v0.15.0

func (i *KeyOnlyIterator) Seek(bound ScanBound)

type LamportClock added in v0.6.0

type LamportClock struct {
	// contains filtered or unexported fields
}

LamportClock implements Lamport's logical clock algorithm.

IMPORTANT: One clock per Database instance, shared across all attributes. This provides global temporal ordering for point-in-time queries, causal debugging, and as-of functionality.

Rules (from Lamport 1978):

  1. Before each local event: L = L + 1
  2. When sending: include L in message
  3. When receiving message with L_remote: L = max(L, L_remote) + 1

This ensures: if A causally precedes B, then L(A) < L(B)

Thread Safety: All methods are safe for concurrent use. The counter is updated using atomic operations, ensuring correctness under high write contention from multiple goroutines.

func NewLamportClock added in v0.6.0

func NewLamportClock(replicaID uint64) *LamportClock

NewLamportClock creates a clock for the given replica. The clock starts at 0; the first call to Next() returns Lamport=1.

func (*LamportClock) Current added in v0.6.0

func (c *LamportClock) Current() uint64

Current returns the current clock value without incrementing. This is the Lamport timestamp that would be returned by the PREVIOUS Next() call. The NEXT Next() call will return Current() + 1.

Thread-safe: uses atomic load.

func (*LamportClock) Next added in v0.6.0

func (c *LamportClock) Next() ElementID

Next generates the next ElementID for a local event. Implements rule 1: L = L + 1 before each event.

This is the primary method for generating timestamps during writes. Each call returns a unique, monotonically increasing ElementID.

Thread-safe: uses atomic increment.

func (*LamportClock) Peek added in v0.6.0

func (c *LamportClock) Peek() ElementID

Peek returns what the next ElementID would be without consuming it. Equivalent to ElementID{Lamport: Current() + 1, ReplicaID: ReplicaID()}.

Note: In a concurrent environment, the actual next ElementID may differ if another goroutine calls Next() between Peek() and your Next() call.

func (*LamportClock) ReadPoint added in v0.16.2

func (c *LamportClock) ReadPoint() ElementID

ReadPoint returns the greatest ElementID at the clock's current Lamport time. Every ElementID drawn so far, on any replica, is at or below it, and every ElementID this clock draws later is above it.

Thread-safe: uses atomic load.

func (*LamportClock) Receive added in v0.6.0

func (c *LamportClock) Receive(remote ElementID)

Receive updates the clock when receiving data from another node. Implements rule 3: L = max(L, L_remote) + 1

This is the critical Lamport property: receiving a message is an event, and the receiver's clock must be greater than both its previous value AND the sender's timestamp.

Use this when merging data from another replica. After Receive(), subsequent Next() calls will return ElementIDs greater than the remote.

Thread-safe: uses compare-and-swap loop.

func (*LamportClock) ReplicaID added in v0.6.0

func (c *LamportClock) ReplicaID() uint64

ReplicaID returns this clock's replica identifier. This is set at construction time and never changes.

func (*LamportClock) Restore added in v0.6.0

func (c *LamportClock) Restore(maxSeen ElementID)

Restore sets the clock to a known value without incrementing. Used ONLY for restoring state on database reopen, NOT for receiving messages.

Unlike Receive(), Restore() does not add 1. This is intentional: - Receive() models a causal event (message receipt) → must advance - Restore() models state recovery (not a causal event) → just set value

After Restore(maxSeen), the next Next() call returns maxSeen.Lamport + 1.

Thread-safe: uses compare-and-swap loop.

type MemoryTreeStore added in v0.15.2

type MemoryTreeStore struct {
	// contains filtered or unexported fields
}

MemoryTreeStore is the typed in-memory backend: eight sorted trees of datalog.Datom, one per index order, published together as a version.

It is the other half of a pair, and the in-process backend the engine opens by default. BadgerEmulatorStore emulates Badger — it holds the same binary index keys a disk store would, so a process with no disk pays BinaryKeyEncoder on every assert and scan. This one compares typed values directly and encodes only at the JDZL and EDN boundaries. Both satisfy Store, so the backend contract runs against both and any divergence is a test failure rather than a discovery.

See MEMORY_DATOM_INDEXES.md. The one place the two backends genuinely differ is scan coverage for values whose encodings are prefixes of one another: BUG_V_PAYLOAD_NOT_PREFIX_FREE.

func NewMemoryTreeStore added in v0.15.2

func NewMemoryTreeStore(encoder *BinaryKeyEncoder) *MemoryTreeStore

NewMemoryTreeStore returns an empty typed store.

func (*MemoryTreeStore) Assert added in v0.15.2

func (s *MemoryTreeStore) Assert(datoms []datalog.Datom) error

Assert adds datoms. Every datom enters all eight orders in one batch, so the version that publishes them is indivisible.

func (*MemoryTreeStore) AssertEach added in v0.15.2

func (s *MemoryTreeStore) AssertEach(produce func(add func(*datalog.Datom) error) error) error

AssertEach adds what produce yields to the open batch, leaving the batch open for the next call; FinishBatch publishes it, and until then the run is invisible: readers hold the previously published version.

A batch over an empty base gathers its datoms — duplicate-free, the dump format's own property, trusted rather than swept for — and builds every index in one sorted pass at FinishBatch, leaves packed full. A batch over existing data inserts per datom into the version being built, paying one copy-on-write generation for the whole run.

func (*MemoryTreeStore) BeginTx added in v0.15.2

func (s *MemoryTreeStore) BeginTx() (StoreTx, error)

func (*MemoryTreeStore) Close added in v0.15.2

func (s *MemoryTreeStore) Close() error

func (*MemoryTreeStore) DatomsAfter added in v0.15.2

func (s *MemoryTreeStore) DatomsAfter(eid datalog.ElementID) (after []datalog.Datom, afterErr error)

DatomsAfter returns every datom committed after eid, in TAEV order.

TAEV orders Tx descending, so the walk starts at the newest and stops at the first entry not after eid — everything past that point is older still.

func (*MemoryTreeStore) DeleteDatoms added in v0.15.2

func (s *MemoryTreeStore) DeleteDatoms(datoms []datalog.Datom) (int, error)

DeleteDatoms removes exactly the datoms given — matched on every component, Tx included — and reports how many were there.

func (*MemoryTreeStore) Encoder added in v0.15.2

func (s *MemoryTreeStore) Encoder() *BinaryKeyEncoder

func (*MemoryTreeStore) FinishBatch added in v0.15.2

func (s *MemoryTreeStore) FinishBatch() error

FinishBatch publishes an open batch, and is what makes a run of AssertEach calls visible. With no batch open there is nothing to publish and it says so by doing nothing.

func (*MemoryTreeStore) Fork added in v0.15.4

func (s *MemoryTreeStore) Fork(ceiling datalog.ElementID) (Store, error)

Fork returns a branch store holding what this store's published version held as of ceiling, and writing independently of this store. It starts from the published version and deletes from its own trees every datom written past ceiling, the rewind TruncateTo performs. Neither store sees what the other commits afterward: a version is immutable, and a builder copies every node it did not create, so the two share each node until one of them writes beneath it. A batch AssertEach left open is not published, and the fork does not hold it.

The fork starts with no metadata. The replica id is a store's write identity, and a branch writes as a replica of its own, assigned by the Database that opens it. The encoder is shared: it is configuration, read and never written.

func (*MemoryTreeStore) GetMetadataUint64 added in v0.15.2

func (s *MemoryTreeStore) GetMetadataUint64(key string) (uint64, bool, error)

GetMetadataUint64 and SetMetadataUint64 sit outside the version. The only key is the replica id, written and read at database open, so it never reaches a query path and cannot participate in a torn snapshot.

func (*MemoryTreeStore) MaxElementID added in v0.15.2

func (s *MemoryTreeStore) MaxElementID() (datalog.ElementID, error)

func (*MemoryTreeStore) MaxTxForEntity added in v0.15.2

func (s *MemoryTreeStore) MaxTxForEntity(e datalog.Identity) (datalog.ElementID, bool, error)

func (*MemoryTreeStore) NewReadSession added in v0.15.2

func (s *MemoryTreeStore) NewReadSession() (ReadSession, error)

NewReadSession retains the current version. Every read through the session walks that one state however many batches commit afterward — the snapshot is a held pointer, not a lock and not a copy.

func (*MemoryTreeStore) Retract added in v0.15.2

func (s *MemoryTreeStore) Retract(datoms []datalog.Datom) error

Retract physically removes every stored datom sharing an (E, A, V) with one of those given, whatever its Tx.

This is deletion, not a CRDT Remove: nothing is tombstoned and the history is gone. Visible deletion in a user transaction is an OpCRDTRemove datom, which Assert writes like any other; truncate removes through DeleteDatoms. See BUG_RETRACT_NAMES_TWO_OPPOSITE_OPERATIONS on the name.

func (*MemoryTreeStore) Scan added in v0.15.2

func (s *MemoryTreeStore) Scan(bound ScanBound) (Iterator, error)

func (*MemoryTreeStore) ScanKeysOnly added in v0.15.2

func (s *MemoryTreeStore) ScanKeysOnly(bound ScanBound) (Iterator, error)

ScanKeysOnly is Scan. The distinction exists for a backend whose values live apart from its keys; a tree holds whole datoms, so there is nothing to skip fetching and nothing to defer.

func (*MemoryTreeStore) SetMetadataUint64 added in v0.15.2

func (s *MemoryTreeStore) SetMetadataUint64(key string, value uint64) error

type ParseCache added in v0.9.0

type ParseCache struct {
	// contains filtered or unexported fields
}

ParseCache caches the parsed form of the text a query is given: a query string's *query.Query and rule text's *query.RuleSet. Unlike PlanCache, no TTL is needed — the mapping from text to its parsed form is immutable. The two arms hold at most maxSize entries together, and a store into a full cache evicts the oldest entry of either arm, the one stored earliest in the cache's sequence of stores.

func NewParseCache added in v0.9.0

func NewParseCache(maxSize int) *ParseCache

NewParseCache creates a new parse cache.

func (*ParseCache) Clear added in v0.9.0

func (c *ParseCache) Clear()

Clear removes every cached query and rule set.

func (*ParseCache) Get added in v0.9.0

func (c *ParseCache) Get(queryString string) (*query.Query, bool)

Get retrieves a cached *query.Query for the given query string.

func (*ParseCache) GetRules added in v0.16.0

func (c *ParseCache) GetRules(text string) (*query.RuleSet, bool)

GetRules retrieves the cached *query.RuleSet for the given rule text.

func (*ParseCache) Set added in v0.9.0

func (c *ParseCache) Set(queryString string, q *query.Query)

Set stores a parsed *query.Query for the given query string, under the cache's one bound.

func (*ParseCache) SetRules added in v0.16.0

func (c *ParseCache) SetRules(text string, rs *query.RuleSet)

SetRules stores a parsed *query.RuleSet for the given rule text, under the cache's one bound.

func (*ParseCache) Stats added in v0.9.0

func (c *ParseCache) Stats() (hits, misses int64, size int)

Stats returns cache statistics: the hits and misses across both arms, and the entries both hold.

type PatternMatcher added in v0.15.2

type PatternMatcher struct {
	// contains filtered or unexported fields
}

PatternMatcher implements executor.PatternMatcher over a storage Store.

func NewPatternMatcher added in v0.15.2

func NewPatternMatcher(store Store) *PatternMatcher

NewPatternMatcher creates a new pattern matcher for a storage backend. The tuple-builder cache initializes lazily on first use; Database-minted matchers arrive with the database's shared cache already set.

func NewPatternMatcherWithOptions added in v0.15.2

func NewPatternMatcherWithOptions(store Store, opts executor.ExecutorOptions) *PatternMatcher

NewPatternMatcherWithOptions creates a new pattern matcher with specific options.

The handler comes in with the options and is materialized into a field here: the scan paths read it per datom, and the relations this matcher builds carry the options on to whatever consumes them. Both are the same value fixed at construction — a matcher's observer never changes over its life.

func (*PatternMatcher) AsOf added in v0.15.2

AsOf is a copy of this matcher reading the view as of target: each attribute resolved over the datoms at or before it.

func (*PatternMatcher) AttachReadSession added in v0.15.2

func (m *PatternMatcher) AttachReadSession(session ReadSession, readPoint datalog.ElementID)

AttachReadSession routes every subsequent storage read this matcher performs through the session, so the whole query observes one snapshot. readPoint is the read point the caller took before it opened the session: cache reads bound themselves to it (see cacheBound), so the shared EA cache serves this matcher only entries on whose key nothing arrived after it, and stores its resolutions only when nothing arrived after it. The caller owns the session's lifecycle; the matcher only reads through it.

func (*PatternMatcher) AttributeLookup added in v0.16.3

func (m *PatternMatcher) AttributeLookup(attr datalog.Keyword) (query.AttributeLookup, error)

AttributeLookup returns the lookup of attr, which answers at each entity what LookupAttribute answers for that entity and attr.

func (*PatternMatcher) CanFuseAttributeFetch added in v0.15.2

func (m *PatternMatcher) CanFuseAttributeFetch(_ query.Symbol, attr datalog.Keyword) bool

CanFuseAttributeFetch reports whether a match of attr on the matcher answers, per entity, the one value LookupAttribute answers: true in the latest and as-of views for an attribute the matcher resolves cardinality-one, whether its schema defines it so or holds no definition of it, since LookupAttribute and CRDTResolvingIterator both resolve an attribute with no definition by LWW. On an as-of matcher both read the datoms at or before its target. False for a cardinality-many attribute, whose match answers each member as a tuple of its own, for a cardinality-vector attribute, and on a History matcher, whose match answers every datom. The matcher is one source and answers for itself under any name, so the source is not read.

func (*PatternMatcher) ForceJoinStrategy added in v0.15.2

func (m *PatternMatcher) ForceJoinStrategy(strategy *JoinStrategy)

ForceJoinStrategy overrides the join strategy selection for testing Pass nil to restore default behavior

func (*PatternMatcher) GetCardinality added in v0.15.2

func (m *PatternMatcher) GetCardinality(attr datalog.Keyword) datalog.Keyword

GetCardinality returns the cardinality an attribute resolves under: the one its definition declares, and CardinalityOne for an attribute the schema does not declare or a matcher with no schema.

func (*PatternMatcher) History added in v0.15.2

func (m *PatternMatcher) History() *PatternMatcher

History is a copy of this matcher reading every datom as written, with no resolution: every version is visible, superseded and withdrawn ones included.

func (*PatternMatcher) LookupAllAttributes added in v0.15.2

func (m *PatternMatcher) LookupAllAttributes(
	entity datalog.Identity,
	attr datalog.Keyword,
) (values []interface{}, lookupErr error)

LookupAllAttributes retrieves all values of a cardinality-many attribute for an entity. Returns all matching values, or empty slice if none found.

func (*PatternMatcher) LookupAttribute added in v0.15.2

func (m *PatternMatcher) LookupAttribute(entity datalog.Identity, attr datalog.Keyword) (interface{}, bool, error)

LookupAttribute answers an entity's value of an attribute, the lookup the database functions get-else, missing? and get-some read. It answers (value, true) when the attribute holds a value and (nil, false) otherwise. A cardinality-one attribute, and one the schema holds no definition of, answers its current value by LWW. A cardinality-many attribute answers its current members as one value, in no fixed order. A cardinality-vector attribute answers its vector, typed by the schema's value type, once the vector has been written, an emptied vector included.

func (*PatternMatcher) LookupSource added in v0.16.2

LookupSource implements executor.EntityLookupMatcher: the matcher is one source, so it is the lookup of the source under any name.

func (*PatternMatcher) Match added in v0.15.2

func (m *PatternMatcher) Match(q *query.Query, bindings executor.Relations) (executor.Relation, error)

Match implements PatternMatcher.Match - returns a Relation directly

func (*PatternMatcher) MatchWithConstraints added in v0.15.2

func (m *PatternMatcher) MatchWithConstraints(
	q *query.Query,
	bindings executor.Relations,
	constraints []executor.StorageConstraint,
) (executor.Relation, error)

MatchWithConstraints implements predicate-aware matching with storage-level filtering

func (*PatternMatcher) PrefetchEntities added in v0.15.2

func (m *PatternMatcher) PrefetchEntities(entities []datalog.Identity) error

PrefetchEntities loads all attributes for a set of entities into the EA cache in a single-pass EATV scan per entity. Entities are sorted by raw hash byte order (= disk order) for efficient forward-only iteration through BadgerDB.

CRDT resolution happens inline as attribute boundaries are crossed during iteration — no second pass. Each (E, A) group is dispatched to Cache.PopulateFromDatoms which uses the canonical Resolve*FromDatoms functions.

See ALGEBRA.md "Rule 6: Entity Prefetch into EA Cache" for design rationale.

The first storage failure aborts the prefetch and returns it: an error from the store means the store is broken, and callers fail the query — prefetch being an optimization does not make store failures ignorable. Groups cached before the failure hold valid data.

func (*PatternMatcher) RefusesUnknownAttributes added in v0.16.0

func (m *PatternMatcher) RefusesUnknownAttributes(_ query.Symbol) bool

RefusesUnknownAttributes implements executor.AttributeVocabulary: the matcher is one source and answers for itself under any name, refusing on a database opened with RefuseUnknownAttributes.

func (*PatternMatcher) ResolveAddWins added in v0.15.2

func (m *PatternMatcher) ResolveAddWins(e datalog.Identity, a datalog.Keyword, report *scanReport) (map[string]any, map[string]datalog.ElementID, datalog.ElementID, bool, error)

ResolveAddWins returns the current set members for cardinality-many, each under its memberKey, and each member's newest assertion's Tx under the same key, through resolveAddWinsSet.

func (*PatternMatcher) ResolveLWW added in v0.15.2

func (m *PatternMatcher) ResolveLWW(e datalog.Identity, a datalog.Keyword, def *schema.AttributeDefinition, report *scanReport) (v any, tx datalog.ElementID, version datalog.ElementID, present bool, resolveErr error)

ResolveLWW returns the current value for cardinality-one (highest ElementID wins), given def, the attribute's definition, nil for an attribute the schema does not declare.

When def declares no unique, and in History, it uses the EATV first-entry shortcut (highest Tx due to descending sort). History reads raw assertions with no resolution, so the unique walk would produce values corresponding to no datom. When def declares a unique and the matcher reads resolved state, it delegates to walkUniqueEntityValue for walk-based resolution: the returned value is the entity's first non-superseded assertion, which may fall back to an older entry if the latest has been claimed by another entity.

tx is the Tx of the datom the pair resolves to, the assertion the value was read from or the withdrawal that left it nil, and is what a pattern's transaction position reads. version is the highest ElementID the resolution read, the datom's own Tx under the plain rule and, under the walk, other entities' superseding claims included; it is what the cache's freshness reads. A unique pair no assertion of which survives the rule resolves to no datom, so its tx is the zero ElementID.

func (*PatternMatcher) ResolveRGA added in v0.15.2

func (m *PatternMatcher) ResolveRGA(e datalog.Identity, a datalog.Keyword, report *scanReport) ([]any, []datalog.ElementID, datalog.ElementID, bool, error)

ResolveRGA returns the ordered vector for cardinality-vector Returns (values, positionToElementID, maxElementID, present, error)

func (*PatternMatcher) SetCache added in v0.15.2

func (m *PatternMatcher) SetCache(c *Cache)

SetCache sets the CRDT resolution cache for O(1) access to resolved views. When cache is set, LookupAttribute() and related methods check the cache before scanning storage, providing O(1) access for cache hits.

func (*PatternMatcher) SetSchema added in v0.15.2

func (m *PatternMatcher) SetSchema(s schema.SchemaProvider)

SetSchema sets the schema for cardinality-aware index selection. When schema is set, the matcher uses EATV for cardinality-one attributes and EAVT for cardinality-many attributes (for add-wins resolution).

func (*PatternMatcher) TypeDefault added in v0.15.2

func (m *PatternMatcher) TypeDefault(attr datalog.Keyword, defaultVal interface{}) interface{}

TypeDefault converts a default value to match the attribute's schema type. For vector attributes with TypeString, converts []interface{} to []string, etc. This implements query.TypedDefaulter, which get-else asks of the lookup of the source it reads.

func (*PatternMatcher) UnknownAttributes added in v0.16.0

func (m *PatternMatcher) UnknownAttributes(_ query.Symbol, attrs []datalog.Keyword) error

UnknownAttributes implements executor.AttributeVocabulary. On a database opened with RefuseUnknownAttributes it refuses every attribute of the roster that the store does not declare and the engine does not reserve, joined in order. On any other database it refuses nothing, and the roster is not read. The matcher is one source and answers for itself under any name.

type QueryOption added in v0.5.6

type QueryOption func(*queryOptions)

QueryOption configures query execution.

func WithContext added in v0.16.0

func WithContext(ctx context.Context) QueryOption

WithContext cancels the query with ctx. A query whose ctx is done is refused at entry, before any work, and a rule evaluation whose ctx is done stops at its next iteration. ctx's error is the error Query, QueryInto, QueryOneInto and Analyze return. Explain and ExplainAlgebra accept the option and plan without executing.

Example:

ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
rel, err := db.Query(q, storage.WithContext(ctx))

func WithSources added in v0.5.6

func WithSources(sources map[query.Symbol]executor.PatternMatcher) QueryOption

WithSources adds additional data sources for multi-source queries. The default source ($) is always the database itself.

type RGAElement added in v0.6.0

type RGAElement struct {
	ID        ElementID  // This element's unique identifier
	Value     any        // The actual value
	AfterRef  ElementID  // What this element was inserted after (HEAD = zero value)
	Tombstone *ElementID // Non-nil if deleted, value is the delete timestamp
}

RGAElement represents an element in a Replicated Growable Array (RGA). RGA is a CRDT for ordered sequences that supports concurrent inserts while maintaining a deterministic final order across all replicas.

Each element has:

  • ID: Unique identifier for this element (from Lamport clock)
  • Value: The actual data stored at this position
  • AfterRef: The element ID this was inserted after (HEAD for first elements)
  • Tombstone: If non-nil, this element has been deleted (the value is when it was deleted)

RGA maintains order through the AfterRef chain:

  • Elements form a tree rooted at HEAD
  • Siblings (same AfterRef) are sorted by their ID
  • DFS traversal produces the final ordered list

func DecodeRGAElement added in v0.6.0

func DecodeRGAElement(id ElementID, data []byte) (RGAElement, error)

DecodeRGAElement deserializes an RGAElement from storage. The ID is passed in from the datom's Tx field.

func (RGAElement) IsDeleted added in v0.6.0

func (e RGAElement) IsDeleted() bool

IsDeleted returns true if this element has been tombstoned.

func (RGAElement) String added in v0.6.0

func (e RGAElement) String() string

String returns a human-readable representation of the element.

type RGAElementWithPosition added in v0.6.0

type RGAElementWithPosition struct {
	Element  RGAElement
	Position int
}

RGAElementWithPosition pairs an element with its position in the reconstructed list. Used for building position indexes.

func ReconstructRGAWithIDs added in v0.6.0

func ReconstructRGAWithIDs(elements []RGAElement) []RGAElementWithPosition

ReconstructRGAWithIDs builds an ordered list with element IDs preserved. This is used for building position indexes (position -> ElementID mapping).

Returns a slice of (ElementID, Value) pairs in the correct order. Tombstoned elements are excluded from the result but kept in the tree structure so their children remain reachable.

type RGAStats added in v0.6.0

type RGAStats struct {
	TotalElements   int // Total including tombstones
	LiveElements    int // Non-tombstoned elements
	TombstoneCount  int // Number of tombstoned elements
	MaxID           datalog.ElementID
	UniqueAfterRefs int // Number of unique insertion points
}

RGAStats provides statistics about an RGA collection.

func ComputeRGAStats added in v0.6.0

func ComputeRGAStats(elements []RGAElement) RGAStats

ComputeRGAStats calculates statistics for debugging and monitoring.

type ReadSession added in v0.15.1

type ReadSession interface {
	StoreReader
	Close() error
}

ReadSession is a consistent read view over the store: every read through the session observes one snapshot of the data, regardless of writes committed after the session opened. A query executes all its reads through one session, so it can never observe two different states mid-execution.

Close releases the snapshot. Sessions carry a finalizer backstop, but deterministic release comes from the query result: the session closes when the result relation is exhausted or closed, whichever comes first.

type Resolver

type Resolver interface {
	ResolveEntity(Entity) datalog.Identity
	ResolveAttribute(Attribute) datalog.Keyword
	ResolveTx(Tx) datalog.ElementID
}

Resolver provides mappings from storage to user representations

type ReuseStrategy

type ReuseStrategy struct {
	Type     ReuseType
	Position int       // Which position is changing (0=E, 1=A, 2=V, 3=T)
	Index    IndexType // Which index to use
	// NeedsValidation is set when the run is narrowed to the binding's V or T
	// and so holds no pair's group whole: each datom it emits is a candidate
	// validated against its pair's resolved datom (matchWithVValidation).
	NeedsValidation bool
	BoundA          any // The bound A value (for validation lookups)
}

ReuseStrategy describes how iterator reuse can be applied

type ReuseType

type ReuseType int

ReuseType indicates the type of iterator reuse strategy

const (
	NoReuse ReuseType = iota
	SinglePositionReuse
	MultiPositionReuse
)

func (ReuseType) String

func (rt ReuseType) String() string

type ScanBound added in v0.15.2

type ScanBound struct {
	Index  IndexType
	Prefix []datalog.Value
}

ScanBound names the datoms of one index whose leading components — in that index's component order — equal the bound's values. The k-th Prefix element binds the k-th component of Index's order, so elements carry no position tag of their own. An empty Prefix names the whole index.

This is a logical set, not a byte range, and the difference is load-bearing for any backend that projects the bound onto keys. A V payload carries no length, so the keys for "abcd" sort inside the range for "abc" interleaved with them, and no choice of endpoints separates the two. Narrowing to the datoms the bound actually names is the backend's obligation, not the caller's: a scan yields exactly these datoms, and how a backend achieves that is not seam vocabulary. See EncodedRun and runMembership for how the binary-key backends discharge it.

Bound components are always leading components, never an interior selection. Every index layout places Op last and AfterRef (when present) immediately before it, so neither is ever a bound component; the orderable positions are exactly E, A, V and Tx, and binding one requires binding every position ahead of it.

Prefix elements are ordinary datalog values — Identity for E, Keyword for A, any domain value for V, ElementID for Tx — so a bound introduces no value kinds beyond the closed domain.

type SetEntry added in v0.6.0

type SetEntry struct {
	Value interface{}
	Op    uint8
}

SetEntry represents a value in a cardinality-many set with its operation. Used for CRDT add-wins set semantics.

func DecodeSetEntry added in v0.6.0

func DecodeSetEntry(data []byte) (SetEntry, error)

DecodeSetEntry decodes a set entry from storage format. Returns the entry and any error encountered.

func (SetEntry) IsAdd added in v0.6.0

func (e SetEntry) IsAdd() bool

IsAdd returns true if this is an add operation

func (SetEntry) IsRemove added in v0.6.0

func (e SetEntry) IsRemove() bool

IsRemove returns true if this is a remove operation

func (SetEntry) String added in v0.6.0

func (e SetEntry) String() string

String returns a human-readable representation

type SetResolutionResult added in v0.6.0

type SetResolutionResult struct {
	// Members holds each member's original Go value under its memberKey, the
	// one key per value under datalog.ValuesEqual, so a lookup keyed through
	// memberKey finds a member by any representation of its value.
	// Iteration: for _, v := range Members
	Members map[string]interface{}

	// MemberTx holds, under each member's memberKey, the Tx of the member's
	// newest assertion, the datom the member resolves to. A pattern's
	// transaction position binds it and a bound T is compared with it.
	MemberTx map[string]datalog.ElementID

	// Present reports whether the (E, A) carries any datom at all within this
	// matcher's temporal bound, and it is the resolved answer to whether the
	// attribute exists.
	//
	// A set that was never written and one whose every member has been removed
	// both resolve to zero members, so Members cannot separate them; only the
	// presence of datoms can, and the two are different states — the first has
	// no value, the second has the empty set. Resolved alongside the members
	// rather than counted afterwards: a statistic describes a resolution, it
	// does not constitute one.
	Present bool

	// MaxElementID is the highest ElementID seen during resolution
	// Used for cache freshness tracking
	MaxElementID datalog.ElementID

	// Bound is the run this resolution walked. The pattern arm announces it,
	// and it travels back rather than being rebuilt there so the announced run
	// and the walked run are the same value and cannot drift.
	Bound ScanBound
}

SetResolutionResult contains the resolved set membership and metadata

type SnapshotInfo added in v0.13.3

type SnapshotInfo struct {
	Name    string
	At      datalog.ElementID // captured high-water point (the AsOf read point)
	Created time.Time
}

SnapshotInfo is the decoded snapshot marker.

type StorageDatom

type StorageDatom struct {
	E        Entity         // Entity identifier (20 bytes)
	A        Attribute      // Attribute identifier (32 bytes)
	V        datalog.Value  // The actual value (unbounded, stored last)
	Tx       Tx             // Transaction/time identifier (16 bytes = ElementID)
	Op       datalog.CRDTOp // CRDT operation (0=none, 1=add, 2=remove, 3=rga-insert, 4=rga-tombstone)
	AfterRef Tx             // RGA position reference (16 bytes, only used when Op.HasAfterRef() is true)
}

StorageDatom is the internal storage representation It uses fixed-size byte arrays for efficient storage and indexing

func StorageDatomFromBytes

func StorageDatomFromBytes(data []byte) (*StorageDatom, error)

StorageDatomFromBytes deserializes a datom from bytes

func ToStorageDatom

func ToStorageDatom(d datalog.Datom) StorageDatom

ToStorageDatom converts a user-facing datom to storage representation

func (StorageDatom) Bytes

func (d StorageDatom) Bytes() []byte

Bytes returns the serialized form of the storage datom Format: E(20) + A(32) + Tx(16) + VSize(2) + VType(1) + V(variable)

func (StorageDatom) String

func (d StorageDatom) String() string

String returns a string representation of the StorageDatom

func (StorageDatom) ToDatom

func (d StorageDatom) ToDatom(resolver Resolver) datalog.Datom

ToDatom converts a storage datom to user-facing representation This requires a resolver to map hashes back to meaningful names

type Store

type Store interface {
	Encoder() *BinaryKeyEncoder

	// Write operations
	Assert(datoms []datalog.Datom) error

	// AssertEach writes the datoms produce yields as one write unit, without the
	// caller gathering them first. The store calls produce once, handing it an
	// add to call per datom, and owns the unit's setup and teardown, so an early
	// return from produce cannot skip either.
	//
	// This is the write path for a producer whose extent is neither known nor
	// bounded — a decoder walking a dump chunk, where an entity is the floor and
	// nothing caps the top. What arrival means is each backend's own: the tree
	// store inserts into the version it is building, Badger encodes into a write
	// batch that splits at its own transaction ceiling.
	//
	// The datom add receives is the producer's workspace, valid for that call
	// only; a backend that retains one past the call copies it.
	//
	// A backend may leave the write open when the call returns, in which case
	// FinishBatch is what completes it. A failed producer stores an unspecified
	// part of its run; re-asserting is safe.
	AssertEach(produce func(add func(*datalog.Datom) error) error) error

	// FinishBatch completes a run of AssertEach calls, and is what makes the run
	// visible for a backend that left it open. A caller writing in a run — the
	// dump importers — calls it once when the run ends; a backend that completed
	// each AssertEach as it returned has nothing to do and returns nil.
	//
	// The tree store leaves runs open because publishing a version costs a
	// copy-on-write clone of every node that version touches, so a run that
	// publishes once pays for those nodes once rather than per call.
	FinishBatch() error

	Retract(datoms []datalog.Datom) error
	DeleteDatoms(datoms []datalog.Datom) (int, error)

	// Read operations
	//
	// There is no point lookup. A complete index key names one (E, A, V, Tx),
	// but Tx is what CRDT resolution determines — first entry of the ordered
	// group wins — so a reader that already knew it would have nothing left to
	// ask. Every read is a prefix scan; a scan whose range binds all four
	// components is the point lookup, and returns at most one datom because Tx
	// is unique per operation.
	//
	// CountKeys is likewise not on Store — it remains *BadgerStore-only
	// (debug/test).
	//
	// A scan yields EXACTLY the datoms whose bound components equal the
	// ScanBound's values, and narrowing to them is the implementation's
	// obligation. It is not free for a backend that projects the bound onto
	// byte keys: a V payload carries no length, so the keys for "abcd" sort
	// inside the range for "abc" interleaved with them and no endpoints
	// separate the two — the in-tree stores narrow by key length (EncodedRun,
	// runMembership). A backend comparing typed components directly has no such
	// gap. Returning everything inside a range returns datoms the caller did not
	// ask for, and no test above this seam will say so.
	Scan(bound ScanBound) (Iterator, error)
	ScanKeysOnly(bound ScanBound) (Iterator, error)
	DatomsAfter(eid datalog.ElementID) ([]datalog.Datom, error)
	MaxTxForEntity(e datalog.Identity) (datalog.ElementID, bool, error)
	GetMetadataUint64(key string) (uint64, bool, error)
	SetMetadataUint64(key string, value uint64) error

	// MaxElementID returns the highest ElementID in the store.
	// Used to restore the Lamport clock on database open.
	// Returns zero ElementID if store is empty.
	MaxElementID() (datalog.ElementID, error)

	// NewReadSession opens a consistent read view at the store's current
	// state: every read through the session observes one snapshot,
	// regardless of writes committed after it opened. A query executes all
	// its storage reads through one session, so it can never observe two
	// different database states mid-execution.
	NewReadSession() (ReadSession, error)

	// Transaction support
	BeginTx() (StoreTx, error)

	// Lifecycle
	Close() error
}

Store is the interface for datom storage

type StoreReader added in v0.15.1

type StoreReader interface {
	Scan(bound ScanBound) (Iterator, error)
	ScanKeysOnly(bound ScanBound) (Iterator, error)
	MaxElementID() (datalog.ElementID, error)
	MaxTxForEntity(e datalog.Identity) (datalog.ElementID, bool, error)
}

StoreReader is the read subset of Store. Both a Store (each call opens its own storage transaction) and a ReadSession (every call observes one shared snapshot) satisfy it, so read paths written against StoreReader run identically in either mode.

The vocabulary is store-agnostic: a scan names a typed ScanBound, not a byte range, and no binary encoder is exposed. A backend that keys on bytes projects the bound through BinaryKeyEncoder.EncodeScanBound at its own boundary; a backend that compares typed components directly does neither.

type StoreTx

type StoreTx interface {
	Assert(datoms []datalog.Datom) error
	Retract(datoms []datalog.Datom) error
	Commit() error
	Rollback() error
}

StoreTx represents a storage transaction

type TestDataConfig

type TestDataConfig struct {
	NumSymbols      int       // Number of stock symbols
	NumDays         int       // Number of days of data
	BarsPerDay      int       // Number of bars per day (1=daily, 24=hourly, 390=minute)
	OutputPath      string    // Where to store the database
	AttributePrefix string    // e.g., "price/" or "trade/"
	StartDate       time.Time // Start date for data generation
}

TestDataConfig specifies what kind of test database to build

func DefaultOHLCConfig

func DefaultOHLCConfig() TestDataConfig

DefaultOHLCConfig returns a small realistic OHLC dataset for profiling Size: ~10 symbols × 30 days × 24 hours = 7,200 bars = 50,400 datoms (~8 MB)

func LargeOHLCConfig

func LargeOHLCConfig() TestDataConfig

LargeOHLCConfig returns a large dataset for stress testing

func MediumOHLCConfig

func MediumOHLCConfig() TestDataConfig

MediumOHLCConfig returns a medium-sized dataset for profiling Size: ~50 symbols × 30 days × 24 hours = 36,000 bars = 252,000 datoms (~40 MB)

type Transaction

type Transaction struct {
	// contains filtered or unexported fields
}

Transaction represents a write transaction

func (*Transaction) Add

func (t *Transaction) Add(e datalog.Identity, a datalog.Keyword, v interface{}) error

Add adds a value using schema-aware CRDT semantics. The schema determines the semantics - callers don't need to know the cardinality.

Behavior by cardinality: - Cardinality-One: LWW semantics (highest ElementID wins) - Cardinality-Many: Add-wins set semantics (value added with OpCRDTAdd) - Cardinality-Vector: RGA append semantics (element chained after previous in tx)

For schemaless attributes (no schema or attribute not defined in schema), Add() stores the raw value with LWW semantics (cardinality-one behavior).

func (*Transaction) AddEntity

func (t *Transaction) AddEntity(e datalog.Identity, attrs map[datalog.Keyword]interface{}) error

AddEntity adds all datoms for an entity map

func (*Transaction) AddMap

func (t *Transaction) AddMap(attrs map[string]interface{}) (datalog.Identity, error)

AddMap is a convenience method that creates an entity ID and adds the attributes

func (*Transaction) Commit

func (t *Transaction) Commit() (datalog.ElementID, error)

Commit commits the transaction's writes and a :db/txInstant mark in one storage transaction, and returns the mark's ElementID, the point at which AsOf reads the commit. A transaction holding no write commits nothing and returns the store's highest ElementID.

func (*Transaction) LookupAllAttributes added in v0.16.2

func (t *Transaction) LookupAllAttributes(e datalog.Identity, a datalog.Keyword) ([]interface{}, error)

LookupAllAttributes answers e's values of a as this transaction holds them, resolved as LookupAttribute resolves them: a cardinality-many attribute's members in no fixed order, a vector's elements in order, and any other attribute's value alone. It answers no values when the attribute holds none. Like LookupAttribute, it holds t.mu only while it copies the transaction's own datoms of a, and a closed transaction answers an error.

func (*Transaction) LookupAttribute added in v0.16.2

func (t *Transaction) LookupAttribute(e datalog.Identity, a datalog.Keyword) (interface{}, bool, error)

LookupAttribute answers e's value of a as this transaction holds it: the resolution of the datoms of (e, a) in the value it opened on together with its own writes. It answers (value, true) when the attribute holds a value and (nil, false) otherwise. A cardinality-one attribute, and one the schema holds no definition of, answers its value. A cardinality-many attribute answers its members as one value, in no fixed order. A cardinality-vector attribute answers its vector, typed by the schema's value type, once the view holds a datom of it, an emptied vector included. It holds t.mu only while it copies the transaction's own datoms of a (pendingOf), and reads the value it opened on through its read session without the lock. A closed transaction answers an error.

func (*Transaction) Remove added in v0.6.0

func (t *Transaction) Remove(e datalog.Identity, a datalog.Keyword, v interface{}) error

Remove removes a value using CRDT tombstone semantics.

Behavior by cardinality: - Cardinality-One: Writes OpCRDTRemove; attribute doesn't exist if tombstone has highest Tx - Cardinality-Many: Writes OpCRDTRemove; add-wins resolves per-value conflicts - Cardinality-Vector: RGA LIFO tombstone; removes most recently added matching element - Schemaless/undefined: Defaults to CardinalityOne

func (*Transaction) Retract

func (t *Transaction) Retract(e datalog.Identity, a datalog.Keyword, v interface{}) error

Retract removes a datom

func (*Transaction) Rollback

func (t *Transaction) Rollback() error

Rollback aborts the transaction

func (*Transaction) SaveStruct added in v0.2.0

func (t *Transaction) SaveStruct(v interface{}) (datalog.Identity, error)

SaveStruct persists a struct to the database with upsert semantics. Each field is compared against this transaction's view of the entity (LookupAttribute, LookupAllAttributes): the value it opened on with its own writes applied. It writes only what the struct changes in that view, so a field the struct holds unchanged leaves another writer's change to it standing. If the struct has an ID field (tagged with `datalog:"-,id"`) and it's empty, a new ID is generated. The generated or existing ID is returned and also set on the struct's ID field.

Upsert behavior:

  • Cardinality-one fields: asserts the value when it differs from the view's
  • Cardinality-many fields (nil slice): leaves existing values unchanged
  • Cardinality-many fields (empty slice): withdraws every member the view holds
  • Cardinality-many fields (non-empty): adds the members the view lacks and withdraws the ones the slice lacks

Example:

person := Person{Name: "Alice", Age: 30}
id, err := tx.SaveStruct(&person)
// person.ID is now set to the generated identity

// Later, modify and save again
person.Name = "Alice Smith"
id, err = tx.SaveStruct(&person)  // Updates name, age unchanged

func (*Transaction) Set added in v0.6.0

func (t *Transaction) Set(e datalog.Identity, a datalog.Keyword, v interface{}) error

Set sets a value for a cardinality-one attribute using CRDT LWW semantics. For cardinality-one: just appends the new value. The storage layer handles Last-Writer-Wins resolution - no read-before-write or retraction needed.

This method is part of the CRDT implementation: - Cardinality-One: Highest ElementID wins (LWW semantics) - Cardinality-Many: Replaces entire set (diffs against the set the transaction holds, generates Add/Remove ops) - Cardinality-Vector: Replaces entire vector

func (*Transaction) SetTime

func (t *Transaction) SetTime(txTime time.Time)

SetTime sets a custom transaction time for this transaction This is useful for backdated data (e.g., historical prices)

type Tx

type Tx [16]byte

Tx represents a transaction/CRDT identifier (16 bytes = ElementID) Layout: Lamport (8 bytes big-endian) + ReplicaID (8 bytes big-endian)

func NewTxFromElementID added in v0.6.0

func NewTxFromElementID(eid datalog.ElementID) Tx

NewTxFromElementID creates a Tx from an ElementID

func NewTxFromTime

func NewTxFromTime(t time.Time) Tx

NewTxFromTime creates a transaction ID from a timestamp Encodes the time as nanoseconds in Lamport position (first 8 bytes) ReplicaID is set to 0

func (Tx) String

func (tx Tx) String() string

String returns a string representation of Tx (ElementID format)

func (Tx) ToElementID added in v0.6.0

func (tx Tx) ToElementID() datalog.ElementID

ToElementID converts Tx to ElementID

func (Tx) Uint64

func (tx Tx) Uint64() uint64

Uint64 returns the Lamport component of the transaction ID as uint64

type VectorResolutionResult added in v0.6.0

type VectorResolutionResult struct {
	// Elements contains the ordered values after RGA reconstruction
	Elements []any

	// Present reports whether the (E, A) carries any RGA datom at all, and it
	// is the resolved answer to whether the attribute exists.
	//
	// A vector that was never set and one whose every element was tombstoned
	// both reconstruct to zero live elements, so Elements cannot separate them;
	// only the presence of datoms can, and the two are different states — the
	// first has no value, the second has the empty vector. This is resolved
	// alongside the value rather than counted afterwards: a statistic describes
	// a resolution, it does not constitute one.
	Present bool

	// MaxElementID is the highest ElementID seen (for cache versioning)
	MaxElementID datalog.ElementID

	// Bound is the run this resolution walked. The pattern arm announces it,
	// and it travels back rather than being rebuilt there so the announced run
	// and the walked run are the same value and cannot drift.
	Bound ScanBound
}

VectorResolutionResult contains the resolved vector and metadata

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址