engine

package
v0.6.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

Module: pkg/engine

Purpose

The high-level orchestration layer that unifies vector CRUD, property graph operations, hybrid vector+text search, and human-memory-inspired ranking into a single coherent API. Sits between the low-level core package (HNSW, KV, metadata) and the persistence package (AOF, snapshots), providing the primary interface for all data operations in KektorDB.

Key Types & Critical Paths

Critical structs:

  • Engine -- Top-level orchestrator: *core.DB, *persistence.LazyAOFWriter, graphLocks [128]sync.Mutex, adminMu sync.Mutex, dirtyCounter int64, aofBaseSize int64, isRewriting atomic.Bool, closeOnce sync.Once, wg sync.WaitGroup.
  • GraphEdge -- {TargetID, CreatedAt, DeletedAt int64, Weight float32, Props json.RawMessage} -- Stored in KV store, not in core's graph.
  • GraphQuery -- {RootID, Relations []string, Direction string, MaxDepth int} -- Traversal criteria for graph-filtered search.
  • EventBus -- Fan-out pub/sub: subscribers map[int]chan Event, mu sync.RWMutex. Non-blocking sends.

Critical paths (hot functions):

  • VSearch() / searchWithFusion() -- Hybrid search: parallel vector + text goroutines, score normalization, alpha fusion, time-decay.
  • VLink() / VUnlink() -- Graph edge read-modify-write under 128-shard locks. AOF write after mutation.
  • VReinforce() -- Updates _last_accessed and _access_count. High-frequency call for memory ranking.
  • FindPath() -- Bidirectional BFS. Default max depth 4. Requires at least one relation type.
  • RewriteAOF() -- Deadlock-free v2: Snapshot + Release pattern. Uses atomic.CompareAndSwap on isRewriting.

Architecture & Data Flow

Hybrid search pipeline (VSearch): Parses SQL-like boolean filters and CONTAINS(field, 'text') regex clauses, builds a roaring bitmap allowlist from metadata filters, optionally intersects with a graph-traversal allowlist (BFS from GraphQuery.RootID, max depth 5), then runs vector search and text search in parallel goroutines with sync.WaitGroup. Results are normalized (vector: 1/(1+distance), text: min-max to 0..1) and fused with configurable alpha weight. Time-decay factors are applied if memory mode is enabled.

Graph operations: Edges are stored in the KV store with namespace-aware keys (indexName::nodeID) using rel: and rev: prefixes for forward and reverse indices. VLink/VUnlink perform read-modify-write cycles under 128-shard graph locks. VExtractSubgraph uses BFS (max depth 3) with optional semantic gating -- a guideQuery vector prunes neighbors that are semantically dissimilar from the guide concept.

Pathfinding (FindPath): Bidirectional BFS for shortest-path discovery. Maintains separate forward and backward search frontiers, expands both simultaneously, and checks for intersection at each level. Default max depth: 4; requires at least one relation type (no blind traversal).

Memory reinforcement (VReinforce): Updates _last_accessed timestamp and increments _access_count. Four decay models are supported: Exponential (2^(-age/halfLife)), Linear (max(0, 1 - age/halfLife)), Step (cliff-edge at half-life), and Ebbinghaus (e^(-age/S) where S = halfLife * (1 + ln(1 + accessCount)) -- the spacing effect). Decay uses the newer of _created_at or _last_accessed, so frequently accessed memories stay fresh.

Persistence: AOF uses lazy batching (100ms flush, 1s fsync). Snapshot writes to .tmp then atomic os.Rename. AOF Rewrite uses the "Snapshot + Release" pattern to avoid deadlocks.

Cross-Module Dependencies

Depends on:

  • pkg/core -- Primary dependency. All vector/graph/metadata operations delegate to core.DB.
  • pkg/persistence -- AOF writer (LazyAOFWriter), snapshot saving, AOF rewriting, crash recovery.
  • pkg/textanalyzer -- Indirectly via core's BM25 full-text index.

Used by:

  • internal/server -- Primary consumer. Every HTTP handler calls engine methods (VAdd, VSearch, VLink, etc.).
  • internal/mcp -- Primary consumer. All 57 MCP tools (49 agent + 8 admin) delegate to engine methods.
  • pkg/proxy -- Uses VSearch, VSearchGraph, VDelete for RAG, cache invalidation, and semantic firewall.
  • pkg/rag -- KektorAdapter wraps engine to implement the Store and AdaptiveStore interfaces.
  • pkg/cognitive -- Gardener uses VSearch, VLink, VUnlink, VGetLinks, VGetRelations, VReinforce for reflection analysis.

Concurrency & Locking Rules

Multi-tiered locking:

  1. adminMu (sync.Mutex) -- Protects engine-level administrative operations (snapshot, AOF rewrite). Rare, long-running, needs global consistency.
  2. graphLocks ([128]sync.Mutex) -- 128 shard mutexes keyed by FNV-1a hash of the graph key. Allows high concurrency on different nodes while protecting read-modify-write cycles.
  3. Core DB locks -- Delegated to core.DB's internal granular locks (sync.RWMutex for store, per-index metaMu for HNSW).

Atomic counters: dirtyCounter (int64), aofBaseSize (int64), isRewriting (bool) all use sync/atomic for lock-free coordination with background tasks.

Deadlock-free AOF Rewrite (v2 fix): Uses "Snapshot + Release" -- briefly acquires DB.RLock() to copy data, releases it, then iterates over HNSW indexes independently. This prevents the classic deadlock where Rewrite held s.mu while acquiring h.metaMu, blocking AddBatch which held h.metaMu and waited for s.mu.

Background goroutines for cascade operations: Cascade delete and lazy self-repair launch go func() goroutines to avoid blocking the API response. These are best-effort and thread-safe via VUnlink.

Non-blocking EventBus: Fan-out pub/sub with sync.RWMutex protecting the subscriber map. Uses non-blocking sends (select with default) to drop events for slow consumers rather than blocking the write path.

Close-once pattern: closeOnce sync.Once ensures Close() is idempotent. sync.WaitGroup + closed channel for clean shutdown of all background tasks.

Common Patterns for Contributors

HNSW Index Type Assertion: Use the getHNSWIndex() helper instead of manual type assertions:

// Good - centralized helper
hnswIdx, err := getHNSWIndex(idx)
if err != nil {
    return nil, err
}

// Avoid - duplicate pattern
hnswIdx, ok := idx.(*hnsw.Index)
if !ok {
    return nil, fmt.Errorf("not hnsw")
}

Sorting Results: Always use sort.Slice() from the standard library, never manual O(n²) sorts:

// Good - O(n log n)
sort.Slice(results, func(i, j int) bool {
    return results[i].Score > results[j].Score  // descending
})

Known Pitfalls / Gotchas

  • NEVER hold adminMu while calling core methods that acquire metaMu -- This was the v1 deadlock. The v2 fix copies data under brief RLock, releases it, then processes. If you add a new admin operation, follow this pattern.
  • VImport bypasses AOF -- Data is NOT durable until VImportCommit is called (which forces a snapshot). If the process crashes between import and commit, all imported data is lost.
  • Cascade delete is fire-and-forget -- VDelete launches a background goroutine to soft-delete incoming edges. If the process crashes before the goroutine completes, dangling edges remain. Lazy self-repair in VGetConnections cleans them on next read.
  • EventBus drops events for slow subscribers -- If the cognitive gardener's event channel (buffer: 64) fills up, events are silently dropped. This means the gardener may miss write events during sustained high-throughput periods.
  • RewriteAOF uses atomic.CompareAndSwap -- Only one rewrite can run at a time. If you call it while another is running, it returns immediately with no error. Check the return value.
  • Graph edges use json.RawMessage for props -- This avoids GC pressure but means you must manually json.Unmarshal when reading. Never store untrusted JSON in edge props -- it's passed directly to the KV store.
  • Parallel search goroutines share no state -- Vector and text search run in parallel goroutines with sync.WaitGroup. They are safe because both are read-only. Do NOT add shared mutable state between them.

Design Trade-offs

Trade-off Decision Rationale
AOF lazy batching Up to 1s of data loss on crash 10-100x throughput improvement; AOF is append-only so replay recovers everything up to the last flush
Background cascade delete Temporary graph inconsistency after node deletion Non-blocking API; self-repair cleans up on next read
EventBus drops slow consumers Event loss under load Prevents backpressure from blocking the database write path
Map-Reduce AOF replay Higher memory during startup Clean state reconstruction; later commands naturally override earlier ones
JSON RawMessage for edge props Manual unmarshaling required Avoids GC scanning millions of edge map structures, drastically reducing STW pauses
Max depth caps (BFS=5, subgraph=3, traverse=10) Cannot traverse arbitrarily deep graphs Prevents runaway traversals and stack overflows in production
VImport bypasses AOF Data not durable until VImportCommit Maximum throughput for bulk loading; commit forces snapshot for durability
Parallel vector+text search Goroutines + WaitGroup for concurrent execution Both are read-only operations; parallelism cuts latency in half for hybrid queries

Documentation

Overview

Package engine provides the high-level, embedded interface for KektorDB.

It orchestrates the in-memory vector data structures (Core) and the on-disk persistence layer (AOF/Snapshot), providing a thread-safe database instance that can be used directly within Go applications without network overhead.

Basic usage: 

opts := engine.DefaultOptions("./data")
db, err := engine.Open(opts)
if err != nil {
    log.Fatal(err)
}
defer db.Close()

This file implements the operational methods of the Engine, wrapping core database actions (KV set/get, Vector add/search) with persistence logic. It ensures that every modification is written to the Append-Only File (AOF) before or after being applied to the in-memory state, maintaining data durability.

Index

Constants

View Source 
const (
	StateCrystallized = "crystallized"
	StateStable       = "stable"
	StateVolatile     = "volatile"
	StateContested    = "contested"
)

EpistemicStates constants for the state machine.

View Source 
const (
	RelationContradicts    = "contradicts"
	RelationContradictedBy = "contradicted_by"
	RelationInvalidates    = "invalidates"
	RelationInvalidatedBy  = "invalidated_by"
)

Relation types for epistemic friction calculation.

Variables

View Source 
var ErrDimensionUnknown = fmt.Errorf("cannot add entity without vector to an empty index (dimension unknown)")

ErrDimensionUnknown is returned when attempting to add an entity without a vector to an empty index whose dimension cannot be inferred.

Functions

func CalculateConsensus added in v0.5.2 

func CalculateConsensus(nodes []EpistemicNode) (score, variance float64, centroid []float32)

CalculateConsensus computes the vector density score (Pilastro 1). It calculates the centroid of all node vectors and measures variance.

func CalculateFriction added in v0.5.2 

func CalculateFriction(nodes []EpistemicNode, indexName string, getIncomingEdges func(string, string, string, int64) ([]GraphEdge, bool)) (score float64, contradictions, invalidations int)

CalculateFriction computes topological contradiction penalties (Pilastro 3). Returns score (1.0 = no friction), and counts of negative relations.

func CalculateStability added in v0.5.2 

func CalculateStability(nodes []EpistemicNode, decayModel string) (score, avgAgeDays float64)

CalculateStability computes temporal robustness (Pilastro 2). Uses access count and age with configurable decay model.

func CosineDistance added in v0.5.2 

func CosineDistance(a, b []float32) float64

CosineDistance calculates cosine distance (1 - similarity) between two vectors.

func DetermineEpistemicState added in v0.5.2 

func DetermineEpistemicState(confidence float64, contradictions, invalidations int, thresholds EpistemicThresholds) string

DetermineEpistemicState selects the state based on final confidence and friction.

func GenerateCaveat added in v0.5.2 

func GenerateCaveat(state string, contradictions, invalidations int, consensusScore, variance float64) string

GenerateCaveat creates a human-readable explanation of the epistemic state.

Types

type ConsensusEvidence added in v0.5.2 

type ConsensusEvidence struct {
	Score          float64   `json:"score"`              // 0.0-1.0
	Sources        int       `json:"sources"`            // Number of nodes analyzed
	VectorVariance float64   `json:"vector_variance"`    // Raw variance metric
	Centroid       []float32 `json:"centroid,omitempty"` // Computed centroid (optional, for debugging)
}

ConsensusEvidence tracks vector density (semantic convergence).

type EdgeList added in v0.4.7 

type EdgeList []GraphEdge

EdgeList is the structure actually stored in the KV Value. It replaces the old []string format.

type Engine 

type Engine struct {
	// DB is the underlying in-memory core.
	// While exported, it is recommended to use Engine methods (e.g., VAdd, VSearch)
	// to ensure operations are correctly persisted to disk.
	DB *core.DB

	// AOF handles the append-only log using lazy batching for better write performance.
	// The lazy writer buffers operations and flushes them periodically (every 100ms or 1000 entries)
	// while ensuring data durability through periodic fsync operations (every 1 second).
	// This provides 10-100x throughput improvement over synchronous flushing with minimal
	// durability risk (max 1 second of data loss in case of crash).
	AOF *persistence.LazyAOFWriter

	// EventBus provides pub/sub for engine write operations.
	// Consumers (Gardener, SSE handler) subscribe to receive events.
	EventBus *EventBus
	// contains filtered or unexported fields
}

Engine is the main entry point for KektorDB. It coordinates the in-memory Core and the on-disk Persistence.

Use Open() to initialize an Engine and Close() to shut it down gracefully.

func Open 

func Open(opts Options) (*Engine, error)

Open initializes a new Engine instance using the provided options.

It performs the following actions: 1. Creates DataDir if missing. 2. Loads the latest Snapshot (.kdb) if available. 3. Replays the AOF (.aof) to recover recent data. 4. Starts background goroutines for auto-saving and compaction.

This method blocks until the database is fully loaded and ready.

func (*Engine) AOFPath added in v0.5.3 

func (e *Engine) AOFPath() string

AOFPath returns the path to the AOF file.

func (*Engine) Close 

func (e *Engine) Close() error

Close performs a clean shutdown of the Engine.

It stops background maintenance tasks, runs registered close hooks (so components holding engine memory can drain first), and closes the AOF file. Note: It does not force a final snapshot, but all data is already persisted in the AOF file, ensuring durability on restart.

func (*Engine) DataDir added in v0.6.0 

func (e *Engine) DataDir() string

DataDir returns the root data directory used by the engine.

func (*Engine) FindPath added in v0.4.7 

func (e *Engine) FindPath(indexName, sourceID, targetID string, relations []string, maxDepth int, atTime int64) (*PathResult, error)

FindPath finds the shortest path between source and target using bidirectional BFS. relations: list of edge types to traverse. Cannot be empty.

func (*Engine) GetArenaStats added in v0.6.3 

func (e *Engine) GetArenaStats(indexName string) (mmap.ArenaStats, error)

GetArenaStats exposes the physical state of an index's arena (chunks, slots, fragmentation, disk footprint). Read-only and cheap enough for monitoring. Returns an error for indexes without an arena (in-memory / no arenaDir).

func (*Engine) GetEpistemicConfig added in v0.5.2 

func (e *Engine) GetEpistemicConfig(indexName string) EpistemicConfig

GetEpistemicConfig retrieves the config for an index. Returns default config if not explicitly set.

func (*Engine) GetIndexLanguage added in v0.5.1 

func (e *Engine) GetIndexLanguage(name string) string

GetIndexLanguage returns the text language configured for an index. Returns empty string if index doesn't exist or has no language configured.

func (*Engine) IndexExists added in v0.4.5 

func (e *Engine) IndexExists(name string) bool

IndexExists checks if an index with the given name exists in the database.

func (*Engine) IndexGeneration added in v0.6.3 

func (e *Engine) IndexGeneration() uint64

IndexGeneration returns a counter that changes whenever any index is created or deleted. A background worker holding per-index state (cursors, caches) must discard that state when the generation moves: there is no delete-index event to subscribe to, and stale state silently makes the worker skip most of the data it should process.

func (*Engine) KVDelete 

func (e *Engine) KVDelete(key string) error

KVDelete removes a key from the key-value store. The operation is persisted to the AOF log.

func (*Engine) KVGet 

func (e *Engine) KVGet(key string) ([]byte, bool)

KVGet retrieves a value from the key-value store. Returns the value and a boolean indicating if the key was found.

func (*Engine) KVSet 

func (e *Engine) KVSet(key string, value []byte) error

KVSet stores a key-value pair in the database. The operation is persisted to the AOF log.

func (*Engine) ListIndexes added in v0.6.0 

func (e *Engine) ListIndexes() []string

ListIndexes returns the names of all vector indexes currently in the database.

func (*Engine) RegisterCloseHook added in v0.6.3 

func (e *Engine) RegisterCloseHook(name string, fn func(timeout time.Duration) error)

RegisterCloseHook registers fn to run during Engine.Close, before the storage layers are torn down (arenas unmapped). Hooks are called in registration order and must not block indefinitely: Close waits for them.

This exists so components that hold references into engine memory (e.g. the compiler's async workers reading vector slices) can drain their goroutines while the memory is still mapped. Without it they would read an unmapped arena and crash with SIGSEGV (C1).

func (*Engine) RewriteAOF 

func (e *Engine) RewriteAOF() error

RewriteAOF compacts the AOF file in the background.

FIX DEADLOCK (v2): Usa il pattern "Snapshot + Release" per evitare di tenere sia s.mu che h.metaMu contemporaneamente, eliminando un deadlock classico con AddBatch.

Il meccanismo precedente teneva DB.RLock() per tutta la durata dell'operazione. Durante quella finestra, IterateVectorIndexesUnlocked → hnsw.Iterate() acquisiva h.metaMu.RLock(). Se AddBatch era in coda per h.metaMu.Lock(), il runtime Go bloccava qualsiasi nuovo reader su h.metaMu — incluso RewriteAOF stesso — causando un blocco permanente.

Soluzione: acquisire s.mu.RLock brevemente solo per copiare i dati necessari, poi rilasciare il lock PRIMA di iterare su HNSW e scrivere su disco.

func (*Engine) RunGraphVacuum added in v0.4.7 

func (e *Engine) RunGraphVacuum()

RunGraphVacuum executes the cleanup of old soft-deleted edges natively in RAM.

func (*Engine) SaveSnapshot 

func (e *Engine) SaveSnapshot() error

SaveSnapshot creates a .kdb snapshot and truncates the AOF.

func (*Engine) SetEpistemicConfig added in v0.5.2 

func (e *Engine) SetEpistemicConfig(indexName string, cfg EpistemicConfig)

SetEpistemicConfig sets the epistemic config for an index.

func (*Engine) VAdd 

func (e *Engine) VAdd(indexName, id string, vector []float32, metadata map[string]any) error

VAdd inserts or updates a single vector in the specified index.

This operation updates the in-memory graph immediately and appends the operation to the AOF (Append-Only File) for durability.

'metadata' can be nil. If provided, it enables filtering and hybrid search.

func (*Engine) VAddBatch 

func (e *Engine) VAddBatch(indexName string, items []types.BatchObject) error

VAddBatch inserts multiple vectors concurrently into an index. It writes to the AOF log for each vector, ensuring durability but with higher I/O overhead. Use VImport for faster initial loading.

func (*Engine) VBeliefState added in v0.5.2 

func (e *Engine) VBeliefState(indexName string, queryVec []float32, k int, cfg EpistemicConfig) (*EpistemicState, error)

VBeliefState performs an epistemic assessment of a belief. It evaluates the truth and robustness of memories matching the query vector by analyzing three pillars: Consensus (vector density), Stability (temporal), and Friction (topological contradictions).

Parameters:

  • indexName: The vector index to search
  • queryVec: The query vector to search for
  • k: Number of candidate memories to analyze (default 10, max 50)
  • cfg: Epistemic configuration (weights and thresholds)

Returns:

  • EpistemicState: Complete assessment with confidence, state, evidence, and nodes

func (*Engine) VCompress 

func (e *Engine) VCompress(indexName string, newPrecision distance.PrecisionType) error

VCompress changes the precision of an existing index (e.g., float32 -> int8). This operation rebuilds the index in memory.

func (*Engine) VCreate 

func (e *Engine) VCreate(name string, metric distance.DistanceMetric, m, efC int, prec distance.PrecisionType, lang string, config *hnsw.AutoMaintenanceConfig, autoLinks []hnsw.AutoLinkRule, memoryConfig *hnsw.MemoryConfig) error

VCreate initializes a new vector index with the specified configuration.

'metric' defines the distance function (e.g., distance.Cosine, distance.Euclidean). 'prec' defines the storage precision (float32, float16, int8). 'lang' enables hybrid search features (e.g., "english", "italian") or "" to disable. accept AutoLinkRules.

Returns an error if an index with the same name already exists.

func (*Engine) VDelete 

func (e *Engine) VDelete(indexName, id string) error

VDelete marks a vector as deleted in the specified index. The node remains in the graph but is excluded from search results. It also performs a CASCADE DELETE on the graph: all edges to and from this node are soft-deleted to prevent ghost nodes and dangling references.

func (*Engine) VDeleteIndex 

func (e *Engine) VDeleteIndex(name string) error

VDeleteIndex completely removes an index and all its data. The operation is persisted to the AOF log as a VDROP command.

func (*Engine) VEvolve added in v0.5.2 

func (e *Engine) VEvolve(indexName, oldID string, newVector []float32, newMetadata map[string]any, reason string) (string, error)

VEvolve performs a semantic evolution of a memory node. It creates a new node with updated data and links it to the old node, preserving the graph topology by copying incoming edges to the new node. The old node is marked as historical.

func (*Engine) VExtractSubgraph added in v0.4.5 

func (e *Engine) VExtractSubgraph(indexName, rootID string, relations []string, maxDepth int, atTime int64, guideQuery []float32, threshold float64) (*SubgraphResult, error)

VExtractSubgraph performs a BFS to retrieve the local neighborhood.

func (*Engine) VFilter added in v0.5.1 

func (e *Engine) VFilter(indexName string, filter string, limit int) ([]string, error)

VFilter returns vector IDs matching the given metadata filter without vector similarity. This is useful for exact metadata-based queries without semantic search.

func (*Engine) VGet 

func (e *Engine) VGet(indexName, id string) (core.VectorData, error)

VGet retrieves the full data (vector + metadata) for a single ID. 

func (e *Engine) VGetAutoLinks(indexName string) ([]hnsw.AutoLinkRule, error)

VGetAutoLinks returns the current auto-linking rules for an index.

func (*Engine) VGetConnections added in v0.3.0 

func (e *Engine) VGetConnections(indexName, sourceID, relationType string) ([]core.VectorData, error)

VGetConnections retrieves the full data of nodes linked to a source node. It performs a Graph Traversal (1-hop) and Hydration in one step. SELF-REPAIR: If it encounters links to non-existent nodes, it removes them in background.

func (*Engine) VGetEdges added in v0.4.7 

func (e *Engine) VGetEdges(indexName, sourceID, relationType string, atTime int64) ([]GraphEdge, bool)

VGetEdges retrieves full edge details for outgoing links.

func (*Engine) VGetIDsByCursor added in v0.5.1 

func (e *Engine) VGetIDsByCursor(indexName string, cursor uint32, limit int) ([]string, uint32, error)

VGetIDsByCursor exposes the fast cursor-based iteration for background jobs and API pagination.

func (*Engine) VGetIncoming added in v0.4.5 

func (e *Engine) VGetIncoming(indexName, targetID, relationType string) ([]string, bool)

VGetIncoming retrieves ACTIVE incoming links. (O(1) Reverse Index)

func (*Engine) VGetIncomingEdges added in v0.4.7 

func (e *Engine) VGetIncomingEdges(indexName, targetID, relationType string, atTime int64) ([]GraphEdge, bool)

VGetIncomingEdges retrieves full edge details for incoming links (Lazy Hydration). NOTE: By contract, the returned GraphEdge.TargetID contains the ID of the SOURCE node.

func (*Engine) VGetIncomingRelations added in v0.5.1 

func (e *Engine) VGetIncomingRelations(indexName, targetID string) map[string][]string

VGetIncomingRelations retrieves ALL active incoming relationships for a node. 

func (e *Engine) VGetLinks(indexName, sourceID, relationType string) ([]string, bool)

VGetLinks retrieves ACTIVE links. (O(1) Memory lookup)

func (*Engine) VGetMany 

func (e *Engine) VGetMany(indexName string, ids []string) ([]core.VectorData, error)

VGetMany retrieves data for multiple IDs in parallel.

func (*Engine) VGetRelations added in v0.3.0 

func (e *Engine) VGetRelations(indexName, sourceID string) map[string][]string

VGetRelations retrieves ALL active outgoing relationships for a node.

func (*Engine) VImport 

func (e *Engine) VImport(indexName string, items []types.BatchObject) error

VImport performs a high-speed bulk insertion of vectors.

Unlike VAddBatch, this method bypasses the AOF log to maximize throughput.

Use this method for initial dataset loading or massive restores. Note: This operation acquires a global administrative lock to ensure consistency during the snapshot.

func (*Engine) VImportCommit added in v0.5.1 

func (e *Engine) VImportCommit(indexName string) error

VImportCommit finalizes a bulk import transaction. It forces a disk snapshot to make data durable, and kicks off the Turbo Refine. 

func (e *Engine) VLink(indexName, sourceID, targetID, relationType, inverseRelationType string, weight float32, props map[string]any) error

VLink creates a rich directed edge between two nodes and automatically updates the reverse index.

func (*Engine) VReinforce added in v0.4.7 

func (e *Engine) VReinforce(indexName string, ids []string) error

VReinforce updates the usage statistics of a memory node to keep it fresh. It updates '_last_accessed' to Now and increments '_access_count'. THREAD-SAFE: Uses fine-grained locking and lightweight VMETA AOF commands.

func (*Engine) VRestore added in v0.6.3 

func (e *Engine) VRestore(indexName, id string) error

VRestore reverses a supersede/archive, making a memory visible to default retrieval again. It clears the hiding flags written by VEvolve (_is_historical) and by the consolidation/session-archival paths (_archived).

It deliberately does NOT resurrect a VDelete'd node: Delete removes the external->internal mapping and the node metadata, so that state is not recoverable. Callers get an explicit error rather than a silent no-op.

Provenance metadata (_consolidated_into, invalidated_by, ...) is left intact: it is history, and only the visibility flags decide what retrieval returns.

func (*Engine) VSearch 

func (e *Engine) VSearch(indexName string, query []float32, k int, filter string, explicitTextQuery string, efSearch int, alpha float64, graphQuery *GraphQuery) ([]string, error)

VSearch performs a vector, text, or hybrid search.

'k': number of results to return. 'filter': supports SQL-like metadata filtering and "CONTAINS(field, 'text')". 'efSearch': tuning parameter for HNSW accuracy (0 = default). 'alpha': weight for hybrid fusion (1.0 = vector only, 0.0 = text only, 0.5 = balanced).

Returns a list of external IDs sorted by relevance.

On a memory index, superseded and archived records are hidden (safe default). Use VSearchWithOptions to include them.

func (*Engine) VSearchGraph added in v0.3.0 

func (e *Engine) VSearchGraph(indexName string, query []float32, k int, filter string, explicitTextQuery string, efSearch int, alpha float64, relations []string, hydrate bool, graphQuery *GraphQuery) ([]GraphSearchResult, error)

VSearchGraph performs a search and traverses the graph based on relation paths. relations example: ["prev", "next", "parent.child"]

The memory-visibility guard applies to the SEED results (this is a search), but the traversal itself stays raw: expansion follows whatever edges exist, including edges to superseded nodes, so evolution chains and graph topology remain inspectable.

func (*Engine) VSearchWithOptions added in v0.6.3 

func (e *Engine) VSearchWithOptions(indexName string, query []float32, k int, filter string, explicitTextQuery string, efSearch int, alpha float64, graphQuery *GraphQuery, opts SearchOptions) ([]string, error)

VSearchWithOptions is VSearch with explicit SearchOptions.

func (*Engine) VSearchWithScores added in v0.3.0 

func (e *Engine) VSearchWithScores(indexName string, query []float32, k int, filter string, efSearch int) ([]SearchResult, error)

VSearchWithScores performs a search and returns results with their scores. If the index has MemoryConfig enabled, it applies time decay ranking.

func (*Engine) VSearchWithScoresOptions added in v0.6.3 

func (e *Engine) VSearchWithScoresOptions(indexName string, query []float32, k int, filter string, efSearch int, opts SearchOptions) ([]SearchResult, error)

VSearchWithScoresOptions is VSearchWithScores with explicit SearchOptions. On a memory index, superseded and archived records are hidden unless opts.IncludeObsolete is set.

func (*Engine) VSetMetadata added in v0.5.1 

func (e *Engine) VSetMetadata(indexName, id string, newProps map[string]any) error

VSetMetadata updates the properties of an existing node without altering its vector. It merges the new properties with the existing ones and persists the change via VMETA.

func (*Engine) VTraverse added in v0.3.1 

func (e *Engine) VTraverse(indexName, startID string, paths []string) (*GraphNode, error)

VTraverse performs a deep graph traversal starting from a specific node ID. Unlike VSearchGraph, this does not perform a vector search but starts from a known ID. paths example: ["parent", "parent.child", "next"]

func (*Engine) VTriggerMaintenance added in v0.3.0 

func (e *Engine) VTriggerMaintenance(indexName string, taskType string) error
func (e *Engine) VUnlink(indexName, sourceID, targetID, relationType, inverseRelationType string, hardDelete bool) error

VUnlink removes a directed edge and its reverse entry. If hardDelete is true, the record is physically removed. If hardDelete is false, it sets DeletedAt (Soft Delete). 

func (e *Engine) VUpdateAutoLinks(indexName string, rules []hnsw.AutoLinkRule) error

VUpdateAutoLinks updates the auto-linking rules for an index at runtime.

func (*Engine) VUpdateIndexConfig added in v0.3.0 

func (e *Engine) VUpdateIndexConfig(indexName string, config hnsw.AutoMaintenanceConfig) error

type EpistemicConfig added in v0.5.2 

type EpistemicConfig struct {
	Enabled    bool                `json:"enabled"`
	Weights    EpistemicWeights    `json:"weights"`
	Thresholds EpistemicThresholds `json:"thresholds"`
	DecayModel string              `json:"decay_model,omitempty"` // "ebbinghaus", "exponential", "linear"
}

EpistemicConfig holds per-index configuration for the epistemic engine. It follows the same pattern as MemoryConfig and can be set during VCreate.

func DefaultEpistemicConfig added in v0.5.2 

func DefaultEpistemicConfig() *EpistemicConfig

DefaultEpistemicConfig returns a sensible default configuration.

type EpistemicEvidence added in v0.5.2 

type EpistemicEvidence struct {
	Consensus ConsensusEvidence `json:"consensus"`
	Stability StabilityEvidence `json:"stability"`
	Friction  FrictionEvidence  `json:"friction"`
}

EpistemicEvidence holds the detailed evidence for each pillar.

type EpistemicNode added in v0.5.2 

type EpistemicNode struct {
	ID           string
	Vector       []float32
	Metadata     map[string]any
	CreatedAt    float64 // Unix timestamp
	AccessCount  int
	IsHistorical bool
	Similarity   float64 // Cosine similarity to query
}

EpistemicNode is an internal representation of a candidate node for analysis.

type EpistemicNodeResult added in v0.5.2 

type EpistemicNodeResult struct {
	ID             string  `json:"id"`
	Content        string  `json:"content"`    // From metadata["content"]
	Score          float64 `json:"score"`      // Cosine similarity
	CreatedAt      int64   `json:"created_at"` // Unix timestamp
	AccessCount    int     `json:"access_count"`
	IsHistorical   bool    `json:"is_historical"`
	Contradictions int     `json:"contradictions_count"`
	Invalidations  int     `json:"invalidations_count"`
}

EpistemicNodeResult is the public representation returned in API responses.

type EpistemicState added in v0.5.2 

type EpistemicState struct {
	Confidence float64               `json:"confidence"` // 0.0-1.0 aggregated score
	State      string                `json:"state"`      // "crystallized", "stable", "volatile", "contested"
	Evidence   EpistemicEvidence     `json:"evidence"`
	Caveat     string                `json:"caveat,omitempty"`
	Nodes      []EpistemicNodeResult `json:"nodes"`
}

EpistemicState represents the complete epistemic assessment of a belief.

type EpistemicThresholds added in v0.5.2 

type EpistemicThresholds struct {
	Crystallized float64 `json:"crystallized" yaml:"crystallized"` // Default: 0.85 (above = crystallized)
	Volatile     float64 `json:"volatile" yaml:"volatile"`         // Default: 0.40 (below = volatile)
}

EpistemicThresholds defines the confidence thresholds for state determination.

type EpistemicWeights added in v0.5.2 

type EpistemicWeights struct {
	Consensus float64 `json:"consensus" yaml:"consensus"` // Default: 0.40
	Stability float64 `json:"stability" yaml:"stability"` // Default: 0.30
	Friction  float64 `json:"friction" yaml:"friction"`   // Default: 0.30
}

EpistemicWeights defines the mathematical weights for the three pillars of epistemic state.

type Event added in v0.5.1 

type Event struct {
	Type      EventType `json:"type"`
	IndexName string    `json:"index_name"`
	ID        string    `json:"id"`
	TargetID  string    `json:"target_id,omitempty"`
	RelType   string    `json:"rel_type,omitempty"`
	Timestamp int64     `json:"timestamp"`
}

Event represents a single write operation in the engine.

type EventBus added in v0.5.1 

type EventBus struct {
	// contains filtered or unexported fields
}

EventBus implements a fan-out pub/sub pattern for engine events. Each subscriber gets its own channel. Events are broadcast to all active subscribers. If a subscriber's buffer is full, the event is silently dropped for that subscriber to prevent slowing down the database write-path.

func NewEventBus added in v0.5.1 

func NewEventBus() *EventBus

NewEventBus creates a new event bus.

func (*EventBus) Close added in v0.5.1 

func (eb *EventBus) Close()

Close shuts down the bus and closes all subscriber channels.

func (*EventBus) Emit added in v0.5.1 

func (eb *EventBus) Emit(e Event)

Emit broadcasts the event to all active subscribers. If a subscriber's buffer is full, the event is silently dropped for that subscriber to prevent slowing down the database write-path.

func (*EventBus) Subscribe added in v0.5.1 

func (eb *EventBus) Subscribe(bufferSize int) chan Event

Subscribe returns a new channel that will receive all future events. The caller is responsible for reading from this channel to avoid blocking, and MUST call Unsubscribe when done.

func (*EventBus) Unsubscribe added in v0.5.1 

func (eb *EventBus) Unsubscribe(ch chan Event)

Unsubscribe removes a subscriber and closes its channel.

type EventType added in v0.5.1 

type EventType string

EventType categorizes engine events for pub/sub consumers. 

const (
	EventVectorAdd    EventType = "vector.add"
	EventVectorDelete EventType = "vector.delete"
	EventEdgeCreate   EventType = "edge.create"
	EventEdgeDelete   EventType = "edge.delete"
	EventVectorAccess EventType = "vector.access"
	EventVectorUpdate EventType = "vector.update"
	EventEvolution    EventType = "memory.evolution"
)

type FrictionEvidence added in v0.5.2 

type FrictionEvidence struct {
	Score          float64 `json:"score"`          // 0.0-1.0 (1.0 = no friction)
	Contradictions int     `json:"contradictions"` // Count of contradicts relations
	Invalidations  int     `json:"invalidations"`  // Count of invalidated_by relations
}

FrictionEvidence tracks topological contradictions.

type GraphEdge added in v0.4.7 

type GraphEdge struct {
	TargetID  string  `json:"t"`           // Target Node ID
	CreatedAt int64   `json:"c"`           // Creation Timestamp (Unix Nano)
	DeletedAt int64   `json:"d,omitempty"` // Deletion Timestamp (0 = Active)
	Weight    float32 `json:"w,omitempty"` // Weight (default 1.0, omitted if 0/1 logic handled externally)
	// Props stores arbitrary metadata.
	// We use json.RawMessage ([]byte) instead of map[string]interface{}.
	// This prevents the Go Garbage Collector from scanning complex map structures
	// for millions of edges, drastically improving STW (Stop The World) pause times.
	Props json.RawMessage `json:"p,omitempty"` // property graph attributes
}

GraphEdge represents a rich, versioned connection between nodes. Used to store the adjacency list in the KV store.

func (GraphEdge) GetProps added in v0.5.1 

func (e GraphEdge) GetProps() map[string]interface{}

GetProps decodes the raw JSON bytes into a usable map. This should only be called when the user explicitly requests edge properties.

func (GraphEdge) IsActive added in v0.4.7 

func (e GraphEdge) IsActive() bool

Active returns true if the edge is valid at the current time (not soft-deleted).

type GraphNode added in v0.3.1 

type GraphNode struct {
	core.VectorData
	// Nested relationship map: "child" -> [List of Nodes]
	Connections map[string][]GraphNode `json:"connections,omitempty"`
}

GraphNode represents a node in the graph with its data and nested connections. This allows representing trees like Chunk -> Parent -> Children.

type GraphQuery added in v0.4.5 

type GraphQuery struct {
	RootID string `json:"root_id"`
	// Relation types to follow (e.g., "mentions", "parent").
	// If empty, follows all relations (though usually you want to be specific).
	Relations []string `json:"relations"`

	// Direction of traversal: "out" (Source->Target), "in" (Target<-Source), or "both".
	// Default: "out".
	Direction string `json:"direction"`

	// How deep to traverse. 1 = immediate neighbors. -1 = infinite (dangerous, use limit).
	MaxDepth int `json:"max_depth"`
}

GraphQuery defines the criteria to select a subset of nodes based on graph topology.

type GraphSearchResult added in v0.3.0 

type GraphSearchResult struct {
	ID    string    `json:"id"`
	Score float64   `json:"score"`
	Node  GraphNode `json:"node"`
}

Public structure for graph results

type Options 

type Options struct {
	// DataDir is the directory where .aof and .kdb files will be stored.
	// It is created automatically if it does not exist.
	DataDir string

	// AofFilename is the name of the Append-Only File (default: "kektordb.aof").
	// The snapshot file will effectively be named <AofFilename>.kdb.
	AofFilename string

	// AutoSaveInterval defines how much time must pass since the last save
	// before a new snapshot is triggered (if AutoSaveThreshold is also met).
	// Set to 0 to disable auto-saving based on time.
	AutoSaveInterval time.Duration

	// AutoSaveThreshold defines how many write operations must occur
	// before a new snapshot is triggered (if AutoSaveInterval is also met).
	// Set to 0 to disable auto-saving based on write count.
	AutoSaveThreshold int64

	// AofRewritePercentage triggers an automatic AOF compaction (rewrite) when the
	// AOF file size exceeds the base size by this percentage.
	// E.g., 100 means rewrite when size doubles. Set to 0 to disable.
	AofRewritePercentage int

	// MaintenanceInterval defines how often the background graph optimization runs.
	// Default: 10 seconds.
	MaintenanceInterval time.Duration

	// EpistemicConfig provides default settings for the epistemic engine (belief assessment).
	// Can be overridden per-index during VCreate.
	EpistemicConfig *EpistemicConfig

	// AOFWriteBufferSize controls the size of the write buffer for the AOF file in bytes.
	// A larger buffer reduces syscall frequency under high write throughput.
	// Set to 0 to use the default (4096 bytes).
	AOFWriteBufferSize int
}

Options configures the behavior of the Engine, including persistence paths and automatic maintenance policies.

func DefaultOptions 

func DefaultOptions(dataDir string) Options

DefaultOptions returns a standard configuration suitable for most use cases.

Defaults:

  • DataDir: provided path
  • AofFilename: "kektordb.aof"
  • AutoSave: Every 60s if at least 1000 changes occurred
  • AofRewrite: At 100% growth

type PathResult added in v0.4.7 

type PathResult struct {
	Source string         `json:"source"`
	Target string         `json:"target"`
	Path   []string       `json:"path"`  // Sequence of Node IDs
	Edges  []SubgraphEdge `json:"edges"` // Details of edges traversed
}

type ScoreBreakdown added in v0.6.1 

type ScoreBreakdown struct {
	Similarity  float64 `json:"similarity"`
	DecayFactor float64 `json:"decay_factor"`
}

ScoreBreakdown explains how a search score was computed: the raw similarity from vector distance and the time-decay factor applied on top.

type SearchOptions added in v0.6.3 

type SearchOptions struct {
	// IncludeObsolete disables the memory-visibility guard, returning also
	// records hidden by VEvolve / consolidation / archival.
	IncludeObsolete bool
}

SearchOptions tunes a search. The zero value is the safe default: on a memory index, superseded (_is_historical) and archived (_archived) records are hidden, which is what every ordinary caller wants. Set IncludeObsolete to see them as well (audit, debugging, re-consolidation).

type SearchResult added in v0.3.0 

type SearchResult struct {
	ID        string          `json:"id"`
	Score     float64         `json:"score"`
	Breakdown *ScoreBreakdown `json:"score_breakdown,omitempty"`
}

SearchResult represents a match with its similarity score/distance.

type StabilityEvidence added in v0.5.2 

type StabilityEvidence struct {
	Score       float64 `json:"score"`              // 0.0-1.0
	AvgAgeDays  float64 `json:"avg_age_days"`       // Average age in days
	TotalAccess int     `json:"total_access_count"` // Sum of access counts
}

StabilityEvidence tracks temporal robustness (age + reinforcements).

type SubgraphEdge added in v0.4.5 

type SubgraphEdge struct {
	Source   string `json:"source"`
	Target   string `json:"target"`
	Relation string `json:"relation"`
	Dir      string `json:"dir"`
}

type SubgraphNode added in v0.4.5 

type SubgraphNode struct {
	ID       string         `json:"id"`
	Metadata map[string]any `json:"metadata"`
}

type SubgraphResult added in v0.4.5 

type SubgraphResult struct {
	RootID string         `json:"root_id"`
	Nodes  []SubgraphNode `json:"nodes"`
	Edges  []SubgraphEdge `json:"edges"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.
⌂ 换网址