core

package
v0.6.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

Module: pkg/core

Purpose

The central data engine of KektorDB, unifying four storage paradigms into a single process: an HNSW vector index for approximate nearest neighbor search, a sharded property graph with soft deletes and time-travel queries, a thread-safe key-value store, and three secondary indexes (roaring bitmap inverted index, B-tree for range queries, and BM25 full-text search). The DB struct is the top-level orchestrator that binds all subsystems together.

Key Types & Critical Paths

Critical structs:

  • DB (core.go:775) -- Top-level container: holds KVStore, vector indexes map, graph shards [128], inverted index, B-tree, text index.
  • Index (hnsw) -- HNSW graph: nodes atomic.Value, shardsMu [128]sync.RWMutex, metaMu sync.RWMutex, closed atomic.Bool.
  • Node -- HNSW node: InternalID uint32, Connections [][]uint32, Deleted atomic.Bool, vector pointers into mmap arena.
  • GraphShard -- {mu sync.RWMutex, nodes map[string]*GraphNode} -- one of 128 independent partitions.
  • GraphNode -- {ID, OutEdges map[string][]GraphEdge, InEdges map[string][]ReverseEdge}.
  • BitSet -- Custom bitmap using []uint64 buckets for visited-node tracking (bitwise ops, no allocation).
  • minHeap / maxHeap -- Store types.Candidate by value (16 bytes), not pointer, for CPU cache locality.

Critical paths (hot functions):

  • searchLayerUnlocked() -- The innermost HNSW search loop. Gets/pools BitSet and heaps on every call. Zero allocations target.
  • AddBatch() -- Parallel insertion: spawns runtime.NumCPU() workers, local accumulators, sharded commit. Throughput-critical.
  • LockTwoShards() -- Deadlock prevention: always locks lower-indexed shard first.
  • BytesToFloat32Slice() -- Zero-copy unsafe.Slice casting from mmap bytes to []float32.

Architecture & Data Flow

Vector data enters via Add() or AddBatch() and flows through a dual-path insertion system. Single inserts go through the fine-grained per-shard lock path; batch inserts spawn runtime.NumCPU() workers that write to local []LinkRequest accumulators (no channel contention), then shard by node ID and commit in parallel. Vectors are stored in the mmap-backed VectorArena (see pkg/storage/mmap), with Node.VectorF32 pointing directly into the mmap region via unsafe.Slice -- zero copies, zero heap pressure.

The graph layer stores forward edges (GraphEdge, ~64 bytes with TargetID, timestamps, weight, and json.RawMessage props) and reverse edges (ReverseEdge, ~32 bytes with only SourceID and timestamps). This asymmetric design saves 50% RAM on incoming edges, shifting the burden to lazy hydration during read-time traversal.

Secondary indexes are maintained in parallel: the inverted index maps string metadata values to roaring.Bitmap of node IDs for O(1) equality/OR/AND operations; the B-tree handles numerical range queries (<, <=, >, >=, =); and the full-text index uses tokenized BM25 ranking with English/Italian stemming support.

Search uses devirtualized distance functions -- the metric function is captured as a closure at query start, avoiding switch-in-loop overhead in the hot path. Pre-filtering via roaring bitmap allowlists intersects metadata constraints before vector search begins, pruning the search space. When the global HNSW entrypoint is excluded by the allowlist, smart entrypoint selection finds the nearest allowed node to start the search.

Cross-Module Dependencies

Depends on:

  • pkg/storage/mmap -- VectorArena for zero-copy vector storage. The Index holds a reference to the arena and calls GetBytes() for vector access.
  • pkg/core/distance -- Distance functions (Euclidean, Cosine) dispatched by precision type (F32/F16/I8).
  • pkg/textanalyzer -- Stemmers for BM25 full-text index tokenization.

Used by:

  • pkg/engine -- Primary consumer. Calls VCreate, VAdd, VSearch, graph ops, metadata ops. Wraps core with persistence and high-level APIs.
  • internal/server -- Indirectly via engine. Exposes core operations over HTTP.
  • internal/mcp -- Indirectly via engine. Exposes core operations as MCP tools.
  • pkg/persistence -- Reads DB.Snapshot() for gob serialization; core provides the data, persistence writes it.

Concurrency & Locking Rules

Five-level lock hierarchy (always acquire in order, release in reverse):

  1. DB.mu (sync.RWMutex) -- Top-level lock protecting the map of vector indexes, graph shards, and secondary index maps. Held briefly for map lookups.

  2. DB.indexLocks (map[string]*sync.RWMutex) -- Per-index granular locks for metadata operations. Allows concurrent metadata writes to different indexes without contending on DB.mu.

  3. Index.metaMu (sync.RWMutex) -- Global HNSW index lock protecting ID maps, node slice pointer, entrypoint, maxLevel, and quantizer.

  4. Index.shardsMu ([128]sync.RWMutex) -- 128 shard locks for node-level operations. Node internalID & 127 determines shard. This is the finest-grained lock, enabling 128 concurrent node operations.

  5. GraphShard.mu (sync.RWMutex) -- 128 graph shard locks. GetShardIndex() uses FNV-1a hash with & (NumGraphShards-1) for O(1) shard lookup.

Strict cross-shard lock ordering: LockTwoShards(id1, id2) always locks the lower-indexed shard first, mathematically preventing deadlocks during cross-shard edge operations.

Copy-on-Write for atomic node slices: The nodes slice is stored as atomic.Value. On growth, a new backing array is allocated, old contents copied, and the atomic pointer swapped. Readers always see a consistent snapshot without any lock.

Copy-Before-Release pattern: During neighbor linking, a node's connections are copied under RLock, then the lock is released before CPU-intensive distance calculations. Only the final pointer swap requires a brief write lock.

sync.Pool for zero-allocation hot paths: BitSet, minHeap, maxHeap, and neighbor buffers are pooled. The search loop (searchLayerUnlocked) gets/puts these on every call, eliminating heap allocations in the critical path.

Atomic lifecycle flags: Index.closed (atomic.Bool) is checked at the start of every public method. Set before acquiring locks in Close() to prevent new operations from starting during shutdown.

Value-semantics heaps: minHeap and maxHeap store types.Candidate by value (not pointer). The struct is only 16 bytes (uint32 + float64), so copying is cheaper than GC pressure from pointer indirection.

Known Pitfalls / Gotchas

  • NEVER hold metaMu while acquiring adminMu -- This was the root cause of the v1 AOF Rewrite deadlock. The fix (v2) uses "Snapshot + Release": copy data under RLock, release it, then iterate indexes independently.
  • AddBatch mutates Index.needsRefine -- After bulk import, the index is marked as needing refinement. Search auto-boosts efSearch when this flag is set. Don't forget to call TurboRefine after VImportCommit.
  • Node.VectorF32 points into mmap -- Never modify vector bytes directly. Use arena.SetBytes() which handles the unsafe cast safely. Direct mutation bypasses the arena's slot table and can corrupt data during compaction.
  • BitSet must be returned to pool -- Every poolBitSet.Get() must have a matching poolBitSet.Put() via defer. Leaking BitSets causes unbounded memory growth under sustained search load.
  • closed flag is set BEFORE acquiring locks -- In Close(), the atomic flag is set first to reject new operations, then locks are acquired for cleanup. If you add a new public method, check closed.Load() as the very first action.
  • gob serialization excludes vectors -- Node.GobEncode() explicitly skips vector slices. On restore, vectors are re-linked via the arena. If you add new fields to Node, remember to update both GobEncode and GobDecode.

Design Trade-offs

Trade-off Decision Rationale
Soft deletes over hard deletes atomic.Bool Deleted flag + DeletedAt timestamps Maintains graph structural stability; enables time-travel queries; vacuum runs asynchronously
128 shards (power-of-two) Fixed count, not dynamic & (N-1) is faster than % N; 128 provides high concurrency with minimal memory overhead
gob + JSON for metadata Custom GobEncode serializes map[string]any as JSON bytes Avoids gob's inability to handle []interface{} and nested maps natively
Two HNSW insert paths Add() (single, latency) vs AddBatch() (parallel, throughput) Single insert prioritizes low latency; batch prioritizes throughput with sharded commit
Smart dispatch for distance Rust CGO for vectors >= 128 dims, pure Go/Gonum for shorter CGO call overhead is amortized only for longer vectors where SIMD benefits dominate
No vector data in snapshots Vectors live in mmap, excluded from gob serialization Snapshots stay small; vectors are re-mapped on restore from persistent .bin files
BM25 with linear scan for TF Posting list is []PostingEntry, not a map Simpler, lower memory overhead for typical document lengths; O(N) scan is acceptable
Reverse edges store only SourceID ReverseEdge is ~32 bytes vs GraphEdge at ~64 bytes Saves 50% RAM on incoming edges; full edge data is lazily hydrated from forward edges during traversal

Documentation

Overview

Package core provides the fundamental data structures and logic for the KektorDB engine.

This file defines the main DB struct, which orchestrates all data storage, including the KV store, vector indexes, and secondary indexes for metadata (inverted, B-Tree, and text). It also implements core functionalities like snapshotting, filtering, full-text search, and index management.

Package core provides the fundamental data structures and logic for the KektorDB engine.

File: pkg/core/graph.go

Index

Constants

View Source 
const NumGraphShards = 128

NumGraphShards defines the number of independent partitions for the graph. 128 is a good power-of-two number that provides high concurrency with low memory footprint.

Variables

This section is empty.

Functions

func GetShardIndex added in v0.5.1 

func GetShardIndex(id string) uint32

GetShardIndex calculates a deterministic shard ID for a given string key. It uses FNV-1a which is very fast and has excellent distribution properties.

Types

type BTreeItem 

type BTreeItem struct {
	Value  float64
	NodeID uint32
}

BTreeItem is a struct used by the B-Tree to associate a numerical metadata value with a node ID.

type BruteForceIndex 

type BruteForceIndex struct {
	// contains filtered or unexported fields
}

BruteForceIndex is a simple implementation of VectorIndex that stores all vectors and calculates the distance to every one of them during a search. It is not optimized for performance but is useful for testing and small datasets.

func NewBruteForceIndex 

func NewBruteForceIndex() *BruteForceIndex

NewBruteForceIndex creates and returns a new, empty BruteForceIndex.

func (*BruteForceIndex) Add 

func (idx *BruteForceIndex) Add(id string, vector []float32) (uint32, error)

Add adds a vector to the index

func (*BruteForceIndex) Close added in v0.5.1 

func (idx *BruteForceIndex) Close() error

Close for BruteForceIndex is a no-op since it only uses GC RAM.

func (*BruteForceIndex) Delete 

func (idx *BruteForceIndex) Delete(id string)

Delete removes a vector by its ID.

func (*BruteForceIndex) GetArenaDir added in v0.5.1 

func (idx *BruteForceIndex) GetArenaDir() string

GetArenaDir for BruteForceIndex returns empty string since it doesn't use mmap.

func (*BruteForceIndex) SearchWithScores 

func (idx *BruteForceIndex) SearchWithScores(query []float32, k int, allowList *roaring.Bitmap, efSearch int) []types.SearchResult

SearchWithScores finds the K nearest vectors to the query vector.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is the main container for all KektorDB data types. It orchestrates the KV store, vector indexes, and all secondary indexes for metadata.

func NewDB 

func NewDB() *DB

NewDB creates and returns a new, initialized DB instance.

func (*DB) AddEdge added in v0.5.1 

func (db *DB) AddEdge(sourceID, targetID, relationType string, weight float32, props []byte, timestamp int64)

AddEdge creates or updates a directed edge from sourceID to targetID. It automatically updates the reverse index (InEdges) on the target node.

func (*DB) AddMetadata 

func (s *DB) AddMetadata(indexName string, nodeID uint32, metadata map[string]any) error

AddMetadata associates metadata with a node ID and updates the secondary indexes.

func (*DB) AddMetadataUnlocked 

func (s *DB) AddMetadataUnlocked(indexName string, nodeID uint32, metadata map[string]any) error

AddMetadataUnlocked adds metadata without acquiring a lock. The caller must ensure thread safety.

func (*DB) Close added in v0.5.1 

func (s *DB) Close() error

Close gracefully shuts down all vector indexes, releasing OS resources like mmap.

func (*DB) Compress 

func (s *DB) Compress(indexName string, newPrecision distance.PrecisionType) error

Compress converts an existing index to a new precision

func (*DB) CreateVectorIndex 

func (s *DB) CreateVectorIndex(name string, metric distance.DistanceMetric, m, efConstruction int, precision distance.PrecisionType, textLang string, arenaDir string) error

CreateVectorIndex creates a new vector index with the specified configuration.

func (*DB) DeleteMetadata added in v0.5.1 

func (s *DB) DeleteMetadata(indexName string, nodeID uint32) error

DeleteMetadata removes all metadata and index entries associated with a node. This should be called when a node is deleted to prevent memory leaks.

func (*DB) DeleteVectorIndex 

func (s *DB) DeleteVectorIndex(name string) error

DeleteVectorIndex removes an entire vector index and all of its associated data.

func (*DB) FindIDsByFilter 

func (s *DB) FindIDsByFilter(indexName string, filter string) (*roaring.Bitmap, error)

FindIDsByFilter acts as a query planner for metadata filters. It supports AND and OR logic. OR has lower precedence (the filter is first split by OR, and each resulting block is evaluated as an AND of its sub-filters). FindIDsByFilter acts as a query planner for metadata filters.

Grammar: `expr := term (OR term)*`, `term := atom (AND atom)*`, `atom := '(' expr ')' | comparison`. AND binds tighter than OR (standard).

Parentheses are supported. They matter because callers routinely append a global condition to a user-supplied expression, e.g. building `layer='a' OR layer='b'` and then appending ` AND _is_historical!='true'`. Without parentheses the AND binds only to the last OR branch (standard precedence), silently returning hidden records from the other branches.

func (*DB) FindIDsByTextSearch 

func (db *DB) FindIDsByTextSearch(indexName, fieldName, queryText string) ([]types.SearchResult, error)

FindIDsByTextSearch performs a search on the text index for a given query. It uses the BM25 ranking algorithm to score and sort documents based on relevance.

func (*DB) GetAllRelations added in v0.5.1 

func (db *DB) GetAllRelations(nodeID string, direction string) map[string][]string

GetAllRelations returns a map of all active relationships for a node. direction can be "out" or "in". Complexity: O(1) shard lock + slice iteration. Massively faster than prefix scan.

func (*DB) GetAllValidNodeIDs added in v0.6.3 

func (s *DB) GetAllValidNodeIDs(indexName string) (*roaring.Bitmap, error)

GetAllValidNodeIDs returns a bitmap of every live (non-deleted) node in an index. Caller-visible counterpart of getAllValidNodeIDsLocked; used by the memory-visibility guard to build a starting set when no caller filter is present.

func (*DB) GetGraphShard added in v0.5.1 

func (db *DB) GetGraphShard(id string) *GraphShard

GetGraphShard returns a specific shard for read-only or single-node operations.

func (*DB) GetInEdges added in v0.5.1 

func (db *DB) GetInEdges(targetID, relationType string, atTime int64) ([]string, bool)

GetInEdges returns all incoming source IDs for a target node. Resolves the Reverse Index instantly.

func (*DB) GetKVStore 

func (s *DB) GetKVStore() *KVStore

GetKVStore returns the underlying key-value store

func (*DB) GetMetadataForNode added in v0.5.1 

func (s *DB) GetMetadataForNode(indexName string, nodeID uint32) map[string]any

GetMetadataForNode exposes the internal metadata lookup.

func (*DB) GetMetadataForNodeUnlocked added in v0.4.7 

func (s *DB) GetMetadataForNodeUnlocked(indexName string, nodeID uint32) map[string]any

GetMetadataForNodeUnlocked exposes the internal metadata lookup. Warning: Caller must hold RLock on DB or ensure safety.

func (*DB) GetOutEdges added in v0.5.1 

func (db *DB) GetOutEdges(sourceID, relationType string, atTime int64) ([]GraphEdge, bool)

GetOutEdges returns all outgoing edges for a node of a specific type. It applies Time Travel filtering if atTime > 0. Returns a COPY of the active edges to prevent race conditions during iteration.

func (*DB) GetSingleVectorIndexInfoAPI 

func (s *DB) GetSingleVectorIndexInfoAPI(name string) (IndexInfo, error)

GetSingleVectorIndexInfoAPI returns information for a single index by name.

func (*DB) GetTextIndexMap added in v0.4.0 

func (s *DB) GetTextIndexMap(indexName string) (map[string]map[string]PostingList, bool)

GetTextIndexMap returns a copy of the available fields in the text index for a given index name. It returns map[fieldName]map[token]PostingList (though usually we just need keys). This is used for auto-detecting which field contains text.

func (*DB) GetVector 

func (s *DB) GetVector(indexName, vectorID string) (VectorData, error)

GetVector retrieves the complete data for a single vector given its external ID.

func (*DB) GetVectorIndex 

func (s *DB) GetVectorIndex(name string) (VectorIndex, bool)

GetVectorIndex retrieves a vector index by its name.

func (*DB) GetVectorIndexInfo 

func (s *DB) GetVectorIndexInfo() ([]IndexInfo, error)

GetVectorIndexInfo returns a slice containing the configuration and status of all existing vector indexes.

func (*DB) GetVectorIndexInfoAPI 

func (s *DB) GetVectorIndexInfoAPI() ([]IndexInfo, error)

GetVectorIndexInfoAPI returns configuration and status information for all vector indexes, suitable for an API response.

func (*DB) GetVectorIndexInfoUnlocked 

func (s *DB) GetVectorIndexInfoUnlocked() ([]IndexInfo, error)

GetVectorIndexInfoUnlocked returns index information without acquiring a lock. The caller is responsible for ensuring thread safety.

func (*DB) GetVectorIndexUnlocked added in v0.4.7 

func (s *DB) GetVectorIndexUnlocked(name string) (VectorIndex, bool)

GetVectorIndexUnlocked retrieves a vector index without acquiring a lock. The caller is responsible for ensuring thread safety.

func (*DB) GetVectors 

func (s *DB) GetVectors(indexName string, vectorIDs []string) ([]VectorData, error)

GetVectors retrieves the complete data for a slice of vector IDs in parallel. If an ID is not found, it is simply omitted from the returned slice.

func (*DB) IterateGraphEdges added in v0.5.1 

func (s *DB) IterateGraphEdges(cb func(source, target, rel string, weight float32, props []byte, cTime, dTime int64))

IterateGraphEdges iterates over all active and soft-deleted edges safely.

func (*DB) IterateKV 

func (s *DB) IterateKV(callback func(pair KVPair))

IterateKV iterates over all key-value pairs in the store, passing each to a callback function. The iteration is performed under a read lock. CONTRACT: the callback must NOT write to the KVStore (the RLock is held for the whole iteration — a write would deadlock) and must NOT retain or mutate the value slice (it is a defensive copy).

func (*DB) IterateKVUnlocked 

func (s *DB) IterateKVUnlocked(callback func(pair KVPair))

iterateKVUnlocked performs the iteration without acquiring locks. The caller is responsible for ensuring thread safety.

func (*DB) IterateVectorIndexes 

func (s *DB) IterateVectorIndexes(callback func(indexName string, index *hnsw.Index, data VectorData))

IterateVectorIndexes iterates over all vector indexes and their contents. The caller is responsible for locking.

func (*DB) IterateVectorIndexesUnlocked 

func (s *DB) IterateVectorIndexesUnlocked(callback func(indexName string, index *hnsw.Index, data VectorData))

iterateVectorIndexesUnlocked performs the iteration without acquiring locks. The caller is responsible for ensuring thread safety.

func (*DB) LoadFromSnapshot 

func (s *DB) LoadFromSnapshot(reader io.Reader, basePath string) error

LoadFromSnapshot deserializes a gob snapshot from an io.Reader and restores the store's state. It clears the current store state before loading.

func (*DB) Lock 

func (s *DB) Lock()

Lock acquires a write lock on the store.

func (*DB) LockTwoShards added in v0.5.1 

func (db *DB) LockTwoShards(id1, id2 string) (*GraphShard, *GraphShard)

LockTwoShards safely acquires exclusive write locks for two potentially different shards. It implements a Strict Lock Ordering (always lock the lower shard index first) to mathematically guarantee that Deadlocks cannot happen during cross-shard operations.

func (*DB) RLock 

func (s *DB) RLock()

RLock acquires a read lock on the store.

func (*DB) RUnlock 

func (s *DB) RUnlock()

RUnlock releases the read lock.

func (*DB) RemoveEdge added in v0.5.1 

func (db *DB) RemoveEdge(sourceID, targetID, relationType string, hardDelete bool, timestamp int64)

RemoveEdge removes a directed edge. If hardDelete is true, it physically filters the slice (O(N)). If hardDelete is false, it uses Soft Delete (O(1) in-place update) for Time Travel support.

func (*DB) RunMaintenance added in v0.3.0 

func (s *DB) RunMaintenance() bool

RunMaintenance triggers background tasks (Vacuum/Refine) on all vector indexes. It should be called periodically by the Engine.

func (*DB) Snapshot 

func (s *DB) Snapshot(writer io.Writer) error

Snapshot serializes the current state of the store in gob format to an io.Writer. This function expects the caller to handle locking.

func (*DB) Unlock 

func (s *DB) Unlock()

Unlock releases the write lock.

func (*DB) UnlockTwoShards added in v0.5.1 

func (db *DB) UnlockTwoShards(id1, id2 string)

UnlockTwoShards releases the locks acquired by LockTwoShards. The order of unlocking does not affect deadlock prevention, but we keep it clean.

func (*DB) VacuumGraph added in v0.5.1 

func (db *DB) VacuumGraph(cutoffTime int64) int

VacuumGraph physically removes edges that have been soft-deleted for longer than the cutoff time. It returns the number of edges physically deleted. It cleans up empty nodes to prevent memory leaks over time.

type GraphEdge added in v0.5.1 

type GraphEdge struct {
	TargetID  string          `json:"t"`
	CreatedAt int64           `json:"c"`
	DeletedAt int64           `json:"d,omitempty"`
	Weight    float32         `json:"w,omitempty"`
	Props     json.RawMessage `json:"p,omitempty"` // json.RawMessage, shares underlying array
}

GraphEdge represents the full, outgoing edge. It acts as the Single Source of Truth (SSOT) for relationship data (weight, properties). Size: ~64 bytes.

type GraphNode added in v0.5.1 

type GraphNode struct {
	ID       string
	OutEdges map[string][]GraphEdge   // RelationType -> List of outgoing edges
	InEdges  map[string][]ReverseEdge // RelationType -> List of incoming edges
}

GraphNode holds the adjacency lists for a single entity in RAM. We use slices instead of maps for the inner collections to avoid pointer-chasing and GC overhead. Soft deletes will be handled by updating 'DeletedAt' in place.

func NewGraphNode added in v0.5.1 

func NewGraphNode(id string) *GraphNode

NewGraphNode initializes a node with empty maps to prevent panics.

type GraphShard added in v0.5.1 

type GraphShard struct {
	// contains filtered or unexported fields
}

GraphShard represents an independent, lockable partition of the graph.

type GraphSnapshot added in v0.5.1 

type GraphSnapshot struct {
	Nodes map[string]*GraphNode
}

GraphSnapshot represents the state of a single graph shard for serialization.

type IndexConfig 

type IndexConfig struct {
	Metric         distance.DistanceMetric
	Precision      distance.PrecisionType
	M              int
	EfConstruction int
	TextLanguage   string // e.g., "english", "italian", or "" to disable
	AutoLinks      []hnsw.AutoLinkRule
	MemoryConfig   hnsw.MemoryConfig `json:"memory_config,omitempty"`
}

IndexConfig holds the configuration parameters for a vector index.

type IndexInfo 

type IndexInfo struct {
	Name           string                  `json:"name"`
	Metric         distance.DistanceMetric `json:"metric"`
	Precision      distance.PrecisionType  `json:"precision"`
	M              int                     `json:"m"`
	EfConstruction int                     `json:"ef_construction"`
	VectorCount    int                     `json:"vector_count"`
	TextLanguage   string                  `json:"text_language"`
}

IndexInfo models the public-facing information about a vector index, intended for serialization in API responses.

type IndexSnapshot 

type IndexSnapshot struct {
	Config             IndexConfig
	VectorDim          int
	Nodes              map[uint32]*NodeSnapshot // Using a map for simpler serialization/deserialization.
	ExternalToInternal map[string]uint32
	InternalCounter    uint32
	EntrypointID       uint32
	MaxLevel           int
	QuantizerState     *distance.Quantizer // Saves the quantizer's state.
	QuantizedNorms     []float32
	MaintenanceConfig  hnsw.AutoMaintenanceConfig
	ArenaState         mmap.ArenaState `json:"arena_state,omitempty"`
}

IndexSnapshot represents the serializable state of a single vector index. It includes the index configuration and all its node data.

type InvertedIndex 

type InvertedIndex map[string]map[string]map[string]PostingList

InvertedIndex is the data structure for full-text search. It maps a token (word) to a list of documents containing it. Structure: map[vector_index_name] -> map[metadata_key] -> map[token] -> PostingList

type KVPair 

type KVPair struct {
	Key   string
	Value []byte
}

KVPair is a struct for returning key-value pairs.

type KVStore 

type KVStore struct {
	// contains filtered or unexported fields
}

KVStore is a thread-safe, in-memory key-value store. It uses a sync.RWMutex to manage concurrent access, allowing for multiple concurrent readers or a single exclusive writer.

func NewKVStore 

func NewKVStore() *KVStore

NewKVStore creates and returns a new, empty KVStore instance.

func (*KVStore) Delete 

func (s *KVStore) Delete(key string)

Delete removes a key and its associated value from the store. This is a write operation and is fully thread-safe

func (*KVStore) Get 

func (s *KVStore) Get(key string) ([]byte, bool)

Get retrieves the value for a given key. It returns the value and a boolean indicating whether the key was found. This is a read operation and is safe for concurrent use. The returned slice is a copy: mutating it never affects the stored state.

func (*KVStore) GetKeysWithPrefix added in v0.5.1 

func (s *KVStore) GetKeysWithPrefix(prefix string, limit int) []string

GetKeysWithPrefix returns a slice of keys that start with the given prefix. It limits the number of returned keys to 'limit' to prevent memory spikes. If limit <= 0, returns all matching keys.

func (*KVStore) Keys added in v0.4.7 

func (s *KVStore) Keys() []string

Keys returns a snapshot of all keys currently in the store. This operation holds a read lock while copying keys, so it's relatively fast but linear to the number of keys.

func (*KVStore) RLock 

func (s *KVStore) RLock()

RLock locks the store for reading. It should be used in conjunction with RUnlock for operations that require a consistent read-only view across multiple steps at a higher level.

func (*KVStore) RUnlock 

func (s *KVStore) RUnlock()

RUnlock unlocks the store after a read lock. It must be called once for every call to RLock.

func (*KVStore) Set 

func (s *KVStore) Set(key string, value []byte)

Set adds or updates a value for a given key. This is a write operation and is fully thread-safe. The value is copied: later mutations of the caller's slice never affect the stored state (defensive copy — the KV holds auth keys and tokens).

type NodeSnapshot 

type NodeSnapshot struct {
	NodeData *hnsw.Node             // The graph data (ID, connections, etc.)
	Metadata map[string]interface{} // The associated metadata
}

NodeSnapshot contains all the necessary data to restore a single node.

func (*NodeSnapshot) GobDecode added in v0.5.1 

func (ns *NodeSnapshot) GobDecode(data []byte) error

GobDecode implements custom gob decoding to restore metadata from JSON.

func (NodeSnapshot) GobEncode added in v0.5.1 

func (ns NodeSnapshot) GobEncode() ([]byte, error)

GobEncode implements custom gob encoding to handle interface{} metadata. Converts metadata to JSON to avoid gob serialization issues with []interface{}.

type PostingEntry 

type PostingEntry struct {
	DocID         uint32
	TermFrequency int
}

PostingEntry contains the document ID and term frequency for an entry in a posting list.

type PostingList 

type PostingList []PostingEntry

PostingList is a slice of PostingEntry structs.

type ReverseEdge added in v0.5.1 

type ReverseEdge struct {
	SourceID  string `json:"s"`
	CreatedAt int64  `json:"c"`
	DeletedAt int64  `json:"d,omitempty"`
}

ReverseEdge represents a lightweight incoming edge. It is used exclusively for topology navigation and time-travel filtering. Size: ~32 bytes (saves 50% RAM compared to full GraphEdge).

type Snapshot 

type Snapshot struct {
	KVData     map[string][]byte
	VectorData map[string]*IndexSnapshot
	GraphData  [NumGraphShards]GraphSnapshot
}

Snapshot represents the complete serializable state of the database.

type TextIndexStats 

type TextIndexStats struct {
	// Total number of documents in an indexed field
	TotalDocs int
	// Average length of a field across all documents
	AvgFieldLength float64
	// TotalDocLength is the sum of all DocLengths — an incremental counter
	// that keeps AvgFieldLength computable in O(1) on every add/remove
	// (bulk ingestion previously recomputed it with an O(N) scan per call).
	TotalDocLength int64
	// Map of DocID -> field length
	DocLengths map[uint32]int
}

TextIndexStats holds statistics for a text index, used by ranking algorithms like BM25.

type VectorData 

type VectorData struct {
	ID       string         `json:"id"`
	Vector   []float32      `json:"vector"`
	Metadata map[string]any `json:"metadata"`
}

VectorData is a struct for returning the complete data of a single vector.

type VectorIndex 

type VectorIndex interface {
	// Add inserts a new vector with a unique external ID into the index.
	// It returns an internal ID used for efficient lookups.
	Add(id string, vector []float32) (uint32, error)
	// Delete removes a vector from the index using its external ID.
	Delete(id string)
	// SearchWithScores finds the K nearest neighbors to a query vector,
	// returning their internal IDs and scores (distances).
	// It supports pre-filtering with an allowList and index-specific search parameters like efSearch.
	SearchWithScores(query []float32, k int, allowList *roaring.Bitmap, efSearch int) []types.SearchResult

	// Metric returns the distance metric used by the index (e.g., Euclidean, Cosine).
	Metric() distance.DistanceMetric
	// Precision returns the data type precision used for storing vectors (e.g., float32, int8).
	Precision() distance.PrecisionType
	// GetArenaDir returns the directory path for the vector arena (mmap files).
	// Returns empty string if no arena is used.
	GetArenaDir() string
	Close() error
}

VectorIndex defines the operations that a vector index must support. This interface allows for different index implementations (e.g., Brute Force, HNSW) to be used interchangeably within the database.

Source Files

View all Source files

Directories

Path Synopsis
Package distance provides functions for calculating vector distances.
 Package distance provides functions for calculating vector distances.
gen command
File: internal/store/distance/gen/main.go
 File: internal/store/distance/gen/main.go
Package hnsw provides the implementation of the Hierarchical Navigable Small World graph algorithm for efficient approximate nearest neighbor search.
 Package hnsw provides the implementation of the Hierarchical Navigable Small World graph algorithm for efficient approximate nearest neighbor search.
Package text provides utilities for text processing, such as document chunking.
 Package text provides utilities for text processing, such as document chunking.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.
⌂ 换网址