Documentation
¶
Overview ¶
Package storefts wraps github.com/blevesearch/bleve/v2 to provide the full-text search surface consumed by IMAP SEARCH and JMAP Email/query. One index, two query paths (REQ-PROTO-47).
Ownership: storage-implementor.
Index ¶
- Constants
- type Composite
- func (c *Composite) Commit(ctx context.Context) error
- func (c *Composite) IndexMessage(ctx context.Context, msg store.Message, text string) error
- func (c *Composite) Query(ctx context.Context, principalID store.PrincipalID, q store.Query) ([]store.MessageRef, error)
- func (c *Composite) ReadChangeFeedForFTS(ctx context.Context, cursor uint64, max int) ([]store.FTSChange, error)
- func (c *Composite) RemoveMessage(ctx context.Context, id store.MessageID) error
- type Index
- func (i *Index) Close() error
- func (i *Index) Commit(ctx context.Context) error
- func (i *Index) Delete(ctx context.Context, id store.MessageID) error
- func (i *Index) IndexChatMessage(ctx context.Context, msg store.ChatMessage) error
- func (i *Index) IndexMessage(ctx context.Context, msg store.Message, text string) error
- func (i *Index) IndexMessageFull(ctx context.Context, principalID store.PrincipalID, msg store.Message, ...) error
- func (i *Index) PendingBytes() int
- func (i *Index) PendingSize() int
- func (i *Index) Query(ctx context.Context, principalID store.PrincipalID, q store.Query) ([]store.MessageRef, error)
- func (i *Index) RemoveChatMessage(ctx context.Context, id store.ChatMessageID) error
- func (i *Index) RemoveMessage(ctx context.Context, id store.MessageID) error
- func (i *Index) SearchChatMessages(ctx context.Context, q string, conversationIDs []store.ConversationID, ...) ([]store.ChatMessageID, error)
- type MailparseExtractor
- type TextExtractor
- type Worker
- type WorkerOptions
Constants ¶
const ( DefaultBatchSize = 500 DefaultMaxBatchBytes = 256 * 1024 * 1024 DefaultFlushInterval = 500 * time.Millisecond DefaultCursorKey = "fts" )
Default WorkerOptions values. Originally derived from docs/design/server/notes/spike-fts-cadence.md (peak throughput at batch=2000, sub-second visibility at 500 ms commit ceiling). Lowered 2026-05-09 after a 144 GiB resident incident: the spike measured throughput on small synthetic docs but did not bound peak memory; a real corpus with multi-MiB extracted bodies can hold many GiB in a single 2000-doc Bleve batch before the doc-count ceiling trips. The new default pairs a smaller doc-count ceiling with a hard byte ceiling (DefaultMaxBatchBytes) so neither dimension can run away.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Composite ¶
type Composite struct {
// contains filtered or unexported fields
}
Composite implements store.FTS by routing the Bleve-backed surface to *Index and the SQL-backed change-feed read to a per-backend delegate (the storesqlite / storepg ftsStub). The split exists because the change feed lives in the relational state_changes table the backend owns, while the search corpus lives in the Bleve index storefts owns.
Production wires this up in admin/server.go after constructing the *Index, replacing store.Store.FTS() so the JMAP Email/query and IMAP SEARCH paths read from Bleve instead of the substring stub.
func NewComposite ¶
NewComposite returns a store.FTS that serves IndexMessage, RemoveMessage, Query, and Commit from idx and forwards ReadChangeFeedForFTS to delegate. delegate is typically the per-backend stub returned by store.FTS() before composition.
func (*Composite) IndexMessage ¶
IndexMessage writes the FTS document via the underlying *Index. The principal-scoped variant (IndexMessageFull) used by Worker stays on *Index directly; this method exists for callers that hold only a store.FTS and accepts the principal-less signature the interface defines.
func (*Composite) Query ¶
func (c *Composite) Query(ctx context.Context, principalID store.PrincipalID, q store.Query) ([]store.MessageRef, error)
func (*Composite) ReadChangeFeedForFTS ¶
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index is a Bleve-backed implementation of store.FTS. One index covers all principals; queries filter by principal_id so a single index serves both the IMAP SEARCH and JMAP Email/query surfaces (REQ-PROTO-47).
Writes use a pending batch that is flushed on Commit, on Delete, or when size ceiling is reached. The worker drives the cadence (size OR 500 ms) per docs/design/server/notes/spike-fts-cadence.md.
func New ¶
New opens (or creates) a Bleve index at dir/bleve. The directory is created if it does not exist. The clock is used for produced-at-style timestamps in tests; production uses clock.NewReal.
func (*Index) Close ¶
Close flushes any pending batch and closes the underlying Bleve index. Must be called exactly once on shutdown.
func (*Index) Commit ¶
Commit flushes the pending batch to the index backend. Safe to call on an empty batch (returns nil). Callers trigger it on size OR time per the spike recommendations.
func (*Index) Delete ¶
Delete is an alias for RemoveMessage that flushes immediately. Used by call sites that want "gone now" semantics (e.g. a reindex rebuild clearing a stale doc). Callers inside the worker use RemoveMessage + Commit to keep deletions batched with creates.
func (*Index) IndexChatMessage ¶
IndexChatMessage writes (or replaces) the FTS document for a chat message (Wave 2.9.6 Track D, REQ-CHAT-80..82). The document carries the conversation id (for membership-scoped retrieval), the sender principal id, the plain-text body (REQ-CHAT-81 — HTML is too noisy), and the creation instant. System messages and soft-deleted rows are rejected here; the caller (typically the FTS worker) is expected to hand them to RemoveChatMessage instead so a stale doc does not linger.
func (*Index) IndexMessage ¶
IndexMessage writes (or replaces) the FTS document for msg. The text argument is the pre-extracted plain text (body + attachment text); the worker performs the extraction upstream so this method does not touch the blob store. The batch is accumulated in memory; call Commit to flush, or let the worker's size/time trigger drive it.
func (*Index) IndexMessageFull ¶
func (i *Index) IndexMessageFull( ctx context.Context, principalID store.PrincipalID, msg store.Message, text string, ) error
IndexMessageFull is the worker-facing variant that takes the principal explicitly — the worker has it from the FTSChange and passes it in so the index can filter by owner. Kept separate from IndexMessage to stay bug-compatible with the store.FTS interface signature.
func (*Index) PendingBytes ¶
PendingBytes returns the cumulative body-text byte count of the pending batch. The worker pairs it with PendingSize to flush on whichever ceiling trips first; the byte ceiling exists because Bleve retains every doc's field values in memory until the batch flushes, so a 2000-doc batch of large messages can hold many GiB before the doc-count threshold trips.
func (*Index) PendingSize ¶
PendingSize returns the current accumulated batch size. The worker uses it to decide whether to flush on the size ceiling.
func (*Index) Query ¶
func (i *Index) Query( ctx context.Context, principalID store.PrincipalID, q store.Query, ) ([]store.MessageRef, error)
Query runs q against principalID's documents and returns hits in descending Score order. An empty Query matches all of the principal's messages (up to the limit). The principal filter is applied via a conjunction with a keyword term query on principal_id, so an empty free-text Query still scopes correctly.
func (*Index) RemoveChatMessage ¶
RemoveChatMessage deletes the chat-message document identified by id. The deletion is accumulated in the pending batch; Commit flushes it. Idempotent: removing a doc that was never indexed (e.g. a system message reaching the worker via the change feed) is a no-op.
func (*Index) RemoveMessage ¶
RemoveMessage deletes the document identified by id. The deletion is accumulated in the pending batch; Commit flushes it.
func (*Index) SearchChatMessages ¶
func (i *Index) SearchChatMessages( ctx context.Context, q string, conversationIDs []store.ConversationID, limit int, ) ([]store.ChatMessageID, error)
SearchChatMessages returns chat-message ids whose plain-text body matches q, scoped to the supplied conversation id set (REQ-CHAT-82 membership scope). The caller passes the conversations the requesting principal is a member of; the index restricts hits to that set via a disjunction on conversation_id so a non-member cannot search-hit a conversation they are not in. An empty conversationIDs slice returns no hits — the membership check happens in the caller.
Hits are returned in descending relevance (Bleve Score) order, capped at limit (defaultQueryLimit when limit <= 0, hardQueryLimit ceiling).
type MailparseExtractor ¶
type MailparseExtractor struct {
// Options controls the mailparse strictness caps. Zero value uses
// NewParseOptions defaults; operators may tighten or relax
// per-deployment (e.g. tolerate StrictCharset failures that come
// from the wild).
Options mailparse.ParseOptions
// PerAttachmentMaxBytes caps the extracted-text length for a single
// attachment; the default is 5 MiB per the FTS architecture spec.
// A zero value is treated as the default; pass a negative value
// (or use the test seam SetCaps) to disable the cap.
PerAttachmentMaxBytes int
// PerMessageMaxBytes caps the total extracted-text length across
// all attachments in a single message; default 20 MiB.
PerMessageMaxBytes int
}
MailparseExtractor is the production TextExtractor: it parses the blob through internal/mailparse, concatenates the text of every text/* leaf, then walks the message attachments and appends extracted text from supported binary formats (HTML, DOCX, XLSX, PPTX). PDF extraction is gated behind a separate optional extractor so the default build does not pull in a PDF parser.
Architecture: docs/design/server/architecture/02-storage-architecture.md §FTS specifies per-attachment and per-message extraction caps with a counter on truncation; PerAttachmentMaxBytes and PerMessageMaxBytes implement those bounds.
func NewMailparseExtractor ¶
func NewMailparseExtractor() *MailparseExtractor
NewMailparseExtractor returns a MailparseExtractor with production defaults. Callers override Options after construction if needed.
The FTS extractor uses mailparse.NewLenientParseOptions: the strict modes (Base64/QP/Boundary/Charset) NewParseOptions defaults to exist to reject silently corrupted messages on the delivery hot path, where letting a parse succeed against bad bytes would mask data loss. The FTS worker runs post-delivery: a message with a malformed-base64 attachment is already in the store and the user wants to find it via search. Aborting the extraction would leave the message un-indexed for the entire life of its row. Lenient parsing here means a malformed leaf is recorded in the part's DecodeErrors and skipped, while the cleanly-parseable leaves still contribute their text to the index.
func (*MailparseExtractor) Extract ¶
func (e *MailparseExtractor) Extract( ctx context.Context, _ store.Message, body io.Reader, ) (string, error)
Extract implements TextExtractor. The message argument is currently unused (the parser reads everything from body) but is retained in the signature so future extractors can tailor handling by flags, size, or MIME-from-cache without an API break.
type TextExtractor ¶
type TextExtractor interface {
// Extract consumes body and returns the plain-text representation of
// the message suitable for indexing. Implementations must not close
// body; the caller manages the reader lifecycle.
Extract(ctx context.Context, msg store.Message, body io.Reader) (string, error)
}
TextExtractor turns a message body byte stream into the plain text the FTS index will consume. Phase 1 wraps internal/mailparse to pull text/* leaves; Phase 1.5 layers on attachment extractors (PDF, DOCX, XLSX) that wrap an existing TextExtractor and append extracted attachment text to its output.
The extension point is deliberate: a new backend registers by implementing Extract and composing on top of MailparseExtractor; nothing in the worker assumes a specific implementation.
type Worker ¶
type Worker struct {
// contains filtered or unexported fields
}
Worker is the single async indexing goroutine. It reads the store's FTS change feed with a durable cursor, resolves each change to a message + extracted text, accumulates a Bleve batch, and flushes it on size OR time (whichever fires first). One worker covers the v1 100 msg/s target per the spike; sharding by principal is a future concern.
Lifecycle: construct with NewWorker; call Run(ctx) in a goroutine owned by the server lifecycle manager; cancel ctx to drain and return.
func NewWorker ¶
func NewWorker( idx *Index, st store.Store, extract TextExtractor, logger *slog.Logger, clk clock.Clock, opts WorkerOptions, ) *Worker
NewWorker constructs a worker. idx must be a *storefts.Index; st is the composed store.Store exposing the FTS change feed, metadata (GetMessage), and blob surfaces. extract turns a blob body into plain text for the index.
func (*Worker) CurrentFlushSignal ¶
func (w *Worker) CurrentFlushSignal() <-chan struct{}
CurrentFlushSignal returns the channel that will be closed by the next signalFlush. Capture this BEFORE the action that should trigger the flush (e.g. clock.Advance, a new change-feed insert) and then receive from it; this guarantees the channel observed by the caller is the same one the worker will close on its next commit, closing the race window WaitForFlush has when the worker outpaces the caller.
func (*Worker) Cursor ¶
Cursor returns the worker's current durable cursor. Exposed for tests and the operator diag surface; the worker mutates it under the hood.
func (*Worker) FlushForTest ¶
FlushForTest reads one batch of FTS changes from the feed, processes them, and commits. It returns true if any changes were found and indexed, false if the feed was empty (worker is idle).
TEST SEAM — do not call from production paths. Tests that need a deterministic, wall-clock-free way to drain the index use this method in a loop rather than driving the async Run goroutine via clock advances:
for {
changed, err := w.FlushForTest(ctx)
if err != nil { t.Fatal(err) }
if !changed { break }
}
Callers must NOT start Run concurrently on the same worker, as processBatch is not safe to call from multiple goroutines simultaneously.
func (*Worker) Lag ¶
Lag returns the wall-time delta between now and the ProducedAt of the most recent change indexed. Zero if the worker has never processed a change. Used by the metrics exporter to expose herold_fts_indexing_lag_seconds.
func (*Worker) Run ¶
Run consumes the FTS change feed until ctx is cancelled. It is a long-running call; callers run it in a single goroutine managed by the server lifecycle. Returns nil on ctx cancellation; non-nil on a fatal error from the store or index.
func (*Worker) WaitForFlush ¶
WaitForFlush blocks until the worker has committed at least one batch after the call. Tests use it to synchronise deterministically with the worker's single goroutine; it returns ctx.Err() if the context is cancelled first. Not part of the store.FTS surface.
Race note: WaitForFlush captures the current flush broadcast at call time, so a flush that races ahead of the call (worker scheduled between the test's trigger and this call) is missed. For race-correct usage, capture CurrentFlushSignal() BEFORE the trigger and select on that channel directly. This entry point is retained for backward compatibility and the simple "WaitForFlush BEFORE trigger" pattern where the worker has not yet been awakened.
type WorkerOptions ¶
type WorkerOptions struct {
// BatchSize is the document-count ceiling that triggers a commit.
BatchSize int
// MaxBatchBytes is the cumulative body-text byte ceiling that
// triggers a commit. Bleve retains every doc's field values in the
// pending batch until it flushes, so a doc-count-only ceiling lets
// a backlog of large messages pin many GiB of resident memory.
// Whichever of BatchSize or MaxBatchBytes trips first wins.
MaxBatchBytes int64
// FlushInterval is the wall-time ceiling, measured from the first
// un-flushed change in the current batch, that triggers a commit.
FlushInterval time.Duration
// CursorKey names the worker's durable cursor slot. Kept in
// WorkerOptions for forward compatibility with a future per-shard
// worker layout; Phase 1 uses a single "fts" slot.
CursorKey string
}
WorkerOptions tunes the single async indexing worker. Zero fields fall back to the recommended defaults (500 / 256 MiB / 500 ms / "fts").