storefts

package
v0.0.0-...-abe11a3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package storefts wraps github.com/blevesearch/bleve/v2 to provide the full-text search surface consumed by IMAP SEARCH and JMAP Email/query. One index, two query paths (REQ-PROTO-47).

Ownership: storage-implementor.

Index

Constants

View Source
const (
	DefaultBatchSize     = 500
	DefaultMaxBatchBytes = 256 * 1024 * 1024
	DefaultFlushInterval = 500 * time.Millisecond
	DefaultCursorKey     = "fts"
)

Default WorkerOptions values. Originally derived from docs/design/server/notes/spike-fts-cadence.md (peak throughput at batch=2000, sub-second visibility at 500 ms commit ceiling). Lowered 2026-05-09 after a 144 GiB resident incident: the spike measured throughput on small synthetic docs but did not bound peak memory; a real corpus with multi-MiB extracted bodies can hold many GiB in a single 2000-doc Bleve batch before the doc-count ceiling trips. The new default pairs a smaller doc-count ceiling with a hard byte ceiling (DefaultMaxBatchBytes) so neither dimension can run away.

Variables

This section is empty.

Functions

This section is empty.

Types

type Composite

type Composite struct {
	// contains filtered or unexported fields
}

Composite implements store.FTS by routing the Bleve-backed surface to *Index and the SQL-backed change-feed read to a per-backend delegate (the storesqlite / storepg ftsStub). The split exists because the change feed lives in the relational state_changes table the backend owns, while the search corpus lives in the Bleve index storefts owns.

Production wires this up in admin/server.go after constructing the *Index, replacing store.Store.FTS() so the JMAP Email/query and IMAP SEARCH paths read from Bleve instead of the substring stub.

func NewComposite

func NewComposite(idx *Index, delegate store.FTS) *Composite

NewComposite returns a store.FTS that serves IndexMessage, RemoveMessage, Query, and Commit from idx and forwards ReadChangeFeedForFTS to delegate. delegate is typically the per-backend stub returned by store.FTS() before composition.

func (*Composite) Commit

func (c *Composite) Commit(ctx context.Context) error

func (*Composite) IndexMessage

func (c *Composite) IndexMessage(ctx context.Context, msg store.Message, text string) error

IndexMessage writes the FTS document via the underlying *Index. The principal-scoped variant (IndexMessageFull) used by Worker stays on *Index directly; this method exists for callers that hold only a store.FTS and accepts the principal-less signature the interface defines.

func (*Composite) Query

func (c *Composite) Query(ctx context.Context, principalID store.PrincipalID, q store.Query) ([]store.MessageRef, error)

func (*Composite) ReadChangeFeedForFTS

func (c *Composite) ReadChangeFeedForFTS(ctx context.Context, cursor uint64, max int) ([]store.FTSChange, error)

func (*Composite) RemoveMessage

func (c *Composite) RemoveMessage(ctx context.Context, id store.MessageID) error

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index is a Bleve-backed implementation of store.FTS. One index covers all principals; queries filter by principal_id so a single index serves both the IMAP SEARCH and JMAP Email/query surfaces (REQ-PROTO-47).

Writes use a pending batch that is flushed on Commit, on Delete, or when size ceiling is reached. The worker drives the cadence (size OR 500 ms) per docs/design/server/notes/spike-fts-cadence.md.

func New

func New(dir string, logger *slog.Logger, clk clock.Clock) (*Index, error)

New opens (or creates) a Bleve index at dir/bleve. The directory is created if it does not exist. The clock is used for produced-at-style timestamps in tests; production uses clock.NewReal.

func (*Index) Close

func (i *Index) Close() error

Close flushes any pending batch and closes the underlying Bleve index. Must be called exactly once on shutdown.

func (*Index) Commit

func (i *Index) Commit(ctx context.Context) error

Commit flushes the pending batch to the index backend. Safe to call on an empty batch (returns nil). Callers trigger it on size OR time per the spike recommendations.

func (*Index) Delete

func (i *Index) Delete(ctx context.Context, id store.MessageID) error

Delete is an alias for RemoveMessage that flushes immediately. Used by call sites that want "gone now" semantics (e.g. a reindex rebuild clearing a stale doc). Callers inside the worker use RemoveMessage + Commit to keep deletions batched with creates.

func (*Index) IndexChatMessage

func (i *Index) IndexChatMessage(ctx context.Context, msg store.ChatMessage) error

IndexChatMessage writes (or replaces) the FTS document for a chat message (Wave 2.9.6 Track D, REQ-CHAT-80..82). The document carries the conversation id (for membership-scoped retrieval), the sender principal id, the plain-text body (REQ-CHAT-81 — HTML is too noisy), and the creation instant. System messages and soft-deleted rows are rejected here; the caller (typically the FTS worker) is expected to hand them to RemoveChatMessage instead so a stale doc does not linger.

func (*Index) IndexMessage

func (i *Index) IndexMessage(ctx context.Context, msg store.Message, text string) error

IndexMessage writes (or replaces) the FTS document for msg. The text argument is the pre-extracted plain text (body + attachment text); the worker performs the extraction upstream so this method does not touch the blob store. The batch is accumulated in memory; call Commit to flush, or let the worker's size/time trigger drive it.

func (*Index) IndexMessageFull

func (i *Index) IndexMessageFull(
	ctx context.Context,
	principalID store.PrincipalID,
	msg store.Message,
	text string,
) error

IndexMessageFull is the worker-facing variant that takes the principal explicitly — the worker has it from the FTSChange and passes it in so the index can filter by owner. Kept separate from IndexMessage to stay bug-compatible with the store.FTS interface signature.

func (*Index) PendingBytes

func (i *Index) PendingBytes() int

PendingBytes returns the cumulative body-text byte count of the pending batch. The worker pairs it with PendingSize to flush on whichever ceiling trips first; the byte ceiling exists because Bleve retains every doc's field values in memory until the batch flushes, so a 2000-doc batch of large messages can hold many GiB before the doc-count threshold trips.

func (*Index) PendingSize

func (i *Index) PendingSize() int

PendingSize returns the current accumulated batch size. The worker uses it to decide whether to flush on the size ceiling.

func (*Index) Query

func (i *Index) Query(
	ctx context.Context,
	principalID store.PrincipalID,
	q store.Query,
) ([]store.MessageRef, error)

Query runs q against principalID's documents and returns hits in descending Score order. An empty Query matches all of the principal's messages (up to the limit). The principal filter is applied via a conjunction with a keyword term query on principal_id, so an empty free-text Query still scopes correctly.

func (*Index) RemoveChatMessage

func (i *Index) RemoveChatMessage(ctx context.Context, id store.ChatMessageID) error

RemoveChatMessage deletes the chat-message document identified by id. The deletion is accumulated in the pending batch; Commit flushes it. Idempotent: removing a doc that was never indexed (e.g. a system message reaching the worker via the change feed) is a no-op.

func (*Index) RemoveMessage

func (i *Index) RemoveMessage(ctx context.Context, id store.MessageID) error

RemoveMessage deletes the document identified by id. The deletion is accumulated in the pending batch; Commit flushes it.

func (*Index) SearchChatMessages

func (i *Index) SearchChatMessages(
	ctx context.Context,
	q string,
	conversationIDs []store.ConversationID,
	limit int,
) ([]store.ChatMessageID, error)

SearchChatMessages returns chat-message ids whose plain-text body matches q, scoped to the supplied conversation id set (REQ-CHAT-82 membership scope). The caller passes the conversations the requesting principal is a member of; the index restricts hits to that set via a disjunction on conversation_id so a non-member cannot search-hit a conversation they are not in. An empty conversationIDs slice returns no hits — the membership check happens in the caller.

Hits are returned in descending relevance (Bleve Score) order, capped at limit (defaultQueryLimit when limit <= 0, hardQueryLimit ceiling).

type MailparseExtractor

type MailparseExtractor struct {
	// Options controls the mailparse strictness caps. Zero value uses
	// NewParseOptions defaults; operators may tighten or relax
	// per-deployment (e.g. tolerate StrictCharset failures that come
	// from the wild).
	Options mailparse.ParseOptions

	// PerAttachmentMaxBytes caps the extracted-text length for a single
	// attachment; the default is 5 MiB per the FTS architecture spec.
	// A zero value is treated as the default; pass a negative value
	// (or use the test seam SetCaps) to disable the cap.
	PerAttachmentMaxBytes int

	// PerMessageMaxBytes caps the total extracted-text length across
	// all attachments in a single message; default 20 MiB.
	PerMessageMaxBytes int
}

MailparseExtractor is the production TextExtractor: it parses the blob through internal/mailparse, concatenates the text of every text/* leaf, then walks the message attachments and appends extracted text from supported binary formats (HTML, DOCX, XLSX, PPTX). PDF extraction is gated behind a separate optional extractor so the default build does not pull in a PDF parser.

Architecture: docs/design/server/architecture/02-storage-architecture.md §FTS specifies per-attachment and per-message extraction caps with a counter on truncation; PerAttachmentMaxBytes and PerMessageMaxBytes implement those bounds.

func NewMailparseExtractor

func NewMailparseExtractor() *MailparseExtractor

NewMailparseExtractor returns a MailparseExtractor with production defaults. Callers override Options after construction if needed.

The FTS extractor uses mailparse.NewLenientParseOptions: the strict modes (Base64/QP/Boundary/Charset) NewParseOptions defaults to exist to reject silently corrupted messages on the delivery hot path, where letting a parse succeed against bad bytes would mask data loss. The FTS worker runs post-delivery: a message with a malformed-base64 attachment is already in the store and the user wants to find it via search. Aborting the extraction would leave the message un-indexed for the entire life of its row. Lenient parsing here means a malformed leaf is recorded in the part's DecodeErrors and skipped, while the cleanly-parseable leaves still contribute their text to the index.

func (*MailparseExtractor) Extract

func (e *MailparseExtractor) Extract(
	ctx context.Context,
	_ store.Message,
	body io.Reader,
) (string, error)

Extract implements TextExtractor. The message argument is currently unused (the parser reads everything from body) but is retained in the signature so future extractors can tailor handling by flags, size, or MIME-from-cache without an API break.

type TextExtractor

type TextExtractor interface {
	// Extract consumes body and returns the plain-text representation of
	// the message suitable for indexing. Implementations must not close
	// body; the caller manages the reader lifecycle.
	Extract(ctx context.Context, msg store.Message, body io.Reader) (string, error)
}

TextExtractor turns a message body byte stream into the plain text the FTS index will consume. Phase 1 wraps internal/mailparse to pull text/* leaves; Phase 1.5 layers on attachment extractors (PDF, DOCX, XLSX) that wrap an existing TextExtractor and append extracted attachment text to its output.

The extension point is deliberate: a new backend registers by implementing Extract and composing on top of MailparseExtractor; nothing in the worker assumes a specific implementation.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker is the single async indexing goroutine. It reads the store's FTS change feed with a durable cursor, resolves each change to a message + extracted text, accumulates a Bleve batch, and flushes it on size OR time (whichever fires first). One worker covers the v1 100 msg/s target per the spike; sharding by principal is a future concern.

Lifecycle: construct with NewWorker; call Run(ctx) in a goroutine owned by the server lifecycle manager; cancel ctx to drain and return.

func NewWorker

func NewWorker(
	idx *Index,
	st store.Store,
	extract TextExtractor,
	logger *slog.Logger,
	clk clock.Clock,
	opts WorkerOptions,
) *Worker

NewWorker constructs a worker. idx must be a *storefts.Index; st is the composed store.Store exposing the FTS change feed, metadata (GetMessage), and blob surfaces. extract turns a blob body into plain text for the index.

func (*Worker) CurrentFlushSignal

func (w *Worker) CurrentFlushSignal() <-chan struct{}

CurrentFlushSignal returns the channel that will be closed by the next signalFlush. Capture this BEFORE the action that should trigger the flush (e.g. clock.Advance, a new change-feed insert) and then receive from it; this guarantees the channel observed by the caller is the same one the worker will close on its next commit, closing the race window WaitForFlush has when the worker outpaces the caller.

func (*Worker) Cursor

func (w *Worker) Cursor() uint64

Cursor returns the worker's current durable cursor. Exposed for tests and the operator diag surface; the worker mutates it under the hood.

func (*Worker) FlushForTest

func (w *Worker) FlushForTest(ctx context.Context) (bool, error)

FlushForTest reads one batch of FTS changes from the feed, processes them, and commits. It returns true if any changes were found and indexed, false if the feed was empty (worker is idle).

TEST SEAM — do not call from production paths. Tests that need a deterministic, wall-clock-free way to drain the index use this method in a loop rather than driving the async Run goroutine via clock advances:

for {
    changed, err := w.FlushForTest(ctx)
    if err != nil { t.Fatal(err) }
    if !changed { break }
}

Callers must NOT start Run concurrently on the same worker, as processBatch is not safe to call from multiple goroutines simultaneously.

func (*Worker) Lag

func (w *Worker) Lag() time.Duration

Lag returns the wall-time delta between now and the ProducedAt of the most recent change indexed. Zero if the worker has never processed a change. Used by the metrics exporter to expose herold_fts_indexing_lag_seconds.

func (*Worker) Run

func (w *Worker) Run(ctx context.Context) error

Run consumes the FTS change feed until ctx is cancelled. It is a long-running call; callers run it in a single goroutine managed by the server lifecycle. Returns nil on ctx cancellation; non-nil on a fatal error from the store or index.

func (*Worker) WaitForFlush

func (w *Worker) WaitForFlush(ctx context.Context) error

WaitForFlush blocks until the worker has committed at least one batch after the call. Tests use it to synchronise deterministically with the worker's single goroutine; it returns ctx.Err() if the context is cancelled first. Not part of the store.FTS surface.

Race note: WaitForFlush captures the current flush broadcast at call time, so a flush that races ahead of the call (worker scheduled between the test's trigger and this call) is missed. For race-correct usage, capture CurrentFlushSignal() BEFORE the trigger and select on that channel directly. This entry point is retained for backward compatibility and the simple "WaitForFlush BEFORE trigger" pattern where the worker has not yet been awakened.

type WorkerOptions

type WorkerOptions struct {
	// BatchSize is the document-count ceiling that triggers a commit.
	BatchSize int
	// MaxBatchBytes is the cumulative body-text byte ceiling that
	// triggers a commit. Bleve retains every doc's field values in the
	// pending batch until it flushes, so a doc-count-only ceiling lets
	// a backlog of large messages pin many GiB of resident memory.
	// Whichever of BatchSize or MaxBatchBytes trips first wins.
	MaxBatchBytes int64
	// FlushInterval is the wall-time ceiling, measured from the first
	// un-flushed change in the current batch, that triggers a commit.
	FlushInterval time.Duration
	// CursorKey names the worker's durable cursor slot. Kept in
	// WorkerOptions for forward compatibility with a future per-shard
	// worker layout; Phase 1 uses a single "fts" slot.
	CursorKey string
}

WorkerOptions tunes the single async indexing worker. Zero fields fall back to the recommended defaults (500 / 256 MiB / 500 ms / "fts").

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址