compaction

package
v1.0.0-beta.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package compaction trades an event store's old history for one full-state event per surviving aggregate, and moves everything it retires into a portable archive file so nothing is actually lost.

An event store that has run for a long time carries history nobody reads any more: a resource that changed five hundred times still replays five hundred events to answer one question. Compact collapses that history at or below a watermark into a single compaction event carrying the aggregate's full state, and writes the retired events to an archive in the same newline-delimited JSON format the migration package exports.

The order of work is the safety argument

For each batch the archive is written and fsynced first, the compaction events are appended second, and only then is anything deleted. Every failure before the delete therefore leaves the store exactly as it was, and every delete happens with the archive already durable. Because deletes come last, positions are never reused: a retired position becomes a permanent gap in the global feed rather than a number a later event might claim.

As a sequence (mermaid; rendered in docs/decisions/0011):

sequenceDiagram
    participant C as Compact
    participant P as StateProvider
    participant A as Archive (JSONL)
    participant S as CompactableEventStore
    Note over C: Build the whole plan first. A provider failure aborts here.
    loop every aggregate in the plan
        C->>P: state(aggregateID)
        P-->>C: full state
    end
    loop every batch
        C->>A: 1. write retired events
        C->>A: 2. fsync
        C->>S: 3. Append(compaction event, snapshot metadata)
        rect rgb(235, 245, 255)
            Note right of S: one transaction
            C->>S: 4. RetireEvents(ids, manifest)
            S->>S: delete retired rows
            S->>S: record manifest
        end
    end

Deleting is transactional per batch, and each batch's archive segment is recorded as a manifest in the same transaction, so an interrupted run resumes at the first batch it never recorded instead of archiving and compacting the same history twice. A batch that fails after its bytes are already in the archive has them cut back off (see ArchiveFile), so the archive holds only segments a manifest accounts for.

One window stays open: a hard crash between a batch's fsync and its delete commit leaves a durable segment that no manifest covers and no rollback ever ran for. The next run resumes from the last recorded manifest and archives those events again, so an archive that survived a crash can hold an event twice. Restore such an archive with migration.ImportOptions.SkipExisting, which skips an event already present instead of colliding on its ID; the duplicated span is the one below the first manifest's FromPosition, and each manifest's Checksum identifies the segment it does account for.

Compaction does not consult subscriber checkpoints

Compact deletes the history below its watermark whether or not a background subscriber has read it. A subscriber whose checkpoint is below the watermark never sees the retired events: its next ReadAfter skips straight over the gap they left, and it then receives the compaction events as ordinary feed entries carrying the caller's own event type. Choose a watermark below every live subscriber's checkpoint, or make those handlers treat the compaction event type as a full-state replacement.

What stays out

What to retain is policy, and this package only provides the hook (Retain) and honours it. What the state of an aggregate actually is belongs to the caller, which is why StateProvider is a parameter rather than something compaction works out for itself.

Index

Constants

View Source
const (
	// MetadataCompactedFrom is the compaction event's metadata key holding the
	// lowest global position the event replaces for its own aggregate.
	MetadataCompactedFrom = "compacted_from"

	// MetadataCompactedTo is the compaction event's metadata key holding the
	// run's watermark — the global position the run compacted up to. It is the
	// same on every compaction event a run appends.
	MetadataCompactedTo = "compacted_to"

	// DefaultBatchSize is how many retiring events one batch archives, appends
	// for, and deletes when Options.BatchSize is not set.
	DefaultBatchSize = 500
)

Variables

View Source
var (
	// ErrArchiveNotSyncable is returned when the archive destination cannot be
	// fsynced. A plain writer would report a clean success while the archive
	// sat in a page cache, so compaction refuses it rather than deleting
	// behind it.
	ErrArchiveNotSyncable = errors.New("compaction archive cannot be fsynced")

	// ErrMissingArchive is returned when Options.Archive is nil.
	ErrMissingArchive = errors.New("compaction requires an archive destination")

	// ErrMissingEventType is returned when Options.EventType is empty. The
	// compaction event's type is the caller's to choose (e.g.
	// "Resource.Compacted") because it lands in the caller's own event stream.
	ErrMissingEventType = errors.New("compaction requires an event type")

	// ErrMissingStateProvider is returned when Options.State is nil.
	ErrMissingStateProvider = errors.New("compaction requires a state provider")
)

Functions

func DefaultIsDelete

func DefaultIsDelete(event domain.EventEnvelope[any]) bool

DefaultIsDelete recognises an event as retiring its aggregate when its type ends in "Deleted", the convention the library's own event types follow. A caller whose domain retires things differently supplies its own predicate through Options.IsDelete rather than renaming its events to suit compaction.

Types

type ArchiveFile

type ArchiveFile interface {
	Stat() (os.FileInfo, error)
	Truncate(size int64) error
}

ArchiveFile is the optional ability of an archive to report its current length and be cut back to a previous one. *os.File satisfies it, and an archive that does buys two things a bare writer cannot have:

  • the version header goes only into an archive that is empty, so a run appending to an archive an earlier run started never writes a second header into the middle of the file, where no importer would read it;
  • a batch that fails after its bytes are already down is rolled back, so the archive never keeps a segment that no manifest accounts for and a later run cannot archive the same events twice.

Without it Compact falls back to deciding the header from the resume position, and reports a failed batch's stray segment in the error instead of removing it.

type Options

type Options struct {
	// Watermark is the global position the run compacts up to. Every event at
	// or below it is a candidate; everything above it is left alone.
	Watermark int64

	// FromPosition resumes an archive that already holds the history up to
	// this position, so the run starts above it. Leave it zero for a fresh
	// archive.
	//
	// It is a floor, not the whole resume story: recorded manifests raise the
	// starting position further, so an interrupted run resumes correctly even
	// when the caller passes nothing. It does not decide whether the archive
	// gets a version header — that follows from whether the destination is
	// empty (see ArchiveFile), so leaving this zero while resuming from a
	// manifest cannot put a header in the middle of a file.
	FromPosition int64

	// EventType is the type given to every compaction event the run appends,
	// e.g. "Resource.Compacted". Required.
	EventType string

	// State supplies the full state each compaction event carries. Required.
	State StateProvider

	// Archive is where retired events are written, as newline-delimited JSON
	// in the same shape migration.Export produces. It must implement Syncer.
	// Required.
	Archive io.Writer

	// Retain holds chosen events back from both compaction and the archive.
	Retain Retain

	// IsDelete decides whether an event retires its aggregate. An aggregate
	// whose last event is a delete gets no compaction event — its history goes
	// to the archive only. Defaults to DefaultIsDelete.
	IsDelete func(event domain.EventEnvelope[any]) bool

	// BatchSize is how many retiring events one batch archives, appends for,
	// and deletes together — and so how much of a run is lost when one batch
	// fails. Values <= 0 use DefaultBatchSize.
	//
	// It does not bound the run's memory: Compact reads every event in scope
	// before it writes anything, because the whole plan (each aggregate's
	// state included) has to exist before the first byte is archived for a
	// failure to leave the store untouched. Memory therefore scales with the
	// number of events at or below the watermark, not with BatchSize; compact
	// a long history in several runs with rising watermarks rather than one.
	BatchSize int
}

Options configures one compaction run.

type Report

type Report struct {
	// Archived is the number of events written to the archive and deleted.
	Archived int

	// CompactionEvents is the number of compaction events appended.
	CompactionEvents int

	// Manifests are the batch records this run wrote, in the order it wrote
	// them. On a failure it holds the batches that did complete.
	Manifests []domain.CompactionManifest

	// LastPosition is the highest global position the run retired, or the
	// position it resumed from when it retired nothing.
	LastPosition int64
}

Report summarizes a completed — or partially completed — run.

func Compact

func Compact(ctx context.Context, store domain.EventStore, opts Options) (Report, error)

Compact collapses store's history at or below opts.Watermark into one full-state event per surviving aggregate and moves the retired events to opts.Archive.

store must implement domain.CompactableEventStore; anything else is refused with domain.ErrCompactionNotSupported before a single event is read, since archiving history a store can never delete would be pure loss of time and disk. An archive that cannot be fsynced is refused just as early.

The returned Report describes what the run did, and is meaningful even when the run fails partway: every batch it lists was archived, appended for, and deleted, and every batch it does not list left the store untouched.

type Retain

type Retain struct {
	// EventTypes retains events whose type is exactly one of these.
	EventTypes []string

	// EventTypeMatch retains events whose type the predicate accepts, for
	// policies a fixed list cannot express (a prefix, a suffix, a namespace).
	EventTypeMatch func(eventType string) bool

	// NotBefore retains events created at or after this time. The zero time
	// retains nothing on this condition.
	NotBefore time.Time
}

Retain holds chosen events back from both compaction and the archive: a retained event is neither archived nor deleted, and stays in the live store below the compaction event. What to retain is policy set by the caller; this type is only the hook.

The conditions are a union — an event is retained if any of them matches. The zero value retains nothing.

func (Retain) Holds

func (r Retain) Holds(event domain.EventEnvelope[any]) bool

Holds reports whether the retention policy keeps this event out of the run.

type StateProvider

type StateProvider func(ctx context.Context, aggregateID string) (any, error)

StateProvider returns an aggregate's current full state — the payload the compaction event will carry.

It is asked for the state as the aggregate stands now, including any events above the watermark. The compaction event is appended at sequence max + 1, so it is the last thing replay applies; a snapshot of the state as at the watermark would sit above the surviving events and silently undo them on every rehydration.

An error aborts the whole run before anything is written or deleted.

type Syncer

type Syncer interface {
	Sync() error
}

Syncer is the archive's durability contract. Compact refuses an archive that does not implement it (*os.File does).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址