journalstore

package
v0.0.0-...-37ef50c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package journalstore provides append-only metering fact journal adapters: in-memory for tests/local use, and Bun-backed SQLite/Postgres for durable deployments. Adapters implement pkg/aipsdk/metering Recorder and Querier without coupling to live authority counters or proprietary financial ledgers (requirements 13.1, 13.5, 13.8).

Index

Constants

View Source
const (
	ObservationFaultAfterOutbox = "after_outbox"
	// ObservationFaultBeforeClaimRelease is the failpoint of the orderly
	// shutdown release of the process-owned claims. It exists so the release
	// failure path is provable without a crashed process.
	ObservationFaultBeforeClaimRelease = "before_claim_release"
)
View Source
const AccountWindowProjectionMigrationName = "20260914000000"

AccountWindowProjectionMigrationName identifies the additive search-column migration for provider account-window gauges. The canonical observation envelope remains the sole immutable source; these columns are only indexed query accelerators and consistency checks.

View Source
const BaselineMigrationName = "20260714000000"
View Source
const LinkedStatementCandidateIndexMigrationName = "20260920000000"

LinkedStatementCandidateIndexMigrationName identifies the additive partial index that makes the economic relay's linked-statement lookup selective on verified importer provenance. Without it a single B-leg's unrelated usage observations are enumerated as candidates; with it only verified statement-line rows for the trusted store + B-leg are visited, and its trailing keyset columns still serve the ListObservations ORDER BY. The canonical observation envelope remains the sole immutable source; this index is only a query accelerator.

View Source
const LinkedStatementIndexMigrationName = "20260918000000"

LinkedStatementIndexMigrationName identifies the additive partial index that makes correlated B-leg observation lookups (verified statement/correction evidence) selective. The canonical observation envelope remains the sole immutable source; this index is only a query accelerator.

View Source
const LinkedStatementOrderedIndexMigrationName = "20260919000000"

LinkedStatementOrderedIndexMigrationName extends the linked-statement B-leg partial index with the ListObservations keyset columns. One index then serves both the store-scoped B-leg bound and the deterministic page ordering, so SQLite keeps the plan out of a temp B-tree. It replaces the interim store_id/b_leg_id-only index recorded by LinkedStatementIndexMigrationName. The canonical observation envelope remains the sole immutable source; this index is only a query accelerator.

Lock window, for the operator running the cutover. On PostgreSQL this builds with a plain CREATE INDEX, which takes a ShareLock on metering_facts and blocks INSERT for the duration of the build. That is accepted deliberately: the build is fast relative to the append rate this journal sees, the DROP below cannot be made concurrent either (see below), and the alternative - CREATE INDEX CONCURRENTLY - is unavailable inside this migrator because bun runs a registered Go migration on a pooled connection with no surrounding transaction to escape from, and CONCURRENTLY is illegal in a transaction block. Choosing it would mean either a bespoke non-transactional migration path for this one index or accepting a failed CONCURRENTLY build leaving an INVALID index behind for VerifySchema to reject.

The DROP/CREATE pair is the reason the interim index is rebuilt twice on a database that has not yet run 20260919000000: 20260918000000 builds the store_id/b_leg_id shape, this migration drops and rebuilds it with the keyset columns, and 20260920000000 adds the separate statement candidate index. Only the first two touch the same index name, so a database already past 20260919000000 performs one build here, not three. Between the DROP and the CREATE the B-leg lookup has no index and falls back to a scan; the window is one index build.

View Source
const ObservationEconomicOutboxMigrationName = "20260916000000"

ObservationEconomicOutboxMigrationName identifies the durable relay outbox. Observation and outbox rows are written by the same local transaction, so a crash cannot acknowledge an observation without leaving a restartable economic trigger.

View Source
const ObservationEconomicOutboxPendingIndexName = "idx_metering_observation_outbox_pending"
View Source
const ObservationProjectionMigrationName = "20260912000000"

ObservationProjectionMigrationName identifies the additive V2 observation projection migration. It extends metering_facts; it does not introduce a second authoritative observation journal.

View Source
const ObservationProjectionVersion = 1

ObservationProjectionVersion is bumped only when the deterministic component projection shape changes.

View Source
const PresenceBooleanRepairMigrationName = "20260917000000"

PresenceBooleanRepairMigrationName identifies the forward repair converting legacy INTEGER presence flags to logical BOOLEAN on PostgreSQL and ensuring the observation-projection indexes exist on both dialects. SQLite keeps its dialect-native INTEGER representation for booleans.

View Source
const SchemaV2MigrationName = "20260719000000"
View Source
const StoreScopedFiltersMigrationName = "20260718120000"
View Source
const StoreScopedSourceKeyMigrationName = "20260718000000"

Variables

View Source
var (
	ErrObservationOutboxBackpressure = errors.New("metering/journalstore: observation economic outbox is full")
	ErrObservationOutboxClaimLost    = errors.New("metering/journalstore: observation economic outbox claim lost")
)
View Source
var ErrIdentityCollision = errors.New("metering/journalstore: fact identity collision")

ErrIdentityCollision is returned when Append sees the same SourceEventKey (or legacy IdempotencyKey row) with a different Sequence, Kind, or double-count-sensitive payload than the stored fact (requirements 3.1, 3.4, 13.4).

View Source
var ErrInvalidCursor = errors.New("metering/journalstore: invalid cursor")

ErrInvalidCursor identifies a cursor that is malformed or was created for a different store/filter. V2 cursors are opaque and filter-bound.

View Source
var ErrPageSizeExceeded = errors.New("metering/journalstore: page size exceeds hard maximum")

ErrPageSizeExceeded is returned by V2 bounded queries when a caller asks for more than the hard safety bound.

View Source
var ErrQueryOutOfScope = errors.New("metering/journalstore: query out of scope")

ErrQueryOutOfScope identifies a query that attempts to use another store's namespace or an untrusted subject scope.

View Source
var ErrQueryTooBroad = metering.ErrQueryTooBroad

ErrQueryTooBroad is returned when List lacks a required selective bound so the store cannot safely page without scanning (requirement 14.4/14.8).

View Source
var ErrSQLiteBusyRetryExhausted = errors.New("sqlite_busy_retry_exhausted")

ErrSQLiteBusyRetryExhausted classifies a busy append that used its retry budget.

View Source
var ErrSQLiteRetryCanceled = errors.New("sqlite_retry_canceled")

ErrSQLiteRetryCanceled classifies cancellation while a busy append is waiting to retry.

View Source
var ErrSupersessionCycle = errors.New("metering/journalstore: supersession cycle")

ErrSupersessionCycle is returned when supersession edges would form a cycle (requirements 6.7, D7).

View Source
var ErrSupersessionTarget = errors.New("metering/journalstore: supersession target invalid")

ErrSupersessionTarget is returned when a correction/replacement supersedes a missing fact or a fact outside the same stream (requirements 6.6, D7).

View Source
var ErrUniqueRaceMissingRow = errors.New("metering/journalstore: unique race missing winner row")

ErrUniqueRaceMissingRow is returned when a unique-constraint race on Append is followed by a lookup that finds no winning row. Callers should treat this as a transient inconsistency and retry.

RequiredMigrationNames are bun history names that must be present after a successful Migrate. Collapsed legacy histories may retain extra baseline rows.

View Source
var V2BoundedIndexNames = []string{
	"metering_facts_store_stream_fact_id_key",
	"idx_metering_facts_store_stream_seq",
	"idx_metering_facts_store_attempt",
	"idx_metering_facts_store_recorded",
	"idx_metering_facts_store_plane",
	"idx_metering_fact_supersessions_to",
	"metering_facts_store_id_key",
	"metering_facts_store_observation_revision_key",
	"idx_metering_components_store_subject",
	"idx_metering_components_store_component",
	"idx_metering_components_store_provider_account",
	"idx_metering_components_observation",
	accountWindowFactsIndex,
	meteringFactsStoreBLegIndex,
	meteringFactsStoreBLegStatementIndex,
}

V2BoundedIndexNames are store-scoped indexes required by VerifySchema (task 3.4 / D12).

Functions

func BackfillSchemaV2IdentityForTest

func BackfillSchemaV2IdentityForTest(ctx context.Context, db *bun.DB) error

BackfillSchemaV2IdentityForTest re-runs identity column backfill (tests only).

func BackfillSchemaV2SupersessionsForTest

func BackfillSchemaV2SupersessionsForTest(ctx context.Context, db *bun.DB) error

BackfillSchemaV2SupersessionsForTest re-runs supersession edge backfill (tests only).

func Migrate

func Migrate(ctx context.Context, db *bun.DB) error

Migrate applies the metering-journal schema on an admin connection.

func NewObservationSink

func NewObservationSink(store *DurableStore) sdkmetering.ObservationSink

NewObservationSink adapts the V2 observation journal to the SDK capture ports. The returned adapter implements both ObservationSink for source compatibility and AtomicObservationSink for retry-safe checkpoint batches. DurableStore already has a legacy Fact Append method, so the adapter keeps the two append contracts distinct.

func NewObservationSinkWithOutbox

func NewObservationSinkWithOutbox(store *DurableStore) coremetering.EconomicObservationSink

NewObservationSinkWithOutbox adapts the V2 observation journal and enables the atomic durable economic trigger outbox. It is deliberately a separate constructor so disabled or non-economic metering remains a no-op with no relay rows.

func VerifySchema

func VerifySchema(ctx context.Context, db *bun.DB) error

VerifySchema checks required runtime relations without applying migrations.

Types

type AccountWindowObservationPage

type AccountWindowObservationPage = coremetering.AccountWindowObservationPage

These aliases expose the domain query contract at the journal adapter boundary without duplicating DTOs in infrastructure.

type AccountWindowProjection

type AccountWindowProjection = coremetering.AccountWindowProjection

These aliases expose the domain query contract at the journal adapter boundary without duplicating DTOs in infrastructure.

type AccountWindowProjectionPage

type AccountWindowProjectionPage = coremetering.AccountWindowProjectionPage

These aliases expose the domain query contract at the journal adapter boundary without duplicating DTOs in infrastructure.

type AccountWindowQuery

type AccountWindowQuery = coremetering.AccountWindowQuery

These aliases expose the domain query contract at the journal adapter boundary without duplicating DTOs in infrastructure.

type ComponentPage

type ComponentPage struct {
	Components []ComponentProjection
	NextCursor string
}

ComponentPage is one deterministic page of projected observation items.

type ComponentProjection

type ComponentProjection struct {
	ID                     int64
	StoreID                string
	ObservationRowID       int64
	ObservationID          string
	ObservationRevision    uint64
	ObservationFingerprint string
	ItemKind               string
	ItemID                 string
	ComponentKey           string
	CanonicalComponentKey  string
	ComponentKeyHash       string
	Coefficient            string
	Scale                  uint8
	ValuePresent           bool
	MoneyPresent           bool
	Currency               string
	ChargeCoverageJSON     string
	SubjectKind            metering.SubjectKind
	SubjectID              string
	TenantID               string
	ProviderAccountKey     string
	StreamID               string
	Sequence               uint64
	Origin                 string
	Acquisition            string
	Authority              string
	ProjectionVersion      int
}

ComponentProjection is a rebuildable, indexed projection of one V2 measure or reported charge. ComponentKey is the canonical key bytes; the hash is only an accelerator and is never authoritative.

type ComponentQuery

type ComponentQuery struct {
	StoreID               string
	SubjectKind           metering.SubjectKind
	SubjectID             string
	Subject               *metering.SubjectRef
	TenantID              string
	ProviderAccountKey    string
	StreamID              string
	ComponentKey          string
	ComponentCanonicalKey string
	ComponentKeyHash      string
	ItemKind              string
	Limit                 int
	Cursor                string
}

ComponentQuery is a store-scoped, bounded component projection query.

type DurableConfig

type DurableConfig struct {
	StoreID         string
	DefaultPageSize int
	MaxPageSize     int
	// ObservationOutboxMaxPending bounds unacknowledged observation relay
	// entries. Zero uses the safe default; a negative value is rejected.
	ObservationOutboxMaxPending int
	Now                         func() time.Time
	SQLiteRetryNow              func() time.Time
	SQLiteRetrySleep            func(context.Context, time.Duration) error
	SQLiteRetryObserver         SQLiteRetryObserver
	// ObservationFaultHook is an infra-only test/failpoint seam. Returning an
	// error rolls back the caller-owned observation transaction.
	ObservationFaultHook func(string) error
	// ObservationIdentityMint mints this journal's opaque observation
	// identity. Nil uses the crypto-backed default. It is a constructor
	// seam so an owner can fail the open closed deterministically instead of
	// publishing a journal whose identity cannot be proven.
	ObservationIdentityMint func() (metering.ObservationJournalID, error)
}

DurableConfig configures a Bun-backed metering journal.

type DurableStore

type DurableStore struct {
	// contains filtered or unexported fields
}

DurableStore persists metering facts via Bun (SQLite or Postgres).

func NewDurableStore

func NewDurableStore(ctx context.Context, db *bun.DB, cfg DurableConfig) (*DurableStore, error)

NewDurableStore migrates schema and returns a durable journal. Caller owns closing db via DurableStore.Close (closes the bun handle).

func OpenStore

func OpenStore(ctx context.Context, db *bun.DB, cfg DurableConfig) (*DurableStore, error)

OpenStore opens a journal without migrations and without taking ownership of db. The composition root owns the shared runtime pool.

func (*DurableStore) Append

func (s *DurableStore) Append(ctx context.Context, fact metering.Fact) error

Append inserts one fact or applies idempotent/collision rules on UNIQUE source_event_key. SameFactReplay → no-op; same source key otherwise → ErrIdentityCollision.

func (*DurableStore) AppendAccountWindowObservation

func (s *DurableStore) AppendAccountWindowObservation(ctx context.Context, observation aipsdkmetering.Observation) error

AppendAccountWindowObservation validates and appends one provider allowance snapshot. The canonical observation journal remains the sole authority; this spelling only prevents callers from accidentally treating a non-gauge or B-leg observation as an account-window record.

func (*DurableStore) AppendAccountWindowObservationInTx

func (s *DurableStore) AppendAccountWindowObservationInTx(ctx context.Context, tx bun.Tx, observation aipsdkmetering.Observation) error

AppendAccountWindowObservationInTx is the transaction-composing form of AppendAccountWindowObservation. It never commits or rolls back tx.

func (*DurableStore) AppendObservation

func (s *DurableStore) AppendObservation(ctx context.Context, observation metering.Observation) error

AppendObservation appends one immutable V2 observation and all of its measure/charge projections atomically. Exact replay is a no-op; a changed payload for the same source identity/revision is a collision.

func (*DurableStore) AppendObservationInTx

func (s *DurableStore) AppendObservationInTx(ctx context.Context, tx bun.Tx, observation metering.Observation) error

AppendObservationInTx appends an observation using the caller-owned local transaction. It never commits or rolls back tx.

func (*DurableStore) AppendObservations

func (s *DurableStore) AppendObservations(ctx context.Context, observations []metering.Observation) error

AppendObservations appends a bounded batch of immutable V2 observations in one local transaction. Exact replays remain no-ops; a changed payload for an existing source identity is a collision; and any collision or projection failure rolls the complete batch back. The operation is therefore safe to retry after an error, including an ambiguous commit result, and satisfies metering.AtomicObservationSink. An empty batch performs no durable operation.

func (*DurableStore) AppendObservationsWithOutbox

func (s *DurableStore) AppendObservationsWithOutbox(ctx context.Context, observations []metering.Observation) error

AppendObservationsWithOutbox appends observations and their durable relay entries in one local transaction. A failure after either side is written rolls both sides back, preserving the observation-to-work relationship.

func (*DurableStore) CheckReadiness

func (s *DurableStore) CheckReadiness(ctx context.Context) error

CheckReadiness pings the database.

func (*DurableStore) ClaimObservationOutbox

func (s *DurableStore) ClaimObservationOutbox(ctx context.Context, owner string, limit int, lease time.Duration) ([]ObservationOutboxItem, error)

ClaimObservationOutbox leases due entries to one process-owned relay. A lease expiry makes a processing row restartable after a crash.

func (*DurableStore) Close

func (s *DurableStore) Close() error

Close closes the underlying bun DB handle.

func (*DurableStore) CompleteObservationOutbox

func (s *DurableStore) CompleteObservationOutbox(ctx context.Context, id int64, owner string) error

CompleteObservationOutbox acknowledges a successfully enqueued work set.

func (*DurableStore) DB

func (s *DurableStore) DB() *bun.DB

func (*DurableStore) Database

func (s *DurableStore) Database() *bun.DB

Database is an explicit infra spelling of DB for composition checks.

func (*DurableStore) GetObservation

func (s *DurableStore) GetObservation(ctx context.Context, observationID string, revision uint64) (metering.Observation, error)

GetObservation returns one exact V2 observation revision. The durable row is validated against its duplicated identity columns and stored fingerprint; corrupt rows fail closed instead of returning an unvalidated observation.

func (*DurableStore) GetObservationRef

func (s *DurableStore) GetObservationRef(ctx context.Context, ref metering.ObservationRef) (metering.Observation, error)

GetObservationRef is a convenience form for callers holding an immutable observation reference.

func (*DurableStore) List

List returns a bounded page filtered by indexed selective bounds.

func (*DurableStore) ListAccountWindowObservations

func (s *DurableStore) ListAccountWindowObservations(ctx context.Context, query AccountWindowQuery) (AccountWindowObservationPage, error)

ListAccountWindowObservations returns immutable account-window history in effective observed-at order. Provider account is mandatory; pool, window, reset and AsOf are selective refinements.

func (*DurableStore) ListComponents

func (s *DurableStore) ListComponents(ctx context.Context, query ComponentQuery) (ComponentPage, error)

ListComponents is a compatibility alias for ListObservationComponents.

func (*DurableStore) ListObservationComponents

func (s *DurableStore) ListObservationComponents(ctx context.Context, query ComponentQuery) (ComponentPage, error)

ListObservationComponents returns deterministic bounded projection rows.

func (*DurableStore) ListObservations

func (s *DurableStore) ListObservations(ctx context.Context, query ObservationQuery) (ObservationPage, error)

func (*DurableStore) ListPendingObservationOutbox

func (s *DurableStore) ListPendingObservationOutbox(ctx context.Context, limit int) ([]ObservationOutboxItem, error)

ListPendingObservationOutbox returns bounded active entries for diagnostics and tests. It does not claim or acknowledge them.

func (*DurableStore) ObservationJournalIdentity

func (s *DurableStore) ObservationJournalIdentity() metering.ObservationJournalID

ObservationJournalIdentity reports the opaque identity of this journal. The observation sink built over the same store reports the same identity, so a recorder/sink pair built from one journal is provably one journal.

func (*DurableStore) ProjectAccountWindows

func (s *DurableStore) ProjectAccountWindows(ctx context.Context, query AccountWindowQuery) (AccountWindowProjectionPage, error)

ProjectAccountWindows rebuilds one current/as-of projection per matching account/pool/window/reset identity from immutable observations. It never performs SQL arithmetic: percentages, limits and remaining values remain independent gauge measures.

func (*DurableStore) QueryAccountWindowObservations

func (s *DurableStore) QueryAccountWindowObservations(ctx context.Context, query AccountWindowQuery) (AccountWindowObservationPage, error)

QueryAccountWindowObservations is a compatibility spelling for callers that use query-oriented naming.

func (*DurableStore) RebuildObservationProjections

func (s *DurableStore) RebuildObservationProjections(ctx context.Context) error

RebuildObservationProjections reconstructs every component row from the canonical observation JSON while leaving payload_json and its fingerprint untouched.

func (*DurableStore) RebuildProjections

func (s *DurableStore) RebuildProjections(ctx context.Context) error

RebuildProjections is the short form used by migration/repair callers.

func (*DurableStore) ReleaseObservationOutboxClaims

func (s *DurableStore) ReleaseObservationOutboxClaims(ctx context.Context, owner string) error

ReleaseObservationOutboxClaims makes process-owned in-flight entries immediately retryable during orderly shutdown. A crash without this call is still safe because the lease expires and the next process can reclaim it.

func (*DurableStore) RetryObservationOutbox

func (s *DurableStore) RetryObservationOutbox(ctx context.Context, id int64, owner string, delay time.Duration, relayErr error) error

RetryObservationOutbox releases a failed relay attempt without losing the durable payload. The caller may choose a bounded delay for backpressure.

func (*DurableStore) SetObservationFaultHook

func (s *DurableStore) SetObservationFaultHook(hook func(string) error)

SetObservationFaultHook installs a test/failpoint hook. A returned error aborts the active append transaction.

func (*DurableStore) WriteObservationTx

func (s *DurableStore) WriteObservationTx(ctx context.Context, tx bun.Tx, observation metering.Observation) error

WriteObservationTx is a compatibility spelling for infra composition code.

type MemoryConfig

type MemoryConfig struct {
	StoreID         string
	DefaultPageSize int
	MaxPageSize     int
	Now             func() time.Time
}

MemoryConfig configures an in-memory metering journal.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is a concurrent in-memory metering journal.

func NewMemoryStore

func NewMemoryStore(cfg MemoryConfig) (*MemoryStore, error)

NewMemoryStore returns a ready in-memory journal.

func (*MemoryStore) Append

func (s *MemoryStore) Append(ctx context.Context, fact metering.Fact) error

Append validates and stores one fact with source-key idempotency. SameFactReplay → no-op; same source key otherwise → ErrIdentityCollision.

func (*MemoryStore) CheckReadiness

func (s *MemoryStore) CheckReadiness(ctx context.Context) error

CheckReadiness always succeeds for memory.

func (*MemoryStore) Close

func (s *MemoryStore) Close() error

Close is a no-op for the memory store.

func (*MemoryStore) List

List returns a bounded page of facts matching the query.

func (*MemoryStore) SeedSourceKeyForTest

func (s *MemoryStore) SeedSourceKeyForTest(key string, fact metering.Fact) error

SeedSourceKeyForTest indexes fact under an explicit source_event_key for process-local legacy/preload replay tests. It does not rewrite durable rows.

type ObservationOutboxItem

type ObservationOutboxItem struct {
	ID                     int64
	Observation            metering.Observation
	ObservationFingerprint string
	Status                 string
	AttemptCount           int
	NextAttemptAt          time.Time
	LeaseOwner             string
	LeaseUntil             time.Time
	LastError              string
}

ObservationOutboxItem is an immutable observation payload plus mutable relay-attempt metadata. The payload is copied from the canonical observation transaction and is never reconstructed from mutable billing state.

type ObservationPage

type ObservationPage struct {
	Observations []metering.Observation
	NextCursor   string
}

ObservationPage is one deterministic page of immutable V2 observations.

type ObservationQuery

type ObservationQuery struct {
	StoreID            string
	SubjectKind        metering.SubjectKind
	SubjectID          string
	Subject            *metering.SubjectRef
	TenantID           string
	ProviderAccountKey string
	StreamID           string
	CorrelationBLegID  string
	// StatementEvidenceOnly restricts a page to verified statement-line
	// observations (origin=statement, acquisition=statement_importer,
	// authority=verified_statement, subject kind=statement_line). It is only
	// valid together with CorrelationBLegID and exists to keep the economic
	// relay's linked-statement candidate work selective and bounded.
	StatementEvidenceOnly bool
	ComponentKey          string
	ComponentCanonicalKey string
	ComponentKeyHash      string
	ItemKind              string
	Limit                 int
	Cursor                string
}

ObservationQuery is a store-scoped, bounded V2 observation query. At least one selective bound (subject identity, correlation B-leg, stream, tenant, or component) is required. StoreID is optional and, when set, must equal the opened store.

type ObservationTxWriter

type ObservationTxWriter interface {
	AppendObservationInTx(context.Context, bun.Tx, metering.Observation) error
}

ObservationTxWriter is the infra-only seam used by a composition root to append an observation on an already-open local transaction. Bun remains inside infrastructure and is never part of the SDK/domain contracts.

type SQLiteRetryEvent

type SQLiteRetryEvent struct {
	Attempt         int
	Classification  string
	Backoff         time.Duration
	TerminalOutcome string
}

SQLiteRetryEvent describes one append attempt from the narrow SQLite retry policy.

type SQLiteRetryObserver

type SQLiteRetryObserver func(SQLiteRetryEvent)

SQLiteRetryObserver receives bounded append retry diagnostics.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址