database

package module
v0.0.0-...-86254be Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Index

Constants

View Source
const (

	// SiteOwnerRestampSourceDeployment names a restamp whose owner came from
	// the newest packageDeployment naming the site in its deployables[].
	SiteOwnerRestampSourceDeployment = "deployment"
	// SiteOwnerRestampSourcePackage names a restamp whose owner came from the
	// package row, because no deployment named the site.
	SiteOwnerRestampSourcePackage = "package"
)
View Source
const (
	// DefaultRequestPoolSize is the main (request) pool's limit when nothing
	// configures one (MAX_OPEN_CONNS under the database's env prefix).
	DefaultRequestPoolSize = defaultRequestMaxOpenConns
	// DirectPoolSize is the direct pool's fixed limit, opened only when a
	// direct DSN is configured.
	DirectPoolSize = directMaxOpenConns
	// MigrationPoolSize is the connection a startup migration opens.
	MigrationPoolSize = migrationMaxOpenConns
)

The connections a process can open against max_connections, pool by pool, as the deploy gate and the budget documentation count them (scripts/deploy/conn-headroom-check.sh, which conn_headroom_test.go holds to these; docs/public/operate/db-connection-budget.md). With the gate pool (DefaultGatePoolSize, gate_pool.go) they are a process's four pools.

View Source
const (
	// DefaultGatePoolSize is how many write gates one process can hold open at
	// once (MEMQL_DB_GATE_POOL_SIZE).
	DefaultGatePoolSize = defaultGatePoolSize

	// DefaultGateIdleInTxTimeout is how long a gate session may sit idle in its
	// transaction before PostgreSQL ends it (MEMQL_DB_GATE_IDLE_IN_TX_TIMEOUT_MS).
	DefaultGateIdleInTxTimeout = defaultGateIdleInTxTimeoutMs * time.Millisecond

	// DefaultRequestStatementTimeout and DefaultRequestIdleInTxTimeout are the
	// request pool's defaults (MEMQL_DB_STATEMENT_TIMEOUT_MS,
	// MEMQL_DB_IDLE_IN_TX_TIMEOUT_MS).
	DefaultRequestStatementTimeout = defaultStatementTimeoutMs * time.Millisecond
	DefaultRequestIdleInTxTimeout  = defaultIdleInTxTimeoutMs * time.Millisecond
)

The numbers other packages build on, read from here rather than copied: the gate wait must stay under the driver's read deadline, a gated phase under the gate session's idle limit, and the request pool's own limits are what the gate pool exists to be free of.

Variables

View Source
var ErrMaintenanceSessionUnavailable = errors.New("maintenance session unavailable on this database")

ErrMaintenanceSessionUnavailable answers OpenMaintenanceSession on a Database with no DSN, or one built over a custom opener (tests, sqlmock), which has no way to open a session without the request pool's limits.

View Source
var ErrMigrationDeferred = errors.New("deferred to the next migration attempt")

ErrMigrationDeferred marks a migration attempt that ended early with nothing left that the next attempt could overlap, which then carries on from where this one stopped. Three things say so, each for its own reason:

  • The readiness history collapse (20260921000000), whose work can outlast one attempt's MIGRATION_TIMEOUT_MS (memql#5604): it stopped with nothing of its own running, and everything it did committed or rolled back whole.
  • An index ensure (deferInterruptedIndexWork, latest_row_index.go): a build may well still be running, but its backend holds the build lock until it exits, and the next attempt takes that lock before anything else.
  • The runner itself (migrationsForAttempt), when the attempt ended between migrations: what finished is recorded and what follows has not started.

It is neither a context nor a transport error, so the runner releases its lock (migrationMayStillBeRunning). A node starts the next attempt on its monitor tick (tryPing, via migrationsPending); the migrate subcommand, which has no tick, starts it at once (awaitMigrations, subcommand_migrate.go).

View Source
var ErrMigrationLockHeld = errors.New("migration lock held by another migrator")

ErrMigrationLockHeld marks a migration attempt that ran nothing because another migrator's row was already in bun's lock table. bun's Lock is a TRY-lock -- an INSERT into bun_migration_locks, refused by the row's unique key while another migrator holds it -- so "held" says something about the world rather than about this attempt: another process is migrating, or is confirming there is nothing to migrate, right now. A node retries on its monitor tick (tryPing, via migrationsPending). A caller that must not proceed on an unmigrated schema and cannot wait for a tick -- dbtest.EnsureSchema -- tests for this with errors.Is on MigrationError() and waits the holder out.

Functions

func BackfillRecipientEmailDigests

func BackfillRecipientEmailDigests(ctx context.Context, db *bun.DB, logger *slog.Logger) (int, error)

BackfillRecipientEmailDigests stamps the digest onto every recipient whose newest version has none, and returns how many it stamped. Safe to call any number of times, and from any number of replicas at once. db is the pool the caller owns.

A pass whose context is cancelled ends between two batches, after the batches that committed, with an error wrapping the context's; the next call carries on.

func NewGatePool

func NewGatePool(dsn string, size int) (*sql.DB, error)

NewGatePool opens a pool for write gates over dsn, configured as the Database component configures its own: gateConnParams on every session, bounded at size connections (the configured size when size <= 0). The caller closes it.

Exported for the tests of the packages that take gates, which must run them on a pool shaped like the deployed one rather than on whatever pool they have.

func OpenMaintenanceSessionDSN

func OpenMaintenanceSessionDSN(ctx context.Context, dsn string) (*sql.Conn, func(), error)

OpenMaintenanceSessionDSN opens a maintenance session on dsn with the default opener. For the tests of the packages that run maintenance, which must run it on a session shaped like the deployed one rather than on whatever pool they have (NewGatePool's reason).

func RecipientEmailDigest

func RecipientEmailDigest(addr string) string

RecipientEmailDigest is the key a lookup of an address uses: the stamp of the address, or "" when the text cannot be a mailbox (no "@", an empty local part or domain, whitespace inside). It is campaigns.EmailDigest, kept here because this package cannot import that one; see the header for what holds the two together.

func RecipientEmailStamp

func RecipientEmailStamp(text string) string

RecipientEmailStamp is the digest addRecipient stamps on a recipient and this sweep gives the ones that have none: the SHA-256, as lower-case hex, of the text trimmed of surrounding whitespace and lower-cased. Whatever the text is; see the header for why text that is not an address is stamped too.

func RevertSiteOwnerRestamp

func RevertSiteOwnerRestamp(ctx context.Context, db *bun.DB, logger *slog.Logger) error

RevertSiteOwnerRestamp is the down direction: delete exactly the versions this migration appended, identified by their provenance. Every other version of every site is left as it was, so the rows read as they did before the up ran.

staged-data: MUST-NOT-GATE -- a MIGRATION must see every row, by definition (epic memql#3974, task memql#3984).

Types

type BunFactory

type BunFactory func(*sql.DB) (*bun.DB, error)

type BunOption

type BunOption func(*bun.DB)

type Database

type Database struct {
	sync.Mutex

	Logger *slog.Logger
	DB     *sql.DB
	Bun    *bun.DB
	// DirectDB / DirectBun are the optional second, NON-pooled endpoint
	// (DIRECT_DSN). Session-stateful work -- session-scoped advisory
	// locks, leader election, migrations -- rides this so it bypasses the
	// transaction pooler that recycles a server backend out from under a
	// held session (epic memql#1925). nil when DIRECT_DSN is unset, in
	// which case DirectBunDB() falls back to the main (pooled) Bun -- so
	// local/dev without a pooler is unaffected (env-agnostic).
	DirectDB  *sql.DB
	DirectBun *bun.DB
	// Gate is the pool write gates run on (gate_pool.go): connections used
	// for nothing else, so a gate holder never waits for one to do its work
	// and a session sitting idle in a gate's transaction is not held to the
	// request pool's limits. nil when it was not opened; there is no fallback
	// to the request pool.
	Gate *sql.DB
	// contains filtered or unexported fields
}

func NewDatabase

func NewDatabase(componentName common.ComponentName, args ...DatabaseArg) (*Database, error)

func (*Database) BunDB

func (d *Database) BunDB() *bun.DB

BunDB returns the current Bun database handle in a thread-safe manner. Returns nil if the database is not connected.

func (*Database) ClosePool

func (d *Database) ClosePool(ctx context.Context) sql.DBStats

ClosePool stops the connection monitor and closes the underlying pool, handing every open connection back to the database server. It is the explicit, intention-revealing entry point the drain sequence calls so a draining node releases its TigerDB sessions PROMPTLY instead of holding up to MaxOpenConns connections open until the process finally exits at the end of the blue-green scale-down window (memql#1875). Returns the pre-close pool stats so the caller can log how many connections were released.

The monitor is stopped first (via Stop) so the ticker reconnect loop can't race in and re-open the pool we're about to close. Closing is then done directly as well, so the method also works (and stays idempotent) when the lifecycle was never started or has already stopped -- a second call, e.g. the later dependency Stop sweep, is a harmless no-op.

func (*Database) ComponentName

func (d *Database) ComponentName() common.ComponentName

func (*Database) DirectBunDB

func (d *Database) DirectBunDB() *bun.DB

DirectBunDB returns the Bun handle for the direct (non-pooled) endpoint when DIRECT_DSN is configured, else falls back to the main (pooled) Bun handle. Session-stateful work -- session-scoped advisory locks, leader election, migrations -- must resolve its *bun.DB through this getter so it bypasses the transaction pooler the bulk pool rides; transaction-mode PgBouncer recycles a server backend between statements and would drop a held session-scoped lock (epic memql#1925). When DIRECT_DSN is unset the fallback keeps behavior identical to today (single pool), so local/dev is unaffected.

func (*Database) GateDB

func (d *Database) GateDB() *sql.DB

GateDB returns the pool write gates run on, or nil when it was not opened (before the database started, after it stopped, or under a custom SQL opener). There is no fallback to the request pool: a gate that cannot have its own connection must say so (component/memql: "shared write coordination is unavailable"), not take one from the pool its holder works on.

func (*Database) IsRunning

func (d *Database) IsRunning() bool

func (*Database) MigrationError

func (d *Database) MigrationError() error

MigrationError returns the error from the most recent migration attempt, or nil if migrations succeeded or were skipped (migrateOnStart=false). The gated pre-deploy migrate subcommand consults this to fail the deploy when a migration did not complete (#671).

func (*Database) MigrationLockHeldFor

func (d *Database) MigrationLockHeldFor() time.Duration

MigrationLockHeldFor is how long this node has found the same row in bun's lock table in the way of its attempts: zero when nothing is, or once its own attempt got the lock. It reads no database, so a /metrics scrape can call it.

func (*Database) OpenMaintenanceSession

func (d *Database) OpenMaintenanceSession(ctx context.Context) (*sql.Conn, func(), error)

OpenMaintenanceSession opens one dedicated maintenance session (see the header): no server statement_timeout, no client read deadline beyond ctx's, on the direct endpoint when one is configured. The caller owns it and must call the returned func, which closes the session and the single-connection pool behind it. Every statement on it should run under a context with a deadline and a statement_timeout the caller sets.

func (*Database) Order

func (d *Database) Order() int

func (*Database) PoolStats

func (d *Database) PoolStats() (sql.DBStats, bool)

PoolStats returns a snapshot of the underlying *sql.DB connection-pool statistics (open / in-use / idle counts + wait metrics). The bool reports whether a live pool handle was present: false when the database has not connected yet or has already been closed. Surfaces per-node active- connection counts for the periodic pool-stats log and the drain sequence so a deploy can watch the cluster-wide connection total against the TigerDB ceiling (memql#1875).

func (*Database) Ready

func (d *Database) Ready() <-chan struct{}

Ready returns a channel that is closed when the database is ready.

func (*Database) Start

func (d *Database) Start(ctx context.Context)

func (*Database) Stop

func (d *Database) Stop(ctx context.Context)

func (*Database) WithBunFactory

func (d *Database) WithBunFactory(factory BunFactory) DatabaseArg

func (*Database) WithBunOptions

func (d *Database) WithBunOptions(options ...BunOption) DatabaseArg

func (*Database) WithConnMaxIdleTimeMs

func (d *Database) WithConnMaxIdleTimeMs(ms int) DatabaseArg

WithConnMaxIdleTimeMs sets the maximum amount of time a connection may be idle (in milliseconds).

func (*Database) WithConnMaxLifetimeMs

func (d *Database) WithConnMaxLifetimeMs(ms int) DatabaseArg

WithConnMaxLifetimeMs sets the maximum amount of time a connection may be reused (in milliseconds).

func (*Database) WithGateOpener

func (d *Database) WithGateOpener(opener SQLOpener) DatabaseArg

WithGateOpener sets the opener of the gate pool. The default opens a real pool over the DSN with gateConnParams; a test supplies its own (and, like the migration pool, a custom WithSQLOpener alone leaves no gate pool at all).

func (*Database) WithInitHook

func (d *Database) WithInitHook(hook InitHook) DatabaseArg

func (*Database) WithLoggerLevel

func (d *Database) WithLoggerLevel(level slog.Level) DatabaseArg

func (*Database) WithLoggerWriter

func (d *Database) WithLoggerWriter(w io.Writer) DatabaseArg

func (*Database) WithMaxConnRetries

func (d *Database) WithMaxConnRetries(retries int) DatabaseArg

func (*Database) WithMaxConnRetriesInterval

func (d *Database) WithMaxConnRetriesInterval(intervalMs int) DatabaseArg

func (*Database) WithMaxIdleConns

func (d *Database) WithMaxIdleConns(n int) DatabaseArg

WithMaxIdleConns sets the maximum number of idle connections in the pool.

func (*Database) WithMaxOpenConns

func (d *Database) WithMaxOpenConns(n int) DatabaseArg

WithMaxOpenConns sets the maximum number of open connections to the database.

func (*Database) WithMigrateOnStart

func (d *Database) WithMigrateOnStart(enabled bool) DatabaseArg

func (*Database) WithMigrationTimeoutMs

func (d *Database) WithMigrationTimeoutMs(timeoutMs int) DatabaseArg

func (*Database) WithMigrations

func (d *Database) WithMigrations(migrations *migrate.Migrations) DatabaseArg

func (*Database) WithMigrator

func (d *Database) WithMigrator(m Migrator) DatabaseArg

func (*Database) WithPostMigrationHook

func (d *Database) WithPostMigrationHook(hook PostMigrationHook) DatabaseArg

func (*Database) WithSQLDriver

func (d *Database) WithSQLDriver(driverName string) DatabaseArg

func (*Database) WithSQLOpener

func (d *Database) WithSQLOpener(opener SQLOpener) DatabaseArg

func (*Database) WithTickerInterval

func (d *Database) WithTickerInterval(intervalMs int) DatabaseArg

type DatabaseArg

type DatabaseArg common.CtorArg[config]

func EnvOptionsToArgs

func EnvOptionsToArgs(opts DatabaseEnvOptions) ([]DatabaseArg, error)

func OptionalCtorArg

func OptionalCtorArg(name string, fn func(*config)) DatabaseArg

func RequiredCtorArg

func RequiredCtorArg(name string, fn func(*config)) DatabaseArg

func WithDSN

func WithDSN(dsn string) DatabaseArg

func WithDirectDSN

func WithDirectDSN(dsn string) DatabaseArg

WithDirectDSN configures the optional second, NON-pooled DB endpoint (DIRECT_DSN). Session-stateful work (advisory locks, leader election, migrations) resolves its *bun.DB via DirectBunDB() so it bypasses the transaction pooler the main pool rides (epic memql#1925). An empty/blank value is a no-op: DirectBunDB() then falls back to the main pool, keeping behavior identical to today for local/dev without a pooler.

type DatabaseEnvKeys

type DatabaseEnvKeys struct {
	DSN                      string
	DirectDSN                string
	MigrateOnStart           string
	MaxConnRetries           string
	MaxConnRetriesIntervalMs string
	TickerIntervalMs         string
	MigrationTimeoutMs       string
	SQLDriver                string
	LoggerLevel              string
	// Connection pool configuration
	MaxOpenConns      string
	MaxIdleConns      string
	ConnMaxLifetimeMs string
	ConnMaxIdleTimeMs string
}

func DefaultDatabaseEnvKeys

func DefaultDatabaseEnvKeys() DatabaseEnvKeys

type DatabaseEnvLoader

type DatabaseEnvLoader struct {
	Prefix string
	Keys   DatabaseEnvKeys
}

type DatabaseEnvOptions

type DatabaseEnvOptions struct {
	DSN                      string
	DirectDSN                string
	MigrateOnStart           *bool
	MaxConnRetries           *int
	MaxConnRetriesIntervalMs *int
	TickerIntervalMs         *int
	MigrationTimeoutMs       *int
	SQLDriver                string
	LoggerLevel              string
	// Connection pool configuration
	MaxOpenConns      *int
	MaxIdleConns      *int
	ConnMaxLifetimeMs *int
	ConnMaxIdleTimeMs *int
}

func LoadDatabaseEnvOptions

func LoadDatabaseEnvOptions(loader DatabaseEnvLoader) (DatabaseEnvOptions, error)

type InitHook

type InitHook func(context.Context, *sql.DB) error

type Migrator

type Migrator interface {
	Migrate(ctx context.Context, db *bun.DB) error
}

type PostMigrationHook

type PostMigrationHook func(context.Context, *bun.DB, *slog.Logger) error

type PostgresDatabase

type PostgresDatabase struct {
	*Database
}

func NewPostgresDatabase

func NewPostgresDatabase(componentName common.ComponentName, args ...DatabaseArg) (*PostgresDatabase, error)

type SQLOpener

type SQLOpener func(driverName, dsn string) (*sql.DB, error)

type SiteOwnerRestamp

type SiteOwnerRestamp struct {
	SiteId    string
	PackageId string
	// OwnerUserId is the owner written, or empty for an unresolved row.
	OwnerUserId string
	// Source is SiteOwnerRestampSourceDeployment or
	// SiteOwnerRestampSourcePackage, or empty for an unresolved row.
	Source string
}

SiteOwnerRestamp is one site the migration decided about.

type SiteOwnerRestampReport

type SiteOwnerRestampReport struct {
	// Restamped lists every site that received a new version with its owner
	// set.
	Restamped []SiteOwnerRestamp
	// Unresolved lists every blanked site left alone because neither a
	// deployment nor the package row could name its owner.
	Unresolved []SiteOwnerRestamp
}

SiteOwnerRestampReport is what one run of the migration did.

func RestampSiteOwners

func RestampSiteOwners(ctx context.Context, db *bun.DB, logger *slog.Logger) (SiteOwnerRestampReport, error)

RestampSiteOwners runs the repair once, in one transaction, and reports what it did. Safe to call any number of times.

type TimescaleDBDatabase

type TimescaleDBDatabase struct {
	*PostgresDatabase
}

func NewTimescaleDBDatabase

func NewTimescaleDBDatabase(componentName common.ComponentName, args ...DatabaseArg) (*TimescaleDBDatabase, error)

func (*TimescaleDBDatabase) Migrator

func (db *TimescaleDBDatabase) Migrator() (*migrate.Migrator, error)

Directories

Path Synopsis
Package dbtest provides shared setup for the Postgres-gated ("*_db_test.go") suites that CI runs together in the db-tests lane (.github/workflows/ci.yml) against ONE shared database.
Package dbtest provides shared setup for the Postgres-gated ("*_db_test.go") suites that CI runs together in the db-tests lane (.github/workflows/ci.yml) against ONE shared database.
Package memoryNodes -- concept_parser.go
Package memoryNodes -- concept_parser.go
Package storagequota owns durable, cluster-wide physical storage admission.
Package storagequota owns durable, cluster-wide physical storage admission.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址