Documentation
¶
Index ¶
- Constants
- Variables
- func BackfillRecipientEmailDigests(ctx context.Context, db *bun.DB, logger *slog.Logger) (int, error)
- func NewGatePool(dsn string, size int) (*sql.DB, error)
- func OpenMaintenanceSessionDSN(ctx context.Context, dsn string) (*sql.Conn, func(), error)
- func RecipientEmailDigest(addr string) string
- func RecipientEmailStamp(text string) string
- func RevertSiteOwnerRestamp(ctx context.Context, db *bun.DB, logger *slog.Logger) error
- type BunFactory
- type BunOption
- type Database
- func (d *Database) BunDB() *bun.DB
- func (d *Database) ClosePool(ctx context.Context) sql.DBStats
- func (d *Database) ComponentName() common.ComponentName
- func (d *Database) DirectBunDB() *bun.DB
- func (d *Database) GateDB() *sql.DB
- func (d *Database) IsRunning() bool
- func (d *Database) MigrationError() error
- func (d *Database) MigrationLockHeldFor() time.Duration
- func (d *Database) OpenMaintenanceSession(ctx context.Context) (*sql.Conn, func(), error)
- func (d *Database) Order() int
- func (d *Database) PoolStats() (sql.DBStats, bool)
- func (d *Database) Ready() <-chan struct{}
- func (d *Database) Start(ctx context.Context)
- func (d *Database) Stop(ctx context.Context)
- func (d *Database) WithBunFactory(factory BunFactory) DatabaseArg
- func (d *Database) WithBunOptions(options ...BunOption) DatabaseArg
- func (d *Database) WithConnMaxIdleTimeMs(ms int) DatabaseArg
- func (d *Database) WithConnMaxLifetimeMs(ms int) DatabaseArg
- func (d *Database) WithGateOpener(opener SQLOpener) DatabaseArg
- func (d *Database) WithInitHook(hook InitHook) DatabaseArg
- func (d *Database) WithLoggerLevel(level slog.Level) DatabaseArg
- func (d *Database) WithLoggerWriter(w io.Writer) DatabaseArg
- func (d *Database) WithMaxConnRetries(retries int) DatabaseArg
- func (d *Database) WithMaxConnRetriesInterval(intervalMs int) DatabaseArg
- func (d *Database) WithMaxIdleConns(n int) DatabaseArg
- func (d *Database) WithMaxOpenConns(n int) DatabaseArg
- func (d *Database) WithMigrateOnStart(enabled bool) DatabaseArg
- func (d *Database) WithMigrationTimeoutMs(timeoutMs int) DatabaseArg
- func (d *Database) WithMigrations(migrations *migrate.Migrations) DatabaseArg
- func (d *Database) WithMigrator(m Migrator) DatabaseArg
- func (d *Database) WithPostMigrationHook(hook PostMigrationHook) DatabaseArg
- func (d *Database) WithSQLDriver(driverName string) DatabaseArg
- func (d *Database) WithSQLOpener(opener SQLOpener) DatabaseArg
- func (d *Database) WithTickerInterval(intervalMs int) DatabaseArg
- type DatabaseArg
- type DatabaseEnvKeys
- type DatabaseEnvLoader
- type DatabaseEnvOptions
- type InitHook
- type Migrator
- type PostMigrationHook
- type PostgresDatabase
- type SQLOpener
- type SiteOwnerRestamp
- type SiteOwnerRestampReport
- type TimescaleDBDatabase
Constants ¶
const ( // SiteOwnerRestampSourceDeployment names a restamp whose owner came from // the newest packageDeployment naming the site in its deployables[]. SiteOwnerRestampSourceDeployment = "deployment" // SiteOwnerRestampSourcePackage names a restamp whose owner came from the // package row, because no deployment named the site. SiteOwnerRestampSourcePackage = "package" )
const ( // DefaultRequestPoolSize is the main (request) pool's limit when nothing // configures one (MAX_OPEN_CONNS under the database's env prefix). DefaultRequestPoolSize = defaultRequestMaxOpenConns // DirectPoolSize is the direct pool's fixed limit, opened only when a // direct DSN is configured. DirectPoolSize = directMaxOpenConns // MigrationPoolSize is the connection a startup migration opens. MigrationPoolSize = migrationMaxOpenConns )
The connections a process can open against max_connections, pool by pool, as the deploy gate and the budget documentation count them (scripts/deploy/conn-headroom-check.sh, which conn_headroom_test.go holds to these; docs/public/operate/db-connection-budget.md). With the gate pool (DefaultGatePoolSize, gate_pool.go) they are a process's four pools.
const ( // DefaultGatePoolSize is how many write gates one process can hold open at // once (MEMQL_DB_GATE_POOL_SIZE). DefaultGatePoolSize = defaultGatePoolSize // DefaultGateIdleInTxTimeout is how long a gate session may sit idle in its // transaction before PostgreSQL ends it (MEMQL_DB_GATE_IDLE_IN_TX_TIMEOUT_MS). DefaultGateIdleInTxTimeout = defaultGateIdleInTxTimeoutMs * time.Millisecond // DefaultRequestStatementTimeout and DefaultRequestIdleInTxTimeout are the // request pool's defaults (MEMQL_DB_STATEMENT_TIMEOUT_MS, // MEMQL_DB_IDLE_IN_TX_TIMEOUT_MS). DefaultRequestStatementTimeout = defaultStatementTimeoutMs * time.Millisecond DefaultRequestIdleInTxTimeout = defaultIdleInTxTimeoutMs * time.Millisecond )
The numbers other packages build on, read from here rather than copied: the gate wait must stay under the driver's read deadline, a gated phase under the gate session's idle limit, and the request pool's own limits are what the gate pool exists to be free of.
Variables ¶
ErrMaintenanceSessionUnavailable answers OpenMaintenanceSession on a Database with no DSN, or one built over a custom opener (tests, sqlmock), which has no way to open a session without the request pool's limits.
var ErrMigrationDeferred = errors.New("deferred to the next migration attempt")
ErrMigrationDeferred marks a migration attempt that ended early with nothing left that the next attempt could overlap, which then carries on from where this one stopped. Three things say so, each for its own reason:
- The readiness history collapse (20260921000000), whose work can outlast one attempt's MIGRATION_TIMEOUT_MS (memql#5604): it stopped with nothing of its own running, and everything it did committed or rolled back whole.
- An index ensure (deferInterruptedIndexWork, latest_row_index.go): a build may well still be running, but its backend holds the build lock until it exits, and the next attempt takes that lock before anything else.
- The runner itself (migrationsForAttempt), when the attempt ended between migrations: what finished is recorded and what follows has not started.
It is neither a context nor a transport error, so the runner releases its lock (migrationMayStillBeRunning). A node starts the next attempt on its monitor tick (tryPing, via migrationsPending); the migrate subcommand, which has no tick, starts it at once (awaitMigrations, subcommand_migrate.go).
var ErrMigrationLockHeld = errors.New("migration lock held by another migrator")
ErrMigrationLockHeld marks a migration attempt that ran nothing because another migrator's row was already in bun's lock table. bun's Lock is a TRY-lock -- an INSERT into bun_migration_locks, refused by the row's unique key while another migrator holds it -- so "held" says something about the world rather than about this attempt: another process is migrating, or is confirming there is nothing to migrate, right now. A node retries on its monitor tick (tryPing, via migrationsPending). A caller that must not proceed on an unmigrated schema and cannot wait for a tick -- dbtest.EnsureSchema -- tests for this with errors.Is on MigrationError() and waits the holder out.
Functions ¶
func BackfillRecipientEmailDigests ¶
func BackfillRecipientEmailDigests(ctx context.Context, db *bun.DB, logger *slog.Logger) (int, error)
BackfillRecipientEmailDigests stamps the digest onto every recipient whose newest version has none, and returns how many it stamped. Safe to call any number of times, and from any number of replicas at once. db is the pool the caller owns.
A pass whose context is cancelled ends between two batches, after the batches that committed, with an error wrapping the context's; the next call carries on.
func NewGatePool ¶
NewGatePool opens a pool for write gates over dsn, configured as the Database component configures its own: gateConnParams on every session, bounded at size connections (the configured size when size <= 0). The caller closes it.
Exported for the tests of the packages that take gates, which must run them on a pool shaped like the deployed one rather than on whatever pool they have.
func OpenMaintenanceSessionDSN ¶
OpenMaintenanceSessionDSN opens a maintenance session on dsn with the default opener. For the tests of the packages that run maintenance, which must run it on a session shaped like the deployed one rather than on whatever pool they have (NewGatePool's reason).
func RecipientEmailDigest ¶
RecipientEmailDigest is the key a lookup of an address uses: the stamp of the address, or "" when the text cannot be a mailbox (no "@", an empty local part or domain, whitespace inside). It is campaigns.EmailDigest, kept here because this package cannot import that one; see the header for what holds the two together.
func RecipientEmailStamp ¶
RecipientEmailStamp is the digest addRecipient stamps on a recipient and this sweep gives the ones that have none: the SHA-256, as lower-case hex, of the text trimmed of surrounding whitespace and lower-cased. Whatever the text is; see the header for why text that is not an address is stamped too.
func RevertSiteOwnerRestamp ¶
RevertSiteOwnerRestamp is the down direction: delete exactly the versions this migration appended, identified by their provenance. Every other version of every site is left as it was, so the rows read as they did before the up ran.
staged-data: MUST-NOT-GATE -- a MIGRATION must see every row, by definition (epic memql#3974, task memql#3984).
Types ¶
type Database ¶
type Database struct {
sync.Mutex
Logger *slog.Logger
DB *sql.DB
Bun *bun.DB
// DirectDB / DirectBun are the optional second, NON-pooled endpoint
// (DIRECT_DSN). Session-stateful work -- session-scoped advisory
// locks, leader election, migrations -- rides this so it bypasses the
// transaction pooler that recycles a server backend out from under a
// held session (epic memql#1925). nil when DIRECT_DSN is unset, in
// which case DirectBunDB() falls back to the main (pooled) Bun -- so
// local/dev without a pooler is unaffected (env-agnostic).
DirectDB *sql.DB
DirectBun *bun.DB
// Gate is the pool write gates run on (gate_pool.go): connections used
// for nothing else, so a gate holder never waits for one to do its work
// and a session sitting idle in a gate's transaction is not held to the
// request pool's limits. nil when it was not opened; there is no fallback
// to the request pool.
Gate *sql.DB
// contains filtered or unexported fields
}
func NewDatabase ¶
func NewDatabase(componentName common.ComponentName, args ...DatabaseArg) (*Database, error)
func (*Database) BunDB ¶
BunDB returns the current Bun database handle in a thread-safe manner. Returns nil if the database is not connected.
func (*Database) ClosePool ¶
ClosePool stops the connection monitor and closes the underlying pool, handing every open connection back to the database server. It is the explicit, intention-revealing entry point the drain sequence calls so a draining node releases its TigerDB sessions PROMPTLY instead of holding up to MaxOpenConns connections open until the process finally exits at the end of the blue-green scale-down window (memql#1875). Returns the pre-close pool stats so the caller can log how many connections were released.
The monitor is stopped first (via Stop) so the ticker reconnect loop can't race in and re-open the pool we're about to close. Closing is then done directly as well, so the method also works (and stays idempotent) when the lifecycle was never started or has already stopped -- a second call, e.g. the later dependency Stop sweep, is a harmless no-op.
func (*Database) ComponentName ¶
func (d *Database) ComponentName() common.ComponentName
func (*Database) DirectBunDB ¶
DirectBunDB returns the Bun handle for the direct (non-pooled) endpoint when DIRECT_DSN is configured, else falls back to the main (pooled) Bun handle. Session-stateful work -- session-scoped advisory locks, leader election, migrations -- must resolve its *bun.DB through this getter so it bypasses the transaction pooler the bulk pool rides; transaction-mode PgBouncer recycles a server backend between statements and would drop a held session-scoped lock (epic memql#1925). When DIRECT_DSN is unset the fallback keeps behavior identical to today (single pool), so local/dev is unaffected.
func (*Database) GateDB ¶
GateDB returns the pool write gates run on, or nil when it was not opened (before the database started, after it stopped, or under a custom SQL opener). There is no fallback to the request pool: a gate that cannot have its own connection must say so (component/memql: "shared write coordination is unavailable"), not take one from the pool its holder works on.
func (*Database) MigrationError ¶
MigrationError returns the error from the most recent migration attempt, or nil if migrations succeeded or were skipped (migrateOnStart=false). The gated pre-deploy migrate subcommand consults this to fail the deploy when a migration did not complete (#671).
func (*Database) MigrationLockHeldFor ¶
MigrationLockHeldFor is how long this node has found the same row in bun's lock table in the way of its attempts: zero when nothing is, or once its own attempt got the lock. It reads no database, so a /metrics scrape can call it.
func (*Database) OpenMaintenanceSession ¶
OpenMaintenanceSession opens one dedicated maintenance session (see the header): no server statement_timeout, no client read deadline beyond ctx's, on the direct endpoint when one is configured. The caller owns it and must call the returned func, which closes the session and the single-connection pool behind it. Every statement on it should run under a context with a deadline and a statement_timeout the caller sets.
func (*Database) PoolStats ¶
PoolStats returns a snapshot of the underlying *sql.DB connection-pool statistics (open / in-use / idle counts + wait metrics). The bool reports whether a live pool handle was present: false when the database has not connected yet or has already been closed. Surfaces per-node active- connection counts for the periodic pool-stats log and the drain sequence so a deploy can watch the cluster-wide connection total against the TigerDB ceiling (memql#1875).
func (*Database) Ready ¶
func (d *Database) Ready() <-chan struct{}
Ready returns a channel that is closed when the database is ready.
func (*Database) WithBunFactory ¶
func (d *Database) WithBunFactory(factory BunFactory) DatabaseArg
func (*Database) WithBunOptions ¶
func (d *Database) WithBunOptions(options ...BunOption) DatabaseArg
func (*Database) WithConnMaxIdleTimeMs ¶
func (d *Database) WithConnMaxIdleTimeMs(ms int) DatabaseArg
WithConnMaxIdleTimeMs sets the maximum amount of time a connection may be idle (in milliseconds).
func (*Database) WithConnMaxLifetimeMs ¶
func (d *Database) WithConnMaxLifetimeMs(ms int) DatabaseArg
WithConnMaxLifetimeMs sets the maximum amount of time a connection may be reused (in milliseconds).
func (*Database) WithGateOpener ¶
func (d *Database) WithGateOpener(opener SQLOpener) DatabaseArg
WithGateOpener sets the opener of the gate pool. The default opens a real pool over the DSN with gateConnParams; a test supplies its own (and, like the migration pool, a custom WithSQLOpener alone leaves no gate pool at all).
func (*Database) WithInitHook ¶
func (d *Database) WithInitHook(hook InitHook) DatabaseArg
func (*Database) WithLoggerLevel ¶
func (d *Database) WithLoggerLevel(level slog.Level) DatabaseArg
func (*Database) WithLoggerWriter ¶
func (d *Database) WithLoggerWriter(w io.Writer) DatabaseArg
func (*Database) WithMaxConnRetries ¶
func (d *Database) WithMaxConnRetries(retries int) DatabaseArg
func (*Database) WithMaxConnRetriesInterval ¶
func (d *Database) WithMaxConnRetriesInterval(intervalMs int) DatabaseArg
func (*Database) WithMaxIdleConns ¶
func (d *Database) WithMaxIdleConns(n int) DatabaseArg
WithMaxIdleConns sets the maximum number of idle connections in the pool.
func (*Database) WithMaxOpenConns ¶
func (d *Database) WithMaxOpenConns(n int) DatabaseArg
WithMaxOpenConns sets the maximum number of open connections to the database.
func (*Database) WithMigrateOnStart ¶
func (d *Database) WithMigrateOnStart(enabled bool) DatabaseArg
func (*Database) WithMigrationTimeoutMs ¶
func (d *Database) WithMigrationTimeoutMs(timeoutMs int) DatabaseArg
func (*Database) WithMigrations ¶
func (d *Database) WithMigrations(migrations *migrate.Migrations) DatabaseArg
func (*Database) WithMigrator ¶
func (d *Database) WithMigrator(m Migrator) DatabaseArg
func (*Database) WithPostMigrationHook ¶
func (d *Database) WithPostMigrationHook(hook PostMigrationHook) DatabaseArg
func (*Database) WithSQLDriver ¶
func (d *Database) WithSQLDriver(driverName string) DatabaseArg
func (*Database) WithSQLOpener ¶
func (d *Database) WithSQLOpener(opener SQLOpener) DatabaseArg
func (*Database) WithTickerInterval ¶
func (d *Database) WithTickerInterval(intervalMs int) DatabaseArg
type DatabaseArg ¶
func EnvOptionsToArgs ¶
func EnvOptionsToArgs(opts DatabaseEnvOptions) ([]DatabaseArg, error)
func OptionalCtorArg ¶
func OptionalCtorArg(name string, fn func(*config)) DatabaseArg
func RequiredCtorArg ¶
func RequiredCtorArg(name string, fn func(*config)) DatabaseArg
func WithDSN ¶
func WithDSN(dsn string) DatabaseArg
func WithDirectDSN ¶
func WithDirectDSN(dsn string) DatabaseArg
WithDirectDSN configures the optional second, NON-pooled DB endpoint (DIRECT_DSN). Session-stateful work (advisory locks, leader election, migrations) resolves its *bun.DB via DirectBunDB() so it bypasses the transaction pooler the main pool rides (epic memql#1925). An empty/blank value is a no-op: DirectBunDB() then falls back to the main pool, keeping behavior identical to today for local/dev without a pooler.
type DatabaseEnvKeys ¶
type DatabaseEnvKeys struct {
DSN string
DirectDSN string
MigrateOnStart string
MaxConnRetries string
MaxConnRetriesIntervalMs string
TickerIntervalMs string
MigrationTimeoutMs string
SQLDriver string
LoggerLevel string
// Connection pool configuration
MaxOpenConns string
MaxIdleConns string
ConnMaxLifetimeMs string
ConnMaxIdleTimeMs string
}
func DefaultDatabaseEnvKeys ¶
func DefaultDatabaseEnvKeys() DatabaseEnvKeys
type DatabaseEnvLoader ¶
type DatabaseEnvLoader struct {
Prefix string
Keys DatabaseEnvKeys
}
type DatabaseEnvOptions ¶
type DatabaseEnvOptions struct {
DSN string
DirectDSN string
MigrateOnStart *bool
MaxConnRetries *int
MaxConnRetriesIntervalMs *int
TickerIntervalMs *int
MigrationTimeoutMs *int
SQLDriver string
LoggerLevel string
// Connection pool configuration
MaxOpenConns *int
MaxIdleConns *int
ConnMaxLifetimeMs *int
ConnMaxIdleTimeMs *int
}
func LoadDatabaseEnvOptions ¶
func LoadDatabaseEnvOptions(loader DatabaseEnvLoader) (DatabaseEnvOptions, error)
type PostgresDatabase ¶
type PostgresDatabase struct {
*Database
}
func NewPostgresDatabase ¶
func NewPostgresDatabase(componentName common.ComponentName, args ...DatabaseArg) (*PostgresDatabase, error)
type SiteOwnerRestamp ¶
type SiteOwnerRestamp struct {
SiteId string
PackageId string
// OwnerUserId is the owner written, or empty for an unresolved row.
OwnerUserId string
// Source is SiteOwnerRestampSourceDeployment or
// SiteOwnerRestampSourcePackage, or empty for an unresolved row.
Source string
}
SiteOwnerRestamp is one site the migration decided about.
type SiteOwnerRestampReport ¶
type SiteOwnerRestampReport struct {
// Restamped lists every site that received a new version with its owner
// set.
Restamped []SiteOwnerRestamp
// Unresolved lists every blanked site left alone because neither a
// deployment nor the package row could name its owner.
Unresolved []SiteOwnerRestamp
}
SiteOwnerRestampReport is what one run of the migration did.
func RestampSiteOwners ¶
func RestampSiteOwners(ctx context.Context, db *bun.DB, logger *slog.Logger) (SiteOwnerRestampReport, error)
RestampSiteOwners runs the repair once, in one transaction, and reports what it did. Safe to call any number of times.
type TimescaleDBDatabase ¶
type TimescaleDBDatabase struct {
*PostgresDatabase
}
func NewTimescaleDBDatabase ¶
func NewTimescaleDBDatabase(componentName common.ComponentName, args ...DatabaseArg) (*TimescaleDBDatabase, error)
Source Files
¶
- 20260912000000_site_owner_restamp.go
- 20260913000000_memory_nodes_concept_id_created_at_idx.go
- 20260914000000_memory_nodes_concept_index_verified.go
- 20260921000000_module_readiness_history_collapse.go
- 20260926000000_work_recovery_index.go
- 20260926010000_work_journal_lookup_index.go
- 20261012000000_email_digest_lookup_index.go
- 20261012010000_email_provider_lookup_index.go
- 20261013000000_retention_indexes.go
- 20261013040000_secret_retention_index.go
- 20261014000000_work_run_owner_lookup_index.go
- 20261029100000_work_run_goal_lookup_index.go
- 20261029110000_work_run_owner_walk_index.go
- 20261029120000_work_run_signature_lookup_index.go
- 20261029130000_work_run_revision_lookup_index.go
- 20261030000000_site_access_lookup_indexes.go
- 20261030140000_auth_session_hash_lookup_index.go
- 20261111120000_inbound_failed_lookup_index.go
- 20261112110000_account_index.go
- conn_retry.go
- database.go
- gate_pool.go
- latest_row_index.go
- maintenance_session.go
- memory_nodes_compression.go
- payload_lookup_index.go
- payload_walk_index.go
- postgres.go
- recipient_email_digest.go
- timescaledb.go
Directories
¶
| Path | Synopsis |
|---|---|
|
Package dbtest provides shared setup for the Postgres-gated ("*_db_test.go") suites that CI runs together in the db-tests lane (.github/workflows/ci.yml) against ONE shared database.
|
Package dbtest provides shared setup for the Postgres-gated ("*_db_test.go") suites that CI runs together in the db-tests lane (.github/workflows/ci.yml) against ONE shared database. |
|
Package memoryNodes -- concept_parser.go
|
Package memoryNodes -- concept_parser.go |
|
Package storagequota owns durable, cluster-wide physical storage admission.
|
Package storagequota owns durable, cluster-wide physical storage admission. |