Documentation
¶
Overview ¶
Package lease is the coordination primitive the framework's distributed capabilities stand on: a name that at most one healthy process among those sharing the primary database holds at a time. The scheduler claims a name per job and instant, so a cluster claims each instant once, and a second time only for a round cut short; a capability that keeps a name for as long as its holder stays healthy, or claims one for the duration of a call, is built the same way.
A lease is one row of gst_leases, updated in place: who holds the name (holder, a token minted per claim and never reused), which process that is (instance, for reading logs), the term (incremented every time the name changes hands, and logged with every round and tenure) and when the lease expires (by the database clock, never a process clock). Four single-row statements do all the work; no lock outlives its own statement:
claim UPDATE gst_leases SET holder = :holder, instance = :instance,
term = term + 1, expires_at_ms = :now + 15000
WHERE name = :name AND expires_at_ms <= :now
1 row: claimed. 0 rows: held by someone.
No row yet: INSERT, worded to do nothing when the name is there
(INSERT IGNORE, ON CONFLICT DO NOTHING); 0 rows: someone was first.
renew UPDATE gst_leases SET expires_at_ms = :now + 15000
WHERE name = :name AND holder = :holder AND expires_at_ms > :now
0 rows: lost.
verify SELECT term FROM gst_leases
WHERE name = :name AND holder = :holder AND expires_at_ms > :now
Run as the first statement of every database.Transaction opened
under the lease; 0 rows: lost, and not one business statement
runs. Writes outside such a transaction are not checked: like
a Kubernetes leader, work stops through its context, not
through a check on every statement.
release UPDATE gst_leases SET expires_at_ms = 0
WHERE name = :name AND holder = :holder
0 rows: the lease was already gone; the work is done either way.
The scheduler claims a name per job and instant, and keeps three columns more on it: the last instant claimed (slot_ms, never decreasing), that instant again for as long as its round has not run to its end (unfinished_slot_ms, 0 once it has) and the last instant claimed a second time (rerun_slot_ms). A round cut short — its process shut down or died, or its lease was lost — thus stays on record, and runs again once. Only the last instant claimed can be unfinished: an unfinished_slot_ms other than slot_ms is settled. The scheduler's statements take the place of claim, and of release for a round that ran to its end — a round cut short gives its lease back with release — and add three:
claim an instant
SELECT term, slot_ms, unfinished_slot_ms, rerun_slot_ms,
expires_at_ms <= :now FROM gst_leases WHERE name = :name
then, when the lease expired and :slot is past slot_ms:
UPDATE gst_leases SET holder = :holder, instance = :instance,
term = term + 1, expires_at_ms = :now + 15000,
slot_ms = :slot, unfinished_slot_ms = :slot
WHERE name = :name AND term = :term AND expires_at_ms <= :now
1 row: claimed, and an unfinished_slot_ms read equal to the
slot_ms read is an instant given up for this one. 0 rows:
someone claimed first. No row yet: the INSERT, with both slots.
finish UPDATE gst_leases SET expires_at_ms = 0, unfinished_slot_ms = 0
WHERE name = :name AND holder = :holder
the release of a round that ran to its end. 0 rows — the lease
was lost meanwhile — records the end without the lease:
UPDATE gst_leases SET unfinished_slot_ms = 0
WHERE name = :name AND unfinished_slot_ms = :slot
find SELECT name, term, slot_ms FROM gst_leases
WHERE name IN (:names) AND unfinished_slot_ms = slot_ms
AND rerun_slot_ms < slot_ms AND expires_at_ms <= :now
the instants whose round was cut short and that are free to
claim again, for every job of a scheduler at once.
claim again
UPDATE gst_leases SET holder = :holder, instance = :instance,
term = term + 1, expires_at_ms = :now + 15000,
rerun_slot_ms = slot_ms
WHERE name = :name AND term = :term AND unfinished_slot_ms = slot_ms
AND rerun_slot_ms < slot_ms AND expires_at_ms <= :now
1 row: claimed; the term found holds the row to the instant
found. 0 rows: claimed again elsewhere, recorded finished, or
given up for a later instant.
last SELECT slot_ms FROM gst_leases WHERE name = :name
read as the scheduler starts, to tell an instant no replica
claimed from a job that has never run.
The instant claims read before they write because the claim of an instant has to know what it gives up. A term compared on the write makes the pair as exclusive as a single statement: every claim moves the term, so of replicas that read the same row, one write matches.
:now is the database server's clock in UTC milliseconds — MySQL UNIX_TIMESTAMP() with the milliseconds of NOW(3), PostgreSQL clock_timestamp(), SQLite unixepoch() — read without a round trip through any session time zone, so the processes of a deployment need not agree on the time. A ClickHouse primary database has none of this: the claim fails, and with it whatever capability asked for the lease.
The holder renews every 2 seconds, a renewal that failed included, each attempt waiting at most 5 seconds, and gives itself up 10 seconds after the last successful renewal started, 5 seconds before the database lets anyone else claim the name: a database that stalls or drops a statement for a few seconds costs the holder nothing, and a holder that cannot reach it stops before its successor can start. The context Hold returns ends at that moment, and the transactions opened under it end with it — the standard library rolls back a transaction whose context ends and refuses its Commit. Verify is the third line, for the transaction that would open after the loss, and Run the last: work that ignores all three and runs on after the loss fails the process, which then ends without waiting for it.
Importing the package is what brings leases into a process: the table joins the registered models and the transaction guard is installed from init, so a project that links no capability built on leases carries no table, no statement and no goroutine.
Index ¶
- Variables
- func Available() error
- func Hold(parent context.Context, h *Handle, log types.Logger) (ctx context.Context, stop context.CancelFunc)
- func LastSlot(ctx context.Context, name string) (time.Time, bool, error)
- func RetryInterval() time.Duration
- func Run(ctx context.Context, h *Handle, log types.Logger, ...) error
- func SetFail(fn func(error)) (restore func())
- func SetTimings(lease, renew, deadline, grace time.Duration) (restore func())
- func TermFromContext(ctx context.Context) (uint64, bool)
- func ValidateName(name string) error
- func Verify(ctx context.Context, base, tx *gorm.DB) error
- func WithHandle(ctx context.Context, h *Handle) context.Context
- type Handle
- type Unfinished
Constants ¶
This section is empty.
Variables ¶
var ( // ErrLost reports that the lease is no longer held: it expired and // someone else claimed the name, or it was released. Work under the // lease stops on it. ErrLost = errors.New("lease lost") // ErrUnsupportedDatabase reports a primary database the protocol cannot // run on: ClickHouse has neither the row-level updates nor the unique // keys a lease needs. ErrUnsupportedDatabase = errors.New("leases need a MySQL, PostgreSQL or SQLite primary database") )
Functions ¶
func Available ¶
func Available() error
Available reports whether the primary database can carry leases: nil on MySQL, PostgreSQL and SQLite, ErrUnsupportedDatabase on ClickHouse, and an error before the database is initialized. A capability built on leases checks it as it starts, so a deployment that cannot coordinate fails at startup instead of running every replica as if it were alone.
func Hold ¶
func Hold(parent context.Context, h *Handle, log types.Logger) (ctx context.Context, stop context.CancelFunc)
Hold keeps h renewed on a goroutine of its own and returns a context that ends the moment the lease is known to be lost — a renewal reports it gone, or none succeeds within localDeadline of the start of the last one that did — with ErrLost as the cause; parent ending ends it too. The work under the lease runs on that context: the transactions it opens end with it.
A renewal is due renewInterval after the previous attempt started, the claim counting as the first, and a renewal that failed is tried again at the same cadence, the way client-go renews: a database that drops a statement or stalls for a few seconds costs the holder nothing. Each attempt waits at most half of localDeadline — the bound client-go gives each of its requests — and never past the deadline, so an attempt that hangs leaves room for another and cannot carry the holder past it.
The renewals outlive parent on purpose. Parent ending tells the work to stop, and until it has — a round or a tenure winding down, a transaction rolling back — the name must stay this holder's, or another process could start the same work while it is still in flight. stop ends the renewals and returns once they have — a renewal in flight is cut short — without releasing the lease; the holder calls it once its work has returned, then releases.
The framework opens a single connection to SQLite, so there a renewal waits behind the work's own statements: a transaction of the work still open when a renewal falls due keeps the renewal from the connection, and once localDeadline has passed since the last renewal the lease counts as lost, the same as when the database cannot be reached. A transaction shorter than localDeadline minus renewInterval can never do that on its own, and one longer than localDeadline always does. Transactions back to back leave the renewals no turn of their own: a renewal that waited out one transaction succeeds, and the attempt due right after it waits out the next, bounded by half of localDeadline — so each transaction must be shorter than that.
log is the holder's own logger: a renewal that failed without losing the lease is what explains a round or a tenure cut short, so the entry belongs beside the holder's own rather than in the global stream.
func LastSlot ¶
LastSlot returns the last instant on record under name — the last claimed, or the last a round that ran to its end took along, see Finish — and whether the name has ever been claimed. The scheduler reads it as it starts, to tell an instant no replica claimed or skipped from a job that has never run at all.
func RetryInterval ¶
RetryInterval is how soon a caller tries again after the database failed to answer: the cadence the protocol already keeps against it, which a caller waiting for the same database has no reason to beat.
func Run ¶
func Run(ctx context.Context, h *Handle, log types.Logger, work func(ctx context.Context) error) error
Run runs work on ctx — the context Hold returned, or one derived from it — and returns what work returned. It is the protocol's last line. The deadline stops the renewals, the context ends the work and Verify refuses its next transaction; work that ignores all three and runs on once the lease is lost would run beside its successor's, the very thing the lease exists to rule out. So once ctx ends with ErrLost the work has stepDownGrace to return, and past that the process fails through lifecycle.FailNow — bootstrap ends Run without waiting for anything still running, this work included, and the orchestrator restarts the replica — while the wait goes on for as long as the process lasts, so the caller never claims again beside work of its own still running. ctx ending for any other reason — the process shutting down — is not a loss: Run waits for the work, and whoever stops the process bounds that wait. The renewals go on while the work winds down, though, and a lease they find lost meanwhile — h is lost, see Handle.Lost — leaves the work the grace like any other loss.
The work runs on a goroutine of its own, so that the lease being lost can be watched while it runs; a panic in it is recovered into an error carrying the stack of the panic site, which Run returns.
h is the handle the work runs under: its name is what a failure names. log is the holder's own logger: work that will not stop is the last thing said about a round or a tenure, so the entry belongs beside the holder's own rather than in the global stream.
func SetFail ¶
func SetFail(fn func(error)) (restore func())
SetFail replaces what Run does once work under a lost lease will not stop, and returns the function that restores it. The tests record the failure instead of ending the test process: the process-wide failure is one-way, so a test that tripped it could not run twice.
func SetTimings ¶
SetTimings replaces the protocol's timings and returns the function that restores them. The tests of the capabilities built on leases play the protocol out in milliseconds; a deployment runs what the declarations in lease.go say.
func TermFromContext ¶
TermFromContext returns the term of the lease ctx carries, and whether it carries one.
func ValidateName ¶
ValidateName reports whether name can be a coordinated name: not empty, and within the width of the name column. The capabilities validate the names they register at startup, so a name that could never be claimed fails the process instead of leaving work that silently never runs — MySQL would truncate the name on the claim's insert and PostgreSQL refuse it, every time — and claim checks again, so no path reaches the table with a name it cannot hold.
func Verify ¶
Verify is the transaction guard: run as the first statement of a transaction opened under a lease, it reports ErrLost when the lease is no longer this holder's — taken by another, released, or expired with no one to take it yet — so not one business statement of the transaction runs. A context carrying no lease passes. It is the third line behind the holder's own deadline and the context cancellation, for the transaction that would open after the loss.
The lease rows live on the primary database, so a transaction opened there runs the check on itself, and one opened on another instance — which has no lease table — is checked against the primary instead, still before its first statement.
Types ¶
type Handle ¶
type Handle struct {
// contains filtered or unexported fields
}
Handle is a claimed lease: the proof of holding a name from the claim until the release, or the loss.
func Claim ¶
Claim tries to take name for this process, once, without waiting: it returns the handle and true when the name was free — never claimed, expired or released — and false when someone holds it. An error means the database could not answer.
func ClaimRerun ¶
ClaimRerun claims u's instant a second time, u as UnfinishedSlots found it, and returns the handle and true when the instant is still free to claim a second time: the name has not changed hands since it was found. It returns false when the instant was claimed again elsewhere, its first round was recorded finished after all, a later instant was claimed, or the name is held. The round runs under the handle like any other and ends with Finish, or with Release when it is cut short again, which leaves the instant unfinished for good.
func ClaimSlot ¶
ClaimSlot is Claim for an instant of a scheduler's job, slot, under name: the name must be free, and slot later than the last instant on record under it — the last instant claimed, or the last one a round that ran to its end recorded as having come while it held the name, see Finish. The claim records slot as the last instant claimed, and as unfinished until Finish; it is refused when someone holds the name or slot is not later than the instant on record. A cluster then claims each instant of a job once, whichever replica gets there first, and an instant already claimed, or skipped while a round held the name, is refused everywhere; only a round cut short is claimed a second time, see ClaimRerun.
The instant claimed before under name, when its round has not run to its end — cut short, and not run a second time yet, or cut short the second time too — is given up by the claim and never runs again; the handle names it, see Handle.Superseded.
func FromContext ¶
FromContext returns the lease ctx carries, if any.
func (*Handle) Finish ¶
Finish records that the round for the instant h was claimed for ran to its end, so the instant never runs again, and gives the name back, the way Release does. passed is the last instant of the round's schedule to have come by the time the round ended. When it is later than the round's own instant, the round overran the instants up to it: none of them was claimed — a claim of one would have refused the round's own, and the round's lease refused every claim while it held the name — and the record takes passed along as the last instant on record, so they stay skipped, every later claim of them refused, a start-up catch-up included. An instant the database clock has yet to reach is not taken along, so a caller whose clock runs ahead does not skip an instant the other replicas are still to claim.
When the name is no longer this holder's the end is recorded all the same, unless a later instant has been claimed since, and passed is not taken along: other replicas may have claimed the name meanwhile. ErrLost then reports the name was not given back — unless h is known lost, see Lost, and giving it back was not tried. Only a handle of ClaimSlot or ClaimRerun has an instant to finish.
func (*Handle) Lost ¶
Lost reports whether the lease is known lost — a renewal found it gone, or none succeeded before the local deadline — whether or not the context of the work under it had already ended for another reason, such as the process shutting down. Only Hold learns of a loss, so a handle never held is never lost.
func (*Handle) Release ¶
Release gives the name up at once instead of letting the lease run out, so the next claimant need not wait. ErrLost reports the lease was already gone; the work it protected is done either way, so a caller logs it and moves on.
func (*Handle) Renew ¶
Renew extends the lease by leaseDuration from the database's now. ErrLost reports the lease expired — claimed by someone else since, or not yet, which the holder's own deadline makes unreachable in normal operation — or was released; any other error means the database could not answer, which is not yet a loss — Hold keeps trying until the local deadline.
func (*Handle) Superseded ¶
func (h *Handle) Superseded() (Unfinished, bool)
Superseded returns the instant the claim of h gave up, and whether it gave one up: the instant claimed before under the name, whose round had not run to its end — cut short, and not run a second time yet, or cut short the second time too — and that never runs again now. Only a handle of ClaimSlot gives one up.
type Unfinished ¶
type Unfinished struct {
// Name is the name the instant was claimed under.
Name string
// Slot is the instant.
Slot time.Time
// Rerun reports whether the instant had been claimed a second time
// already, and that round was cut short too. UnfinishedSlots never
// returns such an instant.
Rerun bool
// contains filtered or unexported fields
}
Unfinished is an instant claimed under a scheduler's name whose round has not run to its end: the process running it shut down or died, or its lease was lost, before the round returned.
func UnfinishedSlots ¶
func UnfinishedSlots(ctx context.Context, names []string) ([]Unfinished, error)
UnfinishedSlots returns the instants, under the names given, whose round was cut short and that are free to claim a second time: the round has not run to its end, the instant has not been claimed a second time already, no later instant has been claimed since, and the lease has expired or been released. One statement answers for all the names.
Source Files