cronjob

Overview ¶

Package cronjob runs the jobs a project registers on a schedule.

A job is a func(ctx) error registered from a package init function with a cron expression — six fields, seconds first — or a descriptor such as "@hourly" or "@every 5m". Importing the package is what enables scheduling: the scheduler joins the process lifecycle from init, starts once the tables are ready and stops as soon as the process begins to drain; a project that registers no job never links it.

Every schedule is read in UTC, and "@every" runs on the multiples of its period counted from the Unix epoch, so every replica of a deployment computes the same instants for a job; the period is a whole number of seconds, at least one, and a schedule that never fires is refused at registration. An expression that must follow another wall clock says so itself, with a CRON_TZ= prefix. The instants are read off each replica's own clock — the lease decides who runs one, on the database's clock — so the replicas' clocks must agree to within the schedule's granularity, as any clock-synchronized deployment's do: a replica running ahead claims an instant early, and the others find it taken when their clocks reach it.

Each instant of a job is claimed once across the deployment: the replicas share the instant's lease through the primary database (see the lease package), the first to claim it runs the round while the others skip the instant, and a round still holding the lease — a second run included — keeps the next instants from everyone. A job that must run on every replica — refreshing a process-local cache, cleaning a local directory — registers with RegisterPerInstance and runs without a lease; the lease table comes with the package all the same, and stays empty for such a project. A ClickHouse primary database cannot carry leases, so a job under a lease fails the start there.

A round counts once it has run to its end: the job returned nil or an error of its own, or panicked. A round cut short — the job returned the ending of its context, ctx.Err() or context.Cause(ctx), as the process shut down or the lease was lost, or never returned because its process crashed or failed — runs a second time, on whichever replica finds it first: every replica looks for such rounds every 15 seconds or so, with one query for all its jobs. A failure of the job's own returned beside the ending counts only joined with it by errors.Join: attached by errors.CombineErrors it goes unseen, and the round reads as cut short. A round a shutdown cut short gives its lease back as it stops, and runs again within about 15 seconds; one whose process died waits for its lease to expire first, up to 15 seconds more, and so does one that ran to its end but whose end the database failed to record. The rule, exactly: the round did not run to its end, or its end is not on record; the instant has not run a second time already (a second round cut short is given up, so a job that brings its process down cannot bring the replicas down one after another); and no later instant has been claimed since (the next instant starting first gives the unfinished one up, with a warning naming it). A job under a lease may therefore run twice for one instant, and must be idempotent: the second round starts over, whatever the first did before it was cut short.

On start-up the scheduler catches up the most recent instant of a job when no replica claimed it, which is what a rolling deployment or an outage owes the job. The rule, exactly: the job has run before (its lease row exists — a job never run starts with its next instant, the instants before its first deployment were never its to run); the most recent instant that passed lies within the last day (an older one is history, not a missed round); and no replica claimed that instant, nor did a round that ran to its end overrun it (such a round records, as it ends, the instants it overran, and they stay skipped). The catch-up is one round, on one replica, and never repeats an instant already claimed; cut short, it runs a second time like any other round.

Each job runs in a loop of its own: the loop waits for the next instant of the schedule, or for a round of the job cut short that its replica found, runs the job on a context that ends when the process begins shutting down or the lease is lost, then computes the next instant from the moment the run ended — an instant that passed while a run was still in flight is skipped, never piled on top of it, and a round that ran to its end puts the skip on record, so no replica starting afterwards catches the instant up. The parser is the cron library's; the loop is this package's, because the library's runtime neither tells a job which instant it runs for nor lets a test drive the clock.

A registration that cannot be honored — no name, no schedule, a schedule that does not parse or names the process's own zone, a name already taken — fails the process at startup rather than dropping the job: the name is the job's identity in every log line about it and the name of its lease.