Documentation
¶
Overview ¶
Package checkjobsvc provides check job queue operations for the distributed check runner system.
Index ¶
- Constants
- Variables
- func LeaseDuration(job *models.CheckJob, defaultTimeout, margin time.Duration) time.Duration
- func NoICMPFromCapabilities(capabilities []string) bool
- func ObserveRelease(ctx context.Context, logger *slog.Logger, job *models.CheckJob, region string, ...)
- func ParamOverlay(ctx context.Context, store secretref.ParamStore, orgUID string, ...) (map[string]any, error)
- func PerCheckTimeout(config map[string]any) (time.Duration, bool)
- func PluginTypesFromCapabilities(capabilities []string) []string
- func PluginTypesFromContext(ctx context.Context) []string
- func ResultReason(outcome SecretMerge, err error) error
- func WithNoICMP(ctx context.Context, noICMP bool) context.Context
- func WithPluginTypes(ctx context.Context, types []string) context.Context
- func WithSealFingerprints(ctx context.Context, fingerprints []string) context.Context
- type AgentScope
- type Option
- type SecretMerge
- func MergeJobSecrets(ctx context.Context, creds credentials.Service, job *models.CheckJob) (SecretMerge, error)
- func OpenJobSecrets(ctx context.Context, creds credentials.Service, job *models.CheckJob) (map[string]any, SecretMerge, error)
- func OpenSecretsEnvelope(ctx context.Context, creds credentials.Service, orgUID string, ...) (map[string]any, SecretMerge, error)
- type Service
Constants ¶
const CheckTimeoutConfigKey = "timeout"
CheckTimeoutConfigKey is the check-config key holding the optional per-check timeout duration string (spec 2026-07-11-05).
const DefaultLeaseCheckTimeout = 15 * time.Second
DefaultLeaseCheckTimeout is the execution ceiling the lease assumes for a check without its own `timeout` when the deployment sets none. It mirrors the default SP_SCHEDULING_CHECK_TIMEOUT_MS.
const DefaultLeaseMargin = 30 * time.Second
DefaultLeaseMargin is the lease slack used when the deployment sets none.
const MaxPerCheckTimeout = 30 * time.Second
MaxPerCheckTimeout is the clamp on a check's own `timeout` config value (spec 2026-07-11-05), shared by the worker's execution budget and the lease. It mirrors the uniform cap enforced at check create/update time; the defensive clamp keeps legacy stored configs written under the old 60s per-checker caps from extending the execution context past 31s.
Variables ¶
var ErrInvalidAgentScope = errors.New("invalid agent claim scope")
ErrInvalidAgentScope is returned when a claim scope is not a scope any agent can legitimately have. It fails the claim closed rather than widening it.
var ErrJobClaimedByAnother = errors.New("job may have been claimed by another worker")
ErrJobClaimedByAnother is returned when a job has been claimed by another worker.
var ErrJobGone = errors.New("job no longer exists")
ErrJobGone is returned by a release when the job row no longer exists: the check was deleted or disabled, or the region removed, while the run was in flight (spec 2026-10-12-05). Its result is stored; there is nothing left to reschedule, so callers treat it as the normal end of that job.
var ErrSecretsOrgKeyUndecryptable = errors.New(
"this organization's encryption key could not be opened on this worker — " +
"re-saving the check will not help; check the worker logs for \"unwrap org DEK\"",
)
ErrSecretsOrgKeyUndecryptable is the reason attached to a job whose ORGANIZATION key could not be opened at all — the layer above this check's envelope. Re-saving the check cannot help (the save path needs the same key), so the message must not send the operator down that road: this is a wrong or missing master key on this process, or an unreadable DEK row, and the answer is in the worker's logs.
"check has encrypted credentials but SP_ENCRYPTION_MASTER_KEY is not configured on this worker",
)
ErrSecretsUnavailable is the reason attached to a job whose config_private envelope cannot be opened because no master key is configured on this process. It is user-visible (it lands in the error result's output on the in-process path), so it names the fix rather than the internals.
var ErrSecretsUndecryptable = errors.New(
"check credentials could not be decrypted — re-save the check's credentials",
)
ErrSecretsUndecryptable is the reason attached to a job whose envelope exists and a master key is configured, but decryption failed (wrong key, missing org DEK, tampered ciphertext). The underlying error is logged, never surfaced — it can carry cryptographic detail.
Functions ¶
func LeaseDuration ¶
LeaseDuration is how long a lease taken on job lasts, counted from max(scheduled_at, now): max(period, check timeout) + margin (spec 2026-11-11-06). The check timeout is the job's own `timeout` when set, defaultTimeout otherwise. Sizing on the timeout too means a check whose probe may run longer than its period still owns its row until it releases it, and the margin absorbs the claim, result and release round-trips — the part a remote region with a slow database link pays on every run.
func NoICMPFromCapabilities ¶
NoICMPFromCapabilities reports whether a capability report is a real "no ICMP": a non-nil (closed) set without the icmp name. A nil set is "unknown" and keeps claiming everything.
func ObserveRelease ¶
func ObserveRelease(ctx context.Context, logger *slog.Logger, job *models.CheckJob, region string, err error)
ObserveRelease records the outcome of the release that ends a check run (spec 2026-11-11-06): the claim-to-release time, when the job carries its claim time, and a lost lease — the release refused with ErrJobClaimedByAnother because the lease expired and a peer re-claimed the job. The lost-lease log line carries what is needed to size SP_CHECKERS_LEASE_MARGIN from it alone: when the lease expired, when the release came, by how much it was missed, the check period and timeout.
region is the region of the result (the job's, falling back to the worker's); an empty region is reported as such.
func ParamOverlay ¶
func ParamOverlay( ctx context.Context, store secretref.ParamStore, orgUID string, config map[string]any, ) (map[string]any, error)
ParamOverlay resolves the `${param:KEY}` references in a config and returns ONLY the keys whose value actually changed — the overlay to lay over the stored config to obtain the effective one.
It is the server half of "store the reference, resolve at execution" (spec 2026-09-11-03). `param:` is org data the API owns, so it is resolved here, at the claim/dispatch boundary, and never written back to any row:
- on the in-process path the overlay is merged into the claimed job's in-memory config, next to the secrets MergeJobSecrets already merged;
- on the agent path the overlay is folded into the SEALED payload, where the agent's own merge puts it over the reference still sitting in the public wire config — so a resolved value never crosses the wire in clear.
`${env:}` is deliberately left alone (secretref.APIResolver skips it): it belongs to whichever process executes the check, which for a deported agent is that agent's own environment.
An unresolvable reference is an error the caller turns into an explicit error result, exactly as it does for an unopenable envelope: running the check with a literal "${param:…}" in its body would quietly probe the wrong thing, and skipping it silently would wedge the job with nothing in the check's history to explain it.
func PerCheckTimeout ¶
PerCheckTimeout extracts the optional per-check `timeout` duration from a job's config. Returns (0, false) when the key is absent, not a duration string, or non-positive. Valid values are clamped at MaxPerCheckTimeout.
func PluginTypesFromCapabilities ¶
PluginTypesFromCapabilities picks the plugin check types out of an agent's capability report: a plugin type is a valid capability name (`x-…`).
func PluginTypesFromContext ¶
PluginTypesFromContext returns the plugin types advertised on ctx.
func ResultReason ¶
func ResultReason(outcome SecretMerge, err error) error
ResultReason maps a failed secrets open to the static, user-visible reason to record on the check result. It names WHICH layer failed — the org key or this check's envelope — because the two have opposite remedies. The underlying error is only ever logged: it can carry cryptographic detail.
func WithNoICMP ¶
WithNoICMP returns a context for a claim by a worker whose ICMP probe said no: it must not claim icmp jobs (spec 2026-10-28-06).
func WithPluginTypes ¶
WithPluginTypes returns a context advertising the plugin check types the claiming worker can run (spec 2026-10-17-05 §17). A claim made without it advertises none.
func WithSealFingerprints ¶
WithSealFingerprints returns a context for a claim by an agent that can open blobs sealed to these fingerprints: its own key and the region keys it advertises (spec 2026-11-08-01 §7). A sealed job is then only handed out if its blob lists one of them, so an agent never claims a job it cannot open. Without it, no filter applies.
Types ¶
type AgentScope ¶
type AgentScope struct {
// OrgUID pins the claim to exactly one organization. It is empty ONLY for a
// system agent.
OrgUID string
// Region is matched on exact equality for both kinds — never a prefix,
// never a NULL-region fallback.
Region string
// System marks a platform-operated agent, whose scope is the region alone
// (the same scope the in-cluster DirectBackend claims with). Tenant-private
// agents leave it false.
System bool
// PluginTypes are the plugin check types the agent loaded (from its
// capability report). A plugin-type job is only handed to an agent that
// advertises its type (spec 2026-10-17-05 §17).
PluginTypes []string
// NoICMP is set when the agent reported it cannot open an ICMP socket: it
// is not handed icmp jobs (spec 2026-10-28-06).
NoICMP bool
}
AgentScope is the claim scope of one enrolled agent. It exists so the widening a system agent needs (serving a shared cloud region across every org) can never happen by accident: an org agent's scope carries its OrgUID, and dropping the organization predicate requires System to be set explicitly.
type Option ¶
type Option func(*serviceImpl)
Option tunes a check job service.
func WithCheckTimeout ¶
WithCheckTimeout sets the execution ceiling the lease assumes for a check without its own `timeout`. A non-positive value keeps DefaultLeaseCheckTimeout.
func WithLeaseMargin ¶
WithLeaseMargin sets the slack added to every lease. A non-positive value keeps DefaultLeaseMargin.
type SecretMerge ¶
type SecretMerge int
SecretMerge is the outcome of MergeJobSecrets.
const ( // SecretMergeNoop means the job carries no config_private envelope at all: // the check simply has no secret fields (secrets, when present, are always // split into config_private now — as an AES-GCM envelope with a master key, // or a plaintext envelope without one). The job is dispatchable as-is. SecretMergeNoop SecretMerge = iota // SecretMergeMerged means the envelope was decrypted and merged into // job.Config, and the envelope was stripped off the job. SecretMergeMerged // configured on this process, so it can never be opened. SecretMergeUnavailable // SecretMergeFailed means an envelope exists and a key is configured, but // decryption failed. SecretMergeFailed )
func MergeJobSecrets ¶
func MergeJobSecrets( ctx context.Context, creds credentials.Service, job *models.CheckJob, ) (SecretMerge, error)
MergeJobSecrets decrypts a claimed job's config_private envelope for its own org and merges the plaintext into job.Config in place, mirroring the dispatch-time semantics every executor depends on: a checker must receive one merged config map and must never see the envelope.
It is the merge rule shared by every claim path that dispatches a job to a checker — currently the in-process claim path (checkworker/backend.DirectBackend) — so the strip-the-envelope invariant and the failure taxonomy live here once.
It deliberately does NOT decide what happens on failure: the caller does (the in-process backend writes an explicit error result). It never logs, and never returns a config value in an error — the decrypted map is a secret and only ever reaches job.Config.
creds may be nil (a worker built without a credentials service): that is treated exactly like a disabled service.
func OpenJobSecrets ¶
func OpenJobSecrets( ctx context.Context, creds credentials.Service, job *models.CheckJob, ) (map[string]any, SecretMerge, error)
OpenJobSecrets decrypts a job's config_private envelope and RETURNS the plaintext secret map without touching the job. It is the shared half of MergeJobSecrets: the same envelope taxonomy (plaintext envelope / no key / decrypt failure) for callers that must not merge the secrets into the config.
The system-agent claim path (spec 2026-07-27-01) is exactly such a caller: it re-seals these secrets to the claiming agent's X25519 key instead of shipping them merged in the wire config, so the plaintext never leaves the server on the agent transport. SecretMergeNoop means "no envelope, nothing to seal"; SecretMergeMerged means the returned map is the opened plaintext.
Like MergeJobSecrets it never logs and never puts a config value in an error.
func OpenSecretsEnvelope ¶
func OpenSecretsEnvelope( ctx context.Context, creds credentials.Service, orgUID string, envelope *string, ) (map[string]any, SecretMerge, error)
OpenSecretsEnvelope is the envelope-opening primitive behind MergeJobSecrets and OpenJobSecrets, taking the envelope and its owning org directly so a caller holding a check row (not a job row) — e.g. the SSH-tunnel block a system agent's claim must re-seal — reuses the exact same taxonomy.
type Service ¶
type Service interface {
// ClaimJobs atomically claims due check jobs for the given worker with
// per-lane reservation (spec 2026-07-01-03 D3/D6). fastLimit is the total
// claim capacity (normally the worker's free runner slots). slowLimit is
// the slow lane's reservation budget: a worker passing slowLimit =
// (poolSize − fastLaneReserved) − busySlow guarantees slow jobs can never
// occupy the reserved fast floor. The slow lane claims its budgeted share
// FIRST (capped at min(slowLimit, fastLimit)); fast then fills every
// remaining slot. Slow-first is load-bearing: fast claim-ahead keeps the
// fast lane permanently eligible on fleets whose periods fit inside
// maxAhead, so a leftovers-only slow allowance would be zero forever and
// starve the slow lane outright. Both SELECTs run in one transaction.
//
// The bulk lane (website crawls) is never claimed here: a crawl run is a
// worker task (spec 2026-10-22-03), materialized by workertasks.Claim.
// Lease duration is calculated per job as scheduled_at + period + 30s.
// Returns claimed jobs (nil if none available) plus a next-eligible hint:
// how long until the earliest still-unleased job in scope becomes
// claimable (0 = none within the hint horizon). The worker's fetcher
// sleeps on that hint instead of a flat fallback poll, which is what
// keeps sub-minute periods honest on an otherwise idle worker.
ClaimJobs(
ctx context.Context,
workerUID string,
region *string,
fastLimit int,
slowLimit int,
maxAhead time.Duration,
) ([]*models.CheckJob, time.Duration, error)
// ClaimJobsForCheck atomically claims any due check_jobs rows for the
// given checkUID. Used by the express runner that wakes up on
// check.created events and bypasses the regular runner pool so a
// freshly-created check produces its first real result without
// queueing behind in-flight long-running checks.
ClaimJobsForCheck(
ctx context.Context,
workerUID string,
region *string,
checkUID string,
) ([]*models.CheckJob, error)
// ClaimJobsForAgent atomically claims due jobs for a deported agent
// (spec 2026-07-16-02, generalized by 2026-07-27-01). The scope is HARD in
// both directions: the region always matches on exact equality — no prefix
// matching, no NULL-region fallback — and an org agent additionally matches
// exactly its own organization, so a compromised or misconfigured
// tenant-private agent can never claim another org's or another region's
// work. A system (platform-operated) agent serves a SHARED cloud region and
// therefore claims that region across all orgs, exactly like the in-cluster
// DirectBackend; see AgentScope, which fails closed on an ambiguous scope.
//
// checkUID, when non-empty, further pins the claim to one check (the agent
// express path). No lane split applies (an agent pool is not the server's
// fast/slow reservation); ordering stays cost-aware via
// effective_scheduled_at. The second return is the same next-eligible hint
// as ClaimJobs, computed over the agent's whole scope regardless of any
// checkUID pin, so the agent's fetcher always learns when to come back.
ClaimJobsForAgent(
ctx context.Context,
workerUID string,
scope AgentScope,
checkUID string,
limit int,
maxAhead time.Duration,
) ([]*models.CheckJob, time.Duration, error)
// ClaimPassiveJobs claims due passive (heartbeat, email) jobs for the jobs
// node's passive evaluator — their ONLY claimer (spec 2026-09-25-04). Every
// other claim (cloud workers, their express path, org and system agents)
// excludes passive types, so a dead region can no longer silence a
// dead-man's switch and an agent can no longer turn its evaluation into an
// error.
//
// Scope: region IS NULL and a passive type, due now (no claim-ahead — the
// evaluator does not sleep in-slot), unleased or lease expired. Rows are
// locked FOR UPDATE SKIP LOCKED on Postgres and leased with the normal
// lease (scheduled_at + period + 30s) inside one transaction, so two jobs
// nodes can never evaluate the same tick twice. The second return is the
// usual next-eligible hint over the same scope.
ClaimPassiveJobs(
ctx context.Context,
workerUID string,
limit int,
) ([]*models.CheckJob, time.Duration, error)
// ReleaseLease releases the lease and reschedules the job for next execution.
// It folds in no fresh cost/delay sample (the probe was skipped, or ran on a
// backend that does not measure cost), so it re-anchors effective_scheduled_at
// to the new schedule; the cost offset is reapplied on the next
// post-exec write via ReleaseLeaseWithSchedulingState.
//
// This is the *terminal* release: the attempt is over (an error result was
// written, or the reaper gave up on a stuck job) and the job starts its next
// period with a clean slate. A job that was merely DEFERRED — the per-org
// rate limiter turned it away before the probe ran — must use
// DeferLeaseRateLimited instead, or it loses the overdue-ness that earns it
// priority next window.
ReleaseLease(ctx context.Context, jobUID string, workerUID string, nextScheduledAt time.Time) error
// DeferLeaseRateLimited releases the lease of a job the per-org
// MaxChecksPerMinute gate turned away before it ran, advancing scheduled_at
// to the next tick but DELIBERATELY PRESERVING effective_scheduled_at
// (spec 2026-08-26-02).
//
// That one omitted assignment is the whole fairness mechanism. Claim order
// is `ORDER BY effective_scheduled_at ASC`, so a job whose ordering key
// stays pinned at the tick it missed keeps receding relative to now and
// sorts ahead of its on-time org siblings in the next window. An org over
// its cap therefore rotates the deficit round-robin instead of starving the
// same deterministic-phase losers forever. Re-anchoring here (what
// ReleaseLease does) is what let a 1-minute check go 7.5 hours without a
// single execution while its org ran at full rate.
//
// No busy-loop risk: the claim predicate still gates on
// `scheduled_at <= now + ahead`, so a deferred job cannot be re-claimed
// inside the window it was just turned away from.
//
// cost_ewma_ms, delay_ewma_ms and lane are left untouched as well: the probe
// never ran, so there is no sample to fold in, and the delay EWMA is the
// diagnostic that exposed this pathology in production.
DeferLeaseRateLimited(ctx context.Context, jobUID string, workerUID string, nextScheduledAt time.Time) error
// ReleaseWorkerLeases frees every check-job lease workerUID holds that was
// taken before claimedBefore (spec 2026-11-14-03). A restarted process
// comes back under the same worker UID but does not know it holds those
// leases, so without this nobody runs them until lease_expires_at, up to a
// whole period later.
//
// The job becomes due at once: scheduled_at and effective_scheduled_at are
// left alone, and the job is already overdue. A "Capture now" request the
// lease carried goes back to pending, and a parked run request stays (no
// probe answered either). lease_starts is kept, not reset: the lease ended
// without a release, which is what the crash-loop counter counts, and the
// next claim adds one to it. The multi-step pointers (step_run_*) are kept
// too, like the stuck-job reaper does, so the next claim resumes the run.
//
// claimedBefore guards against a sibling sharing the slug (two pods with
// the same SP_NODE_NAME during a rolling update): leases it claims after
// this process started are never touched. A claim sets updated_at, so a
// lease with updated_at at or after claimedBefore is left to expire.
// Returns the number of jobs freed.
ReleaseWorkerLeases(ctx context.Context, workerUID string, claimedBefore time.Time) (int, error)
// ReleaseLeasesOnNewBoot records bootID, the random ID a deported agent
// process sends on enroll and reconnect, on its workers row, and frees
// the worker's leases (ReleaseWorkerLeases) only when it replaces a
// DIFFERENT stored boot ID (spec 2026-11-14-03). A reconnect from the
// same process (a network blip) keeps its in-flight leases, so the result
// it submits afterwards is still accepted. An empty bootID (an agent too
// old to send one) does nothing. A first report on a row that has none
// only stores it: there is no way to tell whether the process changed.
// Both writes share one transaction, so a failed release leaves the old
// boot ID in place and the next connection tries again.
ReleaseLeasesOnNewBoot(ctx context.Context, workerUID string, bootID string) (int, error)
// ReleaseLeaseWithSchedulingState releases the lease, reschedules, and folds
// the post-exec cost and delay signals into the row in the same write: it
// stores the updated cost and delay EWMAs, the recomputed
// effective_scheduled_at used for cost-aware, plan-weighted claim ordering
// (spec 2026-06-30-09), and the hysteresis-classified lane (spec
// 2026-07-01-03). No extra query on the hot path — it reuses the single
// post-exec UPDATE. Callers without a fresh sample (rate-limit deferral,
// reaper, remote backends) use ReleaseLease instead, which leaves the lane
// unchanged.
ReleaseLeaseWithSchedulingState(
ctx context.Context,
jobUID string,
workerUID string,
nextScheduledAt time.Time,
costEWMAMs float64,
delayEWMAMs float64,
effectiveScheduledAt time.Time,
lane uint8,
) error
}
Service provides check job queue operations.