Documentation
¶
Overview ¶
Package watchdog is the platform's self-monitoring detector set (spec 2026-08-24-10).
The worst failure mode of a monitoring product is going blind, and that is exactly the failure that produces zero signal: a check that stops being executed keeps its last state forever, so the dashboard shows green rows with quietly aging timestamps. Check-level alerting (incidents → escalation policies) cannot report on itself. This package is the channel that can.
It holds only the parts that are testable without a job runner: the detectors, the anti-flood transition state machine, and the digest renderer. Scheduling and delivery live in the `platform_watchdog` job (internal/jobs/jobtypes/job_platform_watchdog.go).
Index ¶
- Constants
- Variables
- func ClaimNotification(ctx context.Context, dbService db.Service, fingerprint string, ...) (bool, error)
- func ClearNotification(ctx context.Context, dbService db.Service, fingerprint string) (bool, error)
- func DarkRegionFingerprint(region string) string
- func PublishMetrics(report *Report)
- func ValidateParameter(value any) error
- type Anomaly
- type Config
- type Digest
- type RegionHealthReporter
- type Report
- type Service
- type Severity
- type Transition
- type TransitionKind
Constants ¶
const ( // DefaultInterval is how often the job wakes up. DefaultInterval = time.Hour // DefaultRenotifyAfter bounds re-notification of an ONGOING anomaly. DefaultRenotifyAfter = 24 * time.Hour // DefaultDarkRegionMinOverdueJobs is the blast-radius floor: 419 stranded // jobs is page-worthy, one job 90 seconds late is not. DefaultDarkRegionMinOverdueJobs = 5 // DefaultDarkRegionMinOverdueAge is how late the oldest overdue job must // be before the region counts as an anomaly. DefaultDarkRegionMinOverdueAge = 10 * time.Minute // DefaultDarkRegionCriticalJobs escalates a dark region to critical. DefaultDarkRegionCriticalJobs = 50 // DefaultDarkRegionCriticalAge escalates a dark region to critical on // duration alone — a handful of jobs stranded for half an hour is still // a region nobody is serving. DefaultDarkRegionCriticalAge = 30 * time.Minute // DefaultFleetDropPercent is the drop, against the same hour a day // earlier, that counts as a collapse. DefaultFleetDropPercent = 50.0 // DefaultFleetMinBaseline keeps a quiet instance (or a brand-new one) // from reporting a collapse: without a floor, 4 results vs. 10 is a 60% // "collapse". DefaultFleetMinBaseline = 100 // DefaultFleetCriticalDropPercent escalates a collapse to critical. DefaultFleetCriticalDropPercent = 80.0 // DefaultStaleIncidentMinAge is the floor of max(N × period, this). DefaultStaleIncidentMinAge = 15 * time.Minute // DefaultStaleIncidentPeriodMultiplier is the N of max(N × period, floor). DefaultStaleIncidentPeriodMultiplier = 3 // DefaultStaleIncidentCriticalCount escalates the stale-incident anomaly // to critical — 61 frozen incidents was the 2026-08-24 number. DefaultStaleIncidentCriticalCount = 10 // DefaultStaleChecksCriticalCount escalates the stale-checks anomaly // (stale checks in regions that are NOT dark) to critical. DefaultStaleChecksCriticalCount = 10 // DefaultStaleIncidentScanLimit bounds the active-incident scan so the // run stays cheap on an instance with a genuinely huge outage. DefaultStaleIncidentScanLimit = 2000 )
Watchdog defaults. Every one of them is overridable through the system parameter; these are the values an operator gets by writing nothing but `{"enabled": true, "recipients": ["<uid>"]}`.
const ( // DetectorDarkRegion reports regions whose assigned work nothing live can // claim (the 2026-08-24 stranding). DetectorDarkRegion = "dark-region" // DetectorFleetCollapse reports an instance-wide drop in produced results // against the same hour a day earlier — the catch-all for stranding // causes this spec did not imagine. DetectorFleetCollapse = "fleet-collapse" // DetectorStaleIncidents reports active incidents whose check has stopped // producing results (the "frozen incident" symptom). DetectorStaleIncidents = "stale-incidents" // DetectorStaleChecks reports checks in the `stale` ("No data") status // whose regions are NOT dark (spec 2026-09-25-02): a platform bug — // scheduler, leases, rate limiting — rather than a region outage. DetectorStaleChecks = "stale-checks" )
Detector names. They are the stable half of an anomaly fingerprint and of the per-detector metric label, so they must never be reworded casually — a rename resets every operator's anti-flood state to "new".
const ( // SeverityTokenInfo is the "info" token. SeverityTokenInfo = "info" // SeverityTokenWarning is the "warning" token, and the default bar. SeverityTokenWarning = "warning" // SeverityTokenCritical is the "critical" token. SeverityTokenCritical = "critical" )
Severity tokens, as they appear in the system parameter and the digest.
const ParamPlatformWatchdog = "platform_watchdog"
ParamPlatformWatchdog is the system-parameter key the watchdog is configured through. It rides the existing super-admin /api/v1/system/parameters CRUD, so there is no new settings surface and no redeploy needed to add an operator.
const StateKeyPrefix = "watchdog:anomaly:"
StateKeyPrefix namespaces the per-anomaly anti-flood markers. They are GLOBAL state entries (organization_uid IS NULL) because the watchdog reports on the platform, not on a tenant.
const SubjectFleet = "instance"
SubjectFleet is the stable subject of the instance-wide execution-rate anomaly. Instance-wide detectors need a constant subject: a fingerprint that churned between runs would re-notify forever and never resolve.
const SubjectStaleChecks = "unexplained"
SubjectStaleChecks is the stable subject of the stale-checks anomaly: one anomaly per run carrying a count, never one per check.
const SubjectStaleIncidents = "active"
SubjectStaleIncidents is the stable subject of the frozen-incident anomaly. One anomaly per run carrying a count, not one per incident: 61 frozen incidents must produce one line in one digest, never 61 pages.
Variables ¶
var ( // ErrInvalidParameterShape is returned when the value is not a JSON object. ErrInvalidParameterShape = errors.New("platform_watchdog must be a JSON object") // ErrInvalidRecipient is returned for a blank entry in recipients. ErrInvalidRecipient = errors.New("platform_watchdog recipients must be non-empty user UIDs") // ErrInvalidMinSeverity is returned for an unknown minSeverity token. ErrInvalidMinSeverity = errors.New(`platform_watchdog minSeverity must be one of "info", "warning", "critical"`) )
Parameter validation errors.
ErrRegionHealthUnavailable is reported by the dark-region detector when no RegionHealth reporter was wired. It is deliberately an ERROR rather than a silent skip: a watchdog that quietly stops looking at the one failure mode it was written for is worse than no watchdog.
Functions ¶
func ClaimNotification ¶
func ClaimNotification( ctx context.Context, dbService db.Service, fingerprint string, severity Severity, headline string, now time.Time, escalation bool, ) (bool, error)
ClaimNotification records, in the shared anti-flood ledger, that a notification for fingerprint is being delivered NOW at severity, and reports whether the caller should deliver it.
It is the entry point the per-minute region sweep uses, so its transitions and the hourly digest read the same marker:
- no marker: nobody told the operator yet — write it (exactly the shape Reconcile writes for a notified anomaly) and deliver;
- a marker, and escalation false: the operator already heard about this outage (from the digest, which grades severity on its own scale) — stay quiet;
- a marker, escalation true (the caller's own stalled → dark), notified at a LOWER severity: that is news, so update the notified severity and deliver.
Only call it when there is someone to deliver to: a marker written for a notification that never went out would suppress the real one later.
func ClearNotification ¶
ClearNotification removes a fingerprint from the shared ledger and reports whether it was there — i.e. whether the operator was told about the condition and so must now be told it is over.
func DarkRegionFingerprint ¶
DarkRegionFingerprint is the anti-flood fingerprint of a CLOUD region's dark-region anomaly — the marker the hourly detector and the per-minute region sweep share, so one outage is announced to the operator once, whichever of the two sees it first (spec 2026-09-25-03).
func PublishMetrics ¶
func PublishMetrics(report *Report)
PublishMetrics exports the run's findings on the Prometheus registry.
It publishes EVERY anomaly, not just the ones past minSeverity: the out-of-band path exists so an external Prometheus can alert on its own thresholds, independently of what the in-band digest was configured to deliver.
func ValidateParameter ¶
ValidateParameter rejects a platform_watchdog value the watchdog could not decode at run time.
This runs on the write, not on the read, on purpose. A malformed value only surfaces at the next hourly run — as a failing job on an alerting path that nobody is watching, which is precisely the failure mode this spec exists to remove. Catching it in the PUT hands the mistake back to the person making it, while they are still looking.
Types ¶
type Anomaly ¶
type Anomaly struct {
// Detector is one of the Detector* constants.
Detector string
// Subject identifies WHAT is broken within the detector's domain — a
// region slug, or the sentinel "fleet"/"global" for instance-wide
// detectors. Together with Detector it forms the fingerprint, so it must
// be stable across runs for the same underlying condition: an anomaly
// whose subject churns would re-notify forever.
Subject string
Severity Severity
// Headline is the one-line summary carrying the numbers that matter.
Headline string
// Detail is optional extra context (the three oldest frozen incidents,
// the last-worker-seen timestamp, …).
Detail string
// Remediation is the ready-to-run fix, when there is one.
Remediation string
// Count is the detector's headline magnitude — overdue jobs for a dark
// region, frozen incidents for stale-incidents, lost results for a fleet
// collapse. Kept as a first-class field rather than re-parsed out of the
// rendered headline so the gauges and the digest read the same number.
Count int
}
Anomaly is one thing the platform found wrong with itself.
func Filter ¶
Filter returns the anomalies at or above minSeverity. Filtering applies to DELIVERY only — every anomaly is logged and metered regardless, because the out-of-band Prometheus path must see everything.
func (*Anomaly) Fingerprint ¶
Fingerprint is the anti-flood identity of an anomaly: `<detector>:<subject>`.
type Config ¶
type Config struct {
// Enabled gates the whole job. False means the run has NO side effects at
// all: no queries, no state writes, no metrics, no delivery.
Enabled bool `json:"enabled"`
// Recipients are user UIDs. Delivery goes through each user's own enabled
// notification routes — no new medium, no hardcoded webhook.
Recipients []string `json:"recipients"`
// MinSeverity filters DELIVERY only.
MinSeverity string `json:"minSeverity"`
IntervalMinutes int `json:"intervalMinutes,omitempty"`
RenotifyAfterMinutes int `json:"renotifyAfterMinutes,omitempty"`
DarkRegionMinOverdueJobs int `json:"darkRegionMinOverdueJobs,omitempty"`
DarkRegionMinOverdueMinutes int `json:"darkRegionMinOverdueMinutes,omitempty"`
DarkRegionCriticalJobs int `json:"darkRegionCriticalJobs,omitempty"`
DarkRegionCriticalMinutes int `json:"darkRegionCriticalMinutes,omitempty"`
FleetDropPercent float64 `json:"fleetDropPercent,omitempty"`
FleetMinBaseline int `json:"fleetMinBaseline,omitempty"`
FleetCriticalDropPercent float64 `json:"fleetCriticalDropPercent,omitempty"`
StaleIncidentMinMinutes int `json:"staleIncidentMinMinutes,omitempty"`
StaleIncidentPeriodMultiplier int `json:"staleIncidentPeriodMultiplier,omitempty"`
StaleIncidentCriticalCount int `json:"staleIncidentCriticalCount,omitempty"`
StaleIncidentScanLimit int `json:"staleIncidentScanLimit,omitempty"`
StaleChecksCriticalCount int `json:"staleChecksCriticalCount,omitempty"`
}
Config is the decoded `platform_watchdog` system parameter.
Every threshold lives here rather than in koanf config for one reason: the people who tune a watchdog are the people running the instance, and they can already edit system parameters live. A threshold that needs a redeploy is a threshold nobody adjusts.
func DefaultConfig ¶
func DefaultConfig() *Config
DefaultConfig is the watchdog with everything at its documented default and the feature OFF — an instance that never writes the parameter must not suddenly start mailing whoever happens to be user #1.
func LoadConfig ¶
LoadConfig reads and decodes the `platform_watchdog` system parameter. A missing parameter yields the disabled default, which is the correct behavior for every instance that has never opted in.
func (*Config) DarkRegionCriticalAge ¶
DarkRegionCriticalAge is the age at which a dark region becomes critical.
func (*Config) DarkRegionMinOverdueAge ¶
DarkRegionMinOverdueAge is how late the oldest overdue job must be.
func (*Config) RenotifyAfter ¶
RenotifyAfter is how long an ONGOING anomaly stays silent.
func (*Config) StaleIncidentMinAge ¶
StaleIncidentMinAge is the floor of max(N × period, floor).
type Digest ¶
Digest is ONE message per run, never one per anomaly. A watchdog that sends a message per anomaly is a watchdog that gets muted the first time a region strands 400 jobs.
func BuildDigest ¶
func BuildDigest(transitions []Transition, report *Report, generatedAt time.Time) Digest
BuildDigest renders the notifiable transitions of one run.
Returns an empty digest when nothing transitioned — a run where every anomaly is a known, still-ongoing one inside its quiet window sends nothing at all, which is the entire point of the transition model.
type RegionHealthReporter ¶
type RegionHealthReporter interface {
RegionHealth(ctx context.Context) (*checks.RegionHealthReport, error)
}
RegionHealthReporter is the spec-09 ghost-detection function, injected as an interface for one reason only: tests need to make it fail on demand (to prove detector independence) without a broken database. Production always passes the real *checks.Service, so "dark" has exactly one definition — checks.Service.RegionHealth — and this package never re-derives it.
type Report ¶
type Report struct {
GeneratedAt time.Time
Anomalies []Anomaly
// Failed maps a detector name to the error it returned. A detector that
// errored is explicitly NOT reported as "clean": its fingerprints are
// excluded from resolution reconciliation, so a transient query failure
// can never be laundered into a false recovery notice.
Failed map[string]error
// StrandedJobs is the total overdue-job count across every dark region,
// exported as its own gauge because it is the number that told the story
// on 2026-08-24.
StrandedJobs int
// StaleIncidents is the count of frozen active incidents.
StaleIncidents int
// StaleChecks is the count of stale checks whose region is not dark.
StaleChecks int
}
Report is the outcome of one watchdog evaluation.
func (*Report) DetectorSucceeded ¶
DetectorSucceeded reports whether a given detector completed without error.
func (*Report) Filtered ¶
Filtered returns a copy of the report keeping only the anomalies at or above minSeverity, with the detector-failure map (and therefore the resolution-reconciliation guard) carried over untouched.
The anti-flood ledger tracks the FILTERED set on purpose: a marker written for an anomaly the digest never mentions would record a notification that never happened, and would then suppress the real one when the anomaly escalates past the bar.
func (*Report) HasFailures ¶
HasFailures reports whether any detector errored this run.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service evaluates the platform's own vitals.
func NewService ¶
func NewService(dbService db.Service, regionHealth RegionHealthReporter) *Service
NewService builds a watchdog service. regionHealth may be nil, in which case the dark-region detector reports a failure rather than silently skipping.
func (*Service) Evaluate ¶
Evaluate runs every detector and collects their anomalies.
Detector independence is the contract: each runs in its own call, and a detector that returns an error contributes to Report.Failed while the others still produce their anomalies. A watchdog that goes silent because one query broke would reproduce the exact failure mode it exists to catch.
func (*Service) Reconcile ¶
Reconcile diffs this run's anomalies against the persisted fingerprints, writes the updated markers, and returns what moved.
The load-bearing rule: a fingerprint is only considered RESOLVED when the detector that would have produced it actually ran successfully. A query error must never be laundered into a false "recovered" notice — that would tell an operator the outage is over while it is still going.
type Severity ¶
type Severity int
Severity orders the three levels an anomaly can carry. Ordered so minSeverity filtering is a plain comparison.
const ( // SeverityInfo is informational — never pages, always logged. SeverityInfo Severity = iota // SeverityWarning is the default bar: something is wrong but bounded. SeverityWarning // SeverityCritical is blast-radius territory: hundreds of stranded jobs, // a halved fleet, a wall of frozen incidents. SeverityCritical )
Severity levels, ascending.
func ParseSeverity ¶
ParseSeverity maps a configured token to a Severity. Anything unrecognized (including the empty string) falls back to warning — the documented default bar, never "off".
type Transition ¶
type Transition struct {
Fingerprint string
Kind TransitionKind
// Anomaly is the current anomaly. Zero-valued for TransitionResolved,
// which by definition has no current anomaly — read Fingerprint and
// FirstSeenAt instead.
Anomaly Anomaly
FirstSeenAt time.Time
// Notify reports whether this transition belongs in the digest.
Notify bool
}
Transition is one anomaly's movement across this run.
func NotifiableTransitions ¶
func NotifiableTransitions(transitions []Transition) []Transition
NotifiableTransitions filters the transitions that belong in a digest.
type TransitionKind ¶
type TransitionKind string
TransitionKind is what happened to one anomaly between the previous run and this one. The watchdog reports TRANSITIONS, never state: an anomaly that persists must not re-page every hour, and a resolution must be announced because "watchdog sees 0 stranded jobs" is the operator's exit criterion.
const ( // TransitionNew is a fingerprint seen for the first time — always notifies. TransitionNew TransitionKind = "new" // TransitionOngoing is a known fingerprint inside its re-notify window — // recorded, never notified. TransitionOngoing TransitionKind = "ongoing" // TransitionRenotify is a known fingerprint past the re-notify window (or // whose severity has escalated) — notifies as "still broken since …". TransitionRenotify TransitionKind = "renotify" // TransitionResolved is a fingerprint that was present and is now gone — // notifies exactly once, then the marker is cleared. TransitionResolved TransitionKind = "resolved" )
Transition kinds.