watchdog

package
v0.0.0-...-df0d7b6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package watchdog is the platform's self-monitoring detector set (spec 2026-08-24-10).

The worst failure mode of a monitoring product is going blind, and that is exactly the failure that produces zero signal: a check that stops being executed keeps its last state forever, so the dashboard shows green rows with quietly aging timestamps. Check-level alerting (incidents → escalation policies) cannot report on itself. This package is the channel that can.

It holds only the parts that are testable without a job runner: the detectors, the anti-flood transition state machine, and the digest renderer. Scheduling and delivery live in the `platform_watchdog` job (internal/jobs/jobtypes/job_platform_watchdog.go).

Index

Constants

View Source
const (
	// DefaultInterval is how often the job wakes up.
	DefaultInterval = time.Hour
	// DefaultRenotifyAfter bounds re-notification of an ONGOING anomaly.
	DefaultRenotifyAfter = 24 * time.Hour

	// DefaultDarkRegionMinOverdueJobs is the blast-radius floor: 419 stranded
	// jobs is page-worthy, one job 90 seconds late is not.
	DefaultDarkRegionMinOverdueJobs = 5
	// DefaultDarkRegionMinOverdueAge is how late the oldest overdue job must
	// be before the region counts as an anomaly.
	DefaultDarkRegionMinOverdueAge = 10 * time.Minute
	// DefaultDarkRegionCriticalJobs escalates a dark region to critical.
	DefaultDarkRegionCriticalJobs = 50
	// DefaultDarkRegionCriticalAge escalates a dark region to critical on
	// duration alone — a handful of jobs stranded for half an hour is still
	// a region nobody is serving.
	DefaultDarkRegionCriticalAge = 30 * time.Minute

	// DefaultFleetDropPercent is the drop, against the same hour a day
	// earlier, that counts as a collapse.
	DefaultFleetDropPercent = 50.0
	// DefaultFleetMinBaseline keeps a quiet instance (or a brand-new one)
	// from reporting a collapse: without a floor, 4 results vs. 10 is a 60%
	// "collapse".
	DefaultFleetMinBaseline = 100
	// DefaultFleetCriticalDropPercent escalates a collapse to critical.
	DefaultFleetCriticalDropPercent = 80.0

	// DefaultStaleIncidentMinAge is the floor of max(N × period, this).
	DefaultStaleIncidentMinAge = 15 * time.Minute
	// DefaultStaleIncidentPeriodMultiplier is the N of max(N × period, floor).
	DefaultStaleIncidentPeriodMultiplier = 3
	// DefaultStaleIncidentCriticalCount escalates the stale-incident anomaly
	// to critical — 61 frozen incidents was the 2026-08-24 number.
	DefaultStaleIncidentCriticalCount = 10
	// DefaultStaleChecksCriticalCount escalates the stale-checks anomaly
	// (stale checks in regions that are NOT dark) to critical.
	DefaultStaleChecksCriticalCount = 10
	// DefaultStaleIncidentScanLimit bounds the active-incident scan so the
	// run stays cheap on an instance with a genuinely huge outage.
	DefaultStaleIncidentScanLimit = 2000
)

Watchdog defaults. Every one of them is overridable through the system parameter; these are the values an operator gets by writing nothing but `{"enabled": true, "recipients": ["<uid>"]}`.

View Source
const (
	// DetectorDarkRegion reports regions whose assigned work nothing live can
	// claim (the 2026-08-24 stranding).
	DetectorDarkRegion = "dark-region"
	// DetectorFleetCollapse reports an instance-wide drop in produced results
	// against the same hour a day earlier — the catch-all for stranding
	// causes this spec did not imagine.
	DetectorFleetCollapse = "fleet-collapse"
	// DetectorStaleIncidents reports active incidents whose check has stopped
	// producing results (the "frozen incident" symptom).
	DetectorStaleIncidents = "stale-incidents"
	// DetectorStaleChecks reports checks in the `stale` ("No data") status
	// whose regions are NOT dark (spec 2026-09-25-02): a platform bug —
	// scheduler, leases, rate limiting — rather than a region outage.
	DetectorStaleChecks = "stale-checks"
)

Detector names. They are the stable half of an anomaly fingerprint and of the per-detector metric label, so they must never be reworded casually — a rename resets every operator's anti-flood state to "new".

View Source
const (
	// SeverityTokenInfo is the "info" token.
	SeverityTokenInfo = "info"
	// SeverityTokenWarning is the "warning" token, and the default bar.
	SeverityTokenWarning = "warning"
	// SeverityTokenCritical is the "critical" token.
	SeverityTokenCritical = "critical"
)

Severity tokens, as they appear in the system parameter and the digest.

View Source
const ParamPlatformWatchdog = "platform_watchdog"

ParamPlatformWatchdog is the system-parameter key the watchdog is configured through. It rides the existing super-admin /api/v1/system/parameters CRUD, so there is no new settings surface and no redeploy needed to add an operator.

View Source
const StateKeyPrefix = "watchdog:anomaly:"

StateKeyPrefix namespaces the per-anomaly anti-flood markers. They are GLOBAL state entries (organization_uid IS NULL) because the watchdog reports on the platform, not on a tenant.

View Source
const SubjectFleet = "instance"

SubjectFleet is the stable subject of the instance-wide execution-rate anomaly. Instance-wide detectors need a constant subject: a fingerprint that churned between runs would re-notify forever and never resolve.

View Source
const SubjectStaleChecks = "unexplained"

SubjectStaleChecks is the stable subject of the stale-checks anomaly: one anomaly per run carrying a count, never one per check.

View Source
const SubjectStaleIncidents = "active"

SubjectStaleIncidents is the stable subject of the frozen-incident anomaly. One anomaly per run carrying a count, not one per incident: 61 frozen incidents must produce one line in one digest, never 61 pages.

Variables

View Source
var (
	// ErrInvalidParameterShape is returned when the value is not a JSON object.
	ErrInvalidParameterShape = errors.New("platform_watchdog must be a JSON object")
	// ErrInvalidRecipient is returned for a blank entry in recipients.
	ErrInvalidRecipient = errors.New("platform_watchdog recipients must be non-empty user UIDs")
	// ErrInvalidMinSeverity is returned for an unknown minSeverity token.
	ErrInvalidMinSeverity = errors.New(`platform_watchdog minSeverity must be one of "info", "warning", "critical"`)
)

Parameter validation errors.

View Source
var ErrRegionHealthUnavailable = errors.New("watchdog: region health reporter not wired")

ErrRegionHealthUnavailable is reported by the dark-region detector when no RegionHealth reporter was wired. It is deliberately an ERROR rather than a silent skip: a watchdog that quietly stops looking at the one failure mode it was written for is worse than no watchdog.

Functions

func ClaimNotification

func ClaimNotification(
	ctx context.Context, dbService db.Service, fingerprint string,
	severity Severity, headline string, now time.Time, escalation bool,
) (bool, error)

ClaimNotification records, in the shared anti-flood ledger, that a notification for fingerprint is being delivered NOW at severity, and reports whether the caller should deliver it.

It is the entry point the per-minute region sweep uses, so its transitions and the hourly digest read the same marker:

  • no marker: nobody told the operator yet — write it (exactly the shape Reconcile writes for a notified anomaly) and deliver;
  • a marker, and escalation false: the operator already heard about this outage (from the digest, which grades severity on its own scale) — stay quiet;
  • a marker, escalation true (the caller's own stalled → dark), notified at a LOWER severity: that is news, so update the notified severity and deliver.

Only call it when there is someone to deliver to: a marker written for a notification that never went out would suppress the real one later.

func ClearNotification

func ClearNotification(ctx context.Context, dbService db.Service, fingerprint string) (bool, error)

ClearNotification removes a fingerprint from the shared ledger and reports whether it was there — i.e. whether the operator was told about the condition and so must now be told it is over.

func DarkRegionFingerprint

func DarkRegionFingerprint(region string) string

DarkRegionFingerprint is the anti-flood fingerprint of a CLOUD region's dark-region anomaly — the marker the hourly detector and the per-minute region sweep share, so one outage is announced to the operator once, whichever of the two sees it first (spec 2026-09-25-03).

func PublishMetrics

func PublishMetrics(report *Report)

PublishMetrics exports the run's findings on the Prometheus registry.

It publishes EVERY anomaly, not just the ones past minSeverity: the out-of-band path exists so an external Prometheus can alert on its own thresholds, independently of what the in-band digest was configured to deliver.

func ValidateParameter

func ValidateParameter(value any) error

ValidateParameter rejects a platform_watchdog value the watchdog could not decode at run time.

This runs on the write, not on the read, on purpose. A malformed value only surfaces at the next hourly run — as a failing job on an alerting path that nobody is watching, which is precisely the failure mode this spec exists to remove. Catching it in the PUT hands the mistake back to the person making it, while they are still looking.

Types

type Anomaly

type Anomaly struct {
	// Detector is one of the Detector* constants.
	Detector string
	// Subject identifies WHAT is broken within the detector's domain — a
	// region slug, or the sentinel "fleet"/"global" for instance-wide
	// detectors. Together with Detector it forms the fingerprint, so it must
	// be stable across runs for the same underlying condition: an anomaly
	// whose subject churns would re-notify forever.
	Subject  string
	Severity Severity
	// Headline is the one-line summary carrying the numbers that matter.
	Headline string
	// Detail is optional extra context (the three oldest frozen incidents,
	// the last-worker-seen timestamp, …).
	Detail string
	// Remediation is the ready-to-run fix, when there is one.
	Remediation string
	// Count is the detector's headline magnitude — overdue jobs for a dark
	// region, frozen incidents for stale-incidents, lost results for a fleet
	// collapse. Kept as a first-class field rather than re-parsed out of the
	// rendered headline so the gauges and the digest read the same number.
	Count int
}

Anomaly is one thing the platform found wrong with itself.

func Filter

func Filter(anomalies []Anomaly, minSeverity Severity) []Anomaly

Filter returns the anomalies at or above minSeverity. Filtering applies to DELIVERY only — every anomaly is logged and metered regardless, because the out-of-band Prometheus path must see everything.

func (*Anomaly) Fingerprint

func (a *Anomaly) Fingerprint() string

Fingerprint is the anti-flood identity of an anomaly: `<detector>:<subject>`.

type Config

type Config struct {
	// Enabled gates the whole job. False means the run has NO side effects at
	// all: no queries, no state writes, no metrics, no delivery.
	Enabled bool `json:"enabled"`
	// Recipients are user UIDs. Delivery goes through each user's own enabled
	// notification routes — no new medium, no hardcoded webhook.
	Recipients []string `json:"recipients"`
	// MinSeverity filters DELIVERY only.
	MinSeverity string `json:"minSeverity"`

	IntervalMinutes      int `json:"intervalMinutes,omitempty"`
	RenotifyAfterMinutes int `json:"renotifyAfterMinutes,omitempty"`

	DarkRegionMinOverdueJobs    int `json:"darkRegionMinOverdueJobs,omitempty"`
	DarkRegionMinOverdueMinutes int `json:"darkRegionMinOverdueMinutes,omitempty"`
	DarkRegionCriticalJobs      int `json:"darkRegionCriticalJobs,omitempty"`
	DarkRegionCriticalMinutes   int `json:"darkRegionCriticalMinutes,omitempty"`

	FleetDropPercent         float64 `json:"fleetDropPercent,omitempty"`
	FleetMinBaseline         int     `json:"fleetMinBaseline,omitempty"`
	FleetCriticalDropPercent float64 `json:"fleetCriticalDropPercent,omitempty"`

	StaleIncidentMinMinutes       int `json:"staleIncidentMinMinutes,omitempty"`
	StaleIncidentPeriodMultiplier int `json:"staleIncidentPeriodMultiplier,omitempty"`
	StaleIncidentCriticalCount    int `json:"staleIncidentCriticalCount,omitempty"`
	StaleIncidentScanLimit        int `json:"staleIncidentScanLimit,omitempty"`

	StaleChecksCriticalCount int `json:"staleChecksCriticalCount,omitempty"`
}

Config is the decoded `platform_watchdog` system parameter.

Every threshold lives here rather than in koanf config for one reason: the people who tune a watchdog are the people running the instance, and they can already edit system parameters live. A threshold that needs a redeploy is a threshold nobody adjusts.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig is the watchdog with everything at its documented default and the feature OFF — an instance that never writes the parameter must not suddenly start mailing whoever happens to be user #1.

func LoadConfig

func LoadConfig(ctx context.Context, dbSvc db.Service) (*Config, error)

LoadConfig reads and decodes the `platform_watchdog` system parameter. A missing parameter yields the disabled default, which is the correct behavior for every instance that has never opted in.

func (*Config) DarkRegionCriticalAge

func (c *Config) DarkRegionCriticalAge() time.Duration

DarkRegionCriticalAge is the age at which a dark region becomes critical.

func (*Config) DarkRegionMinOverdueAge

func (c *Config) DarkRegionMinOverdueAge() time.Duration

DarkRegionMinOverdueAge is how late the oldest overdue job must be.

func (*Config) Interval

func (c *Config) Interval() time.Duration

Interval is the self-reschedule delay.

func (*Config) RenotifyAfter

func (c *Config) RenotifyAfter() time.Duration

RenotifyAfter is how long an ONGOING anomaly stays silent.

func (*Config) Severity

func (c *Config) Severity() Severity

Severity is the parsed delivery bar.

func (*Config) StaleIncidentMinAge

func (c *Config) StaleIncidentMinAge() time.Duration

StaleIncidentMinAge is the floor of max(N × period, floor).

type Digest

type Digest struct {
	Subject string
	Text    string
}

Digest is ONE message per run, never one per anomaly. A watchdog that sends a message per anomaly is a watchdog that gets muted the first time a region strands 400 jobs.

func BuildDigest

func BuildDigest(transitions []Transition, report *Report, generatedAt time.Time) Digest

BuildDigest renders the notifiable transitions of one run.

Returns an empty digest when nothing transitioned — a run where every anomaly is a known, still-ongoing one inside its quiet window sends nothing at all, which is the entire point of the transition model.

func (Digest) Empty

func (d Digest) Empty() bool

Empty reports whether there is nothing worth sending.

type RegionHealthReporter

type RegionHealthReporter interface {
	RegionHealth(ctx context.Context) (*checks.RegionHealthReport, error)
}

RegionHealthReporter is the spec-09 ghost-detection function, injected as an interface for one reason only: tests need to make it fail on demand (to prove detector independence) without a broken database. Production always passes the real *checks.Service, so "dark" has exactly one definition — checks.Service.RegionHealth — and this package never re-derives it.

type Report

type Report struct {
	GeneratedAt time.Time
	Anomalies   []Anomaly
	// Failed maps a detector name to the error it returned. A detector that
	// errored is explicitly NOT reported as "clean": its fingerprints are
	// excluded from resolution reconciliation, so a transient query failure
	// can never be laundered into a false recovery notice.
	Failed map[string]error
	// StrandedJobs is the total overdue-job count across every dark region,
	// exported as its own gauge because it is the number that told the story
	// on 2026-08-24.
	StrandedJobs int
	// StaleIncidents is the count of frozen active incidents.
	StaleIncidents int
	// StaleChecks is the count of stale checks whose region is not dark.
	StaleChecks int
}

Report is the outcome of one watchdog evaluation.

func (*Report) DetectorSucceeded

func (r *Report) DetectorSucceeded(detector string) bool

DetectorSucceeded reports whether a given detector completed without error.

func (*Report) Filtered

func (r *Report) Filtered(minSeverity Severity) *Report

Filtered returns a copy of the report keeping only the anomalies at or above minSeverity, with the detector-failure map (and therefore the resolution-reconciliation guard) carried over untouched.

The anti-flood ledger tracks the FILTERED set on purpose: a marker written for an anomaly the digest never mentions would record a notification that never happened, and would then suppress the real one when the anomaly escalates past the bar.

func (*Report) HasFailures

func (r *Report) HasFailures() bool

HasFailures reports whether any detector errored this run.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service evaluates the platform's own vitals.

func NewService

func NewService(dbService db.Service, regionHealth RegionHealthReporter) *Service

NewService builds a watchdog service. regionHealth may be nil, in which case the dark-region detector reports a failure rather than silently skipping.

func (*Service) Evaluate

func (s *Service) Evaluate(ctx context.Context, cfg *Config) *Report

Evaluate runs every detector and collects their anomalies.

Detector independence is the contract: each runs in its own call, and a detector that returns an error contributes to Report.Failed while the others still produce their anomalies. A watchdog that goes silent because one query broke would reproduce the exact failure mode it exists to catch.

func (*Service) Now

func (s *Service) Now() time.Time

Now returns the service's current time.

func (*Service) Reconcile

func (s *Service) Reconcile(ctx context.Context, report *Report, cfg *Config) ([]Transition, error)

Reconcile diffs this run's anomalies against the persisted fingerprints, writes the updated markers, and returns what moved.

The load-bearing rule: a fingerprint is only considered RESOLVED when the detector that would have produced it actually ran successfully. A query error must never be laundered into a false "recovered" notice — that would tell an operator the outage is over while it is still going.

func (*Service) SetNow

func (s *Service) SetNow(now func() time.Time)

SetNow overrides the service clock. Exported because the watchdog's windows ("the last completed hour", "the same hour yesterday", "re-notify after 24h") are only testable against a pinned clock — the same reason checks.Service carries one for RegionHealth.

type Severity

type Severity int

Severity orders the three levels an anomaly can carry. Ordered so minSeverity filtering is a plain comparison.

const (
	// SeverityInfo is informational — never pages, always logged.
	SeverityInfo Severity = iota
	// SeverityWarning is the default bar: something is wrong but bounded.
	SeverityWarning
	// SeverityCritical is blast-radius territory: hundreds of stranded jobs,
	// a halved fleet, a wall of frozen incidents.
	SeverityCritical
)

Severity levels, ascending.

func ParseSeverity

func ParseSeverity(value string) Severity

ParseSeverity maps a configured token to a Severity. Anything unrecognized (including the empty string) falls back to warning — the documented default bar, never "off".

func (Severity) String

func (s Severity) String() string

String renders the severity as the lowercase token used in the system parameter and in the digest.

type Transition

type Transition struct {
	Fingerprint string
	Kind        TransitionKind
	// Anomaly is the current anomaly. Zero-valued for TransitionResolved,
	// which by definition has no current anomaly — read Fingerprint and
	// FirstSeenAt instead.
	Anomaly     Anomaly
	FirstSeenAt time.Time
	// Notify reports whether this transition belongs in the digest.
	Notify bool
}

Transition is one anomaly's movement across this run.

func NotifiableTransitions

func NotifiableTransitions(transitions []Transition) []Transition

NotifiableTransitions filters the transitions that belong in a digest.

type TransitionKind

type TransitionKind string

TransitionKind is what happened to one anomaly between the previous run and this one. The watchdog reports TRANSITIONS, never state: an anomaly that persists must not re-page every hour, and a resolution must be announced because "watchdog sees 0 stranded jobs" is the operator's exit criterion.

const (
	// TransitionNew is a fingerprint seen for the first time — always notifies.
	TransitionNew TransitionKind = "new"
	// TransitionOngoing is a known fingerprint inside its re-notify window —
	// recorded, never notified.
	TransitionOngoing TransitionKind = "ongoing"
	// TransitionRenotify is a known fingerprint past the re-notify window (or
	// whose severity has escalated) — notifies as "still broken since …".
	TransitionRenotify TransitionKind = "renotify"
	// TransitionResolved is a fingerprint that was present and is now gone —
	// notifies exactly once, then the marker is cleared.
	TransitionResolved TransitionKind = "resolved"
)

Transition kinds.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址