run

package
v0.46.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package run is the domain model of a run and its steps.

The rule is explicit: "do not use plain booleans such as running = true". States are a type, and transitions are validated — a `Run` cannot go from SUCCESS to RUNNING through carelessness or through a race.

Index

Constants

View Source
const Unmapped = -1

Unmapped is the map_index of a step that is not mapped, which is most of them. It is -1 and not NULL because two NULLs are never equal in SQL, and the unique on (run, node, attempt, map_index) would silently stop protecting unmapped steps.

Variables

This section is empty.

Functions

func Validate

func Validate(from, to Status) error

Validate returns an error when a RUN's transition does not exist in the graph.

func ValidateStep

func ValidateStep(from, to Status) error

ValidateStep is the same for a STEP. A step that has not started is StatusPending, which is where a skipped one comes from.

Types

type AutoParams

type AutoParams struct {
	// ScheduledAt is the slot this run represents: what the cron said. Absent
	// on a manual run, which belongs to no slot.
	ScheduledAt *time.Time `json:"scheduled_at,omitempty"`

	// StartedAt is when this ATTEMPT began.
	//
	// Named for the column it comes from (`started_at`) and for what every
	// other orchestrator calls it, rather than `running_at`: a run is not
	// running for the whole time this timestamp describes -- it is the instant
	// it started, and a retry moves it.
	StartedAt *time.Time `json:"started_at,omitempty"`

	// AdjustedAt is the clock to read instead of now(). See the type's comment.
	AdjustedAt time.Time `json:"adjusted_at"`

	// DelaySeconds is how late this attempt was: StartedAt minus ScheduledAt.
	// Zero on a manual run and never negative.
	//
	// It is here so nobody subtracts two timestamps to find out, and because a
	// pipeline that decides to widen its window when it is very late needs one
	// number rather than two.
	DelaySeconds int `json:"delay_seconds"`

	// IntervalStart and IntervalEnd are the window this run covers: from the
	// PREVIOUS slot up to this one, end excluded.
	//
	// A pipeline that asks the vendor for [start, end) never overlaps and never
	// gaps, however late it runs and however often it retries. That is the
	// whole point, and it is why they come from the CRON and not from the
	// history: a backfill of a slot from March has to produce the window March
	// had, not the window this workflow's runs happen to describe today.
	//
	// Absent when the workflow has no schedule.
	IntervalStart *time.Time `json:"interval_start,omitempty"`
	IntervalEnd   *time.Time `json:"interval_end,omitempty"`

	// PreviousError says the previous run of this workflow did not succeed.
	PreviousError bool `json:"previous_error"`

	// PreviousSuccessAt is the slot of the last run that DID succeed, which is
	// what makes PreviousError actionable: a boolean says something is wrong, a
	// timestamp says how far back to catch up.
	PreviousSuccessAt *time.Time `json:"previous_success_at,omitempty"`

	// Landed is what arrived to start this run, for a run a landing started.
	//
	// Absent on every other kind, and `omitempty` rather than an empty list:
	// the JSON is read by Python and by jq, and `null` versus `[]` versus
	// absent is three spellings of one thing. Absent is the one that matches
	// how the optional timestamps above behave.
	Landed []string `json:"landed,omitempty"`

	// Date is AdjustedAt as YYYY-MM-DD, in UTC. It is here because it is the
	// single most-used value in the whole of data engineering -- a partition, a
	// folder, a WHERE clause -- and because formatting it in eight pipelines is
	// eight chances to pick a different timezone.
	Date string `json:"date"`
}

AutoParams are what the ENGINE knows about a run and every pipeline would otherwise work out for itself.

The one that matters is AdjustedAt, and the bug it removes is common enough to be worth naming. A fetcher reads `time.Now()`, subtracts its window and asks the vendor for the last two hours. On a run that starts on time that is right. On a run the queue delayed by forty minutes it is forty minutes wrong -- and the forty minutes it skipped belong to no run at all, because the next slot reads its own `now()` too. The data is simply missing, nothing failed, and it is found weeks later.

So the engine hands over a clock instead: AdjustedAt is what a pipeline should read INSTEAD of now(). For a scheduled run it is the slot, so it does not move when the run is late and does not move when the run is retried three hours later. For a manual run there is no slot, so it is when the run started -- which is the only honest answer.

They are a SNAPSHOT, computed when the run starts and stored on it. Not derived on read: PreviousError is a fact about the moment this run began, and recomputing it tomorrow -- after the previous run was retried and passed -- would answer a different question with the same name.

func Auto

func Auto(r Run, started time.Time, prev Previous, window Interval) AutoParams

Auto computes a run's automatic params.

Pure: everything it needs is passed in, so the rule can be read and tested without a database and without a clock.

func (AutoParams) Env

func (a AutoParams) Env() map[string]string

Env is the auto params as environment variables, one each.

One variable per value, and not only the JSON blob: a `bash` step with no library reads `$BREVIS_AUTO_ADJUSTED_AT`, and requiring `jq` to answer "what time is it for this run" would put a dependency in the way of the simplest possible step.

func (AutoParams) String

func (a AutoParams) String() string

String is what the screen shows for one value, so the UI and a log line agree.

type ErrInvalidTransition

type ErrInvalidTransition struct {
	From, To Status
}

ErrInvalidTransition carries both states so the error says what happened, and not merely that something was refused.

func (ErrInvalidTransition) Error

func (e ErrInvalidTransition) Error() string

type Interval

type Interval struct{ Start, End time.Time }

Interval is the window a schedule gives this slot. Zero means the workflow has no schedule.

type Previous

type Previous struct {
	Failed    bool
	SuccessAt *time.Time
}

Previous is what the history says about the run before this one.

type Run

type Run struct {
	ID             uuid.UUID
	WorkflowSlug   string
	IdempotencyKey string
	Status         Status
	Attempt        int
	Definition     []byte

	// TriggerType says why the Run exists (section 12). Without it there is no
	// telling a backfill from a scheduled run while investigating an incident.
	TriggerType string

	// Params are the values used in THIS run. Stored alongside the run because
	// "what parameters did this run with?" is the first question of any backfill
	// investigation.
	Params map[string]string

	// TriggerTargets are the landings that started this run, for a run whose
	// TriggerType is `landed`. Empty for every other kind.
	//
	// THE TARGETS AND NOT THE LANDINGS. A target is an identity and it is what
	// a step acts on -- "rebuild from bronze.orders". The row counts and the
	// instants belong to the catalog, which already holds them and which a
	// step reads on /data; carrying them here would put three values in an
	// environment variable for a question nobody asked at this point.
	//
	// A SNAPSHOT, like MaxActive below: what the window held when the run was
	// created. A landing that arrives afterwards belongs to the next window
	// and starts the next run.
	TriggerTargets []string

	// MaxActive is the workflow's concurrency limit at the instant of the
	// trigger. A snapshot: lowering the limit later does not change runs that
	// are already queued.
	MaxActive int

	// Group is the workflow's concurrency group at the instant of the trigger,
	// a snapshot like MaxActive. Empty: the run contends only with runs of its
	// own workflow.
	Group string

	// LogicalDate is the slot this Run represents. Nil for a manual trigger,
	// which belongs to no slot at all.
	LogicalDate *time.Time
	CreatedAt   time.Time
	StartedAt   *time.Time
	FinishedAt  *time.Time
	Err         string

	// Auto is what the engine worked out for this run so no pipeline has to:
	// the clock to read instead of now(), the window it covers, how late it
	// was, and whether the run before it failed. See AutoParams.
	Auto AutoParams

	// CancelRequestedAt is when someone first asked to cancel the run; nil
	// when nobody did. A run asked while a dispatcher held it is still going
	// until the dispatcher reads the request and stops its steps.
	CancelRequestedAt *time.Time
	// CanceledBy is who asked first: a console user, `cli`, or empty for a
	// console without login.
	CanceledBy string
}

Run is one execution of a workflow.

`Definition` holds the snapshot of the graph at the moment the Run came into being. The rule requires it: editing the workflow later must not change the meaning of a past run.

func (*Run) Transition

func (r *Run) Transition(to Status, now time.Time) error

Transition moves the Run, validating. It stamps the times here, and not in the caller, so that no path exists that changes the state without recording when.

type SkipCause added in v0.46.0

type SkipCause string

SkipCause says why a step was skipped. Both are a skipped step, and they are not the same event for whoever watches the data: a step that stood down because there was nothing to do kept its schedule, and one cut off because something it needs failed did not.

const (
	// SkippedByDesign: nothing for the step to do. An empty `unless_empty`
	// value or `for_each` list, a `when: any_failed` that saw no failure, or an
	// upstream that was itself skipped by design.
	SkippedByDesign SkipCause = "design"

	// SkippedUpstream: a step it depends on failed, was canceled, or was itself
	// skipped upstream.
	SkippedUpstream SkipCause = "upstream"
)

type Status

type Status string

Status is a run's state.

const (
	StatusCreated  Status = "created"
	StatusQueued   Status = "queued"
	StatusRunning  Status = "running"
	StatusSuccess  Status = "success"
	StatusFailed   Status = "failed"
	StatusRetrying Status = "retrying"
	StatusCanceled Status = "canceled"

	// StatusPending and StatusSkipped are STEP states. A run is never either
	// one, and the two maps below are what enforce that.
	//
	// StatusPending is a step that has not started. It has always existed as
	// the string "pending" in the graph payload and in the screen's palette,
	// with no constant behind it -- which is how the UI ended up owning a state
	// the domain did not know about.
	StatusPending Status = "pending"

	// StatusSkipped is a step that did not run because its trigger rule was not
	// satisfied.
	//
	// It is a state of its own and not a flavour of success or failure, and
	// that distinction is the reason this state exists at all. A step whose
	// upstream failed did not fail -- it was never given the chance -- and
	// calling it success is a lie that reaches the run's own status and the
	// screen. It is not `canceled` either: nobody stopped it. And it is not
	// `pending`, which means "has not run YET" and is the state it would
	// otherwise be left in forever.
	StatusSkipped Status = "skipped"
)

func RunStatuses

func RunStatuses() []Status

RunStatuses is every state a RUN can be in.

It exists so code outside this package can enumerate them -- the retention query builds its status list from here rather than typing three strings into SQL, which is the shape that drifts the day a fourth one arrives.

func Settled

func Settled() []Status

Settled is every run state nothing will touch again: the complement of InFlight, and what retention is allowed to delete.

func (Status) CanGo

func (s Status) CanGo(to Status) bool

CanGo says whether the transition is allowed for a RUN.

func (Status) CanStepGo

func (s Status) CanStepGo(to Status) bool

CanStepGo says whether the transition is allowed for a STEP.

func (Status) InFlight

func (s Status) InFlight() bool

InFlight says the system may still act on this run: a dispatcher may claim it, a runner may be inside it, a retry may pick it up.

It is NOT the complement of Terminal, and the gap between them is the reason this exists. A run that FAILED is not terminal -- the state machine lets it become `retrying`, which is what makes a retry possible at all -- but nothing is going to retry a run from last March on its own. Retention needs "is anybody still holding this", and asking Terminal would keep every failure ever recorded, forever, which is the opposite of what a failure deserves.

Enumerated rather than derived, because the question is about OWNERSHIP and not about the shape of the graph. TestEveryRunStatusIsClassified is what keeps that honest: a new status has to be put on one side or the other before it compiles green.

func (Status) Terminal

func (s Status) Terminal() bool

Terminal says whether the state ends a RUN's life.

FAILED is not terminal for a run: it can go to RETRYING. What decides whether there is an attempt left is the retry policy, not the state machine.

Derived from the map rather than listed again, so the two cannot disagree.

func (Status) TerminalStep

func (s Status) TerminalStep() bool

TerminalStep says whether a STEP is finished, and it differs from Terminal in exactly one place: a step's FAILED is final.

A step's retries are a loop inside the runner with a row per attempt, so the row itself never goes anywhere. A run's failure can become a retry, and that is why the two questions have different answers -- which was found by a trigger rule reading `all_done` and deciding a failed step had not finished.

type StepKey

type StepKey struct {
	Node     string
	MapIndex int
}

StepKey identifies one INSTANCE of a step within a run.

A mapped step runs once per element of a list, and each of those runs has its own row, its own exit code and its own retry. Keying by node alone was enough until `for_each:` existed; it is not any more, and the two places that care are the resume (which of them already succeeded) and the screen (how many there are).

func Step

func Step(node string) StepKey

Step is the key of an unmapped step.

type TaskRun

type TaskRun struct {
	ID         uuid.UUID
	RunID      uuid.UUID
	NodeID     string
	Status     Status
	Attempt    int
	ExitCode   *int
	StartedAt  *time.Time
	FinishedAt *time.Time
	Err        string
}

TaskRun is the execution of one node inside a Run.

func (*TaskRun) Transition

func (t *TaskRun) Transition(to Status, now time.Time) error

Transition moves the TaskRun.

Against the STEP's graph, not the run's. The two shared one until `skipped` arrived -- a state a step has and a run does not -- and a single map could no longer say the truth about both.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址