crawltask

package
v0.0.0-...-df0d7b6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: AGPL-3.0 Imports: 23 Imported by: 0

Documentation

Overview

Package crawltask runs the website crawl on worker tasks (spec 2026-10-22-03): one crawl run is one worker_tasks row of kind "crawl", claimed once per slice by a cloud worker or by an agent of the check's private location.

The check_jobs row stays the schedule (period, "run now", lanes for the billing projections). The server side (Server) materializes a due job into a task, leases the job to it, reserves and refunds the org's rate-limit units, diffs and stores the report, and writes the run's one result. The worker side (Executor) runs one StepChecker slice per claim.

Index

Constants

View Source
const (
	// Kind is the worker_tasks.kind of a crawl run.
	Kind = "crawl"
	// Capability is what a worker running the executor reports.
	Capability = "crawl"
)

Names of the kind.

View Source
const OutputKeyReportFileUID = "reportFileUid"

OutputKeyReportFileUID names the stored report of a finished run in the result output.

Variables

This section is empty.

Functions

This section is empty.

Types

type Deps

type Deps struct {
	DB      db.Service
	Tasks   *workertasks.Service
	Results ResultRecorder
	Reports ReportStore
	// Entitlements reserves the slices' units. Nil reserves nothing.
	Entitlements *entitlements.Service
	// SliceBudget is the wall-clock budget of one slice.
	SliceBudget time.Duration
}

Deps are what the server side of the kind needs.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor is the worker side of the crawl kind: it runs one StepChecker slice per claim. The loop puts the worker's egress guard on its context.

func NewExecutor

func NewExecutor(allowEnv secretref.EnvAllowFunc) *Executor

NewExecutor builds the executor over the checker registry. allowEnv is the process's `${env:}` allowlist (config.Config.EnvRefAllowed): a config's env references resolve against the environment of the worker running the slice, which for an agent is its own.

func (*Executor) Capability

func (*Executor) Capability() string

Capability implements tasks.Executor.

func (*Executor) Kind

func (*Executor) Kind() string

Kind implements tasks.Executor.

func (*Executor) Run

func (e *Executor) Run(ctx context.Context, claim *workertasks.Claim, rep tasks.Reporter) error

Run implements tasks.Executor: decode the input, run one slice, then report the next state, the run's result, or the failure.

type Input

type Input struct {
	Type         string         `json:"type"`
	Config       map[string]any `json:"config"`
	ParamOverlay map[string]any `json:"paramOverlay,omitempty"`
}

Input is the claim input of a crawl slice: the job's type and stored config, and the server-resolved `${param:}` values laid over it. The overlay lives only in the claim (never in a row) and is merged by the executor, which resolves `${env:}` itself.

type ReportStore

type ReportStore interface {
	PutCrawlReport(ctx context.Context, orgUID, checkUID string, report []byte, details models.JSONMap) (string, error)
	LatestCrawlReport(ctx context.Context, orgUID, checkUID string) ([]byte, error)
}

ReportStore keeps the crawl reports (attachments.Service).

type Result

type Result struct {
	Status     int             `json:"status"`
	DurationMs float64         `json:"durationMs"`
	Metrics    map[string]any  `json:"metrics,omitempty"`
	Output     map[string]any  `json:"output,omitempty"`
	Report     json.RawMessage `json:"report,omitempty"`
	Units      int             `json:"units"`
}

Result is the finish body of a crawl run: the checker's result, its report and the units the last slice used.

type ResultRecorder

type ResultRecorder interface {
	RecordResult(ctx context.Context, job *models.CheckJob, req *workers.SubmitResultRequest) error
}

ResultRecorder writes a run's result through the submit path (incidents, notifications): workers.Service.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the server side of the crawl kind.

func Register

func Register(deps Deps) *Server

Register registers the crawl kind on deps.Tasks and returns its server side.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址