Documentation
¶
Overview ¶
Package crawltask runs the website crawl on worker tasks (spec 2026-10-22-03): one crawl run is one worker_tasks row of kind "crawl", claimed once per slice by a cloud worker or by an agent of the check's private location.
The check_jobs row stays the schedule (period, "run now", lanes for the billing projections). The server side (Server) materializes a due job into a task, leases the job to it, reserves and refunds the org's rate-limit units, diffs and stores the report, and writes the run's one result. The worker side (Executor) runs one StepChecker slice per claim.
Index ¶
Constants ¶
const ( // Kind is the worker_tasks.kind of a crawl run. Kind = "crawl" // Capability is what a worker running the executor reports. Capability = "crawl" )
Names of the kind.
const OutputKeyReportFileUID = "reportFileUid"
OutputKeyReportFileUID names the stored report of a finished run in the result output.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Deps ¶
type Deps struct {
DB db.Service
Tasks *workertasks.Service
Results ResultRecorder
Reports ReportStore
// Entitlements reserves the slices' units. Nil reserves nothing.
Entitlements *entitlements.Service
// SliceBudget is the wall-clock budget of one slice.
SliceBudget time.Duration
}
Deps are what the server side of the kind needs.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor is the worker side of the crawl kind: it runs one StepChecker slice per claim. The loop puts the worker's egress guard on its context.
func NewExecutor ¶
func NewExecutor(allowEnv secretref.EnvAllowFunc) *Executor
NewExecutor builds the executor over the checker registry. allowEnv is the process's `${env:}` allowlist (config.Config.EnvRefAllowed): a config's env references resolve against the environment of the worker running the slice, which for an agent is its own.
func (*Executor) Capability ¶
Capability implements tasks.Executor.
type Input ¶
type Input struct {
Type string `json:"type"`
Config map[string]any `json:"config"`
ParamOverlay map[string]any `json:"paramOverlay,omitempty"`
}
Input is the claim input of a crawl slice: the job's type and stored config, and the server-resolved `${param:}` values laid over it. The overlay lives only in the claim (never in a row) and is merged by the executor, which resolves `${env:}` itself.
type ReportStore ¶
type ReportStore interface {
PutCrawlReport(ctx context.Context, orgUID, checkUID string, report []byte, details models.JSONMap) (string, error)
LatestCrawlReport(ctx context.Context, orgUID, checkUID string) ([]byte, error)
}
ReportStore keeps the crawl reports (attachments.Service).
type Result ¶
type Result struct {
Status int `json:"status"`
DurationMs float64 `json:"durationMs"`
Metrics map[string]any `json:"metrics,omitempty"`
Output map[string]any `json:"output,omitempty"`
Report json.RawMessage `json:"report,omitempty"`
Units int `json:"units"`
}
Result is the finish body of a crawl run: the checker's result, its report and the units the last slice used.
type ResultRecorder ¶
type ResultRecorder interface {
RecordResult(ctx context.Context, job *models.CheckJob, req *workers.SubmitResultRequest) error
}
ResultRecorder writes a run's result through the submit path (incidents, notifications): workers.Service.