compactor

package
v4.0.0-rc.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: AGPL-3.0 Imports: 43 Imported by: 0

Documentation

Overview

Package compactor contains the dataobj compaction planner and worker service skeletons. The planner and worker are two roles in the broader dataobj compaction system.

The compaction planner and compaction worker are both opt-in via Loki targets and share the same dataobj.compaction.enabled gate.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Enabled gates both the dataobj-compaction-planner and the
	// dataobj-compaction-worker targets. When false, both modules are no-ops
	// even if their target is selected.
	Enabled bool `yaml:"enabled"`

	// MaxRunningCompactionTasks caps how many IndexMerge tasks the coordinator
	// runs concurrently per tenant. Zero means unlimited (one goroutine per task
	// with no admission throttle). Negative values are rejected at config
	// validation.
	MaxRunningCompactionTasks int `yaml:"max_running_compaction_tasks"`

	// LogMaxRunningCompactionTasks caps how many LogMerge tasks the coordinator
	// runs concurrently per tenant. Zero means unlimited. Negative values are
	// rejected at config validation.
	LogMaxRunningCompactionTasks int `yaml:"logs_max_running_compaction_tasks"`

	// PollingInterval is the cadence of the coordinator's main loop. Each
	// tick reads the most-recent ToC and plans compaction per tenant.
	PollingInterval time.Duration `yaml:"polling_interval"`

	// MinBackoff is the minimum wait a per-tenant worker applies between
	// compaction phases. It is both the floor between productive phases and the
	// starting point of the exponential backoff that idle or failing tenants
	// grow toward MaxBackoff.
	MinBackoff time.Duration `yaml:"min_backoff"`

	// MaxBackoff caps the exponential backoff a per-tenant worker applies after
	// consecutive no-work (converged or empty) or failing phases, so a worker
	// with nothing to do stops hammering object storage.
	MaxBackoff time.Duration `yaml:"max_backoff"`

	// WindowLookback is the number of metastore windows prior to the current one the coordinator
	// compacts in addition to the current window. It must be at least 1 (the
	// default), so tenants stay discoverable through the preceding window while
	// the current window has no ToC yet.
	WindowLookback int `yaml:"window_lookback"`

	// MaxRunsPerTask (K in the K-way merge) is the maximum number of runs a
	// single IndexMerge task may consume. Memory grows linearly with K.
	MaxRunsPerTask int `yaml:"max_runs_per_task"`

	// LogMaxRunsPerTask (K for log compaction) is the maximum number of runs a
	// single LogMerge task may consume. A size level must also hold K runs
	// before log compaction starts. Kept separate from index max runs.
	LogMaxRunsPerTask int `yaml:"logs_max_runs_per_task"`

	// LogMinCompactionSize is the minimum total compactable data (sum of all
	// runs' uncompressed size) that justifies log compaction. Windows below
	// this floor are skipped.
	LogMinCompactionSize flagext.Bytes `yaml:"logs_min_compaction_size"`

	// ToCConsolidateTimeout bounds the coordinator's inline ReplaceIndexPointers
	// call. NOT a task TTL — applied as context.WithTimeout around the metastore
	// RPC; on expiry the cycle aborts inline and the next polling tick re-plans.
	ToCConsolidateTimeout time.Duration `yaml:"toc_consolidate_timeout"`

	// DryRun, when true, skips the post-compaction ReplaceIndexPointers ToC
	// swap. Planning and IndexMerge task execution still run and the
	// per-output audit log is still emitted, but the ToC is never mutated.
	// Intended for testing index compaction.
	DryRun bool `yaml:"dry_run"`

	// PlanVersion is hashed into IndexMerge output paths so a planner
	// change invalidates previously-written outputs. Bump on any
	// breaking change to the planner or merge semantics. Stored as uint
	// because the standard library's flag package does not provide
	// Uint32Var.
	PlanVersion uint `yaml:"plan_version"`

	// Scheduler holds the scheduler-side knobs: advertise_addr and
	// endpoint for the embedded scheduler.
	Scheduler SchedulerConfig `yaml:"scheduler"`

	// Worker holds the worker-side knobs for the dataobj-compaction-worker
	// target. Independent of Scheduler: a process can be a planner-only
	// (scheduler+coordinator) or worker-only deployment, selected via
	// -target.
	Worker WorkerConfig `yaml:"worker"`

	// IndexobjBuilder controls index object construction parameters (page sizes,
	// target object/section sizes, etc.) used by the compactor worker when
	// merging postings and stats sections into a new index object.
	IndexobjBuilder logsobj.BuilderBaseConfig `yaml:"indexobj_builder" category:"experimental"`

	// LogsobjBuilder controls log object construction parameters (page sizes,
	// target object/section sizes, etc.) used by the compactor worker when
	// merging streams and logs sections into a new logs object.
	LogsobjBuilder logsobj.BuilderBaseConfig `yaml:"logsobj_builder" category:"experimental"`
}

Config is the top-level configuration for dataobj compaction.

func (*Config) RegisterFlags

func (cfg *Config) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers the compaction config flags under the given prefix.

func (*Config) RegisterFlagsWithPrefix

func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)

RegisterFlagsWithPrefix registers the compaction config flags using prefix as the flag-name prefix.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate returns nil while the compaction planner is disabled.

type Limits

type Limits interface {
	// CompactionPhases reports which compaction phases run for the tenant.
	// runIndex is true when either index or log compaction is enabled; runLog
	// is true only when log compaction is enabled. The invalid combination
	// (log without index) is rejected at config validation.
	CompactionPhases(userID string) (runIndex, runLog bool)
}

Limits is the per-tenant configuration the compaction coordinator consults. *github.com/grafana/loki/v4/pkg/validation.Overrides satisfies it.

type Planner

type Planner struct {
	*services.BasicService
	// contains filtered or unexported fields
}

Planner is the dataobj-compaction-planner target service. It hosts an embedded scheduler that compaction workers connect to and drives the stateless per-cycle coordinator.

func New

func New(params PlannerParams) (*Planner, error)

New constructs a compaction Planner. Returns an error if the scheduler cannot be constructed or if any required dependency is missing.

The disabled-compactor case (Config.Enabled = false) is handled at the module-init layer by returning nil from initDataObjCompactionPlanner; New is only reached on the enabled path.

func (*Planner) Scheduler

func (c *Planner) Scheduler() *Scheduler

Scheduler returns the embedded compactor Scheduler. Used by the Loki module-init wiring to register the scheduler's HTTP handler on the Loki router (when running in remote-transport mode) and to register scheduler metrics.

type PlannerParams

type PlannerParams struct {
	Config          Config
	Bucket          objstore.Bucket                  // required
	MetastoreWriter *metastore.TableOfContentsWriter // required
	Limits          Limits                           // required
	Logger          log.Logger
	Registerer      prometheus.Registerer
}

PlannerParams collects the constructor arguments for New.

Config.Enabled is the user-facing YAML/CLI gate consulted by the Loki module-init wiring: when false, the planner module returns a nil service and New is never called. New therefore treats every Planner it constructs as enabled — Bucket and MetastoreWriter are unconditionally required.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler is the compactor-side scheduler wrapper. It gives the coordinator access to the internal *scheduler.Scheduler.

func (*Scheduler) RegisterMetrics

func (s *Scheduler) RegisterMetrics(reg prometheus.Registerer) error

RegisterMetrics registers the scheduler's Prometheus metrics with reg.

func (*Scheduler) RegisterSchedulerServer

func (s *Scheduler) RegisterSchedulerServer(router *mux.Router)

RegisterSchedulerServer installs the wire.Listener HTTP handler on the supplied router at the configured endpoint. No-op when no advertise address was provided (in-process-only mode).

func (*Scheduler) Service

func (s *Scheduler) Service() services.Service

Service returns the lifecycle service for the embedded scheduler. The Loki module wiring drives Start / Stop through this.

func (*Scheduler) UnregisterMetrics

func (s *Scheduler) UnregisterMetrics(reg prometheus.Registerer)

UnregisterMetrics unregisters the scheduler's Prometheus metrics from reg.

type SchedulerConfig

type SchedulerConfig struct {
	// AdvertiseAddr is the host:port the embedded scheduler advertises to
	// remote workers. Empty string keeps the scheduler in-process-only
	// (no HTTP listener registered).
	AdvertiseAddr string `yaml:"advertise_addr"`

	// Endpoint is the absolute path on the Loki HTTP router where the
	// embedded scheduler listens for worker frame traffic. Defaults to
	// "/api/v2/compaction-frame".
	Endpoint string `yaml:"endpoint"`
}

SchedulerConfig holds the parameters of the embedded compaction scheduler.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker is the dataobj-compaction-worker target service. It wraps an engine.Worker pointed at the compaction scheduler's DNS-SRV record.

func NewWorker

func NewWorker(params WorkerParams) (*Worker, error)

NewWorker constructs a compaction Worker. Returns an error if the worker config is unusable or if the engine worker constructor rejects the parameters.

Worker-specific config preconditions are enforced here rather than in Config.Validate() so that planner-only deployments (which keep the worker fields at their defaults) validate cleanly.

func (*Worker) RegisterMetrics

func (w *Worker) RegisterMetrics(reg prometheus.Registerer) error

RegisterMetrics registers metrics about w to report to reg.

func (*Worker) RegisterWorkerServer

func (w *Worker) RegisterWorkerServer(router *mux.Router)

RegisterWorkerServer registers the worker's HTTP frame handler on the provided router.

func (*Worker) Service

func (w *Worker) Service() services.Service

Service returns the service used to manage the lifecycle of the Worker.

func (*Worker) UnregisterMetrics

func (w *Worker) UnregisterMetrics(reg prometheus.Registerer)

UnregisterMetrics unregisters metrics about w from reg.

type WorkerConfig

type WorkerConfig struct {
	// WorkerThreads is the per-pod task concurrency. Zero means use
	// GOMAXPROCS.
	WorkerThreads int `yaml:"worker_threads"`

	// SchedulerLookupAddress is the DNS-SRV address used to discover the
	// compaction scheduler(s). Required when the worker module actually
	// initializes (enforced in NewWorker, not in Config.Validate, so
	// planner-only deployments validate cleanly with the default empty
	// value).
	// Example: dnssrv+_compaction-frame._tcp.compactor-scheduler.svc.cluster.local
	SchedulerLookupAddress string `yaml:"scheduler_lookup_address"`

	// SchedulerLookupInterval is how often the worker re-runs the DNS-SRV
	// lookup to discover scheduler changes. Defaults to 10s.
	SchedulerLookupInterval time.Duration `yaml:"scheduler_lookup_interval"`

	// AdvertiseAddr is the host:port the embedded worker advertises to
	// schedulers and other workers. Required when the worker module
	// actually initializes (enforced in NewWorker).
	AdvertiseAddr string `yaml:"advertise_addr"`

	// Endpoint is the absolute path on the worker's HTTP router where the
	// wire frame handler is registered. Defaults to
	// "/api/v2/compaction-frame".
	Endpoint string `yaml:"endpoint"`
}

WorkerConfig holds the worker-side parameters that get passed to engine.NewWorker when the dataobj-compaction-worker target boots. The compaction worker discovers the compaction scheduler via DNS-SRV lookup.

The worker module is gated by Loki target -target=dataobj-compaction-worker AND the same dataobj.compaction.enabled flag the planner uses. There is no separate worker enable flag — the role separation comes from the Loki target.

func (*WorkerConfig) RegisterFlagsWithPrefix

func (cfg *WorkerConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)

RegisterFlagsWithPrefix registers the worker config flags using prefix as the flag-name prefix. Typically called via Config.RegisterFlagsWithPrefix with prefix = "<parent>worker.".

type WorkerParams

type WorkerParams struct {
	Config WorkerConfig

	Bucket       objstore.Bucket
	DataBucket   objstore.Bucket // unprefixed bucket for reading source log objects; nil falls back to Bucket
	ScratchStore scratch.Store

	Logger     log.Logger
	Registerer prometheus.Registerer

	// IndexobjCfg controls index object construction parameters for compaction.
	IndexobjCfg logsobj.BuilderBaseConfig
	// LogsobjCfg controls log object construction parameters for compaction.
	LogsobjCfg logsobj.BuilderBaseConfig
}

WorkerParams collects the constructor arguments for NewWorker.

Directories

Path Synopsis
internal
compactortest
Package compactortest provides object-backed compaction integration fixtures.
Package compactortest provides object-backed compaction integration fixtures.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址