lifecycle

package
v0.11.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 75 Imported by: 0

Documentation

Overview

Package lifecycle provides the lifecycle migration service.

Package lifecycle implements row-level replay paths used by migration visitors when a source segment maps to more than one target segment. The receiving data node owns segment creation per-row, which makes the per-target sender loop unnecessary.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewCommand

func NewCommand() *cobra.Command

NewCommand creates a new lifecycle command.

func NewCommandWithRegistry added in v0.11.0

func NewCommandWithRegistry() (*cobra.Command, observability.MetricsRegistry)

NewCommandWithRegistry creates a new lifecycle command and also returns the metrics registry it wires. Tests and embedders use the registry to inspect the lifecycle and banyandb_lifecycle_migration_* metric families (e.g. via its observability.PrometheusHandlerProvider) without scraping a live HTTP port.

func NewService

func NewService(
	meta metadata.Repo,
	metricsRegistry observability.MetricsRegistry,
	metricsClient queue.Client,
) run.Unit

NewService creates a new lifecycle service. metricsRegistry replaces the previous BypassRegistry, so the protector memory metrics and the lifecycle proof counter emit real series. metricsClient is the native-metrics pipeline to the co-located data node; it is only exercised when native mode is enabled.

Types

type GroupConfig added in v0.9.0

type GroupConfig struct {
	*commonv1.Group
	NodeSelector   node.Selector
	QueueClient    queue.Client
	AccumulatedTTL *commonv1.IntervalRule
	// SegmentInterval is the current stage's segment interval, used to read
	// source segments on this node.
	SegmentInterval *commonv1.IntervalRule
	// TargetSegmentInterval is the next stage's segment interval. It differs
	// from SegmentInterval on 3-stage deployments (e.g. warm->cold), so any
	// computation against the target tier's segment grid must use this field.
	TargetSegmentInterval *commonv1.IntervalRule
	// SourceStage and TargetStage are the migration's source and target stage
	// names (e.g. "hot" -> "warm"), surfaced in the migration report's errors.
	SourceStage    string
	TargetStage    string
	TargetShardNum uint32
	TargetReplicas uint32
}

GroupConfig encapsulates the parsed lifecycle configuration for a Group. It contains all necessary information for migration and deletion operations.

func (*GroupConfig) Close added in v0.9.0

func (gc *GroupConfig) Close()

Close releases resources held by the GroupConfig.

type MigrationError added in v0.11.0

type MigrationError struct {
	Shard       *uint32 `json:"shard,omitempty"`
	Part        *uint64 `json:"part,omitempty"`
	SourceStage string  `json:"source_stage"`
	TargetStage string  `json:"target_stage"`
	Group       string  `json:"group"`
	Catalog     string  `json:"catalog"`
	Scope       string  `json:"scope"`
	Segment     string  `json:"segment,omitempty"`
	Interval    string  `json:"interval,omitempty"`
	Node        string  `json:"node,omitempty"`
	Error       string  `json:"error"`
}

MigrationError is a single structured migration error for the report. It carries the error location (group/segment/shard/part/node) as named fields plus the source/target stage and a human-readable segment directory name and segment interval. Shard/Part are pointers so a series- or node-scoped error omits them while a part-scoped error on shard 0 still emits "shard": 0.

type Progress

type Progress struct {
	CompletedGroups             map[string]bool                                          `json:"completed_groups"`
	DeletedStreamGroups         map[string]bool                                          `json:"deleted_stream_groups"`
	DeletedMeasureGroups        map[string]bool                                          `json:"deleted_measure_groups"`
	DeletedTraceGroups          map[string]bool                                          `json:"deleted_trace_groups"`
	CompletedStreamParts        map[string]map[string]map[common.ShardID]map[uint64]bool `json:"completed_stream_parts"`
	CompletedStreamSeries       map[string]map[string]map[common.ShardID]bool            `json:"completed_stream_series"`
	CompletedStreamElementIndex map[string]map[string]map[common.ShardID]bool            `json:"completed_stream_element_index"`
	StreamPartCounts            map[string]int                                           `json:"stream_part_counts"`
	StreamPartProgress          map[string]int                                           `json:"stream_part_progress"`
	StreamSeriesCounts          map[string]int                                           `json:"stream_series_counts"`
	StreamSeriesProgress        map[string]int                                           `json:"stream_series_progress"`
	StreamElementIndexCounts    map[string]int                                           `json:"stream_element_index_counts"`
	StreamElementIndexProgress  map[string]int                                           `json:"stream_element_index_progress"`
	// Measure part-specific progress tracking
	CompletedMeasureParts  map[string]map[string]map[common.ShardID]map[uint64]bool `json:"completed_measure_parts"`
	MeasurePartCounts      map[string]int                                           `json:"measure_part_counts"`
	MeasurePartProgress    map[string]int                                           `json:"measure_part_progress"`
	CompletedMeasureSeries map[string]map[string]map[common.ShardID]bool            `json:"completed_measure_series"`
	MeasureSeriesCounts    map[string]int                                           `json:"measure_series_counts"`
	MeasureSeriesProgress  map[string]int                                           `json:"measure_series_progress"`
	// Trace part-specific progress tracking
	CompletedTraceShards map[string]map[string]map[common.ShardID]bool `json:"completed_trace_shards"`
	TraceShardCounts     map[string]int                                `json:"trace_shard_counts"`
	TraceShardProgress   map[string]int                                `json:"trace_shard_progress"`
	CompletedTraceSeries map[string]map[string]map[common.ShardID]bool `json:"completed_trace_series"`
	TraceSeriesCounts    map[string]int                                `json:"trace_series_counts"`
	TraceSeriesProgress  map[string]int                                `json:"trace_series_progress"`

	// Per-source completion tracking (advances *Progress idempotently once per source item).
	SourceCompletedStreamParts        map[string]map[string]map[common.ShardID]map[uint64]bool `json:"source_completed_stream_parts"`
	SourceCompletedStreamSeries       map[string]map[string]map[common.ShardID]bool            `json:"source_completed_stream_series"`
	SourceCompletedStreamElementIndex map[string]map[string]map[common.ShardID]bool            `json:"source_completed_stream_element_index"`
	SourceCompletedMeasureParts       map[string]map[string]map[common.ShardID]map[uint64]bool `json:"source_completed_measure_parts"`
	SourceCompletedMeasureSeries      map[string]map[string]map[common.ShardID]bool            `json:"source_completed_measure_series"`
	SourceCompletedTraceShards        map[string]map[string]map[common.ShardID]bool            `json:"source_completed_trace_shards"`
	SourceCompletedTraceSeries        map[string]map[string]map[common.ShardID]bool            `json:"source_completed_trace_series"`

	// Sync-mode breakdown per group: how many parts (measure/stream) or shards
	// (trace) were migrated via chunk-sync vs row-replay, plus the real number
	// of rows replayed. Accumulated across resume cycles, fed into the report's
	// summary.<catalog>.sync_breakdown.
	StreamChunkSyncParts  map[string]uint64 `json:"stream_chunk_sync_parts"`
	StreamRowReplayParts  map[string]uint64 `json:"stream_row_replay_parts"`
	StreamRowReplayRows   map[string]uint64 `json:"stream_row_replay_rows"`
	MeasureChunkSyncParts map[string]uint64 `json:"measure_chunk_sync_parts"`
	MeasureRowReplayParts map[string]uint64 `json:"measure_row_replay_parts"`
	MeasureRowReplayRows  map[string]uint64 `json:"measure_row_replay_rows"`
	TraceChunkSyncShards  map[string]uint64 `json:"trace_chunk_sync_shards"`
	TraceRowReplayParts   map[string]uint64 `json:"trace_row_replay_parts"`
	TraceRowReplayRows    map[string]uint64 `json:"trace_row_replay_rows"`

	// OrphanRows reports, per catalog -> group -> deleted measure/stream subject,
	// how many rows were archived or discarded because the schema was deleted from
	// the registry. Accumulated across resume cycles (orphan parts are marked
	// completed, so each row is counted once), fed into the report's orphans.
	OrphanRows map[string]map[string]map[string]uint64 `json:"orphan_rows"`

	// MigrationErrors is the single source of structured migration errors for both
	// the report (stage/group/catalog/scope/segment/interval/location/message) and
	// the per-resource error counts. Reset by ClearErrors at the start of each run
	// so the report reflects the current attempt; persisted only for crash
	// durability within a run.
	MigrationErrors []MigrationError `json:"migration_errors"`

	SnapshotStreamDir  string   `json:"snapshot_stream_dir"`
	SnapshotMeasureDir string   `json:"snapshot_measure_dir"`
	SnapshotTraceDir   string   `json:"snapshot_trace_dir"`
	GroupsToProcess    []string `json:"groups_to_process"`
	// contains filtered or unexported fields
}

Progress tracks the lifecycle migration progress to support resume after crash.

func LoadProgress

func LoadProgress(path string, l *logger.Logger) *Progress

LoadProgress loads progress from a file if it exists.

func NewProgress

func NewProgress(path string, l *logger.Logger) *Progress

NewProgress creates a new Progress tracker.

func (*Progress) AddMeasureChunkSyncPart added in v0.11.0

func (p *Progress) AddMeasureChunkSyncPart(group string)

AddMeasureChunkSyncPart records one measure part migrated via chunk-sync.

func (*Progress) AddMeasureRowReplay added in v0.11.0

func (p *Progress) AddMeasureRowReplay(group string, rows int)

AddMeasureRowReplay records one measure part migrated via row-replay and the number of rows it republished.

func (*Progress) AddMigrationError added in v0.11.0

func (p *Progress) AddMigrationError(e MigrationError)

AddMigrationError appends a structured migration error to the report-facing list under the progress lock.

func (*Progress) AddOrphanRows added in v0.11.0

func (p *Progress) AddOrphanRows(group, catalog string, bySubject map[string]uint64)

AddOrphanRows accumulates per-subject orphan row counts for one (catalog, group) into the report's orphans section. Additive so resume cycles — each replaying a different subset of parts — sum to the true total.

func (*Progress) AddStreamChunkSyncPart added in v0.11.0

func (p *Progress) AddStreamChunkSyncPart(group string)

AddStreamChunkSyncPart records one stream part migrated via chunk-sync.

func (*Progress) AddStreamRowReplay added in v0.11.0

func (p *Progress) AddStreamRowReplay(group string, rows int)

AddStreamRowReplay records one stream part migrated via row-replay and the number of rows it republished.

func (*Progress) AddTraceChunkSyncShard added in v0.11.0

func (p *Progress) AddTraceChunkSyncShard(group string)

AddTraceChunkSyncShard records one trace shard migrated via chunk-sync.

func (*Progress) AddTraceRowReplay added in v0.11.0

func (p *Progress) AddTraceRowReplay(group string, parts, rows int)

AddTraceRowReplay records the parts and rows of a trace shard migrated via row-replay (one shard may contain several parts).

func (*Progress) AllGroupsNotFullyCompleted added in v0.9.0

func (p *Progress) AllGroupsNotFullyCompleted(groups []*commonv1.Group) []string

AllGroupsNotFullyCompleted find is there have any group not fully completed.

func (*Progress) ClearErrors added in v0.9.0

func (p *Progress) ClearErrors()

ClearErrors resets the recorded migration errors at the start of a run so a fresh cycle does not inherit the previous run's errors.

func (*Progress) GetMeasurePartCount added in v0.9.0

func (p *Progress) GetMeasurePartCount(group string) int

GetMeasurePartCount returns the total number of parts for a measure.

func (*Progress) GetMeasurePartProgress added in v0.9.0

func (p *Progress) GetMeasurePartProgress(group string) int

GetMeasurePartProgress returns the number of completed parts for a measure.

func (*Progress) GetMeasureSeriesCount added in v0.9.0

func (p *Progress) GetMeasureSeriesCount(group string) int

GetMeasureSeriesCount returns the total number of series segments for a measure.

func (*Progress) GetMeasureSeriesProgress added in v0.9.0

func (p *Progress) GetMeasureSeriesProgress(group string) int

GetMeasureSeriesProgress returns the number of completed series segments for a measure.

func (*Progress) GetStreamElementIndexCount added in v0.9.0

func (p *Progress) GetStreamElementIndexCount(group string) int

GetStreamElementIndexCount returns the total number of element index files for a stream.

func (*Progress) GetStreamElementIndexProgress added in v0.9.0

func (p *Progress) GetStreamElementIndexProgress(group string) int

GetStreamElementIndexProgress returns the number of completed element index files for a stream.

func (*Progress) GetStreamPartCount added in v0.9.0

func (p *Progress) GetStreamPartCount(group string) int

GetStreamPartCount returns the total number of parts for a stream.

func (*Progress) GetStreamPartProgress added in v0.9.0

func (p *Progress) GetStreamPartProgress(group string) int

GetStreamPartProgress returns the number of completed parts for a stream.

func (*Progress) GetStreamSeriesCount added in v0.9.0

func (p *Progress) GetStreamSeriesCount(group string) int

GetStreamSeriesCount returns the total number of series segments for a stream.

func (*Progress) GetStreamSeriesProgress added in v0.9.0

func (p *Progress) GetStreamSeriesProgress(group string) int

GetStreamSeriesProgress returns the number of completed series segments for a stream.

func (*Progress) GetTraceSeriesCount added in v0.10.0

func (p *Progress) GetTraceSeriesCount(group string) int

GetTraceSeriesCount gets the total number of series segments for the current trace.

func (*Progress) GetTraceSeriesProgress added in v0.10.0

func (p *Progress) GetTraceSeriesProgress(group string) int

GetTraceSeriesProgress gets the number of completed series segments for the current trace.

func (*Progress) GetTraceShardProgress added in v0.10.0

func (p *Progress) GetTraceShardProgress(group string) int

GetTraceShardProgress gets the number of completed shards for the current trace.

func (*Progress) GetTraceShards added in v0.10.0

func (p *Progress) GetTraceShards(group string) int

GetTraceShards gets the total number of shards for the current trace.

func (*Progress) IsGroupCompleted

func (p *Progress) IsGroupCompleted(group string) bool

IsGroupCompleted checks if a group has been completed.

func (*Progress) IsMeasureGroupDeleted

func (p *Progress) IsMeasureGroupDeleted(group string) bool

IsMeasureGroupDeleted checks if a measure group segments have been deleted.

func (*Progress) IsMeasurePartCompleted added in v0.9.0

func (p *Progress) IsMeasurePartCompleted(group string, segmentID string, shardID common.ShardID, partID uint64) bool

IsMeasurePartCompleted checks if a specific part of a measure has been completed.

func (*Progress) IsMeasureSeriesCompleted added in v0.9.0

func (p *Progress) IsMeasureSeriesCompleted(group string, segmentID string, shardID common.ShardID) bool

IsMeasureSeriesCompleted checks if a specific series segment of a measure has been completed.

func (*Progress) IsStreamElementIndexCompleted added in v0.9.0

func (p *Progress) IsStreamElementIndexCompleted(group string, segmentID string, shardID common.ShardID) bool

IsStreamElementIndexCompleted checks if a specific element index file of a stream has been completed.

func (*Progress) IsStreamGroupDeleted

func (p *Progress) IsStreamGroupDeleted(group string) bool

IsStreamGroupDeleted checks if a stream group segments have been deleted.

func (*Progress) IsStreamPartCompleted added in v0.9.0

func (p *Progress) IsStreamPartCompleted(group string, segmentID string, shardID common.ShardID, partID uint64) bool

IsStreamPartCompleted checks if a specific part of a stream has been completed.

func (*Progress) IsStreamSeriesCompleted added in v0.9.0

func (p *Progress) IsStreamSeriesCompleted(group string, segmentID string, shardID common.ShardID) bool

IsStreamSeriesCompleted checks if a specific series segment of a stream has been completed.

func (*Progress) IsTraceGroupDeleted added in v0.10.0

func (p *Progress) IsTraceGroupDeleted(group string) bool

IsTraceGroupDeleted checks if a trace group has been deleted.

func (*Progress) IsTraceSeriesCompleted added in v0.10.0

func (p *Progress) IsTraceSeriesCompleted(group string, segmentID string, shardID common.ShardID) bool

IsTraceSeriesCompleted checks if a specific series segment of a trace has been completed.

func (*Progress) IsTraceShardCompleted added in v0.10.0

func (p *Progress) IsTraceShardCompleted(group string, segmentID string, shardID common.ShardID) bool

IsTraceShardCompleted checks if a specific part of a trace has been completed.

func (*Progress) MarkGroupCompleted

func (p *Progress) MarkGroupCompleted(group string)

MarkGroupCompleted marks a group as completed.

func (*Progress) MarkMeasureGroupDeleted

func (p *Progress) MarkMeasureGroupDeleted(group string)

MarkMeasureGroupDeleted marks a measure group segments as deleted.

func (*Progress) MarkMeasurePartCompleted added in v0.9.0

func (p *Progress) MarkMeasurePartCompleted(group string, segmentID string, shardID common.ShardID, partID uint64)

MarkMeasurePartCompleted marks a specific part of a measure as completed.

func (*Progress) MarkMeasureSeriesCompleted added in v0.9.0

func (p *Progress) MarkMeasureSeriesCompleted(group string, segmentID string, shardID common.ShardID)

MarkMeasureSeriesCompleted marks a specific series segment of a measure as completed.

func (*Progress) MarkSourceMeasurePartCompleted added in v0.10.3

func (p *Progress) MarkSourceMeasurePartCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID, partID uint64)

MarkSourceMeasurePartCompleted records that one source measure part finished every target write successfully; idempotent.

func (*Progress) MarkSourceMeasureSeriesCompleted added in v0.10.3

func (p *Progress) MarkSourceMeasureSeriesCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID)

MarkSourceMeasureSeriesCompleted records that one source measure series file finished every target write successfully; idempotent.

func (*Progress) MarkSourceStreamElementIndexCompleted added in v0.10.3

func (p *Progress) MarkSourceStreamElementIndexCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID)

MarkSourceStreamElementIndexCompleted records that one source stream element-index visit finished every target write successfully; idempotent.

func (*Progress) MarkSourceStreamPartCompleted added in v0.10.3

func (p *Progress) MarkSourceStreamPartCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID, partID uint64)

MarkSourceStreamPartCompleted records that one source stream part finished every target write successfully; idempotent (++Progress on first call only).

func (*Progress) MarkSourceStreamSeriesCompleted added in v0.10.3

func (p *Progress) MarkSourceStreamSeriesCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID)

MarkSourceStreamSeriesCompleted records that one source stream series file finished every target write successfully; idempotent.

func (*Progress) MarkSourceTraceSeriesCompleted added in v0.10.3

func (p *Progress) MarkSourceTraceSeriesCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID)

MarkSourceTraceSeriesCompleted records that one source trace series file finished every target write successfully; idempotent.

func (*Progress) MarkSourceTraceShardCompleted added in v0.10.3

func (p *Progress) MarkSourceTraceShardCompleted(group string, sourceSegmentID string, sourceShardID common.ShardID)

MarkSourceTraceShardCompleted records that one source trace shard finished every target write successfully; idempotent.

func (*Progress) MarkStreamElementIndexCompleted added in v0.9.0

func (p *Progress) MarkStreamElementIndexCompleted(group string, segmentID string, shardID common.ShardID)

MarkStreamElementIndexCompleted marks a specific element index file of a stream as completed.

func (*Progress) MarkStreamGroupDeleted

func (p *Progress) MarkStreamGroupDeleted(group string)

MarkStreamGroupDeleted marks a stream group segments as deleted.

func (*Progress) MarkStreamPartCompleted added in v0.9.0

func (p *Progress) MarkStreamPartCompleted(group string, segmentID string, shardID common.ShardID, partID uint64)

MarkStreamPartCompleted marks a specific part of a stream as completed.

func (*Progress) MarkStreamSeriesCompleted added in v0.9.0

func (p *Progress) MarkStreamSeriesCompleted(group string, segmentID string, shardID common.ShardID)

MarkStreamSeriesCompleted marks a specific series segment of a stream as completed.

func (*Progress) MarkTraceGroupDeleted added in v0.10.0

func (p *Progress) MarkTraceGroupDeleted(group string)

MarkTraceGroupDeleted marks a trace group as deleted.

func (*Progress) MarkTraceSeriesCompleted added in v0.10.0

func (p *Progress) MarkTraceSeriesCompleted(group string, segmentID string, shardID common.ShardID)

MarkTraceSeriesCompleted marks a specific series segment of a trace as completed.

func (*Progress) MarkTraceShardCompleted added in v0.10.0

func (p *Progress) MarkTraceShardCompleted(group string, segmentID string, shardID common.ShardID)

MarkTraceShardCompleted marks a specific shard of a trace as completed.

func (*Progress) Remove

func (p *Progress) Remove(path string, l *logger.Logger)

Remove deletes the progress file.

func (*Progress) Save

func (p *Progress) Save(path string, l *logger.Logger)

Save writes the progress to the specified file.

func (*Progress) SetGroupsToProcess added in v0.10.3

func (p *Progress) SetGroupsToProcess(groups []string)

SetGroupsToProcess records the set of groups picked up by this migration cycle. Used as the denominator for migration_status.completion_rate so the report reflects "completed / scheduled" rather than overlapping per-catalog buckets.

func (*Progress) SetMeasurePartCount added in v0.9.0

func (p *Progress) SetMeasurePartCount(group string, totalParts int)

SetMeasurePartCount sets the total number of parts for a measure.

func (*Progress) SetMeasureSeriesCount added in v0.9.0

func (p *Progress) SetMeasureSeriesCount(group string, totalSegments int)

SetMeasureSeriesCount sets the total number of series segments for a measure.

func (*Progress) SetStreamElementIndexCount added in v0.9.0

func (p *Progress) SetStreamElementIndexCount(group string, totalIndexFiles int)

SetStreamElementIndexCount sets the total number of element index files for a stream.

func (*Progress) SetStreamPartCount added in v0.9.0

func (p *Progress) SetStreamPartCount(group string, totalParts int)

SetStreamPartCount sets the total number of parts for a stream.

func (*Progress) SetStreamSeriesCount added in v0.9.0

func (p *Progress) SetStreamSeriesCount(group string, totalSegments int)

SetStreamSeriesCount sets the total number of series segments for a stream.

func (*Progress) SetTraceSeriesCount added in v0.10.0

func (p *Progress) SetTraceSeriesCount(group string, totalSegments int)

SetTraceSeriesCount sets the total number of series segments for the current trace.

func (*Progress) SetTraceShardCount added in v0.10.0

func (p *Progress) SetTraceShardCount(group string, totalShards int)

SetTraceShardCount sets the total number of shards for the current trace.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址