builder

package
v4.0.0-rc.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: AGPL-3.0 Imports: 34 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// LogsobjBuilder controls the construction of the data objects built from
	// the consumed records.
	LogsobjBuilder logsobj.BuilderBaseConfig `yaml:"logsobj_builder"`

	// IndexobjBuilder controls the construction of the index object built for
	// each data object.
	IndexobjBuilder logsobj.BuilderBaseConfig `yaml:"indexobj_builder"`

	IdleFlushTimeout time.Duration `yaml:"idle_flush_timeout"`
	MaxBuilderAge    time.Duration `yaml:"max_builder_age"`

	// FlushConcurrency is the maximum number of data objects that one flush
	// builds, uploads and indexes at the same time. It must be at least 1.
	FlushConcurrency int `yaml:"flush_concurrency"`

	// This is temporary until we move to kafkav2.
	Topic string `yaml:"topic"`

	// PartitionID overrides the Kafka partition this instance consumes. A negative value (the
	// default) derives the partition from the hostname.
	PartitionID int `yaml:"partition_id"`

	// RequiredReplicasRefreshInterval is the time between two refreshes of
	// loki_dataobj_builder_required_replicas. Only the builder that derives
	// partition 0 from its hostname uses it.
	RequiredReplicasRefreshInterval time.Duration `yaml:"required_replicas_refresh_interval"`
}

func (*Config) RegisterFlags

func (cfg *Config) RegisterFlags(f *flag.FlagSet)

func (*Config) RegisterFlagsWithPrefix

func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)

func (*Config) Validate

func (cfg *Config) Validate() error

type MultiObjectBuilder

type MultiObjectBuilder struct {
	// contains filtered or unexported fields
}

MultiObjectBuilder keeps one logsobj.Builder per tenant and ToC window.

func NewMultiObjectBuilder

func NewMultiObjectBuilder(builderFactory builderFactory, maxBufferedBytes int) *MultiObjectBuilder

NewMultiObjectBuilder creates a new builder group scoped by tenant and ToC window. maxBufferedBytes is the combined upper bound across all builders.

func (*MultiObjectBuilder) Append

func (m *MultiObjectBuilder) Append(tenant string, stream logproto.Stream, recTime time.Time) error

func (*MultiObjectBuilder) GetEstimatedSize

func (m *MultiObjectBuilder) GetEstimatedSize() int

func (*MultiObjectBuilder) IsFull

func (m *MultiObjectBuilder) IsFull() bool

IsFull reports whether the combined buffered size across all builders has exceeded the configured target object size.

We intentionally compare the *sum* (rather than each individual builder's IsFull) so that a partition that receives records spanning many tenants and windows is still bounded to roughly MaxBufferedBytes of buffered memory, rather than K*MaxBufferedBytes.

func (*MultiObjectBuilder) Reset

func (m *MultiObjectBuilder) Reset()

type Service

type Service struct {
	services.Service
	// contains filtered or unexported fields
}

func New

func New(kafkaCfg kafka.Config, cfg Config, uploaderCfg dataobj_uploader.Config, mCfg metastore.Config, bucket objstore.Bucket, scratchStore scratch.Store, reg prometheus.Registerer, logger log.Logger, overrides logsobj.TenantOverrides) (*Service, error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址