storage

package
v0.0.0-...-9087d8f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 20 Imported by: 1

Documentation

Index

Constants

View Source
const (
	DefaultOrphanedFileAge = 24 * time.Hour
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Bucket

type Bucket interface {
	objstore.Bucket
	GetReaderAt(ctx context.Context, name string) (io.ReaderAt, error)
}

Bucket is an objstore.Bucket that also supports reading files via a ReaderAt interface.

type BucketReaderAt

type BucketReaderAt struct {
	objstore.Bucket
}

BucketReaderAt implements the Bucket interface.

func NewBucketReaderAt

func NewBucketReaderAt(bucket objstore.Bucket) *BucketReaderAt

NewBucketReaderAt returns a new Bucket.

func (*BucketReaderAt) GetReaderAt

func (b *BucketReaderAt) GetReaderAt(ctx context.Context, name string) (io.ReaderAt, error)

GetReaderAt returns a io.ReaderAt for the given filename.

type FileReaderAt

type FileReaderAt struct {
	objstore.Bucket
	// contains filtered or unexported fields
}

FileReaderAt is a wrapper around a objstore.Bucket that implements the ReaderAt interface.

func (*FileReaderAt) ReadAt

func (b *FileReaderAt) ReadAt(p []byte, off int64) (n int, err error)

ReadAt implements the io.ReaderAt interface.

type Iceberg

type Iceberg struct {
	// contains filtered or unexported fields
}

Iceberg is an Apache Iceberg backed DataSink/DataSource.

func NewIceberg

func NewIceberg(uri string, ctlg catalog.Catalog, bucket objstore.Bucket, options ...IcebergOption) (*Iceberg, error)

NewIceberg creates a new Iceberg DataSink/DataSource. You must provide the URI of the warehouse and the objstore.Bucket that points to that warehouse.

func (*Iceberg) Close

func (i *Iceberg) Close() error

func (*Iceberg) Delete

func (i *Iceberg) Delete(_ context.Context, _ string) error

func (*Iceberg) Maintenance

func (i *Iceberg) Maintenance(ctx context.Context) error

func (*Iceberg) Prefixes

func (i *Iceberg) Prefixes(ctx context.Context, prefix string) ([]string, error)

Prefixes lists all the tables found in the warehouse for the given database(prefix).

func (*Iceberg) Scan

func (i *Iceberg) Scan(ctx context.Context, prefix string, _ *dynparquet.Schema, filter logicalplan.Expr, _ uint64, callback func(context.Context, any) error) error

Scan will load the latest Iceberg table. It will filter out any manifests that do not contain useful data. Then it will read the manifests that may contain useful data. It will then filter out the data file that dot not contain useful data. Finally it has a set of data files that may contain useful data. It will then read the data files and apply the filter to each row group in the data file.

func (*Iceberg) String

func (i *Iceberg) String() string

func (*Iceberg) Upload

func (i *Iceberg) Upload(ctx context.Context, name string, r io.Reader) error

Upload a parquet file into the Iceberg table.

type IcebergOption

type IcebergOption func(*Iceberg)

IcebergOption is a function that configures an Iceberg DataSink/DataSource.

func WithDataFileExpiry

func WithDataFileExpiry(maxAge time.Duration) IcebergOption

WithDataFileExpiry will a maxiumum age for data files. Data files older than the max age will be deleted from the table periodically according to the maintenance schedule.

func WithIcebergPartitionSpec

func WithIcebergPartitionSpec(spec iceberg.PartitionSpec) IcebergOption

WithIcebergPartitionSpec sets the partition spec for the Iceberg table. This is useful for pruning manifests during scans. note that at this time the Iceberg storage engine does not write data in a partition fashion. So this is only useful for setting the upper/lower bounds of columns in the manifest data.

func WithLogger

func WithLogger(l log.Logger) IcebergOption

func WithMaintenanceSchedule

func WithMaintenanceSchedule(schedule time.Duration) IcebergOption

WithMaintenanceSchedule sets the schedule for the maintenance of the Iceberg table. This will spawn a goroutine that will periodically expire data files if WithDataFileExpiry is set. And will delete orphanded files from the table.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址