Documentation
¶
Index ¶
- Constants
- type Bucket
- type BucketReaderAt
- type FileReaderAt
- type Iceberg
- func (i *Iceberg) Close() error
- func (i *Iceberg) Delete(_ context.Context, _ string) error
- func (i *Iceberg) Maintenance(ctx context.Context) error
- func (i *Iceberg) Prefixes(ctx context.Context, prefix string) ([]string, error)
- func (i *Iceberg) Scan(ctx context.Context, prefix string, _ *dynparquet.Schema, ...) error
- func (i *Iceberg) String() string
- func (i *Iceberg) Upload(ctx context.Context, name string, r io.Reader) error
- type IcebergOption
Constants ¶
const (
DefaultOrphanedFileAge = 24 * time.Hour
)
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bucket ¶
type Bucket interface {
objstore.Bucket
GetReaderAt(ctx context.Context, name string) (io.ReaderAt, error)
}
Bucket is an objstore.Bucket that also supports reading files via a ReaderAt interface.
type BucketReaderAt ¶
BucketReaderAt implements the Bucket interface.
func NewBucketReaderAt ¶
func NewBucketReaderAt(bucket objstore.Bucket) *BucketReaderAt
NewBucketReaderAt returns a new Bucket.
func (*BucketReaderAt) GetReaderAt ¶
GetReaderAt returns a io.ReaderAt for the given filename.
type FileReaderAt ¶
FileReaderAt is a wrapper around a objstore.Bucket that implements the ReaderAt interface.
type Iceberg ¶
type Iceberg struct {
// contains filtered or unexported fields
}
Iceberg is an Apache Iceberg backed DataSink/DataSource.
func NewIceberg ¶
func NewIceberg(uri string, ctlg catalog.Catalog, bucket objstore.Bucket, options ...IcebergOption) (*Iceberg, error)
NewIceberg creates a new Iceberg DataSink/DataSource. You must provide the URI of the warehouse and the objstore.Bucket that points to that warehouse.
func (*Iceberg) Prefixes ¶
Prefixes lists all the tables found in the warehouse for the given database(prefix).
func (*Iceberg) Scan ¶
func (i *Iceberg) Scan(ctx context.Context, prefix string, _ *dynparquet.Schema, filter logicalplan.Expr, _ uint64, callback func(context.Context, any) error) error
Scan will load the latest Iceberg table. It will filter out any manifests that do not contain useful data. Then it will read the manifests that may contain useful data. It will then filter out the data file that dot not contain useful data. Finally it has a set of data files that may contain useful data. It will then read the data files and apply the filter to each row group in the data file.
type IcebergOption ¶
type IcebergOption func(*Iceberg)
IcebergOption is a function that configures an Iceberg DataSink/DataSource.
func WithDataFileExpiry ¶
func WithDataFileExpiry(maxAge time.Duration) IcebergOption
WithDataFileExpiry will a maxiumum age for data files. Data files older than the max age will be deleted from the table periodically according to the maintenance schedule.
func WithIcebergPartitionSpec ¶
func WithIcebergPartitionSpec(spec iceberg.PartitionSpec) IcebergOption
WithIcebergPartitionSpec sets the partition spec for the Iceberg table. This is useful for pruning manifests during scans. note that at this time the Iceberg storage engine does not write data in a partition fashion. So this is only useful for setting the upper/lower bounds of columns in the manifest data.
func WithLogger ¶
func WithLogger(l log.Logger) IcebergOption
func WithMaintenanceSchedule ¶
func WithMaintenanceSchedule(schedule time.Duration) IcebergOption
WithMaintenanceSchedule sets the schedule for the maintenance of the Iceberg table. This will spawn a goroutine that will periodically expire data files if WithDataFileExpiry is set. And will delete orphanded files from the table.