block

package
v0.2.16 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 12, 2026 License: BSD-3-Clause Imports: 34 Imported by: 0

Documentation

Index

Constants

View Source 
const AllocationOverheadAllowance = 384

AllocationOverheadAllowance is the amount of extra bytes we budget when fitting a block to an allocator size class. This accounts for metadata (of size cache.ValueMetadataSize+block.MetadataSize) that is allocated together with the block data.

For instance, we may have a block of size 1020B that by itself would fit within a 1KiB class. However, when loaded into the block cache we also allocate space for the cache entry and block metadata. The new allocation may now only fit within a 2KiB class, which increases internal fragmentation.

The exact overhead is cache.ValueMetadataSize + MetadataSize, but we add some extra slack to allow for some metadata growth in future releases without causing fragmentation when loading blocks created by this release.

View Source 
const CategoryMax = 30

CategoryMax is the maximum value of a category, and is also the maximum number of categories that can be registered.

View Source 
const MetadataSize = 280

MetadataSize is the size of the metadata. The value is chosen to fit a colblk.DataBlockDecoder and a CockroachDB colblk.KeySeeker.

View Source 
const TrailerLen = 5

TrailerLen is the length of the trailer at the end of a block.

Variables

View Source 
var NoReadEnv = ReadEnv{}

NoReadEnv is the empty ReadEnv which reports no stats and does not use a buffer pool.

Functions

func CastMetadata 

func CastMetadata[T any](md *Metadata) *T

CastMetadata casts the provided metadata to the type parameter T. If the Metadata has not already been initialized, callers should use CastMetadataZero.

func CastMetadataZero 

func CastMetadataZero[T any](md *Metadata) *T

CastMetadataZero casts the provided metadata to the type parameter T, zeroing the memory backing the metadata first. This zeroing is necessary when first initializing the data structure to ensure that the Go garbage collector doesn't misinterpret any of T's pointer fields, falsely detecting them as invalid pointers.

func ReadRaw 

func ReadRaw(
	ctx context.Context,
	f objstorage.Readable,
	readHandle objstorage.ReadHandle,
	logger base.LoggerAndTracer,
	fileNum base.DiskFileNum,
	buf []byte,
	off int64,
) ([]byte, error)

ReadRaw reads len(buf) bytes from the provided Readable at the given offset into buf. It's used to read the footer of a table.

func ValidateChecksum 

func ValidateChecksum(checksumType ChecksumType, b []byte, bh Handle) error

ValidateChecksum validates the checksum of a block.

Types

type AllocedBuffer 

type AllocedBuffer struct {
	// contains filtered or unexported fields
}

AllocedBuffer is an allocated memory buffer.

type Buf 

type Buf struct {
	// contains filtered or unexported fields
}

A Buf holds a reference to a manually-managed, pooled byte buffer.

func (*Buf) Release 

func (b *Buf) Release()

Release releases the buffer back to the pool.

func (Buf) Valid 

func (b Buf) Valid() bool

Valid returns true if the buf holds a valid buffer.

type BufferHandle 

type BufferHandle struct {
	// contains filtered or unexported fields
}

A BufferHandle is a handle to manually-managed memory. The handle may point to a block in the block cache (h.cv != nil), or a buffer that exists outside the block cache allocated from a BufferPool (b.Valid()).

func CacheBufferHandle 

func CacheBufferHandle(cv *cache.Value) BufferHandle

CacheBufferHandle constructs a BufferHandle from a block cache Handle.

func (BufferHandle) BlockData 

func (bh BufferHandle) BlockData() []byte

BlockData retrieves the buffer for the block data.

func (BufferHandle) BlockMetadata 

func (bh BufferHandle) BlockMetadata() *Metadata

BlockMetadata returns the buffer for the block metadata.

func (BufferHandle) Release 

func (bh BufferHandle) Release()

Release releases the buffer, either back to the block cache or BufferPool. It is okay to call Release on a zero-value BufferHandle (to no effect).

func (BufferHandle) Valid 

func (bh BufferHandle) Valid() bool

Valid returns true if the BufferHandle holds a value.

type BufferPool 

type BufferPool struct {
	// contains filtered or unexported fields
}

A BufferPool holds a pool of buffers for holding sstable blocks. An initial size of the pool is provided on Init, but a BufferPool will grow to meet the largest working set size. It'll never shrink. When a buffer is released, the BufferPool recycles the buffer for future allocations.

A BufferPool should only be used for short-lived allocations with well-understood working set sizes to avoid excessive memory consumption.

BufferPool is not thread-safe.

func (*BufferPool) Alloc 

func (p *BufferPool) Alloc(n int) Buf

Alloc allocates a new buffer of size n. If the pool already holds a buffer at least as large as n, the pooled buffer is used instead.

Alloc is O(MAX(N,M)) where N is the largest number of concurrently in-use buffers allocated and M is the initialSize passed to Init.

func (*BufferPool) Init 

func (p *BufferPool) Init(initialSize int, reason ReasonForBufferPool)

Init initializes the pool with an initial working set buffer size of `initialSize`.

func (*BufferPool) Reason 

func (p *BufferPool) Reason() ReasonForBufferPool

Reason returns the reason passed to Init.

func (*BufferPool) Release 

func (p *BufferPool) Release()

Release releases all buffers held by the pool and resets the pool to an uninitialized state.

type ByKind 

type ByKind[T any] struct {
	DataBlocks  T
	ValueBlocks T
	OtherBlocks T
}

ByKind stores three different instances of T, one for sstable data blocks, one for sstable and blob file value blocks, and one for all other blocks.

func (*ByKind[T]) ForKind 

func (b *ByKind[T]) ForKind(kind Kind) *T

type ByLevel 

type ByLevel[T any] struct {
	L5          T
	L6          T
	OtherLevels T
}

ByLevel stores three different instance of T, one for L5, one for L6, and one for all other levels.

func (*ByLevel[T]) ForLevel 

func (b *ByLevel[T]) ForLevel(level base.Level) *T

type Category 

type Category uint8

Category is a user-understandable string, where stats are aggregated for each category. The cardinality of this should be low, say < 20. The prefix "pebble-" is reserved for internal Pebble categories.

Examples of categories that can be useful in the CockroachDB context are: sql-user, sql-stats, raft, rangefeed, mvcc-gc, range-snapshot. 

const CategoryUnknown Category = 0

CategoryUnknown is the unknown category. It has the latency-sensitive QoS level.

func Categories 

func Categories() []Category

Categories returns all registered categories, including CategoryUnknown.

Can only be called after all categories have been registered. Calling RegisterCategory() after Categories() will result in a panic.

func RegisterCategory 

func RegisterCategory(name string, qosLevel QoSLevel) Category

RegisterCategory registers a new category. Each category has a name and an associated QoS level. The category name must be unique.

Only CategoryMax categories can be registered in total.

func StringToCategoryForTesting 

func StringToCategoryForTesting(s string) Category

StringToCategoryForTesting returns the Category for the string, or panics if the string is not known.

func (Category) QoSLevel 

func (c Category) QoSLevel() QoSLevel

QoSLevel returns the QoSLevel associated with this Category.

func (Category) SafeFormat 

func (c Category) SafeFormat(p redact.SafePrinter, verb rune)

SafeFormat implements the redact.SafeFormatter interface.

func (Category) String 

func (c Category) String() string

type CategoryStats 

type CategoryStats struct {
	// BlockBytes is the bytes in the loaded blocks. If the block was
	// compressed, this is the compressed bytes. Currently, only the index
	// blocks, data blocks containing points, and filter blocks are included.
	// Additionally, value blocks read after the corresponding iterator is
	// closed are not included.
	BlockBytes uint64
	// BlockBytesInCache is the subset of BlockBytes that were in the block
	// cache.
	BlockBytesInCache uint64
	// BlockReadDuration is the total duration to read the bytes not in the
	// cache, i.e., BlockBytes-BlockBytesInCache. When multiple concurrent
	// readers wait for each other, and only one does the read, this will
	// account for the total time spent waiting plus potentially reading for all
	// those readers, so it can over count. Such over counting should be rare.
	BlockReadDuration time.Duration
}

CategoryStats provides stats about a category of reads.

type CategoryStatsAggregate 

type CategoryStatsAggregate struct {
	Category      Category
	CategoryStats CategoryStats
}

CategoryStatsAggregate is the aggregate for the given category.

type CategoryStatsCollector 

type CategoryStatsCollector struct {
	// contains filtered or unexported fields
}

CategoryStatsCollector collects and aggregates the stats per category.

func (*CategoryStatsCollector) Accumulator 

func (c *CategoryStatsCollector) Accumulator(p uint64, category Category) *CategoryStatsShard

Accumulator returns a stats accumulator for the given category. The provided p is used to detrmine which shard to write stats to.

func (*CategoryStatsCollector) GetStats 

func (c *CategoryStatsCollector) GetStats() []CategoryStatsAggregate

GetStats returns the aggregated stats.

type CategoryStatsShard 

type CategoryStatsShard struct {
	// contains filtered or unexported fields
}

CategoryStatsShard holds CategoryStats with a mutex to ensure safe access.

func (*CategoryStatsShard) Accumulate 

func (c *CategoryStatsShard) Accumulate(
	blockBytes, blockBytesInCache uint64, blockReadDuration time.Duration,
)

Accumulate implements the IterStatsAccumulator interface.

type ChecksumType 

type ChecksumType byte

ChecksumType specifies the checksum used for blocks. 

const (
	ChecksumTypeNone     ChecksumType = 0
	ChecksumTypeCRC32c   ChecksumType = 1
	ChecksumTypeXXHash   ChecksumType = 2
	ChecksumTypeXXHash64 ChecksumType = 3
)

The available checksum types. These values are part of the durable format and should not be changed.

func (ChecksumType) String 

func (t ChecksumType) String() string

String implements fmt.Stringer.

type Checksummer 

type Checksummer struct {
	Type ChecksumType
	// contains filtered or unexported fields
}

A Checksummer calculates checksums for blocks.

func (*Checksummer) Checksum 

func (c *Checksummer) Checksum(block []byte, blockType byte) (checksum uint32)

Checksum computes a checksum over the provided block and block type.

func (*Checksummer) Init 

func (c *Checksummer) Init(typ ChecksumType)

type CompressionCounters 

type CompressionCounters struct {
	Compressed   ByLevel[ByKind[LogicalBytesCompressed]]
	Decompressed ByLevel[ByKind[LogicalBytesDecompressed]]
}

CompressionCounters holds running counters for the number of bytes compressed and decompressed. Counters are separated by L5 vs L6 vs other levels, and by data blocks vs value blocks vs other blocks.

The main purpose for these metrics is to allow estimating how overall CPU usage would change with a different compression algorithm (in conjunction with performance information about the algorithms, like that produced by the compression analyzer).

In all cases, the figures refer to the uncompressed ("logical") size; i.e. the *input* size for compression and the *output* size for decompression.

Every block is compressed, so every block written is counted.

func (*CompressionCounters) LoadCompressed 

func (c *CompressionCounters) LoadCompressed() ByLevel[ByKind[uint64]]

func (*CompressionCounters) LoadDecompressed 

func (c *CompressionCounters) LoadDecompressed() ByLevel[ByKind[uint64]]

type CompressionIndicator 

type CompressionIndicator byte

CompressionIndicator is the byte stored physically within the block.Trailer to indicate the compression type.

TODO(jackson): Avoid exporting once all compression and decompression is delegated to the block package. 

const (
	NoCompressionIndicator     CompressionIndicator = 0
	SnappyCompressionIndicator CompressionIndicator = 1
	ZlibCompressionIndicator   CompressionIndicator = 2
	Bzip2CompressionIndicator  CompressionIndicator = 3
	Lz4CompressionIndicator    CompressionIndicator = 4
	Lz4hcCompressionIndicator  CompressionIndicator = 5
	XpressCompressionIndicator CompressionIndicator = 6
	ZstdCompressionIndicator   CompressionIndicator = 7
	MinLZCompressionIndicator  CompressionIndicator = 8
)

The block type gives the per-block compression format. These constants are part of the file format and should not be changed. Every block is compressed with compression.BlockSetting. Only the indicator of the supported algorithm (LZ4) is accepted; blocks carrying any other indicator (including NoCompressionIndicator) are rejected as corrupt.

func CompressionIndicatorFromAlgorithm added in v0.2.0 

func CompressionIndicatorFromAlgorithm(algo compression.Algorithm) CompressionIndicator

CompressionIndicatorFromAlgorithm returns the indicator stored in the trailer of a block compressed with the given algorithm.

func (CompressionIndicator) Algorithm 

func (i CompressionIndicator) Algorithm() compression.Algorithm

Algorithm returns the compression algorithm for the indicator. Unsupported indicators (including NoCompressionIndicator) map to compression.Unknown, whose decompressor returns a corruption error rather than panicking.

func (CompressionIndicator) String 

func (i CompressionIndicator) String() string

String implements fmt.Stringer.

type CompressionStats 

type CompressionStats struct {
	// contains filtered or unexported fields
}

CompressionStats collects compression statistics (either for a single file or for a collection of files).

Compression statistics consist of the total compressed and uncompressed sizes for each distinct compression.Setting.

func ParseCompressionStats 

func ParseCompressionStats(s string) (CompressionStats, error)

ParseCompressionStats parses the output of CompressionStats.String back into CompressionStats.

If the string contains statistics for unknown compression settings, these are accumulated under a special "unknown" setting.

func (*CompressionStats) Add 

func (c *CompressionStats) Add(other *CompressionStats)

Add updates the receiver stats to include the other stats.

func (*CompressionStats) All 

func (c *CompressionStats) All() iter.Seq2[compression.Setting, CompressionStatsForSetting]

All returns an iterator over the collected stats, in arbitrary order.

func (*CompressionStats) Clone 

func (c *CompressionStats) Clone() CompressionStats

Clone returns a copy of the CompressionStats.

func (*CompressionStats) IsEmpty 

func (c *CompressionStats) IsEmpty() bool

func (CompressionStats) ReadPinnedMemory added in v0.2.5 

func (cs CompressionStats) ReadPinnedMemory() uint64

ReadPinnedMemory counts the current immutable compression-stat map backing.

func (*CompressionStats) Reset 

func (c *CompressionStats) Reset()

func (*CompressionStats) Scale 

func (c *CompressionStats) Scale(size uint64, backingSize uint64) CompressionStats

Scale the stats by (size/backingSize). Used to obtain an approximation of the stats for a virtual table.

TODO(radu): the utility of this is questionable; we should only be reporting the statistics for the backing.

func (CompressionStats) String 

func (c CompressionStats) String() string

String returns a string representation of the stats, in the format: "<setting1>:<compressed1>/<uncompressed1>,<setting2>:<compressed2>/<uncompressed2>,..."

The settings are ordered according to the algorithm definition ordering (and by increasing level).

type CompressionStatsForSetting 

type CompressionStatsForSetting struct {
	CompressedBytes   uint64
	UncompressedBytes uint64
}

func (*CompressionStatsForSetting) Add 

func (cs *CompressionStatsForSetting) Add(other CompressionStatsForSetting)

func (CompressionStatsForSetting) CompressionRatio 

func (cs CompressionStatsForSetting) CompressionRatio() float64

CompressionRatio returns the compression ratio for the setting. Returns 0 if the stats are empty.

type Compressor 

type Compressor struct {
	// contains filtered or unexported fields
}

Compressor is used to compress blocks. Every block of every kind is compressed with compression.BlockSetting, unconditionally: there is no minimum reduction below which a block is stored uncompressed, even when compression grows the block. Typical usage: 

c := MakeCompressor()
.. = c.Compress(..)
.. = c.Compress(..)
c.Close()

func MakeCompressor 

func MakeCompressor() Compressor

MakeCompressor returns a Compressor for compression.BlockSetting. Close must be called when the compressor is no longer needed.

func (*Compressor) Close 

func (c *Compressor) Close()

Close must be called when the Compressor is no longer needed. After Close is called, the Compressor must not be used again.

func (*Compressor) Compress 

func (c *Compressor) Compress(dst, src []byte, kind Kind) (CompressionIndicator, []byte)

Compress a block, appending the compressed data to dst[:0].

In addition to the buffer, returns the compression indicator for the algorithm used.

func (*Compressor) InputBytes 

func (c *Compressor) InputBytes() iter.Seq2[Kind, uint64]

InputBytes returns an iterator over the total number of input bytes passed through the compressor, by block kind.

func (*Compressor) Stats 

func (c *Compressor) Stats() *CompressionStats

Stats returns the compression stats. The result can only be used until the next call to the Compressor.

type Decompressor 

type Decompressor = compression.Decompressor

func GetDecompressor 

func GetDecompressor(c CompressionIndicator) Decompressor

GetDecompressor returns the decompressor for a block's compression indicator. An unsupported indicator yields a decompressor that fails with a corruption error.

type FlushGovernor 

type FlushGovernor struct {
	// contains filtered or unexported fields
}

FlushGovernor is used to decide when to flush a block. It takes into consideration a target block size and (optionally) allocation size classes.

When allocation size classes are used, we use the allocation class that is closest to the target block size. We also take into account the next allocation class and use it if it reduces internal fragmentation.

func MakeFlushGovernor 

func MakeFlushGovernor(
	targetBlockSize int,
	blockSizeThreshold int,
	sizeClassAwareThreshold int,
	allocatorSizeClasses []int,
) FlushGovernor

MakeFlushGovernor initializes a flush controller.

There are two cases:

1. No allocation classes. If we don't have any allocatorSizeClasses, or targetBlockSize doesn't fit between two allocation classes, then we flush right before the block would exceed targetBlockSize (except if the block size would be smaller than blockSizeThreshold percent of the target, in which case we flush right after the target block size is exceeded).

2. With allocation classes. We take into account allocation size classes no smaller than sizeClassAwareThreshold percent of the target block size and up to the first class that fits the target block size. We flush near allocation class boundaries to minimize wasted memory space in the block cache (internal fragmentation).

The FlushGovernor is immutable and can be copied by value.

func (*FlushGovernor) LowWatermark 

func (fg *FlushGovernor) LowWatermark() int

LowWatermark returns the minimum size of a block that could be flushed. ShouldFlush will never return true if sizeBefore is below the low watermark.

This can be used in a "fast path" check that uses an easy-to-compute overestimation of the block size.

func (*FlushGovernor) ShouldFlush 

func (fg *FlushGovernor) ShouldFlush(sizeBefore int, sizeAfter int) bool

ShouldFlush returns true if we should flush the current block of sizeBefore instead of adding another KV that would increase the block to sizeAfter.

func (FlushGovernor) String 

func (fg FlushGovernor) String() string

type GetInternalValueForPrefixAndValueHandler 

type GetInternalValueForPrefixAndValueHandler interface {
	// GetInternalValueForPrefixAndValueHandle returns a InternalValue for the
	// given value prefix and value.
	//
	// The result is only valid until the next call to
	// GetInternalValueForPrefixAndValueHandle. Use InternalValue.Clone if the
	// lifetime of the InternalValue needs to be extended. For more details, see
	// the "memory management" comment where LazyValue is declared.
	GetInternalValueForPrefixAndValueHandle(handle []byte) base.InternalValue
}

GetInternalValueForPrefixAndValueHandler is an interface for getting an InternalValue from a value prefix and value.

type Handle 

type Handle struct {
	// Offset identifies the offset of the block within the file.
	Offset uint64
	// Length is the length of the block data (excludes the trailer).
	Length uint64
}

Handle is the file offset and length of a block.

func DecodeHandle 

func DecodeHandle(src []byte) (Handle, int)

DecodeHandle returns the block handle encoded in a variable-width encoding at the start of src, as well as the number of bytes it occupies. It returns zero if given invalid input. A block handle for a data block or a first/lower level index block should not be decoded using DecodeHandle since the caller may validate that the number of bytes decoded is equal to the length of src, which will be false if the properties are not decoded. In those cases the caller should use DecodeHandleWithProperties.

func (Handle) EncodeVarints 

func (h Handle) EncodeVarints(dst []byte) int

EncodeVarints encodes the block handle into dst using a variable-width encoding and returns the number of bytes written.

func (Handle) String 

func (h Handle) String() string

String implements fmt.Stringer.

type HandleWithProperties 

type HandleWithProperties struct {
	Handle
	Props []byte
}

HandleWithProperties is used for data blocks and first/lower level index blocks, since they can be annotated using BlockPropertyCollectors.

func DecodeHandleWithProperties 

func DecodeHandleWithProperties(src []byte) (HandleWithProperties, error)

DecodeHandleWithProperties returns the block handle and properties encoded in a variable-width encoding at the start of src. src needs to be exactly the length that was encoded. This method must be used for data block and first/lower level index blocks. The properties in the block handle point to the bytes in src.

func (HandleWithProperties) EncodeVarints 

func (h HandleWithProperties) EncodeVarints(dst []byte) []byte

EncodeVarints encodes the block handle and properties into dst using a variable-width encoding and returns the number of bytes written.

type InitFileReadStats 

type InitFileReadStats struct {
	// Stats and IterStats are slightly different. Stats is a shared struct
	// supplied from the outside, and represents stats for the whole iterator
	// tree and can be reset from the outside (e.g. when the pebble.Iterator is
	// being reused). It is currently only provided when the iterator tree is
	// rooted at pebble.Iterator. IterStats contains an sstable iterator's
	// private stats that are reported to a CategoryStatsCollector when this
	// iterator is closed. In the important code paths, the
	// CategoryStatsCollector is managed by the fileCacheContainer.
	//
	// Both can be nil.
	Stats     *base.InternalIteratorStats
	IterStats *CategoryStatsShard
}

InitFileReadStats is used to capture file reading stats when a file is initialized in the file cache. The primary use case is when such initialization is triggered because an iterator needs to read that file, and we are able to capture these initialization stats in the iterator stats.

type Kind 

type Kind = blockkind.Kind

Kind is a convenience alias.

type LogicalBytesCompressed 

type LogicalBytesCompressed struct {
	atomic.Uint64
}

LogicalBytesCompressed keeps a count of the logical bytes that were compressed.

type LogicalBytesDecompressed 

type LogicalBytesDecompressed struct {
	atomic.Uint64
}

LogicalBytesDecompressed keeps a count of the logical bytes that were decompressed.

type Metadata 

type Metadata [MetadataSize]byte

Metadata is an in-memory buffer that stores metadata for a block. It is allocated together with the buffer storing the block and is initialized once when the block is read from disk.

Portions of this buffer can be cast to the structures we need (through CastMetadata[Zero]), but note that any pointers in these structures should be considered invisible to the GC for the purpose of preserving lifetime. Pointers to the block's data buffer are ok, since the metadata and the data have the same lifetime (sharing the underlying allocation).

type OwnedPhysicalBlock 

type OwnedPhysicalBlock struct {
	// contains filtered or unexported fields
}

OwnedPhysicalBlock is a wrapper around PhysicalBlock which indicates that whoever is holding it is responsible for releasing the buffer.

func (OwnedPhysicalBlock) Length 

func (b OwnedPhysicalBlock) Length() PhysicalBlockLength

Length returns the length of the data block.

type PhysicalBlock 

type PhysicalBlock struct {
	// contains filtered or unexported fields
}

PhysicalBlock is a block (possibly compressed) as it will be stored physically on disk (including its trailer).

A PhysicalBlock is always backed by a TempBuffer and should be released after it is written out.

func AlreadyEncodedPhysicalBlock 

func AlreadyEncodedPhysicalBlock(dataWithTrailer []byte) PhysicalBlock

AlreadyEncodedPhysicalBlock creates a PhysicalBlock from the provided on-disk data; used when copying existing blocks. The given slice is copied into the physical block.

func (*PhysicalBlock) AppendAndRelease added in v0.2.2 

func (b *PhysicalBlock) AppendAndRelease(dst []byte) []byte

AppendAndRelease appends the block's on-disk data (including its trailer) to dst and releases the underlying TempBuffer. Like CopyAndRelease, it is used when many blocks must be held at once.

func (*PhysicalBlock) CopyAndRelease 

func (b *PhysicalBlock) CopyAndRelease(alloc bytealloc.A) (bytealloc.A, []byte)

CopyAndRelease copies the block's on-disk data (including its trailer) into memory obtained from the given allocator and releases the underlying TempBuffer. It is used when many blocks must be held at once: a pooled TempBuffer has a large capacity, so holding one per block would pin far more memory than the blocks need.

func (PhysicalBlock) Length 

func (b PhysicalBlock) Length() PhysicalBlockLength

Length returns the length of the data block.

func (PhysicalBlock) LengthWithoutTrailer 

func (b PhysicalBlock) LengthWithoutTrailer() int

LengthWithoutTrailer returns the length of the data block, excluding the trailer.

func (*PhysicalBlock) Release 

func (b *PhysicalBlock) Release()

Release the underlying TempBuffer. The PhysicalBlock should not be used again.

func (*PhysicalBlock) Take 

func (b *PhysicalBlock) Take() OwnedPhysicalBlock

Take moves the physical block to an OwnedPhysicalBlock. The receiver must not be used again. This is used to make the ownership transfer more explicit.

type PhysicalBlockLength 

type PhysicalBlockLength struct {
	// contains filtered or unexported fields
}

PhysicalBlockLength represents the length of a physical block.

func WriteAndReleasePhysicalBlock 

func WriteAndReleasePhysicalBlock(
	b OwnedPhysicalBlock, w objstorage.Writable,
) (length PhysicalBlockLength, err error)

WriteAndReleasePhysicalBlock writes the block (including its trailer) to the provided Writable and releases the block. The block is released even in error cases.

func (PhysicalBlockLength) WithTrailer 

func (l PhysicalBlockLength) WithTrailer() int

func (PhysicalBlockLength) WithoutTrailer 

func (l PhysicalBlockLength) WithoutTrailer() int

type PhysicalBlockMaker 

type PhysicalBlockMaker struct {
	Compressor          Compressor
	Checksummer         Checksummer
	CompressionCounters *ByKind[LogicalBytesCompressed]
}

PhysicalBlockMaker is used to create physical blocks from logical block data. It takes care of compression, checksum calculation, and trailer encoding.

It is not thread-safe and should not be used concurrently.

func (*PhysicalBlockMaker) Close 

func (p *PhysicalBlockMaker) Close()

Close must be called when the PhysicalBlockMaker is no longer needed. After Close is called, the PhysicalBlockMaker must not be used again (unless it is initialized again).

func (*PhysicalBlockMaker) Init 

func (p *PhysicalBlockMaker) Init(
	checksumType ChecksumType, compressionCounters *ByKind[LogicalBytesCompressed],
)

Init the physical block maker. Closed must be called when no longer needed.

func (*PhysicalBlockMaker) Make 

func (p *PhysicalBlockMaker) Make(uncompressedData []byte, blockKind Kind) PhysicalBlock

Make creates a PhysicalBlock from the provided uncompressed data. The block is always compressed (see Compressor). The PhysicalBlock should be released to save allocations, at least in the "normal" paths.

The uncompressedData slice is never used directly in the PhysicalBlock.

type QoSLevel 

type QoSLevel uint8

QoSLevel describes whether the read is latency-sensitive or not. Each category must map to a single QoSLevel. While category strings are opaque to Pebble, the QoSLevel may be internally utilized in Pebble to better optimize future reads. 

const (
	// LatencySensitiveQoSLevel is the default when QoSLevel is not specified,
	// and represents reads that are latency-sensitive.
	LatencySensitiveQoSLevel QoSLevel = iota
	// NonLatencySensitiveQoSLevel represents reads that are not
	// latency-sensitive.
	NonLatencySensitiveQoSLevel
)

func StringToQoSForTesting 

func StringToQoSForTesting(s string) QoSLevel

StringToQoSForTesting returns the QoSLevel for the string, or panics if the string is not known.

func (QoSLevel) SafeFormat 

func (q QoSLevel) SafeFormat(p redact.SafePrinter, verb rune)

SafeFormat implements the redact.SafeFormatter interface.

type ReadEnv 

type ReadEnv struct {
	// stats and iterStats are slightly different. stats is a shared struct
	// supplied from the outside, and represents stats for the whole iterator
	// tree and can be reset from the outside (e.g. when the pebble.Iterator is
	// being reused). It is currently only provided when the iterator tree is
	// rooted at pebble.Iterator. iterStats contains an sstable iterator's
	// private stats that are reported to a CategoryStatsCollector when this
	// iterator is closed. In the important code paths, the CategoryStatsCollector
	// is managed by the fileCacheContainer.
	Stats     *base.InternalIteratorStats
	IterStats *CategoryStatsShard

	// BufferPool is not-nil if we read blocks into a buffer pool and not into the
	// cache. This is used during compactions and by the level checker.
	BufferPool *BufferPool
	// Level is the LSM level associated with the operation, when the operation
	// applies to a (possibly virtual) sstable. It is used when interacting with
	// the block cache.
	Level base.Level

	// ExcludeFromCompressionCounters, if set, excludes the decompression done
	// by this read from the reader's CompressionCounters. Debugging reads (the
	// level checker) set it so that they don't skew the counters, which are
	// meant to reflect the workload.
	ExcludeFromCompressionCounters bool

	// ReportCorruptionFn is called with ReportCorruptionArg and the error
	// whenever an SSTable corruption is detected. The argument is used to avoid
	// allocating a separate function for each object. It returns an error with
	// more details.
	ReportCorruptionFn  func(base.ObjectInfo, error) error
	ReportCorruptionArg base.ObjectInfo

	// ValueRetrievalProfile, if non-nil, is used to record stack traces of
	// retrievals of separated values. It's only used by the blob.ValueFetcher.
	ValueRetrievalProfile *bytesprofile.Profile
}

ReadEnv contains arguments used when reading a block which apply to all the block reads performed by a higher-level operation.

ReadEnv can be copied by value; any copy will use the same statistics and buffer pool.

func (*ReadEnv) BlockRead 

func (env *ReadEnv) BlockRead(kind Kind, blockLength uint64, readDuration time.Duration)

BlockRead updates the stats when a block had to be read.

func (*ReadEnv) BlockServedFromCache 

func (env *ReadEnv) BlockServedFromCache(kind Kind, blockLength uint64)

BlockServedFromCache updates the stats when a block was found in the cache.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads blocks from a single file, handling caching, checksum validation and decompression.

func (*Reader) ChecksumType 

func (r *Reader) ChecksumType() ChecksumType

ChecksumType returns the checksum type used by the reader.

func (*Reader) Close 

func (r *Reader) Close() error

Close releases resources associated with the Reader.

func (*Reader) FileNum 

func (r *Reader) FileNum() base.DiskFileNum

FileNum returns the file number of the file being read.

func (*Reader) GetFromCache 

func (r *Reader) GetFromCache(bh Handle, level base.Level) *cache.Value

GetFromCache retrieves the block from the cache, if it is present. It does not mark the block as recently used.

Users should prefer using Read, which handles reading from object storage on a cache miss.

It returns nil if the reader has no block cache.

func (*Reader) Init 

func (r *Reader) Init(readable objstorage.Readable, ro ReaderOptions, checksumType ChecksumType)

Init initializes the Reader to read blocks from the provided Readable.

func (*Reader) Read 

func (r *Reader) Read(
	ctx context.Context,
	env ReadEnv,
	readHandle objstorage.ReadHandle,
	bh Handle,
	kind Kind,
	initBlockMetadataFn func(*Metadata, []byte) error,
) (handle BufferHandle, _ error)

Read reads the block referenced by the provided handle. The readHandle is optional.

func (*Reader) Readable 

func (r *Reader) Readable() objstorage.Readable

Readable returns the underlying objstorage.Readable.

Users should avoid accessing the underlying Readable if it can be avoided.

func (*Reader) UsePreallocatedReadHandle 

func (r *Reader) UsePreallocatedReadHandle(
	readBeforeSize objstorage.ReadBeforeSize, rh *objstorageprovider.PreallocatedReadHandle,
) objstorage.ReadHandle

UsePreallocatedReadHandle returns a ReadHandle that reads from the reader and uses the provided preallocated read handle to back the read handle, avoiding an unnecessary allocation.

type ReaderOptions 

type ReaderOptions struct {
	// CacheOpts contains the information needed to interact with the block
	// cache.
	CacheOpts sstableinternal.CacheOptions
	// LoadBlockSema, if set, is used to limit the number of blocks that can be
	// loaded (i.e. read from the filesystem) in parallel. Each load acquires
	// one unit from the semaphore for the duration of the read.
	LoadBlockSema *fifo.Semaphore
	// LoggerAndTracer is an optional logger and tracer.
	LoggerAndTracer base.LoggerAndTracer

	// CompressionCounters, if non-nil, is used to keep track of how much data was
	// decompressed.
	CompressionCounters *ByLevel[ByKind[LogicalBytesDecompressed]]
}

ReaderOptions configures a block reader.

type ReasonForBufferPool 

type ReasonForBufferPool int8

ReasonForBufferPool identifies the use case for th BufferPool. These are exceptional use cases, as using a buffer pool does not populate the block cache.

The reason determines how the cache hit/miss metrics are updated. 

const (
	ForCompaction ReasonForBufferPool = iota
	ForBlobFileRewrite
	ForSSTableMetadata
	ForBlobFileMetadata
	ForExternalIter
	ForLevelChecking
)

type TempBuffer 

type TempBuffer struct {
	// contains filtered or unexported fields
}

TempBuffer is a buffer that is used temporarily and is released back to a pool for reuse.

func NewTempBuffer 

func NewTempBuffer() *TempBuffer

NewTempBuffer returns a TempBuffer from the pool. The buffer will have zero size and length and arbitrary capacity.

func (*TempBuffer) Append 

func (tb *TempBuffer) Append(v []byte) (startOffset int)

Append appends the contents of v to the buffer, growing the buffer if necessary. Returns the offset at which it was appended.

func (*TempBuffer) Data 

func (tb *TempBuffer) Data() []byte

Data returns the byte slice currently backing the Buffer.

func (*TempBuffer) Release 

func (tb *TempBuffer) Release()

Release releases the buffer back to the pool for reuse.

func (*TempBuffer) Reset 

func (tb *TempBuffer) Reset()

Reset is equivalent to Resize(0).

func (*TempBuffer) Resize 

func (tb *TempBuffer) Resize(length int)

Resize resizes the buffer to the specified length, allocating if necessary. If the length is longer than the current length, the values of the new bytes are arbitrary.

func (*TempBuffer) Size 

func (tb *TempBuffer) Size() int

Size returns the current size of the buffer.

type Trailer 

type Trailer [TrailerLen]byte

Trailer is the trailer at the end of a block, encoding the block type (compression) and a checksum.

func MakeTrailer 

func MakeTrailer(blockType byte, checksum uint32) (t Trailer)

MakeTrailer constructs a trailer from a block type and a checksum.

type Value 

type Value struct {
	// contains filtered or unexported fields
}

Value is a block buffer, either backed by the block cache or a BufferPool.

func Alloc 

func Alloc(n int, p *BufferPool) Value

Alloc allocates a new Value for a block of length n (excluding the block trailer, but including an associated Metadata). If bufferPool is non-nil, Alloc allocates the buffer from the pool. Otherwise it allocates it from the block cache.

func (Value) BlockData 

func (b Value) BlockData() []byte

BlockData returns the byte slice for the block data.

func (Value) BlockMetadata 

func (b Value) BlockMetadata() *Metadata

BlockMetadata returns the block metadata.

func (Value) MakeHandle 

func (b Value) MakeHandle() BufferHandle

MakeHandle constructs a BufferHandle from the Value.

func (Value) Release 

func (b Value) Release()

Release releases the handle.

func (*Value) SetInCacheForTesting 

func (b *Value) SetInCacheForTesting(h *cache.Handle, fileNum base.DiskFileNum, offset uint64)

func (Value) Truncate 

func (b Value) Truncate(n int)

Truncate truncates the block to n bytes.

type ValuePrefix 

type ValuePrefix byte

ValuePrefix is the single byte prefix in values indicating either an in-place value or a value encoding a valueHandle. It encodes multiple kinds of information (see below).

func BlobValueHandlePrefix 

func BlobValueHandlePrefix(setHasSameKeyPrefix bool, attr base.ShortAttribute) ValuePrefix

BlobValueHandlePrefix returns the ValuePrefix for a blob.

func InPlaceValuePrefix 

func InPlaceValuePrefix(setHasSameKeyPrefix bool) ValuePrefix

InPlaceValuePrefix returns the ValuePrefix for an in-place value.

func ValueBlockHandlePrefix 

func ValueBlockHandlePrefix(setHasSameKeyPrefix bool, attribute base.ShortAttribute) ValuePrefix

ValueBlockHandlePrefix returns the ValuePrefix for a valblk.Handle.

func (ValuePrefix) IsBlobValueHandle 

func (vp ValuePrefix) IsBlobValueHandle() bool

IsBlobValueHandle returns true if the ValuePrefix is for a blob.

func (ValuePrefix) IsInPlaceValue 

func (vp ValuePrefix) IsInPlaceValue() bool

IsInPlaceValue returns true if the ValuePrefix is for an in-place value.

func (ValuePrefix) IsValueBlockHandle 

func (vp ValuePrefix) IsValueBlockHandle() bool

IsValueBlockHandle returns true if the ValuePrefix is for a valblk.Handle.

func (ValuePrefix) SetHasSamePrefix 

func (vp ValuePrefix) SetHasSamePrefix() bool

SetHasSamePrefix returns true if the ValuePrefix encodes that the key is a set with the same prefix as the preceding key which also is a set.

func (ValuePrefix) ShortAttribute 

func (vp ValuePrefix) ShortAttribute() base.ShortAttribute

ShortAttribute returns the user-defined base.ShortAttribute encoded in the ValuePrefix.

REQUIRES: !IsInPlaceValue()

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.
⌂ 换网址