Documentation
¶
Overview ¶
Package format defines the shared vocabulary of the on-disk index format: the term iterator contract, postings bitmaps, per-document metadata, and the postings encodings.
Both the version shim and the versioned format implementations import this package, which keeps them free of import cycles.
Index ¶
Constants ¶
const MaxQueryRequestBytes = 8 * 1024 * 1024
MaxQueryRequestBytes enforces the object-storage request contract.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bitmap ¶
Bitmap wraps a roaring bitmap with an optional MatchesAll sentinel. MatchesAll indicates a density-filtered term that matches all documents. When MatchesAll is true, Roaring is nil.
func (Bitmap) And ¶
And intersects this bitmap with another. Sentinel (MatchesAll) bitmaps are identity elements: AND(all, x) = x, AND(x, all) = x, AND(all, all) = all.
type DocumentMetadata ¶
type DocumentMetadata struct {
ID uint32
MinTimeUnix int64 // milliseconds since Unix epoch
MaxTimeUnix int64 // milliseconds since Unix epoch
}
DocumentMetadata contains metadata about a document for storage in the index header. Timestamps are stored as milliseconds since Unix epoch for millisecond precision.
type HeaderInfo ¶
type HeaderInfo struct {
Version uint32 `json:"version"`
Flags uint32 `json:"flags"`
DocumentCount uint32 `json:"document_count"`
TermBlockCount uint32 `json:"term_block_count"`
PostingsBlockCount uint32 `json:"postings_block_count"`
PostingsCompression uint32 `json:"postings_compression"`
TermCount uint64 `json:"term_count"`
PostingsDataSize uint64 `json:"postings_data_size"`
TermDataSize uint64 `json:"term_data_size"`
DocMetadataSize uint64 `json:"doc_metadata_size"`
TermBlockDirSize uint64 `json:"term_block_dir_size"`
PostingsBlockDirSize uint64 `json:"postings_block_dir_size"`
}
HeaderInfo is the meta.json index_header schema: a summary of the on-disk index file header. It is not the wire type; convert at the IndexMeta boundary.
type PostingsEncoding ¶
type PostingsEncoding uint32
PostingsEncoding selects how per-term postings are encoded in the postings section.
const ( // PostingsEncodingBIDX stores one bitmap per term using BitmapIndexV2. PostingsEncodingBIDX PostingsEncoding = iota // PostingsEncodingFastUint32Blocked stores docIDs as blocked []uint32 payloads. PostingsEncodingFastUint32Blocked // PostingsEncodingFastRoaringBlocked stores blocked serialized roaring payloads. PostingsEncodingFastRoaringBlocked // PostingsEncodingBIDXPacked stores N terms per packed BIDX bitmap. PostingsEncodingBIDXPacked // PostingsEncodingFastDeltaVarIntBlocked stores delta+varint encoded docIDs in blocks. PostingsEncodingFastDeltaVarIntBlocked // PostingsEncodingFastEliasFanoBlocked stores Elias-Fano encoded docIDs in // blocks. Blocks are written uncompressed, each followed by an xxh3-64 // checksum. PostingsEncodingFastEliasFanoBlocked )
func ParseFlags ¶
func ParseFlags(flags uint32) (PostingsEncoding, uint8)
ParseFlags extracts the postings encoding and packing factor from header flags.
type QueryMultipleResult ¶
type QueryMultipleResult struct {
Documents []DocumentMetadata
TermBatchesProcessed int
Reason QueryMultipleTerminationReason
}
QueryMultipleResult is the result of a multi-term query against an index.
type QueryMultipleTerminationReason ¶
type QueryMultipleTerminationReason uint8
QueryMultipleTerminationReason indicates why QueryMultiple stopped.
const ( QueryMultipleReasonComplete QueryMultipleTerminationReason = iota QueryMultipleReasonTermMiss QueryMultipleReasonEmptyAnd )
type ReadSection ¶
type ReadSection uint8
ReadSection classifies a byte range within an index file.
const ( ReadSectionUnknown ReadSection = iota ReadSectionHeader // header (v1) or footer (v2) ReadSectionPostings // postings data (bitmaps) ReadSectionTermDict // term dictionary ReadSectionMetadata // doc metadata + block directories )
type TermBitmapPostings ¶
TermBitmapPostings represents a term and its roaring bitmap. Not an on-disk type, but shared between the accumulator (pkg/logline) and the encoder (v1) as a handoff contract.
type TermIterator ¶
TermIterator iterates over all terms in an index.
type WriterConfig ¶
type WriterConfig struct {
// Encoding overrides the postings encoding. Zero value uses the format default.
Encoding PostingsEncoding
// DensityThreshold overrides the density threshold.
// 0 uses the format default; positive values enable at that threshold;
// negative values disable filtering.
// The threshold is applied against the total documents in a full day
// (24h / DocumentInterval), not the current index's document count.
DensityThreshold float32
// DocumentInterval is the time range each document ID covers. Used with
// DensityThreshold to compute the day-based sentinel cutoff:
// sentinelCutoff = (24h / DocumentInterval) * DensityThreshold.
// Zero disables the density filter.
DocumentInterval time.Duration
}
WriterConfig allows overriding per-writer settings beyond the format defaults. A nil *WriterConfig means use version defaults for all settings. Zero-valued fields use the version default for that field.
Source Files