storage

package
v1.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: AGPL-3.0 Imports: 34 Imported by: 0

Documentation

Overview

Package storage defines StorageAPI, the contract for accessing a single drive: metadata files (xl.meta), data parts and volume (bucket) directories. Implementations are the local xl storage (NewLocal) and the RPC client to a remote node (M6). The Formatter service manages the format.json documents that bind the drives of a pool together.

Mocks are generated into ./mock by the repository-wide mockery configuration (.mockery.yaml, run through go generate in internal/service).

Index

Constants

View Source
const (
	// MetaLogDir is the directory of the metadata log in the system volume.
	MetaLogDir = "metalog"

	// DefaultMetaLogSegmentSize is the default size of a segment.
	DefaultMetaLogSegmentSize = 64 << 20
	// MinMetaLogSegmentSize and MaxMetaLogSegmentSize bound the segment
	// size (offsets in a segment are 32-bit).
	MinMetaLogSegmentSize = 1 << 20
	MaxMetaLogSegmentSize = 1 << 30
)

The metadata log of a drive keeps the xl.meta documents of its user volumes in append-only segments instead of one file per object (docs/proposals/small-object-log.md). A writer goroutine owns the active segment: the records waiting when it starts a round are written at once and flushed together (group commit), so concurrent writes share one flush. An in-memory index per volume (a B-tree ordered by key) locates the latest record of every object; it is rebuilt at open by replaying the segments. Overwritten and deleted records are garbage that the compactor reclaims by copying the live records of a mostly dead segment forward, then removing it.

Record layout (little endian):

0  magic   "NXL1"
4  crc32c  over bytes [8, total)
8  total   uint32, the record length
12 type    put, delete or drop (the whole volume)
13 -       reserved
14 volLen  uint16
16 keyLen  uint16
18 -       reserved
20 seq     uint64, the sequence number of the change
28 volume, key, payload (the encoded xl.meta of a put)

A record keeps its sequence number when the compactor copies it, so replay picks, for every key, the record with the highest sequence, and a delete or drop hides the older records whatever segment they are in.

View Source
const (
	// FormatFile is the format.json document, relative to the system volume.
	FormatFile = "format.json"
	// TmpDir holds in-progress writes; it is emptied at startup.
	TmpDir = "tmp"
	// MultipartDir holds in-progress multipart uploads.
	MultipartDir = "multipart"
	// BucketsDir holds the per-bucket metadata documents.
	BucketsDir = "buckets"
	// ConfigDir holds the cluster configuration and the IAM documents.
	ConfigDir = "config"
	// HealingFile is the healing tracker of the drive (HealingTracker
	// encoded as JSON). Its presence marks a drive being healed.
	HealingFile = "healing.bin"
	// QuarantineDir keeps the xl.meta documents found undecodable by a
	// write (see StorageAPI.WriteMetadata): the object starts over on the
	// drive and the copy is kept for inspection; nothing removes it.
	QuarantineDir = "quarantine"
	// MetaFile is the per-object metadata document.
	MetaFile = "xl.meta"
)

Well-known paths inside the system volume (domain.SystemBucket).

View Source
const (
	CheckPartUnknown = iota
	CheckPartSuccess
	CheckPartFileNotFound
	CheckPartFileCorrupt
	CheckPartVolumeNotFound
)

CheckParts result codes.

View Source
const DefaultMetaLogMemoryPercent = 25

DefaultMetaLogMemoryPercent is the default share of the RAM the index of a drive may use.

View Source
const SmallFileThreshold = 128 << 10

SmallFileThreshold is the size below which the object layer keeps the content of a version inline in xl.meta instead of a data directory.

Variables

View Source
var ErrFileCorrupt = fmt.Errorf("%w: file corrupt", domain.ErrFaultyDisk)

ErrFileCorrupt reports a file whose content does not match what its metadata promises (truncated part, bad xl.meta checksum). It wraps domain.ErrFaultyDisk: the drive copy is unusable and must be healed.

Functions

func ObjectDataDir

func ObjectDataDir(key, dataDir string) string

ObjectDataDir returns the path, relative to the volume, of the data directory of an object version (see ObjectDataPath).

func ObjectDataPath

func ObjectDataPath(key, dataDir string, part int) string

ObjectDataPath returns the path, relative to the volume, of a part file of an object version: "<key>/<dataDir>/part.N". Directory objects ("dir/") keep their data under the suffixed directory used for their metadata.

Types

type CheckPartsResult

type CheckPartsResult struct {
	// Results has one entry per part: one of the CheckPart* constants.
	Results []int
}

CheckPartsResult reports the state of each part.

type ChecksumInfo

type ChecksumInfo struct {
	PartNumber int
	Algorithm  string
	Hash       []byte
}

ChecksumInfo is the bitrot checksum of one part.

type DeleteOptions

type DeleteOptions struct {
	Recursive bool
	Immediate bool
	// UndoWrite rolls back a partially written version.
	UndoWrite bool
	// OldDataDir is the data directory to remove when replacing a version.
	OldDataDir string
	// KeepData leaves in place the data directory of the version that
	// DeleteVersion removes (or that a delete marker replaces): the caller
	// purges it once the delete reached its quorum, or restores the version
	// with WriteMetadata when it did not.
	KeepData bool
}

DeleteOptions tunes DeleteVersion and Delete.

type Deps

type Deps struct {
	Clock  clock.Clock
	Logger *slog.Logger
}

Deps are the dependencies of the storage implementations. Every field is optional: nil values are replaced by defaults.

type DiskInfoOptions

type DiskInfoOptions struct {
	// DiskID, when set, must match the drive's id or ErrDiskNotFound is returned.
	DiskID string
	// Metrics requests per-API latency metrics in the result.
	Metrics bool
	// NoOp skips the filesystem statfs call and returns cached values.
	NoOp bool
}

DiskInfoOptions tunes DiskInfo.

type ErasureInfo

type ErasureInfo struct {
	Algorithm    string
	DataBlocks   int
	ParityBlocks int
	BlockSize    int64
	// Index is the 1-based shard index of this drive.
	Index int
	// Distribution is the shard distribution across drives.
	Distribution []int
	// Checksums are the bitrot checksums of each part.
	Checksums []ChecksumInfo
}

ErasureInfo is the erasure layout of a version.

type FileHandle

type FileHandle interface {
	// Size returns the size of the file when it was opened. Pure accessor.
	Size() int64
	// ReadStream returns a reader over length bytes of the file starting
	// at offset (length -1 reads to the end), with the bounds checks of
	// StorageAPI.ReadFileStream. Closing the stream leaves the handle
	// open; several streams may be open at once.
	ReadStream(ctx context.Context, offset, length int64) (io.ReadCloser, error)
	// Close releases the handle: the file is closed once its last stream
	// is closed. Closing twice is a no-op.
	Close(ctx context.Context) error
}

FileHandle is a data file opened by StorageAPI.OpenFile.

type FileInfo

type FileInfo struct {
	Volume string
	Name   string
	// VersionID is the version UUID, empty for unversioned objects.
	VersionID string
	IsLatest  bool
	// Deleted marks a delete marker.
	Deleted bool
	// DataDir is the UUID directory holding the parts.
	DataDir string
	ModTime time.Time
	Size    int64
	// Mode mirrors the fs.FileMode bits kept for compatibility.
	Mode uint32
	// Metadata holds the HTTP metadata (content-type, x-amz-meta-*, ...).
	Metadata map[string]string
	Parts    []PartInfo
	Erasure  ErasureInfo
	// Data is the inline content of small objects.
	Data []byte
	// NumVersions is the total number of versions of the object.
	NumVersions int
	// SuccessorModTime is the mod time of the next newer version.
	SuccessorModTime time.Time
	// Fresh reports a brand new object (no previous versions).
	Fresh bool
	// Idx is the position of the version in the metadata file.
	Idx int
	// Checksum is the composite additional checksum of the object.
	Checksum []byte
	// Versioned reports that the bucket had versioning when written.
	Versioned bool
	// MarkDeleted asks DeleteVersion to write a delete marker.
	MarkDeleted bool
	// ReplicationState and TransitionStatus are opaque to storage.
	ReplicationState map[string]string
	TransitionStatus string
}

FileInfo is one version of an object as stored in xl.meta.

func DecodeFileInfos

func DecodeFileInfos(volume, name string, raw []byte) ([]FileInfo, error)

DecodeFileInfos decodes a raw xl.meta document (as produced by WalkDir, possibly without its data section) into the FileInfos of its versions, latest first, without inline data.

type Formatter

type Formatter interface {
	// EnsureFormats reads the format.json of every drive of layout and makes
	// the deployment consistent: a pool whose drives carry no document is
	// freshly formatted (one document per drive, written atomically), a
	// pool whose drives carry documents is checked with
	// topology.Service.CheckFormats and its unformatted drives (fresh
	// replacements) receive the document of their position and a healing
	// tracker (HealingFile), so that the healing service repopulates them. Drives are
	// matched to the layout by endpoint; endpoints without a drive (remote
	// nodes) are skipped. It returns the deployment id.
	EnsureFormats(ctx context.Context, layout *topology.Layout, drives []StorageAPI) (string, error)
}

Formatter manages the format.json documents of the drives of a deployment.

func NewFormatter

func NewFormatter(deps FormatterDeps) Formatter

NewFormatter returns the formatter. Topology is required; IDs and Logger default to UUID v4 and slog.Default().

type FormatterDeps

type FormatterDeps struct {
	Topology topology.Service
	// IDs generates deployment identifiers (default: UUID v4).
	IDs topology.IDGenerator
	// Clock dates the healing trackers of the replacement drives (default:
	// wall clock).
	Clock  clock.Clock
	Logger *slog.Logger
}

FormatterDeps are the dependencies of the formatter.

type HealingTracker

type HealingTracker struct {
	ID            string
	PoolIndex     int
	SetIndex      int
	DiskIndex     int
	Path          string
	Endpoint      string
	Started       time.Time
	LastUpdate    time.Time
	ObjectsHealed uint64
	ObjectsFailed uint64
	BytesDone     uint64
	BytesFailed   uint64
	// Bucket and Object are the last item healed: a restarted healing
	// resumes after them.
	Bucket string
	Object string
	// HealedBuckets lists the buckets completely healed.
	HealedBuckets []string `json:",omitempty"`
	// RetryAttempts counts the passes started over because objects failed
	// to heal in the previous one (sources momentarily without quorum).
	RetryAttempts int `json:",omitempty"`
	Finished      bool
}

HealingTracker is the persisted healing state of a drive.

func (*HealingTracker) Progress

func (h *HealingTracker) Progress() *domain.HealingProgress

Progress returns the progress the tracker records, for the drive information (domain.DiskInfo.HealingProgress); nil for a nil tracker.

type Janitor

type Janitor interface {
	lifecycle.Runner
	// Sweep removes the stale entries of every drive now and returns how
	// many it removed.
	Sweep(ctx context.Context) (int, error)
}

Janitor sweeps the temporary area (TmpDir) of the local drives of a node: an entry left there longer than the age threshold is the leftover of a write whose coordinator crashed (in a cluster, another node), which the startup cleanup of the drive would only remove at its own restart. In-progress writes keep their entries fresh (the files they write are modified as the data flows), so the threshold (default 24h) only needs to exceed the longest stall of a write.

func NewJanitor

func NewJanitor(deps JanitorDeps, opts ...JanitorOption) (Janitor, error)

NewJanitor returns the janitor of deps.Drives.

type JanitorDeps

type JanitorDeps struct {
	// Drives are the drives to sweep: the local drives of the node.
	Drives []StorageAPI
	// Clock tells the age of the entries (default: wall clock).
	Clock  clock.Clock
	Logger *slog.Logger
}

JanitorDeps are the dependencies of the janitor.

type JanitorOption

type JanitorOption func(*janitorOptions)

JanitorOption configures the janitor.

func WithMaxTmpAge

func WithMaxTmpAge(d time.Duration) JanitorOption

WithMaxTmpAge sets the age past which a temporary entry is stale.

func WithSweepInterval

func WithSweepInterval(d time.Duration) JanitorOption

WithSweepInterval sets the time between two background sweeps (0 disables them: Sweep runs on demand only).

type MetadataLogStats added in v1.0.1

type MetadataLogStats struct {
	Enabled     bool
	Segments    int
	Bytes       int64
	LiveBytes   int64
	Objects     int64
	IndexBytes  uint64
	IndexBudget uint64
	SegmentSize int64
	// Counters since the drive opened: the flushes of the group commit,
	// the records they made durable and the time they took; the segments
	// compacted and the garbage bytes they reclaimed; the damaged areas
	// the replay skipped.
	Flushes        uint64
	Records        uint64
	FlushSeconds   float64
	Compactions    uint64
	CompactedBytes uint64
	DamagedAreas   uint64
}

MetadataLogStats describe the metadata log of a drive.

type MetadataLogTuner added in v1.0.1

type MetadataLogTuner interface {
	// SetMetadataLogSegmentSize sets the size of the next segments
	// (bounded by MinMetaLogSegmentSize and MaxMetaLogSegmentSize).
	SetMetadataLogSegmentSize(n int64)
	// SetMetadataLogMemoryPercent sets the share of the RAM the index may
	// use (1-90; 0 keeps the default) before a warning.
	SetMetadataLogMemoryPercent(p int)
	// MetadataLogStats returns the state of the log (Enabled false when
	// the drive keeps xl.meta files).
	MetadataLogStats() MetadataLogStats
	// CompactMetadataLog compacts now every sealed segment that holds
	// garbage (the background compaction waits for half of a segment):
	// the segments removed and the bytes reclaimed; nothing without the
	// log.
	CompactMetadataLog(ctx context.Context) (segments int, reclaimed int64, err error)
	// Endpoint names the drive.
	Endpoint() domain.Endpoint
}

MetadataLogTuner is implemented by the local drives: it changes the settings of the metadata log of a running drive and reports its state.

type Option

type Option func(*options)

Option configures the local storage implementation.

func WithMetadataLog added in v1.0.1

func WithMetadataLog(segmentSize int64, memPercent int) Option

WithMetadataLog keeps the xl.meta documents of the user volumes in the metadata log of the drive, with segments of segmentSize bytes (0: 64 MiB) and an index expected to stay below memPercent of the RAM (0: 25%). A drive that keeps xl.meta files converts them at open; without this option, a drive whose metadata is in the log converts it back.

func WithoutFsync

func WithoutFsync() Option

WithoutFsync skips fsync calls; data durability is then left to the operating system (tests, benchmarks).

func WithoutTmpCleanup

func WithoutTmpCleanup() Option

WithoutTmpCleanup keeps the content of the system tmp directory at startup.

type PartInfo

type PartInfo struct {
	Number     int
	ETag       string
	Size       int64
	ActualSize int64
	ModTime    time.Time
	// Index is an optional compression/encryption index.
	Index []byte
	// Checksums holds the per-part additional checksums.
	Checksums map[string]string
}

PartInfo describes an erasure-coded part on this drive.

type ReadOptions

type ReadOptions struct {
	// ReadData returns inline data in FileInfo.Data.
	ReadData bool
	// Healing allows reading versions marked as being healed.
	Healing bool
	// InclFreeVersions includes free versions (tiered objects).
	InclFreeVersions bool
}

ReadOptions tunes ReadVersion.

type RenameDataResult

type RenameDataResult struct {
	// OldDataDir is the data dir of the replaced version, to be purged.
	OldDataDir string
	// Signature identifies the written metadata for quorum checks.
	Signature [4]byte
}

RenameDataResult is the outcome of RenameData.

type RenameOptions

type RenameOptions struct {
	// BaseOptions are shared with Delete when replacing an old version.
	BaseOptions DeleteOptions
}

RenameOptions tunes RenameData.

type StatInfo

type StatInfo struct {
	Name    string
	Size    int64
	Dir     bool
	Mode    uint32
	ModTime time.Time
}

StatInfo describes a stat'ed path.

type StorageAPI

type StorageAPI interface {
	// Endpoint returns the drive endpoint this API serves.
	Endpoint() domain.Endpoint
	// IsOnline reports whether the drive is reachable.
	IsOnline() bool

	// DiskInfo returns the drive usage and identity.
	DiskInfo(ctx context.Context, opts DiskInfoOptions) (domain.DiskInfo, error)

	// MakeVol creates a volume directory.
	MakeVol(ctx context.Context, volume string) error
	// ListVols lists the volumes on the drive.
	ListVols(ctx context.Context) ([]VolInfo, error)
	// StatVol returns information about a volume.
	StatVol(ctx context.Context, volume string) (VolInfo, error)
	// DeleteVol removes a volume; with force it removes its content too.
	DeleteVol(ctx context.Context, volume string, force bool) error

	// ReadVersion reads the metadata of one version of an object. An empty
	// versionID selects the latest version. With opts.ReadData the inline
	// data (small objects) is returned in FileInfo.Data.
	ReadVersion(ctx context.Context, volume, path, versionID string, opts ReadOptions) (FileInfo, error)
	// WriteMetadata writes (or appends) a version to the metadata file.
	// Like RenameData and DeleteVersion, it moves an undecodable xl.meta
	// (domain.ErrInvalidFormat) into QuarantineDir and starts the object
	// over on the drive instead of failing: the other versions are healed
	// back from the drives of the erasure set.
	WriteMetadata(ctx context.Context, volume, path string, fi FileInfo) error
	// UpdateMetadata updates the metadata of an existing version in place.
	UpdateMetadata(ctx context.Context, volume, path string, fi FileInfo, opts UpdateMetadataOptions) error
	// DeleteVersion removes one version (or writes a delete marker when
	// fi.MarkDeleted is set) and cleans up its data directory (unless
	// opts.KeepData). On an undecodable xl.meta (quarantined, see
	// WriteMetadata) the drive no longer holds the version: the delete
	// succeeds, and a delete marker becomes the only version.
	DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, opts DeleteOptions) error

	// CreateFile streams size bytes from r into a new data file, written
	// in the temporary area and renamed into place once complete. size may
	// be -1 when unknown; otherwise a shorter stream fails with
	// io.ErrUnexpectedEOF and nothing is left behind.
	CreateFile(ctx context.Context, volume, path string, size int64, r io.Reader) error
	// ReadFileStream returns a reader over length bytes starting at offset
	// (length -1 reads to the end). A range beyond the end of the file is
	// ErrInvalidRange when offset is past the end, ErrFileCorrupt (a
	// domain.ErrFaultyDisk) when only the end is.
	ReadFileStream(ctx context.Context, volume, path string, offset, length int64) (io.ReadCloser, error)
	// OpenFile opens a data file for reads at any offset until the handle
	// is closed: the data stays readable when the file is removed or
	// replaced meanwhile (an open file survives its removal, POSIX), which
	// pins the shards of a read that released its lock (docs/02 §2.4).
	// Opening transfers no data; a remote drive keeps the file open on its
	// node until Close or an idle timeout.
	OpenFile(ctx context.Context, volume, path string) (FileHandle, error)
	// RenameData atomically moves the temporary data directory and metadata
	// of fi into their final location, committing the version. Parent
	// prefixes that are objects, or a destination that is a prefix, are
	// refused with domain.ErrInvalidObjectName (code XNexoParentIsObject /
	// XNexoObjectExistsAsDirectory).
	RenameData(ctx context.Context, srcVolume, srcPath string, fi FileInfo, dstVolume, dstPath string, opts RenameOptions) (RenameDataResult, error)
	// RenameFile atomically moves a single file (multipart parts being
	// assembled into an object data directory). Missing parents of the
	// destination are created.
	RenameFile(ctx context.Context, srcVolume, srcPath, dstVolume, dstPath string) error

	// WalkDir walks the entries below opts.BaseDir in S3 key order and
	// calls fn for each. Returning an error from fn stops the walk with
	// that error; fs.SkipAll stops it without error; fs.SkipDir skips the
	// content of a prefix entry (or the remaining entries of the current
	// directory when returned for an object entry).
	WalkDir(ctx context.Context, opts WalkDirOptions, fn WalkDirFunc) error

	// ReadAll reads a whole small file (configuration, format.json, ...).
	ReadAll(ctx context.Context, volume, path string) ([]byte, error)
	// WriteAll atomically writes a whole small file.
	WriteAll(ctx context.Context, volume, path string, b []byte) error
	// Delete removes a file or, with opts.Recursive, a directory tree.
	Delete(ctx context.Context, volume, path string, opts DeleteOptions) error

	// CheckParts verifies that every part of fi exists with the right size.
	CheckParts(ctx context.Context, volume, path string, fi FileInfo) (*CheckPartsResult, error)
	// StatInfoFile stats a path; with glob, path may contain "*".
	StatInfoFile(ctx context.Context, volume, path string, glob bool) ([]StatInfo, error)

	// Healing returns the healing state of the drive, nil when not healing.
	Healing() *HealingTracker
	// Close releases the drive resources.
	Close(ctx context.Context) error
}

StorageAPI is the low-level drive interface used by the object layer. Every method takes the volume (bucket) and the path (object key) the operation refers to. Errors are domain sentinels (ErrDiskNotFound, ErrFaultyDisk, ErrObjectNotFound, ErrNoSuchVersion, ErrStorageFull, ...).

Paths are slash separated and relative to the volume: empty segments, "." and "..", and symbolic links on disk are refused, so that no operation can reach outside the drive. A version id of "null" designates the null version, like the empty id.

func NewLocal

func NewLocal(deps Deps, ep domain.Endpoint, opts ...Option) (StorageAPI, error)

NewLocal opens (creating it when missing) the drive directory of a local endpoint, prepares the system directories (.nexo.sys, tmp, multipart, buckets, config), empties tmp, probes writability and loads the drive id from format.json when present. The drive is left unformatted when no format.json exists: Formatter.EnsureFormats writes it. In a distributed deployment the drives of this node are URL endpoints marked IsLocal by the topology: they are opened at their path and keep their URL as Endpoint.

type UpdateMetadataOptions

type UpdateMetadataOptions struct {
	// NoPersistence skips the fsync for non-critical updates.
	NoPersistence bool
}

UpdateMetadataOptions tunes UpdateMetadata.

type VolInfo

type VolInfo struct {
	Name    string
	Created time.Time
}

VolInfo describes a volume (bucket directory).

type WalkDirFunc

type WalkDirFunc func(entry WalkEntry) error

WalkDirFunc receives WalkDir entries. Returning fs.SkipDir from a directory entry skips its content.

type WalkDirOptions

type WalkDirOptions struct {
	Bucket  string
	BaseDir string
	// Recursive descends into sub-directories.
	Recursive bool
	// ReportNotFound reports a missing BaseDir instead of an empty walk.
	ReportNotFound bool
	// FilterPrefix only returns entries starting with the prefix.
	FilterPrefix string
	// ForwardTo skips entries before this name (resume a listing).
	ForwardTo string
	// Limit stops after this many entries (0 = unlimited).
	Limit int
}

WalkDirOptions tunes WalkDir.

type WalkEntry

type WalkEntry struct {
	// Name is the entry path relative to the volume. Directory entries end
	// with a slash.
	Name string
	// IsDir reports a directory (prefix).
	IsDir bool
	// Metadata is the raw xl.meta content for object entries, nil for prefixes.
	Metadata []byte
}

WalkEntry is one entry produced by WalkDir.

Directories

Path Synopsis
Package storagetest is the behavior suite of storage.StorageAPI.
Package storagetest is the behavior suite of storage.StorageAPI.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址