bitalosdb

package module
v2.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 10, 2026 License: Apache-2.0 Imports: 40 Imported by: 2

README

bitalos

Bitalosdb is a high-performance KV storage engine. 中文版

Introduction

  • Bitalosdb is a high-performance KV storage engine (self-developed), which is based on the creative IO architecture and storage technology and focuses on solving the problem of LSM-Tree read and write amplification. As a substitute for Rocksdb, Bitalosdb can improve read and write performance greatly.

Team

Key Technology

  • Bithash (KV separation technology), significantly reduce write amplification. For bithash, time complexity of retrieval is O(1). GC can be completed independently, and value and index are decoupled.

  • Bitalostree (high-performance compression index technology), basically eliminate read amplification. If B+ Tree has a number of huge Pages, write amplification is a severe problem. With a creative index compression technology, Bitalostree eliminates B+ Tree write amplification, and improves the read performance.

  • Bitalostable (cold and hot data separation technology), stores cold data, which is calculated according to the data scale and access frequency. Storage engine writes cold data to Bitalostable when QPS becomes low. Improve data compression, reduce index memory consumption, and achieve more rational resource utilization. (open source stable edition has basic features, enterprise edition supports more comprehensive hot and cold separation).

Performance

  • This benchmark is based on Bitloasdb version (v5.0) and rocksdb stable version (v7.6.0).

Hardware

CPU:    Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
Memory: 384GB
Disk:   2*3.5TB NVMe SSD

Program

  • Benchmark thread number: 8

  • Program cpu cgroup: 8 core

  • Comparison standard: QPS on single-core (multi-core QPS / core number), single-core performance reflects cost advantage better.

Data

  • Key-value spec: key-size=32B、value-size=1KB

  • Comparison dimensions: Total data size(25/50/100GB) * IO ratio(100% random write, 100% random read, 50% random write + 50% random read, 30% random write + 70% random read)

Config

  • rocksdb
Memtable：256MB
WAL：enable
Cache：8GB
TargetFileSize：256M
L0CompactTrigger：8
L0StopWritesTrigger：24
  • bitalosdb
Memtable：256MB
WAL：enable
Cache：disable

Result

benchmark

Document

Technical architecture and documentation, refer to the official website: bitalos.zuoyebang.com (Building...)

Documentation

Index

Constants

View Source 
const (
	InternalKeyKindDelete       = base.InternalKeyKindDelete
	InternalKeyKindSet          = base.InternalKeyKindSet
	InternalKeyKindSetBithash   = base.InternalKeyKindSetBithash
	InternalKeyKindLogData      = base.InternalKeyKindLogData
	InternalKeyKindPrefixDelete = base.InternalKeyKindPrefixDelete
	InternalKeyKindExpireAt     = base.InternalKeyKindExpireAt
	InternalKeyKindMax          = base.InternalKeyKindMax
	InternalKeySeqNumMax        = base.InternalKeySeqNumMax
	InternalKeyKindInvalid      = base.InternalKeyKindInvalid
)
View Source 
const (
	DataTypeNone      = kkv.DataTypeNone
	DataTypeString    = kkv.DataTypeString
	DataTypeBitmap    = kkv.DataTypeBitmap
	DataTypeHash      = kkv.DataTypeHash
	DataTypeList      = kkv.DataTypeList
	DataTypeSet       = kkv.DataTypeSet
	DataTypeZset      = kkv.DataTypeZset
	DataTypeZsetIndex = kkv.DataTypeZsetIndex
	DataTypeDKHash    = kkv.DataTypeDKHash
	DataTypeDKSet     = kkv.DataTypeDKSet
)

Variables

View Source 
var (
	ErrNotFound      = base.ErrNotFound
	ErrWrongType     = base.ErrWrongType
	ErrClosed        = errors.New("bitalosdb: closed")
	ErrNoSlotId      = errors.New("bitalosdb: not set slotId")
	ErrSubKeySize    = errors.New("invalid subKey size")
	ErrKeySize       = errors.New("invalid key size")
	ErrSiKeySize     = errors.New("invalid siKey size")
	ErrValueSize     = errors.New("invalid value size")
	ErrZsetMemberNil = errors.New("zset member is nil")
	ErrDKExist       = errors.New("bitalosdb: distributed key exist")
	ErrDKShardNum    = errors.New("invalid shard num")
	ErrBitUnmarshal  = errors.New("ERR bitmap unmarshal fail")
	ErrBitMarshal    = errors.New("ERR bitmap marshal fail")
	ErrBitupleNil    = errors.New("bituple is nil")
)
View Source 
var DefaultComparer = base.DefaultComparer
View Source 
var DefaultLogger = base.DefaultLogger

Functions

func NewFlushStats 

func NewFlushStats() *dbFlushStats

Types

type Bituple 

type Bituple struct {
	// contains filtered or unexported fields
}

func (*Bituple) Close 

func (b *Bituple) Close() (err error)

func (*Bituple) GetDebugInfo 

func (b *Bituple) GetDebugInfo() string

func (*Bituple) GetDirDiskInfo 

func (b *Bituple) GetDirDiskInfo() string

func (*Bituple) GetDiskInfo 

func (b *Bituple) GetDiskInfo() (bitupleDisk, bitpageDisk, bithashDisk uint64)

func (*Bituple) GetInfo 

func (b *Bituple) GetInfo() string

func (*Bituple) NewBitupleIter 

func (b *Bituple) NewBitupleIter() *TupleIterator

func (*Bituple) RemoveDir 

func (b *Bituple) RemoveDir() error

func (*Bituple) RunVtGC 

func (b *Bituple) RunVtGC(force bool)

func (*Bituple) RunVtRehash 

func (b *Bituple) RunVtRehash(force bool)

type BitupleIterator 

type BitupleIterator struct {
	// contains filtered or unexported fields
}

func (*BitupleIterator) Close 

func (i *BitupleIterator) Close() error

func (*BitupleIterator) First 

func (i *BitupleIterator) First() ([]byte, uint8, uint64)

func (*BitupleIterator) GetCursor 

func (i *BitupleIterator) GetCursor() []byte

func (*BitupleIterator) Next 

func (i *BitupleIterator) Next() ([]byte, uint8, uint64)

func (*BitupleIterator) Seek 

func (i *BitupleIterator) Seek(cursor []byte) ([]byte, uint8, uint64)

type CompactEnv 

type CompactEnv struct {
	StartHour            int
	EndHour              int
	BithashDeletePercent float64
	VtGCThreshold        float64
}

type Compare 

type Compare = base.Compare

type Comparer 

type Comparer = base.Comparer

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func Open 

func Open(dirname string, opts *Options) (db *DB, err error)

func (*DB) AsyncMemFlush 

func (d *DB) AsyncMemFlush(flushForce bool) (<-chan struct{}, error)

func (*DB) AsyncVmFlush 

func (d *DB) AsyncVmFlush() (<-chan struct{}, error)

func (*DB) BitCount64 

func (d *DB) BitCount64(key []byte, slotId uint16, begin, end int) (int64, error)

func (*DB) BitPos64 

func (d *DB) BitPos64(key []byte, slotId uint16, on, begin, end int) (int64, error)

func (*DB) Checkpoint 

func (d *DB) Checkpoint(dstDir string) (func(), error)

func (*DB) Close 

func (d *DB) Close() (err error)

func (*DB) DKCreate 

func (d *DB) DKCreate(key []byte, slotId uint16, dataType uint8, shardNum uint32) error

func (*DB) DKCreateShard 

func (d *DB) DKCreateShard(key []byte, slotId uint16, dataType uint8) error

func (*DB) DKIncrBySize 

func (d *DB) DKIncrBySize(key []byte, slotId uint16, incr int64) (int64, error)

func (*DB) DebugInfo 

func (d *DB) DebugInfo() string

func (*DB) Delete 

func (d *DB) Delete(key []byte, slotId uint16) (int64, uint8, error)

func (*DB) DeleteKKV 

func (d *DB) DeleteKKV(key []byte, slotId uint16, dataType uint8, subKeys ...[]byte) (int64, error)

func (*DB) DeleteXKKV 

func (d *DB) DeleteXKKV(key []byte, slotId uint16, dataType uint8, subKeys ...[]byte) (int64, error)

func (*DB) DirDiskInfo 

func (d *DB) DirDiskInfo() string

func (*DB) DiskInfo 

func (d *DB) DiskInfo() (bitupleDisk, bitpageDisk, bithashDisk uint64)

func (*DB) Exist 

func (d *DB) Exist(key []byte, slotId uint16) (bool, error)

func (*DB) ExpireAt 

func (d *DB) ExpireAt(key []byte, slotId uint16, newTs uint64) (int64, uint8, error)

func (*DB) Flush 

func (d *DB) Flush(flushForce bool) error

func (*DB) FlushBitpage 

func (d *DB) FlushBitpage()

func (*DB) FlushMemtable 

func (d *DB) FlushMemtable() error

func (*DB) FlushVm 

func (d *DB) FlushVm() error

func (*DB) Get 

func (d *DB) Get(key []byte, slotId uint16) (value []byte, dataType uint8, timestamp uint64, closer func(), err error)

func (*DB) GetBit64 

func (d *DB) GetBit64(key []byte, slotId uint16, offset uint64) (int64, error)

func (*DB) GetMeta 

func (d *DB) GetMeta(key []byte, slotId uint16) (dataType uint8, timestamp, version, lindex, rindex uint64, size uint32, err error)

func (*DB) GetMetaSize 

func (d *DB) GetMetaSize(key []byte, slotId uint16) (size uint32, err error)

func (*DB) GetStats 

func (d *DB) GetStats() DbStats

func (*DB) GetTimestamp 

func (d *DB) GetTimestamp(key []byte, slotId uint16) (uint8, uint64, error)

func (*DB) HDel 

func (d *DB) HDel(key []byte, slotId uint16, subKeys ...[]byte) (int64, error)

func (*DB) HDelX 

func (d *DB) HDelX(key []byte, slotId uint16, subKeys ...[]byte) (int64, error)

func (*DB) HGet 

func (d *DB) HGet(key []byte, slotId uint16, subKey []byte) ([]byte, func(), error)

func (*DB) HGetAll 

func (d *DB) HGetAll(key []byte, slotId uint16) ([][]byte, func(), error)

func (*DB) HIncrBy 

func (d *DB) HIncrBy(key []byte, slotId uint16, subKey []byte, incr int64) (int64, error)

func (*DB) HKeys 

func (d *DB) HKeys(key []byte, slotId uint16) ([][]byte, func(), error)

func (*DB) HMGet 

func (d *DB) HMGet(key []byte, slotId uint16, subKeys ...[]byte) ([][]byte, []func(), error)

func (*DB) HMSet 

func (d *DB) HMSet(key []byte, slotId uint16, kvs ...[]byte) (int64, error)

func (*DB) HMSetX 

func (d *DB) HMSetX(key []byte, slotId uint16, isExist bool, kvs ...[]byte) (int64, error)

func (*DB) HValues 

func (d *DB) HValues(key []byte, slotId uint16) ([][]byte, func(), error)

func (*DB) IsCheckpointHighPriority 

func (d *DB) IsCheckpointHighPriority() bool

func (*DB) IsCheckpointSyncing 

func (d *DB) IsCheckpointSyncing() bool

func (*DB) IsClosed 

func (d *DB) IsClosed() bool

func (*DB) ListIndex 

func (d *DB) ListIndex(key []byte, slotId uint16, index int64) ([]byte, func(), error)

func (*DB) ListPop 

func (d *DB) ListPop(key []byte, slotId uint16, isLeft bool) ([]byte, func(), error)

func (*DB) ListPush 

func (d *DB) ListPush(key []byte, slotId uint16, isLeft bool, isExist bool, values ...[]byte) (int64, error)

func (*DB) ListRange 

func (d *DB) ListRange(key []byte, slotId uint16, start, stop int64) ([][]byte, func(), error)

func (*DB) ListSet 

func (d *DB) ListSet(key []byte, slotId uint16, index int64, value []byte) error

func (*DB) ListTrim 

func (d *DB) ListTrim(key []byte, slotId uint16, start, stop int64) (err error)

func (*DB) NewBitupleIter 

func (d *DB) NewBitupleIter() *BitupleIterator

func (*DB) NewKKVIterator 

func (d *DB) NewKKVIterator(o *IterOptions) *KKVIterator

func (*DB) RemoveSlot 

func (d *DB) RemoveSlot(slotId uint16) error

func (*DB) SAdd 

func (d *DB) SAdd(key []byte, slotId uint16, kvs ...[]byte) (int64, error)

func (*DB) SAddX 

func (d *DB) SAddX(key []byte, slotId uint16, isExist bool, kvs ...[]byte) (int64, error)

func (*DB) SIsMember 

func (d *DB) SIsMember(key []byte, slotId uint16, subKey []byte) (int64, error)

func (*DB) SMembers 

func (d *DB) SMembers(key []byte, slotId uint16) ([][]byte, func(), error)

func (*DB) SPop 

func (d *DB) SPop(key []byte, slotId uint16, count int64) ([][]byte, func(), error)

func (*DB) SPopX 

func (d *DB) SPopX(key []byte, slotId uint16, count int64) ([][]byte, func(), error)

func (*DB) SRandMember 

func (d *DB) SRandMember(key []byte, slotId uint16, count int64) ([][]byte, func(), error)

func (*DB) SRem 

func (d *DB) SRem(key []byte, slotId uint16, members ...[]byte) (int64, error)

func (*DB) SRemX 

func (d *DB) SRemX(key []byte, slotId uint16, members ...[]byte) (int64, error)

func (*DB) Scan 

func (d *DB) Scan(cursor []byte, count int, dt uint8, cmp func(member string) bool) ([]byte, [][]byte, error)

func (*DB) Set 

func (d *DB) Set(key []byte, slotId uint16, dataType uint8, timestamp uint64, value []byte) error

func (*DB) SetAutoCompact 

func (d *DB) SetAutoCompact(newVal bool)

func (*DB) SetBit64 

func (d *DB) SetBit64(key []byte, slotId uint16, offset uint64, on int) (int64, error)

func (*DB) SetMeta 

func (d *DB) SetMeta(
	key []byte, slotId uint16, dataType uint8,
	hi, lo, timestamp, version uint64,
	lindex, rindex uint64, size uint32,
) error

func (*DB) VtableGC 

func (d *DB) VtableGC(slotId uint16)

func (*DB) VtableRehash 

func (d *DB) VtableRehash(slotId uint16)

func (*DB) ZAdd 

func (d *DB) ZAdd(key []byte, slotId uint16, kvs ...[]byte) (int64, error)

func (*DB) ZCount 

func (d *DB) ZCount(key []byte, slotId uint16, min float64, max float64, leftClose bool, rightClose bool) (int64, error)

func (*DB) ZIncrBy 

func (d *DB) ZIncrBy(key []byte, slotId uint16, member []byte, incr float64) (float64, error)

func (*DB) ZLexCount 

func (d *DB) ZLexCount(
	key []byte, slotId uint16, min []byte, max []byte,
	leftClose bool, rightClose bool,
) (int64, error)

func (*DB) ZRange 

func (d *DB) ZRange(key []byte, slotId uint16, start int64, stop int64, reverse bool) ([]ScorePair, func(), error)

func (*DB) ZRangeByLex 

func (d *DB) ZRangeByLex(
	key []byte, slotId uint16, min []byte, max []byte,
	leftClose bool, rightClose bool, offset int, count int,
) ([][]byte, func(), error)

func (*DB) ZRangeByScore 

func (d *DB) ZRangeByScore(
	key []byte, slotId uint16, min float64, max float64,
	leftClose bool, rightClose bool, offset int, count int,
) ([]ScorePair, func(), error)

func (*DB) ZRank 

func (d *DB) ZRank(key []byte, slotId uint16, member []byte, reverse bool) (int64, error)

func (*DB) ZRem 

func (d *DB) ZRem(key []byte, slotId uint16, members ...[]byte) (int64, error)

func (*DB) ZRemRangeByLex 

func (d *DB) ZRemRangeByLex(
	key []byte, slotId uint16,
	min []byte, max []byte,
	leftClose bool, rightClose bool,
) (int64, error)

func (*DB) ZRemRangeByRank 

func (d *DB) ZRemRangeByRank(
	key []byte, slotId uint16, start int64, stop int64,
) (int64, error)

func (*DB) ZRemRangeByScore 

func (d *DB) ZRemRangeByScore(
	key []byte, slotId uint16,
	min float64, max float64,
	leftClose bool, rightClose bool,
) (int64, error)

func (*DB) ZRevRangeByScore 

func (d *DB) ZRevRangeByScore(
	key []byte, slotId uint16, min float64, max float64,
	leftClose bool, rightClose bool, offset int, count int,
) ([]ScorePair, func(), error)

func (*DB) ZScore 

func (d *DB) ZScore(key []byte, slotId uint16, member []byte) (float64, error)

type DbStats 

type DbStats struct {
	VmTableFlushLastCost  int64
	VmTableFlushAvgCost   int64
	MemTableFlushLastCost int64
	MemTableFlushAvgCost  int64
}

type Equal 

type Equal = base.Equal

type FileNum 

type FileNum = base.FileNum

type FileType 

type FileType = base.FileType

type InternalKKVIterator 

type InternalKKVIterator = kkv.InternalIterator

type InternalKKVKey 

type InternalKKVKey = kkv.InternalKey

type InternalKey 

type InternalKey = base.InternalKey

type InternalKeyKind 

type InternalKeyKind = base.InternalKeyKind

type IterOptions 

type IterOptions = options.IterOptions

type IterValidityState 

type IterValidityState int8
const (
	IterExhausted IterValidityState = iota
	IterValid
)

type IteratorStats 

type IteratorStats struct {
	ForwardSeekCount [NumStatsKind]int
	ReverseSeekCount [NumStatsKind]int
	ForwardStepCount [NumStatsKind]int
	ReverseStepCount [NumStatsKind]int
}

type IteratorStatsKind 

type IteratorStatsKind int8
const (
	InterfaceCall IteratorStatsKind = iota
	InternalIterCall
	NumStatsKind
)

type KKVIterator 

type KKVIterator struct {
	// contains filtered or unexported fields
}

func (*KKVIterator) Close 

func (i *KKVIterator) Close() error

func (*KKVIterator) Error 

func (i *KKVIterator) Error() error

func (*KKVIterator) First 

func (i *KKVIterator) First() bool

func (*KKVIterator) GetListIndex 

func (i *KKVIterator) GetListIndex() uint64

func (*KKVIterator) GetSubKey 

func (i *KKVIterator) GetSubKey() []byte

func (*KKVIterator) GetZsetKey 

func (i *KKVIterator) GetZsetKey() (float64, []byte)

func (*KKVIterator) GetZsetMember 

func (i *KKVIterator) GetZsetMember() []byte

func (*KKVIterator) GetZsetScore 

func (i *KKVIterator) GetZsetScore() (float64, bool)

func (*KKVIterator) Key 

func (i *KKVIterator) Key() []byte

func (*KKVIterator) Last 

func (i *KKVIterator) Last() bool

func (*KKVIterator) Next 

func (i *KKVIterator) Next() bool

func (*KKVIterator) NextWithLimit 

func (i *KKVIterator) NextWithLimit() IterValidityState

func (*KKVIterator) Prev 

func (i *KKVIterator) Prev() bool

func (*KKVIterator) PrevWithLimit 

func (i *KKVIterator) PrevWithLimit() IterValidityState

func (*KKVIterator) SeekGE 

func (i *KKVIterator) SeekGE(key []byte) bool

func (*KKVIterator) SeekGEWithLimit 

func (i *KKVIterator) SeekGEWithLimit(key []byte) IterValidityState

func (*KKVIterator) SeekLT 

func (i *KKVIterator) SeekLT(key []byte) bool

func (*KKVIterator) SetBounds 

func (i *KKVIterator) SetBounds(lower, upper []byte)

func (*KKVIterator) Stats 

func (i *KKVIterator) Stats() IteratorStats

func (*KKVIterator) Valid 

func (i *KKVIterator) Valid() bool

func (*KKVIterator) Value 

func (i *KKVIterator) Value() []byte

type Logger 

type Logger = base.Logger

type Options 

type Options struct {
	BytesPerSync                   int
	Comparer                       *Comparer
	FS                             vfs.FS
	Logger                         Logger
	CompressionType                int
	Verbose                        bool
	LogTag                         string
	DeleteFileInternal             int
	DisableStoreKey                bool
	VectorTableCount               uint16
	VectorTableHashSize            uint32
	VmTableSize                    int
	VmTableStopWritesThreshold     int
	MemTableSize                   int
	MemTableStopWritesThreshold    int
	GetNowTimestamp                func() uint64
	IOWriteLoadThresholdFunc       func() bool
	MaxKeySize                     int
	MaxSubKeySize                  int
	MaxValueSize                   int
	FlushPrefixDeleteKeyMultiplier int
	FlushFileLifetime              int
	BitpageFlushSize               uint64
	BitpageSplitSize               uint64
	BitpageDisableMiniVi           bool
	BitpageTaskWorkerNum           int
	BithashTableSize               int
	AutoCompact                    bool
	CompactInfo                    CompactEnv
	FlushReporter                  func(int)
	// contains filtered or unexported fields
}

func (*Options) Clone 

func (o *Options) Clone() *Options

func (*Options) EnsureDefaults 

func (o *Options) EnsureDefaults() *Options

type ScorePair 

type ScorePair struct {
	Score  float64
	Member []byte
}

type Tuple 

type Tuple struct {
	// contains filtered or unexported fields
}

type TupleIterator 

type TupleIterator struct {
	// contains filtered or unexported fields
}

func (*TupleIterator) Close 

func (i *TupleIterator) Close() error

func (*TupleIterator) First 

func (i *TupleIterator) First() (
	key []byte, h, l, seqNum uint64, dataType uint8,
	timestamp uint64, version uint64, slotId uint16, size uint32,
	pre, next uint64, value []byte, final bool,
)

func (*TupleIterator) Next 

func (i *TupleIterator) Next() (
	key []byte, h, l, seqNum uint64, dataType uint8,
	timestamp uint64, version uint64, slotId uint16, size uint32,
	pre, next uint64, value []byte, final bool,
)

Source Files

View all Source files

Directories

Path Synopsis
hashindex
sklindex
vectorindex
vectorindex64
bdb
internal
arenaskl
arenaskl2
base
bindex
bitask
buffer
bytepools
cache
cache/lfucache
cache/lfucache/internal/arenaskl
cache/lfucache/internal/base
cache/lrucache
compress
consts
crc
errors
fastrand
hash
humanize
invariants
iterator
kkv
list2
manual
mmap
options
os2
simd
sortedkv
statemachine
unsafe2
utils
vectormap
vectortable
vfs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.
⌂ 换网址