Repository navigation
sstable: explore index size optimizations #97
Description
RocksDB has added version 3 and version 4 sstable formats:
- Version 3 changes index block keys to not include the sequence number (which is always 0?)
- Version 4 changes index block values (block handles) from being encoded as
<offset, size>to a delta encoding that usually only stores<size>.
There is no particular urgency to supporting these formats, but we should investigate whether these format changes move the needle on any benchmarks.
Jira issue: PEBBLE-185
Activity
petermattis commentedon May 1, 2019
@ajkr do you have an inside knowledge on this?
ajkr commentedon May 1, 2019
I remember
format_version=4saved noticeable space on indexes. But it looks like it depends on settingindex_block_restart_interval > 1, which we currently don't do. So it shouldn't be the first thing we try if we ever want to optimize index size.petermattis commentedon May 2, 2019
Something I've been meaning to ask you about: we saw a situation once where keys larger than a data block were written to a table. The result was that every data block had 1 entry, and the index block was essentially the same size as all of the data blocks combined. I think we could avoid this situation if there was a bit more complexity in the sstable block boundary heuristics. Specifically, we shouldn't allow a block to contain only a single entry if the key for the entry is larger than the target block size.
On the other hand, perhaps it isn't worth futzing with the sstable block boundary heuristics, and instead put effort into blob (wisc-key) storage of large entries.
github-actions commentedon Mar 2, 2022
We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it
in 10 days to keep the issue queue tidy. Thank you for your
contribution to Pebble!
[-]sstable: support format version 3 and 4[/-][+]sstable: explore index size optimizations[/+]github-actions commentedon Aug 28, 2023
We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it
in 10 days to keep the issue queue tidy. Thank you for your
contribution to Pebble!
colblk: add index block writer, reader, iterator
colblk: add index block writer, reader, iterator
colblk: add index block writer, reader, iterator