Repository navigation
Skip destination read-back for null segments during assembly - #368
Merged
Merged
Conversation
folbricht
force-pushed
the
skip-null-segment-readback
branch
from
July 8, 2026 07:29
5bcd776 to
28f32db
Compare
After writing a seed segment, AssembleFile reads every chunk of the segment back from the destination and hashes it to detect seeds that changed while the extraction was running. Null segments were subject to the same read-back even though they have no external source that could change: they write deterministic zeros (or clone from desync's own temporary block file), so there is nothing to re-validate. For sparse files like VM images this read-back dominates the entire assembly time since SHA512/256 runs at roughly 500 MB/s per core while the null writes themselves are nearly free (skipped entirely on blank files, or reflinked). Skipping the read-back for null segments, identified by an empty FileName like Plan.Validate already does, cuts assembly of a 1 GiB image that is 97% zeros from 1.5s to under 0.1s in local testing.
folbricht
force-pushed
the
skip-null-segment-readback
branch
from
July 8, 2026 08:00
28f32db to
9ef5e13
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After writing a seed segment, the
AssembleFileworkers read every chunk of the segment back from the destination and hash it, to catch seeds whose content changed while the extraction was running. This read-back also runs for null segments, where the rationale doesn't apply: anullChunkSectionwrites deterministic zeros (or clones from desync's own temporary block file), so there is no external file that can change underneath the extraction.Plan.Validatealready treats null segments as always-valid for the same reason.For sparse files such as VM images, this read-back is essentially the entire cost of assembly: the null writes themselves are nearly free (skipped entirely on blank files, or reflinked), while reading back and SHA512/256-hashing the zeroed ranges runs at only ~500 MB/s per core.
Change
Skip the post-write read-back for null segments, identified by an empty
FileName()— the same conventionisFileSeed()uses in the sequencer. Read-back still runs for all file seed segments.Assembling a 1 GiB image that is ~97% zeros (chunked at default sizes, local store,
N: 10) went from 1.5 s to under 0.1 s, with the output verified byte-identical.Also adds a test that assembles a null-heavy index over an existing file full of random data, confirming null sections are still written out correctly without the read-back.