[folbricht]

The README claims "identical output to casync". Two things stopped that from being true for blob indexes.

1. The chunker never tested the boundary at the min chunk size

Both tools apply the same optimization — leave the first min - 48 bytes unhashed, since no boundary can occur before the minimum chunk size. They differ in one step of the seeding.

casync's ca_chunker_scan() fills the 48-byte window so chunk_size reaches exactly min, computes ca_chunker_start(), and then immediately tests it:

v = ca_chunker_start(c, c->window, c->window_size);
k += m;
if (shall_break(c, v))     /* boundary test at chunk_size == min */
        goto now;

desync seeded the identical hash over c.buf[c.min-ChunkerWindowSize : c.min] but never evaluated it — the loop started at base := int(c.min) and rolled buf[min] in before the first test, so min+1 was the smallest chunk it could emit.

Whenever a boundary fell exactly on min the two disagreed. On a 123MB tar of /usr that happened once in 1699 chunks: casync emitted 16384 + 56320 where desync emitted a single 72704. At that offset the window ending at 16384 hashes to 0xad145682, and 0xad145682 % 49535 == 49534 == d-1, so it is a genuine boundary; desync's first test one byte later is not. The two resynced at the next boundary, so it never cascaded.

Fixed by testing the seeded window before entering the roll loop.

2. The blob index header omitted CaFormatExcludeFile

casync includes it in CA_FORMAT_DEFAULT and stamps it into every index, so its flags read 0xb000000000000000 where desync's read 0xa000000000000000 — a one-byte difference even when the chunking agreed.

The flag means the encoder honors .caexclude files while walking a directory tree (caencoder.c:538), so it is inert for a blob index, where there is no tree to walk. It is therefore set only on that path, in make.go.

Archive indexes deliberately keep their current flags. ChunkStream (used only by tar -i) would be claiming .caexclude support that desync doesn't implement — that's still open as #114 — and caidx flags differ from casync's regardless, because desync stores a smaller catar feature set (0xb0000001ffeffe26 vs 0xb00000000100 1f22 on a sample tree). Setting the bit there would buy no parity and assert something untrue.

Verification

casync 2 (Debian 2+20201210-2+b2) and desync run in the same container over three data sets — a 123MB tar of /usr, 400MB of zeros, 400MB of random. .caibx files are now byte-identical in every case, including the tar that previously diverged (both now 1699 chunks, 68064 bytes).

Also identical with --digest=sha256 and with non-default chunk sizes (8:32:128K), and cross-tool extraction works both directions — casync extracting from a desync-built store and vice versa, output compared byte-for-byte, with both stores holding the same 1678 chunks.

chunker_minsize_test.go covers the min-size boundary directly by searching for a window whose hash lands on a boundary and asserting the resulting chunk is exactly min bytes. It fails without the fix (produces 122 bytes for a 64-byte min).

Compatibility

Nothing about the format, chunk IDs or stores changes, and existing indexes and chunks stay valid and readable by both tools.

The chunker change alters boundaries in the rare case above, so re-chunking data that is already in a store won't dedup against those few chunks. On the tar that was 1 chunk in 1699; the expected rate is roughly 1 in discriminator (~49535 at the default average).