Repository navigation
Comparing changes
Open a pull request
base repository: folbricht/desync
base: v1.1.1
head repository: folbricht/desync
compare: v1.1.2
- 13 commits
- 108 files changed
- 2 contributors
Commits on Aug 30, 2026
-
Support storing indexes in OCI registries (#396)
* Support storing indexes in OCI registries Chunks could be kept in a registry but indexes could not, so every user still needed a separate channel for the caibx. A registry can now hold everything needed to distribute a file. The index location is the repository followed by the index name, which becomes the tag: desync extract -s oci+https://ghcr.io/me/store \ oci+https://ghcr.io/me/store/file.iso.caibx file.iso That is the same split indexStoreFromLocation already applies to every other backend, path.Dir for the store and path.Base for the name, so no URL or config handling needed changing. Index names are therefore limited to the OCI tag grammar, which is checked at the store rather than left to the registry to reject. Each index is its own artifact, mirroring the chunk layout: a blob holding the index, referenced by a manifest tagged with the index name and carrying artifact type application/vnd.desync.index.v1. Annotations record the chunk count, indexed size and chunk sizes so the shape of an index can be read from the manifest without fetching it. Indexes and chunks can share a repository. Chunk operations already require the chunk artifact type, so prune leaves indexes alone even if one is named like a chunk ID, and an index never satisfies a chunk lookup. A registry blob must be pushed with its digest and size known upfront, and an index of a large blob runs to hundreds of megabytes, so the index is serialized to a temp file and hashed on the way rather than buffered. The repository and client setup is now shared between the chunk and index stores; encryption options are rejected as they are for other index stores. * Address review findings on the OCI index store Timeouts: the shared http.Client applied the store timeout to the whole exchange, body included. That suits chunks of a few hundred KB but caps the case this feature exists for - an index of a large blob runs to hundreds of megabytes, which a one minute default aborts mid-transfer. Index stores now bound the wait for response headers instead, so a stalled server still fails while a slow one is allowed to finish. Retries: the blob was pushed from an *os.File, which leaves GetBody nil, so oras skips its retry path and re-authentication of a request whose token expires mid-upload fails outright. Indexes up to 32MB are now pushed from a bytes.Reader, which is rewindable, and only larger ones stream. The temp file is gone with it. The index is serialized twice instead, once to size and digest it and once into the push, which needs neither disk nor a large buffer. $TMPDIR is RAM-backed on several distributions, so writing there did not achieve what the comment claimed anyway. Index names are checked before commands do their work. make and tar write the index last, so an unusable name previously surfaced only after the whole file had been chunked and uploaded. Also document that a store-options entry enabling encryption on a shared repository breaks index operations, and how the timeout applies. * Bound the manifest read in the OCI stores Both stores read a manifest response body with io.ReadAll. oras applies its MaxMetadataBytes limit on the paths that fetch other metadata, but FetchReference hands back the raw response body, so a registry answering with an endless body would be read until memory ran out. Read manifests through a shared helper that caps the body at 4MiB, oras' own default and far above any manifest desync writes. The body is still decoded whole rather than streamed off the reader: at these sizes there is nothing to gain from streaming, and decoding the whole body rejects anything trailing the manifest instead of ignoring it. * Bound OCI index transfers by inactivity rather than duration Index stores turn off http.Client.Timeout so a large index isn't cut off part way through a transfer that is progressing normally. Bounding the wait for response headers instead left a gap: a registry that answers with headers and then stops sending body bytes, or accepts an upload and stops reading it, hung the caller forever with no way to time out. Wrap the transport so the timeout applies to the time a request spends making no progress, in either direction. Reads of the request body are how the transport puts it on the wire, so a server that stops reading is caught the same way as one that stops writing. The wrapper clones the request rather than replacing its body outright, leaving GetBody intact for the retry and re-authentication layers above it. Also restore the fetchChunkManifest doc comment, which a helper had been inserted underneath, and document that indexes over 32MiB are streamed and so lose retry and token renewal on the push. * Tighten the comments on the OCI idle timeout
Configuration menu - View commit details
-
Copy full SHA for 724481d - Browse repository at this point
Copy the full SHA 724481dView commit details -
Gate CI on golangci-lint and govulncheck (#397)
* Handle the errors errcheck and staticcheck found Mostly making an intentional discard explicit, but four were worth changing: LocalStore.StoreChunk renamed the temp file into place without checking that closing it succeeded. A close failure means the chunk was never flushed, so the store would publish a chunk it had not written. The RemoteHTTP, S3 and SFTP index stores serialize an index into a pipe from a goroutine and closed the write end with a plain Close. A failing WriteTo therefore ended the stream cleanly and the store uploaded a short index as if it were whole. They now close with the error, as the OCI index store already did. Same pattern in the tar and mtree commands. Cache.Close returned only the upstream store's error, discarding the cache store's, and FormatDecoder.Next ignored the error from advancing past an unread element, leaving the stream misaligned so the next read reported a bogus header instead of the real failure. Deprecated calls in tests replaced: cobra's SetOutput with SetOut and SetErr, minio's Credentials.Get with GetWithContext. * Gate CI on golangci-lint and govulncheck golangci-lint runs the standard set (errcheck, govet, ineffassign, staticcheck, unused) in its own job so a lint failure and a test failure are reported side by side. Two categories are turned off in .golangci.yml, both because they report style rather than defects: staticcheck's QF refactoring suggestions, and errcheck on a deferred Close, which has nowhere to report to. A Close called outright stays flagged, which is where the real bugs turned out to be. The default caps on reported issues are lifted; at three per distinct message they hid most of what was there. govulncheck reports only vulnerabilities the code actually reaches. It runs against the current stable toolchain rather than the go.mod version, since most findings are in the standard library and the fix for those is a Go upgrade. Also enables the go-vet pre-commit hook, which needs no extra tooling.
Configuration menu - View commit details
-
Copy full SHA for 005027c - Browse repository at this point
Copy the full SHA 005027cView commit details -
Lead the README with what desync is for (#398)
The page opened by describing desync as a re-implementation of casync, which only means something to a reader who already knows casync, and never said what problem any of it solves. Someone landing on the repo had no way to tell whether the tool was for them. Opens with the mechanism in plain terms instead, then the four use cases it actually gets used for, a measured example, and a comparison against the tools a reader would otherwise reach for. The numbers are from two adjacent Debian point releases exported as container root filesystems: 18.4MB downloaded against a 125.2MB image, or 50.4MB for the same image without a seed. The commands that produce them are included and run as written, and the section says plainly that the saving depends on how much changed.
Configuration menu - View commit details
-
Copy full SHA for 2d70581 - Browse repository at this point
Copy the full SHA 2d70581View commit details -
Generate the CLI reference and ship the man pages (#399)
* Generate the CLI reference and ship the man pages The README documented flags by hand, and had drifted: 27 of the 62 flags the commands define were never mentioned, and `--writeable` appeared twice for a flag actually spelled `--writable`, once in an index-server line meant to be copied and pasted. A hidden gendocs command writes a markdown page per command into docs/cli, and CI regenerates and diffs it, so the reference can only be wrong by being stale. Generation clears the directory first, otherwise the page for a deleted command would survive and the check would have nothing to catch. The README keeps its option tables, which explain rather than enumerate, and now points at the generated reference for the full flag lists. A second CI check verifies every flag the README mentions actually exists, which is the part the generated reference can't cover. Releases now carry the man pages the manpage command has always been able to produce (#225). They're generated in the goreleaser before hook rather than committed, being a build artifact. Two fixes on the way: the pages were written to section 3, for library calls, rather than section 1, so an installed page went where `man desync` wouldn't look; and neither manpage nor gendocs created the output directory cobra expects to already exist. * Keep argument placeholders in the man pages cobra builds each man page as markdown and runs it through md2man, which parses <index> as an inline HTML tag and drops it. The damage went past the synopsis, which rendered as "desync chop [flags]": prose naming an argument lost it mid-sentence too, leaving "use --ignore which will skip any chunks" and "in a text file with --ignore-chunks ." Escape the angle bracket for the duration of generation so the markdown parser emits it as literal text, then put the descriptions back, since the command tree is shared with the running CLI and the escape would otherwise reach --help. Only placeholder-shaped tokens are escaped. Escaping every angle bracket also caught the process substitution in the completion command's help, where the backslash survived into a code block and broke the command it was demonstrating. Alternation is included in the pattern: mtree takes <catar|index|dir>.
Configuration menu - View commit details
-
Copy full SHA for 5a97116 - Browse repository at this point
Copy the full SHA 5a97116View commit details -
Split the README into a docs directory (#400)
The README was 1,194 lines doing the job of nine documents, roughly 700 of them inside ten <details> blocks — collapsing sections is what a page does once it has outgrown its format. It's now a 177 line landing page: what desync is for, what it saves, how it compares, install, quick start, platform support. Everything else moved to docs/, one page per subject, with an index there and a table on the landing page. Content moved as it was rather than being rewritten, so this is a rearrangement to review, not a rewrite. Plain markdown with no build step. GitHub renders it, the links work in the repository as well as in the web UI, and docs/cli was already in this shape. A generator can be pointed at the same files later without moving anything, which is the reason not to choose one now. Deep links are preserved. GitHub generates anchors for all 58 headings on the old page, including the ones inside <details>, so links from elsewhere do resolve today. A collapsed "Where things moved" section keeps the old section names as headings, so those links still land on the README beside a pointer to the page that now holds the content. Two checks follow the content. The flag check from #399 now covers the docs pages, since 36 of the 40 flag mentions moved out of the README and it would otherwise have been left guarding four. A new test resolves every relative link and heading anchor across all 38 markdown files, which is the failure mode a set of cross-linked pages has and a single page did not.
Configuration menu - View commit details
-
Copy full SHA for acbe947 - Browse repository at this point
Copy the full SHA acbe947View commit details -
Drop the "Where things moved" section from the README (#401)
It documented the page's own history rather than the tool, and the mapping it described is already obvious from the documentation table above it.
Configuration menu - View commit details
-
Copy full SHA for e1fe559 - Browse repository at this point
Copy the full SHA e1fe559View commit details
Commits on Sep 2, 2026
-
Bump google.golang.org/grpc from 1.82.1 to 1.83.1 (#402)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.82.1 to 1.83.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.82.1...v1.83.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 2279379 - Browse repository at this point
Copy the full SHA 2279379View commit details -
Stop TestFailoverMutliple failing on a loaded machine (#404)
The test flips which of two stores fails every 10 ms while 16 goroutines query the group, and asserts no error ever reaches the caller. But GetChunk tries each store at most once per call, so a flip landing between those two attempts leaves that call having seen both stores fail, and it returns an error. That is what FailoverGroup documents it does when every store fails - the test was asking for something the implementation never promised. The window is microseconds wide, so an idle machine almost never hits it. A loaded or emulated one does: descheduling a goroutine mid-call widens the window to however long it waits. Hold the switch still for the duration of each call so a call always sees a consistent pair. Everything the test is actually for survives: 16 goroutines still hammer the group concurrently, failover still happens roughly every 10 ms, and the shared active-store state is still exercised from all of them. Since the mutex now orders the accesses, the atomics it replaces aren't needed. Reproduced beforehand with 12 concurrent copies of the test binary at GOMAXPROCS=1, which fails within a few hundred runs. The same load over 940 runs is clean with this change, as is -race.
Configuration menu - View commit details
-
Copy full SHA for 8b83694 - Browse repository at this point
Copy the full SHA 8b83694View commit details -
Support NetBSD, OpenBSD and DragonFly (#403)
* Support NetBSD, OpenBSD and DragonFly FUSE was the only thing keeping desync off these three platforms. The go-fuse bindings cover linux, darwin and freebsd; everything else in the tree already builds for the remaining BSDs through the existing platform abstractions, without a single change. So exclude the three mount files there and register the same "not available on this platform" stub Windows already used. That stub is now shared rather than windows-only, and gained a Use string so that `desync mount-index` reports why it's unavailable instead of failing with "unknown command". Device node creation gets to use the atomic path rather than Darwin's resolve-then-mknod fallback: all three have mknodat(2), typed with the device number as an int exactly as on Linux, so they can share that file. FreeBSD stays separate only because it types the argument uint64. Extended attributes are the one real gap. pkg/xattr builds its BSD implementation for freebsd and netbsd only; on OpenBSD and DragonFly it compiles in stubs that return nil and do nothing, so tar records no xattrs and untar discards them without an error. OpenBSD has no extended attributes to begin with, so that is honest there. Noted in the platform table since it is silent. File permissions need nothing: FilemodeToStatMode is built from the S_IF*/S_IS* constants, which are identical across all of these. CI grows a VM job per platform alongside the FreeBSD one, and the cross-compile smoke test now covers every BSD release target. Both echo `go version`, because the toolchain comes from each system's package repo and "too old for go.mod" is the failure worth reading off a log. Release builds add netbsd and openbsd on amd64 and arm64, and dragonfly on amd64, which is its only Go port. * Fix the NetBSD and DragonFly CI jobs Two things the VM jobs caught that cross-compiling could not. NetBSD couldn't find the toolchain at all: pkgsrc installs under /usr/pkg, which isn't on the default PATH the way FreeBSD's /usr/local/bin is, so `go` was missing despite pkg_add succeeding. DragonFly can't round-trip device nodes. mknod reports success, but the node reads back with an Rdev of NODEV (0xffffffff) for every major/minor pair the test tries, 0:0 included. TestMkdev passes there, so the encoding desync computes is right and it's the trip through the kernel that loses it. Skip the round-trip test there and say so in the platform table; nothing else in the suite is affected. Worth noting the DragonFly package is Go 1.21, well under what go.mod asks for, and the toolchain download picked up 1.25.0 for dragonfly/amd64 by itself. So the `go version` echo earned its place on the first run. * Point the NetBSD job at the versioned Go, skip VCS stamping Two follow-ups from the second CI round. The NetBSD PATH fix was aimed at the wrong cause. pkgsrc's `go` is a meta-package that installs no files of its own, and the versioned package it depends on lands as /usr/pkg/bin/go126 and /usr/pkg/go126/bin/go. Nothing anywhere provides an unversioned `go`, so no amount of PATH pointing at /usr/pkg/bin would have found it. Glob for the versioned directory instead, without hard-coding a number pkgsrc will bump. NetBSD 10.1 and 11.0 both currently carry 1.26.5. Build with -buildvcs=false in the VMs: git exits 128 on the tree rsynced into the guest, which failed the DragonFly build on version stamping alone after its tests had all passed. Release builds stamp through goreleaser's ldflags, so nothing is lost. Applied to all four VM jobs, since FreeBSD and OpenBSD passing it so far looks incidental rather than guaranteed. * Don't fail a tree walk where the filesystem has no xattrs Reading extended attributes treated every error as fatal, so a walk over a filesystem that simply doesn't implement them failed on its first entry with "operation not supported" instead of reporting a tree with no xattrs. NetBSD is where this shows up - its extattr support is per filesystem and tmpfs has none, so `desync tar` couldn't read anything under /tmp at all - but it isn't NetBSD-specific: the same is true of vfat on Linux. Tell the two cases apart and carry on when support is simply absent. A real failure to read attributes still surfaces. Note that ENOTSUP and EOPNOTSUPP are the same value on Linux but not on the BSDs, so both have to be tested. The write path is deliberately left alone: dropping attributes an archive actually carries is a loss worth an error, while finding none to read is not. * Create device nodes with mknod on NetBSD NetBSD's mknodat(2) is declared to take a uint32_t where every other BSD takes a dev_t, and the device number doesn't survive the call: nodes come back with an rdev of 0, so 1:3 read back as 0:0 and only 0:0 appeared to work. Its plain mknod(2) is the versioned __mknod50 and does carry the full dev_t, so route NetBSD through the same path Darwin already uses. The other two stay where they are. OpenBSD and DragonFly both declare mknodat with a dev_t, and OpenBSD passes the round-trip test as it stands. DragonFly's nodes come back as NODEV rather than 0, which is a different thing and not something the call convention explains, so that one keeps its skip.
Configuration menu - View commit details
-
Copy full SHA for 40c1f28 - Browse repository at this point
Copy the full SHA 40c1f28View commit details -
Verify a device node carries the number it was created with (#405)
CreateDevice passed on whatever error mknod returned and never checked the result, so a filesystem that accepts a device number without recording it produced a node pointing at the wrong device, silently and with a successful exit. Two platforms are known to do exactly that. DragonFly's mknod stores NODEV. NetBSD's did the same through mknodat until #403 switched it to mknod. Neither reported anything: what gave NetBSD away was a test that happens to run as root in a VM job, and nothing would have given away a filesystem that behaved the same way under Linux or macOS, because that test never runs there - GitHub's hosted runners aren't root. So read the node back and compare. Whether a device number survives depends on the kernel and on the filesystem underneath rather than on the operating system, so this isn't something a build tag can express. An extraction that stops and says so beats a device node pointing somewhere else. On DragonFly untar now refuses device entries instead of writing wrong ones, which is the intended trade and is noted in the platform table. Verified as root on Linux, where the round-trip test had never actually run before: it passes, and a deliberate mismatch is caught with the expected message.
Configuration menu - View commit details
-
Copy full SHA for 7a75f5a - Browse repository at this point
Copy the full SHA 7a75f5aView commit details -
Add --no-same-xattrs to untar, and stop dropping xattrs quietly (#406)
* Add --no-same-xattrs to untar, and stop dropping xattrs quietly Extracting an archive that carries extended attributes did one of three things depending on where it ran. It worked on Linux, macOS and FreeBSD. It failed on NetBSD on a filesystem without extattr support. On OpenBSD and DragonFly it reported success and wrote nothing, because pkg/xattr compiles in stubs that return nil there. The third is the one worth fixing: files end up missing what the archive said they should have, with a zero exit code. Refuse instead, which makes the three cases one, and give the caller a way to say they don't want the attributes. Ownership, permissions and timestamps could all already be opted out of; extended attributes were the exception. The flag is on untar only. tar has no owner or permission flags to match, and its --no-time exists to keep timestamps from churning chunks rather than as a metadata opt-out, so there is nothing on that side to be symmetric with. Existing behaviour worth knowing about, left alone here: --no-same-owner already skips extended attributes too, since all four call sites sit inside that check. That looks deliberate, both needing privileges, but it means --no-same-owner has been a partial opt-out all along. Windows never applied extended attributes, so the flag is accepted and ignored there like the other two. * Name the filesystem when it can't store extended attributes Failing to write an attribute reported the errno as it came, so the common case - a filesystem with no support for them - surfaced as f: xattr.FSet /mnt/f user.test: operation not supported which names neither the attributes as the cause nor the fact that they can be skipped. It is also the case most people will meet, since it covers vfat and similar under Linux rather than only the two platforms without any support at all, and those two already had a clear message. Reword that one to match, leaving every other failure to read as itself, and keep the errno wrapped so callers can still test for it. Confirmed against a vfat image, which now gives f: extended attributes are not supported by this filesystem: ... The message deliberately doesn't name --no-same-xattrs: nothing else in the library refers to a command line flag, and the reworded text is enough to find it.
Configuration menu - View commit details
-
Copy full SHA for f3f43f2 - Browse repository at this point
Copy the full SHA f3f43f2View commit details -
Build release binaries static and trimmed, ship Windows as zip (#407)
CGO_ENABLED was unset, which meant cgo was on for the one target matching the release runner (linux/amd64) and off for every other one, since cross-compiling disables it. The published v1.1.1 linux_amd64 binary is dynamically linked against glibc 2.34 while linux_arm64 from the same release is static. -trimpath drops the 1672 references to the runner's build directory that the v1.1.1 binaries carry. Windows archives become zip. tar.gz has no built-in unpacker there before Windows 11.
Configuration menu - View commit details
-
Copy full SHA for 4cbac85 - Browse repository at this point
Copy the full SHA 4cbac85View commit details -
Don't panic when a waiter gets a nil chunk from a failed request (#409)
A store that fails returns a nil chunk alongside its error. DedupQueue publishes that to everyone waiting on the same in-flight request, where it arrives as a *Chunk held in an interface. That is not an untyped nil, so the `case nil` arm never matched it: the *Chunk arm did, and cloning it dereferenced nil. The first requester already guarded against this, the waiters did not. Both waiter paths now share one helper, so they can't drift apart again, and the unreachable `case nil` that made the switch look safe is gone. Reachable through `untar -c <cache>` on Windows, where local stores are wrapped in a WriteDedupQueue, and through chunk-server on any platform, where an upstream failure took down the server process. Fixes #408.
Configuration menu - View commit details
-
Copy full SHA for 458fc17 - Browse repository at this point
Copy the full SHA 458fc17View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v1.1.1...v1.1.2