Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: folbricht/desync
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.1.1
Choose a base ref
...
head repository: folbricht/desync
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.1.2
Choose a head ref
  • 13 commits
  • 108 files changed
  • 2 contributors

Commits on Aug 30, 2026

  1. Support storing indexes in OCI registries (#396)

    * Support storing indexes in OCI registries
    
    Chunks could be kept in a registry but indexes could not, so every user
    still needed a separate channel for the caibx. A registry can now hold
    everything needed to distribute a file.
    
    The index location is the repository followed by the index name, which
    becomes the tag:
    
      desync extract -s oci+https://ghcr.io/me/store \
        oci+https://ghcr.io/me/store/file.iso.caibx file.iso
    
    That is the same split indexStoreFromLocation already applies to every
    other backend, path.Dir for the store and path.Base for the name, so no
    URL or config handling needed changing. Index names are therefore limited
    to the OCI tag grammar, which is checked at the store rather than left to
    the registry to reject.
    
    Each index is its own artifact, mirroring the chunk layout: a blob
    holding the index, referenced by a manifest tagged with the index name
    and carrying artifact type application/vnd.desync.index.v1. Annotations
    record the chunk count, indexed size and chunk sizes so the shape of an
    index can be read from the manifest without fetching it.
    
    Indexes and chunks can share a repository. Chunk operations already
    require the chunk artifact type, so prune leaves indexes alone even if
    one is named like a chunk ID, and an index never satisfies a chunk
    lookup.
    
    A registry blob must be pushed with its digest and size known upfront,
    and an index of a large blob runs to hundreds of megabytes, so the index
    is serialized to a temp file and hashed on the way rather than buffered.
    
    The repository and client setup is now shared between the chunk and index
    stores; encryption options are rejected as they are for other index
    stores.
    
    * Address review findings on the OCI index store
    
    Timeouts: the shared http.Client applied the store timeout to the whole
    exchange, body included. That suits chunks of a few hundred KB but caps
    the case this feature exists for - an index of a large blob runs to
    hundreds of megabytes, which a one minute default aborts mid-transfer.
    Index stores now bound the wait for response headers instead, so a
    stalled server still fails while a slow one is allowed to finish.
    
    Retries: the blob was pushed from an *os.File, which leaves GetBody nil,
    so oras skips its retry path and re-authentication of a request whose
    token expires mid-upload fails outright. Indexes up to 32MB are now
    pushed from a bytes.Reader, which is rewindable, and only larger ones
    stream.
    
    The temp file is gone with it. The index is serialized twice instead,
    once to size and digest it and once into the push, which needs neither
    disk nor a large buffer. $TMPDIR is RAM-backed on several distributions,
    so writing there did not achieve what the comment claimed anyway.
    
    Index names are checked before commands do their work. make and tar
    write the index last, so an unusable name previously surfaced only after
    the whole file had been chunked and uploaded.
    
    Also document that a store-options entry enabling encryption on a shared
    repository breaks index operations, and how the timeout applies.
    
    * Bound the manifest read in the OCI stores
    
    Both stores read a manifest response body with io.ReadAll. oras applies
    its MaxMetadataBytes limit on the paths that fetch other metadata, but
    FetchReference hands back the raw response body, so a registry answering
    with an endless body would be read until memory ran out.
    
    Read manifests through a shared helper that caps the body at 4MiB, oras'
    own default and far above any manifest desync writes. The body is still
    decoded whole rather than streamed off the reader: at these sizes there
    is nothing to gain from streaming, and decoding the whole body rejects
    anything trailing the manifest instead of ignoring it.
    
    * Bound OCI index transfers by inactivity rather than duration
    
    Index stores turn off http.Client.Timeout so a large index isn't cut off
    part way through a transfer that is progressing normally. Bounding the
    wait for response headers instead left a gap: a registry that answers
    with headers and then stops sending body bytes, or accepts an upload and
    stops reading it, hung the caller forever with no way to time out.
    
    Wrap the transport so the timeout applies to the time a request spends
    making no progress, in either direction. Reads of the request body are
    how the transport puts it on the wire, so a server that stops reading is
    caught the same way as one that stops writing. The wrapper clones the
    request rather than replacing its body outright, leaving GetBody intact
    for the retry and re-authentication layers above it.
    
    Also restore the fetchChunkManifest doc comment, which a helper had been
    inserted underneath, and document that indexes over 32MiB are streamed
    and so lose retry and token renewal on the push.
    
    * Tighten the comments on the OCI idle timeout
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    724481d View commit details
    Browse the repository at this point in the history
  2. Gate CI on golangci-lint and govulncheck (#397)

    * Handle the errors errcheck and staticcheck found
    
    Mostly making an intentional discard explicit, but four were worth
    changing:
    
    LocalStore.StoreChunk renamed the temp file into place without checking
    that closing it succeeded. A close failure means the chunk was never
    flushed, so the store would publish a chunk it had not written.
    
    The RemoteHTTP, S3 and SFTP index stores serialize an index into a pipe
    from a goroutine and closed the write end with a plain Close. A failing
    WriteTo therefore ended the stream cleanly and the store uploaded a
    short index as if it were whole. They now close with the error, as the
    OCI index store already did. Same pattern in the tar and mtree commands.
    
    Cache.Close returned only the upstream store's error, discarding the
    cache store's, and FormatDecoder.Next ignored the error from advancing
    past an unread element, leaving the stream misaligned so the next read
    reported a bogus header instead of the real failure.
    
    Deprecated calls in tests replaced: cobra's SetOutput with SetOut and
    SetErr, minio's Credentials.Get with GetWithContext.
    
    * Gate CI on golangci-lint and govulncheck
    
    golangci-lint runs the standard set (errcheck, govet, ineffassign,
    staticcheck, unused) in its own job so a lint failure and a test failure
    are reported side by side.
    
    Two categories are turned off in .golangci.yml, both because they report
    style rather than defects: staticcheck's QF refactoring suggestions, and
    errcheck on a deferred Close, which has nowhere to report to. A Close
    called outright stays flagged, which is where the real bugs turned out
    to be. The default caps on reported issues are lifted; at three per
    distinct message they hid most of what was there.
    
    govulncheck reports only vulnerabilities the code actually reaches. It
    runs against the current stable toolchain rather than the go.mod
    version, since most findings are in the standard library and the fix for
    those is a Go upgrade.
    
    Also enables the go-vet pre-commit hook, which needs no extra tooling.
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    005027c View commit details
    Browse the repository at this point in the history
  3. Lead the README with what desync is for (#398)

    The page opened by describing desync as a re-implementation of casync,
    which only means something to a reader who already knows casync, and
    never said what problem any of it solves. Someone landing on the repo
    had no way to tell whether the tool was for them.
    
    Opens with the mechanism in plain terms instead, then the four use cases
    it actually gets used for, a measured example, and a comparison against
    the tools a reader would otherwise reach for.
    
    The numbers are from two adjacent Debian point releases exported as
    container root filesystems: 18.4MB downloaded against a 125.2MB image,
    or 50.4MB for the same image without a seed. The commands that produce
    them are included and run as written, and the section says plainly that
    the saving depends on how much changed.
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    2d70581 View commit details
    Browse the repository at this point in the history
  4. Generate the CLI reference and ship the man pages (#399)

    * Generate the CLI reference and ship the man pages
    
    The README documented flags by hand, and had drifted: 27 of the 62 flags
    the commands define were never mentioned, and `--writeable` appeared
    twice for a flag actually spelled `--writable`, once in an index-server
    line meant to be copied and pasted.
    
    A hidden gendocs command writes a markdown page per command into
    docs/cli, and CI regenerates and diffs it, so the reference can only be
    wrong by being stale. Generation clears the directory first, otherwise
    the page for a deleted command would survive and the check would have
    nothing to catch.
    
    The README keeps its option tables, which explain rather than enumerate,
    and now points at the generated reference for the full flag lists. A
    second CI check verifies every flag the README mentions actually exists,
    which is the part the generated reference can't cover.
    
    Releases now carry the man pages the manpage command has always been
    able to produce (#225). They're generated in the goreleaser before hook
    rather than committed, being a build artifact. Two fixes on the way:
    the pages were written to section 3, for library calls, rather than
    section 1, so an installed page went where `man desync` wouldn't look;
    and neither manpage nor gendocs created the output directory cobra
    expects to already exist.
    
    * Keep argument placeholders in the man pages
    
    cobra builds each man page as markdown and runs it through md2man, which
    parses <index> as an inline HTML tag and drops it. The damage went past
    the synopsis, which rendered as "desync chop   [flags]": prose naming an
    argument lost it mid-sentence too, leaving "use --ignore  which will
    skip any chunks" and "in a text file with --ignore-chunks ."
    
    Escape the angle bracket for the duration of generation so the markdown
    parser emits it as literal text, then put the descriptions back, since
    the command tree is shared with the running CLI and the escape would
    otherwise reach --help.
    
    Only placeholder-shaped tokens are escaped. Escaping every angle bracket
    also caught the process substitution in the completion command's help,
    where the backslash survived into a code block and broke the command it
    was demonstrating. Alternation is included in the pattern: mtree takes
    <catar|index|dir>.
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    5a97116 View commit details
    Browse the repository at this point in the history
  5. Split the README into a docs directory (#400)

    The README was 1,194 lines doing the job of nine documents, roughly 700
    of them inside ten <details> blocks — collapsing sections is what a page
    does once it has outgrown its format.
    
    It's now a 177 line landing page: what desync is for, what it saves,
    how it compares, install, quick start, platform support. Everything else
    moved to docs/, one page per subject, with an index there and a table on
    the landing page. Content moved as it was rather than being rewritten,
    so this is a rearrangement to review, not a rewrite.
    
    Plain markdown with no build step. GitHub renders it, the links work in
    the repository as well as in the web UI, and docs/cli was already in
    this shape. A generator can be pointed at the same files later without
    moving anything, which is the reason not to choose one now.
    
    Deep links are preserved. GitHub generates anchors for all 58 headings
    on the old page, including the ones inside <details>, so links from
    elsewhere do resolve today. A collapsed "Where things moved" section
    keeps the old section names as headings, so those links still land on
    the README beside a pointer to the page that now holds the content.
    
    Two checks follow the content. The flag check from #399 now covers the
    docs pages, since 36 of the 40 flag mentions moved out of the README and
    it would otherwise have been left guarding four. A new test resolves
    every relative link and heading anchor across all 38 markdown files,
    which is the failure mode a set of cross-linked pages has and a single
    page did not.
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    acbe947 View commit details
    Browse the repository at this point in the history
  6. Drop the "Where things moved" section from the README (#401)

    It documented the page's own history rather than the tool, and the
    mapping it described is already obvious from the documentation table
    above it.
    folbricht authored Aug 30, 2026
    Configuration menu
    Copy the full SHA
    e1fe559 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2026

  1. Bump google.golang.org/grpc from 1.82.1 to 1.83.1 (#402)

    Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.82.1 to 1.83.1.
    - [Release notes](https://github.com/grpc/grpc-go/releases)
    - [Commits](grpc/grpc-go@v1.82.1...v1.83.1)
    
    ---
    updated-dependencies:
    - dependency-name: google.golang.org/grpc
      dependency-version: 1.83.1
      dependency-type: indirect
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    2279379 View commit details
    Browse the repository at this point in the history
  2. Stop TestFailoverMutliple failing on a loaded machine (#404)

    The test flips which of two stores fails every 10 ms while 16 goroutines
    query the group, and asserts no error ever reaches the caller. But
    GetChunk tries each store at most once per call, so a flip landing
    between those two attempts leaves that call having seen both stores
    fail, and it returns an error. That is what FailoverGroup documents it
    does when every store fails - the test was asking for something the
    implementation never promised.
    
    The window is microseconds wide, so an idle machine almost never hits
    it. A loaded or emulated one does: descheduling a goroutine mid-call
    widens the window to however long it waits.
    
    Hold the switch still for the duration of each call so a call always
    sees a consistent pair. Everything the test is actually for survives:
    16 goroutines still hammer the group concurrently, failover still
    happens roughly every 10 ms, and the shared active-store state is still
    exercised from all of them. Since the mutex now orders the accesses,
    the atomics it replaces aren't needed.
    
    Reproduced beforehand with 12 concurrent copies of the test binary at
    GOMAXPROCS=1, which fails within a few hundred runs. The same load over
    940 runs is clean with this change, as is -race.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    8b83694 View commit details
    Browse the repository at this point in the history
  3. Support NetBSD, OpenBSD and DragonFly (#403)

    * Support NetBSD, OpenBSD and DragonFly
    
    FUSE was the only thing keeping desync off these three platforms. The
    go-fuse bindings cover linux, darwin and freebsd; everything else in the
    tree already builds for the remaining BSDs through the existing
    platform abstractions, without a single change.
    
    So exclude the three mount files there and register the same "not
    available on this platform" stub Windows already used. That stub is now
    shared rather than windows-only, and gained a Use string so that
    `desync mount-index` reports why it's unavailable instead of failing
    with "unknown command".
    
    Device node creation gets to use the atomic path rather than Darwin's
    resolve-then-mknod fallback: all three have mknodat(2), typed with the
    device number as an int exactly as on Linux, so they can share that
    file. FreeBSD stays separate only because it types the argument uint64.
    
    Extended attributes are the one real gap. pkg/xattr builds its BSD
    implementation for freebsd and netbsd only; on OpenBSD and DragonFly it
    compiles in stubs that return nil and do nothing, so tar records no
    xattrs and untar discards them without an error. OpenBSD has no
    extended attributes to begin with, so that is honest there. Noted in
    the platform table since it is silent.
    
    File permissions need nothing: FilemodeToStatMode is built from the
    S_IF*/S_IS* constants, which are identical across all of these.
    
    CI grows a VM job per platform alongside the FreeBSD one, and the
    cross-compile smoke test now covers every BSD release target. Both echo
    `go version`, because the toolchain comes from each system's package
    repo and "too old for go.mod" is the failure worth reading off a log.
    Release builds add netbsd and openbsd on amd64 and arm64, and dragonfly
    on amd64, which is its only Go port.
    
    * Fix the NetBSD and DragonFly CI jobs
    
    Two things the VM jobs caught that cross-compiling could not.
    
    NetBSD couldn't find the toolchain at all: pkgsrc installs under
    /usr/pkg, which isn't on the default PATH the way FreeBSD's
    /usr/local/bin is, so `go` was missing despite pkg_add succeeding.
    
    DragonFly can't round-trip device nodes. mknod reports success, but the
    node reads back with an Rdev of NODEV (0xffffffff) for every major/minor
    pair the test tries, 0:0 included. TestMkdev passes there, so the
    encoding desync computes is right and it's the trip through the kernel
    that loses it. Skip the round-trip test there and say so in the platform
    table; nothing else in the suite is affected.
    
    Worth noting the DragonFly package is Go 1.21, well under what go.mod
    asks for, and the toolchain download picked up 1.25.0 for
    dragonfly/amd64 by itself. So the `go version` echo earned its place on
    the first run.
    
    * Point the NetBSD job at the versioned Go, skip VCS stamping
    
    Two follow-ups from the second CI round.
    
    The NetBSD PATH fix was aimed at the wrong cause. pkgsrc's `go` is a
    meta-package that installs no files of its own, and the versioned
    package it depends on lands as /usr/pkg/bin/go126 and
    /usr/pkg/go126/bin/go. Nothing anywhere provides an unversioned `go`,
    so no amount of PATH pointing at /usr/pkg/bin would have found it.
    Glob for the versioned directory instead, without hard-coding a number
    pkgsrc will bump. NetBSD 10.1 and 11.0 both currently carry 1.26.5.
    
    Build with -buildvcs=false in the VMs: git exits 128 on the tree rsynced
    into the guest, which failed the DragonFly build on version stamping
    alone after its tests had all passed. Release builds stamp through
    goreleaser's ldflags, so nothing is lost. Applied to all four VM jobs,
    since FreeBSD and OpenBSD passing it so far looks incidental rather than
    guaranteed.
    
    * Don't fail a tree walk where the filesystem has no xattrs
    
    Reading extended attributes treated every error as fatal, so a walk over
    a filesystem that simply doesn't implement them failed on its first
    entry with "operation not supported" instead of reporting a tree with no
    xattrs. NetBSD is where this shows up - its extattr support is per
    filesystem and tmpfs has none, so `desync tar` couldn't read anything
    under /tmp at all - but it isn't NetBSD-specific: the same is true of
    vfat on Linux.
    
    Tell the two cases apart and carry on when support is simply absent. A
    real failure to read attributes still surfaces. Note that ENOTSUP and
    EOPNOTSUPP are the same value on Linux but not on the BSDs, so both have
    to be tested.
    
    The write path is deliberately left alone: dropping attributes an
    archive actually carries is a loss worth an error, while finding none to
    read is not.
    
    * Create device nodes with mknod on NetBSD
    
    NetBSD's mknodat(2) is declared to take a uint32_t where every other BSD
    takes a dev_t, and the device number doesn't survive the call: nodes come
    back with an rdev of 0, so 1:3 read back as 0:0 and only 0:0 appeared to
    work. Its plain mknod(2) is the versioned __mknod50 and does carry the
    full dev_t, so route NetBSD through the same path Darwin already uses.
    
    The other two stay where they are. OpenBSD and DragonFly both declare
    mknodat with a dev_t, and OpenBSD passes the round-trip test as it
    stands. DragonFly's nodes come back as NODEV rather than 0, which is a
    different thing and not something the call convention explains, so that
    one keeps its skip.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    40c1f28 View commit details
    Browse the repository at this point in the history
  4. Verify a device node carries the number it was created with (#405)

    CreateDevice passed on whatever error mknod returned and never checked
    the result, so a filesystem that accepts a device number without
    recording it produced a node pointing at the wrong device, silently and
    with a successful exit.
    
    Two platforms are known to do exactly that. DragonFly's mknod stores
    NODEV. NetBSD's did the same through mknodat until #403 switched it to
    mknod. Neither reported anything: what gave NetBSD away was a test that
    happens to run as root in a VM job, and nothing would have given away a
    filesystem that behaved the same way under Linux or macOS, because that
    test never runs there - GitHub's hosted runners aren't root.
    
    So read the node back and compare. Whether a device number survives
    depends on the kernel and on the filesystem underneath rather than on
    the operating system, so this isn't something a build tag can express.
    An extraction that stops and says so beats a device node pointing
    somewhere else.
    
    On DragonFly untar now refuses device entries instead of writing wrong
    ones, which is the intended trade and is noted in the platform table.
    
    Verified as root on Linux, where the round-trip test had never actually
    run before: it passes, and a deliberate mismatch is caught with the
    expected message.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    7a75f5a View commit details
    Browse the repository at this point in the history
  5. Add --no-same-xattrs to untar, and stop dropping xattrs quietly (#406)

    * Add --no-same-xattrs to untar, and stop dropping xattrs quietly
    
    Extracting an archive that carries extended attributes did one of three
    things depending on where it ran. It worked on Linux, macOS and FreeBSD.
    It failed on NetBSD on a filesystem without extattr support. On OpenBSD
    and DragonFly it reported success and wrote nothing, because pkg/xattr
    compiles in stubs that return nil there.
    
    The third is the one worth fixing: files end up missing what the archive
    said they should have, with a zero exit code. Refuse instead, which
    makes the three cases one, and give the caller a way to say they don't
    want the attributes. Ownership, permissions and timestamps could all
    already be opted out of; extended attributes were the exception.
    
    The flag is on untar only. tar has no owner or permission flags to
    match, and its --no-time exists to keep timestamps from churning chunks
    rather than as a metadata opt-out, so there is nothing on that side to
    be symmetric with.
    
    Existing behaviour worth knowing about, left alone here: --no-same-owner
    already skips extended attributes too, since all four call sites sit
    inside that check. That looks deliberate, both needing privileges, but
    it means --no-same-owner has been a partial opt-out all along.
    
    Windows never applied extended attributes, so the flag is accepted and
    ignored there like the other two.
    
    * Name the filesystem when it can't store extended attributes
    
    Failing to write an attribute reported the errno as it came, so the
    common case - a filesystem with no support for them - surfaced as
    
      f: xattr.FSet /mnt/f user.test: operation not supported
    
    which names neither the attributes as the cause nor the fact that they
    can be skipped. It is also the case most people will meet, since it
    covers vfat and similar under Linux rather than only the two platforms
    without any support at all, and those two already had a clear message.
    
    Reword that one to match, leaving every other failure to read as itself,
    and keep the errno wrapped so callers can still test for it. Confirmed
    against a vfat image, which now gives
    
      f: extended attributes are not supported by this filesystem: ...
    
    The message deliberately doesn't name --no-same-xattrs: nothing else in
    the library refers to a command line flag, and the reworded text is
    enough to find it.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    f3f43f2 View commit details
    Browse the repository at this point in the history
  6. Build release binaries static and trimmed, ship Windows as zip (#407)

    CGO_ENABLED was unset, which meant cgo was on for the one target
    matching the release runner (linux/amd64) and off for every other one,
    since cross-compiling disables it. The published v1.1.1 linux_amd64
    binary is dynamically linked against glibc 2.34 while linux_arm64 from
    the same release is static.
    
    -trimpath drops the 1672 references to the runner's build directory
    that the v1.1.1 binaries carry.
    
    Windows archives become zip. tar.gz has no built-in unpacker there
    before Windows 11.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    4cbac85 View commit details
    Browse the repository at this point in the history
  7. Don't panic when a waiter gets a nil chunk from a failed request (#409)

    A store that fails returns a nil chunk alongside its error. DedupQueue
    publishes that to everyone waiting on the same in-flight request, where
    it arrives as a *Chunk held in an interface. That is not an untyped nil,
    so the `case nil` arm never matched it: the *Chunk arm did, and cloning
    it dereferenced nil.
    
    The first requester already guarded against this, the waiters did not.
    Both waiter paths now share one helper, so they can't drift apart again,
    and the unreachable `case nil` that made the switch look safe is gone.
    
    Reachable through `untar -c <cache>` on Windows, where local stores are
    wrapped in a WriteDedupQueue, and through chunk-server on any platform,
    where an upstream failure took down the server process. Fixes #408.
    folbricht authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    458fc17 View commit details
    Browse the repository at this point in the history
Loading
⌂ 换网址