Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: folbricht/desync
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.1.0
Choose a base ref
...
head repository: folbricht/desync
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.1.1
Choose a head ref
  • 10 commits
  • 27 files changed
  • 1 contributor

Commits on Aug 29, 2026

  1. Drop the cgo zstd implementation (#386)

    Compression had two implementations: the pure-Go klauspost/compress
    default, and a DataDog/zstd variant selected with the "datadog" build
    tag. The cgo variant meant the build could produce binaries needing a C
    toolchain, which complicates cross-compilation for no benefit that the
    pure-Go implementation doesn't already provide.
    
    Remove compress_datadog.go and the build constraint on compress.go, so
    the pure-Go implementation is the only one, and drop the now-unused
    DataDog/zstd dependency.
    
    The release matrix (linux, darwin, windows on amd64/arm64) plus
    linux/arm and linux/riscv64 all build with CGO_ENABLED=0, and no
    documentation referenced the build tag.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    6b6fcfa View commit details
    Browse the repository at this point in the history
  2. Support FreeBSD (#387)

    go-fuse v2.2.0 only builds on Linux and macOS, so the FreeBSD build was
    broken despite the README listing BSD as expected to work. Upgrade it to
    v2.11.0, which supports FreeBSD. No source changes were needed for the
    upgrade itself, the fs and fuse APIs desync uses are unchanged.
    
    That uncovered a second problem: createDeviceNode for non-Linux passed
    the device number to unix.Mknod as an int, but FreeBSD takes a uint64.
    FreeBSD has mknodat(2), so rather than widening the fallback, give it its
    own implementation using the same Root-relative creation as Linux, which
    also confines it better than the darwin path that resolves the parent and
    then creates on the real path.
    
    FreeBSD is now built by goreleaser and validated in CI: a cross-compile
    in the main matrix for a fast signal, and the full test suite in a
    FreeBSD VM. NetBSD, OpenBSD and DragonFly remain unsupported since
    go-fuse does not build there, so the README now says so instead of
    lumping all BSDs together as untested.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    9aafd9b View commit details
    Browse the repository at this point in the history
  3. Use the platform's own device number encoding (#388)

    mkdev built device numbers with Linux's dev_t layout on every platform,
    and the read side split Rdev with Linux's layout too. The encodings are
    not the same everywhere, so on macOS this produced wrong device nodes:
    a catar entry for major 1, minor 3 was created as major 0, minor 259,
    and reading /dev/null back reported major 0, minor 12290 rather than
    3, 2. FreeBSD's encoding happens to agree with Linux's for the small
    major/minor values in practical use, so it was unaffected.
    
    Defer to x/sys/unix, which implements Mkdev, Major and Minor per
    platform. On Linux the encoding is unchanged; the read path there now
    also stops truncating majors above 12 bits.
    
    Encoding now returns an error when a major or minor from the archive is
    too large for the platform to represent, instead of silently truncating
    it to a different device.
    
    TestMkdev covers the encode/decode round-trip on every platform and
    fails on macOS with the old layout. TestCreateDeviceRoundTrip creates
    real device nodes and reads them back, which needs root, so it runs in
    the FreeBSD CI job and skips elsewhere.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    8949ae9 View commit details
    Browse the repository at this point in the history
  4. Upgrade minio-go from v6 to v7 (#389)

    minio-go v6 has been end of life for years. v7 moves every object
    operation onto a context and renames a few things, so the store code
    changes with it:
    
      - minio.NewWithOptions is now minio.New
      - GetObject, PutObject, StatObject and RemoveObject take a context
      - ListObjectsV2 with a done channel is now ListObjects with a
        ListObjectsOptions, cancelled through its context
      - credentials.Provider gained RetrieveWithCredContext, which the
        static and refreshable providers now implement alongside Retrieve
    
    The store interface has no context to pass down, so the per-object calls
    use context.Background(), which is what v6 did internally. Prune already
    receives one and now uses it to bound the listing as well as the loop.
    
    Dependencies pulled up by v7's own requirements: klauspost/compress,
    logrus, cobra, pflag, x/crypto and x/sys.
    
    Verified against a real MinIO server as well as the existing fake-S3
    tests: make, extract, info, prune and a prefixed store all round-trip,
    indexes stored in S3 are read back and survive a prune of the same
    bucket.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    627a467 View commit details
    Browse the repository at this point in the history
  5. Apply the timeout store option to S3 stores (#390)

    The timeout option was documented for chunk read/write but S3 stores
    ignored it: unlike the HTTP and OCI stores, which set it on their
    http.Client, the minio client takes a context per operation and was
    given context.Background(). An unresponsive endpoint would hang for as
    long as it held the connection open.
    
    Add StoreOptions.contextWithTimeout, which derives a context from the
    same effectiveTimeout() the HTTP stores use, and take care that a
    negative (infinite) timeout produces a context with no deadline rather
    than one that is already expired.
    
    Chunk reads and writes get a fresh timeout per attempt, so it bounds
    each try rather than the whole retry sequence, and it covers the body
    read since minio only issues the request on first read. Index reads
    return a reader that the caller consumes later, so the context is
    cancelled when that reader is closed. The listing in Prune keeps the
    caller's context, since a per-object timeout has no meaning for an
    operation that walks the whole store.
    
    Note this is a behaviour change for existing S3 users: operations that
    previously had no deadline now default to one minute, matching the HTTP
    stores. Set a negative timeout for the old behaviour.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    1aeccdc View commit details
    Browse the repository at this point in the history
  6. Add a version command (#391)

    There was no way to tell which build of desync was running, which makes
    a bug report hard to place. Add a 'version' command printing the
    version, commit, build time, Go toolchain and platform, in plain text or
    JSON, plus --version on the root command as a shorthand.
    
    The version is taken from -X main.version at link time when present, and
    otherwise from the build information the toolchain embeds, so every
    install path reports something useful without a build step:
    
      - 'go install ...@v1.1.0' has no ldflags, but the module version is
        embedded, giving v1.1.0
      - a build from a checkout gets a pseudo-version derived from the last
        tag plus vcs.revision and vcs.time, and +dirty for a modified tree
      - release builds get the exact tag, since the ldflags are now set
        explicitly in .goreleaser.yml rather than left to its defaults
    
    A build from a source tarball has neither, and reports the toolchain's
    own "(devel)".
    
    --version is registered explicitly rather than left to cobra, which
    would otherwise claim -v for it. Users would reasonably read -v as
    --verbose.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    00b3c6b View commit details
    Browse the repository at this point in the history
  7. Release binaries for 32-bit ARM and riscv64 (#392)

    Both architectures build cleanly and are a natural fit for the
    embedded/appliance update case, but neither was in the release matrix,
    so those users had to build from source.
    
    Add arm and riscv64, restricted to Linux. Of the other platforms only
    freebsd/arm is a combination Go supports at all, and FreeBSD is already
    untested at runtime, so a second architecture there would be an artifact
    nobody exercises. goreleaser skips the combinations Go doesn't support
    on its own; the ignore rule is for the deliberate omission.
    
    Both ARM variants are built: v6 runs anywhere 32-bit ARM does, and v7 is
    faster on hardware that supports it, which is worth an extra artifact
    for a tool whose chunking is CPU-bound.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    f70904e View commit details
    Browse the repository at this point in the history
  8. Produce byte-identical indexes to casync (#393)

    Two differences kept desync's blob indexes from matching casync's for
    the same input, despite the README claiming identical output.
    
    The chunker never tested the rolling hash window that ends exactly at the
    min chunk size. Both tools apply the same optimization of leaving the
    first min-48 bytes unhashed, but casync's ca_chunker_scan() seeds the
    window and immediately tests it, so a boundary there yields a chunk of
    exactly min bytes. desync seeded the identical hash and rolled a byte in
    before its first test, making min+1 the smallest chunk it could produce.
    Whenever a boundary fell on min the two disagreed: on a 123MB tar that
    happened once in 1699 chunks, casync emitting 16384+56320 where desync
    emitted a single 72704. Test the seeded window before entering the loop.
    
    The blob index header also omitted CaFormatExcludeFile, which casync
    includes in CA_FORMAT_DEFAULT and stamps into every index, so the feature
    flags read 0xa000000000000000 rather than 0xb000000000000000 and the
    files differed in one byte even when the chunking agreed. The flag means
    the encoder honors .caexclude files while walking a directory tree, so it
    is inert for a blob and only set on that path. Archive indexes keep their
    current flags: desync doesn't implement .caexclude (#114) and shouldn't
    claim to, and their flags differ from casync's regardless because desync
    stores a smaller catar feature set.
    
    With both fixed, casync and desync produce byte-identical caibx files for
    the same input, verified for default and custom chunk sizes and for both
    digest algorithms.
    
    Note for existing stores: the chunker change alters boundaries in the
    rare case above, so re-chunking data already in a store will not dedup
    against those few chunks. Existing indexes and chunks stay valid and
    readable.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    6696dc1 View commit details
    Browse the repository at this point in the history
  9. Correct performance and option claims in the README (#394)

    Several statements no longer matched the code or measurement.
    
    The "up to 10x faster" claim had no qualifier. The speedup comes from
    chunking parts concurrently, so it scales with core count: on a 4-core
    machine make measured 1.8x-3.6x faster than casync 2, and single
    threaded desync is in casync's range or slightly behind, since its
    rolling hash is pure Go against casync's optimized C. Say so, rather
    than leaving readers to expect 10x anywhere.
    
    The per-command table called all-zero data the worst case for make and
    "likely comparable to casync". The mechanism it describes is real, but
    nothing supports the comparison, so keep the former and drop the latter.
    
    Also:
    
      - info defaults to json output, not plain
      - the chunker accepts a min chunk size equal to the rolling hash
        window, not only larger than it (chunker.go:135)
      - the timeout store option is only consulted by the HTTP(S), S3 and
        OCI stores; GCS, SFTP and local ignore it
      - device numbers on macOS were fixed in #388, so only the filemode
        half of the catar interop caveat still stands
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    21a747b View commit details
    Browse the repository at this point in the history
  10. Stop calling desync a drop-in replacement for casync (#395)

    The intro claimed compatibility with casync's data structures "to
    function as a drop-in replacement". The data compatibility is real and
    worth stating, but the CLI is not interchangeable: the options differ and
    desync has commands casync doesn't. The CLI Reference already said as
    much, so the intro contradicted it.
    
    The remaining mention, in Design Philosophy, is a narrow and accurate
    one - desync can stand in for the casync binary on an SSH server serving
    chunks read-only, which is what the pull command implements. Only its
    bold lead-in read like the general claim, so scope that to what it
    describes.
    folbricht authored Aug 29, 2026
    Configuration menu
    Copy the full SHA
    1b2038d View commit details
    Browse the repository at this point in the history
Loading
⌂ 换网址