GCS store: desync make fails with "object doesn't exist" on chunks that should be uploaded #358

Problem

gcs.go detects "object not found" using direct equality (err == storage.ErrObjectNotExist) in three places. With a newer cloud.google.com/go/storage that returns the not-found error wrapped rather than as the bare sentinel, the equality check no longer matches, so a routine "chunk not present" is misclassified as a generic error. During desync make against a GCS store this turns a normal cache-miss into a fatal abort, whereas it should trigger an upload.

What happens

desync make --store gs://<bucket> <index>.caibx <blob> aborts when it encounters a chunk that is not already in the bucket:

Error: storage: object doesn't exist: googleapi: Error 404: No such object: <bucket>/<prefix>/<hash>.cacnk, notFound

The chunk legitimately does not exist yet -- it's a new chunk that make should be uploading. gsutil ls confirms the object is absent.

This was reproduced on master (commit 86eb6d01). The relevant dependency change appears to be the gRPC / Google Cloud bump in commit a452c77, which moved cloud.google.com/go/storage from v1.30.1 to v1.60.0. An earlier master snapshot (commit 0e2fa367, on the prior storage-client version) does not exhibit the issue.

Root cause

In gcs.go, all three not-found checks use ==:

if err == storage.ErrObjectNotExist {
    // treat as missing
} else if err != nil {
    // treat as fatal
}

This is in the reader-creation and read paths of GetChunk, and in HasChunk. HasChunk is the one that bites make: when a chunk is absent it should return (false, nil) so the caller uploads it, but against a wrapped error the == is false, control falls through to the err != nil branch, and the not-found propagates as fatal.

This usually stays latent because make only hits the not-found branch when a chunk isn't already in the store, so a store that already contains the chunks never exercises it.

The S3 backend isn't affected because S3Store.HasChunk returns err == nil, nil, i.e. it doesn't depend on matching this sentinel, so the S3 path doesn't have the equivalent failure.

A potential fix may be to use errors.Is, which unwraps:

if errors.Is(err, storage.ErrObjectNotExist) {

in all three locations in gcs.go.

To reproduce

1. Build desync against a cloud.google.com/go/storage version that wraps ErrObjectNotExist.
2. desync make --store gs://<bucket> /tmp/x.caibx <a file containing data not already chunked into the bucket>
3. Observe the fatal 404 on the first new chunk.