go_storage

Go Storage

A streaming storage abstraction for Go backup tools.

Supported backends:

• Local filesystem
• S3 and S3-compatible services (MinIO, Ceph, Wasabi, ...) via aws-sdk-go-v2
• Azure Blob Storage (and Azurite)
• Google Cloud Storage
• SFTP
• FTP / explicit FTPS

go get github.com/jkaninda/go-storage

Streaming API

Every backend implements the same Storage interface:

type Storage interface {
	Upload(ctx context.Context, localPath, remotePath string) error
	Download(ctx context.Context, remotePath, localPath string) error
	List(ctx context.Context, prefix string) ([]Item, error) // recursive
	Remove(ctx context.Context, remotePath string) error     // missing -> ErrNotFound
	Close() error

	Put(ctx context.Context, key string, r io.Reader, opts *PutOptions) (Item, error)
	Get(ctx context.Context, key string) (io.ReadCloser, Item, error)
	Stat(ctx context.Context, key string) (Item, error)
	PutIf(ctx context.Context, key string, r io.Reader, cond Condition, opts *PutOptions) (Item, error)
	ListDir(ctx context.Context, dir string) ([]Item, error) // non-recursive, files only
}

• Put streams a reader of unknown length with bounded memory (no temp files, no whole-file buffering). If the reader fails or the context is cancelled, Put returns an error and no object (and no partial object) is left at the key: S3 aborts the multipart upload, Azure never commits the block list, GCS cancels the resumable upload, and SFTP/FTP/local write to <key>.<random>.partial and rename on success.
• Get returns a streaming body; always Close it. On FTP the body holds the single control connection, so close it before making another call.
• PutIf is a conditional write for small files such as a backup index (optimistic concurrency): Condition{IfNoneMatch: true} creates only if the key does not exist, Condition{IfMatch: item.ETag} replaces only if the object is unchanged. It returns ErrPreconditionFailed when the condition does not hold and ErrConditionNotSupported when the backend cannot evaluate it (fall back to locking).
• Errors wrap the sentinels ErrNotFound, ErrPreconditionFailed and ErrConditionNotSupported; test them with errors.Is.
• Keys are relative to the backend base (bucket, container, remote directory or local base path), use / separators and may not escape the base (..).

S3 PutIf uses a single PutObject request and is limited to 16 MiB.

Example: dump, compress and upload without temp files

package main

import (
	"compress/gzip"
	"context"
	"errors"
	"io"
	"log"
	"os/exec"
	"strings"

	storage "github.com/jkaninda/go-storage"
)

func main() {
	ctx := context.Background()
	s, err := storage.NewStorage(storage.StorageTypeS3, storage.Config{
		S3Bucket:    "backups",
		S3Region:    "us-east-1",
		S3KeyID:     "...",
		S3Secret:    "...",
		S3Endpoint:  "https://minio.example.com", // optional, for S3-compatible services
		S3ForcePath: true,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer s.Close()

	pr, pw := io.Pipe()
	go func() {
		gz := gzip.NewWriter(pw)
		cmd := exec.CommandContext(ctx, "pg_dump", "mydb")
		cmd.Stdout = gz
		err := cmd.Run()
		if err == nil {
			err = gz.Close()
		}
		pw.CloseWithError(err) // a non-nil error aborts the upload
	}()

	item, err := s.Put(ctx, "db/mydb-2026-01-01.sql.gz", pr, &storage.PutOptions{ContentType: "application/gzip"})
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("uploaded %s (%d bytes, etag %s)", item.Key, item.Size, item.ETag)

	// Restore: stream it back.
	body, _, err := s.Get(ctx, item.Key)
	if errors.Is(err, storage.ErrNotFound) {
		log.Fatal("backup not found")
	} else if err != nil {
		log.Fatal(err)
	}
	defer body.Close()
	zr, _ := gzip.NewReader(body)
	_, _ = io.Copy(io.Discard, zr)

	// Update an index file with optimistic concurrency.
	idx, err := s.Stat(ctx, "db/index.json")
	cond := storage.Condition{IfMatch: idx.ETag}
	if errors.Is(err, storage.ErrNotFound) {
		cond = storage.Condition{IfNoneMatch: true}
	}
	_, err = s.PutIf(ctx, "db/index.json", strings.NewReader(`{"backups":[]}`), cond, nil)
	if errors.Is(err, storage.ErrPreconditionFailed) {
		// someone else updated the index: re-read, merge and retry
	}
}

Configuration

storage.NewStorage(storageType, storage.Config{...}) with storageType one of s3, gcs, azure, sftp, ftp, local.

The library never reads environment variables itself (except what the cloud SDKs do for their default credential chains, and SSH_AUTH_SOCK for the SSH agent).

Testing

go test ./...                       # unit tests: local + in-process SFTP server
go test -tags integration ./...     # + MinIO, Azurite, fake-gcs-server, FTP via docker

Integration tests skip when docker is unavailable. Images can be overridden with GO_STORAGE_MINIO_IMAGE, GO_STORAGE_AZURITE_IMAGE, GO_STORAGE_FAKEGCS_IMAGE and GO_STORAGE_FTP_IMAGE.

Deprecated: pkg/ (v0.1 API)

The packages under pkg/ (pkg/local, pkg/s3, pkg/ssh, pkg/ftp, pkg/azure) are the legacy v0.1 API. They still compile but are deprecated and will be removed in a future release; use the root package instead.

License

MIT License, Copyright (c) 2024 Jonas Kaninda. See LICENSE.