Documentation
¶
Overview ¶
Package sftp provides an object store on a remote directory reached through an SFTP client (github.com/pkg/sftp over golang.org/x/crypto/ssh). It ports the layout and commit protocol of the filesystem backend (backend/fs) to SFTP requests.
Layout ¶
Under the root directory, per bucket:
b-<hex bucket>/sftp.lock write lock file b-<hex bucket>/tmp/ staging files b-<hex bucket>/objects/<key segments>/@object.json metadata (commit point) b-<hex bucket>/objects/<key segments>/@data-<version> immutable bytes b-<hex bucket>/uploads/<upload id>/ multipart parts
Every key segment is a directory, so keys such as "a" and "a/b" coexist. Segments beginning with "@" are stored with one extra leading "@", which keeps reserved "@object.json" and "@data-*" names out of the key space. The object layout matches backend/fs, so objects written by one backend can be read by the other; the two backends lock differently (backend/fs uses OS file locks on b-<hex bucket>/lock), so they must not write to the same directory concurrently.
Writes and reads ¶
Put streams the body into a remote staging file while hashing it (with pipelined SFTP writes, see pkg/sftp File.ReadFromWithConcurrency), verifies PutOptions.Size and PutOptions.SHA256, and only then publishes: under the bucket lock it renames the staging file to a fresh @data-<version> file and replaces @object.json, which names that data file. Replacing @object.json is the single commit point. When the server advertises posix-rename@openssh.com (OpenSSH and pkg/sftp servers do), the replacement is an atomic rename(2): readers never observe partial objects or mismatched data and metadata, and a crash leaves either the old or the new version. Without the extension the store first tries SSH_FXP_RENAME, which many servers implement with rename(2). Where the server refuses to rename over an existing file (SFTP v3 semantics, as in OpenSSH), it renames the current @object.json to @object.previous.json, renames the new file into place, and removes the previous file. That fallback is not one atomic step: lock-free readers that miss @object.json read @object.previous.json instead (alternating between the two names for a few rounds), so they observe the old or the new version, and a crash between the renames leaves the old version readable; but a reader that loses every round to back-to-back publications reports ErrNotFound, absent keys cost up to six metadata requests, and a crash after the second rename can leave an orphaned data file. Files are fsync'ed before publication only when the server advertises fsync@openssh.com.
Readers take no lock. Get reads @object.json and opens the immutable data file it names, retrying when a concurrent writer replaced (and removed) that version in between. Ranged reads use ReadAt on the open data file. An open file stays readable after a concurrent replacement on POSIX servers; servers that cannot remove open files (Windows) leave the old data file behind.
Conditional writes and locking ¶
IfAbsent, IfMatchVersion, versioned Delete, multipart completion, Copy destination conditions, and DeleteMany are evaluated and published under a per-bucket lock. Within one Store the lock is an in-process semaphore, so goroutines queue without polling. Across processes (or Store values) the lock is the file b-<hex bucket>/sftp.lock, created with SSH_FXF_CREAT| SSH_FXF_EXCL and acquired with context-aware, jittered polling backoff (Options.LockPollInterval). The holder rewrites a heartbeat counter in the lock file every StaleLockAge/4; a waiter that sees unchanged lock contents for Options.StaleLockAge on its own monotonic clock treats the holder as dead and removes the lock (best-effort, the check and removal are not atomic).
Guarantees: conditional writes are exact (linearizable compare-and-swap) among goroutines sharing one Store, and among processes as long as the server implements exclusive create atomically (open(O_EXCL) on a local filesystem does; some network filesystems behind the server may not) and no live lock is broken as stale. A holder whose heartbeats cannot reach the server for StaleLockAge (for example a stalled connection) can lose its lock, after which concurrent conditional writes are best-effort. A crashed holder delays other writers for up to StaleLockAge.
Multipart, copies, and batches ¶
Multipart parts are verified while they are written and stored as files in the upload directory. CompleteMultipart assembles them by streaming every part from the server through the client into a staging file while re-hashing it (SFTP has no portable server-side concatenation), so completion transfers the object twice more over the connection. Copy hard-links the immutable source data file with hardlink@openssh.com when the server advertises it (no data transfer); otherwise, or when linking fails, it streams the bytes through the client and verifies them against the source digest. StatMany reads metadata key by key; DeleteMany deletes under one bucket lock. SFTP has no presigned URLs, so Store does not implement objectstore.Presigner.
Contexts ¶
SFTP requests are not context-aware. The store checks the context before and between requests, and wraps every streamed body (uploads, part assembly, copies, and the reader returned by Get) in a reader that fails with the context error once the context ends. A request already sent to the server runs to completion. Bodies returned by Get are bound to the Get context.
Limitations ¶
Keys map to path segments, so empty or "." segments are rejected, case-insensitive server filesystems fold keys that differ only in case, and segment lengths and characters are bounded by the server filesystem. List reads one metadata file per listed key. Crashed writers leave staging files in tmp/ that are never published; they can be deleted when no write is in flight.
Index ¶
- Constants
- type Options
- type Store
- func (store *Store) AbortMultipart(ctx context.Context, upload objectstore.Upload) error
- func (store *Store) AtomicReplace() bool
- func (store *Store) BeginMultipart(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, ...) (objectstore.Upload, error)
- func (store *Store) CompleteMultipart(ctx context.Context, upload objectstore.Upload, ...) (objectstore.Object, error)
- func (store *Store) Copy(ctx context.Context, srcBucket objectstore.Bucket, srcKey objectstore.Key, ...) (objectstore.Object, error)
- func (store *Store) Delete(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, ...) error
- func (store *Store) DeleteMany(ctx context.Context, bucket objectstore.Bucket, keys []objectstore.Key) ([]error, error)
- func (store *Store) Get(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, ...) (io.ReadCloser, objectstore.Object, error)
- func (store *Store) List(ctx context.Context, bucket objectstore.Bucket, ...) (objectstore.Page, error)
- func (store *Store) Put(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, ...) (objectstore.Object, error)
- func (store *Store) PutPart(ctx context.Context, upload objectstore.Upload, number int, body io.Reader, ...) (objectstore.CompletedPart, error)
- func (store *Store) Root() string
- func (store *Store) Stat(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key) (objectstore.Object, error)
- func (store *Store) StatMany(ctx context.Context, bucket objectstore.Bucket, keys []objectstore.Key) ([]objectstore.StatResult, error)
Examples ¶
Constants ¶
const ( DefaultStaleLockAge = time.Minute DefaultLockPollInterval = 100 * time.Millisecond )
Default option values.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
// StaleLockAge is how long a waiter must observe unchanged bucket lock
// contents before it removes the lock as abandoned. Lock holders
// heartbeat every StaleLockAge/4. Zero selects DefaultStaleLockAge; a
// negative value disables stale-lock breaking (and heartbeats), so a
// crashed holder blocks conditional writes until its lock file is
// removed by hand.
StaleLockAge time.Duration
// LockPollInterval caps the backoff between attempts to create a bucket
// lock file held by another process. Attempts start after about 1ms and
// double up to this cap. Zero selects DefaultLockPollInterval.
LockPollInterval time.Duration
}
Options configures a Store. The zero value selects the defaults.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an object store in one directory of an SFTP server. It is safe for concurrent use by many goroutines, and by many processes sharing the remote directory within the guarantees described in the package documentation.
func New ¶
New opens a store rooted at the remote directory root, creating it if needed. A relative root is resolved by the server, normally against the login directory. The client stays owned by the caller, who must keep it open while the store is used and close it afterwards.
Example ¶
package main
import (
"bytes"
"context"
"fmt"
"io"
objectstore "github.com/justusschock/object-storage"
sftpadapter "github.com/justusschock/object-storage/backend/sftp"
"github.com/justusschock/object-storage/backend/sftp/sftptest"
)
func main() {
// An in-process SSH server on 127.0.0.1 stands in for a real SFTP host.
server, err := sftptest.Start(sftptest.Options{})
if err != nil {
panic(err)
}
defer func() { _ = server.Close() }()
client, err := server.Dial(server.ClientConfig())
if err != nil {
panic(err)
}
defer func() { _ = client.Close() }()
store, err := sftpadapter.New(client, "objects-root", sftpadapter.Options{})
if err != nil {
panic(err)
}
ctx := context.Background()
first, err := store.Put(ctx, "bucket", "greeting", bytes.NewReader([]byte("hello")), objectstore.PutOptions{IfAbsent: true})
if err != nil {
panic(err)
}
_, err = store.Put(ctx, "bucket", "greeting", bytes.NewReader([]byte("hello again")), objectstore.PutOptions{IfAbsent: true})
fmt.Println("second IfAbsent put:", err)
if _, err := store.Put(ctx, "bucket", "greeting", bytes.NewReader([]byte("hello, world")), objectstore.PutOptions{IfMatchVersion: first.Version}); err != nil {
panic(err)
}
body, object, err := store.Get(ctx, "bucket", "greeting", objectstore.GetOptions{Range: &objectstore.ByteRange{Offset: 7, Length: 5}})
if err != nil {
panic(err)
}
defer func() { _ = body.Close() }()
data, err := io.ReadAll(body)
if err != nil {
panic(err)
}
fmt.Println(string(data), object.Size, store.AtomicReplace())
}
Output: second IfAbsent put: object storage: precondition failed world 12 true
func (*Store) AbortMultipart ¶
AbortMultipart discards an active upload without changing complete objects.
func (*Store) AtomicReplace ¶
AtomicReplace reports whether the server advertised posix-rename@openssh.com, which makes publication an atomic replacement. Without it the store uses SSH_FXP_RENAME and, on servers that refuse to rename over an existing file, the non-atomic two-rename fallback described in the package documentation.
func (*Store) BeginMultipart ¶
func (store *Store) BeginMultipart(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, options objectstore.PutOptions) (objectstore.Upload, error)
BeginMultipart creates isolated multipart state without mutating objects.
func (*Store) CompleteMultipart ¶
func (store *Store) CompleteMultipart(ctx context.Context, upload objectstore.Upload, parts []objectstore.CompletedPart) (objectstore.Object, error)
CompleteMultipart streams the ordered parts from the server through the client into a staging file while re-verifying their digests, then publishes atomically with the upload's put conditions.
func (*Store) Copy ¶
func (store *Store) Copy(ctx context.Context, srcBucket objectstore.Bucket, srcKey objectstore.Key, dstBucket objectstore.Bucket, dstKey objectstore.Key, options objectstore.CopyOptions) (objectstore.Object, error)
Copy publishes the source bytes under the destination key. It hard-links the immutable source data file when the server supports hardlink@openssh.com and otherwise streams the bytes through the client, verifying them against the source digest.
func (*Store) Delete ¶
func (store *Store) Delete(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, version string) error
Delete removes the matching object version under the bucket lock; an empty version is unconditional. Absent objects return ErrNotFound.
func (*Store) DeleteMany ¶
func (store *Store) DeleteMany(ctx context.Context, bucket objectstore.Bucket, keys []objectstore.Key) ([]error, error)
DeleteMany removes every listed key under one bucket lock; absent keys succeed. Cancellation stops before the next key; keys already deleted stay deleted.
func (*Store) Get ¶
func (store *Store) Get(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, options objectstore.GetOptions) (io.ReadCloser, objectstore.Object, error)
Get opens an object or exact byte range. The returned body stays consistent with the returned metadata even if the object is replaced concurrently, and fails with the context error once ctx ends.
func (*Store) List ¶
func (store *Store) List(ctx context.Context, bucket objectstore.Bucket, options objectstore.ListOptions) (objectstore.Page, error)
List returns a stable key-ordered page. Every listed object carries full metadata, because the metadata file is read anyway to report the version. It walks only the directories that can contain keys after options.After with options.Prefix.
func (*Store) Put ¶
func (store *Store) Put(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key, body io.Reader, options objectstore.PutOptions) (objectstore.Object, error)
Put streams body into a remote staging file while hashing it, then publishes it under the bucket lock.
func (*Store) PutPart ¶
func (store *Store) PutPart(ctx context.Context, upload objectstore.Upload, number int, body io.Reader, size int64, digest [32]byte) (objectstore.CompletedPart, error)
PutPart verifies and stores one numbered part. Uploading a part number again replaces the earlier part.
func (*Store) Stat ¶
func (store *Store) Stat(ctx context.Context, bucket objectstore.Bucket, key objectstore.Key) (objectstore.Object, error)
Stat returns object metadata without opening object bytes.
func (*Store) StatMany ¶
func (store *Store) StatMany(ctx context.Context, bucket objectstore.Bucket, keys []objectstore.Key) ([]objectstore.StatResult, error)
StatMany returns metadata for every key in input order, reading one metadata file per key.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package sftptest runs an in-process SSH server with the SFTP subsystem on a 127.0.0.1 loopback listener, for tests of code that uses SFTP clients.
|
Package sftptest runs an in-process SSH server with the SFTP subsystem on a 127.0.0.1 loopback listener, for tests of code that uses SFTP clients. |