Documentation
¶
Overview ¶
Package remotefs downloads from servers a person already owns: a seedbox over FTP or FTPS, a NAS over SFTP, a Nextcloud or ownCloud over WebDAV.
The protocols share nothing but a tree that can be listed, a file size that can be asked for and a byte stream that can start at an offset. That is the FS interface, and it is why the package is named after a file system rather than after its protocols.
Credentials never travel in a link: a URL with a password is refused (see Parse) because it would end up in the task store, the collector and the log. A username in the link is used only for a host without a stored account, which keeps anonymous public FTP working.
Index ¶
- Constants
- Variables
- func DialError(proto string, addr string, err error) error
- func Join(dir, name string) string
- func LinkOf(t Target) string
- func Name(t Target) string
- func PackageName(raw string) string
- func PartFile(dir, link, taskID string) string
- type Accounts
- type Backend
- func (b *Backend) Download(taskID, link string, headers map[string]string, conns int)
- func (b *Backend) Halt(taskID string) bool
- func (b *Backend) Pause(taskID string)
- func (b *Backend) Remove(taskID string, deleteFiles bool)
- func (b *Backend) Resume(taskID string)
- func (b *Backend) SetLogins(accounts Accounts, dialer Dialer)
- type Dialer
- type Downloader
- type Entry
- type FS
- type Kind
- type Listing
- type Login
- type Logins
- type Resolver
- func (r Resolver) Check(ctx context.Context, urls []string) ([]core.Availability, error)
- func (Resolver) Info() resolver.Info
- func (r Resolver) List(ctx context.Context, raw string) ([]Listing, error)
- func (r Resolver) Match(raw string) bool
- func (r Resolver) Resolve(ctx context.Context, req resolver.Request) (resolver.Result, error)
- type Target
Constants ¶
const ResolverID = "remotefs"
ResolverID is the routing id shared by the app's backend table, its directory expansion and the account catalogue; a mismatch would send these links to the HTTP engine.
Variables ¶
var ( // ErrNoAccount is a server nothing is stored for; nobody was refused. ErrNoAccount = errors.New("no account is stored for this server") // ErrAuth is a credential the server looked at and refused. ErrAuth = errors.New("the server refused this username and password") // ErrNotFound is a path the server says is not there. ErrNotFound = errors.New("the server says this path does not exist") // ErrPasswordInLink is a link carrying its own password. ErrPasswordInLink = errors.New("a password in the link is refused; store it under Accounts instead") )
The failures every protocol has its own words for (FTP 550, SFTP SSH_FX_NO_SUCH_FILE, WebDAV 404), translated at the edge of each implementation. They are always wrapped, so the message still names server and path while errors.Is keeps working.
Functions ¶
func DialError ¶
DialError turns a failed connection into a sentence instead of the operating system's error text, which on Windows arrives in the system's language. The cases are told apart by error type, never by message; anything that is not a dial failure passes through wrapped.
func Join ¶
Join is path.Join for a server path, so no caller reaches for filepath.Join and builds backslash paths on Windows.
func LinkOf ¶
LinkOf rebuilds the canonical, credential-free link for a target, used as a staged task's URL. Without a username, default port or trailing slash, one file has one spelling and the duplicate check can fold it.
func PackageName ¶
PackageName is the package a batch expanded from one directory link is filed under: the directory's name, or the host for the server's root. The app's own guess works from file names, which in a folder of unrelated files share no stem.
Types ¶
type Accounts ¶
Accounts answers which stored login belongs to a host; the app wires in the encrypted account store. ok=false means nothing is stored, which Resolver.Match reads as "leave this https:// link alone".
type Backend ¶
type Backend struct {
// Dir returns the destination for one task; nil or "" falls back to the
// backend's own directory. It is asked per task so a settings change
// applies to the next download.
Dir func(taskID string) string
// RateLimit returns the speed limit in force, in bytes per second, 0 for
// none. It is read once per buffer so a scheduled limit applies to a
// running transfer, and it is the only limit these non-HTTP transfers see.
RateLimit func() int64
// contains filtered or unexported fields
}
Backend fetches what the engine cannot. WebDAV targets resolve to plain https URLs and go to the engine (see Resolver.Resolve); FTP, FTPS and SFTP are downloaded here as a single stream with resume.
func NewBackend ¶
type Dialer ¶
type Dialer struct {
// KnownHostsFile is where SSH host keys are remembered (see sftp.go).
// Empty means ~/.ssh/known_hosts; the app passes a path in its data
// directory because a container has no useful home directory.
KnownHostsFile string
// HTTPClient carries WebDAV's PROPFIND and GET. Nil means an httpx client
// with this package's timeout.
HTTPClient *http.Client
}
Dialer opens connections. The zero value works.
type Downloader ¶
type Downloader interface {
Download(taskID, url string, headers map[string]string, conns int)
Pause(taskID string)
Resume(taskID string)
Remove(taskID string, deleteFiles bool)
}
Downloader is the byte-transfer backend a WebDAV link is handed on to, the embedded engine in production.
type Entry ¶
Entry is one thing a remote directory holds. Size is 0 for a directory and for a server that lists without sizes.
type FS ¶
type FS interface {
// Stat says whether a path is a file or a directory, and how large.
Stat(ctx context.Context, path string) (Entry, error)
// List names what a directory holds, directly, without descending.
List(ctx context.Context, path string) ([]Entry, error)
// Open starts reading a file at offset. An implementation that cannot
// honour a non-zero offset must fail rather than restart at 0, which
// would append the whole file to a part file.
Open(ctx context.Context, path string, offset int64) (io.ReadCloser, error)
io.Closer
}
FS is one open connection to one server. It is read-only, so no bug in the app can delete anything through it.
type Kind ¶
type Kind string
Kind is which protocol a target speaks. The scheme alone does not decide it: webdav:// and webdavs:// are one protocol, and so is https:// for a host an account marks as a WebDAV server (see Resolver.Match).
type Listing ¶
Listing is one file a folder link expands into: a canonical link of its own and what the server said about it.
type Login ¶
Login is one stored credential. Both fields empty is the anonymous case, which only FTP has any use for.
type Resolver ¶
type Resolver struct {
// Accounts is consulted by Match under the app's lock, so it must answer
// from memory; the app wires a snapshot rebuilt on every account change.
Accounts Accounts
Dialer Dialer
}
Resolver claims links on servers the user owns and answers what is behind them: the name, the size, and whether it is a folder to expand.
It keeps no connection: every call dials, asks and hangs up, because an idle FTP control connection times out on the server's schedule. Staging a folder therefore costs one login per file; a connection cache would need exclusive ownership per transfer and idle eviction.
func (Resolver) Check ¶
Check answers whether these links are still there with one connection per server, since a seedbox's connection limit would turn many parallel logins into refusals. Only a definite ErrNotFound is offline; a refused login, a server that is down or any other failure leaves the link uncheckable.
func (Resolver) List ¶
List expands a folder link into the files under it, one task per file. It returns (nil, nil) for a link that is not a folder, meaning "stage it as itself".
It recurses within maxListingDepth and maxListingEntries, so subfolders like "Subs" and "Sample" are not left behind. Equal names from different subfolders are kept apart by the collision policy.
func (Resolver) Match ¶
Match claims ftp, ftps, sftp, webdav and webdavs by scheme, and https only for a host with a stored account. The URL's shape alone would be a guess that takes links from other backends. http:// is never claimed, since an account does not say which scheme the host serves WebDAV on; webdav:// names the plaintext case explicitly.
func (Resolver) Resolve ¶
Resolve confirms the file is there and says how to fetch it.
For WebDAV it returns the plain http(s) URL with an Authorization header, which the engine fetches. For FTP, FTPS and SFTP it returns the credential-free link for Backend, which looks the login up again rather than receiving it through the dispatcher.
A folder is an error: folder links are expanded by List before staging, so one arriving here was staged some other way.
type Target ¶
type Target struct {
Kind Kind
// Host is the bare hostname, lower-cased. It is also the account id a
// stored login is filed under, so it never carries the port.
Host string
Port int
// Path is the absolute path on the server, "/" for the root.
Path string
// User is the username the link named, used only when no account is
// stored for the host.
User string
// TLS is whether the transport is encrypted (ftps://, or WebDAV over
// https). SFTP ignores it.
TLS bool
// ImplicitTLS selects FTPS with TLS from the first byte on port 990. Parse
// sets it from the port.
ImplicitTLS bool
}
Target is one file or directory on one server, with everything needed to reach it except the credential.