remotefs

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: AGPL-3.0 Imports: 29 Imported by: 0

Documentation

Overview

Package remotefs downloads from servers a person already owns: a seedbox over FTP or FTPS, a NAS over SFTP, a Nextcloud or ownCloud over WebDAV.

The protocols share nothing but a tree that can be listed, a file size that can be asked for and a byte stream that can start at an offset. That is the FS interface, and it is why the package is named after a file system rather than after its protocols.

Credentials never travel in a link: a URL with a password is refused (see Parse) because it would end up in the task store, the collector and the log. A username in the link is used only for a host without a stored account, which keeps anonymous public FTP working.

Index

Constants

View Source
const ResolverID = "remotefs"

ResolverID is the routing id shared by the app's backend table, its directory expansion and the account catalogue; a mismatch would send these links to the HTTP engine.

Variables

View Source
var (
	// ErrNoAccount is a server nothing is stored for; nobody was refused.
	ErrNoAccount = errors.New("no account is stored for this server")
	// ErrAuth is a credential the server looked at and refused.
	ErrAuth = errors.New("the server refused this username and password")
	// ErrNotFound is a path the server says is not there.
	ErrNotFound = errors.New("the server says this path does not exist")
	// ErrPasswordInLink is a link carrying its own password.
	ErrPasswordInLink = errors.New("a password in the link is refused; store it under Accounts instead")
)

The failures every protocol has its own words for (FTP 550, SFTP SSH_FX_NO_SUCH_FILE, WebDAV 404), translated at the edge of each implementation. They are always wrapped, so the message still names server and path while errors.Is keeps working.

Functions

func DialError

func DialError(proto string, addr string, err error) error

DialError turns a failed connection into a sentence instead of the operating system's error text, which on Windows arrives in the system's language. The cases are told apart by error type, never by message; anything that is not a dial failure passes through wrapped.

func Join

func Join(dir, name string) string

Join is path.Join for a server path, so no caller reaches for filepath.Join and builds backslash paths on Windows.

func LinkOf

func LinkOf(t Target) string

LinkOf rebuilds the canonical, credential-free link for a target, used as a staged task's URL. Without a username, default port or trailing slash, one file has one spelling and the duplicate check can fold it.

func Name

func Name(t Target) string

Name is the file name a target's path ends in, or the host for the root directory.

func PackageName

func PackageName(raw string) string

PackageName is the package a batch expanded from one directory link is filed under: the directory's name, or the host for the server's root. The app's own guess works from file names, which in a folder of unrelated files share no stem.

func PartFile added in v1.7.0

func PartFile(dir, link, taskID string) string

PartFile is the part file the download taskID of link writes in dir until it finishes, or "" for a link this backend hands to the engine.

Types

type Accounts

type Accounts interface {
	Login(host string) (Login, bool)
}

Accounts answers which stored login belongs to a host; the app wires in the encrypted account store. ok=false means nothing is stored, which Resolver.Match reads as "leave this https:// link alone".

type Backend

type Backend struct {

	// Dir returns the destination for one task; nil or "" falls back to the
	// backend's own directory. It is asked per task so a settings change
	// applies to the next download.
	Dir func(taskID string) string
	// RateLimit returns the speed limit in force, in bytes per second, 0 for
	// none. It is read once per buffer so a scheduled limit applies to a
	// running transfer, and it is the only limit these non-HTTP transfers see.
	RateLimit func() int64
	// contains filtered or unexported fields
}

Backend fetches what the engine cannot. WebDAV targets resolve to plain https URLs and go to the engine (see Resolver.Resolve); FTP, FTPS and SFTP are downloaded here as a single stream with resume.

func NewBackend

func NewBackend(accounts Accounts, dialer Dialer, eng Downloader, dir string, onUpdate func(string, core.Update)) *Backend

func (*Backend) Download

func (b *Backend) Download(taskID, link string, headers map[string]string, conns int)

func (*Backend) Halt added in v1.2.0

func (b *Backend) Halt(taskID string) bool

Halt stops the task's transfer and returns once its part file is closed, so the file can be moved. Resume carries on in the folder Dir names by then. A transfer handed to the engine is not this backend's to halt. It reports whether one was running.

func (*Backend) Pause

func (b *Backend) Pause(taskID string)

func (*Backend) Remove

func (b *Backend) Remove(taskID string, deleteFiles bool)

func (*Backend) Resume

func (b *Backend) Resume(taskID string)

func (*Backend) SetLogins added in v1.7.0

func (b *Backend) SetLogins(accounts Accounts, dialer Dialer)

SetLogins makes the transfers started from now on log in with accounts through dialer. A transfer already going keeps the connection it has.

type Dialer

type Dialer struct {
	// KnownHostsFile is where SSH host keys are remembered (see sftp.go).
	// Empty means ~/.ssh/known_hosts; the app passes a path in its data
	// directory because a container has no useful home directory.
	KnownHostsFile string
	// HTTPClient carries WebDAV's PROPFIND and GET. Nil means an httpx client
	// with this package's timeout.
	HTTPClient *http.Client
}

Dialer opens connections. The zero value works.

func (Dialer) Dial

func (d Dialer) Dial(ctx context.Context, t Target, login Login) (FS, error)

Dial opens a connection to one target.

The zero Login means no credential: FTP falls back to anonymous, WebDAV sends no Authorization header, and SFTP refuses (see Resolver.loginFor).

type Downloader

type Downloader interface {
	Download(taskID, url string, headers map[string]string, conns int)
	Pause(taskID string)
	Resume(taskID string)
	Remove(taskID string, deleteFiles bool)
}

Downloader is the byte-transfer backend a WebDAV link is handed on to, the embedded engine in production.

type Entry

type Entry struct {
	Name string
	Size int64
	Dir  bool
}

Entry is one thing a remote directory holds. Size is 0 for a directory and for a server that lists without sizes.

type FS

type FS interface {
	// Stat says whether a path is a file or a directory, and how large.
	Stat(ctx context.Context, path string) (Entry, error)
	// List names what a directory holds, directly, without descending.
	List(ctx context.Context, path string) ([]Entry, error)
	// Open starts reading a file at offset. An implementation that cannot
	// honour a non-zero offset must fail rather than restart at 0, which
	// would append the whole file to a part file.
	Open(ctx context.Context, path string, offset int64) (io.ReadCloser, error)
	io.Closer
}

FS is one open connection to one server. It is read-only, so no bug in the app can delete anything through it.

type Kind

type Kind string

Kind is which protocol a target speaks. The scheme alone does not decide it: webdav:// and webdavs:// are one protocol, and so is https:// for a host an account marks as a WebDAV server (see Resolver.Match).

const (
	KindFTP    Kind = "ftp"
	KindFTPS   Kind = "ftps"
	KindSFTP   Kind = "sftp"
	KindWebDAV Kind = "webdav"
)

type Listing

type Listing struct {
	URL  string
	Name string
	Size int64
}

Listing is one file a folder link expands into: a canonical link of its own and what the server said about it.

type Login

type Login struct {
	Username string
	Password string
}

Login is one stored credential. Both fields empty is the anonymous case, which only FTP has any use for.

type Logins

type Logins map[string]Login

Logins is an Accounts backed by a map from host to login.

func (Logins) Login

func (m Logins) Login(host string) (Login, bool)

type Resolver

type Resolver struct {
	// Accounts is consulted by Match under the app's lock, so it must answer
	// from memory; the app wires a snapshot rebuilt on every account change.
	Accounts Accounts
	Dialer   Dialer
}

Resolver claims links on servers the user owns and answers what is behind them: the name, the size, and whether it is a folder to expand.

It keeps no connection: every call dials, asks and hangs up, because an idle FTP control connection times out on the server's schedule. Staging a folder therefore costs one login per file; a connection cache would need exclusive ownership per transfer and idle eviction.

func (Resolver) Check

func (r Resolver) Check(ctx context.Context, urls []string) ([]core.Availability, error)

Check answers whether these links are still there with one connection per server, since a seedbox's connection limit would turn many parallel logins into refusals. Only a definite ErrNotFound is offline; a refused login, a server that is down or any other failure leaves the link uncheckable.

func (Resolver) Info

func (Resolver) Info() resolver.Info

func (Resolver) List

func (r Resolver) List(ctx context.Context, raw string) ([]Listing, error)

List expands a folder link into the files under it, one task per file. It returns (nil, nil) for a link that is not a folder, meaning "stage it as itself".

It recurses within maxListingDepth and maxListingEntries, so subfolders like "Subs" and "Sample" are not left behind. Equal names from different subfolders are kept apart by the collision policy.

func (Resolver) Match

func (r Resolver) Match(raw string) bool

Match claims ftp, ftps, sftp, webdav and webdavs by scheme, and https only for a host with a stored account. The URL's shape alone would be a guess that takes links from other backends. http:// is never claimed, since an account does not say which scheme the host serves WebDAV on; webdav:// names the plaintext case explicitly.

func (Resolver) Resolve

func (r Resolver) Resolve(ctx context.Context, req resolver.Request) (resolver.Result, error)

Resolve confirms the file is there and says how to fetch it.

For WebDAV it returns the plain http(s) URL with an Authorization header, which the engine fetches. For FTP, FTPS and SFTP it returns the credential-free link for Backend, which looks the login up again rather than receiving it through the dispatcher.

A folder is an error: folder links are expanded by List before staging, so one arriving here was staged some other way.

type Target

type Target struct {
	Kind Kind
	// Host is the bare hostname, lower-cased. It is also the account id a
	// stored login is filed under, so it never carries the port.
	Host string
	Port int
	// Path is the absolute path on the server, "/" for the root.
	Path string
	// User is the username the link named, used only when no account is
	// stored for the host.
	User string
	// TLS is whether the transport is encrypted (ftps://, or WebDAV over
	// https). SFTP ignores it.
	TLS bool
	// ImplicitTLS selects FTPS with TLS from the first byte on port 990. Parse
	// sets it from the port.
	ImplicitTLS bool
}

Target is one file or directory on one server, with everything needed to reach it except the credential.

func Parse

func Parse(raw string, httpsIsWebDAV bool) (Target, error)

Parse turns a pasted link into a Target, refusing unknown schemes and passwords in the URL.

httpsIsWebDAV is the caller's decision for a plain https:// link: true only when an account is stored for that exact host (see Resolver.Match).

func (Target) Addr

func (t Target) Addr() string

Addr is the host:port the dialers connect to.

func (Target) HTTPURL

func (t Target) HTTPURL() string

HTTPURL is the plain http(s) URL of a WebDAV target, which the engine fetches like any other link. The port is left off when it is the scheme's default, since origin comparisons compare the text.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址