Documentation
¶
Overview ¶
Package rfs provides a unified filesystem abstraction for runq. LocalFS wraps os.* for same-machine operation; SSHFS wraps sftp + ssh.Session for remote clusters. The HPC backend receives an FS at construction and uses it for all file I/O and command execution, making local vs. remote transparent.
Design decisions:
- No cache layer: reads go straight through. Incremental-read state (metric offsets, log offsets) is maintained in-memory by the caller.
- Exec runs a command on the target machine. For SSHFS this means an SSH session; for LocalFS it means os/exec. The caller is responsible for quoting and escaping.
- CopyToLocal transfers a file from the FS to the local machine. For LocalFS this is a plain file copy; for SSHFS it is an sftp download.
Index ¶
- Constants
- Variables
- func AcceptNewHostKeyCallback() (ssh.HostKeyCallback, error)
- func AppendKnownHost(hostname string, key ssh.PublicKey) error
- func EffectiveSSH(host string, port int, user, jump string) (effUser, effHost string, effPort int, ok bool)
- func ResolveAuthMethods(keyPath string, agentSocket ...string) ([]ssh.AuthMethod, error)
- func StrictHostKeyCallback() (ssh.HostKeyCallback, error)
- func TOFUHostKeyCallback(confirm func(host, fingerprint string) bool) (ssh.HostKeyCallback, error)
- func VerifySSHInteractive(ctx context.Context, cfg SSHConfig) error
- type ContactObserved
- type CopyOption
- type FS
- type File
- type ForwardStatus
- type HostKeyMismatchError
- type LocalFS
- func (l *LocalFS) CopyToLocal(remotePath, localPath string, opts ...CopyOption) error
- func (l *LocalFS) Exec(ctx context.Context, cmd string, args ...string) (stdout, stderr []byte, exitCode int, err error)
- func (l *LocalFS) ExecStream(ctx context.Context, cmd string, args ...string) (io.ReadCloser, error)
- func (l *LocalFS) MkdirAll(path string, perm os.FileMode) error
- func (l *LocalFS) Open(path string) (File, error)
- func (l *LocalFS) ReadDir(path string) ([]fs.DirEntry, error)
- func (l *LocalFS) ReadFile(path string) ([]byte, error)
- func (l *LocalFS) Remove(path string) error
- func (l *LocalFS) Rename(oldPath, newPath string) error
- func (l *LocalFS) Stat(path string) (fs.FileInfo, error)
- func (l *LocalFS) WriteFile(path string, data []byte, perm os.FileMode) error
- type OpenSSHConfig
- type RemoteForward
- type RemoteForwardConfig
- type SSHConfig
- type SSHFS
- func (s *SSHFS) Close() error
- func (s *SSHFS) CopyToLocal(remotePath, localPath string, opts ...CopyOption) error
- func (s *SSHFS) DialUnix(ctx context.Context, path string) (net.Conn, error)
- func (s *SSHFS) Exec(parent context.Context, cmd string, args ...string) (stdout, stderr []byte, exitCode int, err error)
- func (s *SSHFS) ExecStream(ctx context.Context, cmd string, args ...string) (io.ReadCloser, error)
- func (s *SSHFS) Home() (string, error)
- func (s *SSHFS) MkdirAll(path string, perm os.FileMode) error
- func (s *SSHFS) Open(path string) (File, error)
- func (s *SSHFS) ReadDir(path string) ([]fs.DirEntry, error)
- func (s *SSHFS) ReadFile(path string) ([]byte, error)
- func (s *SSHFS) Remove(path string) error
- func (s *SSHFS) RemoveContext(ctx context.Context, path string) error
- func (s *SSHFS) Rename(oldPath, newPath string) error
- func (s *SSHFS) SetContactObserver(observe func(generation uint64, err error))
- func (s *SSHFS) Stat(path string) (fs.FileInfo, error)
- func (s *SSHFS) WithContext(ctx context.Context) *SSHFS
- func (s *SSHFS) WriteFile(path string, data []byte, perm os.FileMode) error
- func (s *SSHFS) WriteFileContext(parent context.Context, path string, data []byte, perm os.FileMode) error
- type UnknownHostKeyError
Constants ¶
const ( ForwardUp = "up" // session established, socket bound ForwardReconnecting = "reconnecting" // between sessions (includes pre-first-dial) ForwardClosed = "closed" // Close() called — no further attempts )
Forward state labels (ForwardStatus.State).
const ( // HostKeyStrict is the default: unknown or changed keys refuse the // connection (typed errors naming the fix). HostKeyStrict = "" // HostKeyAcceptNew auto-trusts and records UNKNOWN hosts (no prompt); // a CHANGED key still hard-fails. HostKeyAcceptNew = "accept-new" )
Host key policy values consumed by sshConn/RemoteForward.
Variables ¶
var ErrExecInterrupted = errors.New("remote command interrupted: outcome unknown")
ErrExecInterrupted marks an Exec whose command STARTED on the remote side but whose verdict was lost (connection dropped mid-flight, session torn down). This is neither a clean transport failure (the command may have run to completion) nor a scheduler verdict (there is no exit code). Callers with side-effecting commands (job submission) must treat it as "outcome unknown" — never as retry-safe (RQ-74).
Functions ¶
func AcceptNewHostKeyCallback ¶
func AcceptNewHostKeyCallback() (ssh.HostKeyCallback, error)
AcceptNewHostKeyCallback is the ssh_config `StrictHostKeyChecking accept-new` passthrough (RQ-74): an UNKNOWN host is trusted and recorded without a prompt — exactly what the user's own ssh would do under that setting, so runq is never more ceremonious than ssh. A MISMATCH still hard-fails: OpenSSH's `no` would tolerate even that, but silently accepting a CHANGED key disables the MITM protection entirely — runq caps the passthrough at accept-new semantics on purpose.
func AppendKnownHost ¶
AppendKnownHost appends one accepted host key to ~/.ssh/known_hosts in the standard format.
func EffectiveSSH ¶ added in v0.7.0
func EffectiveSSH(host string, port int, user, jump string) (effUser, effHost string, effPort int, ok bool)
EffectiveSSH asks OpenSSH (ssh -G) which user, host name and port a connection resolves to after ~/.ssh/config: an alias and its HostName, or an omitted user and its default, are the same endpoint. ok is false when ssh cannot answer (the caller keeps the configured values).
func ResolveAuthMethods ¶
func ResolveAuthMethods(keyPath string, agentSocket ...string) ([]ssh.AuthMethod, error)
ResolveAuthMethods builds the auth chain (RQ-45): the configured key file and the ssh-agent's identities, offered through ONE publickey method — golang.org/x/crypto/ssh tries each method name once, so a second PublicKeys method after an agent that holds no matching key was never attempted ("attempted methods [none publickey]") even though plain ssh logged in with the same file. Signing via the agent stays in the agent; an encrypted key file without an agent is a clear, actionable error, not a parse failure.
func StrictHostKeyCallback ¶
func StrictHostKeyCallback() (ssh.HostKeyCallback, error)
StrictHostKeyCallback verifies against ~/.ssh/known_hosts and maps the library's KeyError into the two typed errors above.
func TOFUHostKeyCallback ¶
func TOFUHostKeyCallback(confirm func(host, fingerprint string) bool) (ssh.HostKeyCallback, error)
TOFUHostKeyCallback wraps strict verification with a consent prompt for the UNKNOWN case only (mismatch is never promptable — that one is a security event, not an onboarding step). On consent the key is appended to known_hosts, making all future strict connections succeed.
Types ¶
type ContactObserved ¶ added in v0.7.2
ContactObserved is a filesystem that reports its own transport outcomes.
generation names the connection an outcome speaks for and grows with each new connection. Callbacks run concurrently and outside the manager's locks, so one can arrive after a newer generation's: an observer keeps the newest generation it has applied and discards anything older.
type FS ¶
type FS interface {
// Stat returns file metadata, like os.Stat.
Stat(path string) (fs.FileInfo, error)
// ReadFile reads the named file in full, like os.ReadFile.
ReadFile(path string) ([]byte, error)
// Open opens a seekable file handle for incremental reads.
Open(path string) (File, error)
// ReadDir reads a directory, like os.ReadDir.
ReadDir(path string) ([]fs.DirEntry, error)
// WriteFile writes data to the named file, like os.WriteFile.
WriteFile(path string, data []byte, perm os.FileMode) error
// MkdirAll creates a directory tree, like os.MkdirAll.
MkdirAll(path string, perm os.FileMode) error
// Exec runs a command on the target machine, returning stdout, stderr,
// the exit code, and any transport-level error. A non-zero exit code is
// NOT returned as an error — the caller decides how to interpret it.
Exec(ctx context.Context, cmd string, args ...string) (stdout, stderr []byte, exitCode int, err error)
// ExecStream runs a command and returns a stream of its combined output.
// Used for `runq logs -f` style live tailing over SSH.
ExecStream(ctx context.Context, cmd string, args ...string) (io.ReadCloser, error)
// CopyToLocal copies a file from this FS to a local path. For LocalFS
// this is a file copy; for SSHFS it is an sftp download. The target
// directory must already exist.
CopyToLocal(remotePath, localPath string, opts ...CopyOption) error
}
FS is the filesystem abstraction injected into the HPC backend. It covers file reads/writes, directory operations, command execution, and file transfer. Implementations: LocalFS (same machine) and SSHFS (remote).
type ForwardStatus ¶
type ForwardStatus struct {
SocketPath string `json:"socket_path,omitempty"`
Owner string `json:"owner,omitempty"`
State string `json:"state"` // up | reconnecting | closed
Since int64 `json:"since"` // when the current state was entered
// LastOnline is the last moment the forward was known to be serving:
// set on establish and refreshed after successful keepalive. 0 = never
// online ("no contact yet" ≠ "went down" — doctor renders them apart).
LastOnline int64 `json:"last_online,omitempty"`
// Attempts counts consecutive failed sessions since the last up.
Attempts int `json:"attempts,omitempty"`
LastError string `json:"last_error,omitempty"`
Phase string `json:"phase,omitempty"`
Reason string `json:"reason,omitempty"`
Hint string `json:"hint,omitempty"`
NextRetryAt int64 `json:"next_retry_at,omitempty"`
LastAttemptAt int64 `json:"last_attempt_at,omitempty"`
Transport string `json:"transport,omitempty"`
DebugCommand string `json:"debug_command,omitempty"`
SSHAuthSock string `json:"ssh_auth_sock,omitempty"`
}
ForwardStatus is one forward's observable state (RQ-74): what /health exposes and doctor renders ("↻ reconnecting · attempt 3 · last online 12m ago"). All timestamps unix seconds.
type HostKeyMismatchError ¶
HostKeyMismatchError: the host's key CHANGED. Deliberately alarming — this is either a server reinstall, a login-node pool member with its own key, or an interception. Never promptable and never auto-accepted (even under accept-new): the way out is a copy-paste command, not a click (RQ-74 — the error carries its own fix).
func (*HostKeyMismatchError) Error ¶
func (e *HostKeyMismatchError) Error() string
type LocalFS ¶
type LocalFS struct{}
LocalFS implements FS using the local filesystem and os/exec. It is the default FS for same-machine (login-node) HPC operation.
func (*LocalFS) CopyToLocal ¶
func (l *LocalFS) CopyToLocal(remotePath, localPath string, opts ...CopyOption) error
func (*LocalFS) ExecStream ¶
func (*LocalFS) Remove ¶ added in v0.6.0
Remove is an optional FS extension used by project sync to delete stale allow-listed files without exposing a broad recursive-delete primitive.
type OpenSSHConfig ¶ added in v0.7.0
type RemoteForward ¶
type RemoteForward struct {
// contains filtered or unexported fields
}
RemoteForward supervises one remote socket forward. Create with NewRemoteForward, drive with Run (blocking; run it in a goroutine), stop with Close.
func NewRemoteForward ¶
func NewRemoteForward(cfg RemoteForwardConfig) *RemoteForward
NewRemoteForward creates a RemoteForward. No network traffic until Run.
func (*RemoteForward) Close ¶
func (f *RemoteForward) Close() error
Close stops the supervisor and tears down the current connection (which unblocks Serve). Safe to call multiple times.
func (*RemoteForward) Run ¶
func (f *RemoteForward) Run(parent context.Context)
Run supervises the forward until ctx is cancelled or Close is called: dial → bind remote socket → Serve, then backoff and repeat. Failures are logged, never fatal — the HPC lane works with or without the forward.
func (*RemoteForward) Status ¶
func (f *RemoteForward) Status() ForwardStatus
Status returns a snapshot of the forward's observable state.
type RemoteForwardConfig ¶
type RemoteForwardConfig struct {
// SSH is the connection config. IdleTimeout is ignored — a forward
// listener is persistent by definition (see package comment above).
SSH SSHConfig
// SocketPath is the remote unix socket path. Relative paths resolve
// against the remote user's home (sftp session start dir).
// Default: ".runq/runq.sock".
SocketPath string
Owner string
// Serve consumes the remote listener and blocks until the listener
// dies (the daemon passes http.Serve with the dashboard mux). It is
// called once per established forward; a non-nil return is logged and
// triggers a reconnect cycle.
Serve func(net.Listener) error
// OnEstablished, when set, is called once per established forward
// session, right before Serve — the moment the remote socket is bound
// and listening. The daemon uses it to record target reachability
// (RQ-74): a forward coming up is transport-level proof of contact, so
// doctor's "no contact yet" clears without waiting for the next
// scheduled sensor pass. Must be fast and non-blocking.
OnEstablished func()
Logger *slog.Logger
}
RemoteForwardConfig configures a RemoteForward.
type SSHConfig ¶
type SSHConfig struct {
// OpenSSH is set for user targets. Native SSH remains for injected transports.
OpenSSH *OpenSSHConfig
Host string // original OpenSSH alias; native transports use host[:port]
User string
// AuthMethods are tried in order. Convention (RQ-45): ssh-agent first
// (the daemon never touches key plaintext), key file second. Build
// with ResolveAuthMethods.
AuthMethods []ssh.AuthMethod
// HostKeyCallback: nil = known_hosts verification per HostKeyPolicy
// (the daemon default). `runq connect` passes a TOFU callback with a
// human on the other end. InsecureIgnoreHostKey never appears in this
// codebase.
HostKeyCallback ssh.HostKeyCallback
// HostKeyPolicy applies when HostKeyCallback is nil: HostKeyStrict
// (zero value) or HostKeyAcceptNew — the ssh_config
// `StrictHostKeyChecking accept-new` passthrough (RQ-74). Resolve with
// ResolveHostKeyPolicy against the target's ssh alias.
HostKeyPolicy string
// IdleTimeout controls how long a connection survives with no in-flight
// operations before sshConn tears it down. Zero means never idle-close.
// Callers that want the "look like a normal SSH user" behavior MUST set
// this (the daemon does; see backend.NewSSHBackend).
IdleTimeout time.Duration
// MaxSessions caps the number of concurrent SSH sessions (Exec calls).
// sftp runs on its own subsystem and does NOT consume a session slot.
// Default: 3.
MaxSessions int
// OperationTimeout bounds connection setup and each file operation.
OperationTimeout time.Duration
}
SSHConfig holds the parameters needed to establish an SSH connection. Populated from the target config (see internal/config.TargetConfig).
func ResolveTargetSSH ¶ added in v0.6.0
ResolveTargetSSH retains the alias and explicit overrides. OpenSSH resolves configuration and authentication on every reconnect; resolving the alias in advance would lose Host/Match, ProxyJump and host-key store semantics.
func (SSHConfig) DebugCommand ¶ added in v0.7.0
DebugCommand reproduces fresh noninteractive authentication using the alias and overrides, bypassing existing control masters. No credentials are printed.
type SSHFS ¶
type SSHFS struct {
// contains filtered or unexported fields
}
SSHFS implements FS over an SSH connection. File operations go through sftp; command execution opens SSH sessions. The underlying *sshConn is lazy — no network traffic until the first call — and self-healing: dead connections are detected and redialed transparently (see sshConn.getConn).
Thread safety: safe for concurrent use. sshConn serializes dial/reconnect; sftp.Client is documented as safe for concurrent use; SSH sessions are bounded by the semaphore in sshConn.
Every operation is bracketed by beginOp/endOpErr so the idle watcher never tears down a connection with work in flight; for handle-returning calls (Open, ExecStream) the bracket extends until the handle is closed.
func (*SSHFS) Close ¶
Close tears down the SSH connection. The SSHFS should not be used after Close.
func (*SSHFS) CopyToLocal ¶
func (s *SSHFS) CopyToLocal(remotePath, localPath string, opts ...CopyOption) error
CopyToLocal downloads a remote file via sftp. The busy-marker covers the whole transfer — large checkpoint downloads are never cut by idle teardown.
func (*SSHFS) DialUnix ¶ added in v0.6.0
DialUnix opens a remote socket through SSH. Used to prove that a reverse forward actually reaches the expected controller/target before connect succeeds.
func (*SSHFS) Exec ¶
func (s *SSHFS) Exec(parent context.Context, cmd string, args ...string) (stdout, stderr []byte, exitCode int, err error)
Exec runs a remote command via an SSH session and returns stdout, stderr, and the exit code. A non-zero exit is NOT a Go error; transport failures are. ctx ending closes this session only (the remote command gets EOF/HUP); other work on the connection is not disturbed.
func (*SSHFS) ExecStream ¶
ExecStream runs a remote command and returns a streaming reader of its combined output. The session, semaphore slot, and busy-marker are released when the command exits (or ctx is cancelled); the pipe's read side reports the command's exit status via CloseWithError.
func (*SSHFS) RemoveContext ¶ added in v0.6.0
RemoveContext allows bounded cleanup with a context independent of a failed transfer while retaining the manager's connection/setup time bounds.
func (*SSHFS) Rename ¶
Only atomic replacement or a server's normal rename is permitted. A failed replacement must never delete the existing executable/configuration.
func (*SSHFS) SetContactObserver ¶ added in v0.7.0
SetContactObserver reports transport outcomes, including file browsing. Context-scoped views share the observer. The callback runs outside conn.mu.
func (*SSHFS) WithContext ¶ added in v0.6.0
WithContext scopes file operations without changing the shared connection.
func (*SSHFS) WriteFileContext ¶ added in v0.6.0
func (s *SSHFS) WriteFileContext(parent context.Context, path string, data []byte, perm os.FileMode) error
WriteFileContext uses an explicit transfer budget. Connection setup remains independently bounded; a large binary is not limited by a small-file timeout.
type UnknownHostKeyError ¶
UnknownHostKeyError: the host presented a key we have never seen.
func (*UnknownHostKeyError) Error ¶
func (e *UnknownHostKeyError) Error() string