backup

package
v0.3.36 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: AGPL-3.0 Imports: 31 Imported by: 0

Documentation

Overview

Package backup creates and restores server archives and ships them to storage targets (local/NFS/CIFS mount, SFTP, SMB, S3, GCS, Azure). Credentials are encrypted at rest by the caller; this package never logs them.

Index

Constants

View Source
const PBSArchive = "data"

PBSArchive is the single pxar archive every snapshot contains. One archive rather than one per included path, because backup.include lists FILES as well as directories (Minecraft includes server.properties) and a pxar source must be a directory. Inclusion is expressed as exclusions instead — see PBSExcludes.

View Source
const PBSDefaultPort = 8007

PBSDefaultPort is the port PBS listens on.

View Source
const PBSType = "pbs"

PBSType is the backup_targets.type value for this destination.

View Source
const RcloneConfPath = "/ygg/rclone.conf"

RcloneConfPath is where the config file sits inside the container.

View Source
const RcloneType = "rclone"

RcloneType is the target type for every service reached through rclone.

View Source
const UploadScratchDir = ".ygg_uploads"

UploadScratchDir is the folder inside a server's data directory where a chunked upload accumulates before its final rename. It is scratch space, never content: excluded from archives here, and swept by the API package that writes it. One name, one definition — two copies of it is how an archive quietly starts carrying gigabytes of nothing again.

Variables

View Source
var RcloneProviders = map[string]string{
	"jottacloud": "jottacloud",
	"dropbox":    "dropbox",
	"box":        "box",
	"onedrive":   "onedrive",
	"drive":      "drive",
	"pcloud":     "pcloud",
	"mega":       "mega",
	"webdav":     "webdav",
	"custom":     "",
}

RcloneProviders are the services offered in the panel, with the rclone backend each one is. "custom" is a whole config section pasted as is.

Functions

func Archive

func Archive(dataDir string, include []string, w io.Writer) (skipped []string, err error)

Archive writes a gzip-compressed tar of dataDir to w. When include is non-empty, only those top-level paths (files or directories) are archived — matching a gameskill's backup.include. Paths are stored relative to dataDir.

A file that can't be read (permission denied, or it vanished mid-walk) is recorded in skipped and the archive continues — one unreadable file must not fail the whole backup. The caller should surface skipped so a partial archive isn't mistaken for a complete one. err is reserved for fatal, archive-level failures (a broken tar/gzip stream or a closed sink).

func PBSBackupArgs added in v0.3.6

func PBSBackupArgs(cfg Config, backupID, srcMount string, excludes []string) []string

PBSBackupArgs builds the backup invocation. srcMount is where the caller has bind-mounted the server's data directory inside the container.

func PBSBackupID added in v0.3.6

func PBSBackupID(slug, shortID string) string

PBSBackupID converts a panel server name plus its short id into a legal PBS backup-id. Each panel server gets its own backup group, which is what makes the incremental upload work: PBS dedupes a snapshot against the previous snapshot in the SAME group, so a stable id per server is the whole trick.

func PBSConnectionCheckArgs added in v0.3.6

func PBSConnectionCheckArgs(cfg Config) []string

PBSConnectionCheckArgs is the connection test: list the backup groups in the datastore. One call exercises DNS, TLS, the certificate fingerprint, the auth-id, the token secret AND the datastore name, so a green result means a backup will reach the same place.

It lists rather than asking for `status`, and that is measured rather than assumed: `status` needs privileges beyond the datastore and fails with a bare "permission check failed" for a token that can back up perfectly well. `list` needs Datastore.Audit|Datastore.Backup — exactly what a backup token has — so the test passes for every credential that can actually do the job, and fails for every one that cannot.

func PBSEnv added in v0.3.6

func PBSEnv(cfg Config, passFile string) []string

PBSEnv is the environment for a client invocation.

The token secret is passed by FILE, not by value. An -e on the container would be readable from `docker inspect` for as long as Docker keeps the container's JSON on disk, which is longer than the backup and longer than anyone thinks. passFile is a 0600 file the caller creates and removes.

HOME is set because the client writes tickets and known-fingerprints under $HOME/.config; with no HOME it fails in a way that reads like an auth error.

func PBSExcludes added in v0.3.6

func PBSExcludes(dataDir string, include []string) ([]string, error)

PBSExcludes turns a gameskill's backup.include into the exclusions that leave exactly those paths behind.

pxar takes one source directory and a list of things to leave out, so "back up only these" has to be expressed as "leave out everything else". The list of "everything else" is read from the directory at backup time, which is honest about one edge: a top-level entry created between this read and the walk is included rather than excluded. For a backup that is the safe direction to err, and it beats the alternative of quietly ignoring backup.include — which is what a naive PBS implementation does, and which would silently back up a Minecraft server's entire 40 GB cache alongside its 200 MB world.

An empty include list, or one containing ".", means the whole directory.

func PBSForgetArgs added in v0.3.6

func PBSForgetArgs(cfg Config, snapshot string) []string

PBSForgetArgs removes one snapshot. Note that the space is not reclaimed until the PBS server runs garbage collection — chunks are shared, so nothing else could be safe. Callers must not report freed bytes from this.

func PBSGroup added in v0.3.6

func PBSGroup(backupID string) string

PBSGroup is the backup group a server's snapshots live in.

func PBSPruneArgs added in v0.3.6

func PBSPruneArgs(cfg Config, backupID string, keepN, keepDays int) (args []string, ok bool)

PBSPruneArgs applies retention server-side.

The panel's policy is keep_n ("keep the newest N") and keep_days ("keep anything newer than D days"). PBS has --keep-last, which is exactly keep_n, and --keep-daily, which is "keep the newest snapshot of each of the last D days" — close to keep_days but not identical: with two backups on one day it keeps one of them, and it counts days that HAVE a backup rather than calendar days. That difference is worth stating in the UI, and is why prune is only asked for the rules that were actually set.

Returning ok=false means no policy is configured. Running prune with no --keep flag at all would delete every snapshot in the group, so this must never degrade into "just call prune".

func PBSRepository added in v0.3.6

func PBSRepository(cfg Config) string

PBSRepository builds the repository string the client expects:

[[auth-id@]server[:port]:]datastore

The auth-id itself contains "@" and, for an API token, "!". That is fine here and would not be fine in a shell — which is why every caller passes this as one argv element and never interpolates it into a command line.

An IPv6 literal has to be bracketed or the colons are read as the port separator; a hostname that already looks bracketed is left alone.

func PBSRestoreArgs added in v0.3.6

func PBSRestoreArgs(cfg Config, snapshot, dstMount string) []string

PBSRestoreArgs restores a snapshot's pxar archive into dstMount.

func PBSSnapshotListArgs added in v0.3.6

func PBSSnapshotListArgs(cfg Config, backupID string) []string

PBSSnapshotListArgs lists one server's snapshots.

func PBSSnapshotName added in v0.3.6

func PBSSnapshotName(backupID string, at time.Time) string

PBSSnapshotName is the identifier `restore` and `forget` take.

func RcloneSetup added in v0.3.33

func RcloneSetup(ctx context.Context, run RcloneRunner, cfg Config) ([]byte, error)

RcloneSetup turns what an admin entered into a working rclone.conf with one remote named "ygg". For the OAuth services that means running rclone's own configuration, which for Jottacloud exchanges the single-use personal login token for a refreshable one; questions it asks on the way (which drive, which device) are answered with rclone's default.

func Restore

func Restore(r io.Reader, destDir string) error

func Verify

func Verify(r io.Reader) (entries int, total int64, err error)

Restore extracts a gzip-tar archive from r into destDir, guarding against path-traversal entries. Verify streams a backup archive and confirms it is a well-formed gzip+tar that decompresses in full — a cheap integrity check (a truncated or corrupt archive fails at the gzip CRC or a short tar entry) that reads everything but writes nothing to disk. Returns the file count and total uncompressed size.

Types

type Config

type Config struct {
	Type     string `json:"type"` // local | sftp | smb | pbs | s3 | gcs | azure
	Path     string `json:"path"` // base directory / remote path / share subpath
	Host     string `json:"host,omitempty"`
	Port     int    `json:"port,omitempty"`
	Username string `json:"username,omitempty"` // for pbs: the auth-id, e.g. ygg@pbs!panel
	Password string `json:"password,omitempty"` // for pbs: the API token secret
	Share    string `json:"share,omitempty"`    // SMB share name

	// Object storage. Host is the endpoint (empty = the provider's own), Path is
	// a folder inside the bucket. Username/Password are the access key ID and
	// secret for s3, the account name and access key (or SAS token) for azure;
	// for gcs, Password holds the service account's JSON key.
	Bucket string `json:"bucket,omitempty"` // s3/gcs bucket, azure container
	Region string `json:"region,omitempty"` // s3 only; for rclone/webdav, the server flavour

	// rclone: which service (a key of RcloneProviders), and the rclone.conf the
	// panel built at setup. The conf is written by the panel only — never taken
	// from a request, never sent to the browser.
	Provider   string `json:"provider,omitempty"`
	RcloneConf string `json:"rclone_conf,omitempty"`

	// Proxmox Backup Server only.
	Datastore   string `json:"datastore,omitempty"`
	Namespace   string `json:"namespace,omitempty"`   // optional; empty = the datastore root
	Fingerprint string `json:"fingerprint,omitempty"` // server cert sha256, for self-signed installs
}

Config is the decrypted configuration for opening a target.

It is stored as encrypted JSON, so adding a field is backward compatible: a target written before the field existed decodes with it empty.

type Object

type Object struct {
	Name    string    `json:"name"`
	Size    int64     `json:"size"`
	ModTime time.Time `json:"mod_time"`
}

Object describes a stored archive on a target.

func Retention

func Retention(objects []Object, keepN, keepDays int, now time.Time) []Object

Retention selects which objects to delete given keep-N and keep-days rules. An object is kept if it is within the newest keepN OR newer than keepDays. keepN<=0 disables the count rule; keepDays<=0 disables the age rule.

type PBSSnapshot added in v0.3.6

type PBSSnapshot struct {
	Name      string    `json:"name"` // host/<id>/<rfc3339>, what restore takes
	Time      time.Time `json:"time"`
	Size      int64     `json:"size"`
	Protected bool      `json:"protected"`
}

PBSSnapshot is one restorable point in time.

func ParsePBSSnapshots added in v0.3.6

func ParsePBSSnapshots(out []byte) ([]PBSSnapshot, error)

ParsePBSSnapshots reads `snapshot list --output-format json`, newest first.

The client prints progress to stderr and JSON to stdout, but a mixed stream is easy to end up with, so this tolerates leading noise by seeking to the first "[". Anything that is not an object with a backup-time is skipped rather than failing the whole listing: one unreadable entry must not hide the snapshots an operator is trying to restore from.

type RcloneRunner added in v0.3.33

type RcloneRunner interface {
	RunRclone(ctx context.Context, conf []byte, argv []string, stdin io.Reader, stdout io.Writer) (newConf []byte, stderr string, err error)
}

RcloneRunner runs one rclone invocation. conf is copied in before it starts; the config as rclone left it is returned, also on failure.

type Target

type Target interface {
	Put(ctx context.Context, name string, r io.Reader) (int64, error)
	Get(ctx context.Context, name string) (io.ReadCloser, error)
	List(ctx context.Context) ([]Object, error)
	Delete(ctx context.Context, name string) error
	Close() error
}

Target is a storage backend for archives.

func Open

func Open(cfg Config) (Target, error)

Open connects/opens a target from its config.

func OpenRclone added in v0.3.33

func OpenRclone(cfg Config, run RcloneRunner, mu *sync.Mutex, onConf func([]byte)) (Target, error)

OpenRclone opens an rclone target. onConf is called with the config whenever a run changed it (a refreshed token), so it can be stored. mu serialises runs against the same login; the caller keeps one per target.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址