media

package
v0.0.0-...-6ebc918 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package media decodes, resizes, and stores photos, and defines a job pattern for doing this in the background: fetch a source, derive variants, store them, and record what was stored.

Recording what was stored makes a re-run idempotent: a run that is interrupted and retried does not re-download or re-derive variants that already landed. Fetch failures are split into permanent (the source will never be available again) and transient (retry later), so a permanently missing source is marked as such instead of being retried on every run. Decode enforces a maximum input size, a maximum decoded pixel count, and checks the actual content type of the input rather than a caller-supplied one. All three are required together: a small, validly-encoded file can still decode to a very large pixel buffer, so the byte limit alone does not bound memory use.

Store is the storage interface variants are written through. LocalStore is the default implementation, backed by the local filesystem, with Handler to serve it over HTTP; it requires no external service. GCSStore is an alternative implementation backed by Google Cloud Storage, for a caller that has a bucket. The two implementations do not depend on each other.

Index

Constants

View Source
const DefaultMaxBytes = 24 << 20 // 24 MiB

DefaultMaxBytes caps how much of an untrusted source Decode will buffer before giving up. A typical photo from a phone or a third-party feed is in the hundreds of kilobytes to a few megabytes; this is set well above that and well below a size that would threaten a job's memory limit if a source serves an unexpectedly large file.

This bounds the encoded input only. A small, validly-encoded file can still decode to a very large pixel buffer — see DefaultMaxPixels, the separate bound that exists for exactly that case.

View Source
const DefaultMaxPixels = 60_000_000 // 60 megapixels

DefaultMaxPixels caps the decoded image's width times height. A 168 KB PNG can legitimately decode to well over 100 million pixels if it is mostly a uniform color, since PNG's compression has no relationship to decoded size; DefaultMaxBytes does not protect against this at all. Set well above any real photo (a 48-megapixel phone sensor is under 50 million pixels) and well below a size that would threaten a job's memory limit once decoded to raw pixels.

Variables

View Source
var DefaultAllowedTypes = []string{"image/jpeg", "image/png", "image/gif"}

DefaultAllowedTypes are the content types Decode accepts when Options.AllowedTypes is nil. Limited to what imaging.Decode can actually decode — accepting a type here that the decoder cannot read would turn a clear "unsupported type" error into a confusing decode failure instead.

View Source
var ErrNotFound = errors.New("media: object not found")

ErrNotFound is returned by Get when key does not exist. URL does not check existence in either implementation this package provides, so it never returns ErrNotFound.

View Source
var ErrPermanent = errors.New("media: source permanently unavailable")

ErrPermanent marks a source a Fetcher's origin will never serve again: a 404 from a media host that has withdrawn the asset, as opposed to a timeout or a 5xx. Run treats it as Dropped rather than Failed, so a withdrawn source is not requeued on later runs.

View Source
var ErrTooLarge = errors.New("media: source exceeds max bytes")

ErrTooLarge is returned when a source exceeds Options.MaxBytes.

View Source
var ErrTooManyPixels = errors.New("media: source exceeds max pixels")

ErrTooManyPixels is returned when a source's decoded dimensions exceed Options.MaxPixels.

View Source
var ErrUnsupportedType = errors.New("media: unsupported content type")

ErrUnsupportedType is returned when the sniffed content type is not in Options.AllowedTypes.

Functions

func DownscaleTo

func DownscaleTo(src image.Image, maxWidth int) image.Image

DownscaleTo narrows src to maxWidth, preserving aspect ratio, and never upscales: a source already narrower than maxWidth is returned unchanged. Upscaling would produce a derivative that is larger, blurrier, and heavier than its source, with no benefit. maxWidth <= 0 returns src unchanged.

func EncodeJPEG

func EncodeJPEG(img image.Image, quality int) ([]byte, error)

EncodeJPEG encodes img as a JPEG at the given quality (1-100).

Types

type DecodeOptions

type DecodeOptions struct {
	MaxBytes     int64
	MaxPixels    int64
	AllowedTypes []string
}

DecodeOptions configures Decode. The zero value is a working default: 24MiB byte cap, 60-megapixel decoded-size cap, JPEG/PNG/GIF only.

type Decoded

type Decoded struct {
	Image       image.Image
	ContentType string
	Bytes       int64
}

Decoded is the result of a successful Decode.

func Decode

func Decode(r io.Reader, opts DecodeOptions) (*Decoded, error)

Decode reads r up to opts.MaxBytes, sniffs the real content type from the bytes themselves (never a caller-supplied or header-claimed one), rejects anything outside opts.AllowedTypes, and decodes with EXIF orientation applied — a photo shot in portrait on a phone held sideways comes out upright here, before a resize bakes the wrong orientation into every derivative it produces downstream.

It reads one byte past the limit so an exactly-at-the-limit source is distinguishable from a truncated-because-longer one; DefaultMaxBytes never legitimately fits a real photo that tightly, so tripping it is itself the signal that something is wrong with the source, not the read.

type DerivedVariant

type DerivedVariant struct {
	// Name identifies the variant for Recorder bookkeeping (e.g. "large").
	Name string
	// Key is the full Store key to write Body under.
	Key         string
	Body        []byte
	ContentType string
}

DerivedVariant is one variant a Deriver produced, ready to store.

type Deriver

type Deriver func(ctx context.Context, body []byte) ([]DerivedVariant, error)

Deriver turns fetched source bytes into the variants to store.

type FallbackOptions

type FallbackOptions struct {
	Options
	// Limit caps how many pending items one sweep pulls from the Lister.
	// <= 0 defaults to 200.
	Limit int
}

FallbackOptions configures RunFallback.

type FallbackReport

type FallbackReport struct {
	Attempted, Succeeded, Skipped, Dropped, Failed int
	BacklogBefore, BacklogAfter                    int
}

FallbackReport is one sweep's outcome. BacklogBefore/After are -1 when the Lister could not report depth, which is itself worth surfacing rather than silently treated as zero.

func RunFallback

func RunFallback(ctx context.Context, lister Lister, fetch Fetcher, derive Deriver, st Putter, rec Recorder, opts FallbackOptions) (FallbackReport, error)

RunFallback runs one sweep over whatever the Lister currently reports as pending. It is meant to be scheduled less often, and with a smaller Limit, than whatever primary job drains the same queue; RunFallback retries items the primary job did not get to or could not complete.

RunFallback calls Run internally with a Lister-selected batch, rather than reimplementing retry and idempotence logic. A separate implementation for the fallback path could disagree with Run about what "done" means for an item.

type Fetcher

type Fetcher interface {
	Fetch(ctx context.Context, sourceURL string) ([]byte, error)
}

Fetcher downloads a source. Implementations should retry transient failures internally and wrap ErrPermanent around an answer that will never change.

type GCSOption

type GCSOption func(*GCSStore)

GCSOption configures a GCSStore. GCSStore has no other exported way to set these, keeping NewGCSStore's required arguments to exactly what every caller needs and the optional ones opt-in.

func Public

func Public() GCSOption

Public marks the bucket as one that serves objects at a stable public URL (https://storage.googleapis.com/<bucket>/<key>) with no signing. Use this only for a bucket whose IAM policy already grants public read — it does not change the bucket's ACLs itself.

func WithCacheControl

func WithCacheControl(value string) GCSOption

WithCacheControl sets the Cache-Control header written on every Put. The empty default leaves GCS's own default in place.

func WithDefaultTTL

func WithDefaultTTL(ttl time.Duration) GCSOption

WithDefaultTTL sets the signed URL lifetime URL uses when the caller passes ttl <= 0. Defaults to one hour.

func WithSigner

func WithSigner(signer Signer) GCSOption

WithSigner supplies the Signer a private bucket needs for URL. Required unless Public() is set; URL returns an error without one.

type GCSStore

type GCSStore struct {
	// contains filtered or unexported fields
}

GCSStore implements Store on Google Cloud Storage.

func NewGCSStore

func NewGCSStore(client *storage.Client, bucketName string, opts ...GCSOption) *GCSStore

NewGCSStore wraps bucketName as a Store using client.

func (*GCSStore) Delete

func (s *GCSStore) Delete(ctx context.Context, key string) error

Delete implements Store.

func (*GCSStore) Get

func (s *GCSStore) Get(ctx context.Context, key string) ([]byte, error)

Get implements Store.

func (*GCSStore) Put

func (s *GCSStore) Put(ctx context.Context, key string, body []byte, contentType string) error

Put implements Store.

Every upload carries the body's MD5, which GCS validates server-side, and its hex SHA-256 as object metadata. After the write succeeds, Put reads the stored object's metadata back and compares size, MD5, and SHA-256 against what was sent; on any mismatch it re-uploads — an overwrite of the same key, so the retry is idempotent — up to maxPutAttempts times. A partial upload that succeeds at the HTTP layer but lands truncated is therefore detected and replaced instead of being kept as the canonical variant.

func (*GCSStore) URL

func (s *GCSStore) URL(ctx context.Context, key string, ttl time.Duration) (string, error)

URL implements Store.

type Item

type Item struct {
	// Key identifies this item to the Recorder and typically prefixes the
	// Store keys its variants are written under.
	Key       string
	SourceURL string
	// ExpectedVariants are the variant names this item should end up with.
	// An item where Recorder.Done already reports every one of these is
	// skipped without a fetch — this is what makes a re-run idempotent
	// instead of merely retry-safe.
	ExpectedVariants []string
}

Item is one unit of pipeline work.

type Lister

type Lister interface {
	Pending(ctx context.Context, limit int) ([]Item, error)
	BacklogDepth(ctx context.Context) (int, error)
}

Lister selects the next batch of incomplete items to (re)try, and reports the queue's total depth. Depth is separate from the per-run success and failure counts: a run can report success=N failed=0 on every execution while the backlog stays the same size, if new incomplete items are added at the same rate the run processes them. Depth is the only signal that distinguishes that case from an empty queue.

type LocalStore

type LocalStore struct {
	// contains filtered or unexported fields
}

LocalStore implements Store on the local filesystem: for tests, local development, and any deployment with no object storage at all. It never signs anything — URL just joins BaseURL and key — so whatever serves BaseURL is responsible for actually exposing the store over HTTP (see Handler).

All filesystem access goes through an os.Root opened on the store's root directory. os.Root refuses to resolve any path component, including through a symbolic link, to a location outside that directory — a plain string check on the key cannot catch a symlink placed inside the root that points outside it, since the escape happens at the filesystem level, not in the path string.

func NewLocalStore

func NewLocalStore(root, baseURL string) (*LocalStore, error)

NewLocalStore creates the root directory if needed and returns a LocalStore rooted there. baseURL is prefixed to a key to form the URL LocalStore reports; pass "" if nothing serves these files over HTTP yet.

func (*LocalStore) Close

func (s *LocalStore) Close() error

Close releases the root directory handle. A LocalStore used for the life of a process does not need to call this.

func (*LocalStore) Delete

func (s *LocalStore) Delete(_ context.Context, key string) error

Delete implements Store. Deleting a key that does not exist is not an error.

func (*LocalStore) Get

func (s *LocalStore) Get(_ context.Context, key string) ([]byte, error)

Get implements Store.

func (*LocalStore) Handler

func (s *LocalStore) Handler(prefix string) http.Handler

Handler serves LocalStore's contents over HTTP: GET/HEAD only, no directory listing, the exact content type Put was given (falling back to net/http's own sniff of the bytes if that metadata is missing), and X-Content-Type-Options: nosniff on every response — so a stored file is never reinterpreted by a browser as a more dangerous type (an uploaded .svg served as image/svg+xml, for instance, is active content at the media origin) than the one it was declared to be.

Reads go through the same os.Root LocalStore itself uses for Get and Put, so a symlink placed inside the store cannot be used to read a file outside it through this handler either.

func (*LocalStore) Put

func (s *LocalStore) Put(_ context.Context, key string, body []byte, contentType string) error

Put implements Store. The object and its content type are each written to a uniquely-named temporary file first, and neither is renamed into place until both writes have succeeded — so two concurrent Puts of the same key never race on one temp path, a reader never observes a partially written object, and a Put that fails while overwriting an existing key leaves the previous object exactly as it was rather than deleting it: nothing about the live key is touched until there is a complete replacement ready to swap in. The two renames themselves cannot be made atomic together without OS support neither this filesystem nor most others offer; if the first rename succeeds and the second does not, the object has been updated but the content type may briefly lag behind it, which Handler tolerates by falling back to no explicit Content-Type.

func (*LocalStore) URL

func (s *LocalStore) URL(_ context.Context, key string, _ time.Duration) (string, error)

URL implements Store. ttl is ignored: a local filesystem has no signing concept, so URL always returns the same BaseURL-prefixed path. It does not check that key exists.

type Options

type Options struct {
	Concurrency int
	// Logger receives warnings for problems that do not fail the item outright
	// (for example, a Recorder write failing after the Store write it was
	// recording already succeeded). Falls back to slog.Default() when nil;
	// Options never calls slog.SetDefault — that is the caller binary's
	// decision, not this package's.
	Logger *slog.Logger
}

Options configures Run and RunFallback. The zero value is a working default: concurrency 8, logging through slog.Default().

type Putter

type Putter interface {
	Put(ctx context.Context, key string, body []byte, contentType string) error
}

Putter is the write side of Store — all Run needs to land a variant. Any Store satisfies it for free; declaring it separately keeps a caller who only wants to run the pipeline against a stub from having to fake Get, Delete and URL as well.

type Recorder

type Recorder interface {
	// Done reports the variant names already stored for key, from a
	// previous run.
	Done(ctx context.Context, key string) (map[string]bool, error)
	// RecordVariant marks one variant stored for key.
	RecordVariant(ctx context.Context, key, variantName string) error
	// RecordDropped marks key's source as permanently unavailable.
	RecordDropped(ctx context.Context, key string) error
}

Recorder is the durable record of one Item's progress: which variants have already been stored, so a re-run does not redo finished work, and which sources are permanently gone, so they are not retried. A caller implements Recorder against whatever storage it already has: a Postgres table, a KV store, a map in a test.

type Resized

type Resized struct {
	Variant Variant
	Image   image.Image
}

Resized is one Variant's output image, before encoding.

func Chain

func Chain(src image.Image, variants []Variant) []Resized

Chain resizes src into every variant in order, each one resampled from the previous result rather than from src again.

Callers must list variants widest first. Resampling a 1200px "large" down to a 400px "thumb" costs less than resampling the full source twice, and produces a sharper thumbnail because it is derived from an already-decoded image with no intervening lossy encode. Chain does not sort the input; an incorrectly ordered list produces a visibly wrong result rather than being silently corrected.

type Result

type Result struct {
	Item    Item
	Stored  []string
	Skipped bool
	Dropped bool
	Err     error
}

Result is one Item's outcome. Exactly one of Skipped, Dropped, or Err is meaningful; Stored is set alongside any of them and lists whatever did land this run even if the item did not fully complete.

func Run

func Run(ctx context.Context, items []Item, fetch Fetcher, derive Deriver, st Putter, rec Recorder, opts Options) []Result

Run processes items concurrently, bounded by opts.Concurrency. It never returns an error itself — one item's failure is reported in its own Result, not as a call failure, so one bad source cannot abort a whole batch.

type Signer

type Signer interface {
	SignURL(ctx context.Context, bucket, object string, ttl time.Duration) (string, error)
}

Signer signs a GCS object URL for temporary, unauthenticated access. It is declared here, not imported from a signing library, because the caller supplies it: signing needs a service-account key or IAM credentials that a generic Store implementation should never have to know how to obtain.

type Store

type Store interface {
	// Put writes body under key with the given content type, replacing
	// whatever was there. Calling it twice with the same key and body is not
	// an error — that repeatability is what lets a pipeline re-run stay
	// idempotent.
	Put(ctx context.Context, key string, body []byte, contentType string) error

	// Get reads the object back. Returns ErrNotFound if key does not exist.
	Get(ctx context.Context, key string) ([]byte, error)

	// Delete removes the object. Deleting a key that does not exist is not an
	// error — a caller cleaning up after a failed run should not have to
	// check existence first.
	Delete(ctx context.Context, key string) error

	// URL returns an address a client can fetch key from directly. ttl is a
	// hint: an implementation backed by a public bucket ignores it and
	// returns a stable public URL; one backed by a private bucket returns a
	// signed URL that expires around ttl. ttl <= 0 asks for the
	// implementation's own default. URL does not check that key exists.
	URL(ctx context.Context, key string, ttl time.Duration) (string, error)
}

Store puts and fetches opaque byte objects under a key, and mints a URL a client can fetch the object from directly — a stable public URL for a public bucket, or a signed, expiring one for a private bucket or a local filesystem with no public serving path of its own.

Store has no listing, no metadata beyond content type, and no versioning. A media pipeline needs exactly these four operations; anything more belongs in a dedicated object-storage client rather than in this interface, where every implementation would have to support it.

type Variant

type Variant struct {
	// Name identifies the variant (e.g. "large", "thumb") and is typically
	// folded into the storage key by the caller.
	Name string
	// MaxWidth caps the derivative's width, preserving aspect ratio. Zero
	// keeps the source's width unchanged.
	MaxWidth int
	// Quality is the JPEG encode quality, 1-100.
	Quality int
}

Variant is one derivative size to produce from a decoded source.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址