Documentation
¶
Overview ¶
Package media decodes, resizes, and stores photos, and defines a job pattern for doing this in the background: fetch a source, derive variants, store them, and record what was stored.
Recording what was stored makes a re-run idempotent: a run that is interrupted and retried does not re-download or re-derive variants that already landed. Fetch failures are split into permanent (the source will never be available again) and transient (retry later), so a permanently missing source is marked as such instead of being retried on every run. Decode enforces a maximum input size, a maximum decoded pixel count, and checks the actual content type of the input rather than a caller-supplied one. All three are required together: a small, validly-encoded file can still decode to a very large pixel buffer, so the byte limit alone does not bound memory use.
Store is the storage interface variants are written through. LocalStore is the default implementation, backed by the local filesystem, with Handler to serve it over HTTP; it requires no external service. GCSStore is an alternative implementation backed by Google Cloud Storage, for a caller that has a bucket. The two implementations do not depend on each other.
Index ¶
- Constants
- Variables
- func DownscaleTo(src image.Image, maxWidth int) image.Image
- func EncodeJPEG(img image.Image, quality int) ([]byte, error)
- type DecodeOptions
- type Decoded
- type DerivedVariant
- type Deriver
- type FallbackOptions
- type FallbackReport
- type Fetcher
- type GCSOption
- type GCSStore
- func (s *GCSStore) Delete(ctx context.Context, key string) error
- func (s *GCSStore) Get(ctx context.Context, key string) ([]byte, error)
- func (s *GCSStore) Put(ctx context.Context, key string, body []byte, contentType string) error
- func (s *GCSStore) URL(ctx context.Context, key string, ttl time.Duration) (string, error)
- type Item
- type Lister
- type LocalStore
- func (s *LocalStore) Close() error
- func (s *LocalStore) Delete(_ context.Context, key string) error
- func (s *LocalStore) Get(_ context.Context, key string) ([]byte, error)
- func (s *LocalStore) Handler(prefix string) http.Handler
- func (s *LocalStore) Put(_ context.Context, key string, body []byte, contentType string) error
- func (s *LocalStore) URL(_ context.Context, key string, _ time.Duration) (string, error)
- type Options
- type Putter
- type Recorder
- type Resized
- type Result
- type Signer
- type Store
- type Variant
Constants ¶
const DefaultMaxBytes = 24 << 20 // 24 MiB
DefaultMaxBytes caps how much of an untrusted source Decode will buffer before giving up. A typical photo from a phone or a third-party feed is in the hundreds of kilobytes to a few megabytes; this is set well above that and well below a size that would threaten a job's memory limit if a source serves an unexpectedly large file.
This bounds the encoded input only. A small, validly-encoded file can still decode to a very large pixel buffer — see DefaultMaxPixels, the separate bound that exists for exactly that case.
const DefaultMaxPixels = 60_000_000 // 60 megapixels
DefaultMaxPixels caps the decoded image's width times height. A 168 KB PNG can legitimately decode to well over 100 million pixels if it is mostly a uniform color, since PNG's compression has no relationship to decoded size; DefaultMaxBytes does not protect against this at all. Set well above any real photo (a 48-megapixel phone sensor is under 50 million pixels) and well below a size that would threaten a job's memory limit once decoded to raw pixels.
Variables ¶
var DefaultAllowedTypes = []string{"image/jpeg", "image/png", "image/gif"}
DefaultAllowedTypes are the content types Decode accepts when Options.AllowedTypes is nil. Limited to what imaging.Decode can actually decode — accepting a type here that the decoder cannot read would turn a clear "unsupported type" error into a confusing decode failure instead.
var ErrNotFound = errors.New("media: object not found")
ErrNotFound is returned by Get when key does not exist. URL does not check existence in either implementation this package provides, so it never returns ErrNotFound.
var ErrPermanent = errors.New("media: source permanently unavailable")
ErrPermanent marks a source a Fetcher's origin will never serve again: a 404 from a media host that has withdrawn the asset, as opposed to a timeout or a 5xx. Run treats it as Dropped rather than Failed, so a withdrawn source is not requeued on later runs.
var ErrTooLarge = errors.New("media: source exceeds max bytes")
ErrTooLarge is returned when a source exceeds Options.MaxBytes.
var ErrTooManyPixels = errors.New("media: source exceeds max pixels")
ErrTooManyPixels is returned when a source's decoded dimensions exceed Options.MaxPixels.
var ErrUnsupportedType = errors.New("media: unsupported content type")
ErrUnsupportedType is returned when the sniffed content type is not in Options.AllowedTypes.
Functions ¶
func DownscaleTo ¶
DownscaleTo narrows src to maxWidth, preserving aspect ratio, and never upscales: a source already narrower than maxWidth is returned unchanged. Upscaling would produce a derivative that is larger, blurrier, and heavier than its source, with no benefit. maxWidth <= 0 returns src unchanged.
Types ¶
type DecodeOptions ¶
DecodeOptions configures Decode. The zero value is a working default: 24MiB byte cap, 60-megapixel decoded-size cap, JPEG/PNG/GIF only.
type Decoded ¶
Decoded is the result of a successful Decode.
func Decode ¶
func Decode(r io.Reader, opts DecodeOptions) (*Decoded, error)
Decode reads r up to opts.MaxBytes, sniffs the real content type from the bytes themselves (never a caller-supplied or header-claimed one), rejects anything outside opts.AllowedTypes, and decodes with EXIF orientation applied — a photo shot in portrait on a phone held sideways comes out upright here, before a resize bakes the wrong orientation into every derivative it produces downstream.
It reads one byte past the limit so an exactly-at-the-limit source is distinguishable from a truncated-because-longer one; DefaultMaxBytes never legitimately fits a real photo that tightly, so tripping it is itself the signal that something is wrong with the source, not the read.
type DerivedVariant ¶
type DerivedVariant struct {
// Name identifies the variant for Recorder bookkeeping (e.g. "large").
Name string
// Key is the full Store key to write Body under.
Key string
Body []byte
ContentType string
}
DerivedVariant is one variant a Deriver produced, ready to store.
type Deriver ¶
type Deriver func(ctx context.Context, body []byte) ([]DerivedVariant, error)
Deriver turns fetched source bytes into the variants to store.
type FallbackOptions ¶
type FallbackOptions struct {
Options
// Limit caps how many pending items one sweep pulls from the Lister.
// <= 0 defaults to 200.
Limit int
}
FallbackOptions configures RunFallback.
type FallbackReport ¶
type FallbackReport struct {
Attempted, Succeeded, Skipped, Dropped, Failed int
BacklogBefore, BacklogAfter int
}
FallbackReport is one sweep's outcome. BacklogBefore/After are -1 when the Lister could not report depth, which is itself worth surfacing rather than silently treated as zero.
func RunFallback ¶
func RunFallback(ctx context.Context, lister Lister, fetch Fetcher, derive Deriver, st Putter, rec Recorder, opts FallbackOptions) (FallbackReport, error)
RunFallback runs one sweep over whatever the Lister currently reports as pending. It is meant to be scheduled less often, and with a smaller Limit, than whatever primary job drains the same queue; RunFallback retries items the primary job did not get to or could not complete.
RunFallback calls Run internally with a Lister-selected batch, rather than reimplementing retry and idempotence logic. A separate implementation for the fallback path could disagree with Run about what "done" means for an item.
type Fetcher ¶
Fetcher downloads a source. Implementations should retry transient failures internally and wrap ErrPermanent around an answer that will never change.
type GCSOption ¶
type GCSOption func(*GCSStore)
GCSOption configures a GCSStore. GCSStore has no other exported way to set these, keeping NewGCSStore's required arguments to exactly what every caller needs and the optional ones opt-in.
func Public ¶
func Public() GCSOption
Public marks the bucket as one that serves objects at a stable public URL (https://storage.googleapis.com/<bucket>/<key>) with no signing. Use this only for a bucket whose IAM policy already grants public read — it does not change the bucket's ACLs itself.
func WithCacheControl ¶
WithCacheControl sets the Cache-Control header written on every Put. The empty default leaves GCS's own default in place.
func WithDefaultTTL ¶
WithDefaultTTL sets the signed URL lifetime URL uses when the caller passes ttl <= 0. Defaults to one hour.
func WithSigner ¶
WithSigner supplies the Signer a private bucket needs for URL. Required unless Public() is set; URL returns an error without one.
type GCSStore ¶
type GCSStore struct {
// contains filtered or unexported fields
}
GCSStore implements Store on Google Cloud Storage.
func NewGCSStore ¶
NewGCSStore wraps bucketName as a Store using client.
func (*GCSStore) Put ¶
Put implements Store.
Every upload carries the body's MD5, which GCS validates server-side, and its hex SHA-256 as object metadata. After the write succeeds, Put reads the stored object's metadata back and compares size, MD5, and SHA-256 against what was sent; on any mismatch it re-uploads — an overwrite of the same key, so the retry is idempotent — up to maxPutAttempts times. A partial upload that succeeds at the HTTP layer but lands truncated is therefore detected and replaced instead of being kept as the canonical variant.
type Item ¶
type Item struct {
// Key identifies this item to the Recorder and typically prefixes the
// Store keys its variants are written under.
Key string
SourceURL string
// ExpectedVariants are the variant names this item should end up with.
// An item where Recorder.Done already reports every one of these is
// skipped without a fetch — this is what makes a re-run idempotent
// instead of merely retry-safe.
ExpectedVariants []string
}
Item is one unit of pipeline work.
type Lister ¶
type Lister interface {
Pending(ctx context.Context, limit int) ([]Item, error)
BacklogDepth(ctx context.Context) (int, error)
}
Lister selects the next batch of incomplete items to (re)try, and reports the queue's total depth. Depth is separate from the per-run success and failure counts: a run can report success=N failed=0 on every execution while the backlog stays the same size, if new incomplete items are added at the same rate the run processes them. Depth is the only signal that distinguishes that case from an empty queue.
type LocalStore ¶
type LocalStore struct {
// contains filtered or unexported fields
}
LocalStore implements Store on the local filesystem: for tests, local development, and any deployment with no object storage at all. It never signs anything — URL just joins BaseURL and key — so whatever serves BaseURL is responsible for actually exposing the store over HTTP (see Handler).
All filesystem access goes through an os.Root opened on the store's root directory. os.Root refuses to resolve any path component, including through a symbolic link, to a location outside that directory — a plain string check on the key cannot catch a symlink placed inside the root that points outside it, since the escape happens at the filesystem level, not in the path string.
func NewLocalStore ¶
func NewLocalStore(root, baseURL string) (*LocalStore, error)
NewLocalStore creates the root directory if needed and returns a LocalStore rooted there. baseURL is prefixed to a key to form the URL LocalStore reports; pass "" if nothing serves these files over HTTP yet.
func (*LocalStore) Close ¶
func (s *LocalStore) Close() error
Close releases the root directory handle. A LocalStore used for the life of a process does not need to call this.
func (*LocalStore) Delete ¶
func (s *LocalStore) Delete(_ context.Context, key string) error
Delete implements Store. Deleting a key that does not exist is not an error.
func (*LocalStore) Handler ¶
func (s *LocalStore) Handler(prefix string) http.Handler
Handler serves LocalStore's contents over HTTP: GET/HEAD only, no directory listing, the exact content type Put was given (falling back to net/http's own sniff of the bytes if that metadata is missing), and X-Content-Type-Options: nosniff on every response — so a stored file is never reinterpreted by a browser as a more dangerous type (an uploaded .svg served as image/svg+xml, for instance, is active content at the media origin) than the one it was declared to be.
Reads go through the same os.Root LocalStore itself uses for Get and Put, so a symlink placed inside the store cannot be used to read a file outside it through this handler either.
func (*LocalStore) Put ¶
Put implements Store. The object and its content type are each written to a uniquely-named temporary file first, and neither is renamed into place until both writes have succeeded — so two concurrent Puts of the same key never race on one temp path, a reader never observes a partially written object, and a Put that fails while overwriting an existing key leaves the previous object exactly as it was rather than deleting it: nothing about the live key is touched until there is a complete replacement ready to swap in. The two renames themselves cannot be made atomic together without OS support neither this filesystem nor most others offer; if the first rename succeeds and the second does not, the object has been updated but the content type may briefly lag behind it, which Handler tolerates by falling back to no explicit Content-Type.
type Options ¶
type Options struct {
Concurrency int
// Logger receives warnings for problems that do not fail the item outright
// (for example, a Recorder write failing after the Store write it was
// recording already succeeded). Falls back to slog.Default() when nil;
// Options never calls slog.SetDefault — that is the caller binary's
// decision, not this package's.
Logger *slog.Logger
}
Options configures Run and RunFallback. The zero value is a working default: concurrency 8, logging through slog.Default().
type Putter ¶
type Putter interface {
Put(ctx context.Context, key string, body []byte, contentType string) error
}
Putter is the write side of Store — all Run needs to land a variant. Any Store satisfies it for free; declaring it separately keeps a caller who only wants to run the pipeline against a stub from having to fake Get, Delete and URL as well.
type Recorder ¶
type Recorder interface {
// Done reports the variant names already stored for key, from a
// previous run.
Done(ctx context.Context, key string) (map[string]bool, error)
// RecordVariant marks one variant stored for key.
RecordVariant(ctx context.Context, key, variantName string) error
// RecordDropped marks key's source as permanently unavailable.
RecordDropped(ctx context.Context, key string) error
}
Recorder is the durable record of one Item's progress: which variants have already been stored, so a re-run does not redo finished work, and which sources are permanently gone, so they are not retried. A caller implements Recorder against whatever storage it already has: a Postgres table, a KV store, a map in a test.
type Resized ¶
Resized is one Variant's output image, before encoding.
func Chain ¶
Chain resizes src into every variant in order, each one resampled from the previous result rather than from src again.
Callers must list variants widest first. Resampling a 1200px "large" down to a 400px "thumb" costs less than resampling the full source twice, and produces a sharper thumbnail because it is derived from an already-decoded image with no intervening lossy encode. Chain does not sort the input; an incorrectly ordered list produces a visibly wrong result rather than being silently corrected.
type Result ¶
Result is one Item's outcome. Exactly one of Skipped, Dropped, or Err is meaningful; Stored is set alongside any of them and lists whatever did land this run even if the item did not fully complete.
func Run ¶
func Run(ctx context.Context, items []Item, fetch Fetcher, derive Deriver, st Putter, rec Recorder, opts Options) []Result
Run processes items concurrently, bounded by opts.Concurrency. It never returns an error itself — one item's failure is reported in its own Result, not as a call failure, so one bad source cannot abort a whole batch.
type Signer ¶
type Signer interface {
SignURL(ctx context.Context, bucket, object string, ttl time.Duration) (string, error)
}
Signer signs a GCS object URL for temporary, unauthenticated access. It is declared here, not imported from a signing library, because the caller supplies it: signing needs a service-account key or IAM credentials that a generic Store implementation should never have to know how to obtain.
type Store ¶
type Store interface {
// Put writes body under key with the given content type, replacing
// whatever was there. Calling it twice with the same key and body is not
// an error — that repeatability is what lets a pipeline re-run stay
// idempotent.
Put(ctx context.Context, key string, body []byte, contentType string) error
// Get reads the object back. Returns ErrNotFound if key does not exist.
Get(ctx context.Context, key string) ([]byte, error)
// Delete removes the object. Deleting a key that does not exist is not an
// error — a caller cleaning up after a failed run should not have to
// check existence first.
Delete(ctx context.Context, key string) error
// URL returns an address a client can fetch key from directly. ttl is a
// hint: an implementation backed by a public bucket ignores it and
// returns a stable public URL; one backed by a private bucket returns a
// signed URL that expires around ttl. ttl <= 0 asks for the
// implementation's own default. URL does not check that key exists.
URL(ctx context.Context, key string, ttl time.Duration) (string, error)
}
Store puts and fetches opaque byte objects under a key, and mints a URL a client can fetch the object from directly — a stable public URL for a public bucket, or a signed, expiring one for a private bucket or a local filesystem with no public serving path of its own.
Store has no listing, no metadata beyond content type, and no versioning. A media pipeline needs exactly these four operations; anything more belongs in a dedicated object-storage client rather than in this interface, where every implementation would have to support it.
type Variant ¶
type Variant struct {
// Name identifies the variant (e.g. "large", "thumb") and is typically
// folded into the storage key by the caller.
Name string
// MaxWidth caps the derivative's width, preserving aspect ratio. Zero
// keeps the source's width unchanged.
MaxWidth int
// Quality is the JPEG encode quality, 1-100.
Quality int
}
Variant is one derivative size to produce from a decoded source.