origin_serve

package
v0.0.0-...-a1a4906 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: Apache-2.0 Imports: 76 Imported by: 0

Documentation

Overview

Package origin_serve provides checksum computation and caching for origin servers.

Security: All checksum operations use os.Root (Go 1.24+) to prevent symlink traversal attacks that could escape the storage directory. The os.Root type provides a secure filesystem view that prevents access to files outside the designated root directory, similar to a chroot jail but implemented at the Go runtime level without requiring special system privileges.

Index

Constants

View Source
const (
	MetadataStatusHeader    = "X-Pelican-Metadata-Status"
	MetadataQueryURLHeader  = "X-Pelican-Metadata-Query-Url"
	MetadataManageURLHeader = "X-Pelican-Metadata-Manage-Url"
)

Response headers the origin sets on a PUT so the uploading client learns the initial publish outcome and (when the row persists) its capability URLs.

View Source
const (
	// ObjectCommitEventType — a new object was committed (created).
	ObjectCommitEventType = "object.committed"
	// ObjectUpdatedEventType — an existing object was overwritten (a re-PUT
	// of a path the tracking DB already knew) or changed out-of-band.
	ObjectUpdatedEventType = "object.updated"
	// ObjectDeletedEventType — an object was removed (client DELETE or an
	// out-of-band deletion the origin observed).
	ObjectDeletedEventType = "object.deleted"
)

Event-type discriminators written into the wire `type` field.

View Source
const IdempotencyKeyHeader = "X-Pelican-Idempotency-Key"

IdempotencyKeyHeader is the header that carries the event UUID on the outgoing webhook request. The well-known `Idempotency-Key` header is still an IETF draft (draft-ietf-httpapi-idempotency-key); per Pelican convention any non-RFC header is namespaced with the `X-Pelican-` prefix so receivers can tell at a glance which header belongs to Pelican vs the broader HTTP ecosystem. If the IETF promotes the bare name to an RFC, we'll add an alias.

View Source
const ObjectMetadataHeader = "X-Pelican-Object-Metadata"

ObjectMetadataHeader is the request header clients use to attach custom fields to an upload. Its value is parsed as an RFC 9651 dictionary.

The client side declares the same header name as client.ObjectMetadataHeaderName; the two are deliberately duplicated rather than shared because the client and origin_serve packages do not import each other. Keep the two values identical.

View Source
const PelicanDAVNamespace = "https://pelicanplatform.org/dav/"

PelicanDAVNamespace is the XML namespace URI for Pelican-specific dead properties. Exported so tests and downstream consumers can reference the same URI.

View Source
const PermanentRejectStatus = 422

PermanentRejectStatus is the HTTP status a catalog returns to say "this metadata event is permanently unacceptable — stop retrying it." It is the wire contract shared with external catalog operators (see docs/metadata-publish-design.md). We use 422 Unprocessable Content: the request was well-formed and authenticated, but the event's content is semantically rejected. Every other non-2xx (including auth 401/403 and 5xx) is treated as transient.

View Source
const PropSourceEtag = "source-etag"

PropSourceEtag is the local name of the source-etag property. Full XML name is {PelicanDAVNamespace}source-etag.

Variables

View Source
var ErrEventNotFound = errors.New("metadata publish event not found")

ErrEventNotFound is returned by mutators when no row matches the supplied event_id.

View Source
var ErrMetadataBacklogFull = errors.New("metadata publish queue backlog full")

ErrMetadataBacklogFull is returned by CommitEvent (eventual mode) when the namespace's pending publish queue is at its configured cap. It fails the close hook, so POSC rolls back the object: the origin refuses to accept an upload whose metadata it cannot durably queue.

View Source
var ErrNotSupported = errors.New("operation not supported by backend")

ErrNotSupported is returned when an operation is not supported by the backend (e.g. Mkdir on a plain HTTP server). Callers can test for this with errors.Is(err, ErrNotSupported).

View Source
var ErrObjectMetadataNotFound = errors.New("object_metadata: row not found")

ErrObjectMetadataNotFound is the sentinel some helpers return when callers prefer it over a nil row.

View Source
var ReservedCustomFieldKeys = []string{"path", "size", "etag", "created_at"}

ReservedCustomFieldKeys lists keys that the client cannot set in the header — they are populated by the origin from the storage commit itself.

Functions

func BackendETag

func BackendETag(info os.FileInfo) string

BackendETag asks `info` for its ETag. Returns the empty string if info is nil or the FileInfo's backend declined / errored. The metadata publish path treats an empty ETag as "no etag known" and emits the field anyway (with that empty value); operators who care about a non-empty ETag in their webhook should ensure their backend implements BackendETager.

IMPORTANT: this function deliberately does NOT synthesize an ETag. "How is an ETag computed?" is a backend question; centralizing the answer here would tie this layer to a particular convention (e.g. `<size>-<mtime>`), and that convention is wrong on every backend that has its own canonical ETag (S3, anything object-store-shaped).

func BackendForPrefix

func BackendForPrefix(prefix string) server_utils.OriginBackend

BackendForPrefix returns the storage backend serving a federation prefix, or nil if no export claims it.

The backend is how this process reaches its own storage. Anything that wants to read an export -- listing it to measure disk usage, say -- should go through here rather than addressing the origin over the network: the bytes are already local, and a request that leaves the process needs a federation, a credential, and an initialized client to come back.

func ContextWithContentLength

func ContextWithContentLength(ctx context.Context, size int64) context.Context

ContextWithContentLength returns a child context carrying the expected upload size. The blob filesystem's OpenFile will use this to hint the underlying writer, enabling single-PUT uploads for small objects.

func GetAuthConfig

func GetAuthConfig() *authConfig

GetAuthConfig returns the global auth config

func GetGlobusBackends

func GetGlobusBackends() map[string]GlobusBackendActivator

GetGlobusBackends returns the map of Globus v2 backends keyed by collection ID. This is used by the launcher to activate backends after Globus OAuth is initialized.

func InitAuthConfig

func InitAuthConfig(ctx context.Context, egrp *errgroup.Group, exports []server_utils.OriginExport) error

InitAuthConfig initializes the global auth config

func InitializeChecksummer

func InitializeChecksummer()

InitializeChecksummer initializes the global checksummer

func InitializeHandlers

func InitializeHandlers(ctx context.Context, exports []server_utils.OriginExport) error

InitializeHandlers initializes the WebDAV handlers for each export

func InvalidateChecksums

func InvalidateChecksums(root *os.Root, filename string) error

InvalidateChecksums removes every cached checksum xattr for a file. It must be called after a file is (over)written so a subsequent read recomputes the checksum from the new bytes instead of trusting a stale cached value.

Cached-checksum freshness is otherwise validated only by the file's mtime at one-second resolution (the XRootD XrdCks on-disk format stores mtime as a Unix second — see serializeXRootDChecksum). An overwrite within the same wall-clock second therefore leaves the old checksum looking valid, and the origin would report a digest that does not match the bytes it now serves — which a cache verifying the download rejects. Dropping the xattrs on write closes that race deterministically.

A missing xattr is not an error (nothing was cached yet). Uses the provided os.Root so all file access stays within the storage directory.

func LaunchGlobusv2TokenRefresh

func LaunchGlobusv2TokenRefresh(ctx context.Context, egrp *errgroup.Group)

LaunchGlobusv2TokenRefresh starts a periodic goroutine that refreshes the OAuth2 tokens for all activated Globus v2 backends. The refresh interval defaults to 5 min but can be overridden via Origin.Globusv2TokenRefreshInterval.

func NewGlobusBackend

func NewGlobusBackend(cfg GlobusBackendConfig) *globusBackend

NewGlobusBackend creates a new native Globus backend.

func ParseObjectMetadataHeader

func ParseObjectMetadataHeader(value string) (map[string]any, error)

ParseObjectMetadataHeader parses the value of an X-Pelican-Object-Metadata header. An empty / absent value yields an empty (non-nil) map. The returned map's values are JSON-serializable: strings, integers, floats, booleans, and (for byte sequences) base64-encoded strings.

Reserved keys (see ReservedCustomFieldKeys) are silently dropped with a non-fatal error if present; the caller may inspect the error to log or count rejections.

func RecordCommitCloseHook

func RecordCommitCloseHook(dao *objectMetadataDAO, namespace string, trackExtra bool) func(ctx context.Context, finalPath string, info os.FileInfo) error

RecordCommitCloseHook returns a closure suitable as a POSC / close- notify close hook that funnels the commit through the DAO's durable RecordCommit path. The closure pulls the actor (token sub) and custom-field extra map from the request context — both set by the upstream middlewares (authMiddleware, extractObjectMetadataFromRequest).

Returned hook is nil-safe: if dao is nil it's a no-op.

func RegisterHandlers

func RegisterHandlers(engine *gin.Engine, directorEnabled bool) error

RegisterHandlers registers the HTTP handlers with the Gin engine. When the director is also running in the same server, handlers are registered under /api/v1.0/origin/<prefix> so the director can distinguish between its routing and the origin's file serving. Otherwise, handlers are registered directly at the federation prefix for standalone origins.

func RegisterMetadataAdminAPI

func RegisterMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)

RegisterMetadataAdminAPI registers the metadata-queue admin endpoints on the supplied router group. Pass `gin.HandlerFunc`s for the auth middleware so the registrar doesn't need a hard dependency on web_ui.

func RegisterMetadataStatusAPI

func RegisterMetadataStatusAPI(rg *gin.RouterGroup)

RegisterMetadataStatusAPI mounts the capability endpoints on rg. It is registered WITHOUT auth middleware on purpose: the random token in the path is the capability. Call it with the same router group the admin API uses; the resulting paths are <group>/metadata_publish/:token.

func RegisterObjectMetadataAdminAPI

func RegisterObjectMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)

RegisterObjectMetadataAdminAPI registers the routes on the given router group. Pass any auth middlewares the caller wants applied.

func RegisterStorageAPI

func RegisterStorageAPI(router *gin.RouterGroup, auth ...gin.HandlerFunc)

RegisterStorageAPI mounts the administrative storage-backend routes.

Registering nothing when the backend is not a pstore is deliberate: a 404 tells an operator plainly that this origin has no such interface, which is better than a route that exists and always fails.

The middleware is variadic because admin authorization takes two handlers, not one: web_ui.AdminAuthHandler only inspects the "User" the authentication handler put in the context, so on its own it rejects everyone with a 401 and the authorization it implements never runs. Every other admin route in the server registers the pair (see local_cache.RegisterPersistentCacheAPI, web_ui.RegisterPprofRoutes, director's RegisterDirectorWebAPI).

func ResetHandlers

func ResetHandlers()

ResetHandlers resets the handler state (for testing)

func ResetPStoreState

func ResetPStoreState()

ResetPStoreState drops the shared stores between handler initializations. Only tests re-initialize handlers within one process.

func ShutdownAuthConfig

func ShutdownAuthConfig()

ShutdownAuthConfig stops the auth config's background processes

func WithClientToken

func WithClientToken(ctx context.Context, token string) context.Context

WithClientToken stores the client's bearer token in the context so that the HTTPS backend can forward it to the upstream server.

Types

type AuthError

type AuthError struct {
	Code    int
	Message string
	Path    string
	User    string
	Issuer  string
	// contains filtered or unexported fields
}

AuthError represents an authorization error with optional context

func NewAuthError

func NewAuthError(code int, message string, path string) *AuthError

NewAuthError creates a new authorization error

func (*AuthError) Error

func (ae *AuthError) Error() string

Error implements the error interface

func (*AuthError) WithCause

func (ae *AuthError) WithCause(err error) *AuthError

WithCause adds the underlying error cause

func (*AuthError) WithIssuer

func (ae *AuthError) WithIssuer(issuer string) *AuthError

WithIssuer adds issuer information to the error

func (*AuthError) WithUser

func (ae *AuthError) WithUser(user string) *AuthError

WithUser adds user information to the error

type BackendETager

type BackendETager interface {
	ETag(ctx context.Context) (string, error)
}

BackendETager is the optional interface a FileInfo (or its underlying value) implements to supply a backend-supplied ETag. The contract matches golang.org/x/net/webdav's internal ETager: return the bare ETag string, including any quotes the wire format demands.

type BackendMode

type BackendMode int
const (
	BackendModeUnknown BackendMode = iota
	BackendModeWebDAV
	BackendModeHTTP
)

type BatchedStmt

type BatchedStmt struct {
	SQL  string
	Args []any
}

BatchedStmt is the public alias the DAO uses to compose a multi-statement op via EnqueueDurableBatch / EnqueueBestEffortBatch.

type BatcherHooks

type BatcherHooks struct {
	// IncFlush is called once per successful or failed COMMIT. The
	// `size` is the number of ops in the batch.
	IncFlush func(size int)
	// IncError is called once per failed COMMIT. (Each failed op
	// also reports its own error via its `done` channel.)
	IncError func()
	// ObserveBatchAge records how long the oldest op waited from
	// enqueue to flush.
	ObserveBatchAge func(d time.Duration)
	// ObserveEnqueueWait is called only when an Enqueue had to
	// block on the channel send (i.e., the in-memory buffer was
	// full). The `durability` label is "durable" or "best_effort".
	// A non-zero count here means callers are paying real
	// back-pressure latency; operators should bump
	// Origin.Metadata.BatchBufferSize.
	ObserveEnqueueWait func(durability string, d time.Duration)
}

BatcherHooks lets the metrics package observe batcher activity without the batcher importing prometheus.

type BlobBackendOptions

type BlobBackendOptions struct {
	// Generic gocloud.dev/blob URL — takes precedence over the S3-specific fields.
	BlobURL string

	// S3-specific fields (used only when BlobURL is empty).
	ServiceURL string // e.g. "https://s3.us-east-1.amazonaws.com"
	Region     string
	Bucket     string
	AccessKey  string
	SecretKey  string
	URLStyle   string // "path" or "virtual" (default: "path")

	// Common fields.
	StoragePrefix string // optional key prefix within the bucket/container
}

BlobBackendOptions groups the parameters needed to construct a blob backend. There are two ways to open a bucket:

  1. Set BlobURL to a gocloud.dev URL (e.g. "s3://bucket", "gs://bucket", "azblob://container").
  2. Set the S3-specific fields (ServiceURL, Region, Bucket, etc.) for backwards-compatible S3 config.

If BlobURL is set it takes precedence.

type BlobFileSysInfo

type BlobFileSysInfo struct {
	ETag string
}

BlobFileSysInfo is returned by blobFileInfo.Sys() when metadata is available.

type ChecksumType

type ChecksumType string

ChecksumType represents the type of checksum

const (
	ChecksumTypeMD5    ChecksumType = "md5"
	ChecksumTypeSHA1   ChecksumType = "sha1"
	ChecksumTypeCRC32  ChecksumType = "crc32"
	ChecksumTypeCRC32C ChecksumType = "crc32c"
)

type Checksummer

type Checksummer interface {
	GetChecksum(root *os.Root, filename string, checksumType ChecksumType) (string, error)
}

Checksummer is an interface for fetching and computing checksums Uses os.Root to ensure all file operations stay within the root directory

func GetChecksummer

func GetChecksummer() Checksummer

GetChecksummer returns the global checksummer

type ConditionalWriteFs

type ConditionalWriteFs interface {
	OpenFileConditional(name string, flag int, perm os.FileMode, size int64,
		requireAbsent bool, requireETag string) (afero.File, error)
}

ConditionalWriteFs is optionally implemented by an afero.Fs that can enforce an HTTP conditional-write precondition (RFC 9110 §13.1) atomically, at the moment the new version is installed.

checkPutPreconditions evaluates If-Match / If-None-Match before the write begins, which is inherently a stat-then-write race: two concurrent "If-None-Match: *" PUTs both observe an absent object and both commit. A backend that can re-check the condition inside its commit transaction -- pstore does -- closes that race, so the handler hands the parsed condition down here rather than keeping it to itself.

The condition is expressed with plain scalars rather than a shared struct so that a backend package does not have to import origin_serve to implement it. requireAbsent is "If-None-Match: *"; requireETag is a single-tag "If-Match", carrying the entity tag exactly as the client sent it (the backend, which minted it, is the only layer that knows how to interpret it). size is the Content-Length hint, or -1 when unknown, matching SizeHintFs.

type CustomFields

type CustomFields map[string]any

CustomFields is the fully-translated, JSON-ready custom field map that came in via X-Pelican-Object-Metadata. We keep it as a typed alias for readability.

type ErrorHandler

type ErrorHandler struct {
	// contains filtered or unexported fields
}

ErrorHandler manages consistent error handling and HTTP status code mapping

func NewErrorHandler

func NewErrorHandler() *ErrorHandler

NewErrorHandler creates a new error handler

func (*ErrorHandler) LogError

func (eh *ErrorHandler) LogError(operation string, err error, args ...interface{})

LogError logs an error with appropriate context

func (*ErrorHandler) MapToHTTPStatus

func (eh *ErrorHandler) MapToHTTPStatus(err error) int

MapToHTTPStatus maps an error to an HTTP status code using proper error type checking

type EtagSource

type EtagSource string

EtagSource is the discriminator persisted on every row that tells the operator where the ETag came from. Lets a reader distinguish "this is the backend's canonical answer" from "the origin filled in because the backend declined."

const (
	EtagSourceBackend EtagSource = "backend"
	EtagSourceOrigin  EtagSource = "origin"
)

type GlobusBackendActivator

type GlobusBackendActivator interface {
	// Activate marks the collection as activated with the given tokens and OAuth2 config.
	Activate(collectionToken, transferToken *oauth2.Token, httpsServer string, oauth2Cfg *oauth2.Config)
	// RefreshTokens refreshes both the collection and transfer tokens.
	RefreshTokens() error
	// IsActivated returns whether the Globus collection has been activated.
	IsActivated() bool
}

GlobusBackendActivator provides methods for activating and refreshing a Globus v2 backend. It is the exported interface that external packages (e.g. launchers) use to manage Globus backends.

type GlobusBackendConfig

type GlobusBackendConfig struct {
	// CollectionID is the Globus collection UUID
	CollectionID string
	// HTTPSServer is the HTTPS URL for the collection (e.g. https://g-12345.data.globus.org)
	HTTPSServer string
	// StoragePrefix is the path prefix within the collection
	StoragePrefix string
	// OAuth2Config for refreshing tokens
	OAuth2Config *oauth2.Config
	// CollectionToken is the initial collection access token
	CollectionToken *oauth2.Token
	// TransferToken is the initial transfer access token
	TransferToken *oauth2.Token
}

GlobusBackendConfig holds the parameters needed to construct a Globus backend.

type HTTPSBackendOptions

type HTTPSBackendOptions struct {
	ServiceURL    string
	StoragePrefix string
	TokenMode     HTTPSTokenMode
	// For static tokens:
	StaticTokenFile string
	// For OAuth2 tokens:
	OAuth2Config *oauth2.Config
	OAuth2Token  *oauth2.Token // initial token (with refresh_token)
	// EnableAutoMkdir, when true, causes PUT operations to automatically
	// create missing parent directories via WebDAV MKCOL before retrying.
	EnableAutoMkdir bool
}

HTTPSBackendOptions groups the parameters needed to construct an HTTPS backend.

type HTTPSFileSysInfo

type HTTPSFileSysInfo struct {
	ETag string
}

HTTPSFileSysInfo carries optional metadata (e.g. ETag) from an upstream HTTPS/WebDAV server. Returned by httpsFileInfo.Sys() when populated.

type HTTPSTokenMode

type HTTPSTokenMode int
const (
	// HTTPSTokenNone — no token is sent to the backend.
	HTTPSTokenNone HTTPSTokenMode = iota
	// HTTPSTokenStatic — a static bearer token read from a file is sent.
	HTTPSTokenStatic
	// HTTPSTokenPassthrough — the client-supplied token is forwarded.
	HTTPSTokenPassthrough
	// HTTPSTokenOAuth2 — an OAuth2 access token is acquired and refreshed automatically.
	HTTPSTokenOAuth2
)

type Mapfile

type Mapfile struct {
	Rules []MapfileRule `json:"-"`
	// contains filtered or unexported fields
}

Mapfile represents the complete mapfile configuration

func NewMapfile

func NewMapfile(filePath string) *Mapfile

NewMapfile creates a new mapfile loader

func (*Mapfile) ApplyRule

func (m *Mapfile) ApplyRule(rule MapfileRule, userClaim, groupClaims []string, requestPath string) string

ApplyRule applies a mapfile rule to determine the mapped username Returns the mapped username if the rule matches, empty string if no match

func (*Mapfile) IsStale

func (m *Mapfile) IsStale() bool

IsStale checks if the mapfile has been modified on disk

func (*Mapfile) Load

func (m *Mapfile) Load() error

Load reads and parses the mapfile from disk

func (*Mapfile) MapUsername

func (m *Mapfile) MapUsername(userClaim string, groupClaims []string, requestPath string) string

MapUsername applies mapfile rules to determine the mapped username

type MapfileRule

type MapfileRule struct {
	Sub      *string `json:"sub"`
	Username *string `json:"username"`
	Path     *string `json:"path"`
	Group    *string `json:"group"`
	Result   string  `json:"result"`
	Ignore   *bool   `json:"ignore"`
	Comment  *string `json:"comment"`
}

MapfileRule represents a single rule in the mapfile for username mapping

type MetadataPublishRow

type MetadataPublishRow struct {
	ID                  int64     `gorm:"primaryKey;autoIncrement"`
	EventID             string    `gorm:"column:event_id;uniqueIndex;not null"`
	Namespace           string    `gorm:"column:namespace;not null;index"`
	ObjectPath          string    `gorm:"column:object_path;not null"`
	ObjectSize          int64     `gorm:"column:object_size;not null"`
	ETag                string    `gorm:"column:etag;not null"`
	ObjectCreated       time.Time `gorm:"column:object_created;not null"`
	CustomFields        string    `gorm:"column:custom_fields;not null;default:'{}'"`
	MetadataContentType string    `gorm:"column:metadata_content_type;not null;default:''"`
	MetadataBody        []byte    `gorm:"column:metadata_body;not null;default:''"`
	CreatedAt           time.Time `gorm:"column:created_at;not null;index"`
	NextAttemptAt       time.Time `gorm:"column:next_attempt_at;not null;index"`
	Attempts            int       `gorm:"column:attempts;not null;default:0"`
	LastError           string    `gorm:"column:last_error;not null;default:''"`

	// State is "pending" (the worker keeps retrying) or "rejected" (terminal:
	// the catalog answered 422 / permanently-bad, so the worker stops touching
	// it; the row lingers for status queries and admin/operator deletion).
	State string `gorm:"column:state;not null;default:'pending'"`

	// QueryToken / ManageToken are per-row random capability tokens embedded in
	// the status / manage URLs handed back to the uploading client in eventual
	// mode. Possession of the (unguessable) URL is the authority — no separate
	// bearer token is required. Empty for transactional-mode rows (which never
	// outlive the request).
	QueryToken  string `gorm:"column:query_token;not null;default:''"`
	ManageToken string `gorm:"column:manage_token;not null;default:''"`

	// EventType is the wire `type` (object.committed / object.updated /
	// object.deleted). Persisted so a retry rebuilds the event with its
	// original type, and so the worker knows to skip the "object still exists?"
	// check for deletes.
	EventType string `gorm:"column:event_type;not null;default:'object.committed'"`
}

MetadataPublishRow is the GORM-mapped persistence representation of one queued event. The struct tag `gorm:"unique"` on event_id matches the migration's UNIQUE constraint.

MetadataContentType and MetadataBody are populated only when the upload was multipart/form-data (see splitMultipartParts in metadata_multipart.go); both columns are empty for header-only and no-metadata uploads, and the worker chooses plain-JSON vs multipart/related outbound based on whether the blob is non-empty.

func (MetadataPublishRow) TableName

func (MetadataPublishRow) TableName() string

TableName overrides the default GORM pluralization rule.

type NotFoundError

type NotFoundError struct {
	ResourceType string
	ResourcePath string
}

NotFoundError is a specialized error for missing resources

func NewNotFoundError

func NewNotFoundError(resourceType, resourcePath string) *NotFoundError

NewNotFoundError creates a new not found error

func (*NotFoundError) Error

func (nfe *NotFoundError) Error() string

Error implements the error interface

func (*NotFoundError) HTTPStatus

func (nfe *NotFoundError) HTTPStatus() int

HTTPStatus returns the appropriate HTTP status code

type ObjectChecksumRow

type ObjectChecksumRow struct {
	ID            int64     `gorm:"primaryKey;autoIncrement"`
	ObjectID      int64     `gorm:"column:object_id;not null"`
	Algorithm     string    `gorm:"column:algorithm;not null"`
	Value         string    `gorm:"column:value;not null"`
	ComputedAt    time.Time `gorm:"column:computed_at;not null"`
	EtagAtCompute string    `gorm:"column:etag_at_compute;not null"`
}

ObjectChecksumRow is the GORM-mapped projection of object_checksums.

func (ObjectChecksumRow) TableName

func (ObjectChecksumRow) TableName() string

type ObjectCommitEvent

type ObjectCommitEvent struct {
	// ID is a server-side UUIDv4. Stable across retries; used for
	// receiver-side dedup.
	ID string

	// Type is the event-type discriminator. Today: ObjectCommitEventType.
	Type string

	// Timestamp is when this event was first generated.
	Timestamp time.Time

	// Namespace is the federation prefix of the export this object
	// belongs to.
	Namespace string

	// Federation is the federation discovery URL the origin belongs to.
	// It makes the event self-describing: a receiver uses it to locate
	// the registry (federation discovery -> registry endpoint) and fetch
	// this namespace's registered public keys to verify the webhook JWT,
	// without needing to reach the origin directly. Stamped by the
	// publisher at send time.
	Federation string

	// ObjectPath is the federation-relative path of the object.
	ObjectPath string

	// ObjectSize is the size in bytes as reported by the storage
	// backend after commit.
	ObjectSize int64

	// ETag is the entity tag computed by the storage backend (eg the
	// WebDAV-generated ETag). The origin does not introduce its own
	// hashing scheme.
	ETag string

	// ObjectCreated is the storage-reported commit / mtime.
	ObjectCreated time.Time

	// CustomFields are uploader-supplied fields (parsed from the
	// X-Pelican-Object-Metadata header). Reserved keys are stripped
	// at parse time.
	CustomFields CustomFields

	// MetadataContentType + MetadataBody carry the opaque-blob
	// metadata supplied via the multipart/form-data upload path
	// (see metadata_multipart.go). Empty content type AND empty body
	// means "no blob, post plain JSON outbound." A non-empty body
	// switches the outbound webhook to multipart/related so the
	// receiver gets the bytes byte-for-byte. They are independent
	// of CustomFields: an upload may carry both.
	MetadataContentType string
	MetadataBody        []byte
}

ObjectCommitEvent is the in-memory representation of a publishable object-commit event. Its JSON form (per MarshalJSON below) is the webhook wire shape.

func EventFromRow

func EventFromRow(r *MetadataPublishRow) (*ObjectCommitEvent, error)

EventFromRow reconstructs an in-memory event from a persisted row. Used by both publish paths when handing rows to the HTTP attempt.

func NewObjectCommitEvent

func NewObjectCommitEvent(namespace, objectPath string, size int64, etag string, created time.Time, custom CustomFields) *ObjectCommitEvent

NewObjectCommitEvent constructs a fresh event with a server-generated UUID and Type/Timestamp pre-populated.

func NewObjectDeletedEvent

func NewObjectDeletedEvent(namespace, objectPath string) *ObjectCommitEvent

NewObjectDeletedEvent builds an object.deleted event. The object is gone, so size/etag are zero/empty; created_at carries the deletion time so the wire field is never a zero timestamp. The catalog keys off `type` + `path`.

func NewObjectUpdatedEvent

func NewObjectUpdatedEvent(namespace, objectPath string, size int64, etag string, modified time.Time, custom CustomFields) *ObjectCommitEvent

NewObjectUpdatedEvent builds an object.updated event (overwrite or out-of-band modification). `modified` is the object's current mtime.

func (*ObjectCommitEvent) HasMetadataBlob

func (e *ObjectCommitEvent) HasMetadataBlob() bool

HasMetadataBlob reports whether the event carries an opaque blob, driving the outbound publish path between plain-JSON and multipart/related.

func (*ObjectCommitEvent) MarshalJSON

func (e *ObjectCommitEvent) MarshalJSON() ([]byte, error)

MarshalJSON renders the event in the wire shape:

{
  "id": "...", "type": "object.committed", "timestamp": "...",
  "namespace": "/foo",
  "object": { "path": ..., "size": ..., "etag": ..., "created_at": ..., ...inlined customs }
  // when a blob is attached:
  "metadata": { "content_type": "application/xml", "size": 12345 }
}

The blob *bytes themselves* are not in the JSON — they ride as the second part of a multipart/related body when the outbound mode is switched on (see metadata_publisher.go). The JSON "metadata" sub-object is a *descriptor* the receiver can read before pulling the blob part.

func (*ObjectCommitEvent) WithMetadataBlob

func (e *ObjectCommitEvent) WithMetadataBlob(contentType string, body []byte) *ObjectCommitEvent

WithMetadataBlob attaches an opaque metadata blob to the event in place and returns the same pointer (for fluent construction). Empty contentType+body is a no-op so the blob path stays opt-in.

type ObjectMetadataEvent

type ObjectMetadataEvent string

ObjectMetadataEvent is the high-level discriminator written to object_metadata_history.event_type.

const (
	ObjectEventCommit          ObjectMetadataEvent = "commit"
	ObjectEventDelete          ObjectMetadataEvent = "delete"
	ObjectEventRename          ObjectMetadataEvent = "rename"
	ObjectEventExternalObserve ObjectMetadataEvent = "external_observe"
	ObjectEventExternalModify  ObjectMetadataEvent = "external_modify"
	ObjectEventExternalDelete  ObjectMetadataEvent = "external_delete"
)

type ObjectMetadataEventInput

type ObjectMetadataEventInput struct {
	Namespace    string
	ObjectPath   string // federation-rooted, eg "/exp/data/run99.dat"
	Size         int64
	ETag         string
	EtagSource   EtagSource
	BackendMtime time.Time
	Actor        string         // token sub at request time; "" if unknown
	Extra        map[string]any // X-Pelican-Object-Metadata; ignored if TrackExtra=false for the ns
	TrackExtra   bool           // resolved at call time from per-export overrides
	// SourceEtag is the ETag returned by a remote source during a
	// third-party copy that produced this commit. Empty for direct
	// PUTs; RecordCommit persists NULL in that case, which clears
	// any stale source_etag from a prior TPC.
	SourceEtag string
}

ObjectMetadataEventInput is the call-site struct for every Record* method. Centralizing the field-set keeps the DAO API stable across the four event types we record.

type ObjectMetadataHistoryRow

type ObjectMetadataHistoryRow struct {
	ID            int64      `gorm:"primaryKey;autoIncrement"`
	EventID       string     `gorm:"column:event_id;uniqueIndex;not null"`
	Namespace     string     `gorm:"column:namespace;not null"`
	ObjectPath    string     `gorm:"column:object_path;not null"`
	EventType     string     `gorm:"column:event_type;not null"`
	EventTS       time.Time  `gorm:"column:event_ts;not null"`
	Size          *int64     `gorm:"column:size"`
	ETag          *string    `gorm:"column:etag"`
	EtagSource    *string    `gorm:"column:etag_source"`
	BackendMtime  *time.Time `gorm:"column:backend_mtime"`
	ChecksumsJSON string     `gorm:"column:checksums_json;not null;default:'{}'"`
	Actor         string     `gorm:"column:actor;not null;default:''"`
	Extra         string     `gorm:"column:extra;not null;default:'{}'"`
	// SourceEtag is a snapshot of the live row's source_etag at the
	// moment of the event. Populated on the 'source_etag' event only;
	// other event types leave it NULL.
	SourceEtag *string `gorm:"column:source_etag"`
}

ObjectMetadataHistoryRow is the GORM-mapped projection of object_metadata_history.

func (ObjectMetadataHistoryRow) TableName

func (ObjectMetadataHistoryRow) TableName() string

type ObjectMetadataRow

type ObjectMetadataRow struct {
	ID           int64      `gorm:"primaryKey;autoIncrement"`
	Namespace    string     `gorm:"column:namespace;not null;index"`
	ObjectPath   string     `gorm:"column:object_path;not null"`
	Size         int64      `gorm:"column:size;not null"`
	ETag         string     `gorm:"column:etag;not null"`
	EtagSource   string     `gorm:"column:etag_source;not null;default:'backend'"`
	BackendMtime time.Time  `gorm:"column:backend_mtime;not null"`
	CreatedAt    time.Time  `gorm:"column:created_at;not null"`
	LastModified time.Time  `gorm:"column:last_modified;not null"`
	LastAccessed *time.Time `gorm:"column:last_accessed"`
	Actor        string     `gorm:"column:actor;not null;default:''"`
	DeletedAt    *time.Time `gorm:"column:deleted_at"`
	// SourceEtag is the ETag returned by the remote source during a
	// third-party copy (TPC) that produced this row. NULL for objects
	// uploaded directly. Cleared by any subsequent write that
	// invalidates the object (RecordCommit, RecordExternalChange)
	// because the stored ETag would no longer describe the source.
	SourceEtag *string `gorm:"column:source_etag"`

	// PublishedAt / PublishedEtag record whether this origin has
	// successfully published a commit webhook for the live object, and at
	// what content version. NULL PublishedAt (or PublishedEtag != ETag after
	// an overwrite) marks the row as owing a publish; the reconcile sweep
	// uses this to recover objects whose publish was lost to a crash between
	// the storage rename and the durable queue write.
	PublishedAt   *time.Time `gorm:"column:published_at"`
	PublishedEtag *string    `gorm:"column:published_etag"`
}

ObjectMetadataRow is the GORM-mapped projection of object_metadata. The schema (including the partial unique index that distinguishes live from soft-deleted rows) is defined by the Goose migration, not by GORM's auto-migrate.

func (ObjectMetadataRow) TableName

func (ObjectMetadataRow) TableName() string

type OperationError

type OperationError struct {
	Operation   string
	Path        string
	HTTPStatus  int
	Message     string
	Recoverable bool
	// contains filtered or unexported fields
}

OperationError represents a file operation error with recovery hints

func NewOperationError

func NewOperationError(operation, path string, err error) *OperationError

NewOperationError creates a new operation error

func (*OperationError) Error

func (oe *OperationError) Error() string

Error implements the error interface

func (*OperationError) String

func (oe *OperationError) String() string

String returns a detailed error description

type PermissionDeniedError

type PermissionDeniedError struct {
	Resource string
	Action   string
	Reason   string
}

PermissionDeniedError is a specialized authorization error

func NewPermissionDeniedError

func NewPermissionDeniedError(resource, action, reason string) *PermissionDeniedError

NewPermissionDeniedError creates a new permission denied error

func (*PermissionDeniedError) Error

func (pde *PermissionDeniedError) Error() string

Error implements the error interface

func (*PermissionDeniedError) HTTPStatus

func (pde *PermissionDeniedError) HTTPStatus() int

HTTPStatus returns the appropriate HTTP status code

type PoscMetricsHooks

type PoscMetricsHooks struct {
	IncActive func()
	DecActive func()
	IncExpire func()
	// IncRollbackFailed is called when the close hook fails AND the
	// best-effort RemoveAll of the just-renamed final path also
	// fails. Invoked at most once per close. Receives no namespace
	// label here — the metrics package translates it through the
	// closure that built the POSC layer.
	IncRollbackFailed func()
}

PoscMetricsHooks is a small set of optional hooks the POSC layer calls for observability. Wired up by the metrics package; the layer functions fine if these are nil.

type PrunerHooks

type PrunerHooks struct {
	// IncDeleted is called once per loop iteration that deleted at
	// least one row. `namespace` is the export's federation prefix;
	// `count` is the number of rows deleted in that iteration.
	IncDeleted func(namespace string, count int64)
	// ObservePassDuration is called once per full pass (all
	// namespaces), with the total wall-clock spent.
	ObservePassDuration func(d time.Duration)
}

PrunerHooks lets the metrics package observe pruner activity.

type PublishMode

type PublishMode string

PublishMode is the per-event behavior knob.

const (
	ModeTransactional PublishMode = "transactional"
	ModeEventual      PublishMode = "eventual"
)

func (PublishMode) IsValid

func (m PublishMode) IsValid() bool

IsValid reports whether m is a recognized mode.

type ResourceExhaustedError

type ResourceExhaustedError struct {
	ResourceType string
	Message      string
}

ResourceExhaustedError represents resource limit violations

func NewResourceExhaustedError

func NewResourceExhaustedError(resourceType, message string) *ResourceExhaustedError

NewResourceExhaustedError creates a new resource exhausted error

func (*ResourceExhaustedError) Error

func (ree *ResourceExhaustedError) Error() string

Error implements the error interface

func (*ResourceExhaustedError) HTTPStatus

func (ree *ResourceExhaustedError) HTTPStatus() int

HTTPStatus returns the appropriate HTTP status code

type SizeHintFs

type SizeHintFs interface {
	OpenFileSized(name string, flag int, perm os.FileMode, size int64) (afero.File, error)
}

SizeHintFs is optionally implemented by an afero.Fs whose write path can do better when the object's final size is known up front.

afero.Fs has no context and no size argument, so a backend that would benefit -- pstore picks its storage tier from the size, and buffers to discover it otherwise -- cannot see the Content-Length the handler already has. This carries it across that gap.

A size of -1 means unknown, which is what a chunked-encoding PUT gives.

type TokenValidationError

type TokenValidationError struct {
	Reason   string
	Issuer   string
	Subject  string
	Code     int
	Details  string
	Verified bool // Whether the token signature was verified before extracting metadata
}

TokenValidationError represents token validation failures

func NewTokenValidationError

func NewTokenValidationError(reason string) *TokenValidationError

NewTokenValidationError creates a new token validation error

func (*TokenValidationError) Error

func (tve *TokenValidationError) Error() string

Error implements the error interface

func (*TokenValidationError) String

func (tve *TokenValidationError) String() string

String returns a detailed error description

func (*TokenValidationError) WithDetails

func (tve *TokenValidationError) WithDetails(details string) *TokenValidationError

WithDetails adds additional context details

func (*TokenValidationError) WithIssuer

func (tve *TokenValidationError) WithIssuer(issuer string) *TokenValidationError

WithIssuer adds issuer information

func (*TokenValidationError) WithSubject

func (tve *TokenValidationError) WithSubject(subject string) *TokenValidationError

WithSubject adds subject information

func (*TokenValidationError) WithVerified

func (tve *TokenValidationError) WithVerified(verified bool) *TokenValidationError

WithVerified marks whether the token was cryptographically verified before extracting metadata

type UserInfo

type UserInfo struct {
	User        string
	Groups      []string
	MappedUser  string
	RequestPath string
}

UserInfo represents extracted user and group information from a token

type UserMapper

type UserMapper struct {
	// contains filtered or unexported fields
}

UserMapper handles username and group extraction from tokens with optional mapfile support

func NewUserMapper

func NewUserMapper(usernameClaim, groupsClaim, mapfilePath, defaultUser, unauthenticatedUser string) *UserMapper

NewUserMapper creates a new user mapper usernameClaim: token claim to use for username (default: "sub") groupsClaim: token claim to use for groups (default: "wlcg.groups") mapfilePath: optional path to mapfile for username mapping defaultUser: fallback username when mapfile is enabled but no rule matches;

set to empty string to reject unmatched tokens.

unauthenticatedUser: username for token-less or invalid-token requests;

set to empty string to reject such requests.

func (*UserMapper) ExtractUserInfo

func (um *UserMapper) ExtractUserInfo(tokenClaims map[string]interface{}, requestPath string) *UserInfo

ExtractUserInfo extracts user and group information from token claims

func (*UserMapper) MapTokenToUser

func (um *UserMapper) MapTokenToUser(tokenStr string) *userInfo

MapTokenToUser parses a JWT token and extracts user/group information with optional mapfile mapping The token is parsed without verification (assumes it was already verified upstream) Returns a userInfo struct suitable for caching with token authorization information

func (*UserMapper) RefreshMapfile

func (um *UserMapper) RefreshMapfile() error

RefreshMapfile reloads the mapfile if it has changed on disk

func (*UserMapper) Shutdown

func (um *UserMapper) Shutdown()

Shutdown stops the periodic refresh goroutine and cleans up resources

func (*UserMapper) StartPeriodicRefresh

func (um *UserMapper) StartPeriodicRefresh(interval time.Duration)

StartPeriodicRefresh starts a goroutine that periodically checks and reloads the mapfile if it has been modified on disk. The refresh interval is controlled by the provided interval parameter.

The goroutine will run until the UserMapper's context is cancelled (typically on server shutdown via Shutdown() method).

type XattrChecksummer

type XattrChecksummer struct{}

XattrChecksummer uses extended attributes to store and retrieve checksums

func (*XattrChecksummer) GetChecksum

func (xc *XattrChecksummer) GetChecksum(root *os.Root, filename string, checksumType ChecksumType) (string, error)

GetChecksum retrieves or computes the checksum for a file Uses the provided os.Root to ensure all file operations stay within the root directory

func (*XattrChecksummer) GetChecksumRFC3230

func (xc *XattrChecksummer) GetChecksumRFC3230(root *os.Root, filename string, checksumType ChecksumType) (string, error)

GetChecksumRFC3230 retrieves the checksum in RFC 3230 format (algorithm=value) MD5 and SHA1 are base64-encoded, CRC32 is hex-encoded Uses the provided os.Root to ensure all file operations stay within the root directory

func (*XattrChecksummer) GetChecksumsRFC3230

func (xc *XattrChecksummer) GetChecksumsRFC3230(root *os.Root, filename string, types []ChecksumType) ([]string, error)

GetChecksumsRFC3230 returns a list of RFC 3230 digest strings for requested types. If any requested checksum is missing or stale, computes all requested plus defaults and stores. Uses the provided os.Root to ensure all file operations stay within the root directory

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址