Documentation
¶
Overview ¶
Package origin_serve provides checksum computation and caching for origin servers.
Security: All checksum operations use os.Root (Go 1.24+) to prevent symlink traversal attacks that could escape the storage directory. The os.Root type provides a secure filesystem view that prevents access to files outside the designated root directory, similar to a chroot jail but implemented at the Go runtime level without requiring special system privileges.
Index ¶
- Constants
- Variables
- func BackendETag(info os.FileInfo) string
- func BackendForPrefix(prefix string) server_utils.OriginBackend
- func ContextWithContentLength(ctx context.Context, size int64) context.Context
- func GetAuthConfig() *authConfig
- func GetGlobusBackends() map[string]GlobusBackendActivator
- func InitAuthConfig(ctx context.Context, egrp *errgroup.Group, exports []server_utils.OriginExport) error
- func InitializeChecksummer()
- func InitializeHandlers(ctx context.Context, exports []server_utils.OriginExport) error
- func InvalidateChecksums(root *os.Root, filename string) error
- func LaunchGlobusv2TokenRefresh(ctx context.Context, egrp *errgroup.Group)
- func NewGlobusBackend(cfg GlobusBackendConfig) *globusBackend
- func ParseObjectMetadataHeader(value string) (map[string]any, error)
- func RecordCommitCloseHook(dao *objectMetadataDAO, namespace string, trackExtra bool) func(ctx context.Context, finalPath string, info os.FileInfo) error
- func RegisterHandlers(engine *gin.Engine, directorEnabled bool) error
- func RegisterMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)
- func RegisterMetadataStatusAPI(rg *gin.RouterGroup)
- func RegisterObjectMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)
- func RegisterStorageAPI(router *gin.RouterGroup, auth ...gin.HandlerFunc)
- func ResetHandlers()
- func ResetPStoreState()
- func ShutdownAuthConfig()
- func WithClientToken(ctx context.Context, token string) context.Context
- type AuthError
- type BackendETager
- type BackendMode
- type BatchedStmt
- type BatcherHooks
- type BlobBackendOptions
- type BlobFileSysInfo
- type ChecksumType
- type Checksummer
- type ConditionalWriteFs
- type CustomFields
- type ErrorHandler
- type EtagSource
- type GlobusBackendActivator
- type GlobusBackendConfig
- type HTTPSBackendOptions
- type HTTPSFileSysInfo
- type HTTPSTokenMode
- type Mapfile
- type MapfileRule
- type MetadataPublishRow
- type NotFoundError
- type ObjectChecksumRow
- type ObjectCommitEvent
- func EventFromRow(r *MetadataPublishRow) (*ObjectCommitEvent, error)
- func NewObjectCommitEvent(namespace, objectPath string, size int64, etag string, created time.Time, ...) *ObjectCommitEvent
- func NewObjectDeletedEvent(namespace, objectPath string) *ObjectCommitEvent
- func NewObjectUpdatedEvent(namespace, objectPath string, size int64, etag string, modified time.Time, ...) *ObjectCommitEvent
- type ObjectMetadataEvent
- type ObjectMetadataEventInput
- type ObjectMetadataHistoryRow
- type ObjectMetadataRow
- type OperationError
- type PermissionDeniedError
- type PoscMetricsHooks
- type PrunerHooks
- type PublishMode
- type ResourceExhaustedError
- type SizeHintFs
- type TokenValidationError
- func (tve *TokenValidationError) Error() string
- func (tve *TokenValidationError) String() string
- func (tve *TokenValidationError) WithDetails(details string) *TokenValidationError
- func (tve *TokenValidationError) WithIssuer(issuer string) *TokenValidationError
- func (tve *TokenValidationError) WithSubject(subject string) *TokenValidationError
- func (tve *TokenValidationError) WithVerified(verified bool) *TokenValidationError
- type UserInfo
- type UserMapper
- func (um *UserMapper) ExtractUserInfo(tokenClaims map[string]interface{}, requestPath string) *UserInfo
- func (um *UserMapper) MapTokenToUser(tokenStr string) *userInfo
- func (um *UserMapper) RefreshMapfile() error
- func (um *UserMapper) Shutdown()
- func (um *UserMapper) StartPeriodicRefresh(interval time.Duration)
- type XattrChecksummer
- func (xc *XattrChecksummer) GetChecksum(root *os.Root, filename string, checksumType ChecksumType) (string, error)
- func (xc *XattrChecksummer) GetChecksumRFC3230(root *os.Root, filename string, checksumType ChecksumType) (string, error)
- func (xc *XattrChecksummer) GetChecksumsRFC3230(root *os.Root, filename string, types []ChecksumType) ([]string, error)
Constants ¶
const ( MetadataStatusHeader = "X-Pelican-Metadata-Status" MetadataQueryURLHeader = "X-Pelican-Metadata-Query-Url" MetadataManageURLHeader = "X-Pelican-Metadata-Manage-Url" )
Response headers the origin sets on a PUT so the uploading client learns the initial publish outcome and (when the row persists) its capability URLs.
const ( // ObjectCommitEventType — a new object was committed (created). ObjectCommitEventType = "object.committed" // ObjectUpdatedEventType — an existing object was overwritten (a re-PUT // of a path the tracking DB already knew) or changed out-of-band. ObjectUpdatedEventType = "object.updated" // ObjectDeletedEventType — an object was removed (client DELETE or an // out-of-band deletion the origin observed). ObjectDeletedEventType = "object.deleted" )
Event-type discriminators written into the wire `type` field.
const IdempotencyKeyHeader = "X-Pelican-Idempotency-Key"
IdempotencyKeyHeader is the header that carries the event UUID on the outgoing webhook request. The well-known `Idempotency-Key` header is still an IETF draft (draft-ietf-httpapi-idempotency-key); per Pelican convention any non-RFC header is namespaced with the `X-Pelican-` prefix so receivers can tell at a glance which header belongs to Pelican vs the broader HTTP ecosystem. If the IETF promotes the bare name to an RFC, we'll add an alias.
const ObjectMetadataHeader = "X-Pelican-Object-Metadata"
ObjectMetadataHeader is the request header clients use to attach custom fields to an upload. Its value is parsed as an RFC 9651 dictionary.
The client side declares the same header name as client.ObjectMetadataHeaderName; the two are deliberately duplicated rather than shared because the client and origin_serve packages do not import each other. Keep the two values identical.
const PelicanDAVNamespace = "https://pelicanplatform.org/dav/"
PelicanDAVNamespace is the XML namespace URI for Pelican-specific dead properties. Exported so tests and downstream consumers can reference the same URI.
const PermanentRejectStatus = 422
PermanentRejectStatus is the HTTP status a catalog returns to say "this metadata event is permanently unacceptable — stop retrying it." It is the wire contract shared with external catalog operators (see docs/metadata-publish-design.md). We use 422 Unprocessable Content: the request was well-formed and authenticated, but the event's content is semantically rejected. Every other non-2xx (including auth 401/403 and 5xx) is treated as transient.
const PropSourceEtag = "source-etag"
PropSourceEtag is the local name of the source-etag property. Full XML name is {PelicanDAVNamespace}source-etag.
Variables ¶
var ErrEventNotFound = errors.New("metadata publish event not found")
ErrEventNotFound is returned by mutators when no row matches the supplied event_id.
var ErrMetadataBacklogFull = errors.New("metadata publish queue backlog full")
ErrMetadataBacklogFull is returned by CommitEvent (eventual mode) when the namespace's pending publish queue is at its configured cap. It fails the close hook, so POSC rolls back the object: the origin refuses to accept an upload whose metadata it cannot durably queue.
var ErrNotSupported = errors.New("operation not supported by backend")
ErrNotSupported is returned when an operation is not supported by the backend (e.g. Mkdir on a plain HTTP server). Callers can test for this with errors.Is(err, ErrNotSupported).
var ErrObjectMetadataNotFound = errors.New("object_metadata: row not found")
ErrObjectMetadataNotFound is the sentinel some helpers return when callers prefer it over a nil row.
var ReservedCustomFieldKeys = []string{"path", "size", "etag", "created_at"}
ReservedCustomFieldKeys lists keys that the client cannot set in the header — they are populated by the origin from the storage commit itself.
Functions ¶
func BackendETag ¶
BackendETag asks `info` for its ETag. Returns the empty string if info is nil or the FileInfo's backend declined / errored. The metadata publish path treats an empty ETag as "no etag known" and emits the field anyway (with that empty value); operators who care about a non-empty ETag in their webhook should ensure their backend implements BackendETager.
IMPORTANT: this function deliberately does NOT synthesize an ETag. "How is an ETag computed?" is a backend question; centralizing the answer here would tie this layer to a particular convention (e.g. `<size>-<mtime>`), and that convention is wrong on every backend that has its own canonical ETag (S3, anything object-store-shaped).
func BackendForPrefix ¶
func BackendForPrefix(prefix string) server_utils.OriginBackend
BackendForPrefix returns the storage backend serving a federation prefix, or nil if no export claims it.
The backend is how this process reaches its own storage. Anything that wants to read an export -- listing it to measure disk usage, say -- should go through here rather than addressing the origin over the network: the bytes are already local, and a request that leaves the process needs a federation, a credential, and an initialized client to come back.
func ContextWithContentLength ¶
ContextWithContentLength returns a child context carrying the expected upload size. The blob filesystem's OpenFile will use this to hint the underlying writer, enabling single-PUT uploads for small objects.
func GetGlobusBackends ¶
func GetGlobusBackends() map[string]GlobusBackendActivator
GetGlobusBackends returns the map of Globus v2 backends keyed by collection ID. This is used by the launcher to activate backends after Globus OAuth is initialized.
func InitAuthConfig ¶
func InitAuthConfig(ctx context.Context, egrp *errgroup.Group, exports []server_utils.OriginExport) error
InitAuthConfig initializes the global auth config
func InitializeChecksummer ¶
func InitializeChecksummer()
InitializeChecksummer initializes the global checksummer
func InitializeHandlers ¶
func InitializeHandlers(ctx context.Context, exports []server_utils.OriginExport) error
InitializeHandlers initializes the WebDAV handlers for each export
func InvalidateChecksums ¶
InvalidateChecksums removes every cached checksum xattr for a file. It must be called after a file is (over)written so a subsequent read recomputes the checksum from the new bytes instead of trusting a stale cached value.
Cached-checksum freshness is otherwise validated only by the file's mtime at one-second resolution (the XRootD XrdCks on-disk format stores mtime as a Unix second — see serializeXRootDChecksum). An overwrite within the same wall-clock second therefore leaves the old checksum looking valid, and the origin would report a digest that does not match the bytes it now serves — which a cache verifying the download rejects. Dropping the xattrs on write closes that race deterministically.
A missing xattr is not an error (nothing was cached yet). Uses the provided os.Root so all file access stays within the storage directory.
func LaunchGlobusv2TokenRefresh ¶
LaunchGlobusv2TokenRefresh starts a periodic goroutine that refreshes the OAuth2 tokens for all activated Globus v2 backends. The refresh interval defaults to 5 min but can be overridden via Origin.Globusv2TokenRefreshInterval.
func NewGlobusBackend ¶
func NewGlobusBackend(cfg GlobusBackendConfig) *globusBackend
NewGlobusBackend creates a new native Globus backend.
func ParseObjectMetadataHeader ¶
ParseObjectMetadataHeader parses the value of an X-Pelican-Object-Metadata header. An empty / absent value yields an empty (non-nil) map. The returned map's values are JSON-serializable: strings, integers, floats, booleans, and (for byte sequences) base64-encoded strings.
Reserved keys (see ReservedCustomFieldKeys) are silently dropped with a non-fatal error if present; the caller may inspect the error to log or count rejections.
func RecordCommitCloseHook ¶
func RecordCommitCloseHook(dao *objectMetadataDAO, namespace string, trackExtra bool) func(ctx context.Context, finalPath string, info os.FileInfo) error
RecordCommitCloseHook returns a closure suitable as a POSC / close- notify close hook that funnels the commit through the DAO's durable RecordCommit path. The closure pulls the actor (token sub) and custom-field extra map from the request context — both set by the upstream middlewares (authMiddleware, extractObjectMetadataFromRequest).
Returned hook is nil-safe: if dao is nil it's a no-op.
func RegisterHandlers ¶
RegisterHandlers registers the HTTP handlers with the Gin engine. When the director is also running in the same server, handlers are registered under /api/v1.0/origin/<prefix> so the director can distinguish between its routing and the origin's file serving. Otherwise, handlers are registered directly at the federation prefix for standalone origins.
func RegisterMetadataAdminAPI ¶
func RegisterMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)
RegisterMetadataAdminAPI registers the metadata-queue admin endpoints on the supplied router group. Pass `gin.HandlerFunc`s for the auth middleware so the registrar doesn't need a hard dependency on web_ui.
func RegisterMetadataStatusAPI ¶
func RegisterMetadataStatusAPI(rg *gin.RouterGroup)
RegisterMetadataStatusAPI mounts the capability endpoints on rg. It is registered WITHOUT auth middleware on purpose: the random token in the path is the capability. Call it with the same router group the admin API uses; the resulting paths are <group>/metadata_publish/:token.
func RegisterObjectMetadataAdminAPI ¶
func RegisterObjectMetadataAdminAPI(rg *gin.RouterGroup, mw ...gin.HandlerFunc)
RegisterObjectMetadataAdminAPI registers the routes on the given router group. Pass any auth middlewares the caller wants applied.
func RegisterStorageAPI ¶
func RegisterStorageAPI(router *gin.RouterGroup, auth ...gin.HandlerFunc)
RegisterStorageAPI mounts the administrative storage-backend routes.
Registering nothing when the backend is not a pstore is deliberate: a 404 tells an operator plainly that this origin has no such interface, which is better than a route that exists and always fails.
The middleware is variadic because admin authorization takes two handlers, not one: web_ui.AdminAuthHandler only inspects the "User" the authentication handler put in the context, so on its own it rejects everyone with a 401 and the authorization it implements never runs. Every other admin route in the server registers the pair (see local_cache.RegisterPersistentCacheAPI, web_ui.RegisterPprofRoutes, director's RegisterDirectorWebAPI).
func ResetPStoreState ¶
func ResetPStoreState()
ResetPStoreState drops the shared stores between handler initializations. Only tests re-initialize handlers within one process.
func ShutdownAuthConfig ¶
func ShutdownAuthConfig()
ShutdownAuthConfig stops the auth config's background processes
Types ¶
type AuthError ¶
type AuthError struct {
Code int
Message string
Path string
User string
Issuer string
// contains filtered or unexported fields
}
AuthError represents an authorization error with optional context
func NewAuthError ¶
NewAuthError creates a new authorization error
func (*AuthError) WithIssuer ¶
WithIssuer adds issuer information to the error
type BackendETager ¶
BackendETager is the optional interface a FileInfo (or its underlying value) implements to supply a backend-supplied ETag. The contract matches golang.org/x/net/webdav's internal ETager: return the bare ETag string, including any quotes the wire format demands.
type BackendMode ¶
type BackendMode int
const ( BackendModeUnknown BackendMode = iota BackendModeWebDAV BackendModeHTTP )
type BatchedStmt ¶
BatchedStmt is the public alias the DAO uses to compose a multi-statement op via EnqueueDurableBatch / EnqueueBestEffortBatch.
type BatcherHooks ¶
type BatcherHooks struct {
// IncFlush is called once per successful or failed COMMIT. The
// `size` is the number of ops in the batch.
IncFlush func(size int)
// IncError is called once per failed COMMIT. (Each failed op
// also reports its own error via its `done` channel.)
IncError func()
// ObserveBatchAge records how long the oldest op waited from
// enqueue to flush.
ObserveBatchAge func(d time.Duration)
// ObserveEnqueueWait is called only when an Enqueue had to
// block on the channel send (i.e., the in-memory buffer was
// full). The `durability` label is "durable" or "best_effort".
// A non-zero count here means callers are paying real
// back-pressure latency; operators should bump
// Origin.Metadata.BatchBufferSize.
ObserveEnqueueWait func(durability string, d time.Duration)
}
BatcherHooks lets the metrics package observe batcher activity without the batcher importing prometheus.
type BlobBackendOptions ¶
type BlobBackendOptions struct {
// Generic gocloud.dev/blob URL — takes precedence over the S3-specific fields.
BlobURL string
// S3-specific fields (used only when BlobURL is empty).
ServiceURL string // e.g. "https://s3.us-east-1.amazonaws.com"
Region string
Bucket string
AccessKey string
SecretKey string
URLStyle string // "path" or "virtual" (default: "path")
// Common fields.
StoragePrefix string // optional key prefix within the bucket/container
}
BlobBackendOptions groups the parameters needed to construct a blob backend. There are two ways to open a bucket:
- Set BlobURL to a gocloud.dev URL (e.g. "s3://bucket", "gs://bucket", "azblob://container").
- Set the S3-specific fields (ServiceURL, Region, Bucket, etc.) for backwards-compatible S3 config.
If BlobURL is set it takes precedence.
type BlobFileSysInfo ¶
type BlobFileSysInfo struct {
ETag string
}
BlobFileSysInfo is returned by blobFileInfo.Sys() when metadata is available.
type ChecksumType ¶
type ChecksumType string
ChecksumType represents the type of checksum
const ( ChecksumTypeMD5 ChecksumType = "md5" ChecksumTypeSHA1 ChecksumType = "sha1" ChecksumTypeCRC32 ChecksumType = "crc32" ChecksumTypeCRC32C ChecksumType = "crc32c" )
type Checksummer ¶
type Checksummer interface {
GetChecksum(root *os.Root, filename string, checksumType ChecksumType) (string, error)
}
Checksummer is an interface for fetching and computing checksums Uses os.Root to ensure all file operations stay within the root directory
func GetChecksummer ¶
func GetChecksummer() Checksummer
GetChecksummer returns the global checksummer
type ConditionalWriteFs ¶
type ConditionalWriteFs interface {
OpenFileConditional(name string, flag int, perm os.FileMode, size int64,
requireAbsent bool, requireETag string) (afero.File, error)
}
ConditionalWriteFs is optionally implemented by an afero.Fs that can enforce an HTTP conditional-write precondition (RFC 9110 §13.1) atomically, at the moment the new version is installed.
checkPutPreconditions evaluates If-Match / If-None-Match before the write begins, which is inherently a stat-then-write race: two concurrent "If-None-Match: *" PUTs both observe an absent object and both commit. A backend that can re-check the condition inside its commit transaction -- pstore does -- closes that race, so the handler hands the parsed condition down here rather than keeping it to itself.
The condition is expressed with plain scalars rather than a shared struct so that a backend package does not have to import origin_serve to implement it. requireAbsent is "If-None-Match: *"; requireETag is a single-tag "If-Match", carrying the entity tag exactly as the client sent it (the backend, which minted it, is the only layer that knows how to interpret it). size is the Content-Length hint, or -1 when unknown, matching SizeHintFs.
type CustomFields ¶
CustomFields is the fully-translated, JSON-ready custom field map that came in via X-Pelican-Object-Metadata. We keep it as a typed alias for readability.
type ErrorHandler ¶
type ErrorHandler struct {
// contains filtered or unexported fields
}
ErrorHandler manages consistent error handling and HTTP status code mapping
func NewErrorHandler ¶
func NewErrorHandler() *ErrorHandler
NewErrorHandler creates a new error handler
func (*ErrorHandler) LogError ¶
func (eh *ErrorHandler) LogError(operation string, err error, args ...interface{})
LogError logs an error with appropriate context
func (*ErrorHandler) MapToHTTPStatus ¶
func (eh *ErrorHandler) MapToHTTPStatus(err error) int
MapToHTTPStatus maps an error to an HTTP status code using proper error type checking
type EtagSource ¶
type EtagSource string
EtagSource is the discriminator persisted on every row that tells the operator where the ETag came from. Lets a reader distinguish "this is the backend's canonical answer" from "the origin filled in because the backend declined."
const ( EtagSourceBackend EtagSource = "backend" EtagSourceOrigin EtagSource = "origin" )
type GlobusBackendActivator ¶
type GlobusBackendActivator interface {
// Activate marks the collection as activated with the given tokens and OAuth2 config.
Activate(collectionToken, transferToken *oauth2.Token, httpsServer string, oauth2Cfg *oauth2.Config)
// RefreshTokens refreshes both the collection and transfer tokens.
RefreshTokens() error
// IsActivated returns whether the Globus collection has been activated.
IsActivated() bool
}
GlobusBackendActivator provides methods for activating and refreshing a Globus v2 backend. It is the exported interface that external packages (e.g. launchers) use to manage Globus backends.
type GlobusBackendConfig ¶
type GlobusBackendConfig struct {
// CollectionID is the Globus collection UUID
CollectionID string
// HTTPSServer is the HTTPS URL for the collection (e.g. https://g-12345.data.globus.org)
HTTPSServer string
// StoragePrefix is the path prefix within the collection
StoragePrefix string
// OAuth2Config for refreshing tokens
OAuth2Config *oauth2.Config
// CollectionToken is the initial collection access token
CollectionToken *oauth2.Token
// TransferToken is the initial transfer access token
TransferToken *oauth2.Token
}
GlobusBackendConfig holds the parameters needed to construct a Globus backend.
type HTTPSBackendOptions ¶
type HTTPSBackendOptions struct {
ServiceURL string
StoragePrefix string
TokenMode HTTPSTokenMode
// For static tokens:
StaticTokenFile string
// For OAuth2 tokens:
OAuth2Config *oauth2.Config
OAuth2Token *oauth2.Token // initial token (with refresh_token)
// EnableAutoMkdir, when true, causes PUT operations to automatically
// create missing parent directories via WebDAV MKCOL before retrying.
EnableAutoMkdir bool
}
HTTPSBackendOptions groups the parameters needed to construct an HTTPS backend.
type HTTPSFileSysInfo ¶
type HTTPSFileSysInfo struct {
ETag string
}
HTTPSFileSysInfo carries optional metadata (e.g. ETag) from an upstream HTTPS/WebDAV server. Returned by httpsFileInfo.Sys() when populated.
type HTTPSTokenMode ¶
type HTTPSTokenMode int
const ( // HTTPSTokenNone — no token is sent to the backend. HTTPSTokenNone HTTPSTokenMode = iota // HTTPSTokenStatic — a static bearer token read from a file is sent. HTTPSTokenStatic // HTTPSTokenPassthrough — the client-supplied token is forwarded. HTTPSTokenPassthrough // HTTPSTokenOAuth2 — an OAuth2 access token is acquired and refreshed automatically. HTTPSTokenOAuth2 )
type Mapfile ¶
type Mapfile struct {
Rules []MapfileRule `json:"-"`
// contains filtered or unexported fields
}
Mapfile represents the complete mapfile configuration
func (*Mapfile) ApplyRule ¶
func (m *Mapfile) ApplyRule(rule MapfileRule, userClaim, groupClaims []string, requestPath string) string
ApplyRule applies a mapfile rule to determine the mapped username Returns the mapped username if the rule matches, empty string if no match
type MapfileRule ¶
type MapfileRule struct {
Sub *string `json:"sub"`
Username *string `json:"username"`
Path *string `json:"path"`
Group *string `json:"group"`
Result string `json:"result"`
Ignore *bool `json:"ignore"`
Comment *string `json:"comment"`
}
MapfileRule represents a single rule in the mapfile for username mapping
type MetadataPublishRow ¶
type MetadataPublishRow struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"column:event_id;uniqueIndex;not null"`
Namespace string `gorm:"column:namespace;not null;index"`
ObjectPath string `gorm:"column:object_path;not null"`
ObjectSize int64 `gorm:"column:object_size;not null"`
ETag string `gorm:"column:etag;not null"`
ObjectCreated time.Time `gorm:"column:object_created;not null"`
CustomFields string `gorm:"column:custom_fields;not null;default:'{}'"`
MetadataContentType string `gorm:"column:metadata_content_type;not null;default:''"`
MetadataBody []byte `gorm:"column:metadata_body;not null;default:''"`
CreatedAt time.Time `gorm:"column:created_at;not null;index"`
NextAttemptAt time.Time `gorm:"column:next_attempt_at;not null;index"`
Attempts int `gorm:"column:attempts;not null;default:0"`
LastError string `gorm:"column:last_error;not null;default:''"`
// State is "pending" (the worker keeps retrying) or "rejected" (terminal:
// the catalog answered 422 / permanently-bad, so the worker stops touching
// it; the row lingers for status queries and admin/operator deletion).
State string `gorm:"column:state;not null;default:'pending'"`
// QueryToken / ManageToken are per-row random capability tokens embedded in
// the status / manage URLs handed back to the uploading client in eventual
// mode. Possession of the (unguessable) URL is the authority — no separate
// bearer token is required. Empty for transactional-mode rows (which never
// outlive the request).
QueryToken string `gorm:"column:query_token;not null;default:''"`
ManageToken string `gorm:"column:manage_token;not null;default:''"`
// EventType is the wire `type` (object.committed / object.updated /
// object.deleted). Persisted so a retry rebuilds the event with its
// original type, and so the worker knows to skip the "object still exists?"
// check for deletes.
EventType string `gorm:"column:event_type;not null;default:'object.committed'"`
}
MetadataPublishRow is the GORM-mapped persistence representation of one queued event. The struct tag `gorm:"unique"` on event_id matches the migration's UNIQUE constraint.
MetadataContentType and MetadataBody are populated only when the upload was multipart/form-data (see splitMultipartParts in metadata_multipart.go); both columns are empty for header-only and no-metadata uploads, and the worker chooses plain-JSON vs multipart/related outbound based on whether the blob is non-empty.
func (MetadataPublishRow) TableName ¶
func (MetadataPublishRow) TableName() string
TableName overrides the default GORM pluralization rule.
type NotFoundError ¶
NotFoundError is a specialized error for missing resources
func NewNotFoundError ¶
func NewNotFoundError(resourceType, resourcePath string) *NotFoundError
NewNotFoundError creates a new not found error
func (*NotFoundError) Error ¶
func (nfe *NotFoundError) Error() string
Error implements the error interface
func (*NotFoundError) HTTPStatus ¶
func (nfe *NotFoundError) HTTPStatus() int
HTTPStatus returns the appropriate HTTP status code
type ObjectChecksumRow ¶
type ObjectChecksumRow struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
ObjectID int64 `gorm:"column:object_id;not null"`
Algorithm string `gorm:"column:algorithm;not null"`
Value string `gorm:"column:value;not null"`
ComputedAt time.Time `gorm:"column:computed_at;not null"`
EtagAtCompute string `gorm:"column:etag_at_compute;not null"`
}
ObjectChecksumRow is the GORM-mapped projection of object_checksums.
func (ObjectChecksumRow) TableName ¶
func (ObjectChecksumRow) TableName() string
type ObjectCommitEvent ¶
type ObjectCommitEvent struct {
// ID is a server-side UUIDv4. Stable across retries; used for
// receiver-side dedup.
ID string
// Type is the event-type discriminator. Today: ObjectCommitEventType.
Type string
// Timestamp is when this event was first generated.
Timestamp time.Time
// Namespace is the federation prefix of the export this object
// belongs to.
Namespace string
// Federation is the federation discovery URL the origin belongs to.
// It makes the event self-describing: a receiver uses it to locate
// the registry (federation discovery -> registry endpoint) and fetch
// this namespace's registered public keys to verify the webhook JWT,
// without needing to reach the origin directly. Stamped by the
// publisher at send time.
Federation string
// ObjectPath is the federation-relative path of the object.
ObjectPath string
// ObjectSize is the size in bytes as reported by the storage
// backend after commit.
ObjectSize int64
// ETag is the entity tag computed by the storage backend (eg the
// WebDAV-generated ETag). The origin does not introduce its own
// hashing scheme.
ETag string
// ObjectCreated is the storage-reported commit / mtime.
ObjectCreated time.Time
// CustomFields are uploader-supplied fields (parsed from the
// X-Pelican-Object-Metadata header). Reserved keys are stripped
// at parse time.
CustomFields CustomFields
// MetadataContentType + MetadataBody carry the opaque-blob
// metadata supplied via the multipart/form-data upload path
// (see metadata_multipart.go). Empty content type AND empty body
// means "no blob, post plain JSON outbound." A non-empty body
// switches the outbound webhook to multipart/related so the
// receiver gets the bytes byte-for-byte. They are independent
// of CustomFields: an upload may carry both.
MetadataContentType string
MetadataBody []byte
}
ObjectCommitEvent is the in-memory representation of a publishable object-commit event. Its JSON form (per MarshalJSON below) is the webhook wire shape.
func EventFromRow ¶
func EventFromRow(r *MetadataPublishRow) (*ObjectCommitEvent, error)
EventFromRow reconstructs an in-memory event from a persisted row. Used by both publish paths when handing rows to the HTTP attempt.
func NewObjectCommitEvent ¶
func NewObjectCommitEvent(namespace, objectPath string, size int64, etag string, created time.Time, custom CustomFields) *ObjectCommitEvent
NewObjectCommitEvent constructs a fresh event with a server-generated UUID and Type/Timestamp pre-populated.
func NewObjectDeletedEvent ¶
func NewObjectDeletedEvent(namespace, objectPath string) *ObjectCommitEvent
NewObjectDeletedEvent builds an object.deleted event. The object is gone, so size/etag are zero/empty; created_at carries the deletion time so the wire field is never a zero timestamp. The catalog keys off `type` + `path`.
func NewObjectUpdatedEvent ¶
func NewObjectUpdatedEvent(namespace, objectPath string, size int64, etag string, modified time.Time, custom CustomFields) *ObjectCommitEvent
NewObjectUpdatedEvent builds an object.updated event (overwrite or out-of-band modification). `modified` is the object's current mtime.
func (*ObjectCommitEvent) HasMetadataBlob ¶
func (e *ObjectCommitEvent) HasMetadataBlob() bool
HasMetadataBlob reports whether the event carries an opaque blob, driving the outbound publish path between plain-JSON and multipart/related.
func (*ObjectCommitEvent) MarshalJSON ¶
func (e *ObjectCommitEvent) MarshalJSON() ([]byte, error)
MarshalJSON renders the event in the wire shape:
{
"id": "...", "type": "object.committed", "timestamp": "...",
"namespace": "/foo",
"object": { "path": ..., "size": ..., "etag": ..., "created_at": ..., ...inlined customs }
// when a blob is attached:
"metadata": { "content_type": "application/xml", "size": 12345 }
}
The blob *bytes themselves* are not in the JSON — they ride as the second part of a multipart/related body when the outbound mode is switched on (see metadata_publisher.go). The JSON "metadata" sub-object is a *descriptor* the receiver can read before pulling the blob part.
func (*ObjectCommitEvent) WithMetadataBlob ¶
func (e *ObjectCommitEvent) WithMetadataBlob(contentType string, body []byte) *ObjectCommitEvent
WithMetadataBlob attaches an opaque metadata blob to the event in place and returns the same pointer (for fluent construction). Empty contentType+body is a no-op so the blob path stays opt-in.
type ObjectMetadataEvent ¶
type ObjectMetadataEvent string
ObjectMetadataEvent is the high-level discriminator written to object_metadata_history.event_type.
const ( ObjectEventCommit ObjectMetadataEvent = "commit" ObjectEventDelete ObjectMetadataEvent = "delete" ObjectEventRename ObjectMetadataEvent = "rename" ObjectEventExternalObserve ObjectMetadataEvent = "external_observe" ObjectEventExternalModify ObjectMetadataEvent = "external_modify" ObjectEventExternalDelete ObjectMetadataEvent = "external_delete" )
type ObjectMetadataEventInput ¶
type ObjectMetadataEventInput struct {
Namespace string
ObjectPath string // federation-rooted, eg "/exp/data/run99.dat"
Size int64
ETag string
EtagSource EtagSource
BackendMtime time.Time
Actor string // token sub at request time; "" if unknown
Extra map[string]any // X-Pelican-Object-Metadata; ignored if TrackExtra=false for the ns
TrackExtra bool // resolved at call time from per-export overrides
// SourceEtag is the ETag returned by a remote source during a
// third-party copy that produced this commit. Empty for direct
// PUTs; RecordCommit persists NULL in that case, which clears
// any stale source_etag from a prior TPC.
SourceEtag string
}
ObjectMetadataEventInput is the call-site struct for every Record* method. Centralizing the field-set keeps the DAO API stable across the four event types we record.
type ObjectMetadataHistoryRow ¶
type ObjectMetadataHistoryRow struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"column:event_id;uniqueIndex;not null"`
Namespace string `gorm:"column:namespace;not null"`
ObjectPath string `gorm:"column:object_path;not null"`
EventType string `gorm:"column:event_type;not null"`
EventTS time.Time `gorm:"column:event_ts;not null"`
Size *int64 `gorm:"column:size"`
ETag *string `gorm:"column:etag"`
EtagSource *string `gorm:"column:etag_source"`
BackendMtime *time.Time `gorm:"column:backend_mtime"`
ChecksumsJSON string `gorm:"column:checksums_json;not null;default:'{}'"`
Actor string `gorm:"column:actor;not null;default:''"`
Extra string `gorm:"column:extra;not null;default:'{}'"`
// SourceEtag is a snapshot of the live row's source_etag at the
// moment of the event. Populated on the 'source_etag' event only;
// other event types leave it NULL.
SourceEtag *string `gorm:"column:source_etag"`
}
ObjectMetadataHistoryRow is the GORM-mapped projection of object_metadata_history.
func (ObjectMetadataHistoryRow) TableName ¶
func (ObjectMetadataHistoryRow) TableName() string
type ObjectMetadataRow ¶
type ObjectMetadataRow struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
Namespace string `gorm:"column:namespace;not null;index"`
ObjectPath string `gorm:"column:object_path;not null"`
Size int64 `gorm:"column:size;not null"`
ETag string `gorm:"column:etag;not null"`
EtagSource string `gorm:"column:etag_source;not null;default:'backend'"`
BackendMtime time.Time `gorm:"column:backend_mtime;not null"`
CreatedAt time.Time `gorm:"column:created_at;not null"`
LastModified time.Time `gorm:"column:last_modified;not null"`
LastAccessed *time.Time `gorm:"column:last_accessed"`
Actor string `gorm:"column:actor;not null;default:''"`
DeletedAt *time.Time `gorm:"column:deleted_at"`
// SourceEtag is the ETag returned by the remote source during a
// third-party copy (TPC) that produced this row. NULL for objects
// uploaded directly. Cleared by any subsequent write that
// invalidates the object (RecordCommit, RecordExternalChange)
// because the stored ETag would no longer describe the source.
SourceEtag *string `gorm:"column:source_etag"`
// PublishedAt / PublishedEtag record whether this origin has
// successfully published a commit webhook for the live object, and at
// what content version. NULL PublishedAt (or PublishedEtag != ETag after
// an overwrite) marks the row as owing a publish; the reconcile sweep
// uses this to recover objects whose publish was lost to a crash between
// the storage rename and the durable queue write.
PublishedAt *time.Time `gorm:"column:published_at"`
PublishedEtag *string `gorm:"column:published_etag"`
}
ObjectMetadataRow is the GORM-mapped projection of object_metadata. The schema (including the partial unique index that distinguishes live from soft-deleted rows) is defined by the Goose migration, not by GORM's auto-migrate.
func (ObjectMetadataRow) TableName ¶
func (ObjectMetadataRow) TableName() string
type OperationError ¶
type OperationError struct {
Operation string
Path string
HTTPStatus int
Message string
Recoverable bool
// contains filtered or unexported fields
}
OperationError represents a file operation error with recovery hints
func NewOperationError ¶
func NewOperationError(operation, path string, err error) *OperationError
NewOperationError creates a new operation error
func (*OperationError) Error ¶
func (oe *OperationError) Error() string
Error implements the error interface
func (*OperationError) String ¶
func (oe *OperationError) String() string
String returns a detailed error description
type PermissionDeniedError ¶
PermissionDeniedError is a specialized authorization error
func NewPermissionDeniedError ¶
func NewPermissionDeniedError(resource, action, reason string) *PermissionDeniedError
NewPermissionDeniedError creates a new permission denied error
func (*PermissionDeniedError) Error ¶
func (pde *PermissionDeniedError) Error() string
Error implements the error interface
func (*PermissionDeniedError) HTTPStatus ¶
func (pde *PermissionDeniedError) HTTPStatus() int
HTTPStatus returns the appropriate HTTP status code
type PoscMetricsHooks ¶
type PoscMetricsHooks struct {
IncActive func()
DecActive func()
IncExpire func()
// IncRollbackFailed is called when the close hook fails AND the
// best-effort RemoveAll of the just-renamed final path also
// fails. Invoked at most once per close. Receives no namespace
// label here — the metrics package translates it through the
// closure that built the POSC layer.
IncRollbackFailed func()
}
PoscMetricsHooks is a small set of optional hooks the POSC layer calls for observability. Wired up by the metrics package; the layer functions fine if these are nil.
type PrunerHooks ¶
type PrunerHooks struct {
// IncDeleted is called once per loop iteration that deleted at
// least one row. `namespace` is the export's federation prefix;
// `count` is the number of rows deleted in that iteration.
IncDeleted func(namespace string, count int64)
// ObservePassDuration is called once per full pass (all
// namespaces), with the total wall-clock spent.
ObservePassDuration func(d time.Duration)
}
PrunerHooks lets the metrics package observe pruner activity.
type PublishMode ¶
type PublishMode string
PublishMode is the per-event behavior knob.
const ( ModeTransactional PublishMode = "transactional" ModeEventual PublishMode = "eventual" )
func (PublishMode) IsValid ¶
func (m PublishMode) IsValid() bool
IsValid reports whether m is a recognized mode.
type ResourceExhaustedError ¶
ResourceExhaustedError represents resource limit violations
func NewResourceExhaustedError ¶
func NewResourceExhaustedError(resourceType, message string) *ResourceExhaustedError
NewResourceExhaustedError creates a new resource exhausted error
func (*ResourceExhaustedError) Error ¶
func (ree *ResourceExhaustedError) Error() string
Error implements the error interface
func (*ResourceExhaustedError) HTTPStatus ¶
func (ree *ResourceExhaustedError) HTTPStatus() int
HTTPStatus returns the appropriate HTTP status code
type SizeHintFs ¶
type SizeHintFs interface {
OpenFileSized(name string, flag int, perm os.FileMode, size int64) (afero.File, error)
}
SizeHintFs is optionally implemented by an afero.Fs whose write path can do better when the object's final size is known up front.
afero.Fs has no context and no size argument, so a backend that would benefit -- pstore picks its storage tier from the size, and buffers to discover it otherwise -- cannot see the Content-Length the handler already has. This carries it across that gap.
A size of -1 means unknown, which is what a chunked-encoding PUT gives.
type TokenValidationError ¶
type TokenValidationError struct {
Reason string
Issuer string
Subject string
Code int
Details string
Verified bool // Whether the token signature was verified before extracting metadata
}
TokenValidationError represents token validation failures
func NewTokenValidationError ¶
func NewTokenValidationError(reason string) *TokenValidationError
NewTokenValidationError creates a new token validation error
func (*TokenValidationError) Error ¶
func (tve *TokenValidationError) Error() string
Error implements the error interface
func (*TokenValidationError) String ¶
func (tve *TokenValidationError) String() string
String returns a detailed error description
func (*TokenValidationError) WithDetails ¶
func (tve *TokenValidationError) WithDetails(details string) *TokenValidationError
WithDetails adds additional context details
func (*TokenValidationError) WithIssuer ¶
func (tve *TokenValidationError) WithIssuer(issuer string) *TokenValidationError
WithIssuer adds issuer information
func (*TokenValidationError) WithSubject ¶
func (tve *TokenValidationError) WithSubject(subject string) *TokenValidationError
WithSubject adds subject information
func (*TokenValidationError) WithVerified ¶
func (tve *TokenValidationError) WithVerified(verified bool) *TokenValidationError
WithVerified marks whether the token was cryptographically verified before extracting metadata
type UserMapper ¶
type UserMapper struct {
// contains filtered or unexported fields
}
UserMapper handles username and group extraction from tokens with optional mapfile support
func NewUserMapper ¶
func NewUserMapper(usernameClaim, groupsClaim, mapfilePath, defaultUser, unauthenticatedUser string) *UserMapper
NewUserMapper creates a new user mapper usernameClaim: token claim to use for username (default: "sub") groupsClaim: token claim to use for groups (default: "wlcg.groups") mapfilePath: optional path to mapfile for username mapping defaultUser: fallback username when mapfile is enabled but no rule matches;
set to empty string to reject unmatched tokens.
unauthenticatedUser: username for token-less or invalid-token requests;
set to empty string to reject such requests.
func (*UserMapper) ExtractUserInfo ¶
func (um *UserMapper) ExtractUserInfo(tokenClaims map[string]interface{}, requestPath string) *UserInfo
ExtractUserInfo extracts user and group information from token claims
func (*UserMapper) MapTokenToUser ¶
func (um *UserMapper) MapTokenToUser(tokenStr string) *userInfo
MapTokenToUser parses a JWT token and extracts user/group information with optional mapfile mapping The token is parsed without verification (assumes it was already verified upstream) Returns a userInfo struct suitable for caching with token authorization information
func (*UserMapper) RefreshMapfile ¶
func (um *UserMapper) RefreshMapfile() error
RefreshMapfile reloads the mapfile if it has changed on disk
func (*UserMapper) Shutdown ¶
func (um *UserMapper) Shutdown()
Shutdown stops the periodic refresh goroutine and cleans up resources
func (*UserMapper) StartPeriodicRefresh ¶
func (um *UserMapper) StartPeriodicRefresh(interval time.Duration)
StartPeriodicRefresh starts a goroutine that periodically checks and reloads the mapfile if it has been modified on disk. The refresh interval is controlled by the provided interval parameter.
The goroutine will run until the UserMapper's context is cancelled (typically on server shutdown via Shutdown() method).
type XattrChecksummer ¶
type XattrChecksummer struct{}
XattrChecksummer uses extended attributes to store and retrieve checksums
func (*XattrChecksummer) GetChecksum ¶
func (xc *XattrChecksummer) GetChecksum(root *os.Root, filename string, checksumType ChecksumType) (string, error)
GetChecksum retrieves or computes the checksum for a file Uses the provided os.Root to ensure all file operations stay within the root directory
func (*XattrChecksummer) GetChecksumRFC3230 ¶
func (xc *XattrChecksummer) GetChecksumRFC3230(root *os.Root, filename string, checksumType ChecksumType) (string, error)
GetChecksumRFC3230 retrieves the checksum in RFC 3230 format (algorithm=value) MD5 and SHA1 are base64-encoded, CRC32 is hex-encoded Uses the provided os.Root to ensure all file operations stay within the root directory
func (*XattrChecksummer) GetChecksumsRFC3230 ¶
func (xc *XattrChecksummer) GetChecksumsRFC3230(root *os.Root, filename string, types []ChecksumType) ([]string, error)
GetChecksumsRFC3230 returns a list of RFC 3230 digest strings for requested types. If any requested checksum is missing or stale, computes all requested plus defaults and stores. Uses the provided os.Root to ensure all file operations stay within the root directory
Source Files
¶
- authz.go
- backend.go
- backend_blob.go
- backend_etag.go
- backend_globus.go
- backend_https.go
- backend_pstore.go
- checksum.go
- checksum_xrootd.go
- close_notify_fs.go
- error_handling.go
- filesystem.go
- handlers.go
- metadata_admin.go
- metadata_controller.go
- metadata_event.go
- metadata_metrics.go
- metadata_multipart.go
- metadata_publisher.go
- metadata_queue.go
- metadata_status_api.go
- metrics_file.go
- multiuser_fs.go
- object_metadata.go
- object_metadata_access.go
- object_metadata_admin.go
- object_metadata_header.go
- object_metadata_observation.go
- object_metadata_pruner.go
- posc.go
- preconditions.go
- propfind_pelican_props.go
- rate_limited_fs.go
- share_guard.go
- sqlite_batcher.go
- storage_api.go
- tpc_handler.go
- user_mapfile.go