commons

package
v0.12.10 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: BSD-3-Clause Imports: 25 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DataRootPathDefault                          string        = "/var/lib/irodsfs_pool"
	PIDFilePathDefault                           string        = "/run/irodsfs-pool/irodsfs-pool.pid"
	SessionTimeoutDefault                        time.Duration = 10 * time.Minute
	SessionTimeoutCheckIntervalDefault           time.Duration = 10 * time.Second
	DataBlockSizeDefault                         int64         = 4 * 1024 * 1024          // 4mb
	MaxDataMemCacheSizeDefault                   int64         = 100 * 1024 * 1024 * 1024 // 100gb
	MaxDataMemCacheBufferItemsDefault            int64         = 512
	DataMemCacheTTLDefault                       time.Duration = 12 * time.Hour
	MaxIOConnectionPerSessionDefault             int           = 30
	StartNewTransactionDefault                   bool          = false
	MaxMetadataCacheEntriesPerSessionDefault     int64         = 1000000
	MaxMetadataCacheSizePerSessionDefault        int64         = 10 * 1024 * 1024 // 10mb
	MaxMetadataCacheBufferItemsPerSessionDefault int64         = 256
	MetadataCacheTTLDefault                      time.Duration = 1 * time.Minute
	StagingRootPathDefault                       string        = "staging"
	MaxStagingDataSizeDefault                    int64         = 500 * 1024 * 1024 * 1024 // 500GB
	MaxCacheFileSizeDefault                      int64         = 1 * 1024 * 1024 * 1024   // 1GB
	StagingDataGracePeriodDefault                time.Duration = 10 * time.Second
	SessionCloseGracePeriodDefault               time.Duration = 30 * time.Second
	OperationTimeoutDefault                      time.Duration = 5 * time.Minute

	// Packed directory defaults. Packing is on: the directories it covers hold
	// tens of thousands of small files that each cost a round trip through the
	// per-file staging path, and none of them is meant to be browsed in iRODS.
	// It does change how they are stored there, into one data object per
	// directory, so the name list is worth reviewing for a given deployment.
	PackedDirectoriesEnabledDefault   bool          = true
	PackedDirectorySuffixDefault      string        = ".packedfs.tar"
	PackedDirectoryCompressionDefault string        = "none"
	MaxPackedDirectorySizeDefault     int64         = 5 * 1024 * 1024 * 1024 // 5GB
	PackedSnapshotIntervalDefault     time.Duration = 30 * time.Minute
	ConcurrentPackLimitDefault        int           = 2

	ManagementServiceEndpointDefault string = "http://0.0.0.0:12021"

	// ClientIDMetadataKey is the gRPC metadata a client sends its own id in.
	// The server makes a connection id of its own, but that one changes
	// whenever the transport reconnects, so it cannot name a client for
	// anything that has to outlive a reconnect - the owner of a file lock,
	// most of all.
	ClientIDMetadataKey string = "x-irodsfs-client-id"

	// RoutingKeyMetadataKey is the gRPC metadata a client sends its routing
	// key in. The key is derived from the iRODS user, so a reverse proxy in
	// front of several pool servers can hash on it to send every call of a
	// user to the same server, where the user's session, staged writes, and
	// file locks live. It is not a credential and must not be trusted as one.
	RoutingKeyMetadataKey string = "x-irodsfs-routing-key"

	LogMaxBackupsDefault int = 10
)

Variables

This section is empty.

Functions

func ErrorToStatus added in v0.6.20

func ErrorToStatus(err error) error

ErrorToStatus converts error to grpc status error

func GetServiceVersion

func GetServiceVersion() string

GetServiceVersion returns service version in string

func GetVersionJSON

func GetVersionJSON() (string, error)

GetVersionJSON returns VersionInfo object in JSON string

func IsFileHandleNotFoundError added in v0.6.20

func IsFileHandleNotFoundError(err error) bool

IsFileHandleNotFoundError evaluates if the given error is file handle not found error

func IsReloginRequiredError added in v0.6.20

func IsReloginRequiredError(err error) bool

IsReloginRequiredError returns true if relogin can solve the issue

func IsSessionNotFoundError added in v0.6.20

func IsSessionNotFoundError(err error) bool

IsSessionNotFoundError evaluates if the given error is session not found error

func NewFileHandleNotFoundError added in v0.6.20

func NewFileHandleNotFoundError(handleID string) error

NewFileHandleNotFoundError creates an error for file handle not found error

func NewSessionNotFoundError added in v0.6.20

func NewSessionNotFoundError(sessionID string) error

NewSessionNotFoundError creates SessionNotFoundError struct

func ParseManagementServiceEndpoint added in v0.12.2

func ParseManagementServiceEndpoint(endpoint string) (string, error)

ParseManagementServiceEndpoint validates an HTTP management endpoint and returns the address suitable for net/http's Server.Addr. An empty endpoint disables the management service.

func ParsePoolServiceEndpoint added in v0.5.6

func ParsePoolServiceEndpoint(endpoint string) (string, string, error)

ParsePoolServiceEndpoint parses endpoint string

func StatusToError added in v0.6.20

func StatusToError(err error) error

StatusToError converts grpc status error to error

Types

type Config

type Config struct {
	ServiceEndpoint string `yaml:"service_endpoint,omitempty" json:"service_endpoint,omitempty"`
	DataRootPath    string `yaml:"data_root_path,omitempty" json:"data_root_path,omitempty"`
	PIDFile         string `yaml:"pid_file,omitempty" json:"pid_file,omitempty"`

	RecoveryEncryptionKey string `yaml:"recovery_encryption_key,omitempty" json:"recovery_encryption_key,omitempty"`

	SessionTimeout                        irodsclient_types.Duration                   `yaml:"session_timeout,omitempty" json:"session_timeout,omitempty"`
	SessionTimeoutCheckInterval           irodsclient_types.Duration                   `yaml:"session_timeout_check_interval,omitempty" json:"session_timeout_check_interval,omitempty"`
	DataBlockSize                         int64                                        `yaml:"data_block_size,omitempty" json:"data_block_size,omitempty"`
	MaxDataMemCacheSize                   int64                                        `yaml:"max_data_mem_cache_size,omitempty" json:"max_data_mem_cache_size,omitempty"`
	MaxDataMemCacheBufferItems            int64                                        `yaml:"max_data_mem_cache_buffer_items,omitempty" json:"max_data_mem_cache_buffer_items,omitempty"`
	DataMemCacheTTL                       irodsclient_types.Duration                   `yaml:"data_mem_cache_ttl,omitempty" json:"data_mem_cache_ttl,omitempty"`
	MaxIOConnectionPerSession             int                                          `yaml:"max_io_connection_per_session,omitempty" json:"max_io_connection_per_session,omitempty"`
	MetadataCacheTimeoutSettings          []irodsclient_fs.MetadataCacheTimeoutSetting `yaml:"metadata_cache_timeout_settings,omitempty" json:"metadata_cache_timeout_settings,omitempty"`
	StartNewTransaction                   bool                                         `yaml:"start_new_transaction,omitempty" json:"start_new_transaction,omitempty"`
	MaxMetadataCacheEntriesPerSession     int64                                        `yaml:"max_metadata_cache_entries_per_session,omitempty" json:"max_metadata_cache_entries_per_session,omitempty"`
	MaxMetadataCacheSizePerSession        int64                                        `yaml:"max_metadata_cache_size_per_session,omitempty" json:"max_metadata_cache_size_per_session,omitempty"`
	MaxMetadataCacheBufferItemsPerSession int64                                        `yaml:"max_metadata_cache_buffer_items_per_session,omitempty" json:"max_metadata_cache_buffer_items_per_session,omitempty"`
	MetadataCacheTTL                      irodsclient_types.Duration                   `yaml:"metadata_cache_ttl,omitempty" json:"metadata_cache_ttl,omitempty"`
	StagingRootPath                       string                                       `yaml:"staging_root_path,omitempty" json:"staging_root_path,omitempty"`
	MaxStagingDataSize                    int64                                        `yaml:"max_staging_data_size,omitempty" json:"max_staging_data_size,omitempty"`
	MaxCacheFileSize                      int64                                        `yaml:"max_cache_file_size,omitempty" json:"max_cache_file_size,omitempty"`
	StagingDataGracePeriod                irodsclient_types.Duration                   `yaml:"staging_data_grace_period,omitempty" json:"staging_data_grace_period,omitempty"`
	SessionCloseGracePeriod               irodsclient_types.Duration                   `yaml:"session_close_grace_period,omitempty" json:"session_close_grace_period,omitempty"`
	OperationTimeout                      irodsclient_types.Duration                   `yaml:"operation_timeout,omitempty" json:"operation_timeout,omitempty"`

	PackedDirectories PackedDirectoriesConfig `yaml:"packed_directories,omitempty" json:"packed_directories,omitempty"`

	ManagementServiceEndpoint string `yaml:"management_service_endpoint,omitempty" json:"management_service_endpoint,omitempty"`

	Debug bool `yaml:"debug,omitempty" json:"debug,omitempty"`

	LogRootPath   string `yaml:"log_root_path,omitempty" json:"log_root_path,omitempty"`
	LogMaxBackups int    `yaml:"log_max_backups,omitempty" json:"log_max_backups,omitempty"`
}

Config holds the parameters list which can be configured

func NewConfigFromFile added in v0.12.0

func NewConfigFromFile(config *Config, filePath string) (*Config, error)

NewConfigFromFile creates Config from file

func NewConfigFromJSON added in v0.8.0

func NewConfigFromJSON(config *Config, jsonBytes []byte) (*Config, error)

NewConfigFromJSON creates Config from JSON

func NewConfigFromJSONFile added in v0.12.0

func NewConfigFromJSONFile(config *Config, jsonPath string) (*Config, error)

NewConfigFromJSONFile creates Config from JSON

func NewConfigFromYAML

func NewConfigFromYAML(config *Config, yamlBytes []byte) (*Config, error)

NewConfigFromYAML creates Config from YAML

func NewConfigFromYAMLFile added in v0.12.0

func NewConfigFromYAMLFile(config *Config, yamlPath string) (*Config, error)

NewConfigFromYAMLFile creates Config from YAML

func NewDefaultConfig

func NewDefaultConfig() *Config

NewDefaultConfig returns a default config

func (*Config) GetDataRootPath added in v0.12.0

func (config *Config) GetDataRootPath() string

func (*Config) GetDataStagingRootPath added in v0.12.0

func (config *Config) GetDataStagingRootPath() string

func (*Config) GetLogFilePath added in v0.6.0

func (config *Config) GetLogFilePath() string

GetLogFilePath returns the service log file path.

func (*Config) GetLogRootPath added in v0.12.0

func (config *Config) GetLogRootPath() string

GetLogRootPath returns the directory containing service and session logs.

func (*Config) GetLogWriter added in v0.12.0

func (config *Config) GetLogWriter(foregroundProcess bool) (io.WriteCloser, error)

func (*Config) GetManagementServiceEndpoint added in v0.12.2

func (config *Config) GetManagementServiceEndpoint() string

GetManagementServiceEndpoint returns the HTTP endpoint for the management service. An empty endpoint disables the service.

func (*Config) GetRecoveryEncryptionKey added in v0.12.0

func (config *Config) GetRecoveryEncryptionKey() ([]byte, error)

func (*Config) GetServiceEndpoint added in v0.6.3

func (config *Config) GetServiceEndpoint() string

func (*Config) MakeLogDir added in v0.6.0

func (config *Config) MakeLogDir() error

MakeLogDir makes a log dir required

func (*Config) MakeWorkDirs added in v0.6.0

func (config *Config) MakeWorkDirs() error

MakeWorkDirs makes dirs required

func (*Config) Validate

func (config *Config) Validate() error

Validate validates configuration

type DataFormat added in v0.12.0

type DataFormat string
const (
	FormatJSON    DataFormat = "JSON"
	FormatYAML    DataFormat = "YAML"
	FormatUnknown DataFormat = "Unknown"
)

func DetectFormat added in v0.12.0

func DetectFormat(data []byte) DataFormat

DetectFormat checks format

type FileHandleNotFoundError added in v0.6.20

type FileHandleNotFoundError struct {
	HandleID string
}

FileHandleNotFoundError contains file handle not found error information

func (*FileHandleNotFoundError) Error added in v0.6.20

func (err *FileHandleNotFoundError) Error() string

Error returns error message

func (*FileHandleNotFoundError) Is added in v0.6.20

func (err *FileHandleNotFoundError) Is(other error) bool

Is tests type of error

func (*FileHandleNotFoundError) ToString added in v0.6.20

func (err *FileHandleNotFoundError) ToString() string

ToString stringifies the object

type MultiWriteCloser added in v0.12.0

type MultiWriteCloser struct {
	// contains filtered or unexported fields
}

MultiWriteCloser writes to multiple writers and closes the ones that implement io.Closer.

func NewMultiWriteCloser added in v0.12.0

func NewMultiWriteCloser(writers ...io.Writer) *MultiWriteCloser

func (*MultiWriteCloser) Close added in v0.12.0

func (mw *MultiWriteCloser) Close() error

func (*MultiWriteCloser) Write added in v0.12.0

func (mw *MultiWriteCloser) Write(p []byte) (n int, err error)

type PackedDirectoriesConfig added in v0.12.3

type PackedDirectoriesConfig struct {
	Enabled bool `yaml:"enabled,omitempty" json:"enabled,omitempty"`

	// Names are directory base names that are packed, matched at any depth.
	Names []string `yaml:"names,omitempty" json:"names,omitempty"`

	// Suffix forms the data object name, so ".venv" becomes ".venv.packedfs.tar".
	// Compression appends its own extension on top of this.
	Suffix string `yaml:"suffix,omitempty" json:"suffix,omitempty"`

	// Compression is one of none, gzip or zstd. The default, none, keeps the
	// archive seekable and costs no CPU; the files in these directories are
	// usually already-compressed data, so a codec saves little.
	Compression string `yaml:"compression,omitempty" json:"compression,omitempty"`

	// MaxPackedDirSize caps the staging disk one packed directory may occupy.
	// A larger directory is refused rather than filling the staging area.
	MaxPackedDirSize int64 `yaml:"max_packed_dir_size,omitempty" json:"max_packed_dir_size,omitempty"`

	// SnapshotInterval bounds how much work a crash can lose, by uploading a
	// dirty directory this often even while the session stays open. Zero uses
	// the default; a negative value uploads at session release only.
	SnapshotInterval irodsclient_types.Duration `yaml:"snapshot_interval,omitempty" json:"snapshot_interval,omitempty"`

	// ConcurrentPackLimit caps how many directories are packed at once.
	ConcurrentPackLimit int `yaml:"concurrent_pack_limit,omitempty" json:"concurrent_pack_limit,omitempty"`
}

PackedDirectoriesConfig controls which directories are stored in iRODS as a single archive data object instead of as a collection.

Directories such as .venv or .git hold tens of thousands of small files, and uploading them one at a time costs a round trip each. A packed directory is held on local staging disk for as long as a session uses it and crosses the wire only as one archive, so the per-file cost disappears. The trade is that the directory is not browsable in iRODS on its own, which suits directories that are not read directly there.

func (*PackedDirectoriesConfig) ToPackedFSConfig added in v0.12.3

func (config *PackedDirectoriesConfig) ToPackedFSConfig() *packedfs.Config

ToPackedFSConfig converts to the form the common library consumes.

type SessionNotFoundError added in v0.6.20

type SessionNotFoundError struct {
	SessionID string
}

SessionNotFoundError contains session not found error information

func (*SessionNotFoundError) Error added in v0.6.20

func (err *SessionNotFoundError) Error() string

Error returns error message

func (*SessionNotFoundError) Is added in v0.6.20

func (err *SessionNotFoundError) Is(other error) bool

Is tests type of error

func (*SessionNotFoundError) ToString added in v0.6.20

func (err *SessionNotFoundError) ToString() string

ToString stringifies the object

type VersionInfo

type VersionInfo struct {
	ServiceVersion string `json:"serviceVersion"`
	GitCommit      string `json:"gitCommit"`
	BuildDate      string `json:"buildDate"`
	GoVersion      string `json:"goVersion"`
	Compiler       string `json:"compiler"`
	Platform       string `json:"platform"`
}

VersionInfo object contains version related info

func GetVersion

func GetVersion() VersionInfo

GetVersion returns VersionInfo object

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址