Documentation
¶
Index ¶
- Constants
- func ErrorToStatus(err error) error
- func GetServiceVersion() string
- func GetVersionJSON() (string, error)
- func IsFileHandleNotFoundError(err error) bool
- func IsReloginRequiredError(err error) bool
- func IsSessionNotFoundError(err error) bool
- func NewFileHandleNotFoundError(handleID string) error
- func NewSessionNotFoundError(sessionID string) error
- func ParseManagementServiceEndpoint(endpoint string) (string, error)
- func ParsePoolServiceEndpoint(endpoint string) (string, string, error)
- func StatusToError(err error) error
- type Config
- func NewConfigFromFile(config *Config, filePath string) (*Config, error)
- func NewConfigFromJSON(config *Config, jsonBytes []byte) (*Config, error)
- func NewConfigFromJSONFile(config *Config, jsonPath string) (*Config, error)
- func NewConfigFromYAML(config *Config, yamlBytes []byte) (*Config, error)
- func NewConfigFromYAMLFile(config *Config, yamlPath string) (*Config, error)
- func NewDefaultConfig() *Config
- func (config *Config) GetDataRootPath() string
- func (config *Config) GetDataStagingRootPath() string
- func (config *Config) GetLogFilePath() string
- func (config *Config) GetLogRootPath() string
- func (config *Config) GetLogWriter(foregroundProcess bool) (io.WriteCloser, error)
- func (config *Config) GetManagementServiceEndpoint() string
- func (config *Config) GetRecoveryEncryptionKey() ([]byte, error)
- func (config *Config) GetServiceEndpoint() string
- func (config *Config) MakeLogDir() error
- func (config *Config) MakeWorkDirs() error
- func (config *Config) Validate() error
- type DataFormat
- type FileHandleNotFoundError
- type MultiWriteCloser
- type PackedDirectoriesConfig
- type SessionNotFoundError
- type VersionInfo
Constants ¶
const ( DataRootPathDefault string = "/var/lib/irodsfs_pool" PIDFilePathDefault string = "/run/irodsfs-pool/irodsfs-pool.pid" SessionTimeoutDefault time.Duration = 10 * time.Minute SessionTimeoutCheckIntervalDefault time.Duration = 10 * time.Second DataBlockSizeDefault int64 = 4 * 1024 * 1024 // 4mb MaxDataMemCacheSizeDefault int64 = 100 * 1024 * 1024 * 1024 // 100gb MaxDataMemCacheBufferItemsDefault int64 = 512 DataMemCacheTTLDefault time.Duration = 12 * time.Hour MaxIOConnectionPerSessionDefault int = 30 StartNewTransactionDefault bool = false MaxMetadataCacheEntriesPerSessionDefault int64 = 1000000 MaxMetadataCacheSizePerSessionDefault int64 = 10 * 1024 * 1024 // 10mb MaxMetadataCacheBufferItemsPerSessionDefault int64 = 256 MetadataCacheTTLDefault time.Duration = 1 * time.Minute StagingRootPathDefault string = "staging" MaxStagingDataSizeDefault int64 = 500 * 1024 * 1024 * 1024 // 500GB MaxCacheFileSizeDefault int64 = 1 * 1024 * 1024 * 1024 // 1GB StagingDataGracePeriodDefault time.Duration = 10 * time.Second SessionCloseGracePeriodDefault time.Duration = 30 * time.Second OperationTimeoutDefault time.Duration = 5 * time.Minute // Packed directory defaults. Packing is on: the directories it covers hold // tens of thousands of small files that each cost a round trip through the // per-file staging path, and none of them is meant to be browsed in iRODS. // It does change how they are stored there, into one data object per // directory, so the name list is worth reviewing for a given deployment. PackedDirectoriesEnabledDefault bool = true PackedDirectorySuffixDefault string = ".packedfs.tar" PackedDirectoryCompressionDefault string = "none" MaxPackedDirectorySizeDefault int64 = 5 * 1024 * 1024 * 1024 // 5GB PackedSnapshotIntervalDefault time.Duration = 30 * time.Minute ConcurrentPackLimitDefault int = 2 ManagementServiceEndpointDefault string = "http://0.0.0.0:12021" // ClientIDMetadataKey is the gRPC metadata a client sends its own id in. // The server makes a connection id of its own, but that one changes // whenever the transport reconnects, so it cannot name a client for // anything that has to outlive a reconnect - the owner of a file lock, // most of all. ClientIDMetadataKey string = "x-irodsfs-client-id" // RoutingKeyMetadataKey is the gRPC metadata a client sends its routing // key in. The key is derived from the iRODS user, so a reverse proxy in // front of several pool servers can hash on it to send every call of a // user to the same server, where the user's session, staged writes, and // file locks live. It is not a credential and must not be trusted as one. RoutingKeyMetadataKey string = "x-irodsfs-routing-key" LogMaxBackupsDefault int = 10 )
Variables ¶
This section is empty.
Functions ¶
func ErrorToStatus ¶ added in v0.6.20
ErrorToStatus converts error to grpc status error
func GetServiceVersion ¶
func GetServiceVersion() string
GetServiceVersion returns service version in string
func GetVersionJSON ¶
GetVersionJSON returns VersionInfo object in JSON string
func IsFileHandleNotFoundError ¶ added in v0.6.20
IsFileHandleNotFoundError evaluates if the given error is file handle not found error
func IsReloginRequiredError ¶ added in v0.6.20
IsReloginRequiredError returns true if relogin can solve the issue
func IsSessionNotFoundError ¶ added in v0.6.20
IsSessionNotFoundError evaluates if the given error is session not found error
func NewFileHandleNotFoundError ¶ added in v0.6.20
NewFileHandleNotFoundError creates an error for file handle not found error
func NewSessionNotFoundError ¶ added in v0.6.20
NewSessionNotFoundError creates SessionNotFoundError struct
func ParseManagementServiceEndpoint ¶ added in v0.12.2
ParseManagementServiceEndpoint validates an HTTP management endpoint and returns the address suitable for net/http's Server.Addr. An empty endpoint disables the management service.
func ParsePoolServiceEndpoint ¶ added in v0.5.6
ParsePoolServiceEndpoint parses endpoint string
func StatusToError ¶ added in v0.6.20
StatusToError converts grpc status error to error
Types ¶
type Config ¶
type Config struct {
ServiceEndpoint string `yaml:"service_endpoint,omitempty" json:"service_endpoint,omitempty"`
DataRootPath string `yaml:"data_root_path,omitempty" json:"data_root_path,omitempty"`
PIDFile string `yaml:"pid_file,omitempty" json:"pid_file,omitempty"`
RecoveryEncryptionKey string `yaml:"recovery_encryption_key,omitempty" json:"recovery_encryption_key,omitempty"`
SessionTimeout irodsclient_types.Duration `yaml:"session_timeout,omitempty" json:"session_timeout,omitempty"`
SessionTimeoutCheckInterval irodsclient_types.Duration `yaml:"session_timeout_check_interval,omitempty" json:"session_timeout_check_interval,omitempty"`
DataBlockSize int64 `yaml:"data_block_size,omitempty" json:"data_block_size,omitempty"`
MaxDataMemCacheSize int64 `yaml:"max_data_mem_cache_size,omitempty" json:"max_data_mem_cache_size,omitempty"`
MaxDataMemCacheBufferItems int64 `yaml:"max_data_mem_cache_buffer_items,omitempty" json:"max_data_mem_cache_buffer_items,omitempty"`
DataMemCacheTTL irodsclient_types.Duration `yaml:"data_mem_cache_ttl,omitempty" json:"data_mem_cache_ttl,omitempty"`
MaxIOConnectionPerSession int `yaml:"max_io_connection_per_session,omitempty" json:"max_io_connection_per_session,omitempty"`
MetadataCacheTimeoutSettings []irodsclient_fs.MetadataCacheTimeoutSetting `yaml:"metadata_cache_timeout_settings,omitempty" json:"metadata_cache_timeout_settings,omitempty"`
StartNewTransaction bool `yaml:"start_new_transaction,omitempty" json:"start_new_transaction,omitempty"`
MaxMetadataCacheEntriesPerSession int64 `yaml:"max_metadata_cache_entries_per_session,omitempty" json:"max_metadata_cache_entries_per_session,omitempty"`
MaxMetadataCacheSizePerSession int64 `yaml:"max_metadata_cache_size_per_session,omitempty" json:"max_metadata_cache_size_per_session,omitempty"`
MaxMetadataCacheBufferItemsPerSession int64 `yaml:"max_metadata_cache_buffer_items_per_session,omitempty" json:"max_metadata_cache_buffer_items_per_session,omitempty"`
MetadataCacheTTL irodsclient_types.Duration `yaml:"metadata_cache_ttl,omitempty" json:"metadata_cache_ttl,omitempty"`
StagingRootPath string `yaml:"staging_root_path,omitempty" json:"staging_root_path,omitempty"`
MaxStagingDataSize int64 `yaml:"max_staging_data_size,omitempty" json:"max_staging_data_size,omitempty"`
MaxCacheFileSize int64 `yaml:"max_cache_file_size,omitempty" json:"max_cache_file_size,omitempty"`
StagingDataGracePeriod irodsclient_types.Duration `yaml:"staging_data_grace_period,omitempty" json:"staging_data_grace_period,omitempty"`
SessionCloseGracePeriod irodsclient_types.Duration `yaml:"session_close_grace_period,omitempty" json:"session_close_grace_period,omitempty"`
OperationTimeout irodsclient_types.Duration `yaml:"operation_timeout,omitempty" json:"operation_timeout,omitempty"`
PackedDirectories PackedDirectoriesConfig `yaml:"packed_directories,omitempty" json:"packed_directories,omitempty"`
ManagementServiceEndpoint string `yaml:"management_service_endpoint,omitempty" json:"management_service_endpoint,omitempty"`
Debug bool `yaml:"debug,omitempty" json:"debug,omitempty"`
LogRootPath string `yaml:"log_root_path,omitempty" json:"log_root_path,omitempty"`
LogMaxBackups int `yaml:"log_max_backups,omitempty" json:"log_max_backups,omitempty"`
}
Config holds the parameters list which can be configured
func NewConfigFromFile ¶ added in v0.12.0
NewConfigFromFile creates Config from file
func NewConfigFromJSON ¶ added in v0.8.0
NewConfigFromJSON creates Config from JSON
func NewConfigFromJSONFile ¶ added in v0.12.0
NewConfigFromJSONFile creates Config from JSON
func NewConfigFromYAML ¶
NewConfigFromYAML creates Config from YAML
func NewConfigFromYAMLFile ¶ added in v0.12.0
NewConfigFromYAMLFile creates Config from YAML
func (*Config) GetDataRootPath ¶ added in v0.12.0
func (*Config) GetDataStagingRootPath ¶ added in v0.12.0
func (*Config) GetLogFilePath ¶ added in v0.6.0
GetLogFilePath returns the service log file path.
func (*Config) GetLogRootPath ¶ added in v0.12.0
GetLogRootPath returns the directory containing service and session logs.
func (*Config) GetLogWriter ¶ added in v0.12.0
func (config *Config) GetLogWriter(foregroundProcess bool) (io.WriteCloser, error)
func (*Config) GetManagementServiceEndpoint ¶ added in v0.12.2
GetManagementServiceEndpoint returns the HTTP endpoint for the management service. An empty endpoint disables the service.
func (*Config) GetRecoveryEncryptionKey ¶ added in v0.12.0
func (*Config) GetServiceEndpoint ¶ added in v0.6.3
func (*Config) MakeLogDir ¶ added in v0.6.0
MakeLogDir makes a log dir required
func (*Config) MakeWorkDirs ¶ added in v0.6.0
MakeWorkDirs makes dirs required
type DataFormat ¶ added in v0.12.0
type DataFormat string
const ( FormatJSON DataFormat = "JSON" FormatYAML DataFormat = "YAML" FormatUnknown DataFormat = "Unknown" )
func DetectFormat ¶ added in v0.12.0
func DetectFormat(data []byte) DataFormat
DetectFormat checks format
type FileHandleNotFoundError ¶ added in v0.6.20
type FileHandleNotFoundError struct {
HandleID string
}
FileHandleNotFoundError contains file handle not found error information
func (*FileHandleNotFoundError) Error ¶ added in v0.6.20
func (err *FileHandleNotFoundError) Error() string
Error returns error message
func (*FileHandleNotFoundError) Is ¶ added in v0.6.20
func (err *FileHandleNotFoundError) Is(other error) bool
Is tests type of error
func (*FileHandleNotFoundError) ToString ¶ added in v0.6.20
func (err *FileHandleNotFoundError) ToString() string
ToString stringifies the object
type MultiWriteCloser ¶ added in v0.12.0
type MultiWriteCloser struct {
// contains filtered or unexported fields
}
MultiWriteCloser writes to multiple writers and closes the ones that implement io.Closer.
func NewMultiWriteCloser ¶ added in v0.12.0
func NewMultiWriteCloser(writers ...io.Writer) *MultiWriteCloser
func (*MultiWriteCloser) Close ¶ added in v0.12.0
func (mw *MultiWriteCloser) Close() error
type PackedDirectoriesConfig ¶ added in v0.12.3
type PackedDirectoriesConfig struct {
Enabled bool `yaml:"enabled,omitempty" json:"enabled,omitempty"`
// Names are directory base names that are packed, matched at any depth.
Names []string `yaml:"names,omitempty" json:"names,omitempty"`
// Suffix forms the data object name, so ".venv" becomes ".venv.packedfs.tar".
// Compression appends its own extension on top of this.
Suffix string `yaml:"suffix,omitempty" json:"suffix,omitempty"`
// Compression is one of none, gzip or zstd. The default, none, keeps the
// archive seekable and costs no CPU; the files in these directories are
// usually already-compressed data, so a codec saves little.
Compression string `yaml:"compression,omitempty" json:"compression,omitempty"`
// MaxPackedDirSize caps the staging disk one packed directory may occupy.
// A larger directory is refused rather than filling the staging area.
MaxPackedDirSize int64 `yaml:"max_packed_dir_size,omitempty" json:"max_packed_dir_size,omitempty"`
// SnapshotInterval bounds how much work a crash can lose, by uploading a
// dirty directory this often even while the session stays open. Zero uses
// the default; a negative value uploads at session release only.
SnapshotInterval irodsclient_types.Duration `yaml:"snapshot_interval,omitempty" json:"snapshot_interval,omitempty"`
// ConcurrentPackLimit caps how many directories are packed at once.
ConcurrentPackLimit int `yaml:"concurrent_pack_limit,omitempty" json:"concurrent_pack_limit,omitempty"`
}
PackedDirectoriesConfig controls which directories are stored in iRODS as a single archive data object instead of as a collection.
Directories such as .venv or .git hold tens of thousands of small files, and uploading them one at a time costs a round trip each. A packed directory is held on local staging disk for as long as a session uses it and crosses the wire only as one archive, so the per-file cost disappears. The trade is that the directory is not browsable in iRODS on its own, which suits directories that are not read directly there.
func (*PackedDirectoriesConfig) ToPackedFSConfig ¶ added in v0.12.3
func (config *PackedDirectoriesConfig) ToPackedFSConfig() *packedfs.Config
ToPackedFSConfig converts to the form the common library consumes.
type SessionNotFoundError ¶ added in v0.6.20
type SessionNotFoundError struct {
SessionID string
}
SessionNotFoundError contains session not found error information
func (*SessionNotFoundError) Error ¶ added in v0.6.20
func (err *SessionNotFoundError) Error() string
Error returns error message
func (*SessionNotFoundError) Is ¶ added in v0.6.20
func (err *SessionNotFoundError) Is(other error) bool
Is tests type of error
func (*SessionNotFoundError) ToString ¶ added in v0.6.20
func (err *SessionNotFoundError) ToString() string
ToString stringifies the object
type VersionInfo ¶
type VersionInfo struct {
ServiceVersion string `json:"serviceVersion"`
GitCommit string `json:"gitCommit"`
BuildDate string `json:"buildDate"`
GoVersion string `json:"goVersion"`
Compiler string `json:"compiler"`
Platform string `json:"platform"`
}
VersionInfo object contains version related info