filegateway

package
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 36 Imported by: 0

README

Xbus file gateway
=================

This xbus client provides actors that act as gateway to/from a filesystem.
The filesystem can be local or remote.

Services
--------

The provided actor services are:

``file-gateway-emitter``
''''''''''''''''''''''''

Observe a pattern and send any matching file to the bus, then delete it.

Settings:

- ``source``: The source url/pattern to watch
- ``source.<option>``: Extra option for the source filesystem
- ``message-type``: The message type to send to the bus

Example:

.. code-block:: yaml

   service:
     type: file-gateway-emitter
     settings:
       source: scp://root:password@somehost/factures/*.pdf
       source.host-pubkey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL+BET9m3jCj5yhI1cRrHJF8HDG5ZMnCl/vI07sY+Oyr somehost
       source.polling-interval: 1m
       message-type: printed-invoice

``file-gateway-consumer``
'''''''''''''''''''''''''

Write the received files to a target directory.

Settings:

- ``target``: The target directory uri
- ``target.<option>``: Extra option for the target filesystem
- ``temporary-errors`` - if 'true', any filesystem error inside a consumer is
  returned to xbus as temporary (SERVERTOOBUSY). Xbus will retry sending the
  file until it is a success.

Example:

.. code-block:: yaml

   service:
     type: file-gateway-consumer
     settings:
       target: file:///exports/
       target.prefix: /var/xbus-data/


Filesystems
-----------

A filesystem url is a standard url, in which the query params are ignored.
It can contain user/password, a host, a port number.

Common options
''''''''''''''

- ``polling-interval`` - for polling-based implementations, the interval
  between polls. Supported units are "h", "m", and "s".
  The default is "10s".


``file://`` - local filesystem
''''''''''''''''''''''''''''''

The url host must be empty.

Options:

- ``prefix``: The base path in which to work

``scp://`` - ssh/scp remote filesystem
''''''''''''''''''''''''''''''''''''''

The url host must be set.

Password and private key authentication is supported.

Options:

- ``basedir``: The base path in which to work. Default: "."
- ``private-key``: The private-key to use for authentication, as a PEM encoded string.
- ``host-pubkey``: The remote host public key, in the authorized key format. Mandatory.

``sftp://`` - sftp remote filesystem
''''''''''''''''''''''''''''''''''''

The url host must be set.

Password and private key authentication is supported.

Options:

- ``basedir``: The base path in which to work. Default: "."
- ``private-key``: The private-key to use for authentication, as a PEM encoded string.
- ``host-pubkey``: The remote host public key, in the authorized key format. Mandatory.
- ``maxconn`` (default 2): The maximum number of concurrent connections to the target host.
- ``keep-alive`` (default "1m"): How long to keep a connection opened after last using it.


``ftp://`` - ftp remote filesystem
''''''''''''''''''''''''''''''''''

The url host must be set

A username and a password must be set.

Options:

- ``maxconn`` (default 2): The maximum number of concurrent connections to the
  target host.
- ``flavor`` (default "default"): Specify the kind of ftp server.
  Can be "default" or "vsftpd".


``ftps://`` - ftps remote filesystem
''''''''''''''''''''''''''''''''''''

Same options as ``ftp://``, plus the following:

- ``host-cert`` (required): the pem encoded public key of the remote server.
- ``tls-mode``: can be "explicit" (default) or "implicit". The value must match
  the server tls mode.
- ``tls-servername``: The server name according to the certificate, if different
  from the URL host.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrMaxAttemptReached = errors.New("max attempts reached")
	ErrNoADirectory      = errors.New("not a directory")
)
View Source
var (
	ErrObserveReturnedNil = errors.New("method Observe() returned nil, which should never happen")
)
View Source
var (
	ErrSFtpPoolFull = errors.New("sftp pool is full")
)

Functions

func MustRegisterFilesystem

func MustRegisterFilesystem(scheme string, factory FilesystemFactory)

MustRegisterFilesystem registers a filesystem or panics if it fails

func NewConsumer

func NewConsumer(actor *api.Actor, settings service.Settings, log xbus.Logger) (service.Consumer, error)

NewConsumer creates a Consumer.

func NewEmitter

func NewEmitter(actor *xbus.Actor, settings xbus.Settings) (xbus.ActorService, error)

NewEmitter creates a Emitter

func RegisterFilesystem

func RegisterFilesystem(scheme string, factory FilesystemFactory) error

RegisterFilesystem registers a new filesystem type

Types

type Consumer

type Consumer struct {
	// contains filtered or unexported fields
}

Consumer save the received files to a filesystem.

func (*Consumer) Process

func (c *Consumer) Process(
	ctx context.Context, request *api.ActorProcessRequest, inputs service.Inputs, log xbus.Logger,
) error

Process ...

func (*Consumer) Shutdown added in v1.7.0

func (c *Consumer) Shutdown() error

Shutdown closes the underlying filesystem if it implements io.Closer

type CustomOptions

type CustomOptions map[string]interface{}

CustomOptions ...

type EmitEnvelopeFunc

type EmitEnvelopeFunc func(
	ctx context.Context,
	env envelope.Envelope,
) error

type Emitter

type Emitter struct {
	// contains filtered or unexported fields
}

Emitter watches a pattern, emit the matching files into the bus, and delete the file after emission

func (*Emitter) Shutdown

func (e *Emitter) Shutdown() error

Shutdown stops the observe routine

func (*Emitter) Startup

func (e *Emitter) Startup() error

Startup lauch the observe routine

type FTPFlavor

type FTPFlavor int

FTPFlavor is the type of the remote FTP server

const (
	// FTPFlavorDefault is for modern ftp servers that implements the MLSD command
	FTPFlavorDefault FTPFlavor = iota
	// FTPFlavorVSFTP is for a vsftp server
	FTPFlavorVSFTP
)

type FileObserver

type FileObserver struct {
	// contains filtered or unexported fields
}

FileObserver is a helper for emitting file names to a channel. It takes lists of files, optionnally filter with a pattern, and globally deduplicate the list

func NewFileObserver

func NewFileObserver(ctx context.Context, pattern string, prefix string, c chan<- string) *FileObserver

NewFileObserver creates a FileObserver

func (*FileObserver) AddFiles

func (fo *FileObserver) AddFiles(files ...string) error

AddFiles add a list of files

type Filesystem

type Filesystem interface {
	String() string
	Write(ctx context.Context, path string, size int) (io.WriteCloser, error)
	Observe(ctx context.Context, pattern string, paths chan<- string, logger xbus.Logger) error
	Read(ctx context.Context, path string) (io.ReadCloser, error)
	WaitForDeletion(ctx context.Context, path string) error
	Rename(ctx context.Context, path, newpath string) error
	Delete(ctx context.Context, path string) error
}

Filesystem ...

func GetFilesystem

func GetFilesystem(scheme string, options FilesystemOptions, log zerolog.Logger) (Filesystem, error)

GetFilesystem instanciate a Filesytem for a given scheme and options

type FilesystemFactory

type FilesystemFactory func(FilesystemOptions, zerolog.Logger) (Filesystem, error)

FilesystemFactory is a function that instanciate a Filesystem

type FilesystemOptions

type FilesystemOptions struct {
	Name            string
	Scheme          string
	Host            string
	Port            string
	Username        string
	Password        string
	HasPassword     bool
	Path            string
	PollingInterval time.Duration
	Custom          CustomOptions
}

FilesystemOptions ...

func LoadFilesystemOptions

func LoadFilesystemOptions(settings service.Settings, name string) (FilesystemOptions, error)

LoadFilesystemOptions loads a filesytem options from a actor service settings

func NewFilesystemOptions

func NewFilesystemOptions(sURI string) (FilesystemOptions, error)

NewFilesystemOptions initializes a FilesystemOptions from a uri

func (FilesystemOptions) CustomIntD

func (fo FilesystemOptions) CustomIntD(key string, target *int, defaultValue int) error

CustomIntD reads a int option if present, or defaultValue if not

func (FilesystemOptions) CustomOption

func (fo FilesystemOptions) CustomOption(key string) (interface{}, bool)

CustomOption returns an option, if present

func (FilesystemOptions) CustomStringD

func (fo FilesystemOptions) CustomStringD(key string, target *string, defaultValue string) error

CustomStringD reads a string option, if present, or defaultValue if not

func (FilesystemOptions) GetCustomStringD

func (fo FilesystemOptions) GetCustomStringD(key string, defaultValue string) (value string, err error)

GetCustomStringD returns a string option with a default value

func (FilesystemOptions) String

func (fo FilesystemOptions) String() string

type FtpFilesystem

type FtpFilesystem struct {
	FilesystemOptions
	// contains filtered or unexported fields
}

FtpFilesystem stores files on ftp(s) server

func (*FtpFilesystem) Close added in v1.7.0

func (fs *FtpFilesystem) Close() error

Close releases the FTP client connection

func (*FtpFilesystem) Delete

func (fs *FtpFilesystem) Delete(_ context.Context, filepath string) error

Delete removes a file

func (*FtpFilesystem) Observe

func (fs *FtpFilesystem) Observe(
	ctx context.Context, pattern string, paths chan<- string, log xbus.Logger,
) error

Observe naïvely do a file list every pollingInterval

func (*FtpFilesystem) Read

func (fs *FtpFilesystem) Read(_ context.Context, filepath string) (io.ReadCloser, error)

Read returns an opened File

func (*FtpFilesystem) Rename

func (fs *FtpFilesystem) Rename(_ context.Context, oldpath, newpath string) error

Rename renames a file

func (*FtpFilesystem) WaitForDeletion

func (fs *FtpFilesystem) WaitForDeletion(ctx context.Context, filepath string) error

WaitForDeletion wath the given path every second until it does not exist anymore

func (*FtpFilesystem) Write

func (fs *FtpFilesystem) Write(_ context.Context, filepath string, size int) (io.WriteCloser, error)

Write ...

type GCSFilesystem

type GCSFilesystem struct {
	FilesystemOptions
	// contains filtered or unexported fields
}

GCSFilesystem stores files on a Google Cloud Storage

func (*GCSFilesystem) Close added in v1.7.0

func (fs *GCSFilesystem) Close() error

Close releases the GCS client resources

func (*GCSFilesystem) Delete

func (fs *GCSFilesystem) Delete(ctx context.Context, filepath string) error

Delete implements the Filesystem interface

func (*GCSFilesystem) Observe

func (fs *GCSFilesystem) Observe(
	ctx context.Context, pattern string, paths chan<- string, log xbus.Logger,
) error

Observe implements the Filesystem interface

func (*GCSFilesystem) Read

func (fs *GCSFilesystem) Read(ctx context.Context, filepath string) (io.ReadCloser, error)

Read returns an opened File

func (*GCSFilesystem) Rename

func (fs *GCSFilesystem) Rename(ctx context.Context, oldpath, newpath string) error

Rename renames a file

func (*GCSFilesystem) WaitForDeletion

func (fs *GCSFilesystem) WaitForDeletion(ctx context.Context, filepath string) error

WaitForDeletion implements the Filesystem interface

func (*GCSFilesystem) Write

func (fs *GCSFilesystem) Write(ctx context.Context, filepath string, size int) (io.WriteCloser, error)

type LocalFilesystem

type LocalFilesystem struct {
	// contains filtered or unexported fields
}

LocalFilesystem stores files on the local filesystem

func (*LocalFilesystem) Delete

func (fs *LocalFilesystem) Delete(_ context.Context, filepath string) error

Delete removes a file

func (*LocalFilesystem) Observe

func (fs *LocalFilesystem) Observe(
	ctx context.Context, pattern string, paths chan<- string, log xbus.Logger,
) error

Observe naïvely do a file list every second

func (*LocalFilesystem) Read

func (fs *LocalFilesystem) Read(_ context.Context, filepath string) (io.ReadCloser, error)

Read returns an opened File

func (*LocalFilesystem) Rename

func (fs *LocalFilesystem) Rename(_ context.Context, oldpath, newpath string) error

Rename renames a file

func (*LocalFilesystem) String

func (fs *LocalFilesystem) String() string

func (*LocalFilesystem) WaitForDeletion

func (fs *LocalFilesystem) WaitForDeletion(ctx context.Context, filepath string) error

WaitForDeletion wath the given path every second until it does not exist anymore

func (*LocalFilesystem) Write

func (fs *LocalFilesystem) Write(_ context.Context, filepath string, size int) (io.WriteCloser, error)

Write ...

type OnDiskBuffer

type OnDiskBuffer struct {
	// contains filtered or unexported fields
}

func NewOnDiskBuffer

func NewOnDiskBuffer(bufmaxsize int64, dir, pattern string) *OnDiskBuffer

func (*OnDiskBuffer) Read

func (b *OnDiskBuffer) Read(data []byte) (int, error)

func (*OnDiskBuffer) Reset

func (b *OnDiskBuffer) Reset() error

func (*OnDiskBuffer) Seek

func (b *OnDiskBuffer) Seek(offset int64, whence int) (int64, error)

func (*OnDiskBuffer) Write

func (b *OnDiskBuffer) Write(data []byte) (int, error)

type RenameTargetVars

type RenameTargetVars struct {
	Full       string
	Dir        string
	Base       string
	Ext        string
	WithoutExt string

	EnvelopeID string
}

RenameTargetVars holds the variables available when rendering a rename target path template

type SFtpFilesystem

type SFtpFilesystem struct {
	FilesystemOptions
	// contains filtered or unexported fields
}

SFtpFilesystem stores files on the local filesystem

func (*SFtpFilesystem) Close added in v1.7.0

func (fs *SFtpFilesystem) Close() error

Close releases the sftp connection pool resources

func (*SFtpFilesystem) Delete

func (fs *SFtpFilesystem) Delete(ctx context.Context, filepath string) error

Delete removes a file

func (*SFtpFilesystem) Observe

func (fs *SFtpFilesystem) Observe(
	ctx context.Context, pattern string, paths chan<- string, logger xbus.Logger,
) error

Observe naïvely do a file list every pollingInterval

func (*SFtpFilesystem) Read

func (fs *SFtpFilesystem) Read(ctx context.Context, filepath string) (io.ReadCloser, error)

Read returns an opened File

func (*SFtpFilesystem) Rename

func (fs *SFtpFilesystem) Rename(ctx context.Context, oldpath, newpath string) error

Rename renames a file

func (*SFtpFilesystem) WaitForDeletion

func (fs *SFtpFilesystem) WaitForDeletion(ctx context.Context, filepath string) error

WaitForDeletion wath the given path every second until it does not exist anymore

func (*SFtpFilesystem) Write

func (fs *SFtpFilesystem) Write(ctx context.Context, filepath string, size int) (io.WriteCloser, error)

Write ...

type SFtpPool

type SFtpPool struct {
	// contains filtered or unexported fields
}

SFtpPool is a pool of sftp connections

func NewSFtpPool

func NewSFtpPool(address string, config *ssh.ClientConfig, maxsize int32, keepAlive time.Duration) *SFtpPool

NewSFtpPool creates a sftp connection pool

func (*SFtpPool) Close

func (p *SFtpPool) Close() error

Close closes the idle connections currently sitting in the pool. Connections in use are not closed here; they are cleaned up when released, through their keep-alive timer.

func (*SFtpPool) Exec

func (p *SFtpPool) Exec(ctx context.Context, run func(ctx context.Context, client *sftp.Client) error) error

Exec runs a function on a sftp client. The ctx is propagated to Get (so acquiring a connection is cancelable) and passed to run, so the callback may check ctx.Err() itself.

func (*SFtpPool) Get

func (p *SFtpPool) Get(ctx context.Context) (*SFtpPoolConn, error)

Get returns a sftp connection. It blocks until a connection is available or ctx is canceled.

func (*SFtpPool) Glob

func (p *SFtpPool) Glob(ctx context.Context, pattern string) (matches []string, err error)

Glob runs Glob on a connection

func (*SFtpPool) Stat

func (p *SFtpPool) Stat(ctx context.Context, path string) (info os.FileInfo, err error)

Stat runs Stat on a connection

type SFtpPoolConn added in v1.7.0

type SFtpPoolConn struct {
	// contains filtered or unexported fields
}

SFtpPoolConn is a single connection managed by the pool. It holds both the sftp client and the underlying ssh client: sftp.Client.Close() only closes the sftp session pipe, not the ssh connection. Keeping the ssh client here lets the pool close the real connection when a connection is discarded, avoiding leaked ssh sessions.

func (*SFtpPoolConn) Client added in v1.7.0

func (c *SFtpPoolConn) Client() *sftp.Client

Client returns the underlying sftp client.

func (*SFtpPoolConn) Close added in v1.7.0

func (c *SFtpPoolConn) Close() error

Close releases the connection back to the pool for reuse. If it is not retrieved again before keepAlive elapses, both the sftp session and the underlying ssh connection are closed.

func (*SFtpPoolConn) ForceClose added in v1.7.0

func (c *SFtpPoolConn) ForceClose() error

ForceClose closes the connection immediately and removes it from the pool. Safe to call on an idle connection; do not call on a connection currently in use, as it would close it under the caller. Idempotent: a concurrent keep-alive timeout or a repeated call is a no-op.

type ScpFilesystem

type ScpFilesystem struct {
	FilesystemOptions
	// contains filtered or unexported fields
}

ScpFilesystem stores files on the local filesystem

func (*ScpFilesystem) Delete

func (fs *ScpFilesystem) Delete(_ context.Context, filepath string) error

Delete removes a file

func (*ScpFilesystem) Observe

func (fs *ScpFilesystem) Observe(
	ctx context.Context, pattern string, paths chan<- string, log xbus.Logger,
) error

Observe naïvely do a file list every pollingInterval

func (*ScpFilesystem) Read

func (fs *ScpFilesystem) Read(_ context.Context, filepath string) (io.ReadCloser, error)

Read returns an opened File

func (*ScpFilesystem) Rename

func (fs *ScpFilesystem) Rename(_ context.Context, oldpath, newpath string) error

Rename renames a file

func (*ScpFilesystem) WaitForDeletion

func (fs *ScpFilesystem) WaitForDeletion(ctx context.Context, filepath string) error

WaitForDeletion wath the given path every second until it does not exist anymore

func (*ScpFilesystem) Write

func (fs *ScpFilesystem) Write(_ context.Context, filepath string, size int) (io.WriteCloser, error)

Write ...

type StartupErrorPolicy added in v1.5.0

type StartupErrorPolicy string
const (
	StartupErrorPolicyStopProcess StartupErrorPolicy = "stop-process"
	StartupErrorPolicyReattempt   StartupErrorPolicy = "reattempt"
)

Source Files

  • consumer.go
  • emitter.go
  • filesystem.go
  • ftp.go
  • gcs.go
  • localfs.go
  • on_disk_buffer.go
  • scp.go
  • sftp.go
  • sftp_pool.go
  • utils.go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址