converter

package
v4.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 1, 2026 License: MIT Imports: 14 Imported by: 0

README

Conveters

Overview

The conveters package is a core component of the ETL project, providing a versatile and efficient mechanism for defining and executing data conversion tasks within a pipeline. A conveter is responsible for transforming data from one format to another, enabling seamless integration between different stages and ensuring data compatibility throughout the pipeline.

What's a Conveter?

A conveter is a self-contained unit of data conversion within a pipeline. It encapsulates a specific conversion logic that transforms input data of type In into output data of type Out. Conveters are highly reusable and can be easily plugged into different stages of the pipeline.

The conveter package provides a generic interface IConverter and a concrete implementation Converter that can be instantiated with custom conversion functions.

How It Works

The core of the conveters package revolves around the IConverter interface and the Converter struct. The IConverter interface defines the contract for a conveter, specifying methods for running the conversion, managing metrics, and handling callbacks.

To create a conveter, you instantiate a new Converter using the New factory function, specifying the conveter name, description, and the conversion function. The conversion function is defined as Convert[In, Out], where In represents the input data type and Out represents the output data type.

When the Run method is called on a conveter, it executes the following steps:

  1. It sets up observability, including tracing, metrics, status, and logging, to monitor and track the conveter's execution.

  2. It invokes the conversion function, passing the input data and receiving the converted output data. If an error occurs during the conversion, it is handled appropriately.

  3. After the conversion is complete, it updates the observability metrics, such as counters for created, running, failed, and done conveters, as well as the duration and status.

  4. If an OnFinished callback is defined, it is invoked with the original input data and the converted output data, allowing for custom post-conversion processing.

  5. Finally, the conveter returns the converted output data.

Features

  1. Generic and Reusable: Conveters are implemented using Go's generic types, allowing for flexibility in handling various data types. They can be easily reused across different stages and pipelines.

  2. Customizable Conversion Logic: The conversion logic is encapsulated in a custom function that can be defined when creating a conveter. This allows for tailored data transformations specific to the project's requirements.

  3. Observability: The conveters package provides comprehensive observability features, including tracing, metrics, status tracking, and logging. This enables effective monitoring and debugging of the conversion process.

  4. Metrics: Conveter metrics are tracked with expvar variables (registered globally on /debug/vars only when ETLER_METRICS_PUBLISH=true; unpublished per-instance variables otherwise), allowing for easy integration with monitoring systems. Metrics include counters for created, running, failed, and done conveters, as well as duration and status.

  5. Callbacks: The package supports an OnFinished callback function that is invoked after the conversion is complete. This callback receives the original input data and the converted output data, enabling custom post-conversion processing.

  6. Error Handling: The conveters package includes robust error handling mechanisms, propagating errors that occur during the conversion process and providing informative error messages.

  7. Factory Functions: The package provides factory functions New and Default for creating conveters with custom or default configurations. The MustDefault function is also available for creating a conveter that panics on error.

  8. Thorough Testing: The codebase includes comprehensive unit tests to ensure the correctness and reliability of the conveters package. The tests cover various scenarios and validate the conveter's behavior and metrics.

Documentation

Overview

Package converter wraps value conversions with lifecycle metrics, tracing, logging, and completion callbacks.

Stages call converters concurrently for individual items, so converter implementations, conversion functions, and callbacks must be safe for concurrent use.

Index

Examples

Constants

View Source
const Type = "converter"

Type identifies converter entities.

Variables

This section is empty.

Functions

This section is empty.

Types

type Convert

type Convert[In, Out any] func(ctx context.Context, in In) (out Out, err error)

Convert converts `in`, returning the converted data and any error. It must be safe for concurrent use because stages invoke it from multiple goroutines.

type Converter

type Converter[In, Out any] struct {
	// Description describes the converter.
	Description string `json:"description"`

	// Func converts one input and must be safe for concurrent use.
	Func Convert[In, Out] `json:"-" validate:"required"`

	// Logger records converter lifecycle events.
	Logger sypl.ISypl `json:"-" validate:"required"`

	// Name identifies the converter.
	Name string `json:"name" validate:"required"`

	// Lifecycle metrics.
	CounterCreated *expvar.Int `json:"counterCreated"`
	CounterRunning *expvar.Int `json:"counterRunning"`
	CounterFailed  *expvar.Int `json:"counterFailed"`
	CounterDone    *expvar.Int `json:"counterDone"`

	CreatedAt time.Time      `json:"createdAt"`
	Duration  *expvar.Int    `json:"duration"`
	Status    *expvar.String `json:"status"`
	// contains filtered or unexported fields
}

Converter wraps a conversion function with lifecycle observability.

func (*Converter[In, Out]) GetCounterCreated

func (c *Converter[In, Out]) GetCounterCreated() *expvar.Int

GetCounterCreated returns the converter's created counter.

func (*Converter[In, Out]) GetCounterDone

func (c *Converter[In, Out]) GetCounterDone() *expvar.Int

GetCounterDone returns the converter's done counter.

func (*Converter[In, Out]) GetCounterFailed

func (c *Converter[In, Out]) GetCounterFailed() *expvar.Int

GetCounterFailed returns the converter's failed counter.

func (*Converter[In, Out]) GetCounterRunning

func (c *Converter[In, Out]) GetCounterRunning() *expvar.Int

GetCounterRunning returns the converter's running counter.

func (*Converter[In, Out]) GetCreatedAt

func (c *Converter[In, Out]) GetCreatedAt() time.Time

GetCreatedAt returns when the converter was constructed.

func (*Converter[In, Out]) GetDescription

func (c *Converter[In, Out]) GetDescription() string

GetDescription returns the converter's description.

func (*Converter[In, Out]) GetDuration

func (c *Converter[In, Out]) GetDuration() *expvar.Int

GetDuration returns the most recent successful conversion duration in milliseconds. A failed conversion does not replace it.

func (*Converter[In, Out]) GetLogger

func (c *Converter[In, Out]) GetLogger() sypl.ISypl

GetLogger returns the converter's logger.

func (*Converter[In, Out]) GetMetrics

func (c *Converter[In, Out]) GetMetrics() map[string]string

GetMetrics returns a snapshot of the converter's metrics as strings.

func (*Converter[In, Out]) GetName

func (c *Converter[In, Out]) GetName() string

GetName returns the `Name` of the converter.

func (*Converter[In, Out]) GetOnFinished

func (c *Converter[In, Out]) GetOnFinished() OnFinished[In, Out]

GetOnFinished returns the `OnFinished` function.

func (*Converter[In, Out]) GetStatus

func (c *Converter[In, Out]) GetStatus() *expvar.String

GetStatus returns the `Status` metric.

func (*Converter[In, Out]) GetType

func (c *Converter[In, Out]) GetType() string

GetType returns the entity type.

func (*Converter[In, Out]) Run

func (c *Converter[In, Out]) Run(ctx context.Context, in In) (Out, error)

Run converts one value and updates lifecycle observability. It may be called concurrently; Func and OnFinished callbacks must therefore be concurrency safe.

func (*Converter[In, Out]) SetOnFinished

func (c *Converter[In, Out]) SetOnFinished(onFinished OnFinished[In, Out])

SetOnFinished sets the `OnFinished` function.

type Func

type Func[In, Out any] func(p IConverter[In, Out]) IConverter[In, Out]

Func configures a converter and returns the implementation to retain. An option must return a usable implementation; returning one whose embedded interface is nil will panic when it is used, as it would anywhere in Go.

func WithOnFinished

func WithOnFinished[In, Out any](onFinished OnFinished[In, Out]) Func[In, Out]

WithOnFinished returns a Func that sets the completion callback.

type IConverter

type IConverter[In, Out any] interface {
	shared.IMeta

	shared.IMetrics

	// GetOnFinished returns the `OnFinished` function.
	GetOnFinished() OnFinished[In, Out]

	// SetOnFinished sets the `OnFinished` function.
	SetOnFinished(onFinished OnFinished[In, Out])

	// Run converts one input value.
	Run(ctx context.Context, in In) (Out, error)
}

IConverter defines converter lifecycle, metrics, and execution. Implementations must be safe for concurrent use because stages call Run from multiple goroutines.

func Default

func Default[In, Out any](
	fn Convert[In, Out],
	opts ...Func[In, Out],
) (IConverter[In, Out], error)

Default creates a converter whose name and description are Type.

func MustDefault

func MustDefault[In, Out any](
	fn Convert[In, Out],
	opts ...Func[In, Out],
) IConverter[In, Out]

MustDefault creates a default converter or panics if validation fails.

func New

func New[In, Out any](
	name, description string,
	fn Convert[In, Out],
	opts ...Func[In, Out],
) (IConverter[In, Out], error)

New creates a converter. It applies options in order, retains each returned implementation, and then validates the result. Nil options and results are rejected. An option must return a usable implementation; returning one whose embedded interface is nil will panic when it is used, as it would anywhere in Go. Structural validation runs before lifecycle metric getters; the final implementation must expose non-nil lifecycle metrics. fn must be safe for concurrent use.

Example

Demonstrates a converter that changes a value's type.

type person struct {
	Name string
	Age  int
}

format, err := New(
	"example-person-string",
	"formats a person as text",
	func(_ context.Context, in person) (string, error) {
		return fmt.Sprintf("%s (%d)", in.Name, in.Age), nil
	},
)
if err != nil {
	log.Fatalln(err)
}

out, err := format.Run(context.Background(), person{Name: "Ada", Age: 37})
if err != nil {
	log.Fatalln(err)
}

fmt.Println(out)
Output:
Ada (37)

type OnFinished

type OnFinished[In, Out any] func(ctx context.Context, c IConverter[In, Out], originalIn In, convertedOut Out)

OnFinished runs after a conversion succeeds. It may be called concurrently.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址