pipeline

package
v4.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 1, 2026 License: MIT Imports: 17 Imported by: 0

README

Pipeline

Overview

The pipeline package is a core component of the ETL project, providing a powerful and flexible mechanism for defining and executing data processing pipelines. It is designed to be modular, extensible, and production-ready, with a focus on observability, concurrency, and adherence to best practices.

What's a Pipeline?

A pipeline is a series of stages that define a sequence of data processing operations. Each stage consists of a converter and one or more processors, which transform and enhance the data as it flows through the pipeline. Pipelines are highly configurable and can be tailored to meet specific data processing requirements.

How It Works

The pipeline package revolves around the IPipeline interface, which defines the contract for a pipeline. The Pipeline struct implements this interface and provides the core functionality for creating and running pipelines.

To create a pipeline, you instantiate a new Pipeline using the New factory function, specifying the pipeline name, description, concurrency mode, and a variadic list of stages. Each stage is defined using the stage.IStage interface, which encapsulates a converter and a set of processors.

When the Run method is called on a pipeline, it iterates through the stages, passing the output of each stage as the input to the next. The pipeline supports both synchronous and concurrent execution modes, allowing for efficient processing of large datasets.

Features

  1. Modular and Extensible: The pipeline package is designed with modularity and extensibility in mind. It leverages interfaces and generic types to enable easy integration of custom converters and processors.

  2. Concurrent Execution: Pipelines support concurrent execution of stages, enabling parallel processing of data for improved performance.

  3. Observability: The package provides comprehensive observability features, including metrics, logging, and tracing, to monitor and debug pipeline execution.

  4. Metrics: Pipeline metrics are tracked with expvar variables (registered globally on /debug/vars only when ETLER_METRICS_PUBLISH=true; unpublished per-instance variables otherwise), allowing for easy integration with monitoring systems. Metrics include counters for created, running, failed, and done pipelines, as well as duration and progress tracking.

  5. Logging: The package utilizes the sypl library for structured logging, providing rich context and consistent log levels throughout the codebase.

  6. Tracing: The pipeline package integrates with the customapm package, which utilizes Elastic APM (Application Performance Monitoring) under the hood. This enables distributed tracing of the pipeline execution, allowing developers to gain deep insights into the performance and behavior of their pipelines. The tracing data can be visualized, analyzed, and correlated with other application metrics.

  7. Error Handling: The pipeline package includes robust error handling mechanisms, with detailed error messages and proper propagation of errors throughout the pipeline.

  8. Pause and Resume: Pipelines support pausing and resuming execution, allowing for graceful handling of system events or maintenance tasks.

  9. Progress Tracking: The package provides progress tracking capabilities, including absolute progress and percentage completion, enabling real-time monitoring of pipeline execution.

  10. Flexible Configuration: Pipelines can be configured with various options, such as concurrency mode, on-finished callbacks, and custom loggers, using a functional options pattern.

  11. Thorough Testing: The codebase includes comprehensive unit tests, ensuring the reliability and correctness of the pipeline functionality.

  12. Production-Ready: The pipeline package is production-ready, with a focus on performance, reliability, and scalability.

  13. Well-Documented: The code is thoroughly documented, with clear comments explaining the purpose and functionality of each component. The package also includes usage examples and test cases.

  14. Idiomatic Go: The codebase follows idiomatic Go practices, leveraging the language's features and conventions for clean and efficient code.

  15. Typed Errors: The package utilizes typed errors, providing more context and facilitating error handling and debugging.

Architectural Modularity and Flexibility

The pipeline package is designed with architectural modularity and flexibility in mind. It leverages Go's interfaces and generic types to provide a highly extensible and customizable pipeline framework.

The IPipeline interface defines the contract for a pipeline, allowing for easy integration of custom pipeline implementations. The stage.IStage interface enables the creation of custom stages with specific converters and processors.

The use of generic types for ProcessedData and ConvertedOut allows pipelines to handle various data types, making the package adaptable to different data processing scenarios.

The functional options pattern, used in the New factory function and various configuration methods, provides a clean and flexible way to customize pipeline behavior without modifying the core pipeline struct.

Concurrency

Concurrency is a powerful feature of the pipeline package, enabling efficient parallel processing of data. Pipelines support both synchronous and concurrent execution modes, controlled by the ConcurrentStage flag.

When ConcurrentStage is set to true, the pipeline executes stages concurrently using the concurrentloop package. This allows for parallel processing of data across stages, improving overall performance. In this mode, the same input data is fed to all stages concurrently, and the results are merged in a slice, at the end of the pipeline.

When ConcurrentStage is set to false, the pipeline executes stages synchronously, processing data sequentially through each stage. This mode is useful for scenarios where data dependencies or ordering are important. In this mode, the output of one stage becomes the input of the next stage, and so on.

The package handles concurrency safely, ensuring proper synchronization and avoiding race conditions. Errors that occur during concurrent execution are propagated and handled appropriately.

Applied Best Practices

The pipeline package adheres to best practices and idiomatic Go programming principles:

  • Interface-Driven Design: The package heavily relies on interfaces, such as IPipeline and stage.IStage, to provide abstraction and extensibility. This allows for easy integration of custom implementations and facilitates testing.

  • Functional Options: The package utilizes the functional options pattern for pipeline configuration, providing a clean and flexible way to customize pipeline behavior.

  • Error Handling: The package follows Go's error handling conventions, returning errors from functions and methods when necessary. Errors are propagated and handled appropriately throughout the pipeline.

  • Testing: The package includes comprehensive unit tests, covering various scenarios and edge cases. The tests ensure the correctness and reliability of the pipeline functionality.

  • Naming Conventions: The codebase follows Go's naming conventions, using descriptive and meaningful names for variables, functions, and types.

  • Code Organization: The package is well-organized, with separate files for different components and concerns. This promotes code readability and maintainability.

By applying these best practices, the pipeline package maintains a high level of code quality, reliability, and ease of use.

Documentation

Overview

Package pipeline composes stages into an observable ETL pipeline.

Sequential pipelines feed each stage's full task into the next stage. Concurrent pipelines give every stage the original task. Both modes return one result per stage in declaration order.

A pipeline can pause processors before their next execution. Concurrent Run calls on one Pipeline are serialized; completion callbacks run after that serialization is released and must be concurrency-safe.

Index

Examples

Constants

View Source
const Type = "pipeline"

Type identifies pipeline entities.

Variables

This section is empty.

Functions

This section is empty.

Types

type Func

type Func[ProcessedData, ConvertedOut any] func(p IPipeline[ProcessedData, ConvertedOut]) IPipeline[ProcessedData, ConvertedOut]

Func configures a pipeline and returns the implementation to retain.

func WithOnFinished

func WithOnFinished[ProcessedData, ConvertedOut any](onFinished OnFinished[ProcessedData, ConvertedOut]) Func[ProcessedData, ConvertedOut]

WithOnFinished returns a Func that sets the completion callback.

type IPipeline

type IPipeline[ProcessedData, ConvertedOut any] interface {
	shared.IMeta

	shared.IMetrics

	// GetProgress returns the number of stages completed in the current or
	// most recent run.
	GetProgress() *expvar.Int

	// GetProgressPercent returns the completed-stage percentage.
	GetProgressPercent() *expvar.String

	// SetProgressPercent recalculates the completed-stage percentage.
	SetProgressPercent()

	// GetPaused returns Paused when execution is paused and Runnning otherwise.
	// Pause state is independent from GetStatus, which never returns Paused.
	GetPaused() status.Status

	// SetPause pauses or resumes execution; setting the current state is a
	// no-op.
	SetPause(state bool)

	// GetOnFinished returns the `OnFinished` function.
	GetOnFinished() OnFinished[ProcessedData, ConvertedOut]

	// SetOnFinished sets the `OnFinished` function.
	SetOnFinished(onFinished OnFinished[ProcessedData, ConvertedOut])

	// Run executes the stages and returns one task per stage in stage order.
	// Lifecycle, progress, and duration describe the active or most recently
	// completed serialized run. Completion callbacks may overlap and must be
	// concurrency-safe.
	Run(ctx context.Context, processedData []ProcessedData) ([]task.Task[ProcessedData, ConvertedOut], error)
}

IPipeline defines pipeline lifecycle, metrics, pause control, and execution. Concurrent Run calls on one implementation are serialized.

func New

func New[ProcessedData, ConvertedOut any](
	name string,
	description string,
	concurrentStage bool,
	stages ...stage.IStage[ProcessedData, ConvertedOut],
) (IPipeline[ProcessedData, ConvertedOut], error)

New creates a pipeline with at least one stage. concurrentStage selects whether stages receive the original task concurrently or run as a chain.

Example

Demonstrates a two-stage sequential pipeline.

increment, err := processor.New(
	"example-pipeline-increment",
	"increments every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value + 1
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

double, err := processor.New(
	"example-pipeline-double",
	"doubles every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value * 2
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

first, err := stage.New(
	"example-pipeline-first",
	"increments numbers",
	passthru.Must[int](),
	increment,
)
if err != nil {
	log.Fatalln(err)
}

second, err := stage.New(
	"example-pipeline-second",
	"doubles numbers",
	passthru.Must[int](),
	double,
)
if err != nil {
	log.Fatalln(err)
}

flow, err := New(
	"example-sequential-pipeline",
	"increments, then doubles",
	false,
	first,
	second,
)
if err != nil {
	log.Fatalln(err)
}

tasks, err := flow.Run(context.Background(), []int{1, 2, 3})
if err != nil {
	log.Fatalln(err)
}

fmt.Println(tasks[len(tasks)-1].ConvertedData)
Output:
[4 6 8]
Example (ConcurrentStages)

Demonstrates stages that independently process the same input concurrently.

addTen, err := processor.New(
	"example-concurrent-add-ten",
	"adds ten to every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value + 10
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

double, err := processor.New(
	"example-concurrent-double",
	"doubles every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value * 2
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

addStage, err := stage.New(
	"example-concurrent-add-stage",
	"adds ten",
	passthru.Must[int](),
	addTen,
)
if err != nil {
	log.Fatalln(err)
}

doubleStage, err := stage.New(
	"example-concurrent-double-stage",
	"doubles",
	passthru.Must[int](),
	double,
)
if err != nil {
	log.Fatalln(err)
}

flow, err := New(
	"example-concurrent-pipeline",
	"runs both stages independently",
	true,
	addStage,
	doubleStage,
)
if err != nil {
	log.Fatalln(err)
}

tasks, err := flow.Run(context.Background(), []int{1, 2, 3})
if err != nil {
	log.Fatalln(err)
}

results := make([][]int, len(tasks))
for i, result := range tasks {
	results[i] = slices.Clone(result.ConvertedData)
	slices.Sort(results[i])
}
sort.Slice(results, func(i, j int) bool {
	return slices.Compare(results[i], results[j]) < 0
})

fmt.Println(results)
Output:
[[2 4 6] [11 12 13]]

type OnFinished

type OnFinished[ProcessedData, ConvertedOut any] func(ctx context.Context, p IPipeline[ProcessedData, ConvertedOut], originalTask task.Task[ProcessedData, ConvertedOut], tasksOut []task.Task[ProcessedData, ConvertedOut])

OnFinished runs after a pipeline succeeds. originalTask is built from the input data; tasksOut contains one result per stage in stage order. Callbacks run after serialized execution is released, may overlap with a later run's callback, and must be safe for concurrent use.

type Pipeline

type Pipeline[ProcessedData any, ConvertedOut any] struct {
	// ConcurrentStage determines whether all stages receive the original task
	// concurrently or each stage receives the preceding stage's output.
	ConcurrentStage bool `json:"concurrentStage"`

	// Logger records pipeline lifecycle events.
	Logger sypl.ISypl `json:"-" validate:"required"`

	// Description describes the pipeline.
	Description string `json:"description"`

	// Name identifies the pipeline.
	Name string `json:"name" validate:"required"`

	// Stages contains the stages run by the pipeline, in declaration order.
	Stages []stage.IStage[ProcessedData, ConvertedOut] `json:"stages" validate:"required,gt=0"`

	// Lifecycle and progress metrics.
	CounterCreated *expvar.Int `json:"counterCreated"`
	CounterRunning *expvar.Int `json:"counterRunning"`
	CounterFailed  *expvar.Int `json:"counterFailed"`
	CounterDone    *expvar.Int `json:"counterDone"`

	CreatedAt       time.Time      `json:"createdAt"`
	Duration        *expvar.Int    `json:"duration"`
	Progress        *expvar.Int    `json:"progress"`
	ProgressPercent *expvar.String `json:"progressPercent"`
	Status          *expvar.String `json:"status"`
	// contains filtered or unexported fields
}

Pipeline runs a configured sequence of stages and records lifecycle metrics. Concurrent Run calls are serialized. Shared observability updates in place: status changes before task validation, progress resets after validation, and duration is replaced before a successful completion callback or as an error returns. Completion callbacks run after serialized execution is released and may overlap.

func (*Pipeline[ProcessedData, ConvertedOut]) GetCounterCreated

func (p *Pipeline[ProcessedData, ConvertedOut]) GetCounterCreated() *expvar.Int

GetCounterCreated returns the pipeline's created counter.

func (*Pipeline[ProcessedData, ConvertedOut]) GetCounterDone

func (p *Pipeline[ProcessedData, ConvertedOut]) GetCounterDone() *expvar.Int

GetCounterDone returns the pipeline's done counter.

func (*Pipeline[ProcessedData, ConvertedOut]) GetCounterFailed

func (p *Pipeline[ProcessedData, ConvertedOut]) GetCounterFailed() *expvar.Int

GetCounterFailed returns the pipeline's failed counter.

func (*Pipeline[ProcessedData, ConvertedOut]) GetCounterRunning

func (p *Pipeline[ProcessedData, ConvertedOut]) GetCounterRunning() *expvar.Int

GetCounterRunning returns the pipeline's running counter.

func (*Pipeline[ProcessedData, ConvertedOut]) GetCreatedAt

func (p *Pipeline[ProcessedData, ConvertedOut]) GetCreatedAt() time.Time

GetCreatedAt returns the created at time.

func (*Pipeline[ProcessedData, ConvertedOut]) GetDescription

func (p *Pipeline[ProcessedData, ConvertedOut]) GetDescription() string

GetDescription returns the `Description` of the pipeline.

func (*Pipeline[ProcessedData, ConvertedOut]) GetDuration

func (p *Pipeline[ProcessedData, ConvertedOut]) GetDuration() *expvar.Int

GetDuration returns the pipeline's most recent run duration in milliseconds.

func (*Pipeline[ProcessedData, ConvertedOut]) GetLogger

func (p *Pipeline[ProcessedData, ConvertedOut]) GetLogger() sypl.ISypl

GetLogger returns the `Logger` of the pipeline.

func (*Pipeline[ProcessedData, ConvertedOut]) GetMetrics

func (p *Pipeline[ProcessedData, ConvertedOut]) GetMetrics() map[string]string

GetMetrics returns a snapshot of the pipeline's lifecycle metrics as strings.

func (*Pipeline[ProcessedData, ConvertedOut]) GetName

func (p *Pipeline[ProcessedData, ConvertedOut]) GetName() string

GetName returns the `Name` of the pipeline.

func (*Pipeline[ProcessedData, ConvertedOut]) GetOnFinished

func (p *Pipeline[ProcessedData, ConvertedOut]) GetOnFinished() OnFinished[ProcessedData, ConvertedOut]

GetOnFinished returns the `OnFinished` function.

func (*Pipeline[ProcessedData, ConvertedOut]) GetPaused

func (p *Pipeline[ProcessedData, ConvertedOut]) GetPaused() status.Status

GetPaused returns this pipeline controller's pause state. It is the only source of pause state; lifecycle status is reported separately by GetStatus.

func (*Pipeline[ProcessedData, ConvertedOut]) GetProgress

func (p *Pipeline[ProcessedData, ConvertedOut]) GetProgress() *expvar.Int

GetProgress returns the number of stages completed in the current or most recent run.

func (*Pipeline[ProcessedData, ConvertedOut]) GetProgressPercent

func (p *Pipeline[ProcessedData, ConvertedOut]) GetProgressPercent() *expvar.String

GetProgressPercent returns the completed-stage percentage.

func (*Pipeline[ProcessedData, ConvertedOut]) GetStatus

func (p *Pipeline[ProcessedData, ConvertedOut]) GetStatus() *expvar.String

GetStatus returns the run's lifecycle status. Pause state is independent and is reported by GetPaused; GetStatus never returns paused.

func (*Pipeline[ProcessedData, ConvertedOut]) GetType

func (p *Pipeline[ProcessedData, ConvertedOut]) GetType() string

GetType returns the entity type.

func (*Pipeline[ProcessedData, ConvertedOut]) Run

func (p *Pipeline[ProcessedData, ConvertedOut]) Run(ctx context.Context, processingData []ProcessedData) ([]task.Task[ProcessedData, ConvertedOut], error)

Run executes the pipeline. Sequential mode feeds each stage's full task into the next; concurrent mode gives every stage the original task. Both modes return one task per stage in stage order. Concurrent calls on one pipeline are serialized. Status changes before task validation, progress resets after validation, and duration is replaced before a successful completion callback or as an error returns. Completion callbacks run after serialization is released and must be concurrency-safe. Separate instances run independently unless their configured stages share component instances or published metrics.

func (*Pipeline[ProcessedData, ConvertedOut]) SetOnFinished

func (p *Pipeline[ProcessedData, ConvertedOut]) SetOnFinished(onFinished OnFinished[ProcessedData, ConvertedOut])

SetOnFinished sets the `OnFinished` function.

Example

Demonstrates observing every stage result after a pipeline finishes.

increment, err := processor.New(
	"example-callback-increment",
	"increments every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value + 1
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

triple, err := processor.New(
	"example-callback-triple",
	"triples every number",
	func(_ context.Context, in []int) ([]int, error) {
		out := make([]int, len(in))
		for i, value := range in {
			out[i] = value * 3
		}

		return out, nil
	},
)
if err != nil {
	log.Fatalln(err)
}

first, err := stage.New(
	"example-callback-first",
	"increments numbers",
	passthru.Must[int](),
	increment,
)
if err != nil {
	log.Fatalln(err)
}

second, err := stage.New(
	"example-callback-second",
	"triples numbers",
	passthru.Must[int](),
	triple,
)
if err != nil {
	log.Fatalln(err)
}

flow, err := New(
	"example-callback-pipeline",
	"observes stage results",
	false,
	first,
	second,
)
if err != nil {
	log.Fatalln(err)
}

flow.SetOnFinished(func(
	_ context.Context,
	_ IPipeline[int, int],
	_ task.Task[int, int],
	tasks []task.Task[int, int],
) {
	for i, result := range tasks {
		fmt.Printf("stage %d: %v\n", i+1, result.ConvertedData)
	}
})

if _, err := flow.Run(context.Background(), []int{1, 2}); err != nil {
	log.Fatalln(err)
}
Output:
stage 1: [2 3]
stage 2: [6 9]

func (*Pipeline[ProcessedData, ConvertedOut]) SetPause

func (p *Pipeline[ProcessedData, ConvertedOut]) SetPause(state bool)

SetPause changes only this pipeline controller's pause state. It never changes the run lifecycle status reported by GetStatus. Processors pause before their next execution and resume immediately on unpause.

func (*Pipeline[ProcessedData, ConvertedOut]) SetProgressPercent

func (p *Pipeline[ProcessedData, ConvertedOut]) SetProgressPercent()

SetProgressPercent recalculates the completed-stage percentage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址