storage

package
v4.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 1, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package storage provides a processor that concurrently creates input items in a storage backend without changing the processor output.

Index

Examples

Constants

View Source
const Name = "storage"

Name identifies the storage processor.

Variables

This section is empty.

Functions

This section is empty.

Types

type Func

type Func[In any] func(c *config[In])

Func configures a storage processor.

func WithIDFieldName

func WithIDFieldName[In any](idFieldName string) Func[In]

WithIDFieldName sets the exported struct field read for an item's storage ID. String, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, and uint64 fields are supported after pointer and interface dereferencing; fmt.Stringer is not used implicitly.

func WithIDFunc

func WithIDFunc[In any](idFunc IDFunc[In]) Func[In]

WithIDFunc sets the first and exclusive source of storage IDs. The function is called concurrently and must be safe for concurrent use.

func WithProcessorOptions

func WithProcessorOptions[In any](opts ...processor.Func[In]) Func[In]

WithProcessorOptions applies options to the underlying processor.

func WithTarget

func WithTarget[In any](target string) Func[In]

WithTarget sets the storage target used by Create. The default is "etl"; New rejects a blank target.

type IDFunc

type IDFunc[In any] func(ctx context.Context, in In) (string, error)

IDFunc derives a storage ID from an item. Returning an error or a blank ID aborts that write.

type Storage

type Storage[In any] struct {
	// IProcessor provides the processor lifecycle and execution API.
	processor.IProcessor[In] `json:"processor" validate:"required"`
}

Storage creates each input item in a storage backend and, on success, returns the input slice unchanged.

Example (IdFieldName)

Demonstrates selecting the item field used as the storage key.

store := &stubStorage{}
processor, err := New[exampleItem](
	store,
	1,
	"item-",
	WithIDFieldName[exampleItem]("Key"),
)
if err != nil {
	panic(err)
}

_, err = processor.Run(
	context.Background(),
	[]exampleItem{{Key: "7", Name: "notebook"}},
)
if err != nil {
	panic(err)
}

fmt.Println("selected key:", store.createdIDs()[0] == "item-7")
Output:
selected key: true

func Must

func Must[In any](
	s storage.IStorage,
	concurrency int,
	idPrefix string,
	opts ...Func[In],
) *Storage[In]

Must creates a storage processor or panics if validation fails.

func New

func New[In any](
	s storage.IStorage,
	concurrency int,
	idPrefix string,
	opts ...Func[In],
) (*Storage[In], error)

New creates a storage processor. For each item it resolves the ID from WithIDFunc, then the configured or default ID/Id field, then a SHA-256 key derived from the item's JSON. It prepends idPrefix before calling Create.

The field path is stable for an unchanged field. The content path is stable only when JSON serialization is deterministic; a custom MarshalJSON method can vary it. A custom IDFunc determines its own stability. Whether repeated writes are idempotent depends on the storage backend's Create implementation.

NOTE: Concurrency values below 1 are treated as 1. NOTE: An idPrefix can namespace keys, for example "news-".

Example

Demonstrates storing a slice while passing its items through unchanged.

store := &stubStorage{}
processor, err := New[exampleItem](store, 1, "item-")
if err != nil {
	panic(err)
}

items := []exampleItem{{Name: "pencil"}, {Name: "notebook"}}
out, err := processor.Run(context.Background(), items)
if err != nil {
	panic(err)
}

stored := store.createdValues()

fmt.Println("items:", out)
fmt.Println("written:", store.created.Load())
fmt.Println(
	"values stored:",
	len(stored) == len(items) &&
		stored[0] == items[0] &&
		stored[1] == items[1],
)
Output:
items: [{ pencil} { notebook}]
written: 2
values stored: true

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址