Documentation
¶
Overview ¶
Package storage provides a processor that concurrently creates input items in a storage backend without changing the processor output.
Index ¶
Examples ¶
Constants ¶
const Name = "storage"
Name identifies the storage processor.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Func ¶
type Func[In any] func(c *config[In])
Func configures a storage processor.
func WithIDFieldName ¶
WithIDFieldName sets the exported struct field read for an item's storage ID. String, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, and uint64 fields are supported after pointer and interface dereferencing; fmt.Stringer is not used implicitly.
func WithIDFunc ¶
WithIDFunc sets the first and exclusive source of storage IDs. The function is called concurrently and must be safe for concurrent use.
func WithProcessorOptions ¶
WithProcessorOptions applies options to the underlying processor.
func WithTarget ¶
WithTarget sets the storage target used by Create. The default is "etl"; New rejects a blank target.
type IDFunc ¶
IDFunc derives a storage ID from an item. Returning an error or a blank ID aborts that write.
type Storage ¶
type Storage[In any] struct { // IProcessor provides the processor lifecycle and execution API. processor.IProcessor[In] `json:"processor" validate:"required"` }
Storage creates each input item in a storage backend and, on success, returns the input slice unchanged.
Example (IdFieldName) ¶
Demonstrates selecting the item field used as the storage key.
store := &stubStorage{}
processor, err := New[exampleItem](
store,
1,
"item-",
WithIDFieldName[exampleItem]("Key"),
)
if err != nil {
panic(err)
}
_, err = processor.Run(
context.Background(),
[]exampleItem{{Key: "7", Name: "notebook"}},
)
if err != nil {
panic(err)
}
fmt.Println("selected key:", store.createdIDs()[0] == "item-7")
Output: selected key: true
func Must ¶
func Must[In any]( s storage.IStorage, concurrency int, idPrefix string, opts ...Func[In], ) *Storage[In]
Must creates a storage processor or panics if validation fails.
func New ¶
func New[In any]( s storage.IStorage, concurrency int, idPrefix string, opts ...Func[In], ) (*Storage[In], error)
New creates a storage processor. For each item it resolves the ID from WithIDFunc, then the configured or default ID/Id field, then a SHA-256 key derived from the item's JSON. It prepends idPrefix before calling Create.
The field path is stable for an unchanged field. The content path is stable only when JSON serialization is deterministic; a custom MarshalJSON method can vary it. A custom IDFunc determines its own stability. Whether repeated writes are idempotent depends on the storage backend's Create implementation.
NOTE: Concurrency values below 1 are treated as 1. NOTE: An idPrefix can namespace keys, for example "news-".
Example ¶
Demonstrates storing a slice while passing its items through unchanged.
store := &stubStorage{}
processor, err := New[exampleItem](store, 1, "item-")
if err != nil {
panic(err)
}
items := []exampleItem{{Name: "pencil"}, {Name: "notebook"}}
out, err := processor.Run(context.Background(), items)
if err != nil {
panic(err)
}
stored := store.createdValues()
fmt.Println("items:", out)
fmt.Println("written:", store.created.Load())
fmt.Println(
"values stored:",
len(stored) == len(items) &&
stored[0] == items[0] &&
stored[1] == items[1],
)
Output: items: [{ pencil} { notebook}] written: 2 values stored: true