ingest

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 19, 2026 License: Apache-2.0, MIT Imports: 45 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Unfreeze added in v0.5.5

func Unfreeze(unfrozen map[peer.ID]cid.Cid, dstore datastore.Datastore) error

Types

type AdState added in v0.13.0

type AdState struct {
	Known      bool
	Processed  bool
	Skipped    bool
	SkipReason string
	Resync     bool
	Frozen     bool
}

func (AdState) Indexed added in v0.13.0

func (s AdState) Indexed() bool

Indexed reports whether the advertisement's content should be considered available from this indexer.

An ad is indexed only when it was fully processed while the indexer was not frozen. Ads marked for resync are treated as not indexed because a resync invalidates the previous processing result until the ad is processed again. Ads that were permanently skipped (malformed, decode failure, etc.) are not indexed even though they are marked as processed in the marker layer.

type Ingester

type Ingester struct {
	// contains filtered or unexported fields
}

Ingester is a type that uses dagsync for the ingestion protocol.

## Advertisement Ingestion Constraints

1. If an Ad is processed, all older ads referenced by this ad (towards the start of the ad chain), have also been processes. For example, given some chain A <- B <- C, the indexer will never be in the state that it indexed A & C but not B.

2. An indexer will index an Ad chain, but will not make any guarantees about consistency in the presence of multiple ad chains for a given provider. For example if a provider publishes two ad chains at the same time chain1 and chain2 the indexer will apply whichever chain it learns about first, then apply the other chain.

3. An indexer will not index the same Ad twice. An indexer will be resilient to restarts. If the indexer goes down and comes back up it should not break constraint 1.

func NewIngester

func NewIngester(cfg config.Ingest, h host.Host, idxr indexer.Interface, reg *registry.Registry, ds, dsTmp datastore.Batching) (*Ingester, error)

NewIngester creates a new Ingester that uses a dagsync Subscriber to handle communication with providers.

func (*Ingester) AllSyncStatuses added in v0.13.0

func (ing *Ingester) AllSyncStatuses() map[peer.ID]*syncTracker

AllSyncStatuses returns a snapshot of all tracked sync statuses keyed by publisher. The map is a copy; the trackers themselves are live and safe for concurrent readers via their MarshalJSON method.

func (*Ingester) Announce

func (ing *Ingester) Announce(ctx context.Context, nextCid cid.Cid, pubAddrInfo peer.AddrInfo) error

Announce sends an announce message to directly to dagsync, instead of through pubsub.

func (*Ingester) Close

func (ing *Ingester) Close() error

Close stops the ingester and all its workers.

func (*Ingester) EndProcessing added in v0.13.0

func (ing *Ingester) EndProcessing(pubID peer.ID, err error)

EndProcessing marks the processing phase finished for the given publisher and closes any ongoing download run. The tracker is retained so completed stats remain visible until the next sync.

func (*Ingester) EndScan added in v0.13.0

func (ing *Ingester) EndScan(publisherID peer.ID, err error)

EndScan marks the scan phase finished for the given publisher.

func (*Ingester) GetAdState added in v0.13.0

func (ing *Ingester) GetAdState(ctx context.Context, adCid cid.Cid) (state AdState, err error)

GetAdState returns the known state for an advertisement.

Processed is true when the ad is marked fully processed (/adProcessed/ value 1 or 3). Skipped is true when the ad was permanently skipped (value 3). SkipReason contains the reason for the skip. Resync is true when the ad is marked for resync (value 2). Frozen is true when the ad was processed while the indexer was in frozen mode (/adF/ key present); frozen processing updates provider metadata but does not index entry multihashes. Known is false when the ad has never been seen (datastore.ErrNotFound).

func (*Ingester) GetAdStates added in v0.13.0

func (ing *Ingester) GetAdStates(ctx context.Context, adCids []cid.Cid) ([]AdState, error)

GetAdStates returns the known state for a batch of advertisements. Results are returned in the same order as the input slice. Duplicate CIDs are read once and answered per position. On error, returns nil and an error wrapping the failing CID; no partial results are returned.

func (*Ingester) GetAllIngestRates added in v0.8.0

func (ing *Ingester) GetAllIngestRates() map[string]rate.Rate

func (*Ingester) GetExposableFilestore added in v0.13.0

func (ing *Ingester) GetExposableFilestore() filestore.Interface

GetExposableFilestore returns the filestore that can be exposed to other indexers for reading CAR files.

func (*Ingester) GetIngestRate added in v0.8.0

func (ing *Ingester) GetIngestRate(peerID peer.ID) (rate.Rate, bool)

func (*Ingester) GetLatestSync

func (ing *Ingester) GetLatestSync(publisherID peer.ID) (cid.Cid, error)

GetLatestSync gets the latest CID synced for the peer. If no error is returned, then returned CID is never cid.Undef.

func (*Ingester) MarkAdProcessed added in v0.8.10

func (ing *Ingester) MarkAdProcessed(publisher peer.ID, adCid cid.Cid) error

MarkAdProcessed explicitly marks an advertisement as processed. This is used to avoid ingesting this and previous advertisements.

func (*Ingester) MarkAdSkipped added in v0.13.0

func (ing *Ingester) MarkAdSkipped(publisher peer.ID, adCid cid.Cid, reason string, frozen bool) error

MarkAdSkipped explicitly marks an advertisement as permanently skipped. This is used to record that an ad was processed but could not be indexed due to a permanent error (decode failure, malformed, entry chunk error, etc.).

func (*Ingester) MultihashesFromMirror added in v0.5.14

func (ing *Ingester) MultihashesFromMirror() uint64

func (*Ingester) RecordAdScanned added in v0.13.0

func (ing *Ingester) RecordAdScanned(publisherID, providerID peer.ID, adCid cid.Cid) int

RecordAdScanned records that adCid was scanned during the ad-chain walk for the given publisher. providerID is the ad's main provider (ad.Provider), not an extended provider. It creates and initializes the tracker while holding the map mutex so readers never observe an uninitialized tracker.

func (*Ingester) RunWorkers

func (ing *Ingester) RunWorkers(n int)

func (*Ingester) Skip500EntriesError added in v0.8.19

func (ing *Ingester) Skip500EntriesError(skip bool)

func (*Ingester) Sync

func (ing *Ingester) Sync(ctx context.Context, peerInfo peer.AddrInfo, depth int, resync bool, adCid cid.Cid) (cid.Cid, error)

Sync syncs advertisements, up to the latest advertisement, from a publisher. Sync returns the final CID that was synced by the call to Sync.

Sync works by first fetching each advertisement from the specified peer starting at the most recent and traversing to the advertisement last seen by the indexer, or until the advertisement depth limit is reached. Then the entries in each advertisement are synced and the multihashes in each entry are indexed.

The depth argument specifies the recursion depth limit to use during sync. Its value may less than -1 for no limit, 0 to use the indexer's configured value, or greater than 1 for an explicit limit.

The resync argument specifies whether to stop the traversal at the latest known advertisement that is already synced. If set to true, the traversal will continue until either there are no more advertisements left or the recursion depth limit is reached.

The adCid argument, if not cid.Undef, specifies that the sync should only include advertisements that are more recent than the ad identified by adCid.

The reference to the latest synced advertisement returned by GetLatestSync is only updated if the given depth is zero and resync is set to false.

The Context argument controls the lifetime of the sync. Canceling it cancels the sync and causes the multihash channel to close without any data.

func (*Ingester) SyncStatus added in v0.13.0

func (ing *Ingester) SyncStatus(publisherID peer.ID) *syncTracker

SyncStatus returns the live sync tracker for the given publisher, or nil if none is tracked. It does not create a tracker.

func (*Ingester) SyncStatusFor added in v0.13.0

func (ing *Ingester) SyncStatusFor(publisherID peer.ID) *syncTracker

SyncStatusFor returns the live sync tracker for the given publisher, creating it if it does not yet exist. One tracker per publisher; it reflects a single main provider at a time. Prefer SyncStatus for read-only lookups that should not create a tracker.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址