provider

package
v0.43.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 40 Imported by: 3

Documentation

Overview

Package provider implements a content provider for the libp2p Kademlia DHT.

The core component is SweepingProvider, which efficiently announces content availability to the DHT network through a region-based approach. Instead of providing each key individually, it divides the keyspace into regions and provides all keys within a region together, reducing network overhead.

Key features:

  • Batched providing: Groups keys by keyspace regions for efficient batch operations
  • Automatic reproviding: Continuously reprovides content at configurable intervals
  • Adaptive scheduling: Balances load across time using keyspace-aware scheduling
  • Network resilience: Handles connectivity issues and peer churn gracefully
  • Performance monitoring: Comprehensive statistics collection and reporting
  • Worker pool management: Configurable parallelism for provide operations

The provider tracks performance metrics including operation counts, timing information, and network statistics over configurable time windows.

Index

Constants

View Source
const (
	// DefaultMaxReprovideDelay is the default maximum delay allowed when
	// reproviding a region. The interval between 2 reprovides of the same region
	// is at most ReprovideInterval+MaxReprovideDelay. This variable is necessary
	// since regions can grow and shrink depending on the network churn.
	DefaultMaxReprovideDelay = 1 * time.Hour

	// DefaultOfflineDelay is the default delay after which a disconnected node
	// is considered as Offline.
	DefaultOfflineDelay = 2 * time.Hour
	// DefaultConnectivityCheckOnlineInterval is the default minimum interval for
	// checking whether the node is still online. Such a check is performed when
	// a network operation fails, and the ConnectivityCheckOnlineInterval limits
	// how often such a check is performed.
	DefaultConnectivityCheckOnlineInterval = 1 * time.Minute

	// DefaultSendProviderRecordTimeout is the default per-peer timeout applied
	// to a single ADD_PROVIDER RPC. Healthy peers complete the round-trip in
	// well under a second; this leaves significant headroom for slow links
	// while still bounding the time a hung peer can pin a provide worker.
	DefaultSendProviderRecordTimeout = 10 * time.Second

	// DefaultLoggerName is the default logger name for the DHT provider.
	DefaultLoggerName = internal.DefaultLoggerName
)

Variables

View Source
var (
	// ErrClosed is returned when the provider is closed.
	ErrClosed = errors.New("provider: closed")
)

Functions

This section is empty.

Types

type KadClosestPeersRouter

type KadClosestPeersRouter interface {
	GetClosestPeers(context.Context, string) ([]peer.ID, error)
}

type Option

type Option func(opt *config) error

func WithAddLocalRecord

func WithAddLocalRecord(f func(context.Context, mh.Multihash) error) Option

WithAddLocalRecord sets the function that adds a provider record to the local provider record store with the provider's lifecycle context.

func WithConnectivityCallbacks added in v0.36.0

func WithConnectivityCallbacks(onOnline, onDisconnected, onOffline func()) Option

WithConnectivityCallbacks sets the connectivity state change callbacks for the provider's internal connectivity checker state machine.

The connectivity checker tracks three states: OFFLINE, DISCONNECTED, and ONLINE. The callbacks are invoked during state transitions:

  • onOnline: Called when the node transitions to ONLINE state (from either OFFLINE or DISCONNECTED). After this callback, the provider measures the network prefix length and refreshes the reprovide schedule.

  • onDisconnected: Called when the node transitions from ONLINE to DISCONNECTED state (i.e., connectivity check failed while online). No provider action is triggered for this transition.

  • onOffline: Called when the node transitions from DISCONNECTED to OFFLINE state (after remaining disconnected for OfflineDelay duration). After this callback, the provider clears the provide queue and invalidates cached metrics.

User-supplied callbacks are invoked before the provider's internal actions. All callbacks should be fast and non-blocking to avoid delaying state transitions and provider operations. Long-running operations should be dispatched to separate goroutines.

func WithConnectivityCheckOnlineInterval

func WithConnectivityCheckOnlineInterval(d time.Duration) Option

WithConnectivityCheckOnlineInterval sets the minimal interval for checking whether the node is still online. Such a check is performed when a network operation fails, and the ConnectivityCheckOnlineInterval limits how often such a check is performed.

func WithDatastore added in v0.36.0

func WithDatastore(ds datastore.Batching) Option

WithDatastore defines the datastore used to keep track of the keyspace region reprovides and persist the provide queue on close.

func WithDedicatedBurstWorkers

func WithDedicatedBurstWorkers(n int) Option

WithDedicatedBurstWorkers sets the number of workers dedicated to burst operations. Burst operations consist in work that isn't scheduled beforehands, such as initial provides and catching up with reproviding after the node went offline for a while.

func WithDedicatedPeriodicWorkers

func WithDedicatedPeriodicWorkers(n int) Option

WithDedicatedPeriodicWorkers sets the number of workers dedicated to periodic region reprovides.

func WithDhtType added in v0.36.0

func WithDhtType(dhtType string) Option

WithDhtType sets the DHT type label used in the prometheus metrics. It should be used when multiple providers are running in the same process to differentiate them.

func WithHost added in v0.36.0

func WithHost(h host.Host) Option

WithHost sets the libp2p host running the provider. It is useful to protect the open connections and keep addresses in the peerstore during provide operations. Additionally, it is used to get the peer.ID if missing.

func WithKeystore

func WithKeystore(ks keystore.Keystore) Option

WithKeystore defines the Keystore used to keep track of the keys that need to be reprovided.

func WithLoggerName added in v0.36.0

func WithLoggerName(name string) Option

WithLoggerName sets the go-log logger name for the DHT provider.

func WithMaxProvideConnsPerWorker

func WithMaxProvideConnsPerWorker(n int) Option

WithMaxProvideConnsPerWorker sets the maximum number of connections to distinct peers that can be opened by a single worker during a provide operation.

func WithMaxReprovideDelay

func WithMaxReprovideDelay(d time.Duration) Option

WithMaxReprovideDelay sets the maximum delay allowed when reproviding a region. The interval between 2 reprovides of the same region is at most ReprovideInterval+MaxReprovideDelay.

This parameter is necessary since regions can grow and shrink depending on the network churn.

func WithMaxWorkers

func WithMaxWorkers(n int) Option

WithMaxWorkers sets the maximum number of workers that can be used for provide and reprovide jobs. The job of a worker is to explore a region of the keyspace and (re)provide the keys matching the region to the closest peers.

You can configure a number of workers dedicated to periodic jobs, and a number of workers dedicated to burst jobs. MaxWorkers should be greater or equal to DedicatedPeriodicWorkers+DedicatedBurstWorkers. The additional workers that aren't dedicated to specific jobs can be used for either job type where needed.

func WithMessageSender

func WithMessageSender(m pb.MessageSender) Option

WithMessageSender sets the message sender used to send messages out to the DHT swarm.

func WithOfflineDelay

func WithOfflineDelay(d time.Duration) Option

WithOfflineDelay sets the delay after which a disconnected node is considered as offline. When a node cannot connect to peers, it is set to `Disconnected`, and after `OfflineDelay` it still cannot connect to peers, its state changes to `Offline`.

func WithPeerID

func WithPeerID(p peer.ID) Option

WithPeerID sets the peer ID of the node running the provider.

func WithReplicationFactor

func WithReplicationFactor(n int) Option

WithReplicationFactor sets the replication factor for provider records. It means that during provide and reprovide operations, each provider records is allocated to the ReplicationFactor closest peers in the DHT swarm.

func WithReprovideInterval

func WithReprovideInterval(d time.Duration) Option

WithReprovideInterval sets the interval at which regions are reprovided.

Set to 0 to disable the periodic reprovide schedule entirely. In that no-schedule mode the SweepingProvider runs in burst-only operation:

  • ProvideOnce and StartProviding continue to publish provider records to the DHT immediately, via the burst worker pool.
  • StartProviding collapses to ProvideOnce semantics: keys are published immediately but NOT persisted in the keystore. Without a schedule, persisting keys would only grow the keystore on disk without any reader to consume them.
  • No schedule is constructed, no periodic reprovide loop runs, and AddToSchedule and RefreshSchedule become no-ops.

Use the no-schedule mode when an external system is responsible for driving reprovides (for example, by calling ProvideOnce on a custom schedule), or when only one-shot ad-hoc announcements are wanted.

Negative durations are rejected.

func WithResumeCycle added in v0.36.0

func WithResumeCycle(resume bool) Option

WithResumeCycle sets whether the reprovider should resume the previous reprovide cycle from where it left off when the node was last shut down.

When set to true (default), the provider will restore its state from the datastore and continue reproviding regions according to their previous schedule. This ensures continuity across node restarts and prevents gaps in provider record availability.

When set to false, the provider starts a fresh reprovide cycle, ignoring any previously persisted state. All regions will be scheduled for reprovision from scratch. This is useful for testing or when you want to reset the reprovide schedule.

func WithRouter

func WithRouter(r KadClosestPeersRouter) Option

WithRouter sets the router used to find closest peers in the DHT.

func WithSelfAddrs

func WithSelfAddrs(f func() []ma.Multiaddr) Option

WithSelfAddrs sets the function that returns the self addresses of the node. These addresses are written in the provider records advertised by the node.

func WithSendProviderRecordTimeout added in v0.40.0

func WithSendProviderRecordTimeout(d time.Duration) Option

WithSendProviderRecordTimeout sets the per-peer timeout applied to a single ADD_PROVIDER RPC. A peer that accepts the libp2p stream but never reads the request must not pin a provide worker goroutine indefinitely; this timeout bounds the wait. Defaults to DefaultSendProviderRecordTimeout.

func WithSkipBootstrapReprovide added in v0.36.0

func WithSkipBootstrapReprovide(skip bool) Option

WithSkipBootstrapReprovide sets whether the initial reprovide should be skipped when the provider starts.

This option should be set to true if the node has a low memory profile or low bandwidth configuration and can afford to wait a full reprovide cycle for all keys to be provided to the DHT. It essentially avoids a spike in resource usage (memory & bandwidth) at startup.

type SweepingProvider

type SweepingProvider struct {
	// contains filtered or unexported fields
}

func New

func New(opts ...Option) (*SweepingProvider, error)

New creates a new SweepingProvider instance with the supplied options.

When WithReprovideInterval is set to 0, the provider operates in no-schedule (burst-only) mode: ProvideOnce and StartProviding still publish records to the DHT, but no schedule is built and no periodic reprovide loop runs. See WithReprovideInterval for details.

func (*SweepingProvider) AddToSchedule

func (s *SweepingProvider) AddToSchedule(keys ...mh.Multihash) error

AddToSchedule makes sure the prefixes associated with the supplied keys are scheduled to be reprovided.

In no-schedule mode (WithReprovideInterval = 0) this is a no-op and returns nil.

Returns an error if the provider is closed or if the node is currently Offline (either never bootstrapped, or disconnected since more than `OfflineDelay`). The schedule depends on the network size, hence recent network connectivity is essential.

func (*SweepingProvider) Clear

func (s *SweepingProvider) Clear() int

Clear clears the all the keys from the provide queue and returns the number of keys that were cleared.

The keys are not deleted from the keystore, so they will continue to be reprovided as scheduled.

func (*SweepingProvider) Close

func (s *SweepingProvider) Close() error

Close stops the provider and releases all resources.

func (*SweepingProvider) ProvideOnce

func (s *SweepingProvider) ProvideOnce(keys ...mh.Multihash) error

ProvideOnce only sends provider records for the given keys out to the DHT swarm. It does NOT take the responsibility to reprovide these keys.

Returns an error if the keys could not be added to the provide queue. This can happen if the provider is closed or if the node is currently Offline (either never bootstrapped, or disconnected since more than `OfflineDelay`). The schedule and provide queue depend on the network size, hence recent network connectivity is essential.

func (*SweepingProvider) RefreshSchedule

func (s *SweepingProvider) RefreshSchedule() error

RefreshSchedule scans the Keystore for any keys that are not currently scheduled for reproviding. If such keys are found, it schedules their associated keyspace region to be reprovided.

This function doesn't remove prefixes that have no keys from the schedule. This is done automatically during the reprovide operation if a region has no keys.

In no-schedule mode (WithReprovideInterval = 0) this is a no-op and returns nil.

Returns an error if the provider is closed or if the node is currently Offline (either never bootstrapped, or disconnected since more than `OfflineDelay`). The schedule depends on the network size, hence recent network connectivity is essential.

func (*SweepingProvider) StartProviding

func (s *SweepingProvider) StartProviding(force bool, keys ...mh.Multihash) error

StartProviding provides the given keys to the DHT swarm unless they were already provided in the past. The keys will be periodically reprovided until StopProviding is called for the same keys or user defined garbage collection deletes the keys.

When the provider is in no-schedule mode (WithReprovideInterval = 0), StartProviding behaves identically to ProvideOnce: keys are published to the DHT immediately but NOT persisted in the keystore.

Returns an error if the keys could not be added to the provide queue. This can happen if the provider is closed or if the node is currently Offline (either never bootstrapped, or disconnected since more than `OfflineDelay`). The schedule and provide queue depend on the network size, hence recent network connectivity is essential.

func (*SweepingProvider) Stats added in v0.36.0

func (s *SweepingProvider) Stats(ctx context.Context) (stats.Stats, error)

Stats returns a snapshot of the provider's current operational metrics.

Pass a ctx with a deadline. The key count is fetched through the keystore worker, which processes operations one at a time, so Stats can stall behind a slow datastore op queued ahead of it. If ctx is canceled before the key count returns, Stats returns stats.Stats{} and ctx.Err().

If the key count lookup fails for a non-ctx reason, the snapshot's Schedule.Keys is -1 and the returned error is nil.

func (*SweepingProvider) StopProviding

func (s *SweepingProvider) StopProviding(keys ...mh.Multihash) error

StopProviding stops reproviding the given keys to the DHT swarm. The node stops being referred as a provider when the provider records in the DHT swarm expire.

Directories

Path Synopsis
Package buffered provides a buffered provider implementation that queues operations and processes them in batches for improved performance.
Package buffered provides a buffered provider implementation that queues operations and processes them in batches for improved performance.
Package dual provides a SweepingProvider for dual DHT setups (LAN and WAN).
Package dual provides a SweepingProvider for dual DHT setups (LAN and WAN).
Package internal defines the core provider interface for DHT content routing.
Package internal defines the core provider interface for DHT content routing.
connectivity
Package connectivity provides thread-safe network connectivity checking with state management and callback notifications for online/offline transitions.
Package connectivity provides thread-safe network connectivity checking with state management and callback notifications for online/offline transitions.
keyspace
Package keyspace provides utilities for Kademlia keyspace operations, including keys, prefixes, and trie operations.
Package keyspace provides utilities for Kademlia keyspace operations, including keys, prefixes, and trie operations.
queue
Package queue provides queues for batching DHT provide operations by Kademlia prefix, with optional datastore persistence.
Package queue provides queues for batching DHT provide operations by Kademlia prefix, with optional datastore persistence.
timeseries
Package timeseries provides time-windowed data structures for collecting and analyzing performance metrics in the libp2p Kademlia DHT provider.
Package timeseries provides time-windowed data structures for collecting and analyzing performance metrics in the libp2p Kademlia DHT provider.
Package keystore provides persistent storage for multihashes used by the DHT provider system.
Package keystore provides persistent storage for multihashes used by the DHT provider system.
Package stats defines the data structures for provider performance metrics and statistics exported by the libp2p kademlia DHT provider.
Package stats defines the data structures for provider performance metrics and statistics exported by the libp2p kademlia DHT provider.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址