Repository navigation
feat(provider/keystore): alt datastore wipe - #1233
Conversation
97ce04b to
dd1e876
Compare
callers that need a prefix should wrap the datastore with namespace.Wrap before passing it to WithDatastore.
| @@ -446,6 +525,23 @@ func (s *ResettableKeystore) Close() error { | |||
| if err = s.ds.Sync(context.Background(), sizeKey); err != nil { | |||
| return fmt.Errorf("error syncing size on close: %w", err) | |||
| } | |||
There was a problem hiding this comment.
hm.. late on my end but if persistSize or Sync fails here, the early return will skip closing the factory-owned datastores at lines 530-543, and leak
perhaps wrap the factory close block in a defer at the top of the default branch, so it runs regardless of the early returns
|
There are 3 namespaces in used in the Updated so that the meta datastore isn't its own dedicated datastore, but has to be supplied to the constructor (to avoid a dedicated datastore file for storing 2 values). |
|
Added a recommendation of when to use factory datastore mode in 7e3affa. Shared mode is still useful for datastore which don't defer compaction, and easier to use. |
- prepareAltDs: defensively destroy stale alt directory before create - teardownAltDs: always nil altDs and attempt destroyDs even if Close fails - fix "deffering" typo in godoc - add TestKeystoreFactoryModeCrashRecovery - add TestKeystoreFactoryModeTeardownCloseError
provider/keystore data is ephemeral and fully reconstructed on the next
reprovide cycle, so we can safely recover from a corrupted marker
instead of returning a hard error that would brick the repo. this is
especially relevant for users on inexpensive hardware where bitflips
are more common than previously assumed.
- reject marker values outside {0, 1} instead of blindly using them
(prevents panic in shared mode, bogus suffixes in factory mode)
- on corruption: log error, fall back to namespace 0, persist corrected
marker, and in factory mode destroy both stale namespaces
- fix metaDs field comment (size is persisted in primary ds, not metaDs)
- use %w instead of %s in option error wrapping for proper error chains
- add TestKeystoreCorruptedMarkerRecovery (shared + factory modes)
- add TestKeystoreOptionAdapter (verifies KeystoreOption pass-through)
- log active namespace on init (factory mode) - log new datastore creation during reset - log namespace swap with new active index and size - log old datastore removal after swap
lidel
left a comment
There was a problem hiding this comment.
Lgtm, manually tested in ipfs/kubo#11198 (review) and works as expected.
Pushed small extra safety changes (we can recover from bitflips by just resetting the cycle rather than hard crash and bricked repo), take a look before merging:
- 7f11bf3 hardened factory mode against crash and teardown failures (stale alt cleanup, nil-safe teardown)
- 1bede74 added marker corruption detection and recovery (fall back to namespace 0 instead of bricking the repo)
- 6584c8f nil guard in
teardownAltDs - 7b44139 added lifecycle logging for namespace operations (init, create, swap, removal)
The default kubo provide queue (Provide.DHT.SweepEnabled=true) is the go-libp2p-kad-dht buffered SweepingProvider, which drains its dsqueue via GetN(1024), not per-item Out(). Only the legacy boxo provider uses Out(). - add GetN(1024) restart-drain benchmarks (the real default batch size) on LevelDB and Pebble - README: distinguish the default (sweep, GetN) and legacy (boxo, Out) paths, add the GetN(1024) numbers, and note that the separate provider keystore datastores (libp2p/go-libp2p-kad-dht#1233, ipfs/kubo#11198) fix the same LSM tombstone cost for a different component
Context
The
ResettableKeystorehas a datastore where it stores keys that should be advertised to the DHT swarm. Its reset process takes a chan of the keys that should replace the existing ones, write them to a new datastore, and once it has read all the keys atomically swaps with the datastore previously in use. After the swap, the old datastore is cleared, all keys are individually removed.Some datastore implementations (such as leveldb or pebble) don't reclaim the space straight away and simply add tombstones so that removed keys are collected during compaction at a later point. This leads to high disk usage since the swap process increases disk footprint by the keystore size at each reset operation, until compaction happens (usually on process shutdown).
This problem is tracked in ipfs/kubo#11096
Kubo companion PR ipfs/kubo#11198
Proposed fix
Instead of having a single datastore and use
namespace.Wrap()to create the temporary datastore used for swap, we create and destroy datastores whenever needed. This allows reclaiming disk space as soon as the reset operation is over by removing the corresponding directory.The main change is that the
ResettableKeystorecan be supplied with a datastore factory (createanddestroyfunctions) instead of the datastore.The old behavior remains possible, but it is now possible to use datastore factory for optimized disk footprint.