Documentation
¶
Overview ¶
Package botevent holds the bot-event "doorbell" primitive shared by the producer side (modules/robot enqueue chokepoints) and the consumer side (modules/bot_api long-poll on POST /v1/bot/events).
It is a leaf package on purpose: modules/bot_api already imports modules/robot, so the key format cannot live in either module without creating an import cycle or a second copy that can drift.
What the doorbell is (and is not) ¶
The doorbell is a **hint that something was enqueued**, never the event itself. The authoritative queue remains the `robotEvent:{robotID}` sorted set read by ZRangeByScore from the caller's cursor. A consumer that wakes on the bell still performs that authoritative read; a consumer that never sees a bell simply waits out its hold and returns an empty batch, and the next poll picks the event up from the sorted set.
That asymmetry is the whole safety argument: a lost, stolen, or stale bell can only cost latency, never an event. Nothing in this package may be promoted into a delivery path.
Index ¶
- Constants
- Variables
- func Activate(deadline context.Context, ctx *config.Context, floor, observedMax int64) (bool, uint64, error)
- func AgeModeBeliefForTest()
- func AuthorityReads() int64
- func AuthorityUnreadable() int64
- func BellKey(robotID string) string
- func CounterFoundWithoutAuthority() int64
- func CountersLost() int64
- func ExpectedModeSpellings() map[string]int
- func HighWaterSeqKey(robotID string) string
- func HighWaterWriteFailures() int64
- func LegacySeqSweepPrefix() string
- func MirrorRebuilds() int64
- func MirrorUnauthorized() int64
- func MirrorValue(epoch uint64) string
- func NegativeBeliefTTL() time.Duration
- func NextEventID(ctx *config.Context, robotID string) (int64, error)
- func Notify(cfg *config.Config, robotID string)
- func ParseMirrorEpoch(v string) (uint64, bool)
- func QueueKey(robotID string) string
- func ResetModeBeliefForTest()
- func ResetSeededForTest()
- func Ring(r Ringer, robotID string) error
- func RollbacksDetected() int64
- func SeedFailures() int64
- func SeqKey(robotID string) string
- func SetExpectedModeForTest(raw string) func()
- type Ringer
- type Scripter
- type State
Constants ¶
const ( // SeqKeyPrefix namespaces the per-bot event id counter. // // The `counter:` segment is load-bearing, not decoration. It was `botEventSeq:` // while ModeKey was `botEventSeq:mode`, which made SeqKey("mode") *equal* to // ModeKey: a bot whose id was `mode` would have seeded and INCRed the global // activation flag as its own counter, coupling one bot's data path to the gate // for every bot. Robot ids are UUID-hex in practice, so it was unreachable — and // a namespace where one legal input collides with a global key is not something // to leave standing on that basis. TestCounterKeyCannotCollideWithModeKey pins it. SeqKeyPrefix = "botEventSeq:counter:" // ModeKey is the Redis **mirror** of the authoritative mode in // octo_bot_event_seq_state (see state.go). // // The mirror exists so the mode can be checked inside the same Lua script as the // INCR — atomically, without a DB round trip on the hot path. It is not the // authority: it lives in a Redis running `appendonly no`, so it regresses with an // RDB snapshot, and an absent mirror must therefore mean "consult the DB", never // "fall back to legacy". Falling back would issue GenSeq ids below everything the // counter already handed out — #697 mirrored. // // Its value is `incr:{epoch}`, not a bare `incr`, and it is only ever believed // after the authority has confirmed that generation. mode.go has the rules; the // short version is that a mirror alone cannot activate anything. ModeKey = "botEventSeq:mode" // ModeIncr is the activated state: allocate from the monotonic counter. Also the // prefix of the mirror value, and an accepted ExpectedModeEnv value. ModeIncr = "incr" // ModeLegacy is the pre-activation state: delegate to GenSeq. // // It is an accepted ExpectedModeEnv value — a replica may assert that it expects // *not* to be activated, which fails closed if the authority says otherwise. It is // deliberately not a mirror value: the mirror's only job is to claim activation, and // "absent" already means "consult the authority" (an earlier revision claimed the // operator tool wrote this string so preflight could tell "not yet activated" from // "never configured" — it never did, and preflight reads the DB row for that). ModeLegacy = "legacy" // HighWaterKeyPrefix names the durable high-water row in the `seq` table. // // It reuses that table rather than adding one: the schema (`key`, `min_seq`) is // exactly right, the seed already reads that table for the legacy ceiling, and // it needs no migration. The key namespace is disjoint from GenSeq's // (`seq:robotEventSeq:…`), so neither reads the other's rows. HighWaterKeyPrefix = "botEventHigh:" // ExpectedModeEnv makes a lost or rolled-back mode key fail closed instead of // silently degrading to legacy. // // `ModeKey` lives in Redis, so it regresses under exactly the RDB rollback // described above — and dropping to legacy then issues *lower* ids beneath live // cursors, the same loss mirrored. Once activation is verified, roll this out as // `incr` and an unconfirmable mode refuses the enqueue instead. Same shape as // #627's OCTO_MESSAGE_EXTRA_VERSION_EXPECTED_MODE, including that a **malformed** // value fails closed rather than reading as "unset" — see mode.go. Accepted values // are `incr` and `legacy`; leave it unset until the flip, because a replica // expecting `incr` beforehand fails every enqueue closed. ExpectedModeEnv = "OCTO_BOTEVENT_EXPECTED_MODE" // QueueKeyPrefix is the authoritative per-bot event queue. Defined here so the // producers and the consumer stop each spelling it with their own // fmt.Sprintf — the seed below has to read the very key the producers write. QueueKeyPrefix = "robotEvent:" )
const ( // StateTable is the singleton state row's table. StateTable = "octo_bot_event_seq_state" // StateModeLegacy delegates to GenSeq. Seeded value, so a deploy is inert. StateModeLegacy = cutover.ModeInactive // StateModeIncr allocates from the Redis counter. StateModeIncr = cutover.ModeActive )
const BellKeyPrefix = "robotEventBell:"
BellKeyPrefix namespaces the per-bot doorbell list. Kept distinct from `robotEvent:` so the authoritative queue and the hint can never collide.
const BellTTL = 5 * time.Minute
BellTTL bounds how long an unread doorbell token survives. It only has to outlive one long-poll hold (30s ceiling) — a bell that expires while an event still sits in the queue costs nothing, because consumers read the sorted set before they ever wait. Five minutes is slack, not a requirement.
const MaxCutoverFloor int64 = 1 << 50
MaxCutoverFloor is the highest floor this allocator may be activated with.
Queue membership is a Redis sorted set, whose scores are float64: above 2^53 distinct int64 ids stop having distinct scores, which recreates the pagination skip and multi-member ack #697 exists to remove. 2^50 leaves three orders of magnitude above any plausible id and is still nowhere near the boundary.
It lives here, not in the operator command, because it is a property of how ids are stored rather than of how the flip is invoked: Activate passes it to the shared primitive so any caller — a future admin path, a recovery script, a test helper — is bounded by it, not just the one CLI that remembers to ask.
Variables ¶
var ErrFloorTooHigh = errors.New("botevent: cutover floor exceeds the float64-distinct score ceiling")
ErrFloorTooHigh is returned when the requested floor exceeds MaxCutoverFloor.
var ErrFloorTooLow = errors.New("botevent: cutover floor is below the observed maximum")
ErrFloorTooLow mirrors msgextraseq: a cutover floor at or below what has already been issued would let post-activation ids land under a live consumer cursor.
var ErrStateMissing = errors.New("botevent: allocator state row is missing")
ErrStateMissing means the migration has not run. Treated as legacy by readers so a pre-migration deploy behaves like the old binary, and as a hard error by the operator tool so activation cannot be attempted blind.
Functions ¶
func Activate ¶
func Activate(deadline context.Context, ctx *config.Context, floor, observedMax int64) (bool, uint64, error)
Activate flips the state row from legacy to incr under a row lock, validating the floor first.
The `FOR UPDATE` (inside cutover.Flip) is **not** a drain barrier — see the file header. It only serialises concurrent operators and makes the read-validate-write atomic. observedMax is gathered by the operator command BEFORE the lock — with no drain barrier there is nothing a locked recompute could pin down — and the floor must clear it strictly: a floor at or below the observed maximum puts the first activated ids under cursors clients already hold. Returns flipped=false with a nil error when the row is already activated, so the operator command is idempotent. deadline bounds the flip: a wedged MySQL must not hang the operator command mid-procedure, with bot-event writes paused.
func AgeModeBeliefForTest ¶
func AgeModeBeliefForTest()
AgeModeBeliefForTest ages the cached belief past negativeBeliefTTL and past repairCooldown without sleeping, so a test can prove the caches expire rather than persist. Both timestamps move together: a test that needs one of them stale almost always means "make this belief old".
func AuthorityReads ¶
func AuthorityReads() int64
AuthorityReads reports how many times the authoritative state row has been read.
func AuthorityUnreadable ¶
func AuthorityUnreadable() int64
AuthorityUnreadable reports authority reads that failed.
func BellKey ¶
BellKey returns the doorbell list key for robotID. Callers MUST pass the robot identity resolved from the authenticated request context, never a value taken from a request body — the bell key shares the bot-ownership boundary of the event queue itself.
func CounterFoundWithoutAuthority ¶
func CounterFoundWithoutAuthority() int64
CounterFoundWithoutAuthority reports refusals caused by a regressed authority.
func CountersLost ¶
func CountersLost() int64
CountersLost reports how many allocations found the counter key missing.
func ExpectedModeSpellings ¶ added in v1.15.0
ExpectedModeSpellings is the authoritative set of values ExpectedModeEnv accepts, and the mode each one asserts. Anything else is malformed and fails closed.
Exported so the operator command reports the guard using the same table the allocator enforces it with, rather than a second hand-written copy that could disagree with the running server about whether a value is valid.
func HighWaterSeqKey ¶
HighWaterSeqKey returns the `seq` table key holding robotID's durable high-water mark. The `seq:` prefix matches how GenSeq namespaces its own rows.
func HighWaterWriteFailures ¶
func HighWaterWriteFailures() int64
HighWaterWriteFailures reports durable high-water writes that failed.
func LegacySeqSweepPrefix ¶ added in v1.15.0
func LegacySeqSweepPrefix() string
LegacySeqSweepPrefix is the `seq` table prefix under which the retired GenSeq allocator kept its rows, for the operator command that computes a cutover floor by sweeping them table-wide.
It exists so that command does not have to name common.RobotEventSeqKey itself. TestNoGenSeqForBotEventIDs forbids that key everywhere outside this file, and the ban is only strong while it is unconditional: an exemption for a "reader" has to fall back to matching the allocation call shape, which a two-line `key := …; GenSeq(key)` walks straight past. Handing out the prefix keeps the reader honest and the guard absolute.
func MirrorRebuilds ¶
func MirrorRebuilds() int64
MirrorRebuilds reports how many times the mode mirror was rebuilt from the DB.
func MirrorUnauthorized ¶
func MirrorUnauthorized() int64
MirrorUnauthorized reports mirrors that claimed an unconfirmed activation.
func MirrorValue ¶
MirrorValue is the exact ModeKey value that activates the given generation.
Exported for the operator tool, which must write the same spelling the allocator validates. A bare `incr` would be accepted as a *claim* and then force every replica into an authority read on every allocation until one of them removed the key.
func NegativeBeliefTTL ¶
NegativeBeliefTTL is the longest interval for which an allocator may reuse a pre-activation belief while the mode mirror is absent.
The operator tool uses this bound in its cutover failure instructions: once the authority has been flipped, a failed mirror write is not a successful activation.
func NextEventID ¶
NextEventID allocates the next event id for robotID's queue.
The returned value is used both as the sorted-set score and as the payload's `event_id`. Those two must stay equal: the consumer's cursor is a payload `event_id` while its reads and acks are bounded by score (`modules/bot_api/events.go`).
func Notify ¶
Notify asks for a wake-up hint for robotID and returns immediately.
It performs no I/O on the caller's goroutine and cannot block: the ring is handed to a bounded worker pool, and if that pool is saturated the ring is dropped rather than queued without limit. Producers therefore pay a mutex and a channel send — never a Redis round trip — no matter what state Redis is in.
Call it after a successful ZADD and after the queue's own EXPIRE, so a producer crash cannot leave the authoritative key without a TTL. There is no error to handle: a producer cannot act on a failed hint, and failing an already-accepted enqueue over one would trade a bounded latency cost for a lost event.
func ParseMirrorEpoch ¶
ParseMirrorEpoch reports whether a raw ModeKey value claims activation, and for which generation.
Exported so the operator tool can recognise a mirror that claims activation the authority has not granted, and refuse to flip while one is present.
func ResetModeBeliefForTest ¶
func ResetModeBeliefForTest()
ResetModeBeliefForTest clears the cached authority belief.
func ResetSeededForTest ¶
func ResetSeededForTest()
ResetSeededForTest clears the process-local seed cache. Tests that seed against different fixtures need it; production never does.
func Ring ¶
Ring publishes a wake-up hint for robotID.
It is deliberately **best-effort and non-fatal**: producers call it after a successful ZADD, and a failure here must never fail or roll back an enqueue that already succeeded. The cost of a dropped bell is bounded by the consumer's hold duration; the cost of failing an accepted enqueue is a lost event.
The list is trimmed to a single element after every push. One token is all a waiter needs ("something changed, go re-read the queue"), and the constant length means an unattended bell cannot grow without bound when no consumer is long-polling.
The error is returned rather than swallowed, and every producer now logs it at Warn. That matters more since the ring moved onto its own connection pool: there is a state where the ZADD succeeds through a warm shared-pool connection while the ring pool cannot get one at all, so bells vanish while nothing else looks wrong. A dropped bell is not free — it costs each waiter up to one chunk (5s), because the hold falls back to re-reading the authoritative queue at chunk boundaries rather than waiting out the whole `wait`. A counter still belongs to the card-ingress observability work (G1), which owns the metric namespace; one log line does not.
func RollbacksDetected ¶
func RollbacksDetected() int64
RollbacksDetected reports how many counter regressions were caught and healed.
func SeedFailures ¶
func SeedFailures() int64
SeedFailures reports how many allocations failed at the seed step.
func SeqKey ¶
SeqKey returns the event-id counter key for robotID. As with BellKey, robotID must be the identity resolved from the authenticated context, never a request-body value: the counter shares the queue's bot-ownership boundary.
func SetExpectedModeForTest ¶
func SetExpectedModeForTest(raw string) func()
SetExpectedModeForTest overrides the env-derived expectation and returns a restore function. Takes the raw env spelling so a test can exercise a malformed value.
Types ¶
type Ringer ¶
type Ringer interface {
Eval(script string, keys []string, args ...interface{}) *rd.Cmd
EvalSha(sha1 string, keys []string, args ...interface{}) *rd.Cmd
ScriptExists(hashes ...string) *rd.BoolSliceCmd
ScriptLoad(script string) *rd.StringCmd
}
Ringer is the Redis surface Ring needs: go-redis's own scripting set, which is what *rd.Script.Run requires. octo-lib's *redis.Conn exposes none of it, so producers pass the instrumented raw client from RingClient.
Declared here rather than using go-redis's unexported `scripter` so a test can substitute a fake, and so the dependency this package places on its callers is written down.
func RingClient ¶
RingClient returns the shared doorbell client. Built through octoredis so TLS options are honoured and pkg/redis's raw-client chokepoint guard stays satisfied.
Producers should call Notify, not Ring with this client: the client's timeouts bound one ring, while Notify is what keeps the ring off the producer's goroutine entirely.
type Scripter ¶
type Scripter interface {
Get(key string) *rd.StringCmd
Set(key string, value interface{}, expiration time.Duration) *rd.StatusCmd
Incr(key string) *rd.IntCmd
ZRevRangeWithScores(key string, start, stop int64) *rd.ZSliceCmd
Eval(script string, keys []string, args ...interface{}) *rd.Cmd
EvalSha(sha1 string, keys []string, args ...interface{}) *rd.Cmd
ScriptExists(hashes ...string) *rd.BoolSliceCmd
ScriptLoad(script string) *rd.StringCmd
}
Scripter is the go-redis surface the allocator needs: INCR, the sorted-set read used to observe the current ceiling, and the scripting set *rd.Script.Run requires. octo-lib's *redis.Conn exposes none of the scripting calls, so the allocator uses the instrumented raw client like the ring does.
Declared as an interface so tests can substitute a fake — in particular one whose INCR succeeds while its seed fails, since the two have different consequences.
func SeqClient ¶
SeqClient returns the shared allocator client, built through octoredis so TLS options are honoured and pkg/redis's raw-client chokepoint guard stays satisfied.
It is built from the same *config.Config as every other Redis user in the process, which is what keeps the counter in the queue's instance and db. See the co-recovery argument at the top of this file before changing that.
type State ¶
State is the authoritative allocator state.
func ReadState ¶
ReadState reads the singleton row with no deadline. For the operator tool and tests; the allocator uses ReadStateContext so a stalled MySQL cannot hold a msgSem slot open (review P1-1).
func ReadStateContext ¶
ReadStateContext reads the singleton row under the caller's deadline.
A missing row returns ErrStateMissing rather than a zero State, so callers choose what it means: readers treat it as legacy (which is what a pre-migration deploy is), while the operator tool refuses to flip. Callers must distinguish it from every other error — "the table is not there yet" and "the authority is unreachable" have opposite safe answers once the counter era has begun.