Repository navigation
Description
Bug Report
Current Behavior
A reactive command whose reply is an error can hang forever: the subscriber never receives onError (nor onComplete), and the command timeout does not fire either, because the command itself has already completed.
The error is dropped in RedisPublisher.State.onError:
void onError(RedisSubscription<?> subscription, Throwable t) {
State state;
while ((state = subscription.state()) != COMPLETED && subscription.changeState(state, COMPLETED)) {
...
subscriber.onError(t);
...
}
}If the CAS fails, the loop condition is false and the method returns without signalling anything. The CAS can fail because the subscribing thread moves the state concurrently: DEMAND.request() dispatches the command and then calls onDataAvailable(), whose read() flips DEMAND -> READING and afterRead() flips it back. When the error reply arrives on the event loop between state() and changeState(...) in onError, the event loop's CAS fails and the error is lost. The success path does not have this problem, because onAllDataRead() sets allDataRead and the reading thread picks it up.
The window is small, so it needs an error reply that comes back almost immediately. We hit it with EVALSHA answering NOSCRIPT on a fresh Redis (first script call, cold JVM, machine under heavy CPU load): Mono.from(reactive.evalsha(...)) (in our case via RxJava's Single.fromPublisher) never terminated; the thread dump showed the test parked in blockingGet(), the netty event loop idle, and no pending command. We confirmed the cause by running the same code with a copy of RedisPublisher that pauses between state() and changeState(): the publisher then hangs every time (well past the 60 s command timeout), and terminates with the error once the CAS is retried.
Thread dump (relevant part)
"ForkJoinPool-1-worker-1" WAITING (parking)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at io.reactivex.rxjava3.internal.observers.BlockingMultiObserver.blockingGet(BlockingMultiObserver.java:86)
at io.reactivex.rxjava3.core.Single.blockingGet(Single.java:3645)
at ...RateLimitImplIT.timedConsumptionExecutesTheProductionScriptAndSetsExpiry(RateLimitImplIT.java:62)
"lettuce-nioEventLoop-4-1" RUNNABLE (idle in select)
"lettuce-timer-3-1" TIMED_WAITING (sleeping)
Input Code
Input Code
// First call on a Redis that does not have the script cached: the reply is a NOSCRIPT error.
Mono.from(connection.reactive().evalsha(sha1, ScriptOutputType.MULTI, keys, args))
.block(); // occasionally never returnsA deterministic unit test that forces the interleaving (the subscribing thread moving DEMAND -> READING right before the error's CAS) is in the linked pull request.
Expected behavior/code
The subscriber receives onError with the RedisCommandExecutionException, regardless of what the subscribing thread does to the state concurrently.
Environment
- Lettuce version(s): 7.8.0.RELEASE; the same
State.onErrorcode is in 7.7.0, 6.8.0 and currentmain - Redis version: 8.6.4
Possible Solution
Retry the CAS instead of leaving the loop when it fails:
State state;
while ((state = subscription.state()) != COMPLETED) {
if (!subscription.changeState(state, COMPLETED)) {
continue;
}
readData(subscription);
Subscriber<?> subscriber = subscription.subscriber;
if (subscriber != null) {
subscriber.onError(t);
}
return;
}I have a pull request ready with this change and a unit test that fails without it.
Additional context
Commands that rarely get error replies are only exposed when Redis is already failing (LOADING, READONLY during failover, OOM, MISCONF, WRONGTYPE), so the realistic trigger is EVALSHA + NOSCRIPT, which is the normal first call after a Redis restart, failover or SCRIPT FLUSH.
Activity
Use explicit Mockito imports in RedisPublisherUnitTests redis#3942
RedisPublisher must not drop an error when the state changes concurre…