Skip to content

Reactive command never terminates when RedisPublisher.State.onError loses its CAS to a concurrent state change #3942

Closed

Description

@zenios

Bug Report

Current Behavior

A reactive command whose reply is an error can hang forever: the subscriber never receives onError (nor onComplete), and the command timeout does not fire either, because the command itself has already completed.

The error is dropped in RedisPublisher.State.onError:

void onError(RedisSubscription<?> subscription, Throwable t) {

    State state;
    while ((state = subscription.state()) != COMPLETED && subscription.changeState(state, COMPLETED)) {
        ...
            subscriber.onError(t);
        ...
    }
}

If the CAS fails, the loop condition is false and the method returns without signalling anything. The CAS can fail because the subscribing thread moves the state concurrently: DEMAND.request() dispatches the command and then calls onDataAvailable(), whose read() flips DEMAND -> READING and afterRead() flips it back. When the error reply arrives on the event loop between state() and changeState(...) in onError, the event loop's CAS fails and the error is lost. The success path does not have this problem, because onAllDataRead() sets allDataRead and the reading thread picks it up.

The window is small, so it needs an error reply that comes back almost immediately. We hit it with EVALSHA answering NOSCRIPT on a fresh Redis (first script call, cold JVM, machine under heavy CPU load): Mono.from(reactive.evalsha(...)) (in our case via RxJava's Single.fromPublisher) never terminated; the thread dump showed the test parked in blockingGet(), the netty event loop idle, and no pending command. We confirmed the cause by running the same code with a copy of RedisPublisher that pauses between state() and changeState(): the publisher then hangs every time (well past the 60 s command timeout), and terminates with the error once the CAS is retried.

Thread dump (relevant part)
"ForkJoinPool-1-worker-1" WAITING (parking)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
    at io.reactivex.rxjava3.internal.observers.BlockingMultiObserver.blockingGet(BlockingMultiObserver.java:86)
    at io.reactivex.rxjava3.core.Single.blockingGet(Single.java:3645)
    at ...RateLimitImplIT.timedConsumptionExecutesTheProductionScriptAndSetsExpiry(RateLimitImplIT.java:62)
"lettuce-nioEventLoop-4-1" RUNNABLE (idle in select)
"lettuce-timer-3-1" TIMED_WAITING (sleeping)

Input Code

Input Code
// First call on a Redis that does not have the script cached: the reply is a NOSCRIPT error.
Mono.from(connection.reactive().evalsha(sha1, ScriptOutputType.MULTI, keys, args))
        .block(); // occasionally never returns

A deterministic unit test that forces the interleaving (the subscribing thread moving DEMAND -> READING right before the error's CAS) is in the linked pull request.

Expected behavior/code

The subscriber receives onError with the RedisCommandExecutionException, regardless of what the subscribing thread does to the state concurrently.

Environment

  • Lettuce version(s): 7.8.0.RELEASE; the same State.onError code is in 7.7.0, 6.8.0 and current main
  • Redis version: 8.6.4

Possible Solution

Retry the CAS instead of leaving the loop when it fails:

State state;
while ((state = subscription.state()) != COMPLETED) {
    if (!subscription.changeState(state, COMPLETED)) {
        continue;
    }
    readData(subscription);
    Subscriber<?> subscriber = subscription.subscriber;
    if (subscriber != null) {
        subscriber.onError(t);
    }
    return;
}

I have a pull request ready with this change and a unit test that fails without it.

Additional context

Commands that rarely get error replies are only exposed when Redis is already failing (LOADING, READONLY during failover, OOM, MISCONF, WRONGTYPE), so the realistic trigger is EVALSHA + NOSCRIPT, which is the normal first call after a Redis restart, failover or SCRIPT FLUSH.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址