Skip to content

DisconnectedBehavior.REJECT_COMMANDS does not fail fast for Redis Cluster commands waiting on lazy node connection establishment #3747

Closed

Description

@Vaibhav026

DisconnectedBehavior.REJECT_COMMANDS does not fail fast for Redis Cluster commands waiting on lazy node connection establishment

Bug Report

Current Behavior

When using Lettuce with Redis Cluster and DisconnectedBehavior.REJECT_COMMANDS, commands do not always fail fast when the target Redis node/connection is unavailable.

We are seeing this during client startup / lazy cluster-node connection establishment. If the client is not able to establish a connection to the Redis Cluster node for a given slot, Lettuce appears to attach new commands to the pending connection-establishment future instead of rejecting them immediately.

For example, Redis Cluster commandssuch as GET / SET can wait until the connection attempt fails, e.g. after our configured CONNECT_TIMEOUT of 1 second, instead of failing immediately with REJECT_COMMANDS.

This causes latency spikes in our clients during network problems or when the cache is unavailable. In these scenarios, our desired behavior is fail-fast so the request can fall back to the database immediately, instead of waiting for the Redis connect timeout.

From code inspection, this seems to happen in the Redis Cluster path:

As a result, the command may not reach DefaultEndpoint.validateWrite(...) while the node connection is still being established, so the configured REJECT_COMMANDS behavior is not applied in this path.

Observed behavior:

  1. Configure Redis Cluster client with:
    • disconnectedBehavior(REJECT_COMMANDS)
    • connect timeout = 1 second
  2. Start the client while Redis/cache is unavailable, or simulate network issues to one or more cluster nodes.
  3. Issue GET / SET commands.
  4. Commands wait for the connection establishment future / connect timeout instead of failing immediately.
  5. Client requests experience approximately 1 second latency spikes before falling back to DB.
Stack trace
// No stack trace is produced in the fail-fast path.
// The issue is that commands remain pending until the cluster node connection
// attempt fails/times out, instead of immediately completing exceptionally.

Input Code

Input Code
import io.lettuce.core.ClientOptions;
import io.lettuce.core.RedisURI;
import io.lettuce.core.SocketOptions;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.RedisClusterClient;
import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;
import io.lettuce.core.cluster.api.async.RedisAdvancedClusterAsyncCommands;

import java.time.Duration;
import java.util.concurrent.TimeUnit;

public class LettuceClusterRejectCommandsRepro {

    public static void main(String[] args) throws Exception {

        RedisURI redisUri = RedisURI.Builder
                .redis("redis-cluster-host", 6379)
                .withTimeout(Duration.ofSeconds(1))
                .build();

        RedisClusterClient client = RedisClusterClient.create(redisUri);

        SocketOptions socketOptions = SocketOptions.builder()
                .connectTimeout(Duration.ofSeconds(1))
                .build();

        ClusterClientOptions clientOptions = ClusterClientOptions.builder()
                .autoReconnect(true)
                .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
                .socketOptions(socketOptions)
                .build();

        client.setOptions(clientOptions);

        StatefulRedisClusterConnection<String, String> connection = client.connect();
        RedisAdvancedClusterAsyncCommands<String, String> commands = connection.async();

        // Reproduction setup:
        //
        // 1. Make the Redis/cache cluster unavailable, or block network connectivity
        //    to the Redis Cluster node that owns the slot for this key.
        //
        // 2. Issue the command below during startup / lazy node connection establishment.
        //
        // 3. Expected:
        //    The command fails immediately because REJECT_COMMANDS is configured.
        //
        // 4. Actual:
        //    The command waits for the pending connection attempt and completes only
        //    after the connect timeout, e.g. approximately 1 second.

        long startNanos = System.nanoTime();

        try {
            commands.get("some-key")
                    .toCompletableFuture()
                    .get(5, TimeUnit.SECONDS);
        } catch (Exception e) {
            long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
            System.out.println("Command failed after " + elapsedMillis + " ms");
            e.printStackTrace();
        } finally {
            connection.close();
            client.shutdown();
        }
    }
}

Expected behavior/code

With DisconnectedBehavior.REJECT_COMMANDS, Redis commands should fail fast whenever the target node connection is not currently available.

In the Redis Cluster lazy connection-establishment path, if the target node connection future is not already successfully completed, Lettuce should not attach the command to that pending connection future when REJECT_COMMANDS is configured. Instead, the command should complete exceptionally immediately.

Basically the REJECT_COMMAND flag works if connection moved from Connected to Disconnected. But if client was never able to create a connection, then commands don't fail fast.

Expected behavior in our setup:

Redis/cache unavailable or target node connection unavailable
GET/SET issued
REJECT_COMMANDS configured
Command fails immediately
Application falls back to DB without waiting for CONNECT_TIMEOUT

This is important because our cache layer is not the source of truth. During network issues or cache unavailability, we want to fail fast and route the request to the DB instead of adding Redis connection timeout latency to the user request.

Environment

  • Lettuce version(s): 6.8.2
  • Redis version: 7.2.11
  • Redis mode: Redis Cluster
  • Client option: ClusterClientOptions.disconnectedBehavior(REJECT_COMMANDS)
  • Socket connect timeout: 1 second
  • Auto reconnect: enabled

Possible Solution

One possible fix is to apply DisconnectedBehavior.REJECT_COMMANDS in ClusterDistributionChannelWriter before attaching commands to a pending cluster node connection future.

Current behavior appears to be roughly:

CompletableFuture<StatefulRedisConnection<K, V>> connectFuture =
        clusterConnectionProvider.getConnectionAsync(connectionIntent, hash);

if (isSuccessfullyCompleted(connectFuture)) {
    writeCommand(commandToSend, false, connectFuture.join(), null);
} else {
    connectFuture.whenComplete((connection, throwable) ->
            writeCommand(commandToSend, false, connection, throwable));
}

Suggested behavior:

CompletableFuture<StatefulRedisConnection<K, V>> connectFuture =
        clusterConnectionProvider.getConnectionAsync(connectionIntent, hash);

if (isSuccessfullyCompleted(connectFuture)) {
    writeCommand(commandToSend, false, connectFuture.join(), null);
} else if (clientOptions.getDisconnectedBehavior() == ClientOptions.DisconnectedBehavior.REJECT_COMMANDS) {
    commandToSend.completeExceptionally(
            new RedisException("Connection not established. Commands are rejected."));
} else {
    connectFuture.whenComplete((connection, throwable) ->
            writeCommand(commandToSend, false, connection, throwable));
}

Alternatively, if maintainers consider this behavior a compatibility-sensitive change, it may make sense to add a separate cluster/client option to reject commands while lazy node connection establishment is still in progress.

Additional context

The production impact is latency spikes during Redis/cache network problems or cache unavailability.

Our intended behavior is:

Redis healthy:
  Use Redis normally.

Redis unavailable / target node connection unavailable:
  Fail fast.

Application behavior:
  Immediately fall back to DB.

Currently, with REJECT_COMMANDS and a 1 second connect timeout, commands can still wait around 1 second while Lettuce attempts to establish the Redis Cluster node connection. This defeats the purpose of fail-fast behavior for our use case.

This seems specific to the Redis Cluster command routing / lazy node connection path. Once a command reaches a concrete endpoint, DefaultEndpoint appears to honor rejectCommandsWhileDisconnected, but commands waiting on cluster node connection establishment may not reach that validation path.

Activity

  1. a-TODO-rov commented on May 15, 2026

    @a-TODO-rov
    Collaborator

    Hey, @Vaibhav026
    Thanks for reporting this.
    Since you have investigated this behavior in depth by now, would you mind taking a look at that one #2866. I know that these are two different scenarios here, but both are related to the REJECT_COMMANDS policy. I am wondering if we should try to aim for unified solution or just fixing separately.

  2. Vaibhav026 commented on May 29, 2026

    @Vaibhav026
    ContributorAuthor

    Hi @a-TODO-rov ,

    Thanks for pointing me to #2866.

    I had a look at that flow. My understanding is that #2866 was related to commands that had already reached DefaultEndpoint / CommandHandler.stack and were then drained after channelInactive. In older versions, those commands were not rejected with REJECT_COMMANDS when autoReconnect=true because the logic was gated on AT_MOST_ONCE. But it looks like that was already addressed in newer Lettuce versions by changing the condition to rely directly on rejectCommandsWhileDisconnected(code).

    For this issue, the command seems to be stuck earlier in the cluster path, before it reaches DefaultEndpoint.write(...) / validateWrite(). In ClusterDistributionChannelWriter, the command is attached to a pending connectFuture, so the existing REJECT_COMMANDS handling in DefaultEndpoint does not get a chance to reject it.

    So I think both issues are related semantically, but the missing enforcement point here is different. Would it be reasonable to proceed with a focused fix for this issue in the cluster writer path, where we reject immediately instead of registering the command on a pending connection future when REJECT_COMMANDS is configured?

  3. Vaibhav026 commented on Jun 4, 2026

    @Vaibhav026
    ContributorAuthor

    @a-TODO-rov I also wanted to call out one related path in DefaultEndpoint around RetryListener.

    Scenario:
    A command can pass DefaultEndpoint.validateWrite(...) because the endpoint appears connected at that moment. Then DefaultEndpoint.writeToChannelAndFlush(...) attempts the Netty write. Since autoReconnect(true) maps to AT_LEAST_ONCE reliability, Lettuce attaches RetryListener to the ChannelFuture.

    The flow is roughly:

    DefaultEndpoint.write(command)
      -> validateWrite(...) passes
      -> writeToChannelAndFlush(command)
      -> channel.writeAndFlush(command)
      -> RetryListener attached for AT_LEAST_ONCE
    

    If the local Netty write fails, RetryListener#doComplete(...) is invoked. For non-encoder / non-fatal failures, it currently goes into the requeue path:

    RetryListener#doComplete(...)
      -> potentiallyRequeueCommands(...)
      -> endpoint.write(command) again
    

    The concern is that with disconnectedBehavior(REJECT_COMMANDS), if the endpoint is already known to be disconnected at the point where RetryListener is about to requeue, we probably should not requeue the command. Instead, we should fail it immediately with the same behavior/message as the disconnected validateWrite path.

    Suggested behavior:

    if REJECT_COMMANDS is configured
    and the Netty local write fails
    then fail the command instead of requeueing it.
    

    Pseudo-code:

    if (endpoint.rejectCommandsWhileDisconnected) {
        complete(new RedisException("Currently not connected. Commands are rejected."));
        return;
    }
    
    potentiallyRequeueCommands(channel, sentCommand, sentCommands);
    

    So the intended change is narrow:

    • Preserve existing retry/requeue behavior for DEFAULT and ACCEPT_COMMANDS.
    • Fail fast when REJECT_COMMANDS is configured and netty local write fails instead of requeueing the command.

    This would make the local write failure path consistent with the existing DefaultEndpoint.validateWrite(...) behavior, where commands are rejected when rejectCommandsWhileDisconnected is true.

    Let me know if this change too makes sense, these two changes in together will help in failing fast when the REJECT_COMMANDS flag is enabled.

  4. Vaibhav026 commented on Jun 8, 2026

    @Vaibhav026
    ContributorAuthor

    @a-TODO-rov @tishun can you help with this? We are currently facing issues in production hence want to mitigate the behvaiour.

  5. a-TODO-rov commented on Jun 23, 2026

    @a-TODO-rov
    Collaborator

    Hey @Vaibhav026,

    I spent time reviewing the request and proposed solution.
    REJECT_COMMANDS is defined as rejecting commands in disconnected state. Lazy cluster node connection establishment, or an uncached per-node connection, is not necessarily a disconnected state. It can simply mean the connection has not been opened yet.

    Applying REJECT_COMMANDS to every pending lazy connection future would therefore be a behavior change, not just a bug fix. It may satisfy your fail-fast use case, but it can break users who rely on Lettuce lazily opening cluster node connections.

    As a workaround, you can warm up the relevant cluster node connections during startup/readiness so requests do not trigger first-use connection establishment.

    I’ll wait for your feedback, but unless we define a separate option for this behavior, I’m leaning toward closing this as not planned.

  6. added
    status: declinedA suggestion or change that we dont feel we should currently apply
    and removed on Jun 23, 2026
  7. Vaibhav026 commented on Jun 23, 2026

    @Vaibhav026
    ContributorAuthor

    @a-TODO-rov warming up does not help for our use-case since lets say connection to a particular Redis(or a set of Redis) is not established, then to fail fast that Redis, we would need to intercept which hash slots all the commands are routed to and based on that take a call whether we allow the command to go further or not(basically mimicking Lettuce behaviour).

    Would you be open to introducing another option like: .preconnectBehaviour(ClientOptions.PreconnectBehaviour.REJECT_COMMANDS), via which we can keep the default behaviour as is, so that the change is not breaking for clients who have not opted in, and for those who have opted in, can reject commands if connection is not established?

  8. a-TODO-rov commented on Jun 24, 2026

    @a-TODO-rov
    Collaborator

    @Vaibhav026
    Let me try to summarize your usecase:
    I want Redis cache lookup to consume at most N milliseconds of my request budget. If Redis/Lettuce cannot return a cached value within that budget, fall back to DB.
    Is this correct ?

  9. Vaibhav026 commented on Jun 26, 2026

    @Vaibhav026
    ContributorAuthor

    @a-TODO-rov We have built a platform used by different use cases (around 400) and they can set a per use-case timeout and because we use the same connection for all use cases, we have a per request timeout based on which use-cases is sending the request. The timeout set by the users are quite large to accommodate for any blips, like jvm pauses due to GC, safepoint, etc. Usually, these blips last for a few seconds and so it is fine to wait for this large timeout. If the issue is persistent, then we cannot have all requests wait for this timeout. We are willing to wait for the timeout if connection is healthy. If connection is not healthy, then we want to fail fast, even without waiting for the timeout.
    If these is a timeout on the pending future(where the requests were getting stuck), then that property is also not required.

  10. 8 remaining items

  11. raiatul26 commented on Aug 17, 2026

    @raiatul26

    @a-TODO-rov we are waiting on the fix for this issue? Any updates @a-TODO-rov @Vaibhav026

  12. a-TODO-rov commented on Aug 18, 2026

    @a-TODO-rov
    Collaborator

    Sorry for the delay, guys. It's been busy couple of months. Merged the PR and will release it as part of 7.7 shortly.

  13. Vaibhav026 commented on Aug 18, 2026

    @Vaibhav026
    ContributorAuthor

    @a-TODO-rov can we release the changes to 6.8 too? I can raise a PR for the same(from 6.8.x branch) if that sounds okay.

  14. a-TODO-rov commented on Aug 18, 2026

    @a-TODO-rov
    Collaborator

    @Vaibhav026 i still consider this change to be breaking the established behavior and i am not willing to backport it to older versions.
    Whoever needs this change will need to upgrade.
    Is this a problem for you ?

  15. Vaibhav026 commented on Aug 19, 2026

    @Vaibhav026
    ContributorAuthor

    Yes @a-TODO-rov we’re actually on Lettuce 6.8.2 as of now. Upgrading to Lettuce 7 would require upgrading Netty to 4.2, which is a significant change for us since we have multiple dependencies on Netty beyond just Lettuce.

  16. tishun commented on Sep 3, 2026

    @tishun
    Collaborator

    @a-TODO-rov , @uglide ,
    I suggest we make an exception and do a compromise:

    • release 6.9.0.RELEASE with explicit warning there could be behavioral changes when using REJECT_COMMANDS
    • do not address this anywhere else between 6.9.x to 7.7.x

    It is not the conventional way to do it, but it would solve the problem for most people.

  17. Vaibhav026 commented on Sep 8, 2026

    @Vaibhav026
    ContributorAuthor

    Would work for us @tishun, let me know if we are good with this @a-TODO-rov @tishun

  18. Vaibhav026 commented on Oct 15, 2026

    @Vaibhav026
    ContributorAuthor

    Hi team, we are still waiting on this, any possible path forward would help @a-TODO-rov @tishun

  19. a-TODO-rov commented on Oct 16, 2026

    @a-TODO-rov
    Collaborator

    Hey @Vaibhav026
    I will release 6.9.0.RELEASE with this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址