Repository navigation
Description
DisconnectedBehavior.REJECT_COMMANDS does not fail fast for Redis Cluster commands waiting on lazy node connection establishment
Bug Report
Current Behavior
When using Lettuce with Redis Cluster and DisconnectedBehavior.REJECT_COMMANDS, commands do not always fail fast when the target Redis node/connection is unavailable.
We are seeing this during client startup / lazy cluster-node connection establishment. If the client is not able to establish a connection to the Redis Cluster node for a given slot, Lettuce appears to attach new commands to the pending connection-establishment future instead of rejecting them immediately.
For example, Redis Cluster commandssuch as GET / SET can wait until the connection attempt fails, e.g. after our configured CONNECT_TIMEOUT of 1 second, instead of failing immediately with REJECT_COMMANDS.
This causes latency spikes in our clients during network problems or when the cache is unavailable. In these scenarios, our desired behavior is fail-fast so the request can fall back to the database immediately, instead of waiting for the Redis connect timeout.
From code inspection, this seems to happen in the Redis Cluster path:
-
DefaultEndpointhas logic to reject writes when disconnected andrejectCommandsWhileDisconnectedis enabled:
https://github.com/redis/lettuce/blob/main/src/main/java/io/lettuce/core/protocol/DefaultEndpoint.java#L315 -
However, for Redis Cluster keyed commands,
ClusterDistributionChannelWriterobtains the node connection asynchronously and, if the connection future is not completed yet, attaches the command to the pending future:
https://github.com/redis/lettuce/blob/main/src/main/java/io/lettuce/core/cluster/ClusterDistributionChannelWriter.java#L162
As a result, the command may not reach DefaultEndpoint.validateWrite(...) while the node connection is still being established, so the configured REJECT_COMMANDS behavior is not applied in this path.
Observed behavior:
- Configure Redis Cluster client with:
disconnectedBehavior(REJECT_COMMANDS)- connect timeout = 1 second
- Start the client while Redis/cache is unavailable, or simulate network issues to one or more cluster nodes.
- Issue
GET/SETcommands. - Commands wait for the connection establishment future / connect timeout instead of failing immediately.
- Client requests experience approximately 1 second latency spikes before falling back to DB.
Stack trace
// No stack trace is produced in the fail-fast path.
// The issue is that commands remain pending until the cluster node connection
// attempt fails/times out, instead of immediately completing exceptionally.Input Code
Input Code
import io.lettuce.core.ClientOptions;
import io.lettuce.core.RedisURI;
import io.lettuce.core.SocketOptions;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.RedisClusterClient;
import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;
import io.lettuce.core.cluster.api.async.RedisAdvancedClusterAsyncCommands;
import java.time.Duration;
import java.util.concurrent.TimeUnit;
public class LettuceClusterRejectCommandsRepro {
public static void main(String[] args) throws Exception {
RedisURI redisUri = RedisURI.Builder
.redis("redis-cluster-host", 6379)
.withTimeout(Duration.ofSeconds(1))
.build();
RedisClusterClient client = RedisClusterClient.create(redisUri);
SocketOptions socketOptions = SocketOptions.builder()
.connectTimeout(Duration.ofSeconds(1))
.build();
ClusterClientOptions clientOptions = ClusterClientOptions.builder()
.autoReconnect(true)
.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
.socketOptions(socketOptions)
.build();
client.setOptions(clientOptions);
StatefulRedisClusterConnection<String, String> connection = client.connect();
RedisAdvancedClusterAsyncCommands<String, String> commands = connection.async();
// Reproduction setup:
//
// 1. Make the Redis/cache cluster unavailable, or block network connectivity
// to the Redis Cluster node that owns the slot for this key.
//
// 2. Issue the command below during startup / lazy node connection establishment.
//
// 3. Expected:
// The command fails immediately because REJECT_COMMANDS is configured.
//
// 4. Actual:
// The command waits for the pending connection attempt and completes only
// after the connect timeout, e.g. approximately 1 second.
long startNanos = System.nanoTime();
try {
commands.get("some-key")
.toCompletableFuture()
.get(5, TimeUnit.SECONDS);
} catch (Exception e) {
long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
System.out.println("Command failed after " + elapsedMillis + " ms");
e.printStackTrace();
} finally {
connection.close();
client.shutdown();
}
}
}Expected behavior/code
With DisconnectedBehavior.REJECT_COMMANDS, Redis commands should fail fast whenever the target node connection is not currently available.
In the Redis Cluster lazy connection-establishment path, if the target node connection future is not already successfully completed, Lettuce should not attach the command to that pending connection future when REJECT_COMMANDS is configured. Instead, the command should complete exceptionally immediately.
Basically the REJECT_COMMAND flag works if connection moved from Connected to Disconnected. But if client was never able to create a connection, then commands don't fail fast.
Expected behavior in our setup:
Redis/cache unavailable or target node connection unavailable
GET/SET issued
REJECT_COMMANDS configured
Command fails immediately
Application falls back to DB without waiting for CONNECT_TIMEOUT
This is important because our cache layer is not the source of truth. During network issues or cache unavailability, we want to fail fast and route the request to the DB instead of adding Redis connection timeout latency to the user request.
Environment
- Lettuce version(s): 6.8.2
- Redis version: 7.2.11
- Redis mode: Redis Cluster
- Client option:
ClusterClientOptions.disconnectedBehavior(REJECT_COMMANDS) - Socket connect timeout: 1 second
- Auto reconnect: enabled
Possible Solution
One possible fix is to apply DisconnectedBehavior.REJECT_COMMANDS in ClusterDistributionChannelWriter before attaching commands to a pending cluster node connection future.
Current behavior appears to be roughly:
CompletableFuture<StatefulRedisConnection<K, V>> connectFuture =
clusterConnectionProvider.getConnectionAsync(connectionIntent, hash);
if (isSuccessfullyCompleted(connectFuture)) {
writeCommand(commandToSend, false, connectFuture.join(), null);
} else {
connectFuture.whenComplete((connection, throwable) ->
writeCommand(commandToSend, false, connection, throwable));
}Suggested behavior:
CompletableFuture<StatefulRedisConnection<K, V>> connectFuture =
clusterConnectionProvider.getConnectionAsync(connectionIntent, hash);
if (isSuccessfullyCompleted(connectFuture)) {
writeCommand(commandToSend, false, connectFuture.join(), null);
} else if (clientOptions.getDisconnectedBehavior() == ClientOptions.DisconnectedBehavior.REJECT_COMMANDS) {
commandToSend.completeExceptionally(
new RedisException("Connection not established. Commands are rejected."));
} else {
connectFuture.whenComplete((connection, throwable) ->
writeCommand(commandToSend, false, connection, throwable));
}Alternatively, if maintainers consider this behavior a compatibility-sensitive change, it may make sense to add a separate cluster/client option to reject commands while lazy node connection establishment is still in progress.
Additional context
The production impact is latency spikes during Redis/cache network problems or cache unavailability.
Our intended behavior is:
Redis healthy:
Use Redis normally.
Redis unavailable / target node connection unavailable:
Fail fast.
Application behavior:
Immediately fall back to DB.
Currently, with REJECT_COMMANDS and a 1 second connect timeout, commands can still wait around 1 second while Lettuce attempts to establish the Redis Cluster node connection. This defeats the purpose of fail-fast behavior for our use case.
This seems specific to the Redis Cluster command routing / lazy node connection path. Once a command reaches a concrete endpoint, DefaultEndpoint appears to honor rejectCommandsWhileDisconnected, but commands waiting on cluster node connection establishment may not reach that validation path.
Activity
a-TODO-rov commentedon May 15, 2026
Hey, @Vaibhav026
Thanks for reporting this.
Since you have investigated this behavior in depth by now, would you mind taking a look at that one #2866. I know that these are two different scenarios here, but both are related to the REJECT_COMMANDS policy. I am wondering if we should try to aim for unified solution or just fixing separately.
Vaibhav026 commentedon May 29, 2026
Hi @a-TODO-rov ,
Thanks for pointing me to #2866.
I had a look at that flow. My understanding is that #2866 was related to commands that had already reached DefaultEndpoint / CommandHandler.stack and were then drained after channelInactive. In older versions, those commands were not rejected with REJECT_COMMANDS when autoReconnect=true because the logic was gated on AT_MOST_ONCE. But it looks like that was already addressed in newer Lettuce versions by changing the condition to rely directly on rejectCommandsWhileDisconnected(code).
For this issue, the command seems to be stuck earlier in the cluster path, before it reaches DefaultEndpoint.write(...) / validateWrite(). In ClusterDistributionChannelWriter, the command is attached to a pending connectFuture, so the existing REJECT_COMMANDS handling in DefaultEndpoint does not get a chance to reject it.
So I think both issues are related semantically, but the missing enforcement point here is different. Would it be reasonable to proceed with a focused fix for this issue in the cluster writer path, where we reject immediately instead of registering the command on a pending connection future when REJECT_COMMANDS is configured?
Vaibhav026 commentedon Jun 4, 2026
@a-TODO-rov I also wanted to call out one related path in DefaultEndpoint around RetryListener.
Scenario:
A command can pass DefaultEndpoint.validateWrite(...) because the endpoint appears connected at that moment. Then DefaultEndpoint.writeToChannelAndFlush(...) attempts the Netty write. Since autoReconnect(true) maps to AT_LEAST_ONCE reliability, Lettuce attaches RetryListener to the ChannelFuture.
The flow is roughly:
If the local Netty write fails, RetryListener#doComplete(...) is invoked. For non-encoder / non-fatal failures, it currently goes into the requeue path:
The concern is that with disconnectedBehavior(REJECT_COMMANDS), if the endpoint is already known to be disconnected at the point where RetryListener is about to requeue, we probably should not requeue the command. Instead, we should fail it immediately with the same behavior/message as the disconnected validateWrite path.
Suggested behavior:
Pseudo-code:
So the intended change is narrow:
This would make the local write failure path consistent with the existing DefaultEndpoint.validateWrite(...) behavior, where commands are rejected when rejectCommandsWhileDisconnected is true.
Let me know if this change too makes sense, these two changes in together will help in failing fast when the REJECT_COMMANDS flag is enabled.
Vaibhav026 commentedon Jun 8, 2026
@a-TODO-rov @tishun can you help with this? We are currently facing issues in production hence want to mitigate the behvaiour.
Reject pending node connection commands when REJECT_COMMANDS is set r…
Fail fast in RetryListener when REJECT_COMMANDS is configured redis#3747
a-TODO-rov commentedon Jun 23, 2026
Hey @Vaibhav026,
I spent time reviewing the request and proposed solution.
REJECT_COMMANDSis defined as rejecting commands in disconnected state. Lazy cluster node connection establishment, or an uncached per-node connection, is not necessarily a disconnected state. It can simply mean the connection has not been opened yet.Applying
REJECT_COMMANDSto every pending lazy connection future would therefore be a behavior change, not just a bug fix. It may satisfy your fail-fast use case, but it can break users who rely on Lettuce lazily opening cluster node connections.As a workaround, you can warm up the relevant cluster node connections during startup/readiness so requests do not trigger first-use connection establishment.
I’ll wait for your feedback, but unless we define a separate option for this behavior, I’m leaning toward closing this as not planned.
Vaibhav026 commentedon Jun 23, 2026
@a-TODO-rov warming up does not help for our use-case since lets say connection to a particular Redis(or a set of Redis) is not established, then to fail fast that Redis, we would need to intercept which hash slots all the commands are routed to and based on that take a call whether we allow the command to go further or not(basically mimicking Lettuce behaviour).
Would you be open to introducing another option like: .preconnectBehaviour(ClientOptions.PreconnectBehaviour.REJECT_COMMANDS), via which we can keep the default behaviour as is, so that the change is not breaking for clients who have not opted in, and for those who have opted in, can reject commands if connection is not established?
a-TODO-rov commentedon Jun 24, 2026
@Vaibhav026
Let me try to summarize your usecase:
I want Redis cache lookup to consume at most N milliseconds of my request budget. If Redis/Lettuce cannot return a cached value within that budget, fall back to DB.Is this correct ?
Vaibhav026 commentedon Jun 26, 2026
@a-TODO-rov We have built a platform used by different use cases (around 400) and they can set a per use-case timeout and because we use the same connection for all use cases, we have a per request timeout based on which use-cases is sending the request. The timeout set by the users are quite large to accommodate for any blips, like jvm pauses due to GC, safepoint, etc. Usually, these blips last for a few seconds and so it is fine to wait for this large timeout. If the issue is persistent, then we cannot have all requests wait for this timeout. We are willing to wait for the timeout if connection is healthy. If connection is not healthy, then we want to fail fast, even without waiting for the timeout.
If these is a timeout on the pending future(where the requests were getting stuck), then that property is also not required.
8 remaining items
Reject pending node connection commands when REJECT_COMMANDS is set r…
Fail fast in RetryListener when REJECT_COMMANDS is configured redis#3747
raiatul26 commentedon Aug 17, 2026
@a-TODO-rov we are waiting on the fix for this issue? Any updates @a-TODO-rov @Vaibhav026
Fix/cluster reject commands pending future (#3782)
a-TODO-rov commentedon Aug 18, 2026
Sorry for the delay, guys. It's been busy couple of months. Merged the PR and will release it as part of 7.7 shortly.
Vaibhav026 commentedon Aug 18, 2026
@a-TODO-rov can we release the changes to 6.8 too? I can raise a PR for the same(from 6.8.x branch) if that sounds okay.
a-TODO-rov commentedon Aug 18, 2026
@Vaibhav026 i still consider this change to be breaking the established behavior and i am not willing to backport it to older versions.
Whoever needs this change will need to upgrade.
Is this a problem for you ?
Vaibhav026 commentedon Aug 19, 2026
Yes @a-TODO-rov we’re actually on Lettuce 6.8.2 as of now. Upgrading to Lettuce 7 would require upgrading Netty to 4.2, which is a significant change for us since we have multiple dependencies on Netty beyond just Lettuce.
tishun commentedon Sep 3, 2026
@a-TODO-rov , @uglide ,
I suggest we make an exception and do a compromise:
REJECT_COMMANDSIt is not the conventional way to do it, but it would solve the problem for most people.
Vaibhav026 commentedon Sep 8, 2026
Would work for us @tishun, let me know if we are good with this @a-TODO-rov @tishun
Vaibhav026 commentedon Oct 15, 2026
Hi team, we are still waiting on this, any possible path forward would help @a-TODO-rov @tishun
a-TODO-rov commentedon Oct 16, 2026
Hey @Vaibhav026
I will release 6.9.0.RELEASE with this change.