Your timeout is a bet: pricing the tradeoff before you pick a number

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

Ask an engineer why their agent's per-step timeout is set to 30 seconds and the honest answer is usually "it felt long enough." That number is a bet, placed without odds, against a distribution nobody looked at. Set it too low and you cut off calls that were about to succeed — a real result, discarded, paid for in tokens already spent and now retried from scratch. Set it too high and every genuine hang sits there burning wall-clock and holding a worker while nothing happens. Both directions cost money. The number that "feels long enough" is very rarely the number that minimizes either.

This is a failure mode wearing a config value's clothes. A timeout firing early looks identical to a real failure downstream — it's counted the same way in your logs, it triggers the same retry, and it can trip the same circuit breaker as an actual outage. Silent failures already hide inside your outcome taxonomy; a false timeout is one you manufactured with a bad guess at a number. That false timeout is then fed into your retry budget, burning tokens on work that would have succeeded, multiplying costs like any other retry.

Two ways to be wrong, priced differently

A call's true completion time is a distribution, not a constant — and it has a long right tail. Pick a timeout T and you split that distribution into two buckets, each with its own cost:

• False timeout (call would have finished, just after T): you paid for the work done so far, threw it away, and now pay again for a retry — plus whatever the retry's own chance of also timing out costs, compounding.
• True timeout (call was actually hung): you paid to sit idle for the full T before finding out, holding a worker the whole time.

Push T up and false timeouts get rarer but true ones get more expensive to detect. Push it down and detection gets cheap but you manufacture false timeouts on calls that were simply a bit slow. There's a number in between that minimizes the sum — and it's a calculation, not a feeling.

# Cost of a chosen timeout T, given the true call-duration distribution.
# Illustrative rates — swap in your own percentile curve and prices.
import random

IDLE_RATE   = 0.002        # $/second the worker burns just waiting
CALL_COST   = 0.10         # $ in tokens/compute already spent when it's cut off
RETRY_MULT  = 1.2          # a retry isn't a clean redo — some work must repeat

def sample_duration(rng):
    # long-tailed: most calls finish fast, a minority run long, a few truly hang.
    r = rng.random()
    if r < 0.85:  return rng.uniform(1, 6)     # normal
    if r < 0.95:  return rng.uniform(6, 30)    # slow but real
    return rng.uniform(120, 300)                # actually hung — never finishes

def cost_for_timeout(T, trials=20_000, seed=7):
    rng = random.Random(seed)
    total = 0.0
    for _ in range(trials):
        d = sample_duration(rng)
        if d <= T:
            total += IDLE_RATE * d                  # succeeded, just paid to wait
        else:
            total += IDLE_RATE * T + CALL_COST * RETRY_MULT  # cut off + retried
    return total / trials

for T in (5, 10, 15, 20, 30, 45, 60):
    print(f"T={T:3d}s  avg cost/call=${cost_for_timeout(T):.4f}")

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article