The cost of undoing: partial failures and the cleanup bill

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

An agent I was running had a simple job: register a customer in three systems, then send a welcome email. It succeeded at steps 1–3 (customer record in the CRM, account created in the billing system, subscription set in the feature service), then failed at step 4 (email send gateway returned a 429).

The agent's error-recovery logic kicked in: retry the email, wait, retry again. After the retry budget was exhausted, the agent reported failure and stopped.

What it didn't report: three systems now had a customer record that didn't exist in any of the others. The email never sent, so the customer never knew they had an account. The CRM and billing system had data that was out of sync with the feature service. A week later, when a customer support ticket came in ("I thought I signed up but nothing works"), someone had to manually find the orphaned records across three systems, decide whether to roll them back or forward-fix them, and clean up the inconsistency.

The cost wasn't the failed email send. It was finding and fixing the orphaned state.

This is the cleanup bill: the hidden cost of partial failures that standard retry logic doesn't touch.

Why partial failures are expensive to undo

A partial failure — where some steps in a multi-step operation succeed and others fail — creates an asymmetry in cost:

1. The original operation was cheap. One agent, one pass, straightforward execution. Cost: 5 API calls, ~300 tokens, ~$0.01 in LLM spend.
2. The cleanup is not. Now you have to:
   • Detect that cleanup is needed (a human notices, or an alert fires, or a test fails)
   • Find which records were written (dig through logs, query across three systems)
   • Decide the recovery strategy (rollback vs. forward-fix)
   • Implement the fix (write a script, run it carefully, verify it worked)
   • Communicate the fix (notify the customer, update internal docs)

For a single incident, this costs $200–500 in human time. For a fleet of 10,000 agents, if the failure rate is even 0.1%, that's 10 orphaned-state incidents per day. The cleanup bill scales faster than the operations do.

The patterns that create cleanup debt

Pattern 1: No rollback plan baked into the agent

The naive approach:

async def onboard_customer(agent, customer_id):
    # Step 1: Write to CRM
    crm.create_customer(customer_id, data)

    # Step 2: Create billing account
    billing.create_account(customer_id, plan="starter")

    # Step 3: Set feature flags
    features.enable(customer_id, ["core", "api"])

    # Step 4: Send email
    email.send(customer_id, "welcome.html")

    return "success"

If step 4 fails, steps 1–3 are committed. There's no automatic rollback because there's no rollback mechanism defined. The agent doesn't even know that a partial success is a problem — it just knows that step 4 failed and retried.

The fix is to define a rollback strategy and bake it into the flow:

async def onboard_customer(agent, customer_id):
    try:
        # Step 1: Write to CRM
        crm_id = crm.create_customer(customer_id, data)

        # Step 2: Create billing account
        billing_id = billing.create_account(customer_id, plan="starter")

        # Step 3: Set feature flags
        features.enable(customer_id, ["core", "api"])

        # Step 4: Send email
        email.send(customer_id, "welcome.html")

        return {"status": "success", "crm_id": crm_id, "billing_id": billing_id}

    except EmailSendError as e:
        # Rollback: remove from feature service
        try:
            features.disable(customer_id)
        except:
            pass  # Best-effort rollback

        # Rollback: mark account as pending in billing
        try:
            billing.mark_pending(customer_id)
        except:
            pass  # Best-effort rollback

        # CRM stays (it's the source of truth; manual fix if needed)
        # Re-raise so caller knows to retry or escalate
        raise

The cost of defining the rollback and executing it on error is ~50 tokens and one extra API call per failure. The cost of not defining it is $200+ per incident, times the number of incidents. If your fleet has a 1% failure rate, you're paying the cost of cleanup hundreds of times more than you'd pay for preventive rollback definition.

Pattern 2: Rollback is defined but not tested

This is the silent killer. You have a rollback path, you've written it, and it's never been exercised until the day you actually need it.

def rollback_customer(customer_id):
    # Remove from feature service
    features.disable(customer_id)

    # Revert billing status
    billing.revert_to_pending(customer_id)

    # Delete from CRM
    crm.delete_customer(customer_id)  # <-- This call might not exist

When the real failure happens and you call the rollback:

• features.disable() works fine.
• billing.revert_to_pending() works, but turns out this endpoint has a bug where it creates a new pending record instead of reverting the old one. Now you have two billing records.
• crm.delete_customer() fails because the endpoint doesn't support deletion — it only allows marking as inactive.

The rollback itself is partially failed, and now you have a bigger mess than the original problem. The cost balloons: not just cleaning up the original orphaned state, but untangling the broken rollback too.

The fix: test the rollback paths under failure conditions before deploying.

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article