*Scenario: *
- An AI agent calls a payment API.
- The payment succeeds, but the HTTP response times out.
- The agent sees an error and retries.
- The payment succeeds again.
This is one of the failure scenarios that becomes important when AI agents move from conversational applications into real enterprise workflows.
The issue is not necessarily with the LLM. It is a classic distributed systems problem: the caller cannot determine whether an external operation completed when its response is lost.
In this article, we'll design a small execution layer that protects against duplicate operations.
We'll use a payment-processing example, but the same design applies to:
- Creating customer accounts.
- Submitting orders.
- Booking appointments.
- Issuing refunds.
- Provisioning cloud resources.
- Triggering external workflows.
The implementation uses Python, FastAPI, and PostgreSQL-style SQL. The code is illustrative and needs provider-specific integration and production hardening before deployment.
- Define the failure scenario
Suppose an agent sends the following request:
POST /payments
Content-Type: application/json
{
"customer_id": "customer-123",
"invoice_id": "invoice-456",
"amount": 250.00,
"currency": "USD"
}
The application calls the payment provider.
The provider processes the payment, but the response is lost.
The agent retries the same request.
Without additional controls, the provider may process another payment.
The application needs to recognize that both requests represent the same logical business operation.
This is the purpose of idempotency.
- Design the execution flow
Instead of allowing the agent to invoke the payment provider directly, introduce an execution gateway.
AI Agent
|
v
Execution API
|
v
Request Validation
|
v
Operation Store
|
+---------+----------+
| |
v v
Existing Operation New Operation
| |
v v
Return/Check State Execution Worker
|
v
Payment Provider
|
v
State Update
|
v
Agent Result
The agent requests an operation. The execution layer owns its identity, state, and retry lifecycle.
- Create the operation database
We need a persistent record for every logical operation.
A PostgreSQL schema could look like this:
CREATE TABLE agent_operations (
operation_id UUID PRIMARY KEY,
idempotency_key VARCHAR(255) NOT NULL UNIQUE,
agent_id VARCHAR(100) NOT NULL,
operation_type VARCHAR(100) NOT NULL,
customer_id VARCHAR(100) NOT NULL,
request_hash VARCHAR(64) NOT NULL,
request_payload JSONB NOT NULL,
status VARCHAR(30) NOT NULL,
provider_reference VARCHAR(255),
retry_count INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
A few decisions are important here.
The idempotency key is unique.
This prevents two concurrent requests from independently registering the same logical operation.
The request hash is stored.
A client should not be able to reuse the same idempotency key with a different amount or customer.
The provider reference is persistent.
If the application restarts, it can still reconcile the external transaction.
Execution state lives in the database.
The LLM conversation is not the source of truth for whether a payment occurred.
- Define explicit execution states
Start with a small state machine:
PENDING
|
v
IN_PROGRESS
|
+-----------> SUCCEEDED
|
+-----------> FAILED
|
+-----------> UNKNOWN
|
v
RECONCILING
|
+------+------+
| |
v v
SUCCEEDED FAILED
The distinction between FAILED and UNKNOWN is essential.
FAILED means the system has evidence that the operation did not succeed.
UNKNOWN means the system cannot currently establish the outcome.
A timeout should generally produce UNKNOWN, not FAILED, unless the provider contract or transport evidence proves that the operation could not have occurred.
This avoids treating a missing response as proof of a failed transaction.
- Implement request validation and hashing
First, define the request model.
from decimal import Decimal
from pydantic import BaseModel, Field
class PaymentRequest(BaseModel):
customer_id: str
invoice_id: str
amount: Decimal = Field(gt=0)
currency: str = "USD"
Next, create a deterministic hash.
import hashlib
import json
def calculate_request_hash(request: PaymentRequest) -> str:
canonical_payload = {
"customer_id": request.customer_id,
"invoice_id": request.invoice_id,
"amount": str(request.amount),
"currency": request.currency.upper()
}
serialized = json.dumps(
canonical_payload,
sort_keys=True,
separators=(",", ":")
)
return hashlib.sha256(
serialized.encode("utf-8")
).hexdigest()
This makes it possible to detect a dangerous situation:
The same idempotency key is submitted again, but the amount or invoice has changed.
That should be rejected rather than silently treated as a retry.
In a production system, canonicalization must also account for all business-relevant fields, including currency precision, optional fields, and normalized identifiers.
- Create the operation atomically
A naive implementation might do this:
if not operation_exists(key):
create_operation(key)
This has a race condition.
Two workers can both check the database before either inserts a record.
Both can then proceed.
Instead, use a database constraint and an atomic insert.
Illustrative PostgreSQL:
INSERT INTO agent_operations (
operation_id,
idempotency_key,
agent_id,
operation_type,
customer_id,
request_hash,
request_payload,
status
)
VALUES (
:operation_id,
:idempotency_key,
:agent_id,
'PAYMENT',
:customer_id,
:request_hash,
CAST(:request_payload AS JSONB),
'PENDING'
)
ON CONFLICT (idempotency_key)
DO NOTHING;
After the insert, retrieve the operation by its idempotency key.
If it already exists:
Compare the stored request hash.
Reject a mismatched payload.
Return the existing operation state.
Do not initiate another logical payment.
The unique database constraint, not an in-memory check, is what protects the operation registration against concurrent requests.
- Build the execution API
An API request should include an idempotency key.
For example:
POST /agent/operations/payment
Idempotency-Key: payment-customer123-invoice456
A simplified FastAPI endpoint:
from fastapi import FastAPI, Header, HTTPException
app = FastAPI()
@app.post("/agent/operations/payment")
async def create_payment(
request: PaymentRequest,
idempotency_key: str = Header(
alias="Idempotency-Key"
)
):
if not idempotency_key:
raise HTTPException(
status_code=400,
detail="Idempotency-Key is required"
)
request_hash = calculate_request_hash(request)
operation = await register_or_get_operation(
idempotency_key=idempotency_key,
request=request,
request_hash=request_hash
)
if operation.request_hash != request_hash:
raise HTTPException(
status_code=409,
detail="Idempotency key reused with different payload"
)
if operation.status == "SUCCEEDED":
return {
"operation_id": str(operation.operation_id),
"status": "SUCCEEDED",
"provider_reference": operation.provider_reference
}
if operation.status in ("IN_PROGRESS", "UNKNOWN"):
return {
"operation_id": str(operation.operation_id),
"status": operation.status,
"message": "Operation already exists; check its status"
}
return await execute_payment(operation)
The functions register_or_get_operation and execute_payment are intentionally abstract here. Their implementation must use a transactional database layer, concurrency-safe state transitions, and a provider-specific payment adapter.
A useful API design also exposes:
GET /agent/operations/{operation_id}
The agent can retrieve the current operation status without initiating another payment.
- Pass the same idempotency key to the provider
The execution gateway must use the same stable key when calling the external provider.
Conceptually:
async def call_payment_provider(operation):
return await payment_provider.create_payment(
customer_id=operation.customer_id,
amount=operation.request_payload["amount"],
currency=operation.request_payload["currency"],
idempotency_key=operation.idempotency_key
)
This is the second layer of protection.
The local database protects the application's operation registration.
The provider's idempotency mechanism protects the external side effect.
Both are important.
Do not generate a new key for every HTTP attempt.
Also, do not assume every payment provider has identical idempotency behavior. Verify its retention period, request-matching rules, response behavior, and support for retrieving the original transaction.
- Handle timeouts without creating duplicate payments
This is the most important part.
A simplistic implementation might look like this:
try:
response = await call_payment_provider(operation)
await mark_succeeded(operation, response)
except TimeoutError:
await mark_failed(operation)
await retry_payment(operation)
This is unsafe.
A timeout does not establish that the provider failed to process the request.
A safer conceptual implementation:
async def execute_payment(operation):
await mark_in_progress(operation)
try:
response = await call_payment_provider(operation)
await mark_succeeded(
operation,
response.provider_reference
)
return {
"operation_id": str(operation.operation_id),
"status": "SUCCEEDED"
}
except TimeoutError:
await mark_unknown(operation)
return {
"operation_id": str(operation.operation_id),
"status": "UNKNOWN",
"message": (
"Payment outcome is being reconciled. "
"Do not submit a new payment."
)
}
This is not the complete worker implementation. The state updates must be durable and conditional, and provider exceptions must be classified according to the provider's documented semantics.
The important design choice is that the timeout does not trigger a fresh logical payment.
- Add a reconciliation worker
Operations in the UNKNOWN state need a recovery mechanism.
A reconciliation worker periodically checks uncertain operations.
Conceptually:
async def reconcile_operation(operation):
result = await payment_provider.lookup(
idempotency_key=operation.idempotency_key,
provider_reference=operation.provider_reference
)
if result.status == "SUCCEEDED":
await mark_succeeded(
operation,
result.provider_reference
)
elif result.status == "FAILED":
await mark_failed(operation)
elif result.status == "PENDING":
await keep_unknown(operation)
else:
await schedule_manual_review(operation)
The exact lookup mechanism depends on the payment provider.
Some providers allow lookup by idempotency key. Others require a transaction reference or a merchant-side business identifier.
If the provider cannot reliably establish the outcome, the system must not pretend that it knows. It should keep the operation unresolved and escalate it appropriately.
- Protect execution from concurrent workers
Even with idempotency keys, the application must protect its local execution lifecycle.
Imagine two workers pick up the same PENDING operation.
Both might try to call the provider.
A conditional state update can help establish a single local owner:
UPDATE agent_operations
SET
status = 'IN_PROGRESS',
updated_at = NOW()
WHERE
operation_id = :operation_id
AND status = 'PENDING';
Check the affected row count.
Only the worker that successfully transitions the operation should proceed.
For longer-running work, production systems may need:
Lease expiration.
Worker ownership identifiers.
Fencing tokens.
Heartbeats.
Recovery of abandoned operations.
Atomic job claiming, such as PostgreSQL FOR UPDATE SKIP LOCKED.
The external provider's idempotency key remains necessary because a worker can crash after the provider processes the payment but before the local database records success.
Local locking alone cannot solve that distributed transaction problem.
- Add an outbox pattern for reliable asynchronous execution
For a larger system, I would avoid having the HTTP request handler directly own the entire execution lifecycle.
A useful architecture is:
Create the operation record.
Insert an execution event into an outbox table in the same database transaction.
Commit both records together.
A background worker reads the outbox.
The worker invokes the external provider.
The operation record is updated.
The outbox event is marked as processed.
This prevents a common failure:
The database operation is committed, but the application crashes before publishing the execution message.
The outbox pattern makes the database update and the intent to publish an event atomic.
It does not make the external payment atomic with the database. Provider idempotency and reconciliation are still required.
- Add observability
A production execution layer should make it possible to trace one business operation across the agent, gateway, worker, database, and provider.
I would capture:
Attribute
Why it matters
agent_id
Identifies the initiating agent
operation_id
Correlates the business operation
idempotency_key
Identifies logical retries
trace_id
Connects distributed service logs
provider_reference
Supports external reconciliation
attempt_number
Shows retry behavior
state_transition
Reconstructs the operation lifecycle
execution_duration
Identifies slow integrations
Useful metrics include:
Duplicate requests prevented.
Operations in UNKNOWN state.
Reconciliation success rate.
Average execution time.
Retry count per operation.
Provider timeout rate.
Operations requiring human intervention.
Do not put payment credentials, sensitive customer information, or full payment payloads into ordinary application logs.
- What this means for AI agent architecture
The execution layer should not depend on which LLM is being used.
Whether an application uses OpenAI, Anthropic, Gemini, Llama, or another model, the same execution guarantees should apply.
That is an important architectural boundary.
The model is responsible for interpreting intent and requesting an operation.
The execution infrastructure is responsible for:
Validating the operation.
Enforcing permissions.
Establishing operation identity.
Preventing duplicate effects.
Managing execution state.
Handling retries.
Reconciling uncertain outcomes.
Recording an audit trail.
The execution layer should remain deterministic even when the model is probabilistic.
- Final thoughts
AI agents are becoming another type of distributed application client.
They interact with services, trigger workflows, and initiate operations. As a result, many of the hardest problems in agent engineering are not necessarily model problems.
They are problems of state management, concurrency, reliability, and distributed execution.
An agent that can call a payment API is relatively easy to build.
An agent that can safely call that API despite timeouts, retries, crashes, concurrent requests, and ambiguous outcomes requires a much more deliberate architecture.
The goal isn't to make the LLM responsible for exactly-once execution. The goal is to build infrastructure that makes duplicate business effects difficult to produce, detectable when they occur, and recoverable when outcomes are uncertain.
That is an important part of moving AI agents from impressive demonstrations into dependable enterprise systems.
About the author
I'm Ashwin Giridharan, a Cloud & Enterprise Solutions Architect focused on scalable systems, distributed architecture, cloud platforms, Generative AI, and production-grade AI infrastructure.
Website: https://ashwingiridharan.com/
LinkedIn: https://www.linkedin.com/in/ashwingiridharan7/
GitHub: https://github.com/giridharan-ashwin/
Medium: https://medium.com/@gashwin7
Top comments (0)