Make scheduled job handlers idempotent: safe to run twice

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

Any system that calls your endpoint on a schedule, or retries it after a failure, will sooner or later call it twice for the same piece of work. A network timeout after your handler finished, a retry after a slow response, two servers firing the same cron: each of these runs the job again. If your handler sends an email or charges a card, "again" is a real problem. The fix is to make the handler idempotent: running it twice has the same effect as running it once.

Why duplicates are unavoidable

Consider a scheduler calling POST /jobs/send-invoices. Your handler takes 35 seconds. The caller's timeout is 30 seconds, so it records a failure and retries. Your first run finishes anyway. Now the invoices go out twice.

You can't fix this purely on the caller's side. Without the caller knowing exactly what your handler completed, "at least once" delivery is the only safe guarantee, and handling duplicates is the handler's job.

Technique 1: an idempotency key

Give each unit of work a unique key and record it when you process it. If the key has been seen, skip the work.

# Flask + PostgreSQL example
from flask import Flask, request
import psycopg

app = Flask(__name__)

@app.post("/jobs/send-reminder")
def send_reminder():
    payload = request.get_json()
    key = payload["job_key"]  # e.g. "reminder:user42:2026-11-01"
    with psycopg.connect(DSN) as conn:
        cur = conn.execute(
            "INSERT INTO processed_jobs (key) VALUES (%s) ON CONFLICT DO NOTHING",
            (key,),
        )
        if cur.rowcount == 0:
            return {"status": "duplicate, skipped"}, 200
        send_email(payload["user_id"])
    return {"status": "sent"}, 200

CREATE TABLE processed_jobs (
  key text PRIMARY KEY,
  processed_at timestamptz DEFAULT now()
);

The unique constraint does the hard work: if two copies arrive at the same moment, only one insert succeeds.

Return a 2xx for duplicates. If you return an error, the caller retries again.

Choosing the key

The key should describe the work, not the delivery attempt:

• Good: invoice-run:2026-11, reminder:user42:trial-day-3
• Bad: a random UUID generated per request, a timestamp of when the call arrived

For recurring jobs, include the period: daily-report:2026-11-01. Then a second call on the same day is skipped, and tomorrow's call runs normally.

Technique 2: make the work itself naturally idempotent

Some operations are safe to repeat by design:

-- Not idempotent
UPDATE accounts SET credits = credits + 100 WHERE id = 42;

-- Idempotent
UPDATE accounts SET plan = 'pro', renewed_until = '2026-12-01' WHERE id = 42;

Setting a value is safe; incrementing it isn't. Upserts, "mark as done" flags and "recalculate from source" jobs are all naturally safe to run again.

Idempotent handlers still need a scheduler that will retry them when the first attempt fails. delay.run does that retry and keeps the log, so a double-run is something you can see and design for.

Technique 3: claim, then process

For batch jobs, mark each item as claimed in the same step as selecting it:

UPDATE emails
SET status = 'sending', claimed_at = now()
WHERE id IN (
  SELECT id FROM emails
  WHERE status = 'queued'
  ORDER BY id
  LIMIT 100
  FOR UPDATE SKIP LOCKED
)
RETURNING id, recipient;

Two overlapping runs each get different rows, and a re-run picks up only what's left.

External side effects

The idempotency key protects your database, but what about the payment provider or email API you call? Many such APIs accept their own idempotency key header. Pass your job key through so the provider can also drop duplicates:

requests.post(
    "https://api.payments.example/charges",
    headers={"Idempotency-Key": key},
    json={...},
)

If an API has no such option, record "sent" in your database before marking the job complete, and accept that a crash in between could still send twice. Design for the rare duplicate to be harmless.

Respond fast, work later

A long-running handler makes duplicates more likely, because callers time out. Accept the request, store the work, and return 202 Accepted quickly, then process it in a background worker. The idempotency key goes with it.

Test it

The simplest test: call the endpoint twice with the same payload, in parallel.

for i in 1 2; do
  curl -s -X POST localhost:5000/jobs/send-reminder \
    -H 'content-type: application/json' \
    -d '{"job_key":"test-1","user_id":42}' &
done; wait

One response should say "sent" and the other "duplicate".

Where retries come from

A hosted scheduler like delay.run calls your URL on a cron schedule or after a delay and retries failed jobs with backoff, logging each attempt. Retries are what make jobs reliable, and idempotent handlers are what make retries safe.

Related reading

• How to retry failed cron jobs with exponential backoff in Bash and Python
• Heartbeat checks: how to know when a scheduled job didn't run
• How to log cron job output so you can debug it later
• Cron vs systemd timers: which should run your scheduled jobs?
• Dead letter queues for HTTP jobs: what they are and how to use one
• Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC