Short answer: use the every-minute trigger as a small admission loop, keep the payment work in a durable queue, and let workers enforce the provider's rate limit. The nightly reconciliation should be idempotent and observable; the public webhook is only a controlled way to wake the loop, not the worker itself.
The media case is easy to underestimate. A news service may have millions of subscription rows, but its payment provider might allow only a few requests per second. A cron tick that starts a full scan turns a modest schedule into a burst. A webhook that does the scan inline turns a transient network delay into a timeout.
I care about the bill and the retention policy here. Keeping every raw payment response forever makes incident review comfortable, but storage and privacy exposure grow with every run. Keeping only a reconciliation key, status, and a bounded error sample is cheaper and still lets an on-call engineer explain a mismatch. That is a deliberate loss: you may need to fetch an old provider record again when a dispute arrives.
What should an every-minute trigger actually admit?
Treat the scheduler as an admission controller. On each minute, it claims a small window of work, writes one durable job per account or settlement period, and returns quickly. The queue owns retries; the worker owns pacing. A lease or visibility timeout prevents two workers from processing the same claim at once, while an idempotency key such as account_id:statement_date makes a repeated claim harmless.
The public endpoint needs a narrow contract. Authenticate the caller, reject a stale schedule token, and enqueue a bounded batch. Do not accept an arbitrary account list from the internet. Return a request identifier so the scheduler can record that admission happened; the response should not imply that reconciliation finished.
Here is the core worker shape in Python. It uses a generic queue client so the policy stays portable.
from time import monotonic, sleep
MAX_REQUESTS_PER_SECOND = 4
WINDOW_SECONDS = 1.0
def drain(queue, provider, ledger, now):
sent_at = monotonic()
for job in queue.receive(limit=20, visibility_timeout=120):
if ledger.already_applied(job.idempotency_key):
queue.ack(job)
continue
elapsed = monotonic() - sent_at
target = 1 / MAX_REQUESTS_PER_SECOND
if elapsed < target:
sleep(target - elapsed)
result = provider.fetch_statement(job.account_id, job.statement_date)
ledger.record(job.idempotency_key, result, observed_at=now())
queue.ack(job)
sent_at = monotonic()
That loop is intentionally boring. The dangerous part is not the sleep; it is deciding what happens after a timeout, a duplicate delivery, or a provider response that says the statement is still pending. Classify those outcomes separately. Retry transport failures with backoff, send poison messages to a dead-letter queue, and mark a business-level mismatch for review instead of retrying it forever.
How do cron, queue, webhook, and rate limits fit together?
Think in four stages: schedule, admit, process, reconcile. The schedule fires every minute. Admission creates bounded queue work. Processing consumes at the provider's rate. Reconciliation compares the provider result with the local ledger and emits a correction task when the two disagree. This separation lets you change the worker pool without changing the public endpoint.
The minute boundary is not a promise of minute-level completion. If 18,000 accounts arrive and the provider permits four requests per second, the lower bound is about 75 minutes before overhead. That arithmetic should be visible on a dashboard, beside queue age and oldest-message age. A green cron metric with a growing queue is a false sense of health.
Keep it boring.
The useful operational detail is the admission record, not another scheduler feature. Persist the schedule token, the batch upper bound, and the first and last account keys for each tick. When an on-call engineer compares a provider statement with the local ledger, those three fields answer whether work was never admitted, admitted but waiting, or processed and later rejected. Add a correlation ID to the queue message and carry it through the provider request, ledger write, and alert. If the provider has a per-account limit as well as a global limit, partition the queue key so one noisy account cannot consume the whole minute. If a night is interrupted, resume from the last committed cursor rather than rebuilding the entire account list. This is where a rate-limited design earns its keep: it makes the slow path explicit, measurable, and restartable instead of hiding it behind a long-running HTTP request.
For a Kubernetes deployment, CronJob concurrency policy and missed-run behavior deserve an explicit choice. A platform scheduler can prevent overlapping jobs, but it cannot make provider calls idempotent. For a managed queue, a dead-letter queue gives operators a bounded place to inspect repeated failures; retention and replay rules still belong to your team.
The cost-and-retention decision is mostly about what you stop keeping
The dominant cost is usually not the cron invocation. It is the volume of provider payloads and the retention period for them. Store a compact result envelope: account identifier, statement date, provider reference, outcome, attempt count, and a hash of the source payload. Put sensitive raw payloads in short-lived encrypted storage only when a dispute or audit requires them.
| Choice | Latency effect | Cost and risk | Use it when |
|---|---|---|---|
| Large admission batch | Faster queue fill, sharper burst | More throttling and replay work | Provider has generous quotas |
| Small admission batch | Smooth pressure, slower catch-up | More scheduler overhead | Quotas are strict or variable |
| Long raw-payload retention | Easier forensic review | Storage, privacy, deletion work | Regulation or contracts require it |
| Compact ledger plus short evidence window | Slightly more re-fetch latency | Lower storage exposure | The provider supports deterministic lookups |
I would stop retaining raw payloads after the evidence window unless a legal hold exists. Your mileage may vary: a broadcaster with chargeback obligations may need a longer window than a free publication. The trade-off is real, and hiding it in a “cleanup” job does not remove it.
Failure drills that keep the webhook boring
Test the unpleasant paths before production: deliver the same webhook twice, delay a queue acknowledgment, return a provider timeout after the provider accepted the request, and fill the dead-letter queue. Verify that the idempotency key prevents a second ledger write and that an operator can replay one message without replaying the whole night.
I once started with a single “run reconciliation” endpoint and a ten-minute timeout. It passed a small fixture, then failed at 02:00 when the provider slowed down. The fix was not a larger timeout; it was moving work behind the queue and making the endpoint acknowledge admission only. That distinction also keeps an external scheduler from retrying a request that already created work.
Do not put secrets in query strings, log full card or bank fields, or use a 200 response for a rejected signature. Record correlation IDs, queue age, provider response class, and the reason a message entered the dead-letter queue. Those fields are enough to find a stuck settlement without turning logs into a second database.
References
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
- https://www.rfc-editor.org/rfc/rfc9110
Further reading
The AWS SQS dead-letter queue guidance covers redrive and retention choices, while the Kubernetes CronJob documentation explains concurrency and missed schedules. RFC 9110 defines the HTTP semantics that make an authenticated, short-lived admission endpoint predictable.
Top comments (0)