patternTip
Bulk email engine in Postgres: per-recipient claim, at-most-once after crash, local SMTP simulator for bounces
Viewed 0 times
skip lockedat-most-onceVERPList-Unsubscribe-PostMailpitbounce simulationfrequency cap
Problem
Designing a marketing-email send engine on top of a single SMTP transport (local Mailpit in dev) that must survive restarts, never send the same email twice to an address, apply exclusions (unsubscribed, hard bounced, rolling frequency cap, monthly quota) at send time, and still let developers test bounces, deferrals and complaints even though the local catcher accepts everything.
Solution
Keep all state in the database: one row per send, one row per recipient with UNIQUE(send_id, address) plus a partial unique index on (root_email_id, address) so a 'send to more' never re-targets original recipients. A loop claims N 'queued' recipients with FOR UPDATE SKIP LOCKED, sets state='sending' with claimed_until and a claim uuid, COMMITS, then calls SMTP outside any transaction, then writes the result in one short transaction. A recovery loop turns expired 'sending' rows into 'interrupted' and never retries them (at-most-once: a crash between SMTP 250 and the result commit undercounts, never duplicates). Use a deterministic Message-ID per recipient so duplicates are detectable. Dynamic rules (rolling per-contact cap, monthly counter) are checked at claim time under pg_advisory_xact_lock(hashtext(address)) so concurrent sends cannot exceed the cap. Use VERP envelope sender (bounces+<recipientToken>@domain) to match asynchronous DSNs back to the recipient. For local testing, run a tiny SMTP server (node:net) only in dev/tests that answers 550/552/451/timeout by local part on a reserved .test domain and relays everything else to the catcher; never load it in production.
Why
SMTP acceptance and the database commit cannot be atomic; choosing which side of the gap to fail on (duplicate vs missed) must be explicit, and for bulk marketing email a missed count is far less harmful than a duplicate send.
Gotchas
- Preference/unsubscribe pages must not change state on GET: link scanners follow links; only POST (and RFC 8058 one-click POST with body List-Unsubscribe=One-Click) changes status.
- Count unique recipients for rates; keep bot opens as flagged events instead of dropping them so stats can be shown with and without bots.
- Subscription and bounce status should be keyed by email address, not contact id, so they survive contact deletion and recreation.
Revisions (0)
No revisions yet.