Duplicate Sends: Idempotency in Email Pipelines

Everyone has received the same email twice. The retry loops, queue semantics, and race conditions that cause duplicates, and the idempotency patterns that stop them.

The double send is the most visible infrastructure failure in email: recipients see it instantly, screenshots reach social media, and unsubscribe rates spike on the second copy. Yet the cause is almost never someone clicking send twice. Duplicates are what distributed systems do naturally when submission outcomes are ambiguous, and email pipelines are full of ambiguous outcomes. This article maps where duplicates are born and the patterns that prevent them.

Where duplicates come from

The ambiguous timeout

Your service POSTs to the ESP, the connection times out, and you have no idea whether the request landed. The provider may have accepted the message, queued it, and failed only to tell you. Retry, and the recipient gets two copies; do not retry, and maybe nobody gets one. Without an idempotency mechanism, both choices are wrong, and this single pattern accounts for most transactional duplicates.

At-least-once delivery, taken literally

Job queues (SQS, Sidekiq, Celery, Kafka consumers) promise at-least-once processing, and the at-least is not decorative. A worker that sends the email but crashes before acknowledging the job guarantees redelivery to another worker, which sends it again. Any send job that is not explicitly idempotent converts queue semantics into duplicate mail, on a schedule determined by your least reliable worker.

The partial batch and the overlapping trigger

Campaign sends fail in the middle: 60,000 of 200,000 recipients processed, then a crash. Rerunning from the top re-mails the first 60,000 unless progress was durably recorded per recipient. The event-driven cousin: two application events fire for one logical trigger, a signup event and a first-login event both wired to the welcome sequence, or an automation rebuilt in the ESP running alongside its predecessor. The recipient experiences all of these identically, as the same message twice.

The patterns that prevent them

Idempotency keys, done deterministically

Several providers accept an idempotency key per submission and silently deduplicate repeats within a retention window. The key must be deterministic from the logical send, order-1234-confirmation, campaign-567-recipient-890, so that any retry, from any worker, at any time, produces the same key. A random UUID generated at request time defeats the entire mechanism, because the retry generates a fresh one. Where your provider lacks the feature, the same idea implements client-side as a unique constraint on a send log.

1def send_once(send_key: str, build_message, submit):
2    # 1. Claim the key. Unique constraint = the lock.
3    try:
4        db.execute(
5            "INSERT INTO send_log (send_key, state) VALUES (%s, 'claimed')",
6            [send_key],
7        )
8    except UniqueViolation:
9        return  # already claimed: sent or in flight. Do nothing.
10
11    # 2. Submit and record the outcome.
12    try:
13        provider_id = submit(build_message())
14        db.execute(
15            "UPDATE send_log SET state='sent', provider_id=%s WHERE send_key=%s",
16            [provider_id, send_key],
17        )
18    except AmbiguousError:
19        # Timeout etc.: leave 'claimed'. A reconciliation job resolves it
20        # against provider events instead of blind retry.
21        raise

Campaigns as recipient-level state machines

The batch problem dissolves when a campaign send is a set of per-recipient rows moving through states (pending, claimed, sent, failed) rather than a loop over a list. A crashed run resumes by selecting pending rows; a rerun finds nothing pending and does nothing. This is exactly the shape ESP-side campaign engines already have internally, which is why platform-managed campaigns duplicate less often than homegrown loops over the recipients API.

Reconciliation against provider events

Ambiguous outcomes need an arbiter, and the provider's event stream is it. A reconciliation job that matches claimed send-log rows against delivery webhooks or the provider's message API resolves the timeout case correctly: if an accepted event exists for the key's message, mark it sent; if not after a grace period, release it for retry. Webhook consumers need their own idempotency (events also arrive at-least-once), which the same send-key discipline provides for free.

Guardrails for the human layer

Some duplicates are process failures wearing technical costumes: the automation migrated to a new tool while the old one kept running, the segment mailed by two teams in the same week, the re-import that resurrected suppressed recipients into an active flow. The countermeasures are organizational: an automation inventory with owners, a send calendar shared across teams, and frequency capping at the platform level as the final backstop, a rule that no recipient gets more than N marketing messages per day regardless of which system asked.

If a duplicate incident does ship, respond in proportion: for a transactional double, usually nothing, and for a full campaign duplicate, a short apology only if the duplicate was widespread and noticed. A third email apologizing for the second is sometimes the right call and often just triples the volume. What matters more is the retro: every duplicate has a mechanical cause in the list above, and each one fixed is a class of incident retired.