Reliable Webhook Processing: Retries, Idempotency and Queues

Webhook retries idempotency is the pair of problems you solve the moment you stop treating an incoming event as a one-shot HTTP request. Senders retry. Networks drop responses after your handler already did the work. Your process restarts mid-deploy. The reliable pattern is always the same: verify the signature, persist the raw event, return 200 in milliseconds, and process it in a worker that deduplicates on the event ID and treats retries as normal traffic. Mailerflow sends signed real-time webhooks with retries and delivery logs, so you can inspect what was actually attempted when something looks wrong.

What reliable webhook processing actually requires

A webhook consumer has four jobs, and skipping any one of them produces the same symptom: duplicate emails, double-counted events, or events that vanish.

1. Acknowledge fast. The sender is holding a connection open. Do the minimum work needed to accept the event and return 200. Anything slower than a few hundred milliseconds invites timeouts, and a timeout looks like a failure to the sender — which triggers a retry.
2. Persist before processing. Write the raw payload to durable storage before you return. If you accept the event and then crash, an in-memory buffer loses it forever.
3. Make handlers idempotent. The same event may arrive twice — once because the sender retried, once because your own worker retried. Your side effects must produce the same result either way.
4. Treat retries as normal. Retries are not an error condition. They are the delivery mechanism working as designed.

The rest of this article builds that pipeline end to end in Node.js, with a Redis queue, a Postgres dedupe table, and tests that prove the second delivery is a no-op.

Not sure your current tool is the right fit? Compare the Best Email Marketing Platforms by price, features and free plans for your exact list size.

Prerequisites

• Node.js 20+ (the code uses the built-in node:test runner and native fetch).
• Redis 7+ for the queue, or Postgres 15+ if you prefer to keep everything in one datastore.
• Postgres 15+ for the dedupe and event table.
• A public HTTPS endpoint. For local work, a tunnel such as ngrok or cloudflared is enough. There is a full walkthrough of that setup in testing webhooks locally.
• A Mailerflow account with a webhook signing secret. Webhooks are signed, so grab the secret from the dashboard before you write the verifier.

You will also want a scoped API key for outbound calls. Mailerflow's REST API uses scoped keys, an OpenAPI spec, and idempotency keys, and the OpenAPI spec is the source of truth for endpoint and payload shapes.

Setup and authentication

Install the SDK and the two runtime dependencies:

npm init -y
npm install express ioredis pg @mailerflow/sdk

Keep credentials in the environment, never in code:

# .env
MAILERFLOW_API_KEY=mf_live_xxxxxxxxxxxxxxxx
MAILERFLOW_WEBHOOK_SECRET=whsec_xxxxxxxxxxxxxxxx
DATABASE_URL=postgres://localhost:5432/webhooks
REDIS_URL=redis://localhost:6379
PORT=3000

Initialize the client once and reuse it. A scoped key limits the blast radius if it leaks, so scope it to the transactional send endpoint only:

// client.js
import { Mailerflow } from '@mailerflow/sdk';

export const mailerflow = new Mailerflow({
  apiKey: process.env.MAILERFLOW_API_KEY,
});

Receiving the webhook: acknowledge fast, then queue

The receiver does exactly three things: verify, persist, enqueue. No database writes to your business tables, no API calls, no email sends.

// receiver.js
import express from 'express';
import crypto from 'node:crypto';
import Redis from 'ioredis';

const app = express();
const redis = new Redis(process.env.REDIS_URL);
const SECRET = process.env.MAILERFLOW_WEBHOOK_SECRET;

// Raw body is required for signature verification.
app.post(
  '/webhooks/mailerflow',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const signature = req.get('x-mailerflow-signature');
    if (!verify(req.body, signature)) {
      return res.status(401).send('invalid signature');
    }

    // Durable handoff. If this fails, return 500 so the sender retries.
    await redis.lpush('events:incoming', req.body.toString('utf8'));
    return res.status(200).send('ok');
  }
);

function verify(rawBody, signature) {
  if (!signature) return false;
  const expected = crypto
    .createHmac('sha256', SECRET)
    .update(rawBody)
    .digest('hex');
  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(signature, 'utf8');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

app.listen(process.env.PORT ?? 3000);

Verifying signatures correctly

Two mistakes break HMAC verification more often than anything else. First, parsing the JSON before verifying: re-serializing changes whitespace and key order, so the bytes no longer match. That is why the route uses express.raw() and never express.json(). Second, comparing with ===, which leaks timing information. Use crypto.timingSafeEqual on equal-length buffers, as above.

A quick test that a bad signature is rejected:

// test/signature.test.js
import test from 'node:test';
import assert from 'node:assert/strict';

const endpoint = 'http://localhost:3000/webhooks/mailerflow';

test('rejects a bad signature', async () => {
  const res = await fetch(endpoint, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      'x-mailerflow-signature': 'deadbeef',
    },
    body: JSON.stringify({ id: 'evt_1', type: 'email.delivered' }),
  });
  assert.equal(res.status, 401);
});

Designing the queue and worker

The worker pulls from Redis, processes the event, and only removes it on success. Failures go back on the queue with exponential backoff and jitter, and after a capped number of attempts the event moves to a dead-letter list.

// worker.js
import Redis from 'ioredis';
import { Pool } from 'pg';
import { processEvent } from './process-event.js';

const redis = new Redis(process.env.REDIS_URL);
const db = new Pool({ connectionString: process.env.DATABASE_URL });
const MAX_ATTEMPTS = 8;

async function loop() {
  while (true) {
    const [, raw] = await redis.brpop('events:incoming', 5);
    if (!raw) continue;

    const event = JSON.parse(raw);
    const attempts = Number(event.__attempts ?? 0) + 1;

    try {
      await processEvent(db, event);
    } catch (err) {
      if (attempts >= MAX_ATTEMPTS) {
        await redis.lpush('events:dead', JSON.stringify({ event, err: String(err) }));
        continue;
      }
      const delay = Math.min(2 ** attempts * 250, 60_000);
      const jitter = Math.random() * 250;
      await new Promise((r) => setTimeout(r, delay + jitter));
      await redis.lpush(
        'events:incoming',
        JSON.stringify({ ...event, __attempts: attempts })
      );
    }
  }
}

loop();

A durable queue beats an in-memory array for one reason: deploys, crashes and restarts. An array dies with the process and takes every unprocessed event with it.

Idempotency: deduplicating events safely

This is where webhook retries idempotency stops being theoretical. The sender may deliver the same event more than once, and your worker may retry on top of that. Handle it in two layers.

Layer one: dedupe on the event ID before doing any work. A unique constraint on the event ID means the second insert fails silently and you skip processing entirely.

Layer two: make the side effects themselves idempotent. Even with a dedupe table, an upsert is safer than an insert, and a conditional update (WHERE processed_at IS NULL) is safer than an unconditional one. If your dedupe row is ever lost, the side effect still converges.

The mirror image of this applies outbound: Mailerflow's REST API supports idempotency keys, so if your worker sends a transactional email in response to an event and the call times out, you can retry with the same key and avoid a duplicate send.

A dedupe table that survives retries

-- migrations/001_events.sql
CREATE TABLE webhook_events (
  event_id     text PRIMARY KEY,
  type         text NOT NULL,
  payload      jsonb NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz,
  attempts     integer NOT NULL DEFAULT 0
);

CREATE INDEX webhook_events_unprocessed
  ON webhook_events (received_at)
  WHERE processed_at IS NULL;

Claim the event, and treat a conflict as "already handled":

-- claim.sql
INSERT INTO webhook_events (event_id, type, payload)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

If RETURNING gives you no row, another delivery already claimed it — return early. On success, mark it done:

-- complete.sql
UPDATE webhook_events
SET processed_at = now(), attempts = attempts + 1
WHERE event_id = $1 AND processed_at IS NULL;

// process-event.js
export async function processEvent(db, event) {
  const claim = await db.query(
    'INSERT INTO webhook_events (event_id, type, payload) VALUES ($1,$2,$3) ON CONFLICT (event_id) DO NOTHING RETURNING event_id',
    [event.id, event.type, event]
  );
  if (claim.rowCount === 0) return; // duplicate delivery

  // Idempotent side effect: upsert, never blind insert.
  await db.query(
    `INSERT INTO contact_events (contact_id, kind, occurred_at)
     VALUES ($1, $2, $3)
     ON CONFLICT (contact_id, kind, occurred_at) DO NOTHING`,
    [event.data.contact_id, event.type, event.created_at]
  );

  await db.query(
    'UPDATE webhook_events SET processed_at = now(), attempts = attempts + 1 WHERE event_id = $1 AND processed_at IS NULL',
    [event.id]
  );
}

Storing the raw payload pays off later: you can replay an event after fixing a bug instead of asking the sender to redeliver it.

Handling errors, retries and poison messages

Sort failures into three buckets, because they need different responses.

The retry wrapper above handles the first and third. For the second, validate the event type against a known list before enqueueing and dead-letter anything unrecognized rather than looping on it.

Remember that the sender retries too. Your handler must be safe under both your retries and theirs, which is exactly why the dedupe layer sits before the side effect. When events look like they never arrived, Mailerflow's delivery logs show what the sender attempted — useful when the problem is on your side of the wire. There is a longer checklist in debugging webhook deliveries.

Testing and going to production

Prove idempotency with a replay: post the same fixture twice and assert only one row exists.

// test/idempotency.test.js
import test from 'node:test';
import assert from 'node:assert/strict';
import crypto from 'node:crypto';
import { Pool } from 'pg';

const db = new Pool({ connectionString: process.env.DATABASE_URL });
const SECRET = process.env.MAILERFLOW_WEBHOOK_SECRET;

function sign(body) {
  return crypto.createHmac('sha256', SECRET).update(body).digest('hex');
}

test('duplicate delivery creates one row', async () => {
  const body = Buffer.from(
    JSON.stringify({
      id: 'evt_test_1',
      type: 'email.delivered',
      created_at: '2026-01-01T00:00:00Z',
      data: { contact_id: 'c_1' },
    })
  );

  for (let i = 0; i < 2; i++) {
    await fetch('http://localhost:3000/webhooks/mailerflow', {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-mailerflow-signature': sign(body),
      },
      body,
    });
  }

  const { rows } = await db.query(
    'SELECT count(*)::int AS n FROM webhook_events WHERE event_id = $1',
    ['evt_test_1']
  );
  assert.equal(rows[0].n, 1);
});

Before you ship, confirm: HTTPS only, signature verification enabled on every webhook route, queue depth monitored, dead-letter alerts wired to a human, and a replay path for events you need to reprocess. Also check that your auth middleware is not applied to the webhook route — a 401 from your own app looks exactly like a signature failure.

Complete example

Putting it together, the pieces are small and independent:

• receiver.js — verifies the signature, pushes the raw body to events:incoming, returns 200.
• worker.js — pulls, retries with backoff, dead-letters after eight attempts.
• process-event.js — claims the event ID, performs an idempotent upsert, marks it processed.
• migrations/001_events.sql — the table and partial index.
• test/ — signature rejection and duplicate-delivery tests.

The worker can also make outbound calls. If an event should trigger a transactional email, the Mailerflow JS SDK sends it on a high-priority lane with templates and variables, and the OpenAPI spec documents the exact request shape. Use an idempotency key on that call so a worker retry cannot produce a second email. The developer docs cover scoped keys and the SDK surface.

Troubleshooting

• Signature mismatch. You parsed the body before verifying, used the wrong secret, or the sender's clock skew broke a timestamp check. Read the raw bytes and compare with timingSafeEqual.
• Duplicate processing. Missing unique constraint on the event ID, or a non-idempotent side effect such as a blind INSERT. Add the constraint and switch to ON CONFLICT DO NOTHING.
• Queue backlog. The worker crashed or a downstream dependency got slow. Check the dead-letter list and add queue-depth alerting.
• Events lost after a deploy. You used an in-memory queue. Move to Redis or a Postgres table.
• 401/403 from your own endpoint. Auth middleware is applied to the webhook route. Exclude it explicitly.

FAQ

How many times will a sender retry? It depends on the provider — check its documentation. Mailerflow retries webhook deliveries and exposes delivery logs so you can see what was attempted.

Should I return 200 before processing? Yes, once the event is durably persisted. Acknowledge first, process second.

Do I need a queue at low volume? Not necessarily. A durable table can be enough, but the pattern is identical: persist, dedupe, process.

How long should I keep dedupe records? Longer than the sender's retry window. Seven to thirty days is a common range; keep the raw payload if you want replay.

Can I use idempotency keys on outbound calls? Yes. Mailerflow's REST API supports idempotency keys, which prevents duplicate sends when your worker retries.

Send your first transactional email with the Mailerflow REST API — scoped keys, idempotency and an OpenAPI spec. Start free.