Background Work, Queues, Email, and Webhooks

Background work in a Next.js application is any useful action that should not block the page or API response that accepted the user’s request. Sending an email, calling a slow third-party API, generating a thumbnail, importing a CSV, and reacting to a webhook are common examples. The outcome for this chapter is concrete: an App Router application should accept a request quickly, record durable intent, process the work outside the request path, and expose enough state to retry, inspect, or compensate when something fails.

This belongs late in a full-stack Next.js course because it combines route handlers, server-side data access, environment configuration, cache updates, and deployment behavior. A queue does not make unreliable code reliable by itself. It gives you a place to store work, schedule attempts, separate user latency from worker latency, and make retries explicit.

How Background Work Runs

A normal HTTP request has a narrow lifetime: the client connects, Next.js runs a route handler, server action, or rendered route, and the response is returned. If that handler also sends three emails and waits on a payment provider, the user pays for every slow dependency. Worse, if the process is interrupted after the database write but before the email, the system may have no durable record that the email still needs to happen.

The usual queue shape has four parts. A producer validates input and writes a job. A queue stores jobs with status, payload, schedule time, attempt count, and sometimes priority. A worker claims available jobs and performs side effects. A result store records success, failure, or a domain event that the UI can read. In hosted Next.js, the worker is often a separate process, a managed queue consumer, a scheduled route, or a serverless function triggered by queue events. The important internal boundary is that the request records intent; the worker performs the slow or retryable side effect.

Webhooks reverse the direction. Instead of your app asking a provider whether something changed, the provider sends an HTTP request to your app. A webhook handler should verify the signature, deduplicate the event, store the event or derived job, and respond quickly. Heavy processing should happen after durable storage so the provider does not retry because your handler spent too long doing work.

Queue Record Anatomy

A practical job record needs more than a payload. The id identifies one job. The type selects the worker branch, such as send_welcome_email or sync_customer. The payload contains the smallest data needed to re-read current state, often ids rather than full objects. The status usually moves through queued, processing, succeeded, failed, or dead. runAfter supports delay and backoff. attempts and lastError make retries diagnosable.

const emailJob = {
  id: "job_1001",
  type: "send_welcome_email",
  payload: { userId: "user_42" },
  status: "queued",
  attempts: 0,
  maxAttempts: 3,
  runAfter: "2026-09-06T12:00:00.000Z"
};

console.log(`${emailJob.type}:${emailJob.status}:${emailJob.payload.userId}`);

This first example is intentionally small. The expected output is send_welcome_email:queued:user_42. Notice that the payload stores a user id, not the email body. The worker can read the latest user record, skip the email if the account was deleted, and avoid leaking unnecessary personal data into queue storage.

Producing Jobs From Next.js

In the App Router, a producer often lives in a server action or app/api/.../route.ts. The producer must validate the request, authorize the action, write the domain state, and enqueue the job in the same durable step when possible. With a relational database, this usually means one transaction that creates an order and inserts an outbox row. With a managed queue, you may write the database row first and include an idempotency key when sending to the queue.

function createSignupJobs(user) {
  if (!user || typeof user.id !== "string" || typeof user.email !== "string") {
    throw new TypeError("user id and email are required");
  }

  return [
    {
      type: "send_welcome_email",
      payload: { userId: user.id },
      idempotencyKey: `welcome:${user.id}`
    },
    {
      type: "notify_sales",
      payload: { userId: user.id, domain: user.email.split("@")[1] },
      idempotencyKey: `sales:${user.id}`
    }
  ];
}

console.log(JSON.stringify(createSignupJobs({ id: "user_42", email: "ada@example.com" })));

The deterministic output is an array containing two jobs with idempotency keys welcome:user_42 and sales:user_42. In a real route handler, the returned HTTP response should not claim that the email was sent. It should say the signup was accepted or created, while the job status page or audit log records whether each side effect finished.

Worker Processing and Retry

A worker repeatedly claims jobs whose runAfter is in the past. Claiming must be atomic: two workers must not send the same email because they read the same queued row at the same time. Databases solve this with row locks or conditional updates; managed queues solve it with visibility timeouts and acknowledgement. The worker should mark success only after the side effect succeeds, and it should retry only operations that are safe to repeat or protected by provider idempotency keys.

function nextRetryDelayMs(attempts) {
  const capped = Math.min(attempts, 5);
  return 1000 * 2 ** capped;
}

function recordFailure(job, message) {
  const attempts = job.attempts + 1;
  const dead = attempts >= job.maxAttempts;
  return {
    ...job,
    attempts,
    status: dead ? "dead" : "queued",
    lastError: message,
    runAfterMs: dead ? null : nextRetryDelayMs(attempts)
  };
}

console.log(JSON.stringify(recordFailure({ attempts: 1, maxAttempts: 3, status: "processing" }, "SMTP timeout")));

The expected output has attempts set to 2, status set to queued, lastError set to SMTP timeout, and runAfterMs set to 4000. This is exponential backoff. It prevents a temporary provider outage from turning into a tight failure loop that burns function invocations and trips rate limits.

Email and Webhook Details

Email jobs should store template names and entity ids rather than rendered HTML whenever possible. Rendering at worker time lets unsubscribe status, locale, and recipient address be checked immediately before send. The email provider call should include an idempotency key or a custom header tied to the job id. If the provider accepts the message but the worker crashes before marking the job succeeded, a retry may happen. Provider-side idempotency is the difference between a harmless replay and duplicate mail.

Webhook handlers need a different first step: verify that the request really came from the provider. Most providers send a timestamp and HMAC signature over the raw request body. In Next.js, read the raw text with await request.text() before parsing JSON, because changing whitespace or property order changes the signature input. After verification, insert the provider event id into a table with a unique constraint. If the insert conflicts, return success because the event was already seen.

const crypto = require("node:crypto");

function verifyWebhook(rawBody, signature, secret) {
  const expected = crypto.createHmac("sha256", secret).update(rawBody).digest("hex");
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

const body = JSON.stringify({ id: "evt_1", type: "invoice.paid" });
const sig = crypto.createHmac("sha256", "secret").update(body).digest("hex");
console.log(verifyWebhook(body, sig, "secret"));

This example prints true. In a route handler, a false result should return a refusal such as 401 and should not enqueue any work. The raw body and secret are part of the signature contract; parsing first and then stringifying again is a common cause of valid provider events being rejected.

Design Choices and Trade-offs

The simplest option is inline work inside the request. It is acceptable for tiny, fast, noncritical side effects during development, but it couples user latency to dependency latency and gives weak recovery. A database-backed queue is often a strong early production choice because it reuses transactions and makes admin inspection easy. Its limit is throughput: polling and locking can become expensive at high volume. A managed queue improves scale, delayed delivery, and worker fan-out, but it adds another service, local-development setup, and consistency boundaries between the database and queue.

For webhooks, synchronous processing is tempting because it is easy to reason about in one function. The trade-off is that providers retry on slow or failed responses, so long processing multiplies duplicate events. The safer pattern is verify, deduplicate, store, enqueue, and return a fast 2xx. For email, decide whether the product needs exactly-once user experience or merely at-least-once delivery with duplicate suppression. Most systems implement at-least-once jobs plus idempotent side effects because true exactly-once delivery across HTTP providers is not realistic.

Failure Modes and Troubleshooting

Duplicate emails. The symptom is users receiving the same message multiple times, often during provider timeouts. The usual cause is marking the job failed after the provider accepted the message, then retrying without an idempotency key. Diagnose by comparing job ids, provider message ids, timestamps, and worker logs. Correct it by sending a stable idempotency key to the provider and recording provider acceptance before deciding whether a retry is allowed.

Webhook events rejected even though the secret is correct. The symptom is repeated provider retries with signature failures. The cause is often verifying a parsed and reserialized body instead of the raw body, or using the wrong environment secret in preview versus production. Diagnose by logging a bounded event id, header presence, deployment environment, and signature algorithm, never the secret. Correct it by reading request.text(), verifying before JSON parsing, and separating preview and production webhook endpoints.

Jobs stuck in processing. The symptom is a queue depth that never falls and rows with old processing timestamps. The cause is a worker crash after claiming jobs but before acknowledging them. Diagnose with claim timestamps, worker deployment ids, and last heartbeat. Correct it with a visibility timeout or lease column: when the lease expires, another worker may claim the job. Keep the timeout longer than normal processing but shorter than your recovery objective.

Security, Performance, and Reliability

Queue payloads are durable data, so treat them like database rows. Do not store secrets, full credit card details, or unnecessary personal data. Scope worker credentials narrowly: an email worker does not need permission to modify billing records. Rate-limit public webhook endpoints and reject oversized bodies before expensive work. For performance, watch queue depth, oldest job age, processing latency, retry rate, and dead-letter count. For reliability, make every worker branch idempotent, give each job a maximum attempt count, and provide an operator path for replaying or cancelling dead jobs.

Hands-on Lab

Prerequisites: a local Next.js App Router project, Node.js, a test email provider sandbox or a console-based fake sender, and somewhere durable to store jobs. SQLite, Postgres, Redis, or a managed queue all work; for the lab, a database table is easiest to inspect.

  1. Create a jobs table with columns for id, type, payload JSON, status, attempts, max attempts, run-after time, and last error.
  2. Add a signup route or server action that creates a test user and inserts a send_welcome_email job with payload {"userId":"..."}.
  3. Return a response that says the signup was accepted, not that the email was sent.
  4. Create a worker script that claims one queued job, reads the user, renders the email, sends through the sandbox or logs the message, and marks the job succeeded.
  5. Force one failure by making the sender throw once. Verify that attempts increases, last error is stored, and run-after moves into the future.
  6. Add a webhook route that reads the raw body, verifies a test HMAC signature, inserts the provider event id with a unique constraint, and enqueues a follow-up job.

Verification should be evidence-based. After signup, query the jobs table and confirm one queued row exists. After the worker runs, confirm the row is succeeded and the sandbox contains one email. Replay the same webhook request twice and confirm the second request returns success but creates no second event row. Cleanup is to delete the test user, test jobs, provider sandbox messages, and webhook secret from local environment files. If you added a real provider endpoint, disable it or point it back to a development URL.

Assessment Exercises

  1. A checkout route writes an order and then sends a receipt inline. Redesign the flow so a process crash cannot lose the receipt intent, and state what the HTTP response should promise.
  2. A provider retries the same webhook event five times. Describe the database constraint and handler response that prevent duplicate downstream jobs while still stopping provider retries.
  3. An email provider sometimes times out after accepting a message. Explain why a normal retry can duplicate mail and how a job id or idempotency key changes the outcome.
  4. Your queue depth is low, but the oldest job age is high. List two plausible causes and the first diagnostic query or log field you would inspect.
  5. Choose between a database-backed queue and a managed queue for a product sending 200 emails per day plus occasional webhooks. Defend the simpler choice and name the signal that would justify changing it.

Summary

Background work in Next.js is about moving slow, retryable, or externally triggered effects out of the user request while preserving durable intent. Queues store that intent, workers claim and execute it, email jobs need idempotent sending, and webhook routes must verify, deduplicate, and return quickly. The practical design target is not magic exactly-once execution; it is observable at-least-once processing with idempotent side effects, bounded retries, and clear operator recovery.