DNS CDNs Load Balancers and Reverse Proxies

Purpose and outcome

Most large systems do not send a user directly to one application server. A request usually passes through DNS, sometimes a CDN edge, then a load balancer, then a reverse proxy, and finally an application process. Each layer answers a different routing question. DNS answers, where should the client connect first? A CDN answers, can an edge location satisfy this request near the user? A load balancer answers, which healthy backend should receive this connection or request? A reverse proxy answers, how should this HTTP request be normalized, protected, routed, cached, or upgraded before it reaches the service?

In system design interviews and real architecture reviews, these layers matter because they control latency, availability, blast radius, deploy safety, and cost. This lesson teaches the mechanics instead of treating them as boxes in a diagram. By the end, you should be able to explain the path of a request, choose DNS and cache policies, compare L4 and L7 balancing, recognize proxy failure symptoms, and build a small local reverse-proxy lab.

How the request path works

Start with a browser requesting https://www.example.com/catalog. The browser checks local DNS cache, then asks a recursive resolver. The resolver walks or queries cached records from authoritative name servers and returns records such as A, AAAA, CNAME, or service-specific records. A TTL tells resolvers how long they may cache the answer. Low TTLs make traffic shifts faster but increase query volume and do not force every resolver to expire immediately. High TTLs reduce DNS load but slow failover and migrations.

If the name points to a CDN, the answer usually directs the client to an edge endpoint selected by geography, network proximity, or provider routing policy. The CDN terminates TLS or passes it through depending on configuration, checks its cache key, and decides whether the object is a hit, stale-but-servable, or miss. A cache key often includes scheme, host, path, selected query parameters, selected headers, and sometimes cookies. If the edge has a fresh object, it returns it without contacting origin. On a miss, it sends an origin request to a load balancer or reverse proxy.

A load balancer may operate at layer 4 or layer 7. An L4 balancer uses connection-level information such as source IP, destination port, TCP state, and sometimes TLS SNI without parsing full HTTP. It is fast and protocol-neutral. An L7 balancer parses HTTP and can route /api to one pool, /images to another, or canary traffic to a new version. Both maintain health checks. A backend that fails health checks is removed from rotation until it recovers.

A reverse proxy is often the first server controlled by the application team. It may terminate TLS, add X-Forwarded-For or Forwarded headers, enforce body-size limits, compress responses, serve static files, route by path, retry idempotent upstream requests, and protect backends from slow clients. NGINX, Envoy, HAProxy in HTTP mode, Apache httpd, and cloud ingress controllers can all act as reverse proxies.

Configuration anatomy

DNS configuration has records, TTLs, and routing policy. An A record maps a name to IPv4, AAAA maps to IPv6, CNAME aliases one name to another, MX handles mail, and TXT carries verification and policy data. For web traffic, apex domains often need provider-specific alias records because standard CNAME records are not valid at the zone apex when other records must coexist.

CDN configuration has origins, cache behavior, cache key rules, invalidation or purge controls, TLS certificates, compression, and request policies. The hardest decision is usually what not to cache. Static assets with content hashes can use long TTLs. Personalized pages should normally bypass cache or vary on the exact identity-bearing inputs. APIs can be cached safely only when method, authorization, query parameters, and freshness expectations are explicit.

Load balancer configuration has listeners, target pools, algorithms, health checks, connection draining, and sometimes session affinity. Common algorithms include round robin, least connections, weighted round robin, consistent hashing, and random with two choices. Health checks must test a dependency level appropriate for the service. A TCP check proves only that a port is open. An HTTP /healthz check can prove the process can answer. A deeper check can detect database failure, but if every instance depends on the same database, a deep check can also remove the entire fleet at once.

Reverse proxy configuration has server names, TLS settings, route matches, upstream pools, header forwarding, timeouts, buffering, limits, and retry rules. Timeouts are part of system behavior. A proxy with a 30 second upstream timeout in front of a service that legitimately runs 90 second exports will create false failures. A proxy with unlimited request bodies can become a memory, disk, or bandwidth liability.

Worked example 1: DNS cutover

Suppose shop.example.com currently points to an old load balancer and must move to a new one. With a 24 hour TTL, many clients may keep using the old address for most of a day. A careful migration lowers the TTL ahead of the cutover, waits at least the old TTL, changes the record, watches traffic, then raises the TTL after stability. The deterministic behavior is that resolvers respecting the original 86400 second TTL are allowed to hold the old answer until it expires; lowering the TTL at the same moment as the record change does not recall already cached answers.

dig +short shop.example.test A
# 203.0.113.10

dig shop.example.test A +noall +answer
# shop.example.test. 300 IN A 203.0.113.10

In this sample, the answer says clients can cache 203.0.113.10 for 300 seconds. If the address changes to 203.0.113.20, well-behaved recursive resolvers should stop returning the old address after the remaining TTL reaches zero.

Worked example 2: CDN cache behavior

Now put a CDN in front of product images. Files such as /assets/shoe.8f3a1.png are immutable because the hash changes when the content changes. They can use a long edge TTL and a browser cache header. The expected behavior is a miss on the first request at an edge location and hits afterward until expiration or purge. For /cart, caching by path alone would leak one user’s cart to another user, so the route must bypass cache or vary on private identity data in a controlled way.

A useful cache policy separates static assets, public API responses, and private pages. Static assets get long TTLs. Public catalog responses might get a short CDN TTL with stale serving during origin errors. Private account pages should have Cache-Control: private, no-store or an equivalent edge bypass rule. The trade-off is direct: more caching lowers latency and origin load, but each extra cached surface needs a correctness rule for freshness and personalization.

Worked example 3: Reverse proxy path routing

Consider one domain serving a web app and an API. The proxy can route /api/ to application servers and all other paths to a frontend server. It can also preserve the original host and client address so application logs and redirects remain correct.

server {
    listen 8080;
    server_name example.test;

    location /api/ {
        proxy_pass http://api_upstream;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 2s;
        proxy_read_timeout 15s;
    }

    location / {
        proxy_pass http://frontend_upstream;
    }
}

The expected behavior is deterministic by path: /api/orders goes to api_upstream; /pricing goes to frontend_upstream. If proxy_pass is written with a different trailing slash form, path rewriting can change, so proxy path tests should be part of deployment verification.

Worked example 4: Load balancing and draining

A rolling deploy with three backend instances should avoid sending new traffic to an instance that is shutting down. Connection draining marks the instance unavailable for new requests while allowing existing requests to finish until a deadline. Without draining, users may see reset connections during deploys. With draining set too long, deploys stall and capacity remains tied up by old instances.

backend api_pool
    balance roundrobin
    option httpchk GET /healthz
    server api1 10.0.1.11:8080 check weight 100
    server api2 10.0.1.12:8080 check weight 100
    server api3 10.0.1.13:8080 check weight 50

This fragment gives api3 half the scheduling weight of the other two servers. If all are healthy and request costs are similar, api3 should receive about half as many new requests as api1 or api2. Real distributions vary with connection reuse, long requests, and keep-alive behavior.

Design choices and trade-offs

Use DNS for coarse routing and disaster recovery, not per-request decisions. DNS has caches outside your control, so it is poor for instant failover. Use CDNs for static or safely cacheable dynamic content, global latency reduction, TLS edge termination, and origin shielding. Avoid using a CDN cache rule you cannot explain in terms of the exact cache key and freshness policy.

Choose L4 balancing when you need low overhead, many protocols, or simple TCP pass-through. Choose L7 balancing when HTTP routing, header decisions, redirects, authentication hooks, or request-aware metrics matter. Use weighted routing for migrations and canaries, but remember that weights apply to balancing decisions, not necessarily exact user percentages when clients reuse connections.

Keep reverse proxy rules boring and explicit. Centralizing TLS, compression, and request limits is useful, but a proxy can also hide upstream errors, retry unsafe methods, or create confusing path rewrites. Timeouts should form a descending budget from client to edge to proxy to service, with enough room for the service to fail quickly and return a meaningful error.

Failure modes and troubleshooting

Symptom: some users reach the old site after migration. Cause: DNS answers are still cached under the old TTL or a recursive resolver is serving stale data. Diagnose: query multiple resolvers, check authoritative answers, compare TTLs, and inspect access logs on old and new load balancers. Correct: keep the old endpoint serving or redirecting until traffic drains, lower TTL before future migrations, and avoid deleting old infrastructure immediately after a DNS change.

Symptom: users intermittently receive another user’s content. Cause: CDN cache key ignores cookies, authorization, or query parameters that change the response. Diagnose: inspect response headers, CDN cache status, cache key policy, and whether private responses include cacheable headers. Correct: bypass cache for private routes, include required vary dimensions only when safe, purge contaminated objects, and add tests for authenticated cache behavior.

Symptom: sudden 502 or 503 responses from the proxy. Cause: upstream instances are unhealthy, health checks are wrong, connection limits are exhausted, or the proxy cannot resolve the upstream name. Diagnose: check proxy error logs, upstream health status, DNS resolution from the proxy host, and backend listen ports. Correct: fix the health endpoint, restore backend capacity, correct upstream names, and tune connection pools or worker limits.

Symptom: uploads fail at a consistent size. Cause: reverse proxy body limit is lower than the application limit. Diagnose: compare HTTP status codes, proxy logs, and configured request size limits at CDN, load balancer, proxy, and app layers. Correct: set aligned limits and return a clear 413 response when the file is too large.

Security, performance, and reliability implications

TLS termination location affects certificate management and which layers can inspect HTTP. If TLS terminates at the CDN, the CDN can cache and filter HTTP, but the origin connection must still be protected. If TLS passes through to the service, intermediaries lose HTTP visibility. Client IP handling also matters. Applications should trust forwarding headers only from known proxies; otherwise a client can spoof X-Forwarded-For.

Performance gains come from fewer round trips, edge caching, connection reuse, compression, and keeping slow clients away from application workers. Reliability gains come from health checks, multi-zone target pools, origin shielding, retries for safe methods, and graceful draining. The danger is retry amplification: if every layer retries a failing request, one user action can become many backend attempts. Limit retries, add jitter, and retry only operations that are safe or explicitly idempotent.

Hands-on lab: local reverse proxy

Prerequisites: Docker or a local NGINX installation, two simple HTTP backends listening on different ports, and a terminal. The goal is to route one hostname by path and verify headers and failure behavior.

1. Start backend A on port 9001 returning the text frontend.
2. Start backend B on port 9002 returning the text api.
3. Create an NGINX server listening on port 8080. Route /api/ to port 9002 and / to port 9001. Add Host, X-Forwarded-For, and X-Forwarded-Proto headers.
4. Run curl http://localhost:8080/. Verification: the response body should be frontend.
5. Run curl http://localhost:8080/api/status. Verification: the response body should be api.
6. Stop backend B and repeat the API request. Verification: the proxy should return an upstream failure such as 502, and the proxy error log should mention connection refusal or upstream unavailability.
7. Restart backend B and verify that /api/status recovers without changing the client URL.

Cleanup: stop the backend processes and remove the temporary NGINX configuration or Docker containers. If you changed a system NGINX config, restore the previous file and reload NGINX only after testing the config syntax.

Assessment exercises

1. A team lowered a DNS TTL from 86400 to 60 at 10:00 and changed the record at 10:05. Explain why some clients still use the old endpoint at 18:00 and design a safer migration schedule.
2. An authenticated page is accidentally cached at the CDN. List the cache key and response header evidence you would collect before changing configuration.
3. Choose L4 or L7 load balancing for a gRPC API that needs path-based canaries and explain the operational cost of your choice.
4. A reverse proxy retries POST requests after a 3 second upstream timeout. Describe one data corruption risk and one safer alternative.
5. Design health checks for a checkout service whose database sometimes slows down. Explain what your check should and should not test.

Summary

DNS, CDNs, load balancers, and reverse proxies are not interchangeable routing boxes. DNS gives clients an entry point under cache constraints. CDNs move cacheable work to the edge and protect origins when cache keys are correct. Load balancers distribute traffic across healthy targets at connection or request level. Reverse proxies shape HTTP behavior close to the application. Good system design comes from assigning each layer a clear job, testing the request path, and understanding how failures appear at the boundaries between layers.