Stateless Services Sessions and Horizontal Scaling

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

A stateless service is a service whose correctness does not depend on private memory left behind by a previous request. The process may keep caches, connection pools, and metrics in memory, but any user state needed to answer the next request lives somewhere shared: a signed token, a database, a distributed cache, or another durable service. This matters because horizontal scaling adds more identical service replicas behind a load balancer. If any replica can handle any request, the fleet can grow, shrink, restart, and fail over without asking users to begin again.

In this system design course, this lesson connects coordination and scale to an everyday application concern: sessions. A session says, in effect, this request belongs to an authenticated user and carries some continuity from earlier requests. The design question is where that continuity lives. Putting it in a single web server is simple, but it ties the user to that server. Externalizing it makes scaling and recovery easier, but introduces new trade-offs around latency, consistency, revocation, and operational dependency.

How Stateless Request Handling Works

Consider a fleet of three API replicas: A, B, and C. A load balancer receives client requests and chooses one replica using round-robin, least-connections, random choice, or a hash over request attributes. With a stateless service, replica A does not need to remember that it handled the previous request. Each request carries enough identity material, or enough lookup information, for whichever replica receives it to reconstruct the needed context.

The important boundary is not whether the process has memory. Every process has memory. The boundary is whether process-local memory is authoritative. A stateless handler can maintain an LRU cache of recently loaded users because a cache miss still produces the correct answer from the real session store. A stateful handler that stores shopping carts only in a Python dictionary inside one process cannot safely serve the next cart request from another replica.

There are three common session patterns. First, a signed client-side token stores claims such as user id, expiration time, and roles. The service verifies the signature on every request and trusts the claims if valid. Second, an opaque session id points to data in a shared store such as Redis, Memcached, or a database table. The service reads the session data by key. Third, sticky sessions configure the load balancer to keep sending a client to the same backend. Sticky sessions reduce application changes, but they weaken horizontal scaling because a backend failure or rebalance can strand process-local state.

API and Configuration Anatomy

A typical HTTP session flow has four pieces. The client sends a cookie or authorization header. The load balancer selects a backend. The backend validates the token or loads the session record. The backend performs the request and may issue a new token, update the shared session, or leave session state unchanged.

For cookie sessions, the cookie attributes are part of the design. HttpOnly prevents browser JavaScript from reading the cookie. Secure requires HTTPS. SameSite controls cross-site sending behavior. For token sessions, claims usually include sub for subject, exp for expiry, iat for issued-at time, and sometimes authorization claims. For server-side sessions, the stored value often contains user id, authentication strength, creation time, last activity time, and a version used for invalidation.

Load balancer configuration also shapes behavior. Plain round-robin assumes replicas are interchangeable. Least-connections can help when requests vary in duration. A hash policy can make traffic more stable, but should not be required for correctness. Sticky cookies bind a client to one backend and should be treated as an optimization or migration bridge, not as the session source of truth.

Example 1: Process Memory Breaks Under Round-Robin

The first example models the mistake. Login writes a session only into replica A. The next request goes to replica B, which has no record. The deterministic output shows why users see random logouts when traffic is spread across replicas.

class Replica:
    def __init__(self, name):
        self.name = name
        self.local_sessions = {}

    def login(self, user_id):
        session_id = f"s-{user_id}"
        self.local_sessions[session_id] = user_id
        return session_id

    def profile(self, session_id):
        user_id = self.local_sessions.get(session_id)
        return f"{self.name}: user {user_id}" if user_id else f"{self.name}: 401"

replicas = [Replica("A"), Replica("B")]
session = replicas[0].login("u123")
print(replicas[0].profile(session))
print(replicas[1].profile(session))

Expected output: replica A recognizes the session, while replica B returns 401. This is not a load balancer bug. It is a state placement bug. Horizontal scaling exposed that the session lived in the wrong place.

Example 2: Shared Session Store Makes Replicas Interchangeable

The next version moves session state to a shared dictionary standing in for Redis or a database. Both replicas read the same authoritative store. The application instances are still allowed to have process memory, but not as the only copy of required session data.

class SharedSessionStore:
    def __init__(self):
        self.records = {}

    def put(self, session_id, user_id):
        self.records[session_id] = user_id

    def get(self, session_id):
        return self.records.get(session_id)

class StatelessReplica:
    def __init__(self, name, store):
        self.name = name
        self.store = store

    def login(self, user_id):
        session_id = f"s-{user_id}"
        self.store.put(session_id, user_id)
        return session_id

    def profile(self, session_id):
        user_id = self.store.get(session_id)
        return f"{self.name}: user {user_id}" if user_id else f"{self.name}: 401"

store = SharedSessionStore()
replicas = [StatelessReplica("A", store), StatelessReplica("B", store)]
session = replicas[0].login("u123")
print(replicas[1].profile(session))

The expected output is B: user u123. Replica B did not handle login, but it can serve the profile request because the session source of truth is shared. In production, this design needs timeouts, replication, capacity planning, and careful handling of store outages, but it has the basic scaling property we want.

Example 3: Signed Token Removes the Lookup

With a signed token, the request carries the claims and the service verifies that they were produced by a trusted signer. This can reduce latency and remove the session-store dependency from ordinary reads. The trade-off is that revoking a token before expiry becomes harder unless the service checks a revocation list or uses short lifetimes.

import base64
import hashlib
import hmac
import json

SECRET = b"demo-secret"


def b64(data):
    return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")


def sign(payload):
    body = b64(json.dumps(payload, sort_keys=True).encode("utf-8"))
    mac = hmac.new(SECRET, body.encode("ascii"), hashlib.sha256).hexdigest()
    return f"{body}.{mac}"


def verify(token):
    body, mac = token.split(".")
    expected = hmac.new(SECRET, body.encode("ascii"), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(mac, expected):
        return None
    padded = body + "=" * (-len(body) % 4)
    return json.loads(base64.urlsafe_b64decode(padded))

token = sign({"sub": "u123", "role": "member"})
print(verify(token)["sub"])
print(verify(token + "x"))

The deterministic output is u123 followed by None. The tampered token fails verification. This pattern works well for short-lived access tokens and services that can tolerate claim changes taking effect at the next token refresh.

Design Choices and Trade-Offs

Client-side signed tokens favor fast, independent replicas. They are attractive for APIs because verification is CPU-local and avoids a network lookup. Their main weakness is control after issuance. If an administrator disables a user, existing tokens may keep working until expiration unless the system checks token version, revocation, or account status on each request.

Server-side sessions favor immediate control. Deleting a session record can log a user out right away, and updating the record changes what every replica sees. Their weakness is the added dependency on the session store. If the store slows down, every authenticated request may slow down. If the store becomes unavailable and the service has no fallback, the application effectively loses login state.

Sticky sessions favor migration speed. A legacy application that stores session data in memory may become horizontally deployed by configuring load balancer affinity. The hidden cost is uneven load and weaker failover. Some users may cluster on a hot replica, autoscaling may not redistribute existing sessions cleanly, and backend loss can still drop users.

Failure Modes and Troubleshooting

Symptom: users are logged out intermittently after adding replicas. Likely cause: session data is stored in process memory or the load balancer affinity changed. Diagnostic steps: compare failing requests by backend id, inspect whether the session id exists in a shared store, and reproduce by sending the same cookie to two replicas. Correction: move required session state into a shared store or signed token, then treat stickiness as optional.

Symptom: login works, but authenticated pages are slow across the whole fleet. Likely cause: the session store is saturated or geographically far from service replicas. Diagnostic steps: measure session lookup latency separately from request latency, inspect connection pool saturation, and compare cache hit rates by endpoint. Correction: right-size the store, use local connection pooling, reduce session payload size, or use short-lived signed access tokens with a refresh flow.

Symptom: a disabled account can still call APIs for several minutes. Likely cause: stateless signed tokens remain valid until expiration. Diagnostic steps: inspect token expiry, check whether handlers read current account status, and test with a token issued before deactivation. Correction: shorten token lifetimes, include a user token-version claim and compare it with authoritative account state, or maintain a revocation list for sensitive operations.

Security, Performance, and Reliability Implications

Security decisions are tightly coupled to session placement. A client-side token must be signed, must use a strong secret or asymmetric key, and must avoid sensitive data that the client should not see. A server-side session id must be random enough to resist guessing and should be rotated after privilege changes such as login or multi-factor authentication. Cookies carrying either form should normally be HttpOnly, Secure, and scoped narrowly.

Performance depends on what happens per request. Token verification spends CPU but avoids a network hop. Server-side sessions spend network latency and store capacity, but support fast revocation and small cookies. Reliability depends on whether session infrastructure is more available than individual app replicas. A shared store should have replication, timeouts, and clear behavior during partial failure. The application should fail closed for authentication when it cannot verify a session, while still exposing health endpoints that allow orchestration systems to make good decisions.

Hands-On Lab: Prove Replica Independence

Prerequisites: Python 3 and a shell. The lab uses a single Python file to simulate two replicas and a load balancer. No external services are required.

  1. Create a file named session_scaling_lab.py in a temporary directory.
  2. Paste the program below.
  3. Run python3 session_scaling_lab.py.
  4. Verify that local sessions fail on the second replica and shared sessions succeed on both replicas.
  5. Cleanup by deleting the temporary file when finished.
class Replica:
    def __init__(self, name, shared_store=None):
        self.name = name
        self.local = {}
        self.shared_store = shared_store

    def login_local(self, user_id):
        self.local["sid"] = user_id
        return "sid"

    def login_shared(self, user_id):
        self.shared_store["sid"] = user_id
        return "sid"

    def read_local(self, session_id):
        return self.local.get(session_id)

    def read_shared(self, session_id):
        return self.shared_store.get(session_id)

shared = {}
a = Replica("A", shared)
b = Replica("B", shared)

local_session = a.login_local("u123")
print("local on A", a.read_local(local_session))
print("local on B", b.read_local(local_session))

shared_session = a.login_shared("u123")
print("shared on A", a.read_shared(shared_session))
print("shared on B", b.read_shared(shared_session))

Verification: the local section prints u123 for A and None for B. The shared section prints u123 for both A and B. That output proves the design property: a horizontally scaled service can route a request to any replica only when required session state is available outside the individual process.

Assessment Exercises

  1. You operate an API with signed access tokens that expire after one hour. A user whose account was disabled can still call a payment endpoint. Propose a change that preserves most token performance benefits while making payment authorization react quickly to account disablement.
  2. A team wants to enable sticky sessions because Redis latency increased. Explain why stickiness may hide symptoms without removing the dependency, and list two measurements you would collect before changing the load balancer.
  3. Design a session record for a server-side web session. Include only fields that replicas need on authenticated requests, and explain which field supports expiration or invalidation.
  4. An autoscaling group doubles from four replicas to eight, but CPU remains uneven. Existing clients are bound by load balancer affinity. Explain the cause and a migration path toward more even balancing.

Summary

Stateless services scale horizontally because each replica can reconstruct request context without depending on private process history. For sessions, that means using signed tokens, shared session stores, or carefully limited stickiness during migration. Tokens optimize independence and latency, shared stores optimize control and revocation, and sticky sessions trade correctness pressure for operational fragility. The durable design rule is simple: any state required for the next request must travel with the request or be reachable from every replica through a shared authority.