Safe Deployments Feature Flags Configuration and Rollback

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

Safe deployment is the practice of separating shipping code from exposing behavior. A binary can be deployed to every instance while a feature flag, dynamic configuration value, or rollout rule decides who actually receives the new path. Rollback then becomes a control-plane operation: disable the flag, lower the percentage, restore the previous configuration, or route traffic back to the earlier version. In system design, this matters because most incidents are not caused by the act of copying files onto servers; they are caused by newly activated behavior meeting real traffic, real data, and real dependencies.

The outcome of this lesson is the ability to design a release path where a risky checkout rewrite, recommendation model, cache strategy, or schema-dependent workflow can move from developer-only traffic to internal users, a small customer cohort, a regional rollout, and finally full exposure. The design must make each step measurable and reversible. That requires flag evaluation rules, configuration distribution, version compatibility, automated rollback checks, and operational ownership.

How the Mechanism Works

A feature flag system has three main parts. The management plane stores flag definitions, audit history, owners, descriptions, and targeting rules. The distribution plane delivers those definitions to services through polling, streaming, sidecars, SDK caches, or configuration snapshots. The evaluation plane runs inside the application or at an edge proxy and decides which variation a request receives.

A typical evaluation request includes a flag key, a subject, and attributes. The subject might be a user id, account id, device id, tenant id, or request id. Attributes might include country, plan, employee status, app version, region, or a compliance segment. The evaluator checks rules in order: hard off, explicit allow list, explicit deny list, segment match, percentage rollout, and default value. Rule order is important because a deny rule for a blocked tenant should beat a broad percentage rollout.

Percentage rollout usually relies on deterministic hashing, not random selection per request. The evaluator hashes the subject id plus a salt, maps the result into buckets such as 0 through 99, and enables the feature when the bucket is below the configured percentage. Determinism keeps a user in the same experience across requests. The salt prevents every 10 percent rollout from selecting the same users across unrelated features.

Dynamic configuration uses the same distribution ideas but returns typed values instead of booleans: limits, timeouts, queue weights, model names, retry counts, cache TTLs, or endpoint names. Rollback for configuration means restoring the last known-good value. Because configuration can alter resource use immediately, production systems often validate type, range, and dependency constraints before accepting a change.

Flag and Configuration Anatomy

A useful flag definition includes a stable key, type, default value, variations, targeting rules, owner, expiry date, and change audit. Defaults should be conservative. If the flag service is unreachable, a payment service should not suddenly activate an experimental charge path. Many teams treat flags as production controls and require code review or approval for changes to high-risk flags.

{
  "flags": {
    "checkout_v2": {
      "enabled": true,
      "default": false,
      "rules": [
        { "attribute": "internal_user", "equals": true, "value": true },
        { "attribute": "country", "equals": "US", "value": true },
        { "percentage": 5, "salt": "checkout_v2", "value": true }
      ]
    }
  }
}

This fragment shows a boolean flag named checkout_v2. Internal users receive the feature first. US traffic receives it next. Finally, 5 percent of remaining subjects receive it based on a salted percentage rule. If no rule matches, the default is false. In a real control plane, each rule would also carry an id and audit metadata so an operator can see exactly who changed the rollout and when.

Example 1: Internal-Only Release

The first progressive example is a checkout UI rewrite. The code is deployed everywhere, but only employees with internal_user=true receive the new page. Expected behavior is simple: employee sessions render checkout version 2, while customer sessions continue to render version 1. This catches template errors, missing translations, and obvious API contract mistakes without exposing paying customers.

The main design choice is where to evaluate the flag. Evaluating in the frontend can hide or show UI affordances quickly, but the backend must still evaluate authoritative behavior such as discounts, tax calculation, and payment submission. A safe design evaluates presentation flags near the UI and business behavior flags in the service that owns the operation.

Example 2: Stable Percentage Rollout

The next stage exposes the feature to a deterministic slice of real users. The important property is stickiness: a user should not move between old and new checkout on every page refresh.

import hashlib


def flag_enabled(user_id: str, percentage: int, salt: str) -> bool:
    key = f"{salt}:{user_id}".encode("utf-8")
    bucket = int(hashlib.sha256(key).hexdigest()[:8], 16) % 100
    return bucket < percentage


users = ["u-101", "u-102", "u-103", "u-104", "u-105"]
for user in users:
    print(user, flag_enabled(user, 40, "checkout_v2"))

With the fixed user list and salt above, the deterministic output is:

u-101 True
u-102 False
u-103 False
u-104 False
u-105 True

The exact users in the rollout are less important than the stability. Increasing the percentage from 40 to 50 adds buckets; it should not reshuffle users already included. If a release compares conversion rate, latency, or error rate between old and new paths, stable assignment is necessary for meaningful measurement.

Example 3: Metric-Gated Rollback

The third example connects rollout state to rollback criteria. A canary might be allowed to continue only when its error rate and latency stay near baseline.

from dataclasses import dataclass


@dataclass
class Metrics:
    requests: int
    errors: int
    p95_ms: int


def should_rollback(baseline: Metrics, canary: Metrics) -> bool:
    baseline_error = baseline.errors / baseline.requests
    canary_error = canary.errors / canary.requests
    error_regression = canary_error > baseline_error + 0.02
    latency_regression = canary.p95_ms > baseline.p95_ms * 1.25
    return error_regression or latency_regression


baseline = Metrics(requests=10000, errors=80, p95_ms=180)
canary = Metrics(requests=1200, errors=55, p95_ms=240)
print(should_rollback(baseline, canary))

The expected output is:

True

The canary has a higher error rate and a p95 latency regression, so the release controller should rollback. In a production controller, this decision would be made over a time window with minimum sample sizes, protected metrics, and suppression rules for known unrelated incidents. Without those guards, a tiny cohort or a regional outage can trigger misleading rollbacks.

Design Choices and Trade-offs

In-process SDK evaluation is fast and resilient because services can cache the latest flag snapshot locally. The trade-off is consistency: different instances may observe changes a few seconds apart. Centralized evaluation through a network service gives uniform decisions and simpler language support, but introduces latency and a dependency on the evaluator. Edge evaluation works well for routing, experiments, and static content decisions, but it cannot replace backend authorization or business validation.

Short-lived release flags reduce code complexity. Long-lived permission flags, tenant entitlements, and kill switches are legitimate, but they need stronger naming and ownership because they become part of the product model. A common failure is leaving old release flags in code until both branches must be maintained forever. Good flag hygiene includes expiry dates, dashboards of stale flags, and removal work as part of the definition of done.

Configuration rollback has different trade-offs from binary rollback. Lowering a concurrency limit, disabling a model, or restoring a timeout can take effect faster than redeploying. However, dynamic configuration can also break every instance at once. Guardrails should include schema validation, staged propagation, dry-run evaluation, maximum change limits, and automatic reversion when health checks fail.

Failure Modes and Troubleshooting

Symptom: only some pods show the rollback after the flag is disabled. Cause: local SDK caches have different polling intervals or cannot reach the distribution endpoint. Diagnostics: inspect each instance’s flag snapshot version, last refresh time, and network errors to the flag service. Correction: shorten emergency refresh intervals, support a forced refresh endpoint, and keep conservative fallback values for critical flags.

Symptom: a 10 percent rollout affects a much larger share of requests. Cause: hashing is based on request id instead of user or account id, so high-volume users are sampled repeatedly and inconsistently. Diagnostics: log the evaluation key, bucket, and variation for a small sample; compare unique subjects with request counts. Correction: choose the subject that matches the risk boundary, usually account id for tenant-impacting changes and user id for personal UI changes.

Symptom: rollback does not restore previous behavior. Cause: the new code performed a one-way side effect such as writing incompatible data or migrating state eagerly. Diagnostics: compare writes made by flagged and unflagged paths, inspect schema compatibility, and replay a request against the old path in staging with copied data. Correction: design expand-and-contract migrations, dual-read or dual-write carefully, and keep rollback paths compatible with data created during the rollout.

Security, Performance, and Reliability Implications

Feature flags are not access control by themselves. A frontend flag can hide a button, but the backend must enforce entitlement and authorization. Flag payloads may reveal planned features, customer segmentation, or operational kill switches, so clients should receive only the flags they need. Administrative changes should be authenticated, authorized, audited, and sometimes approved by a second operator.

Performance depends on evaluation location and payload size. Local evaluation is usually a map lookup plus rule evaluation, but complex segments and large allow lists can become expensive. Systems commonly precompute segments, compress snapshots, and evaluate hot flags without remote calls. Reliability depends on clear fallback semantics: every flag should define what happens when configuration is missing, stale, or invalid.

Hands-on Lab: Build a Rollbackable Flag Plan

Prerequisites: a service with two code paths for one behavior, a place to store flag definitions, access to service metrics, and a staging environment. The behavior can be as small as choosing between two recommendation functions.

  1. Create a flag named recommendations_v2 with default false, an internal-user rule, and a 0 percent rollout rule.
  2. Deploy code that contains both the old and new paths. Log the flag key, variation, subject id hash bucket, and code path for sampled requests.
  3. Enable the internal-user rule in staging. Verify that internal subjects use version 2 and ordinary test users use version 1.
  4. Move to production at 1 percent of account ids. Watch request rate, error rate, p95 latency, dependency calls, and business-specific output quality.
  5. Increase to 5 percent only after the first window is clean. Record the exact flag version and metric window used for the decision.
  6. Practice rollback by restoring default false and clearing percentage rules.

A command-line rollback should be scripted before the incident, not invented during it:

#!/usr/bin/env bash
set -euo pipefail

payload='{"default": false, "rules": []}'
curl -fsS -X PATCH "$FLAG_API/flags/checkout_v2" -H "Authorization: Bearer $FLAG_TOKEN" -H "Content-Type: application/json" -d "$payload"
curl -fsS "$FLAG_API/flags/checkout_v2"

Verification: after rollback, new sampled logs should show recommendations_v2=false, the new dependency should stop receiving traffic, and customer-facing metrics should return toward baseline. Cleanup: when version 2 is fully launched and stable, remove version 1 code and delete the temporary release flag. If the flag becomes a permanent entitlement, rename and document it as a product permission rather than a release toggle.

Assessment Exercises

  1. A feature writes a new column that old code ignores, but the new code also stops writing an old column. Explain why a flag rollback may fail and design a safer migration sequence.
  2. You are rolling out a latency-sensitive search ranking change. Choose the evaluation location, rollout subject, and fallback value. Defend each choice.
  3. A dashboard reports that 5 percent of users are enabled, but 30 percent of traffic hits the new dependency. List two likely causes and the logs you would inspect.
  4. Design an automatic rollback rule for a payment authorization change. Include metrics, minimum sample size, and one guard against false positives.
  5. Identify which flags in a service should be temporary release flags and which should be permanent operational controls.

Summary

Safe deployment with feature flags is a control system for changing production behavior gradually. The core mechanics are deterministic targeting, typed configuration, local or centralized evaluation, staged rollout, and explicit rollback. The hard design work is choosing the right subject, preserving compatibility across old and new paths, validating configuration before it spreads, and connecting rollout decisions to metrics that actually represent user harm. In a system-design interview or production design review, a strong answer treats flags as reliable infrastructure with ownership, audit, defaults, and cleanup, not as scattered if statements.