Threat Modeling Abuse Prevention and Secure Architecture Review

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

Threat modeling is the discipline of asking how a system can be intentionally misused before attackers, fraud rings, or careless insiders discover the same path in production. In system design, it is not a paperwork exercise. It is a way to convert an architecture diagram into concrete security requirements: what must be authenticated, what must be authorized, what must be rate limited, what must be logged, what must fail closed, and what must be reviewed by a human.

This lesson focuses on abuse prevention and secure architecture review. Abuse prevention covers behaviors that may use legitimate features in hostile ways: credential stuffing, trial farming, scraping, spam, refund fraud, privilege misuse, data exfiltration through exports, and workflow manipulation. Secure architecture review evaluates whether the design gives those abuses easy paths or places meaningful controls close to the assets being protected.

How Threat Modeling Works Internally

A practical threat model has four internal parts. First, define assets and outcomes. An asset can be money, personal data, administrative power, service capacity, brand reputation, model output quality, or customer trust. Second, identify actors. Actors include anonymous users, authenticated customers, administrators, compromised accounts, partner systems, background jobs, and operators with production access. Third, map data flows and trust transitions. A trust transition occurs whenever data or authority crosses from one zone of assumptions into another, such as browser to API, API to service mesh, service to database, support console to customer account, or webhook provider to internal worker. Fourth, enumerate abuse cases and decide controls.

Many teams use STRIDE as a vocabulary: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. For abuse prevention, add business-specific categories: automated signup, inventory hoarding, promotion abuse, spam creation, scraping, fake engagement, payment manipulation, and social engineering. The point is not to fill a matrix perfectly. The point is to force the design review to ask different questions than a feature review asks.

Secure architecture review then binds each threat to an architectural decision. Authentication proves an actor identity. Authorization decides whether that identity may perform an action on a resource. Input validation rejects malformed or dangerous requests before deeper components trust them. Rate limits and quotas reduce automation leverage. Idempotency prevents duplicate financial or state-changing effects. Segmentation limits blast radius. Audit logs support investigation and non-repudiation. Detection pipelines identify patterns too subtle for synchronous blocking.

Review Anatomy

A review usually starts with a small packet: system context, data classification, sequence diagrams for critical flows, API surfaces, role definitions, storage choices, external integrations, operational runbooks, and open assumptions. The reviewer should be able to answer three questions from that packet: what can a user do, what can a compromised user do, and what can an internal or partner component do if it behaves incorrectly?

A useful finding contains a scenario, affected asset, entry point, missing or weak control, exploit path, expected impact, proposed mitigation, owner, and verification method. A weak finding says, add security. A strong finding says, the export endpoint authorizes organization membership but not per-project access, so a compromised analyst account can export projects it cannot view in the UI; enforce resource authorization in the export service and add an integration test for cross-project access denial.

Example 1: Ranking Abuse Cases

Start with a small marketplace. The assets are account integrity, order data, refund money, and seller reputation. Three initial abuse cases are credential stuffing, invoice export tampering, and support impersonation. A simple residual score can help prioritize discussion. It is not a replacement for judgment, but it exposes disagreement about likelihood, impact, and existing control strength.

from dataclasses import dataclass

@dataclass(frozen=True)
class Threat:
    name: str
    likelihood: int
    impact: int
    control_strength: int = 0

    def residual_score(self) -> int:
        return max(0, self.likelihood * self.impact - self.control_strength)

threats = [
    Threat("credential stuffing", 5, 4, 6),
    Threat("invoice export tampering", 3, 5, 4),
    Threat("support impersonation", 4, 4, 2),
]

for threat in sorted(threats, key=lambda t: t.residual_score(), reverse=True):
    print(f"{threat.name}: {threat.residual_score()}")

The deterministic output ranks the threats by remaining risk after existing controls:

credential stuffing: 14
support impersonation: 14
invoice export tampering: 11

The tied top scores should not be ignored because the order is convenient. Credential stuffing may be reduced by bot detection, breached-password checks, adaptive MFA, and per-account throttles. Support impersonation may need stronger staff authentication, customer verification scripts, just-in-time access, session recording, and alerts when staff changes payout details. The design output is a backlog of controls, not a single score.

Example 2: Drawing the Attack Surface

For a customer order system, the architecture review should distinguish public, privileged, and asynchronous paths. A diagram like this is enough to expose review questions:

graph TD
  Browser[Customer browser] --> Edge[CDN and WAF]
  Edge --> Api[Public API]
  Api --> Auth[Auth service]
  Api --> Orders[Orders service]
  Orders --> Db[(Order database)]
  Admin[Support console] --> Api
  Queue[Async job queue] --> Orders

The browser-to-edge path raises bot, injection, and session theft questions. The API-to-auth path raises token validation and replay questions. The support console path is more dangerous than a normal customer path because it may bypass customer intent. The job queue path is often forgotten: if a worker accepts unsigned messages or trusts stale authorization decisions embedded in a job payload, an attacker who reaches the queue can create effects that the API would have rejected.

Expected behavior after review is precise: public APIs authenticate callers before returning personal data; support actions require staff role checks and reason codes; queued jobs carry immutable identifiers and re-fetch authorization-relevant state before sensitive writes; database writes are scoped by tenant and protected by constraints where possible.

Example 3: Detecting Automation Patterns

Abuse prevention often needs both inline controls and asynchronous detection. Inline controls stop obvious harm quickly. Detection jobs find patterns over time. The following small detector flags IP addresses that fail logins for at least three accounts:

import re

EVENTS = [
    {"ip": "203.0.113.10", "user": "a@example.com", "ok": False},
    {"ip": "203.0.113.10", "user": "b@example.com", "ok": False},
    {"ip": "203.0.113.10", "user": "c@example.com", "ok": False},
    {"ip": "198.51.100.4", "user": "a@example.com", "ok": True},
]

def suspicious_login_sources(events, threshold=3):
    failures_by_ip = {}
    for event in events:
        if not event["ok"] and re.match(r"^\d+\.\d+\.\d+\.\d+$", event["ip"]):
            failures_by_ip[event["ip"]] = failures_by_ip.get(event["ip"], 0) + 1
    return {ip: count for ip, count in failures_by_ip.items() if count >= threshold}

print(suspicious_login_sources(EVENTS))

The output is:

{'203.0.113.10': 3}

In a real system, this signal should not automatically ban every source forever. Corporate NATs, mobile carriers, and privacy relays can concentrate many legitimate users behind one address. A better response ladder is to require a challenge, slow requests, demand MFA for affected accounts, compare device fingerprints, and alert analysts when the same source also creates accounts, changes passwords, or tests payment instruments.

Design Choices and Trade-offs

Centralized authorization libraries improve consistency, but service owners still need resource-specific checks because only the service may know whether a user can export, refund, approve, or delete a particular object. Edge rate limits are cheap and fast, but they cannot see deep account state. Application rate limits are context-rich, but they add latency and shared-state complexity. Synchronous fraud checks reduce immediate loss, but they can harm availability if the fraud service is down. Asynchronous review preserves availability, but it allows some bad events before remediation.

Human review is another trade-off. Manual approval for high-value payouts or admin actions can stop subtle attacks, yet it creates queues, inconsistent decisions, and insider-risk concerns. Use human review where impact is high and automation confidence is low. Record the evidence shown to reviewers so later investigations can reconstruct why a decision was made.

Failure Modes and Troubleshooting

Symptom: users can access another tenant’s exported report even though the UI hides it. Cause: the export worker trusts a report ID from the request and does not re-check tenant ownership. Diagnosis: trace the request from API to queue to worker, inspect authorization calls, and run a cross-tenant integration test. Correction: enforce ownership in the worker and database query, include tenant ID in the job, and alert on denied cross-tenant attempts.

Symptom: login protection blocks many legitimate users after a marketing campaign. Cause: an IP-only rule treats a shared carrier or office network as one attacker. Diagnosis: compare blocked events by IP, account age, device, password success rate, geography, and user-agent stability. Correction: replace hard blocks with stepped challenges and combine IP with account and device signals.

Symptom: audit logs cannot prove who approved a refund. Cause: logs capture the service account rather than the initiating human and omit request correlation IDs. Diagnosis: follow one refund through API logs, queue messages, database rows, and admin UI events. Correction: propagate actor, reason, correlation ID, and impersonation state through every state-changing call.

Security, Performance, and Reliability Implications

Controls must be placed where bypass is hard. Client-side checks improve user experience but are not security controls. API checks protect direct calls, but background jobs and administrative tools need equivalent enforcement. Database constraints are useful for invariants such as tenant ownership and uniqueness, but they cannot express every business policy. Logs should avoid secrets while still carrying enough context for investigation.

Performance matters because abuse controls sit on hot paths. Cache low-risk decisions, precompute reputation features, and use bounded calls to external risk systems. Reliability matters because a broken control can either fail open and allow abuse or fail closed and create an outage. Choose the failure mode per action. Viewing public catalog data may fail open with reduced personalization. Changing payout details should fail closed or require delayed human review.

Hands-on Lab: Review a Password Reset Flow

Prerequisites: a text editor, the ability to run Python, and a draft architecture for a password reset flow with a browser, API, email provider, user database, and reset-token store.

  1. List assets: account ownership, email address, reset token, active sessions, and audit trail.
  2. Draw the flow: request reset, generate token, send email, submit token, set new password, revoke sessions, and write audit events.
  3. Mark trust transitions: browser to API, API to email provider, email link back to browser, API to token store, and API to session store.
  4. Write at least six abuse cases: user enumeration, token brute force, token replay, email-link interception, reset flooding, and session persistence after password change.
  5. Assign controls: uniform responses, high-entropy single-use tokens, token expiry, per-account and per-source throttles, session revocation, audit events, and alerts on unusual reset volume.
  6. Verify the model by creating test cases for one valid reset, one reused token, one expired token, one unknown email, and one burst of reset requests.
  7. Cleanup by removing any test accounts, invalidating generated tokens, and deleting test email artifacts from shared mailboxes.

Verification is complete when the model produces at least one concrete control for every abuse case and every sensitive state change has an actor, resource, timestamp, and correlation ID in the audit plan.

Assessment Exercises

  1. A support tool can change a customer’s email address. Design the authorization, audit, and abuse-detection controls. Explain which controls must be synchronous.
  2. A queue worker processes refund requests created by the API. What authorization checks belong in the API, and what checks must be repeated in the worker?
  3. Your signup rate limit blocks a university network during enrollment week. Diagnose the likely design flaw and propose a response ladder that reduces abuse without a hard outage for legitimate users.
  4. Review an object storage design for customer exports. Identify how signed URLs, expiry, tenant scoping, and logging should work together.
  5. Choose fail-open or fail-closed behavior for password change, profile image upload, catalog browsing, and payout account change. Justify each answer by impact.

Summary

Threat modeling turns architecture into adversarial questions. Abuse prevention extends the review beyond technical exploits to hostile use of legitimate product paths. A strong secure architecture review identifies assets, actors, trust transitions, abuse cases, controls, failure modes, and verification. The result should be specific enough to change APIs, worker behavior, data constraints, logs, tests, and operational playbooks.