Design a Social Feed Search and Recommendation Platform
Purpose and Outcome
A social feed search and recommendation platform answers two related questions at very high volume: what should this person see now, and what should appear when they search for a topic, account, or post? The outcome is not a single service. It is a pipeline that accepts writes from creators, builds searchable and rankable representations, retrieves a manageable candidate set, orders those candidates for a specific viewer, and records feedback so future ranking improves.
In this system design course, this case study is useful because it combines several recurring architecture problems in one product surface: write fanout, read latency, inverted indexes, feature stores, ranking models, freshness, privacy filtering, abuse controls, and experimentation. A good design is explicit about which work happens when a post is created, which work happens when a user opens the feed, and which work can run asynchronously without making the product feel stale.
Core Mechanism
The platform begins with the post write path. A post service validates text, media references, visibility settings, and author state. It then stores the canonical post record in a durable database and emits an event such as PostCreated. Downstream consumers tokenize text, detect language, extract hashtags and mentions, generate embeddings, compute safety labels, and update multiple serving stores. The search index stores terms mapped to post identifiers. The recommendation index stores vectors, topic labels, author relationships, and lightweight engagement counters. The feed store may also hold precomputed inbox entries for users who follow the author.
Search usually starts with lexical retrieval. A query such as database indexing is normalized into terms, analyzed with stemming or synonym rules, and looked up in an inverted index. The engine intersects or unions posting lists, scores candidates with term frequency, inverse document frequency, freshness, and engagement signals, then applies filters for visibility, blocks, muted words, and geography. Many platforms also add semantic retrieval. The query is embedded into a vector, nearest-neighbor search finds posts with similar meaning, and a blending stage merges semantic and lexical candidates before ranking.
Recommendations start from several candidate generators rather than a typed query. One generator returns posts from followed authors. Another returns posts liked by similar users. Another returns posts close to the viewer’s recent embedding profile. Another returns trending posts inside topics the viewer often consumes. Candidate generation must be broad enough to discover good items, but small enough for ranking to run within the request budget.
The ranker is the decision layer. It consumes candidates and features: viewer interests, author affinity, post age, predicted click probability, predicted dwell time, reply likelihood, negative feedback risk, creator quality, and diversity constraints. A machine-learned model may output a utility score, but the serving system still applies deterministic business rules. For example, it removes blocked authors, downranks content the viewer has already dismissed, caps repeated authors, and reserves slots for fresh followed content. The final response contains post identifiers, ranking metadata, and tracking tokens for impression logging.
API and Data Anatomy
A minimal serving API separates retrieval intent from ranking context. Feed requests include viewer_id, surface, pagination cursor, device hints, and experiment assignments. Search requests include query text plus filters such as since, media type, language, or account scope. Both paths return stable post ids and opaque cursors rather than exposing internal offsets. Offset pagination is fragile because new posts arrive while a user scrolls; cursor pagination can encode the last score, timestamp, and tie-breaker id.
from dataclasses import dataclass
from math import log
@dataclass(frozen=True)
class Post:
id: str
author: str
text: str
likes: int
age_minutes: int
posts = [
Post("p1", "ana", "python database indexing tutorial", 30, 10),
Post("p2", "ben", "coffee photos from paris", 80, 300),
Post("p3", "cal", "database shard rebalancing notes", 12, 20),
]
def score(post: Post, query_terms: set[str]) -> float:
matches = sum(1 for term in query_terms if term in post.text.split())
freshness = 1 / (1 + post.age_minutes / 60)
engagement = log(1 + post.likes)
return 3 * matches + 2 * freshness + 0.4 * engagement
query = {"database", "indexing"}
ranked = sorted(posts, key=lambda post: score(post, query), reverse=True)
print([post.id for post in ranked])This first example is a tiny lexical ranker. Post p1 wins because it matches both query terms and is fresh. Post p3 matches database but not indexing. Post p2 has engagement, but no query match, so it should not outrank relevant results. The deterministic output is [‘p1’, ‘p3’, ‘p2’].
Worked Example: Fanout Strategy
Feed systems often choose between fanout-on-write and fanout-on-read. Fanout-on-write copies a new post into follower inboxes as soon as it is published. This makes feed reads fast for ordinary accounts. Fanout-on-read stores only the author timeline and gathers recent posts from followed authors when the viewer opens the app. This avoids enormous write amplification for creators with millions of followers.
from collections import defaultdict
followers = {
"ana": {"u1", "u2", "u3"},
"celebrity": {f"u{i}" for i in range(10000)},
}
push_inboxes = defaultdict(list)
pull_authors = defaultdict(set)
def publish(author: str, post_id: str, celebrity_threshold: int = 1000) -> None:
audience = followers.get(author, set())
if len(audience) <= celebrity_threshold:
for user_id in audience:
push_inboxes[user_id].append(post_id)
else:
for user_id in audience:
pull_authors[user_id].add(author)
publish("ana", "p100")
publish("celebrity", "p200")
print(push_inboxes["u1"])
print("celebrity" in pull_authors["u1"])The expected output is [‘p100’] and True. Ana’s post is pushed to u1’s inbox. The celebrity post is not copied ten thousand times in this toy example; instead the reader records that celebrity posts must be pulled at read time. Real systems use a hybrid: ordinary authors are pushed, very large authors are pulled, and medium authors may be pushed in batches with backpressure.
Worked Example: Ranking Filters
Candidate generation is intentionally permissive. The ranking path must still enforce viewer-specific constraints before final ordering. A blocked author with a high model score must disappear. A post already seen in a previous page should not return again unless the product explicitly supports recaps.
from dataclasses import dataclass
@dataclass(frozen=True)
class Candidate:
post_id: str
author_id: str
base_score: float
is_blocked: bool = False
already_seen: bool = False
candidates = [
Candidate("p1", "ana", 0.91),
Candidate("p2", "ben", 0.87, already_seen=True),
Candidate("p3", "mallory", 0.99, is_blocked=True),
Candidate("p4", "cal", 0.63),
]
filtered =
ranked = sorted(filtered, key=lambda c: c.base_score, reverse=True)
print(])The expected output is [‘p1’, ‘p4’]. Candidate p3 has the highest base score, but policy filtering removes it. Candidate p2 is also removed because the viewer has already seen it. This illustrates why final ranking is not only model inference; it is model inference plus product, safety, and privacy logic.
Design Choices and Trade-offs
The first major trade-off is freshness versus ranking richness. A model that uses deep neural features, cross-user aggregates, and media embeddings may improve quality, but it can be too slow for every request. Many platforms use a two-stage or three-stage ranker: cheap retrieval for thousands of candidates, a medium ranker for hundreds, and an expensive ranker for the final dozens.
The second trade-off is consistency versus latency. Search indexes and feature stores are usually updated asynchronously, so a newly published post may be visible on the author’s profile before it appears in search or recommendations. For social feeds this is often acceptable if followed-author feeds remain fresh. For search over breaking news, the indexing service needs stricter latency objectives and a replayable event log so missed updates can be recovered.
The third trade-off is personalization versus diversity. Ranking solely by predicted engagement can overfit to narrow interests and repeated authors. Diversity rules intentionally spend some ranking score on exploration: new authors, adjacent topics, recent posts, or content with fewer historical impressions. This can lower immediate click prediction while improving long-term satisfaction and marketplace health.
Failure Modes and Troubleshooting
Symptom: users report that their new posts do not appear in search. A common cause is lag in the indexing consumer or a poison message that blocks a partition. Diagnose by checking the post table for the canonical record, the event log offset for PostCreated, indexer error counts, and whether the post id exists in the search shard. Correct by replaying the affected partition after fixing the parser or by routing invalid posts to a dead-letter queue with alerting.
Symptom: the feed repeats the same posts while scrolling. Causes include cursor fields that do not uniquely identify the last item, ranking scores that change between pages, or missing impression suppression. Diagnose by logging cursor contents, final scores, and seen-post filters for consecutive requests. Correct with stable tie-breakers such as score, timestamp, and post_id, plus a short-lived seen set keyed by viewer and session.
Symptom: recommendation quality suddenly shifts for a subset of users. Causes include missing feature backfills, an experiment assignment bug, or a feature-store freshness issue. Diagnose by comparing online features with offline training examples for the same viewer and post ids. Correct by rolling back the model or feature definition, replaying feature computation, and adding canary metrics for empty candidate sets, null feature rates, and per-cohort engagement.
Security, Performance, and Reliability
Security is part of serving correctness. The ranker must not rely on upstream retrieval to enforce blocks, private-account visibility, age restrictions, takedowns, or regional legal constraints. These filters belong near final serving because every candidate source can make mistakes. Search indexes also need deletion propagation for removed posts; otherwise a deleted item can remain discoverable through stale index entries.
Performance depends on bounding candidate counts and using cacheable layers. Public trending candidates can be cached by region and topic. Viewer-specific author affinity and seen sets are harder to cache, but they can be stored in low-latency key-value systems. Reliability comes from graceful degradation. If semantic retrieval is down, lexical search can still work. If the heavy ranker times out, a simpler ranker can order followed-author and trending candidates while the incident is investigated.
Hands-on Lab
Prerequisites: Python 3, a terminal, and a text editor. Create a temporary directory and place the three Python examples from this lesson into files named search_rank.py, fanout.py, and filter_rank.py. Run each file with python3. Verify that search_rank.py prints [‘p1’, ‘p3’, ‘p2’], fanout.py prints [‘p100’] and True on separate lines, and filter_rank.py prints [‘p1’, ‘p4’].
- Change the first example by increasing p3 likes to 1000. Run it again and observe whether relevance still beats popularity.
- Change the celebrity_threshold in the second example to 20000. Verify that the celebrity post now appears in u1’s pushed inbox, then explain why that setting would be dangerous at real scale.
- Add a new candidate p5 by a non-blocked author with base_score 0.70. Verify that it appears between p1 and p4 after sorting.
Cleanup is simple: delete the temporary files and directory. In a shared development environment, also remove any scratch metrics or logs created during experimentation so later tests do not read stale output.
Assessment Exercises
- Design a cursor format for feed pagination that prevents duplicates when two posts have the same ranking score. Explain which fields must be signed or kept opaque.
- A product manager asks for new posts to be searchable within one second. Which parts of the write path and indexing path would you measure first, and why?
- Compare fanout-on-write and fanout-on-read for an account with fifty followers versus an account with fifty million followers. Where would you place the hybrid threshold?
- Your recommendation model improves click-through rate but increases hides and blocks. Propose two ranking or evaluation changes that address this conflict.
- Explain why final visibility filtering must happen after candidate generation even if every retrieval source claims to enforce permissions.
Summary
A social feed search and recommendation platform is built from durable post storage, asynchronous indexing, multiple candidate generators, personalized ranking, strict final filters, and feedback loops. The design succeeds when it keeps fresh content moving, bounds expensive ranking work, degrades predictably, and treats privacy and deletion rules as serving-time requirements rather than optional metadata.
