Conversation memory lets an AI application use information from earlier turns without pretending the model has a private human memory. In this lesson, the outcome is practical: keep the short-term transcript available for the current answer, promote only selected facts into durable memory, recall those facts for the correct user, and give users control over what is stored.
In a generative AI application, memory is application state, not a hidden property of the model. The model receives a prompt assembled by your code. If the prompt contains recent messages, a summary, or retrieved facts, the model can use them. If those items are absent, the model cannot reliably know them. That distinction shapes the design: context is temporary evidence in the current request, while durable memory is data you store, govern, retrieve, and delete.
Purpose and Outcome
Conversation memory improves continuity. A tutoring assistant can remember that a learner wants Python examples. A support assistant can keep the current troubleshooting steps in view. A writing assistant can remember a user’s style preference. The feature is successful when recalled information is relevant, current, consented, and scoped to the right identity. It fails when stale summaries override new instructions, private data appears in another user’s session, or the assistant confidently cites something that was never stored.
How Memory Works Internally
A typical memory pipeline has five stages. First, the application records the current turn with role, content, time, session, and user identity. Second, it decides what portion of recent history to include directly in the next prompt. Third, it extracts candidate memories such as preferences, profile facts, or long-running task state. Fourth, it stores approved memories with metadata: owner, source, consent, confidence, timestamps, retention class, and sometimes an embedding for semantic search. Fifth, on later requests, it retrieves a small set of relevant memories and injects them into the prompt as labeled evidence.
Short-term memory is usually a sliding window over recent messages. It is simple and faithful because it uses exact text, but it grows until it hits token limits. Summary memory compresses older turns into fewer tokens. It saves space but can drop details or make ambiguous statements sound certain. Vector memory stores embeddings for semantic recall. It can find related facts even when the user changes wording, but it needs filters for user identity, source type, sensitivity, and freshness. Many production systems combine all three.
API and Data Anatomy
A memory record should be boring and explicit. Useful fields include memory_id, user_id, scope, text, source_message_id, created_at, updated_at, expires_at, confidence, consent, and sensitivity. Retrieval should accept a query, the authenticated user, allowed scopes, and a result budget. Prompt assembly should label memory separately from instructions and current user input, for example: Known user preferences, if relevant:. That label helps avoid treating remembered facts as developer instructions.
Example 1: Recent-Context Memory
The first example keeps only recent turns. The expected output shows the system message, three remembered turns, and the new user question in order. Nothing is stored beyond the process data structure.
from dataclasses import dataclass, field
from time import time
@dataclass(frozen=True)
class Message:
role: str
content: str
created_at: float = field(default_factory=time)
def build_prompt(system: str, history: list[Message], user_message: str, max_history: int = 4) -> list[dict[str, str]]:
if not user_message.strip():
raise ValueError("user_message is required")
recent = history[-max_history:]
prompt = [{"role": "system", "content": system}]
prompt.extend({"role": item.role, "content": item.content} for item in recent)
prompt.append({"role": "user", "content": user_message})
return prompt
history = [
Message("user", "My cat is named Pixel."),
Message("assistant", "I will remember that within this chat."),
Message("user", "I prefer concise answers."),
]
for item in build_prompt("Answer as a helpful tutor.", history, "What is my preference?"):
print(f"{item['role']}: {item['content']}")This pattern is appropriate for chat continuity inside one session. Its main trade-off is the context budget: a long conversation may push out earlier facts. Because the prompt contains exact recent text, it is easier to debug than a generated summary.
Example 2: Summary Memory
The second example extracts only simple user-stated facts. Its deterministic output is My name is Asha; I prefer Python examples. The question about vector databases is not stored because it is a transient request, not a stable preference or profile fact.
from dataclasses import dataclass
@dataclass(frozen=True)
class Turn:
role: str
content: str
IMPORTANT_PREFIXES = ("my name is ", "i prefer ", "remember that ")
def summarize_for_memory(turns: list[Turn], max_chars: int = 120) -> str:
facts: list[str] = []
for turn in turns:
text = turn.content.strip().lower()
if turn.role == "user" and text.startswith(IMPORTANT_PREFIXES):
facts.append(turn.content.strip().rstrip("."))
summary = "; ".join(facts)
return summary[:max_chars]
turns = [
Turn("user", "My name is Asha."),
Turn("assistant", "Nice to meet you."),
Turn("user", "I prefer Python examples."),
Turn("user", "What is a vector database?"),
]
print(summarize_for_memory(turns))Real summarizers often use a model, but the decision policy should still be testable: which facts are eligible, whether consent is required, and how uncertainty is represented. A good summary says less with higher confidence instead of compressing every turn into a misleading biography.
Example 3: Durable, Scoped Recall
The third example adds durable memory with consent and user isolation. The expected output is ['Prefers Python examples'] and then []. User u2 has a spreadsheet preference, but the Python query must not retrieve user u1‘s memory. The unconsented billing note is not stored.
from dataclasses import dataclass
@dataclass(frozen=True)
class Memory:
user_id: str
text: str
consent: bool
score: float
class MemoryStore:
def __init__(self) -> None:
self._items: list[Memory] = []
def add(self, memory: Memory) -> None:
if memory.consent:
self._items.append(memory)
def recall(self, user_id: str, query: str, min_score: float = 0.40) -> list[str]:
query_terms = set(query.lower().split())
matches: list[str] = []
for memory in self._items:
if memory.user_id != user_id or memory.score < min_score:
continue
text_terms = set(memory.text.lower().split())
if query_terms & text_terms:
matches.append(memory.text)
return matches
store = MemoryStore()
store.add(Memory("u1", "Prefers Python examples", True, 0.91))
store.add(Memory("u2", "Prefers spreadsheet examples", True, 0.88))
store.add(Memory("u1", "Mentioned an unverified billing problem", False, 0.95))
print(store.recall("u1", "python preference"))
print(store.recall("u2", "python preference"))The matching logic is intentionally simple so the security property is visible. In a real system, lexical matching might be replaced by embedding similarity, but the identity and consent filters must remain mandatory preconditions, not optional ranking hints.
Design Choices and Trade-offs
Choose direct transcript memory when exact wording matters, such as legal wording, troubleshooting logs, or a multi-step form. Choose summary memory when continuity matters more than verbatim detail. Choose vector recall when the user may refer to stored information indirectly, such as asking for examples in their usual language. Use expiration for facts likely to change, such as project names, roles, locations, or active goals. Use user-visible memory management for personal preferences and profile details.
Be conservative about what becomes durable. A user saying “I am frustrated today” is usually not a durable personality fact. A user saying “remember that I prefer TypeScript examples” is a strong candidate. Store source links so a user or operator can inspect why a memory exists. Keep memory snippets short, atomic, and editable; “prefers concise Python examples” is easier to update than a paragraph that mixes name, job, medical detail, and project context.
Failure Modes and Troubleshooting
Symptom: the assistant forgets a fact from earlier in the same chat. Cause: the sliding window excluded the turn or the summary job never ran. Diagnosis: log the assembled prompt length, included message IDs, and summary version. Correction: increase the recent-turn budget, summarize earlier, or add a retrieval path for important facts.
Symptom: the assistant recalls another user’s preference. Cause: retrieval ranked memories before applying the authenticated user filter, or a cache key omitted user_id. Diagnosis: inspect the query filter, cache key, and returned memory owner IDs. Correction: enforce owner filtering at the storage query layer and add a regression test with two users who have similar preferences.
Symptom: stale memory overrides the current message. Cause: prompt assembly presents old memory as more authoritative than fresh user input. Diagnosis: compare memory timestamps with the latest turn and inspect prompt section ordering. Correction: mark memories as background facts, prefer the current turn on conflict, and update or expire stale records.
Security, Performance, and Reliability
Memory increases privacy risk because it deliberately persists user information. Require consent for sensitive or long-lived facts, provide deletion, and avoid storing secrets. Apply authorization before retrieval, before prompt assembly, and before administrative inspection. Performance is mostly about bounding work: limit transcript length, number of recalled memories, embedding calls, and prompt tokens. Reliability depends on graceful degradation. If the memory store is unavailable, the assistant can still answer from the current message and recent context, while clearly avoiding claims based on missing memory.
Hands-on Lab: Build a Memory Slice
Prerequisites: Python 3, a terminal, and no external services. Step 1: run Example 1 and confirm the prompt includes only the recent conversation plus the new question. Step 2: run Example 2 and add a new turn, I prefer examples with tests.; verify the summary includes the new preference. Step 3: run Example 3 and add a memory for u1 with consent=False; verify recall does not return it. Step 4: change the recall call from u1 to u2; verify no cross-user result appears for the Python query. Cleanup is simple because the examples store data in memory only. If you adapt the lab to a database, delete the test rows by test user ID and remove any generated embeddings.
Assessment Exercises
- Design a memory schema for a coding tutor. Which fields are required to support deletion, source inspection, and user-visible editing?
- A user says,
I used to prefer JavaScript, but now I want Python.Explain how your system should update memory and how prompt assembly should handle the conflict before the update completes. - Write a test that proves two users with similar preferences cannot retrieve each other’s memories.
- Given a 40-turn conversation, decide which turns should remain verbatim, which should be summarized, and which should become durable memories. Justify the boundary.
- Explain when a memory retrieval failure should be invisible to the user and when it should change the response.
Summary
Conversation memory is a retrieval and state-management feature around a generative model. Current context gives the model immediate evidence; durable memory stores selected facts for later recall. Strong implementations keep memories atomic, consented, scoped, timestamped, and easy to delete. The engineering challenge is not making the assistant sound familiar; it is ensuring that familiarity comes only from the right data, at the right time, for the right user.
