Tokens, context windows, and sampling controls are the knobs that turn a trained language model into a usable generative AI component. Tokens decide how text is measured, the context window decides how much information can be considered at once, and sampling decides how the next token is chosen from the model’s probability distribution. After this lesson you should be able to estimate prompt size, design context packing rules, choose generation settings for different product goals, and diagnose common output problems.
In the Generative AI Foundations course, this topic sits at the point where model behavior becomes engineering behavior. Prompt wording matters, but so do length limits, truncation policy, decoding parameters, and repeatability requirements. The same model can behave like a conservative extractor, a varied brainstorming partner, or an unstable answer machine depending on these choices.
How Text Becomes Tokens
A model does not usually read characters or full words directly. It reads token ids from a tokenizer. A tokenizer maps common words, word pieces, punctuation, whitespace patterns, numbers, and fragments from many languages into integer ids. The model then predicts a probability distribution over the next token id. Finally, the client decodes token ids back into text.
Tokenization explains why token counts do not match word counts. A short common word may be one token. A rare technical identifier, long URL, emoji sequence, or base64 string may become many tokens. Spaces and punctuation can be part of tokens. This matters because most provider limits and prices are token based: input tokens plus generated output tokens consume the context budget and often determine cost.
A useful mental model is that the prompt is serialized into a single ordered sequence: system instructions, developer or policy instructions, conversation turns, retrieved documents, tool results, and the current user request. The model attends across that sequence inside its context window. It does not have direct access to information that was omitted, truncated, summarized incorrectly, or placed after a cutoff.
Context Window Internals
The context window is the maximum number of tokens the model can process for one generation, including both supplied input and reserved output. If a model accepts a window of N tokens and you reserve M tokens for the answer, your usable input budget is roughly N minus M, minus protocol overhead added by message formatting. Applications therefore need a budget plan before sending the request.
Attention lets the model condition each generated token on earlier tokens, but longer context is not the same as perfect memory. Important facts can be diluted by irrelevant text, contradictory passages, or poor ordering. Models also show position effects: information near instructions and near the current question is often easier to use than a buried paragraph in the middle. Context design is therefore an information architecture problem, not only a maximum-length problem.
Common context strategies include hard truncation, summarization, retrieval, priority packing, and conversational memory compaction. Hard truncation is simple but can silently remove the one fact that matters. Summarization saves space but may discard exact wording. Retrieval can include only the most relevant chunks, but depends on chunking quality and search recall. Priority packing keeps critical rules first and drops low-value evidence when the budget is tight.
Sampling Anatomy
At each generation step, the model produces logits, which are unnormalized scores for possible next tokens. Decoding converts those scores into a selected token. Greedy decoding chooses the highest probability token. Sampling chooses from a distribution, which creates variation. Temperature rescales logits before probabilities are calculated. Lower temperature sharpens the distribution, making high-probability tokens more dominant. Higher temperature flattens it, increasing the chance of surprising tokens.
Top-p, also called nucleus sampling, keeps the smallest set of highest-probability tokens whose cumulative probability reaches p, then samples from that set. A low top-p value narrows choices to the most likely continuations. A high value admits a longer tail. Temperature and top-p interact: temperature changes the shape of the distribution, and top-p clips that shaped distribution.
Other controls bound the generation. A maximum output token limit stops long answers and protects cost. Stop sequences end generation when a specified text pattern appears. Frequency and presence penalties, when available, alter the likelihood of repeated tokens or already-mentioned concepts. A seed may improve reproducibility for sampled decoding, but exact repeatability can still depend on model version, backend implementation, parallelism, and request shape.
API Fields and Configuration Shape
Although APIs differ, the same concepts appear repeatedly. Input messages carry roles and content. Model selection determines the tokenizer, context size, quality profile, latency, and cost. The maximum output setting reserves answer space. Temperature and top-p set randomness. Stop sequences define custom endings. Structured output settings may constrain the decoded text to JSON or another schema, which changes how you should think about sampling because invalid continuations can be rejected or repaired by the decoder.
A practical request template starts with the outcome, then the budget. For classification or extraction, use low temperature, a tight output limit, and a schema. For brainstorming, allow more output and moderate randomness. For retrieval-augmented question answering, reserve enough output for citations or explanation, keep system instructions short, and spend most input budget on the few passages most likely to answer the question.
Example 1: Estimating Prompt Size
This first example uses a deliberately rough estimator. It is not a replacement for the model’s real tokenizer, but it shows why application code needs a token budget before making a request.
def rough_token_count(text: str) -> int:
normalized = text.replace("\n", " ").strip()
if not normalized:
return 0
pieces = normalized.split(" ")
return sum(1 + max(0, len(piece) - 1) // 8 for piece in pieces)
prompt = "Summarize token budgets in two bullet points."
print(rough_token_count(prompt))The deterministic output is 7. The phrase has six space-separated pieces, and the word Summarize is long enough to count as two rough pieces. Real tokenizers will differ, especially around punctuation and non-English text, but the design lesson remains: count before you send, and keep a margin for message overhead and output.
Example 2: Packing Context by Priority
When there is more evidence than budget, decide what gets included before the model sees it. This example sorts context items by priority and includes only items that fit.
from dataclasses import dataclass
@dataclass(frozen=True)
class ContextItem:
name: str
text: str
priority: int
def rough_token_count(text: str) -> int:
return 0 if not text.strip() else sum(1 + max(0, len(p) - 1) // 8 for p in text.split())
def pack_context(items: list[ContextItem], budget: int) -> list[str]:
chosen: list[str] = []
used = 0
for item in sorted(items, key=lambda x: x.priority):
cost = rough_token_count(item.text)
if used + cost <= budget:
chosen.append(item.name)
used += cost
return chosen
items = [
ContextItem("system rule", "Answer only from supplied notes.", 1),
ContextItem("recent invoice", "Invoice 4421 total is 19 dollars.", 2),
ContextItem("old chat", "The user once asked about refunds and shipping labels.", 5),
]
print(pack_context(items, budget=12))The output is ['system rule', 'recent invoice']. The lower-priority chat history is dropped because it would exceed the budget. In a real application, this decision should be logged as metadata such as selected chunk ids and dropped counts, not by storing sensitive raw text unnecessarily.
Example 3: Temperature Changes Probabilities
The next example starts from three logits and shows how temperature changes the softmax probabilities before token selection.
import math
logits = {"red": 2.0, "blue": 1.0, "green": 0.1}
def softmax(values: dict[str, float], temperature: float) -> dict[str, float]:
scaled = {k: v / temperature for k, v in values.items()}
peak = max(scaled.values())
exp_values = {k: math.exp(v - peak) for k, v in scaled.items()}
total = sum(exp_values.values())
return {k: round(v / total, 3) for k, v in exp_values.items()}
print(softmax(logits, temperature=0.5))
print(softmax(logits, temperature=2.0))The expected output is {'red': 0.864, 'blue': 0.117, 'green': 0.019} and then {'red': 0.501, 'blue': 0.304, 'green': 0.194}. Lower temperature makes red dominate. Higher temperature gives the lower-ranked tokens more probability mass, which can help creative variation but can hurt factual consistency.
Example 4: Top-p Keeps a Nucleus
Top-p does not mean keep tokens whose individual probability is p. It means keep the highest-probability tokens until their cumulative probability reaches p.
def apply_top_p(probabilities: dict[str, float], top_p: float) -> list[str]:
kept: list[str] = []
total = 0.0
for token, probability in sorted(probabilities.items(), key=lambda item: item[1], reverse=True):
kept.append(token)
total += probability
if total >= top_p:
break
return kept
next_token_probs = {"Paris": 0.55, "London": 0.20, "Rome": 0.15, "Tokyo": 0.10}
print(apply_top_p(next_token_probs, 0.70))
print(apply_top_p(next_token_probs, 0.95))The output is ['Paris', 'London'] and ['Paris', 'London', 'Rome', 'Tokyo']. With top_p=0.70, the first two tokens already reach the cumulative threshold. With top_p=0.95, all four are needed. This is why high top-p can produce more varied completions even when the most likely answer is obvious.
Design Choices and Trade-offs
For deterministic workflows such as routing, extraction, grading, and policy checks, prefer low randomness, short outputs, explicit schemas, and tests that assert exact or schema-level behavior. For open-ended ideation, allow more randomness and evaluate for usefulness rather than exact wording. For long-document analysis, spend engineering effort on chunking, retrieval, deduplication, and source ordering instead of simply filling the largest possible window.
There is also a cost and latency trade-off. More input tokens usually increase request time and price. More output tokens can dominate latency because tokens are generated sequentially. A larger context window can reduce the need for summarization, but it can also encourage undisciplined prompts that bury the decisive evidence. Good systems make the budget visible: how many tokens are reserved for instructions, conversation history, retrieved context, tool results, and final output.
Failure Modes and Troubleshooting
One common symptom is an answer that ignores a document the user supplied. The cause may be truncation, failed retrieval, or a chunk placed outside the retained context. Diagnose by logging token counts, selected chunk ids, final message order, and the exact budget decision. Correct it by reserving output first, reducing low-value history, improving retrieval, or raising the budget only when the extra context is actually useful.
Another symptom is inconsistent answers to the same question. Causes include high temperature, broad top-p, changing hidden context, or a model/provider update. Diagnose by comparing full request payloads, generation parameters, seeds where supported, and model identifiers. Correct it by lowering randomness, pinning configuration, using structured output, and testing behavior rather than expecting sampled prose to be byte-for-byte identical.
A third symptom is an answer that ends mid-sentence or omits required fields. The cause is usually too small a maximum output limit or an accidental stop sequence. Diagnose by checking finish reasons, output token counts, and whether the stop text appears inside normal content. Correct it by increasing the output allowance, shortening the prompt, using a schema, or choosing a less collision-prone stop sequence.
Security, Performance, and Reliability
Token budgets are also security boundaries. Retrieved documents and tool outputs may contain prompt injection text that competes with system instructions inside the same context. Keep instructions separate from evidence, label untrusted content, and prefer narrow tools or schemas for high-impact actions. Do not rely on a large context window to make unsafe content harmless.
Performance improves when repeated boilerplate is shortened, long histories are summarized with checks, and retrieval returns compact chunks rather than whole files. Reliability improves when token counts, finish reasons, selected sources, and decoding settings are captured as structured telemetry. Avoid logging sensitive raw prompts by default; store identifiers, sizes, hashes, and redacted samples according to your retention policy.
Hands-on Lab: Build a Budgeted Request Plan
Prerequisites: Python 3, a text editor, and no model API key. The lab uses local code so you can focus on the mechanics before connecting a provider.
- Create three short text snippets: a system instruction, a user question, and two evidence passages.
- Use the rough token counter from Example 1 to estimate each snippet.
- Choose a total context budget such as 60 rough tokens and reserve 20 for output.
- Pack evidence with the priority function from Example 2 until the input budget is full.
- Write down the final request order: instruction, selected evidence, question, and reserved output.
- Run the script and verify that selected item names fit within the input budget.
- Change the output reservation from 20 to 35 and verify that less evidence fits.
Verification is successful when the script prints the selected context names and the sum of their estimated tokens never exceeds the input budget. Cleanup is simply deleting the scratch script. If you used real internal documents for practice, remove them from the working directory and from shell history or notebook outputs if they contain sensitive text.
Assessment Exercises
- A support bot has a 4,000-token context budget and must reserve 800 tokens for the answer. Describe how you would divide the remaining budget among instructions, chat history, retrieved policy, and the current question.
- Given two requests with identical prompts but different temperatures, explain why one is better for invoice extraction and the other is better for marketing tagline generation.
- A model returns valid JSON most of the time but sometimes adds a paragraph after it. Which generation settings and output constraints would you change first, and how would you verify the fix?
- Your retrieval system includes ten chunks, but the answer uses an outdated policy. List the diagnostic data you need before changing the model.
- Design a test that proves a stop sequence will not prematurely terminate an answer containing ordinary punctuation or quoted text.
Summary
Tokens are the accounting unit for model input and output, context windows are the bounded workspace for one generation, and sampling controls how probability becomes text. Good generative AI systems treat these as first-class design constraints. They count and reserve tokens, pack context intentionally, choose decoding settings that match the task, and troubleshoot from finish reasons, selected context, and parameter records instead of guessing from the final answer alone.
