Context Internals: How LLMs Actually Read Your Conversation
This addendum is the "why" companion to Context Management for AI-Native Workflows. The lesson teaches you what to do; this page explains the mechanics underneath. Reading it isn't required to use the Forge — but if you want to understand why the practical advice works the way it does, this is where the model is opened up.
Anatomy of a Conversation
LLMs don't "read" your conversation the way a person reads a memo. Understanding how they actually process text is the foundation of every context management technique in 1.7 — and the only way the failure modes covered later (attention dilution, lost-in-the-middle, context rot) make sense.
From text to vectors
Your input is first split into tokens — roughly word-fragments. "Liatrio enables enterprises" might become ["Li", "atrio", " enables", " enterprises"]. Token count, not word count, is what drives cost and context limits. Code and non-English text tokenize less efficiently.
Each token is then mapped to an embedding: a high-dimensional vector (hundreds or thousands of numbers) — the model's internal numeric representation of that token. Tokens used in similar contexts end up near each other in embedding space — "dog" sits near "puppy," "run" near "ran." That geometric proximity is the model's notion of meaning. It's also what makes semantic search (looking up text by meaning rather than exact-word match) and RAG — Retrieval-Augmented Generation, the pattern of pulling in relevant snippets from an external store and injecting them into the prompt — work.
Attention: how the model decides what to "look at"
When the model is about to produce the next token, it has to decide what parts of your context matter most right now. The mechanism it uses is called attention. You can think of it as a fuzzy lookup: for every token in the window, the model computes a relevance score against what it's currently trying to produce, then blends all the tokens together weighted by those scores. (In practice the model has many parallel attention heads doing this in different subspaces, each free to focus on different relationships — syntax, coreference, topical similarity — but the per-head mechanic is the same.)
Source: Jay Alammar, The Illustrated Transformer
The one detail that matters for everything downstream: those relevance scores are forced to sum to 1. The function that does this normalization is called softmax — it takes the raw scores and squashes them into a probability distribution that adds up to 1. So attention is a fixed budget that has to be spread across every token the model can see. With 2,000 tokens in context, an important token can command a big slice of that budget. With 200,000 tokens, the same important token — no matter how relevant — gets a much smaller slice, because the budget got stretched over 100× more competitors. This is the mechanism behind context dilution, and we'll come back to it.
The context window has limits — and quadratic cost
The context window is the model's working memory. It has a hard maximum size (Claude's is 200K or 1M tokens depending on the model), but more importantly: the cost of using it grows quadratically, not linearly. Doubling your context from 50K to 100K tokens roughly quadruples the compute required, because every token has to be compared against every other token.
The practical consequence: long prompts are both slower (more compute) and dumber (attention dilutes across more tokens). Filling the window isn't free even when you stay under the limit.
Multi-turn conversations are stateless replay
This is the single most important thing to internalize, and it's the part most people miss: LLMs have no memory between turns. The chat experience gives the illusion that the model is learning about you as you talk — that it "remembers" what you said five messages ago. It isn't, and it doesn't.
Every time you hit enter, your entire conversation so far — the system prompt, every prior user message, every assistant reply, every tool call, every tool result — gets shoved back into the model as a single big input. The model reads the whole script from scratch, generates a response, and then forgets everything. The next turn does it all again, with your new message tacked on the end.
"Memory" in ChatGPT, Claude, or Claude Code is a UI illusion built on top of a fundamentally stateless system. The client (the chat app, the CLI, the IDE plugin) is the thing keeping the transcript — the model itself holds nothing between turns.
The system prompt isn't architecturally special, either. It's just the first message in that transcript: it gets positional priority and the model has been trained to treat it carefully, but there's no technical privilege — it can be overridden by enough later context.
Why this matters
Everything in 1.7 follows from these mechanics:
- Finite attention budget → long contexts produce diluted, generic outputs (context rot)
- Positional asymmetry → information buried in the middle gets ignored (lost in the middle)
- Stateless replay → you own memory. Summarize, prune, externalize to files
- Quadratic cost → every frontloaded token of system prompt, tool definitions, and skills is working memory you can't use
Context Gotchas
Armed with the mechanics above, we can name and explain the specific failure modes that plague long-context LLM sessions. These aren't bugs — they're predictions that fall out of how attention works. Knowing the names and mechanisms means you can spot them in the wild.
Context Rot and the Dumb Zone
Practitioners have a name for the experience of watching a capable AI agent become progressively less useful mid-session: the "Dumb Zone." The term was coined by Dex Horthy (HumanLayer) and amplified by Matt Pocock, capturing the frustration of an agent that was performing brilliantly ten minutes ago but now seems to have forgotten how to code.
The broader phenomenon is called context rot — a term that originated on Hacker News in June 2025 and was operationalized by Chroma Research shortly after. It describes the gradient nature of degradation: quality doesn't fall off a cliff, it erodes progressively as context fills.
Source: Chroma Research — Context Rot
The chart above shows performance across multiple models (Claude Sonnet 4, GPT-4.1, Qwen3-32B, Gemini 2.5 Flash) on a repeated-words task as input length increases. All models degrade toward ~0.5 (random chance) by 10K+ tokens. Claude holds the longest but still drops sharply. The takeaway: no model is immune to context rot.
Subsequent research has reinforced this. NoLiMa (ICML 2025, Adobe Research) showed that 11 of 12 frontier models drop below 50% of their short-context baseline by 32K tokens — GPT-4o falls from 99.3% to 69.7%. NVIDIA's RULER benchmark — an extension of the classic needle-in-a-haystack (NIAH) test, where a known fact ("the needle") is hidden in a long irrelevant passage ("the haystack") and the model is asked to retrieve it — adds multi-hop tracing on top and shows the same pattern: models that nail vanilla NIAH degrade steeply as length and complexity grow.
Analyzing ~100K developer sessions, Dex Horthy located the Dumb Zone at roughly the 40–60% middle band of the context window — a range where recall and reasoning collapse well before the window is full. His operational rule: keep utilization below 40% via aggressive compaction. One detail matters when translating that advice: his analysis ran against Claude's 200K window, so the 40–60% band is really the 80–120K token band — squarely where the benchmark evidence above shows most models substantially degraded.
Percent-of-window thresholds don't transfer across window sizes. NoLiMa measured Gemini 1.5 Pro — a 2M-token window — at an effective length of about 2K tokens (~0.1% of its window), while GPT-4o managed 8K on a window one-fifteenth that size. A bigger window did not buy a proportionally bigger effective length. What actually predicts degradation is absolute token count and task difficulty: measurable decline begins in the tens of thousands of tokens, and by roughly 100K tokens most models are substantially degraded — whether their window is 200K or 1M.
That's why this program frames its guidance in tokens. A percent rule that works on a 200K window (40% ≈ 80K tokens) becomes dangerous on a 1M window, where 40% utilization is 400K tokens — far past the point where quality has already collapsed. The exact numbers vary by model and task; the principle doesn't: don't wait until your context is full to act.
Lost in the Middle
The first mechanism behind context rot was formalized by Liu et al. in the TACL 2024 paper "Lost in the Middle: How Language Models Use Long Contexts." They found that models disproportionately use information placed at the start (primacy bias) and end (recency bias) of their context, while accuracy on information in the middle drops by 20+ percentage points — sometimes below the closed-book baseline (the model's accuracy when given no supporting documents at all and forced to answer from training memory alone), meaning extra context actively hurts.
Source: Liu et al. 2024 — Lost in the Middle (TACL). Accuracy is highest when the answer-bearing document is at the top or bottom of the context, and lowest in the middle — the dashed red line is the closed-book baseline.
The phenomenon has been replicated across GPT-4, Claude, LLaMA-2, and MPT. It persists in modern long-context architectures, though frontier 2026 models show partial mitigation. Follow-up work — Found in the Middle (Hsieh et al. 2024), RULER, NoLiMa — attributes the U-curve to a combination of mechanisms we'll unpack in the next section.
Practical implications for prompt structure:
- Place critical instructions at both the top and bottom of long prompts — exploit primacy and recency
- Put long documents above the query; put the actual ask last
- For ranked retrievals, reorder so the strongest content sits at the top and bottom of the list — weakest in the middle
- Use XML tags (
<instructions>,<context>,<document>) so the model can locate sections by structure, not just position - Repeat key constraints at the top and bottom of the prompt — so no matter where attention "sags" in the middle, the constraint gets reinforced by primacy or recency
Anthropic's own long-context guidance confirms: placing 20K+ token documents above the question, with the query last, can improve quality by up to 30%.
Attention Sinks and Attention Dilution
Two mechanisms sit underneath both context rot and lost-in-the-middle. Both fall straight out of the softmax constraint above.
Attention sinks (Xiao et al. 2023, Efficient Streaming Language Models with Attention Sinks). Every input fed to an LLM starts with a special BOS token (beginning-of-sequence) — an invisible marker that says "this is where the sequence starts." Because the attention budget must sum to 1, every token must spend its full share somewhere — even when nothing in the context is actually relevant to what the model is generating. Models learn to dump this excess probability mass onto a fixed, always-visible "parking spot": usually the BOS token, or the first few tokens after it. These are attention sinks — tokens that absorb vast amounts of attention for no semantic reason, acting as a kind of attention ground wire.
Source: Xiao et al. 2023 — StreamingLLM. The bright vertical stripes on the left of each panel are attention sinks: the first few tokens capture attention across nearly every layer and head, regardless of semantic relevance.
This is why the system prompt works: in a decoder-only LLM, every token can only attend to tokens that came before it in the sequence (a property called the causal mask — it prevents the model from "peeking ahead" during training). So early tokens are seen by every later token, while late tokens are seen by no one. Instructions placed at the start therefore have a persistent, architectural advantage. It's also the mechanism behind primacy bias in "lost in the middle."
Attention dilution. The same softmax constraint produces a second problem as context grows. The maximum attention weight any single key can achieve shrinks as the number of keys increases — the distribution flattens. Even if the model "knows" which token matters, the ceiling on its attention weight drops with length. Signal doesn't get louder; the noise floor rises as non-informative tokens collectively absorb probability mass.
This is the mechanistic core of context rot. It's not that the model forgets — it's that under attention's mathematical constraints, specific facts buried in long contexts can't pull enough attention weight to drive generation. This is why long contexts produce vague, generic outputs even when the relevant information is demonstrably present.
This isn't just practitioner intuition. Veličković et al. proved formally that softmax attention must disperse as input length grows — even a task as simple as "find the maximum" can't stay sharp (softmax is not enough). And the length effect is isolable from distraction: even with perfect retrieval and every irrelevant token masked out, raw input length alone degrades performance by 13.9–85% (Context Length Alone Hurts LLM Performance, EMNLP 2025).
| Mechanism | What it produces |
|---|---|
| Sinks on initial tokens | Early instructions "leak through" even when middle context is ignored |
| Recency bias (last tokens seen most recently) | Last-mentioned constraints often dominate outputs |
| Dilution as context grows | Long contexts produce vague, generic answers; specific facts get drowned |
| Softmax flattening | Output specificity decreases even when the relevant fact is clearly present |
Context Poisoning and Context Distraction
Even when the mechanics are cooperating, what's in your context matters as much as how much. Drew Breunig's widely-cited failure taxonomy ("How Long Contexts Fail", June 2025) identifies four distinct ways long contexts degrade:
| Failure | Definition | Example |
|---|---|---|
| Poisoning | A hallucination or error enters the context and gets repeatedly referenced, compounding downstream. | Gemini playing Pokémon hallucinated game state into its "goals" section, then pursued impossible objectives. |
| Distraction | Context grows so long the model over-focuses on it, neglecting trained knowledge. Past ~100K tokens, agents tend to repeat prior actions rather than plan new ones. | Agent keeps re-attempting a failed approach instead of reconsidering. |
| Confusion | Superfluous content — often excess tool definitions — degrades output. Smaller models collapse sharply as tool count grows. | 50 MCP tools loaded when 5 are relevant; model mis-selects or invents arguments. |
| Clash | New information or tools conflict with earlier context. Early failed attempts left in history bias the final output. | Agent follows an initial wrong plan even after you've corrected it, because both plans are in context. |
Source: Drew Breunig, How Long Contexts Fail — each added tool definition is context overhead that can measurably degrade output.
Poisoning vs. distraction — the critical distinction:
- Poisoning is about correctness. Bad facts persist and corrupt downstream reasoning. This is the most dangerous type of pollution because it actively misleads.
- Distraction is about focus. Even correct content, if too voluminous, pulls the model into repetition or recency bias rather than genuine planning.
In security literature, "context poisoning" can also mean adversarial prompt injection — content deliberately crafted by an attacker to manipulate the agent. Breunig's sense (and ours here) is the everyday, accidental version: your own failed attempts, hallucinated facts, or wrong conclusions persisting in context and biasing future output. Both matter; they just have different mitigations.
Why this hierarchy matters for compaction. When you run /compact on a polluted context, the summarizer faithfully condenses everything — including wrong conclusions — as if it were valid work. Distraction (noise) gets converted into poisoning (assertions). This is why curating context before compacting matters so much — see Intentional Compaction in 1.7.
Everything in this section — the token thresholds included — is downstream of how today's transformer attention works: a softmax spread over the full window. Architectures are evolving quickly (sparse attention, sub-quadratic designs, longer-sequence training), and each model generation pushes the degradation curve further out. Treat the specific numbers as a snapshot of current models, and revisit them as the architecture advances. The habits — proactive compaction, curated context, externalized state — are the durable part.
Structured Prompting with XML Tags
In 1.7, we recommend Markdown-first structure (clear headings, lists, labeled code fences) for most prompts — that's all the structure you need 90% of the time, and it's the format Claude Code is already designed around. But Anthropic explicitly trains Claude to recognize XML tags as a parsing mechanism, and there are situations where the extra anchoring is worth the verbosity: programmatic API prompts that mix instructions with multiple variable inputs, prompts with {{...}} template substitutions, or long retrieval prompts where you want the model to address documents by source.
From Anthropic's prompt engineering guide:
XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs.
Canonical tags
Anthropic's documentation calls out a small set of tag names Claude has been trained to recognize:
<instructions>— the directive: what you want Claude to do<context>— background information<input>— the user-supplied data the model should operate on<example>— few-shot examples<document>with<source>and<document_content>subtags — for multi-document inputs<thinking>— for reasoning-extraction patterns
A typical multi-document prompt looks like:
<documents>
<document index="1">
<source>annual_report_2023.pdf</source>
<document_content>{{ANNUAL_REPORT}}</document_content>
</document>
</documents>
Analyze the report. Identify strategic advantages.
This structure cooperates with every mechanism we've discussed: the instruction sits at the end (recency), the documents are clearly delimited (so the model can locate them by structure rather than position), and tags give the model anchors that resist dilution.
Lost-in-the-middle-aware structure
Critical instructions go at both the top and bottom of the prompt. Long documents go above the query. For retrieval, reorder chunks so the strongest sit at the top and bottom of the list, not the middle. Repeat key constraints at the top and bottom — so no matter where attention sags in the middle, the constraint gets reinforced. The cost is a few dozen tokens; the benefit is reliability.
When Markdown is enough
For day-to-day Claude Code work — chatting in the CLI, editing project files — Markdown is almost always sufficient. Headings, lists, and labeled code fences give the model the same kind of structural anchors XML tags do, in a format that's also human-readable and that fits the way Claude Code's own system prompt is structured. Reach for XML when:
- You're constructing a programmatic prompt (API call, automation, batch job)
- You're substituting variable inputs via templating (
{{VARIABLE}}) - The prompt contains multiple distinct documents and you want the model to address them by source
- You're chaining outputs and want the model's response inside a specific tag for downstream parsing
For everything else, Markdown first.