Context Windows

Welcome back to the NEXUS AI Literacy Series. We're starting a new track now — Memory and Knowledge — all about why these models know some things cold, forget others instantly, and occasionally make things up with total confidence. And we begin with the single most practical concept in this entire series, the one that, once you understand it, will immediately change how you use these tools day to day. It's called the context window. By the end of this episode, you'll know exactly what it is, why it matters, and how to work with it instead of against it.

Let me start with a puzzle you may have run into yourself. You're having a long conversation with an AI — really cooking, it's tracking everything, and then somewhere deep in the chat it suddenly seems to forget something you told it twenty minutes ago. Or you paste in a huge document and ask about the first page, and it acts like the first page doesn't exist. What happened? It didn't get dumber. You hit the edge of its context window.

Here's the core idea. The context window is the model's working memory — the amount of text it can actively hold in mind at one time while it's generating a response. Everything inside that window, it can see and use. Anything that falls outside it, it simply cannot see — not "forgets," exactly, but genuinely cannot access, like it was never there.

The analogy I want you to lock in is a desk. Imagine you're working at a desk, and the context window is the size of that desk. Every document relevant to your task has to fit on the desktop in front of you. While a paper is on the desk, you can read it, reference it, cross-check it. But the desk is only so big. When it fills up and you need room for new papers, the oldest ones get pushed off the edge onto the floor. They still exist — but you can't see them anymore while you're working. To use them again, someone would have to physically put them back on the desk. That desk is the context window. The size of the desk is measured in tokens — those chunks of text we talked about, roughly three-quarters of a word each.

And here's the crucial, slightly counterintuitive part: the context window holds the entire conversation, not just your latest message. Every question you've asked, every answer the model gave, the document you pasted, the instructions you set at the start — all of it sits on that desk together. That's actually how the model "remembers" your conversation at all. It's not storing your chat in some memory bank the way a person stores memories. Every single time it generates a reply, it re-reads the whole desk from scratch. It has no memory between turns other than what's sitting in the window. So when the conversation gets long enough that early parts slide off the edge, they're gone from its view — and that's the moment it "forgets" what you told it at the start.

Now, the size of these desks has grown astonishingly fast, and this is genuinely one of the big stories in AI progress. The early models had tiny desks — a few thousand tokens, a couple thousand words. You'd hit the edge constantly. Today's frontier models have enormous ones — hundreds of thousands of tokens, and some now reaching a million or more. To put a million tokens in perspective: that's roughly seven or eight full-length novels sitting on the desk at once. You can drop an entire codebase, a giant contract, a year of meeting notes into the window and have the model reason across all of it. That expansion is a big part of why these tools suddenly feel so much more capable than they did even a year or two ago — we made the desk enormous.

But — and you knew there'd be a but — a bigger desk is not free, and this is where you get practically smart. Two things to know.

First, cost and speed. Remember, the model re-reads the entire window every time it generates a response. So the more you put on the desk, the more work it does on every single turn — which means each response costs more money and takes more time. Stuffing a giant document into the context for a question that only needed one paragraph is like rereading an entire encyclopedia to answer one trivia question. It works, but you're paying for all of it, every turn.

Second — and this is the subtle one that even experienced people miss — a bigger desk doesn't mean perfect attention across the whole thing. Researchers have found that models pay the most attention to the beginning and the end of the context window, and can get a little fuzzy on material buried in the middle. They call it the "lost in the middle" problem. Picture cramming for an exam: you tend to remember the first things you studied and the last things you studied, while the stuff in the middle gets hazy. Models show a similar pattern. So just because something technically fits in the window doesn't guarantee the model is weighing it as strongly as you'd hope.

So how do you use this knowledge? Here's the practical payoff, the stuff that makes you better at this than almost everyone around you.

One: put the most important information at the very start or the very end of what you send — your key instructions, the critical document — not buried in the middle of a giant wall of text. Two: for a brand-new topic, start a fresh conversation instead of continuing a long one. A long chat means the model is dragging all that old, now-irrelevant history around on its desk, which costs you money, slows things down, and risks confusing the new task. A clean desk is a sharp desk. Three: don't dump everything you have into the window "just in case." More context isn't automatically better — it's more expensive, slower, and can actually dilute the model's focus. Give it what's relevant, prominently placed, and no more.

And this sets up beautifully where we're going next. If the context window is a limited desk, you face an obvious problem: what if the knowledge you need is way bigger than any desk — a million-document company knowledge base, an entire library? You can't fit it all in the window at once. The solution is to get clever about fetching only the right papers and placing them on the desk exactly when they're needed. And to do that, you need a way to find the right information by meaning — which is exactly the vector databases we teased back in Episode 2.

So let's bring it home. The context window is the model's working memory — a desk of a certain size, measured in tokens, holding your entire conversation at once. The model re-reads that whole desk every time it replies, and anything that slides off the edge it can no longer see — that's why long chats "forget" the beginning. The desks have grown enormous, up to a million tokens or more, but bigger isn't free: every token costs money and time on every turn, and models attend best to the start and end, not the murky middle. Work with it: put what matters at the edges, start fresh for new topics, and don't overload the desk.

Master this one idea and you'll get noticeably better results from every AI tool you touch — because you'll be managing the desk on purpose instead of by accident.

In the next episode, we solve the "knowledge bigger than the desk" problem head-on with vector databases — the technology that lets an AI search a massive library by meaning and pull exactly the right pages onto the desk. See you there.