Parameters & Model Size

Welcome back to the NEXUS AI Literacy Series. This episode closes out our Foundations track, and we're going to demystify the numbers everybody throws around but few can actually explain. "This model has seventy billion parameters." "The new one is four hundred billion." "It was trained on fifteen trillion tokens." Those numbers get tossed around like horsepower at a car show, and they sound impressive, but what do they actually mean? Is a bigger model always a better model? By the end of this episode, you'll be able to read those specs like someone who knows what they're looking at — and you'll understand why "bigger" is one of the most interesting and misunderstood ideas in all of AI.

Let's start with the word you hear most: parameters. We've been calling them dials, and that's exactly right. A parameter is one tiny adjustable number inside the network — one knob that training nudges into place. When someone says a model has seventy billion parameters, they mean it has seventy billion of those little knobs, all tuned during training. So the parameter count is, roughly, a measure of the model's raw capacity — how much it can store and how nuanced its internal wiring can be. More parameters means more room to capture patterns, more shades of meaning, more knowledge.

Here's an analogy that makes it concrete. Think of parameters as the number of neurons and connections in a brain. A creature with very few connections can learn only simple things. As you add more connections, you get the capacity for richer, more abstract, more flexible thinking. A bigger model, in this sense, has a bigger brain — more capacity to absorb and represent the complexity of the world. And for years, the headline story of AI was exactly this: make it bigger, and it gets better. Researchers found these remarkably consistent patterns — they call them "scaling laws" — showing that as you reliably increased three things together, model size, the amount of training data, and the computing power, the model's performance improved in a smooth, almost predictable curve. That discovery is a huge part of why the field exploded. It meant progress wasn't a matter of waiting for some genius insight — you could buy progress, predictably, by scaling up. So everyone raced to build bigger.

But — and this is the important turn — bigger is not free, and bigger is not always better. Let me give you the other side, because this is where the real sophistication lives.

First, the cost. Every additional parameter makes the model more expensive in two ways. It costs more to train, because there are more dials to nudge trillions of times. And it costs more to run — every single time you use a bigger model, more computation has to happen, which means more money and more time for each answer. A giant model answering a simple question is like firing up a massive industrial furnace to toast a single slice of bread. It works, but it's wildly inefficient if the job didn't need all that power.

Second, and more surprising: a bigger brain with not enough to learn from is actually a problem. Picture a brilliant student locked in a room with only three books. All that mental capacity, nothing to fill it with — they'll end up just memorizing those three books word for word instead of actually learning to think. The same thing happens to an oversized model trained on too little data: it memorizes its training text instead of learning general patterns, and then it's brittle and useless on anything new. We call that overfitting. This is why model size and data have to grow together — a bigger brain demands more to learn from, or the size is wasted. In fact, some influential research showed the field had been building models that were too big for the amount of data they were fed, and that smaller models trained on much more data could actually outperform their larger cousins. Bigger wasn't the whole answer. Better balanced was.

And that insight kicked off the most important shift in how we think about model size today. The race stopped being purely about "who has the most parameters" and became about efficiency — getting more capability out of every parameter. Three big moves came out of that. One: better training data. It turns out a smaller model trained on cleaner, smarter, higher-quality text can beat a bigger model trained on a messy scrape of the internet. Quality of education beats size of brain. Two: smarter architectures — clever designs where the model has a huge total number of parameters but only switches on the small slice it actually needs for each specific question, so you get the knowledge of a giant with the running cost of something far smaller. And three — which gets its own episode later — shrinking finished models down so they can run on a laptop or a phone without much loss in quality.

So let me give you the mental model that will make you sound genuinely sharp on this. Don't think of model size like horsepower, where more is simply better. Think of it like the size of a company. A massive corporation has enormous capability, but it's expensive to run, slow to move, and total overkill for a small job. A lean, well-trained startup can run circles around it on the right task, at a fraction of the cost. The best operators don't just ask "how big is it?" They ask "is this the right-sized tool for the job?" And that's exactly how the smartest AI builders think now. You don't use a four-hundred-billion-parameter model to sort your email; you use a small, fast, cheap one. You save the giant for the genuinely hard problems that need its depth.

This is why you're now seeing whole families of models at different sizes — a big flagship, a mid-size workhorse, a tiny fast one — all from the same company. That's not indecision. That's the whole point. Different jobs need different-sized brains, and matching the model to the task is where the real cost savings and the real performance gains come from.

Let me also quickly translate the other number you hear, so it's not a mystery: training data measured in "tokens." Remember from Episode 1, a token is a chunk of text, roughly three-quarters of a word. So when you hear a model was "trained on fifteen trillion tokens," that's just saying it read on the order of ten or eleven trillion words during training — an amount of reading no human could do in a million lifetimes. Parameters are the size of the brain. Tokens are the amount of reading that brain did. Now you can read any model announcement and know exactly what both numbers mean.

So let's bring it home. Parameters are the adjustable dials inside a model — its raw brain capacity. For years the story was simple: scale everything up together and performance reliably improves, which is why everyone raced to go bigger. But bigger costs more to train and to run, and a brain that's too big for its training data just memorizes instead of learning. So the frontier shifted from "most parameters wins" to "most capability per parameter wins" — driven by better data, smarter architectures, and right-sizing the model to the task. The sharp way to think about it isn't horsepower, it's company size: pick the right-sized tool for the job, and save the giant for the problems that actually need it.

That wraps our Foundations track. You now understand what a language model is, how words become coordinates, how the Transformer weighs them with attention, how training teaches the whole thing, and what model size really means. That's a genuinely solid base — more than most people talking confidently about AI actually have.

In the next track, we turn to memory and knowledge — why these models sometimes know things and sometimes make things up. We'll start with one of the most practical ideas of all: the context window, the model's short-term memory, and why it shapes everything about how you work with these tools. See you there.