Embeddings

Welcome back to the NEXUS AI Literacy Series. In the last episode, we landed on a big idea: a large language model is an autocomplete that got so good at predicting the next word that it accidentally learned to understand the world. But I left a question hanging. A computer doesn't actually know what a word means. A computer only knows numbers. So how on earth does a pile of math end up "understanding" that a king and a queen have something in common, or that Paris and France go together? That's what this episode is about, and it's one of the most beautiful ideas in all of artificial intelligence. It's called embeddings.

Let me start with the problem. Imagine you're a computer, and I hand you the word "dog." To you, that's just three letters — d, o, g. There's nothing in those letters that tells you a dog is an animal, that it's furry, that it's related to a wolf, that it's the kind of thing a person might love. The letters are just symbols. If I also hand you "cat" and "carburetor," you have no way of knowing that "dog" and "cat" are deeply related and "carburetor" is off in a completely different world. Alphabetically, "cat" and "carburetor" are actually closer together — they both start with C-A-R... no wait, c-a-t and c-a-r. They look more similar than "dog" and "cat" do. So if all you have is spelling, meaning is completely invisible to you.

Here's the trick that cracked it wide open. What if, instead of treating a word as a string of letters, we represented every word as a location in space? A set of coordinates. Think about how we locate a city on Earth with latitude and longitude — two numbers, and suddenly every city has a precise spot, and cities that are near each other have similar numbers. Now imagine doing that for words. Every word gets coordinates. And we arrange the whole space so that words with similar meanings end up near each other.

In this space, "dog" and "cat" would be close neighbors — both pets, both animals, both furry. "Puppy" would be right next to "dog." "Carburetor" would be way across town, somewhere near "engine" and "transmission." This map — this enormous space where every word is a point and distance means similarity of meaning — that's what we call an embedding space. And the coordinates for each individual word, that list of numbers, that's the word's embedding.

Now, here's where it gets a little mind-bending, and I want you to just relax and go with it. On Earth, we use two numbers — latitude and longitude — because the surface of the Earth is two-dimensional. But meaning is way more complicated than a flat map. A word isn't just "more like a dog" or "less like a dog." It can be more or less alive, more or less formal, more or less positive, masculine or feminine, concrete or abstract, and a thousand other shades all at once. So an embedding space doesn't have two dimensions. It has hundreds, sometimes thousands. Every word is a point floating in a space with, say, fifteen hundred dimensions.

Don't try to picture that. Nobody can picture a thousand-dimensional space, not even the people who build these systems. Just hold onto the simple version: every word is a point on a giant map, and closeness on that map means closeness in meaning. The extra dimensions just give the map enough room to capture every subtle flavor of meaning at the same time.

Now let me show you the single most famous, jaw-dropping consequence of this idea, because when researchers first saw it, they could hardly believe it. Because words are now points in space, you can do arithmetic with them. Actual math, with meaning.

The classic example goes like this. Take the coordinates for "king." Subtract the coordinates for "man." Then add the coordinates for "woman." And you know what point you land closest to in the space? "Queen." Let that sink in. King, minus man, plus woman, equals queen. The model was never taught that. Nobody sat down and programmed the relationship between kings and queens. It emerged, all on its own, from the geometry — because in learning to predict language, the model arranged the space so that the direction you travel to get from "man" to "woman" is the same direction you travel to get from "king" to "queen," and from "uncle" to "aunt," and from "actor" to "actress." Gender became a direction you can walk in. The same way "Paris" minus "France" plus "Italy" lands you right on "Rome." The relationship "capital city of" became a direction too.

That's the moment the whole field realized something profound was happening. Meaning has a shape. Relationships between concepts turned out to be directions and distances in space, and once you put words into that space, the relationships fall out as pure geometry.

So where do these coordinates come from? Nobody assigns them by hand — there are too many words and far too many dimensions. They're learned, using the exact same principle from Episode 1: predicting words from their neighbors. The system reads enormous amounts of text and notices which words show up in similar contexts. "Dog" and "cat" both appear near "pet," "vet," "fur," "feed," "leash." "King" and "queen" both appear near "throne," "royal," "crown," "rule." The model's logic is essentially: words that keep similar company must mean similar things, so I'll place them near each other on the map. Do that across billions of sentences, and the entire geography of human meaning slowly arranges itself, automatically, with no human ever drawing the map.

And here's why you should care, beyond it being a gorgeous idea. Embeddings are the foundation underneath an enormous amount of the AI you actually touch. When you search for something and the results understand what you meant even though you used different words — that's embeddings, matching meaning instead of matching letters. When a store recommends a product "similar to" the one you're looking at — embeddings. When we get to vector databases in a later episode — that whole technology exists purely to store these coordinates and find nearest neighbors fast. Embeddings are also the very first thing that happens inside a language model: before it can reason about your sentence, it converts every word into its coordinates, because numbers in a meaningful space are the only thing the math can actually work with.

So let's bring it home. A computer can't understand words, only numbers. The breakthrough was to turn every word into a set of coordinates — a point in a vast, high-dimensional space of meaning — arranged so that similar meanings sit close together. We call that an embedding. Nobody draws the map; it's learned automatically by noticing which words keep similar company. And once meaning becomes geometry, something almost magical happens: relationships between ideas turn into directions you can travel, and you can literally do arithmetic with concepts. King minus man plus woman equals queen.

Hold onto this image, because it's the bridge to everything next: language, to a machine, is a landscape. Words are places. Meaning is distance and direction.

In the next episode, we finally open up the engine itself — the Transformer, and the idea called "attention" that made the whole modern AI revolution possible. We'll figure out how the model decides which words in a sentence matter to each other, and why reading everything all at once changed the game. See you there.