Multimodal

Welcome back to the NEXUS AI Literacy Series. This episode closes out our Agent Era track, and it's about a shift that's easy to underestimate but genuinely changes what AI can do. Up to now, almost everything we've discussed has been about text — the model reads words and writes words. But the newest models aren't limited to text anymore. They can see images, hear audio, look at video, read charts and diagrams. We call this multimodal AI — one brain, many senses. By the end of this episode, you'll understand what multimodal really means, how it's possible, and why it's a much bigger deal than just "now it can look at pictures."

Let me start with the word itself, because it demystifies the whole thing. A "modality" is just a type of information — a form that data comes in. Text is one modality. Images are another. Audio is another. Video, another. For most of AI history, a given model could handle exactly one. You had text models that worked with words, and totally separate image models, and separate speech models. Each lived in its own world and couldn't talk to the others. A multimodal model breaks down those walls. It's a single model that can take in, and sometimes produce, several of these types at once. Multi-modal. Many forms of information, one mind.

So how is that even possible? How can the same system that processes a sentence also process a photograph? And here's where it connects beautifully back to something we already learned — Episode 2, embeddings. Remember the big idea: a model turns words into coordinates, into points in a space of meaning. Well, here's the elegant trick at the heart of multimodal AI. You can do the same thing with an image. You can turn a picture into coordinates in a space of meaning, too. And the magic move — the thing that makes it all work — is training the system so that an image and the words describing it land in nearly the same place in that shared space.

Let me make that concrete. A photograph of a golden retriever, and the text "a happy dog" — a multimodal model maps both of those to almost the same spot in its meaning-space. The picture of the dog and the phrase about the dog become neighbors, because they mean the same thing. Once images and text live in one shared space of meaning, the model can move fluidly between them. Show it the photo and ask "what is this?" — it finds the nearby words: "a happy golden retriever." That's the foundation. Everything translates into a common language of meaning, regardless of whether it arrived as pixels or letters or sound. The senses are different doors; they all open into the same room.

And once you have that, the capabilities that unlock are remarkable, and very practical. Let me give you a tour. You can show the model a photo and have a real conversation about it — "what's in this picture, what's wrong with this diagram, what does this chart suggest." You can hand it a screenshot of an error message and it reads the screen and helps you fix it. You can show it a photo of the inside of your fridge and ask what you can cook. You can give it a graph and have it interpret the trend, or a photo of a whiteboard full of messy notes and have it transcribe and organize them. It can listen to a meeting and summarize it, look at a product photo and write the listing, watch a short video and tell you what happened. The model isn't just reading your words anymore — it's perceiving the actual stuff of your world.

Let me give you the analogy that captures why this is such a leap. Imagine a brilliant advisor you could only ever communicate with by writing letters back and forth. Smart as they are, you have to describe everything to them in words — "there's a chart, the line goes up then dips in March, the bars are blue..." Painful, lossy, slow. Now imagine that same advisor can suddenly sit in the room with you, look at the chart on the screen, listen to you talk, watch what you're pointing at. The intelligence didn't change — but the bandwidth between you and it just exploded. That's the jump from text-only to multimodal. We stopped having to translate the whole world into words before the AI could engage with it. We just show it.

And this matters enormously for where AI is going, for two reasons worth understanding. First, it's how AI moves into the physical, visual world we actually live in — most of human experience isn't neatly written down in text; it's things we see and hear. A text-only AI is blind and deaf to the vast majority of reality. Multimodal AI can finally engage with it. Second — and connect this back to last episode — it's a massive enabler for agents. An agent that can see is dramatically more capable. An agent that can look at your actual computer screen, see the buttons, read what's displayed, and act accordingly is a completely different animal than one fumbling blindly. The ability to perceive a screen or a camera feed is foundational to AI that can operate software and eventually navigate the real world. Sight isn't a feature bolted on — it's what lets an agent operate in environments built for human eyes.

Now let me be honest about the limits, because the same cautions from this whole series carry over, sometimes amplified. A multimodal model can absolutely misread an image with full confidence — it's still predicting plausible interpretations, so it can describe something in a photo that isn't actually there, the visual cousin of a hallucination. It can misread blurry text, miscount objects, miss a subtle but crucial detail. So everything we said about verifying what matters applies just as much when the input is a picture as when it's a paragraph — maybe more, because a confident description of an image feels so authoritative. And reasoning about images is genuinely harder than reasoning about text in some ways; these systems are powerful but not infallible eyes.

So here's the balanced takeaway: multimodal is one of the most practically useful frontiers in AI right now, because it lets these tools engage with the visual, audible world instead of demanding everything be typed out first. It supercharges agents by giving them sight. And it carries the same need for a careful human eye on anything important — a confident misreading of an image is still a misreading.

So let's bring it home. A modality is a type of information — text, images, audio, video. Multimodal AI is a single model that can handle several at once, and it works by mapping all of them into one shared space of meaning, so a picture of a dog and the words "a happy dog" land in nearly the same place. That lets the model move fluidly between seeing, hearing, and describing — analyze a photo, read a screenshot, interpret a chart, summarize a meeting. It's the leap from an advisor you write letters to, to one sitting in the room with you — and it's foundational to agents that can actually see and operate in our visual world. Same caution as always: it can confidently misread, so verify what matters.

And that closes our Agent Era track. You now understand how AI gets hands through tool use, how it becomes an autonomous worker as an agent, how MCP standardizes the plumbing, and how multimodal gives it senses. Put those together and you can see the shape of where this is all heading: AI that perceives, reasons, and acts in the real world.

In our final track — Business and Strategy — we step back from how it works and look at how to think about it as a leader. We'll start with a question every executive eventually asks: why is AI so expensive to run? Understanding the economics will change how you make decisions about it. See you there.