Hey — Steve here.

Last week we covered Computer Vision. This week, let's talk about something you've probably heard but maybe never fully understood.

This Week's Term: Multimodal

The quick version: AI that handles multiple types of input — text, images, audio, video — all at once. A Swiss Army knife instead of just a blade.

The deeper version:

Early AI was like someone wearing blinders. A text AI could only read words. An image AI could only look at pictures. A speech AI could only hear audio. None of them could do what a 5-year-old does effortlessly — see a picture, hear someone talking about it, and read a caption all at the same time.

Multimodal AI breaks down those walls. It can process text, images, audio, and video together — understanding the relationships between them. You can show it a photo and ask "what's happening here?" You can upload a chart and ask it to explain the data. You can give it a video and ask for a summary. It understands across formats.

This is a massive deal. Think about how you experience the world — you don't process it as separate text, images, and sounds. You take it all in together. Multimodal AI is getting closer to that kind of holistic understanding. GPT-4, Claude, and Gemini are all multimodal — they can see AND read AND reason about all of it at once.

Why This Matters

Because multimodal AI is what makes AI truly useful in everyday life. Take a photo of a restaurant menu in Japanese and get an English translation. Screenshot an error message and ask AI to fix it. Upload a receipt and have AI categorize your expenses. The more types of input AI can handle, the more it can help with real-world tasks.

Try It Yourself (2 minutes)

Upload a photo of handwritten notes to ChatGPT and ask it to transcribe and summarize. That's multimodal AI: processing images and generating text.

Go Deeper

• Search for "Multimodal explained simply" — you'll find great visual guides
• 🔗 Read the full SpeakNerd term page

The Nerd Corner

Multimodal AI systems process and relate information across multiple modalities (text, images, audio, video) using unified architectures or cross-modal encoders. Approaches include early fusion (combining inputs before processing), late fusion (processing separately then combining), and cross-attention mechanisms. Key models include GPT-4V, Claude 3, Gemini, and LLaVA. Challenges include grounding (connecting language to visual elements), cross-modal hallucination, and computational cost of processing multiple high-dimensional inputs.

Next week: Embedding — we'll break down what it means and why you should care.

— Steve

P.S. Know someone who'd find this useful? Forward this email. They can sign up here.