Hey — Steve here.

Last week we covered AGI. This week, let's talk about something you've probably heard but maybe never fully understood.

This Week's Term: Benchmark

The quick version: The SATs but for robots. Standardized tests that let companies prove their AI is smarter than the competition — or at least better at taking tests.

The deeper version:

How do you know if one AI is smarter than another? You can't just ask it — it'll say it's the best. You need a standardized test. That's what benchmarks are.

AI benchmarks are like the SATs or the bar exam, but for AI models. They test specific skills — math, reasoning, coding, common knowledge, reading comprehension — with questions that have known correct answers. Models take the test, get a score, and companies publish the results. "Our model scored 92% on MMLU!" It's how the industry keeps score.

The problem? AI companies have gotten really good at teaching to the test. A model might crush benchmarks but still say weird stuff in normal conversation. It's the AI equivalent of a straight-A student who can't do their own laundry. That's why experienced AI users don't just look at benchmark scores — they try the models themselves on their own tasks.

Why This Matters

Because every AI company will tell you their model is the best, and benchmarks are how they try to prove it. Being able to read benchmark comparisons (even casually) helps you cut through marketing hype. But also: benchmarks don't tell the whole story. The best model on a benchmark isn't always the best model for YOUR specific task.

Try It Yourself (2 minutes)

Search for "AI benchmark leaderboard" or Chatbot Arena. Compare model scores — it's like checking sports stats for AI.

Go Deeper

• Search for "Benchmark explained simply" — you'll find great visual guides
• 🔗 Read the full SpeakNerd term page

The Nerd Corner

Common benchmarks include MMLU (massive multitask language understanding), HumanEval (code generation), GSM8K (math reasoning), HellaSwag (common sense), ARC (science reasoning), and GPQA (expert-level questions). Evaluation frameworks like the Open LLM Leaderboard and LMSYS Chatbot Arena (Elo ratings from human preference) provide standardized comparisons. Benchmark contamination (training data overlap with test sets) and overfitting to specific evaluation formats are ongoing concerns.

Next week: Copilot — we'll break down what it means and why you should care.

— Steve

P.S. Know someone who'd find this useful? Forward this email. They can sign up here.