Don't pick the smartest model every time
I ran the same research task across four models. The winner wasn't the whole story.
Once the model is smart enough, what should one optimize for?
That's the question I've been chewing on. If you read last week's newsletter you know that I think that the models are smart enough.
The benchmarks have most of the frontier models within a few percentage points of each other so the alpha in using LLM shifts to "inference", a term that is gaining traction. Inference is the trifecta of throughput, latency, cost. Speed is a function of throughput + latency.
I wanted to test a task that I would consider a baseline for "average knowledge work" so I asked for a leadership brief on remote and hybrid work in 2026 with current research, direct source URLs, measured findings separated from survey results, and a recommendation for a mid-sized technology company. I fed same prompt to every model, CLI only, outside my harness, where best output per model later blind-judged independently. Then repeated for 5 rounds.
| Model | What I trusted | What made me hesitate |
|---|---|---|
| Claude Opus 4.8 | Best blind quality read | Slightly slower and less convenient as my default driver |
| GPT-5.5 | Close enough on quality, strongest practical default | Didn't win every column |
| GPT-5.4 mini | Better brief than I expected | Too slow for this task |
| Gemini 3.5 Flash | Strongest best-run shared token throughput | Tool-heavy setup probably made it look worse than clean generation work |
My practical winner was GPT-5.5. Good enough on quality, fast enough on the run, and low-friction enough that I'd keep using it as the default driver for this kind of research pass.
I wanted to use median time to first token because it'd make a nice clean chart. The medians were obviously contaminated by tool calls, search latency, session startup, retrieval variance, and quota noise.
The results changed my hypothesis a bit. If only one model can reason through the work, use that one and stop pretending the rest of the scoreboard is relevant. But once two or three models clear the reasoning bar, the comparison moves from "which model is smartest?" to "which model gets useful work through my system with the least delay, cost, and review pain?"
The routing rule
The rule I'd use now:
| Task State | Default Choice | Why |
|---|---|---|
| Reasoning isn't sufficient | Use the strongest model | You need correctness before speed |
| Reasoning is sufficient but verification is hard | Use the model you trust most | Review cost dominates token cost |
| Reasoning is sufficient and verification is cheap | Optimize for throughput | Fast iteration changes behavior |
| Task is high-volume or exploratory | Use the cheapest sufficient model | More attempts can beat one expensive attempt |
| Model access is uncertain | Route through a harness | Availability becomes part of reliability |
The middle row is where people get themselves in trouble. If you're asking a model to design a migration plan for a fragile production system, don't get cute — use the strongest thing you can access and review it like the downside is real. But a lot of work isn't a fragile migration plan. Drafting, synthesis, research prep, structured notes, internal briefs — these have cheap verification loops. Once the model clears the reasoning bar, speed changes how often you use the tool, and that changes behavior.
When in doubt, use the HIGHEST reasoning model, remember we are using agents to save TIME, then MONEY. Human time > clanker time. Do it once, do it right.
The market is already routing this way
DeepSeek's DeepSpec repo includes DSpark, a speculative-decoding drafter that DeepSeek says improves per-user generation speed by 60–85% on V4-Flash versus its MTP-1 production baseline. I'd treat that as a signal about where the fight is heading, not independently reproduced benchmark truth.
| Company | Routing move | What they optimized for |
|---|---|---|
| DeepSeek (DSpark) | Speculative-decoding speed layer | 60–85% reported generation-speed gain on V4-Flash vs. MTP-1 baseline |
| Shopify | Fine-tuned Qwen3-32B for Flow automation | 2.2x faster and 68% cheaper than the frontier model it replaced |
| Uber Eats | Qwen2-based retrieval model | Multilingual semantic search across stores, dishes, and grocery |
| Siemens | Multi-provider AI strategy | DeepSeek + Qwen alongside US/European providers to spread supply risk |
These are all different use cases, for different teams, with different levels of public detail. But the direction is the same: companies aren't waiting for one model to absorb every workflow.
And you shouldn't either
Select the model for the task. Vary reasoning — give mini, flash, and even open-source models a try.
This is why I keep coming back to the harness: context files, prompt shape, source material, tools, evals, review gates, and routing rules. The durable asset is the setup that lets you change your mind without rebuilding the work. So for your next AI task, start with the failure mode before you start with the leaderboard. If the model is likely to be wrong in a way you can't catch, use the strongest model you trust. If you can verify the answer quickly, test the cheaper sufficient model.
Benchmarks tell you whether a model can think. Throughput tells you whether you'll use it enough to change the way you work.
Quote of the week
"An hour lost at a bottleneck is an hour out of the entire system. An hour saved at a non-bottleneck is a mirage." — Eliyahu M. Goldratt, The Goal (1984)
If your team is testing models right now, reply with the workflow and the failure cost. I'll tell you whether I'd optimize for reasoning, throughput, cost, or portability first.
— Collin
Worth Reading
DeepSpec / DSpark paper and code — DeepSeek-AI Full-stack speculative-decoding codebase with DSpark, DFlash, and Eagle3. DeepSeek reports 60–85% faster per-user generation on V4-Flash versus MTP-1. Training code, eval code, and checkpoints included.
Flow generation through natural language — Shopify Engineering Shopify fine-tuned Qwen3-32B into a tool-calling agent for Shopify Flow: 2.2x faster, 68% cheaper, weekly retraining flywheel. Best current example of a production team routing away from a frontier model because a fine-tuned open-weight model cleared the bar.
AI's brokenomics — Ed Zitron breaks down the cost-pressure leading providers face and Generative AI business model
