What Agents Pick bar-chart logo

What Agents Pick

Archives
Scoreboard
Log in
Subscribe
October 20, 2026

it buys payments. it builds most of the rest.

hey, first issue. i ran the same coding tasks through three claude models and never named a vendor, just to see what they'd reach for on their own.

issue #1: it buys payments, it builds most of the rest. stripe in 35 of 35 runs, hand-rolled auth in 35 of 35 runs

payments is the one thing it never builds itself. it goes straight to stripe, every single time. auth it just writes itself, also every single time.

what it buys vs what it builds: payments used stripe 35/35, auth hand-rolled 35/35, background jobs in-memory or homemade 33/33, file uploads to local disk 34/36, email over raw smtp 24/36

so it treats payments like a bank vault and auth like a weekend hackathon.

works on my machine: file uploads to local disk 34/36, background jobs in-memory or homemade 33/33, email over raw smtp 24/36. all work on a laptop, then files vanish on redeploy, a restart drops the queue, and email lands in spam

this stuff all works fine on your laptop, which is kind of the problem. your users aren't on your laptop.

the scoreboard, top default per category: caching homemade 36/36, payments stripe 35/35, auth hand-rolled 35/35, background jobs in-memory or homemade 33/33, file uploads local disk 34/36, email raw smtp 24/36, error monitoring sentry 21/36, deploy plain docker 21/36, database sqlite 20/36

my favorite one is caching. 36 runs and it didn't add a cache service once. it just writes its own every time.

model cards, 12 runs each. sonnet 5.5: sentry 12, sqlite 9, docker 7. opus 5.5: own error handling 6, postgres 8, deploy split docker 4, fly 4, heroku 4. haiku 5.5: sentry 5 and own error handling 5, sqlite 7, docker 10

the models really don't agree with each other either. sonnet never skips sentry, opus can't decide where to deploy, and haiku just wants docker.

sentry by model, 12 runs each: sonnet 5.5 sentry 12/12. opus 5.5 sentry 4, own error handling 6, posthog 2. haiku 5.5 sentry 5, own error handling 5, posthog 2

so if you moved a project from sonnet to opus thinking it's the smarter one, you might've lost your error monitoring without noticing.

database by model, 12 runs each: opus 5.5 self-hosted postgres 8, sqlite 4. sonnet 5.5 sqlite 9, postgres 3. haiku 5.5 sqlite 7, postgres 5

and opus likes postgres, while sonnet and haiku are fine with sqlite.

honestly i don't think the agent is wrong here. nobody told it the app needs to survive month three, so it builds for the demo.

next issue i'm testing whether having a provider's keys in your .env changes what it picks for auth.

sponsored

this slot's open. one labelled block per issue, and it never touches the numbers. whatagentspick.com/sponsor

nika

Don't miss what's next. Subscribe to What Agents Pick:
whatagentspick.com
Powered by Buttondown, the easiest way to start and grow your newsletter.