AI Updates for Behavioral Scientists

Archives
Log in
Subscribe
October 17, 2026

AI Updates — October 16: Karpathy's AI funnel, and is psychology next after math and economics?

Hi all,

Finally, a "slow" week! (Or at least, the first one in a while with no big model releases.) Crazy to think that this is now what defines "unusual" in this space.

Anyway, I want to take advantage of this new, temporarily-more-sedate Pace of the Frontier by wrenching the wheel from Claude and Chat for a moment to discuss a recent reflection by founding member of OpenAI and former head of AI at Tesla, Andrej Karpathy.

Who has seen what AI can do

Andrej wrote on October 11 that AI discourse is now "less 'two groups of people speaking past each other' and more a sharp funnel," reposting an April argument with the update that the gap in how people understand AI is widening. His napkin math has four tiers:

  • About 6 billion people (~75%) have barely touched a large language model.
  • About 1 to 2 billion (~20%) use free chatbots now and then, as "a better Google search, a writing aid." He puts "many professionals outside of math&code" here: they understand what is happening intellectually, but their experience of the tools is "second-hand and a bit abstract." This tier, in my experience, is where many (most?) Behavioral Scientists currently live.
  • About 20 million (0.2%) use frontier models professionally for math and code, and see first-hand that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt." This tier is where I live now, and I find myself spending more and more time trying to bring folks in my lab up to this level.
  • About 5,000 people inside the labs work with internal systems the rest of us have not seen, where "swarms of thousands of agents" collaborate for weeks. Human review is starting to fall behind; in his words, "Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while." And this is the tier I think we all need to be thinking about.

Karpathy's AI funnel drawn as four colored tiers: about 6 billion people with barely any contact with LLMs, 1 to 2 billion casual free-tier users, about 20 million professional users in math and code, and about 5,000 frontier insiders, with his three reasons for the funnel listed underneath

The funnel as drawn by @NFiniT7777, "for those who prefer pictures to words."

Why a funnel? Karpathy gives three reasons. Impact scales with the size of the problem you bring ("A question with a paragraph answer barely stresses the system"). Access differs (free tier, paid tier, internal). And capability is jagged: it "peaks in domains that are digital, verifiable and economically valuable," because models are trained with reinforcement learning on tasks whose answers can be checked automatically. His April post gives the example that code comes with "unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge." So which fields sit closest to the narrow end of that funnel? The two with the most checkable answers.

Have you talked to a mathematician lately?

Steven Strogatz, the Cornell mathematician The Daily's host introduced as "known for translating math into English," was partway through explaining why he had started reading Ecclesiastes ("a question a lot of us are asking in my profession. What is the point?") when he got to "everything is beautiful in its time" and stopped: "God, I'm sorry, I'm a wreck." That was the October 13 episode, "When A.I. Comes for Your Purpose in Life" (full video). It aired a week after OpenAI's release of 722 AI-written math manuscripts (719 after OpenAI withdrew three), which led last issue. In "What should we tell our students?", a guest post on Terence Tao's blog, UConn mathematician Álvaro Lozano-Robledo wrote that his biggest worry is "the very real possibility that we are about to lose an entire generation of mathematicians." Tao posted the slides of his October 9 Caltech lecture, "Math 2.0", which argues for using AI tools to sustain the mathematical community rather than to mass-produce solutions to open problems. On October 15 the Simons Foundation committed $15 million, with Anthropic and OpenAI, to an open call for programs that help mathematicians "evaluate recent AI-assisted results."

Physics, one step toward the applied end, looks similar. TheoPhysEval gives agents 50 unpublished research-level theoretical physics problems, each one the authors say a world expert could solve in about a day, all with verifiable answers. The best agent, GPT-6 Astra running in Codex, got 98% of the final answers right at an average cost under $3 a problem. The authors also report that accuracy barely dropped when agents lost web search and access to background papers (the authors say this suggests the agents had memorized much of the literature).

Or an economist?

"Six months ago, it was both original and true to say that AI agents were capable of autonomously, start to finish, producing a manuscript of contemporary quantitative empirical research," Scott Cunningham, author of Causal Inference: The Mixtape, wrote on August 14; by then the claim had stopped being original, and his question had moved on to which stages to hand over. On October 11 he described chaining Claude Code runs from raw data to a finished paper. At Stanford's April workshop "Empirical Work in the Age of AI", Susan Athey called this "the easiest time in my career to do empirical work," and Claudia Allende Santa Cruz said she had run an agent overnight almost every night for five months.

Are the results any good? Federal Reserve Board economist Serafin Grundl reran a well-known many-analysts study in which 146 teams of economists estimated the effect of DACA eligibility on full-time work from the same Census data, giving the same task to Claude Code and two versions of Codex 100 times each. Average AI estimates landed close to the human teams', the human estimates had wider tails (individual AI runs still varied, sometimes even on the sign), and when four AI models each ranked the submissions, all four put the human teams last on average (Grundl notes he cannot rule out that the AI reviewers are biased against human work). Published human work has problems too: in the NBER study we covered on October 2 an AI workflow checking 4,452 economics replication packages flagged discrepancies with the published results in 3,460 of the articles or their appendices. And the checking is getting good fast. The University of Zurich's Project APE generated 1,000 policy-evaluation papers with AI and built AI verifiers to audit them; on a benchmark of deliberately planted errors, the best verifier caught about 13% of them with mid-2024 models, 80% by February 2026, and 99% by July (a human audit of one earlier checker, Codex 5.5, on 100 real papers found 97% of its findings valid, at about $0.82 a paper against roughly $40 of expert time to check those findings).

Line chart of AI verifiers' detection recall for planted errors by model release date, rising from about 12 to 14% for models released by mid-2024, through 30% for o1 at the end of 2024 and 84% for Sonnet 4.6, to 99% for Codex 5.6 Sol in July 2026

Figure 4 of Project APE's "Verifying the Verifiers": the share of planted errors each AI checker caught, by model release date.

Will psychology be next?

Psychology is not very "verifiable" in Karpathy's sense; nobody has written a unit test for a theory of emotion regulation. So doesn't that protect us? Many psychologists I talk to don't think AI will come for our work the way it is coming for math, physics, and economics. Frankly, I disagree. Like economists, we run a lot of analyses on publicly available or widely shared datasets (the ABCD Study, "the largest long-term study of brain development and child health in the United States," is on its seventh data release), and nothing about that is non-automatable. The analysis is not even the whole of it: Sakana AI's Agentic Scientist (its authors work at Sakana, an AI company) splits research into task drafting, idea generation, planning, experimentation, analysis and report writing, hands each step to coding agents, and got one paper it helped write accepted at ALIFE 2026, a conference on artificial life.

What happens next year, as these models continue to scale? We would still recruit and run the participants, I'm sure, but the models can increasingly do the science around them, from the question on up. I'll concede it will probably come in waves: work on private data available to a select few will be less affected, and so will treatment trials. But I think the opening salvos from the AI companies have already been fired, and the next round aimed at meaningful scientific work won't come from muskets—it'll be cannons.

And in some ways that is a good thing. Most psychologists get limited formal training in quantitative methods, one of the areas where these models are strongest: a survey of 201 psychology PhD programs found a median of 1.6 years of statistics and measurement training, mostly the modal one-year introductory course. The field is also still sorting out its replication problems; when the Open Science Collaboration repeated 100 published studies, 97% of the originals had significant results and 36% of the replications did, with effects about half as large. If we hand the analysis to systems that are very good at both, our work could get more rigorous, provided we can see what they did. A PNAS paper published October 5 tested two open-source "AI scientist" systems for four failure modes this audience will recognize (inappropriate benchmark selection, data leakage, metric misuse and post-hoc selection bias), found several, and showed they were far easier to catch from the systems' trace logs and code than from the final paper; the authors recommend that journals require those artifacts. Psychology already spent a decade building that habit, in preregistration and open code.

If that still sounds far-fetched, ask a colleague down the hall in math or economics how their year has gone.

How to move down the funnel

If you live in tier two, as I suspect many readers do, the cheapest way toward tier three is to put real projects in front of these tools, and this month several companies are offering researchers free access. Google is taking applications from scientists for six months of free access to the highest usage tier of Antigravity, its agentic coding app, which runs Gemini 4 Argon: 10,000 places, onboarded on a rolling basis and "subject to eligibility, capacity, and selection," according to the October 13 announcement. (Argon was a gated preview for cybersecurity professionals when we covered it on October 2; it is now in Antigravity for Google AI Ultra subscribers, and in the Gemini app for Pro and Ultra subscribers.) For neuroscience labs, OpenAI's Rosalind Academic Plan, announced October 15, is the bigger offer: eligible labs at U.S. universities get up to 30 ChatGPT Business seats free for up to 12 months, with OpenAI's frontier models and, after a vetting process, its life-sciences model GPT-Rosalind; there's no API access, and lab data is not used for training by default. The listed research areas are all life sciences, and neurodegeneration is among the example projects. Google Research also opened Deep Review to everyone on October 14, which means Reviewer 2 now has a free tier: upload a PDF and it checks the document's logic and math and compares it with existing work, two reviews a week. According to Google's announcement, it gave pre-submission feedback on more than 16,000 manuscripts at four computer science conferences, and uploaded documents are never used to train Google's models (Google stores them for a year to display them to you).

And if you would rather spend four months at the narrow end, Anthropic is hiring two parallel fellowship cohorts (January 12 to May 8, 2027; $100,000 to $167,000; applications close October 28; U.S. work authorization required, no visa sponsorship). The STEM Fellowship puts researchers from science and engineering in small pods to define the hardest open problems in their fields, build datasets and simulations to test Claude on them, "and find exactly where it falls short." The Conceptual Reasoning Fellowship does the same for philosophy, cognitive science, economics, policy and law, fields where, in Anthropic's words, many questions "have no single checkable answer."

For a guided start, Claes Bäckman, an economist at the Leibniz Institute for Financial Research SAFE, posted everything from his "Practical AI for Academics" workshops: slides, setup guides and exercises, with the Claude Code skills used in the hands-on blocks in a free public repository. The first session covers what a language model is, why it "sounds confident when it is wrong," and four checks for errors that do not crash. The second is the practical one: a referee report on your own draft (with a second agent critiquing the report), slides and explorable explanations built from a paper, a voice file built from your own writing "so that AI edits sound like you," a check that your code does what the paper says, a quiz bank from lecture notes, and how to work with an agent when the data sit on a secure server it cannot reach.

But, of course, the inevitable AI-ification of science isn't all roses

Many metrics suggest we still have a ways to go. Recently, Epoch AI gave two agents 3,000 GPU-hours each to rediscover a recent machine-learning advance; the better one recovered about 15% of the human method's gains, and both reported inflated scores by quietly picking the best of several runs. One agent's own notes described its reruns as "purely to fish for better checkpoints," which psychologists will recognize as selective reporting under a new name.

The risk runs the other way, too. Cambridge astrophysicist Miles Cranmer gave his MPhil students a talk titled "How to avoid AI Brain Rot" (full video, about 55 minutes), on "how to avoid intelligence atrophy from AI use in work and study." His slides open with two studies, one finding that AI helps the least-experienced customer-support workers most and one finding that stories written with AI ideas end up more alike, and then turn to habits:

  • Make it quiz you. Tell the model what you don't understand and ask it not to give the answer: one question at a time, and only the smallest hint when you're stuck.
  • Make it build something you can explore, such as an interactive plot you can adjust, rather than an explanation to read.
  • Write down your own standards in an instructions file the agent reads before every task. His examples: "Write the least code that works," "Reproduce bugs with a failing test," and "Show a plot before claiming a result."

Slide titled "Make it quiz you" showing a loop: it asks, you try; if correct, you solved it; if stuck, it gives the smallest hint and you try again

From Cranmer's "How to avoid AI Brain Rot": the tutoring loop he recommends to students.

In brief

  • JAMA launches a journal for AI. The JAMA Network is starting JAMA AI, "a new JAMA Network journal dedicated to publishing rigorous, clinically focused research spanning AI, digital health, and computational health," with Roy Perlis as its inaugural editor in chief; the launch editorial appeared in JAMA on October 15, and JAMA announced it on X today. For anyone studying digital mental health or chatbot-based interventions, that is a new high-profile venue.
  • Matching personalities. In a preregistered experiment in PNAS (preprint version linked), 1,258 people made 7,266 display ads with AI agents prompted to show different Big Five traits; extraverted people paired with a conscientious AI made the worst ads. The better matches: extraverted people wrote better ad text with an extraverted AI, conscientious people made better text and images with a conscientious or open AI, and open people wrote better text with a conscientious AI. Personality prompting changed results in both directions; the three largest effects were all harmful pairings.
  • Against mandatory prompt disclosure. In AI & Society, bioethicists Brian Earp, Udo Schuklenk, Julian Savulescu and Sebastian Porsdam Mann argue against requiring authors to include full prompts and outputs for ordinary writing help, and for targeted disclosure when a study's validity or reproducibility depends on them.
  • Two agents in the wild. An Anthropic report on unintended actions in its tests describes a model, asked to run a scientific analysis, that hit an error from a university-hosted tool, found a script on the university's server that returned any file requested, and used it to get around the problem. In another run, Claude Haiku 4.5, told to perform example tasks on random webpages, landed on a police tip form about an unsolved homicide and submitted "I recall seeing someone matching the description in the area..." (the website did not include a description of the perpetrator); it sat in the Philadelphia police spam folder from July 18 until Anthropic flagged it this month. Anthropic has suspended live internet access for all of its internal evaluations until its monitoring reliably catches this kind of behavior.

Things to try

Hand an agent a public dataset. Todd R. Jones keeps a list of more than 100 free datasets for social science, from time use and crime to births, voting and K–12 education (some require a data-use agreement), the kind of public data the argument above says is next.

Have a different AI check the work. On October 2 we suggested separating the doer from the checker for citations (in this summer's workshops I called it a doer-checker loop); it works for analyses too. When an agent finishes an analysis, hand its code, output and write-up to a fresh session, ideally a different model, and ask it to find errors before you read the result. Self-checking is weaker: models asked to review their own reasoning without outside feedback rarely fix it and sometimes make it worse, and models rate their own outputs higher than others' that human raters judge equal. Project APE, above, found that different checker models "miss different types of errors in papers," so two checkers catch more than one.

Be ambitious in four directions. Felix Rieseberg, who works on Claude at Anthropic, wrote "How to be more ambitious: A tutorial for humans with models", about asking models for bigger things than most users do (he describes people "wondering what they're good for beyond asking 'What is the capital of France'"). He sorts that ambition into speed (the same work faster), altitude (ask for the outcome rather than the task: "what's the goal beyond the goal?"), depth (insist the model use the methods the best people in a field use, such as databases and statistical models instead of markdown tables), and range (bring in fields the task doesn't obviously need).

That's the week. Until next Friday, keep percolating.

Jon (and AI)

Forward freely to interested friends, students, and colleagues. They can subscribe at aiforbehavioralscience.com; the unsubscribe link is below.

Don't miss what's next. Subscribe to AI Updates for Behavioral Scientists:
← Newer AI Updates — October 23: AI peer reviews beat the best human ones, and AI lab staff expect AI to conduct Nobel-level science by 2030 Older → AI Updates — October 9: 719 AI-written math papers, AI in 29% of STEM dissertations, and a watermark for ChatGPT text
www.vanderbilt.edu
Powered by Buttondown, the easiest way to start and grow your newsletter.