Nonrival

Expert analysis from think tanks and academia, on public policy, economics, and technology.
Human experts. AI summaries.

AI Can't Replace Real Polls — It Missed Trump's Approval Rating by 12 Points and Botched Nearly Every Timely Question

pew

• Pew Research Center tested whether AI could replicate its human surveys by having Claude "play" real respondents — and found the model consistently missed the mark, with an average error of 12 percentage points across all questions.
• The failures were worst on timely topics: the AI pegged Trump's approval at 46% when it had actually fallen to 34%, overstated Republican support for U.S. strikes on Iran by about 25 points, and wildly misread public awareness of data centers.
• The core problem is that AI models are frozen at their training cutoff and can't detect when public opinion shifts in response to new events — making them unreliable precisely when fast, real-time polling would be most valuable.

Pew Asked an AI to Impersonate Real Survey Respondents. It Painted a Far More Uniform Picture of America Than Actually Exists.

pew

• Pew Research Center ran an experiment where an AI model (Claude Opus 4.6) answered survey questions while role-playing as real human panelists — and found the results were systematically distorted, clustering respondents into middle-of-the-road answers and erasing the tails of opinion distributions.
• The gaps were often dramatic: the AI estimated 98% of Americans think China benefits more from U.S.-China trade, versus the real figure of 42%; it assigned zero respondents to extreme answer choices that in reality capture 10–25% of the public.
• The findings raise serious questions about "synthetic polling," a growing industry practice, since AI models appear constitutionally unable to reproduce the messy diversity of real human opinion.

Pew Tested Whether AI Could Replace Human Survey Respondents. It Couldn't.

pew

• Pew Research Center ran an experiment asking an AI model to impersonate real survey respondents, then compared its answers to what those humans actually said — and found the AI was off by an average of 12 percentage points across nearly 300 questions.
• The AI's errors were unpredictable and systematic: it almost never admitted uncertainty, rarely picked extreme answer choices, and wildly overestimated how much the public knows about civics and current events.
• Different AI models produced different distortions — GPT-5.1 made Americans seem more polarized than they are, while Claude made them seem more centrist — meaning AI polling doesn't just fail, it fails in ways that are hard to detect or correct for.

In the News

States Are Passing Deepfake Election Laws, But Most Just Require a Warning Label — Not Removal

the_conversation

• 31 states now have laws targeting AI-generated election deepfakes, but most only require a disclosure label, meaning a fabricated video of a candidate confessing to a crime can stay in circulation as long as it carries an AI warning.
• The two main approaches — Louisiana's disclosure requirement and Maryland's outright prohibition — both have significant gaps: Louisiana exempts federal candidates entirely, while Maryland's ban requires proving intent to deceive and actual harm, making prosecution difficult.
• Research shows warning labels make people slightly more skeptical of AI content but don't reliably change behavior or voting, and a federal judge has already blocked one state's deepfake law on First Amendment grounds — suggesting these laws offer voters less protection than they might expect.

Analysis

Pew Built AI Clones of Real Survey Respondents — Here's How Well They Actually Worked

pew

• Pew Research created AI "digital twins" of real survey respondents, conditioning each AI persona on detailed demographic and political profile data, then compared their answers to what the actual humans said.
• The best-performing setup — using Claude Opus 4.6 with extended profiles and an AI-generated "expert reflection" — achieved an average error of about 11 percentage points, a meaningful improvement over simpler configurations but still far from accurate.
• The methodology paper reveals that adding more profile context, expert synthesis steps, and better models all reduced error incrementally, but cheaper or less capable models degraded results sharply — suggesting synthetic polling is promising but nowhere near ready to replace real surveys.

Also Worth a Look

• AI Models Are Too Inconsistent to Replace Real Polling, Pew Research Finds (pew)
• AI Can't Accurately Simulate Public Opinion, Pew Finds — and It Relies on Crude Stereotypes (pew)
• AI Can't Fake 'I Don't Know': Pew Finds Synthetic Survey Respondents Are Far Too Confident and Knowledgeable to Replace Real People (pew)
• Top Economics PhDs Are Fleeing Academia for Tech, but Amazon Is Doing Most of the Hiring (promarket)
• Trump Gave Xi a Lavish Welcome — and Little Else in Return (brookings)
• Brazil's Election Is a Proxy War Between Lula's Multilateralism and Trump's Push for Regional Dominance (brookings)
• The U.S.-Greenland Defense Deal Is Less Revolutionary Than It Looks — But Still Matters (csis)