This is a reader-supported newsletter! If we lived in a world of thriving magazine and book publishing, and well-funded public institutions, I wouldn’t need to ask for a cent. Unfortunately we don’t, and I appreciate your support (do feel free to share anything you read here). In addition to the two articles a month free subscribers get by email, premium subscribers receive one bonus post a month. On that note, here’s another reminder that you can purchase my book Control Science from Verso right now.

Earlier this week, Anthropic’s Dario Amodei released a blog post on the future of AI. 

I can’t believe I’m arguing against regulation for once, but AI company’s Anthropic’s suggested regulatory framework, viewed alongside their recent actions, worries me. Claude Fable 5 released this week with a strange shortcoming. Fable runs like their much-vaunted new Mythos model most of the time… but Anthropic insists Mythos is too dangerous for public use, because it can generate cybersecurity and biosecurity threats. So if Claude detects you are getting too close to these topics, it downgrades itself to Opus, the previous model version. 

The impulse among AI critics will be to call this either a marketing tactic, insisting that something is too powerful to release in order to hype it up, or security theater, a trick by the AI firm that wants to keep its reputation as “the good one.” I think it’s something worse. Anthropic’s combination of fear and hype over AI development, linked to the “rationalist” Silicon Valley subculture, has lead the firm to believe it must be a custodian of knowledge and technological capacity generally. 

On the surface, much of Amodei’s blog post seems even-handed. “AI will soon become so capable that I worry it cannot safely be fully entrusted to either governments or companies, and there must be checks and balances on each.” But it contains a damning admission: Amodei is already thinking of governments and firms as peers. And his proposals for what both should do are alarming. 

“Frontier AI models, like airplanes,” Amodei writes, “should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety.” Right now, that might seem possible with normal regulatory tools, because frontier models take a great deal of computing power to train. It is possible for a government to restrict access to power, GPUs, and technical expertise. For now. 

But in the long term, controlling AI development would require enormous government surveillance. Precisely because of how much AI firms have invested in data centers, any efficiency improvements are worth an ungodly amount of money, so we can be sure they will be found. Controlling model development in the way Amodei calls for here would require, ironically, the kind of AI-powered government surveillance he deplores later in the post. Citizens would have to either lose access to general purpose computing– a future of only data centers and dumb terminals at home, where AI companies can play with computing power but you can’t– or have our every device surveilled constantly to make sure we aren’t doing the wrong kind of math on your graphics card. I think this in and of itself would be wrong, but even if you don’t, this regulatory apparatus could be bent toward malign ends easily.

The Fable debacle is an ominous sign of where Anthropic’s safety thinking is leading it. Originally Fable downgraded itself without informing the user. In other words, this is a company that thinks it can change what it is selling you without even notifying you– a company willing to lie to you in order to impose its vision of safety upon you. While they rolled this back thanks to public outcry, the initial decision remains telling. Even their practical judgment is questionable. Fable’s system card boasts that its safeguards have a 95% success rate—by Anthropic’s own reasoning this means there’s a 5% chance it will help someone construct a supposedly devastating cyberattack or biological weapon! They think this is worth compromising user trust over? 

Anthropic’s behavior is not a ploy for money– it looks to me like a collision of confused values, paranoia, and avarice with sheer intellectual arrogance. I would need another post entirely (which I may write later) to explain the problems with Amodei’s macroeconomic and tax analysis, which is strange. Amodei can imagine AI so powerful it can do better than humans at any task, and yet somehow thinks we can solve the problems this would result in with a few tax tweaks and a UBI. But a blinkered view is what you get when– here I check the end of the post– you consult centrist blogger Matt Yglesias instead of a macroeconomist for policy advice.

These are the narrow tastemakers influencing the future of AI. Currently, the ability to control what models exist is the ability to control how hundreds of millions of people experience the world– in many ways this is unfortunate, and I think there’s plenty of room to contest how large a role AI plays in our world, but it plays a major role now and will play some kind of role going forward. One of the reasons AI needs to actually be open is because no narrow slice of human opinion should get to determine all of that. I would not trust Silicon Valley technocrats to do this any more than I would trust white supremacists or a random DSA chapter or my book club. You definitely shouldn’t trust me (although, shameless plug, perhaps Amodei would have seen some of these problems more clearly if he had read my recent book Control Science, which you should pick up if you haven’t yet, and can also review on Amazon even if you didn’t buy it there, apparently, which you should do if you have picked it up).

Let me be clear here: I don’t think we are doomed because of AI and I don’t think it’s impossible to regulate well. I think benevolent regulation is possible. It’s just going to have to look like something we are entirely unfamiliar with. That’s alright. Amodei complains that our regulatory apparatus is too slow– this is true– but it has gotten faster. It used to take generations for human societies to develop new laws to deal with new industries. In the 20th century we managed to do it in single-digit years. We’ve slowed lately, and that’s a problem, but we can be creative again. 

There’s a lot of talk about what democratic governance means restricting– maybe banning data center construction, as Bernie Sanders has suggested (I think this is a bad idea), or restricting new model developments like Amodei argues for. We also have to think about what it means enabling. I think that includes open, transparent, and accountable AI systems, from open weight experimentation on local hardware for experts and hobbyists to user-accountable, well-governed, well-explained platforms for general users. I do not want a democracy of disempowered consumers– that’s what we have with social media, and look what it has done. I want a democracy of empowered users. Anthropic is getting in the way. 

Here’s where I’d like to tell you Amodei’s stated commitment to democracy acts as a counterweight. There is a limit to how much harm you can do if you don’t have the power to make decisions. But there are two problems with this. First, Anthropic’s mistrust of its users and the public undermines any possibility for collective governance. This is a company that had to be pressured to even let you know what model you are using. 

Second, there is a jarring disconnect between Amodei’s optimism about liberal democracy and the dismal state it’s actually currently in. Would you trust the Trump administration with any of the powers I just mentioned? “The state,” Amodei writes “has a legitimate, often existential, interest in protecting its population from internal and external threats. But granting it too much power is the road to tyranny. Modern democracies have largely managed this balance successfully, but it is an uneasy one at the best of times–” Have they? Anthropic recently (to its credit!) sued the government because it did not want them using Claude for domestic surveillance. Dario Amodei is either lying to us or lying to himself. 

Of course, that’s what ordinary CEOs do. But Amodei is not being treated like an ordinary CEO, nor Anthropic like an ordinary company, nor do they think of themselves as ordinary either. I think some of its stated values are, genuinely, good, and certainly they have helped produce a better product: Claude won’t steer you toward transphobia like Grok, and it won’t try to encourage suicide like ChatGPT 4o did. And there are no easy answers to the problems of governance and economics that AI raises. But the answers Anthropics is finding lately don’t seem like the right ones.

Anthropic’s actions and statements suggest the company is acting on a particularly pernicious combination of self-interest and ideology. The firm is ideological enough to compromise its self interest– lying to customers in order to enforce its nebulous and fallible safety regulations– and self-interested enough to compromise its values– Anthropic recently signed a deal to use Elon Musk’s Colossus datacenter in Memphis. While the harms of datacenters in general have often been exaggerated, this one is a gas turbine monstrosity poisoning the city’s air for the financial benefit of a wealthy white supremacist responsible for the deaths of hundreds of thousands of people. Musk is currently instigating race riots in Europe. 

It is not hard to imagine how executives at Anthropic this or any other dubious decision. Think of the good we can do with the money. We have to win the arms race if we want safer AI. You know Sam Altman would do it. Google’s going to do it. And after all we stood up to the government didn’t we? We’re the good guys. We know best. Every decision that marks them as not like other firms becomes another justification for acting like other firms in other areas. Anthropic is worried about what AI might evolve into. But Claude seems to be coming along quite nicely. Dario Amodei should be more worried about what his company is becoming.