There’s a widespread belief that artificial superintelligence (ASI) is inevitable. In this post, I argue that this is false, and believing it is upstream of a lot of bad discourse.

Why do people believe it?

It’s easy to believe. Samuel Butler, already in 1863, observed that machines are improving faster than humans are improving. Today, AI systems are gaining capabilities much faster than humans: in five years we went from systems that can barely string together a sentence to autonomous cybercrime.

Biological nervous systems have many virtues, like being power-efficient (because they run on Brownian motion, i.e. for free), but they were designed by evolution, a greedy optimizer that can’t plan or readily backtrack. Given what we know about the physical limits of computation and exploratory engineering, it is reasonable to think we can do better, and build machines that do the work of the brain, but better, faster, cheaper, and with fewer transitive dependencies (a single human needs the entire biosphere to live, computers need only electricity). If you believe in natural selection, and it’s hard not to, then it seems reasonable to think that, in the long run, smart, fast, and cheap will outcompete dumb, slow, and expensive.

Any rigorous thinking about the long-term future leads to extreme outcomes. It cannot, by construction, produce normal or mundane futures.

“Rigorous thinking” means you have a model: a formalism for representing the state of the world, and a transition function to advance the state. You put the world into the formalism, and turn the crank to get successive states of the world. Models can be qualitative or quantitative. Most thinking about the future is qualitative and informal: people think in terms of trends over vague qualities like “intelligence”. For example, your model might be:

1. Currently, humans are smarter than computers.
2. Computers are getting smarter faster than humans are getting smarter.

From this, you can derive:

So, I have this little app, Epoch, that I use to keep a journal. It’s a tiny Rust web app that runs as a systemd service and uses SQLite as the database. I use a desktop and a laptop regularly, and use Syncthing to synchronize them, including Epoch’s database. That way I can use the app on both devices without needing a server to synchronize them, the tradeoff being that I have to make sure the sync is finished before performing any mutations.

But I had this bug. Say I edit today’s entry on the laptop, come home, wait for Syncthing to finish, then I’d open today’s entry on the desktop, and the text would be missing. It’s not that the server is holding a lock on the file and preventing the sync: opening the database with the sqlite3 command line tool shows the new text is there. If I restart the server, Epoch can read the new text.

My mental model was:

The rusqlite Connection object points to the database file. Syncthing swaps the file’s contents from under it. Subsequent queries go to the new file.

The implicit promise of modernity is: we have to work really hard for a few centuries, but eventually we’ll get to the Star Trek future where we’ve automated all the drudgery, and all work is meaningful, heroic, and dignifies the spirit. We will be artists, scholars, captains of the starship Enterprise, etc. There’s a few problems with this idea:

1. “Drudgery” and “meaningful work” are in the same complexity class, so the technology to automates the former also automates the latter.
2. Our revealed preference seems to be that we prefer solipsistic convenience over human relationships.

The rest of this post elaborates the argument.

Task Complexity and AGI

Many of the people building AI, and many of the people working on AI safety, share a common vision of what a good AI future looks like:

1. We figure out alignment,
2. build superintelligent AI,
3. and it takes over the world, for our benefit.

Many people will tell you that last part outright: they think human disempowerment is a good thing because the AIs will be smarter and “more moral” than us. Others don’t outright cheer for disempowerment, but you can infer it from their influences, e.g. people who say they are inspired by Iain Banks’ Culture series of novels, where benevolent superintelligent machines run the world while humans just party and play video games. This idea of benevolent disempowerment goes back to the origins of alignment as an idea.

In this worldview, alignment is the last and most important task for humans to work on. It is also a thought-terminating cliche, because it lets you avoid any of the hard political or economic or moral questions about the post-AI world. Any objection about the aligned AI utopia can be dismissed by saying “that’s not real alignment”.