For us, humanity was a distant goal toward which all men were moving, whose image no one knew, whose laws were nowhere written down.

— Hermann Hesse, Demian

Despite having spent most of my life marinating in transhumanist ideas, I never cared about AI. Because I’m a classical transhumanist: I want to use technology to change myself, so that I can do more. Because I, personally, want to learn and think and act, and not have a machine learn and think and act on my behalf. And in this worldview, AI is not interesting, except as a stopgap technology to unlock the transhumanism tech tree faster (but of course, technology has plans of its own).

Now, the trajectory of technology has shaken out such that we’re closer to AGI than to transhumanism. But, realizability aside, I find it surprising and demoralizing that the “AGI future” seems to be much more popular than the “transhuman future” (more clearly: transhuman, but human-descended). I thought “become as the gods” would be more attractive than “build a benevolent god to rule over us”.

It has made me more cynical about humanity. Most people don’t want to be more, do more, know more. They want to watch Netflix and die. They want the world of the Culture: an AI-run infinite vacation where they consume content for a few centuries, then autoeuthanize from boredom. When you tell them AGI means disempowerment, the closure of the future, giving up their autonomy, they don’t care. Choice and autonomy are a burden to them. They want to die in every dimension, except the sensual. Chimps wearing diapers, cared for by machines.

If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by explaining my previous beliefs, and why I held them, and then explain why those beliefs changed.

Contents

1. Economics
2. Alignment
3. Control
   1. Writing
   2. Software Engineering
   3. Deferring to AI
4. Stagnation

Economics

There’s a widespread belief that artificial superintelligence (ASI) is inevitable. In this post, I argue that this is false, and believing it is upstream of a lot of bad discourse.

Why do people believe it?

It’s easy to believe. Samuel Butler, already in 1863, observed that machines are improving faster than humans are improving. Today, AI systems are gaining capabilities much faster than humans: in five years we went from systems that can barely string together a sentence to autonomous cybercrime.

Biological nervous systems have many virtues, like being power-efficient (because they run on Brownian motion, i.e. for free), but they were designed by evolution, a greedy optimizer that can’t plan or readily backtrack. Given what we know about the physical limits of computation and exploratory engineering, it is reasonable to think we can do better, and build machines that do the work of the brain, but better, faster, cheaper, and with fewer transitive dependencies (a single human needs the entire biosphere to live, computers need only electricity). If you believe in natural selection, and it’s hard not to, then it seems reasonable to think that, in the long run, smart, fast, and cheap will outcompete dumb, slow, and expensive.

Any rigorous thinking about the long-term future leads to extreme outcomes. It cannot, by construction, produce normal or mundane futures.

“Rigorous thinking” means you have a model: a formalism for representing the state of the world, and a transition function to advance the state. You put the world into the formalism, and turn the crank to get successive states of the world. Models can be qualitative or quantitative. Most thinking about the future is qualitative and informal: people think in terms of trends over vague qualities like “intelligence”. For example, your model might be:

1. Currently, humans are smarter than computers.
2. Computers are getting smarter faster than humans are getting smarter.

From this, you can derive:

So, I have this little app, Epoch, that I use to keep a journal. It’s a tiny Rust web app that runs as a systemd service and uses SQLite as the database. I use a desktop and a laptop regularly, and use Syncthing to synchronize them, including Epoch’s database. That way I can use the app on both devices without needing a server to synchronize them, the tradeoff being that I have to make sure the sync is finished before performing any mutations.

But I had this bug. Say I edit today’s entry on the laptop, come home, wait for Syncthing to finish, then I’d open today’s entry on the desktop, and the text would be missing. It’s not that the server is holding a lock on the file and preventing the sync: opening the database with the sqlite3 command line tool shows the new text is there. If I restart the server, Epoch can read the new text.

My mental model was:

The rusqlite Connection object points to the database file. Syncthing swaps the file’s contents from under it. Subsequent queries go to the new file.

The implicit promise of modernity is: we have to work really hard for a few centuries, but eventually we’ll get to the Star Trek future where we’ve automated all the drudgery, and all work is meaningful, heroic, and dignifies the spirit. We will be artists, scholars, captains of the starship Enterprise, etc. There’s a few problems with this idea:

1. “Drudgery” and “meaningful work” are in the same complexity class, so the technology to automates the former also automates the latter.
2. Our revealed preference seems to be that we prefer solipsistic convenience over human relationships.

The rest of this post elaborates the argument.

Task Complexity and AGI

Many of the people building AI, and many of the people working on AI safety, share a common vision of what a good AI future looks like:

1. We figure out alignment,
2. build superintelligent AI,
3. and it takes over the world, for our benefit.

Many people will tell you that last part outright: they think human disempowerment is a good thing because the AIs will be smarter and “more moral” than us. Others don’t outright cheer for disempowerment, but you can infer it from their influences, e.g. people who say they are inspired by Iain Banks’ Culture series of novels, where benevolent superintelligent machines run the world while humans just party and play video games. This idea of benevolent disempowerment goes back to the origins of alignment as an idea.

In this worldview, alignment is the last and most important task for humans to work on. It is also a thought-terminating cliche, because it lets you avoid any of the hard political or economic or moral questions about the post-AI world. Any objection about the aligned AI utopia can be dismissed by saying “that’s not real alignment”.