Many of the people building AI, and many of the people working on AI safety,
share a common vision of what a good AI future looks like:
- We figure out alignment,
- build superintelligent AI,
- and it takes over the world, for our benefit.
Many people will tell you that last part outright: they think human
disempowerment is a good thing because the AIs will be smarter and “more moral”
than us. Others don’t outright cheer for disempowerment, but you can infer it
from their influences, e.g. people who say they are inspired by Iain Banks’
Culture series of novels, where benevolent superintelligent machines
run the world while humans just party and play video games. This idea of
benevolent disempowerment goes back to the origins of alignment as an
idea.
In this worldview, alignment is the last and most important task for humans to
work on. It is also a thought-terminating cliche, because it lets you
avoid any of the hard political or economic or moral
questions about the post-AI world. Any objection about the aligned AI utopia can
be dismissed by saying “that’s not real alignment”.
