"AI" means bad UX

There is an emergent strain of thought in America right now that people who are skeptical of "Artificial Intelligence" are only hurting themselves. Per this theory, held by both executives trying to roll out 'AI'¹ tools and a broad swathe of computer professionals who regard themselves as pretty well up to speed on automation and its complexities, people who are not putting in the effort to understand how to make LLM-based tools work for them right now are at serious risk of being cast adrift of history's passage. There are different motivations behind this worldview. In many cases it is well-meant and earnestly believed, born from concern over labor disruption and people being left behind. In other cases it is much more venal and ill-considered. But it is an attitude illustrative of the present moment, because it highlights the contradiction of the technologies that most commonly get called 'AI': these large, multifunctional, LLM-based generative models with a natural language interface seem like they must be capable of just about anything, but are fiendishly difficult to get to do anything actually useful. So you end up with people being exhorted not just to use 'AI' in their daily work, but to figure out how to use 'AI' in their daily work, as well as a lot of people using it for purposes ranging from unhelpful to affirmatively antisocial, where the quality and thoughtfulness of the output is not of primary concern.

This is a strange state of affairs, and in many ways unlike previous technological upheavals. When personal computers or smartphones or the internet hit the business world it was mostly quite clear how you might want to use them. They were clunky and frustrating and slow in many cases, but for instance the increased utility of a word processor compared to a typewriter was evident. In cases where the utility wasn't obvious, the tools didn't get used. In other cases of technological advance, like the dawn of the web, the utility of having a web page was obvious but the technical barriers to using the available tools—HTML and web servers—were moderately low but high enough that companies created and hired for specialized roles to use those tools. 'AI', in the form of products like Microsoft's copilot tools or ChatGPT, is an uncomfortable chimera. It is being deployed as if it has broad and obvious utility to everybody, but in fact the technical barriers to getting it working correctly are substantial enough that many if not most people are, quite rightly, disinclined to bother, and see the products as at best an intrusion accompanied by a litany of malign social byproducts.

Affordances of 'AI'

What is behind this strange nonviability, and the huge differences of opinion about usefulness? I think the answer is fundamentally one of user experience. My knowledge of the world of UX is indirect and pretty partial—it's never been a professional focus, and insofar as I've read relevant literature it's the literature one step before the actual UX literature, so JJ Gibson rather than Don Norman—so bear with me, but the classic way to think about UX (since the work of the Norman linked above) is in terms of "affordances". This term was coined by the psychologist Gibson, so I feel on somewhat solid ground talking about it: an "affordance" of an object is the ways that your brain perceives that it could be manipulated and used. So if you see a hammer, say, you will see that it has a place to hold it and a heavy thing suitable for clubbing stuff. If you see a cucumber, similarly you see that you could hold it like a club, but also it looks like something you could eat. If you see a bowl of soup, the latter affordance presents itself but not, one hopes, the former. And so forth. The central premise as elucidated by Gibson is that our brains have evolved to see the world and the things in it in terms of how we might interact with them; perception and interaction are inextricably linked and our understanding of what something in the world comprises is coextensive with our understanding of how we might interact with it.

This understanding of perception has been something of an orphan paradigm in psychology and vision science, regarded by many researchers as underspecified and frankly a little bit too hippy-dippy to guide research, but in the hands of Don Norman and his acolytes, it has been massively, overwhelmingly influential in interaction design. The central premise of the graphical user interface is that the interface itself should teach the user how to use it; if a function is operated by a window control, that control should LOOK LIKE it does what it does. Things that can be moved should seem movable. Things that can be clicked should be appealing and rewarding to click. Controls that do different things should look different. Over all, objects and interfaces should make it easy to make them work correctly and difficult to make them work incorrectly. User interface metaphors—this is a window, this is a folder, this is an icon—are designed around the presentation of clear and immediately understandable affordances to the user.

When Gibson talked about affordances and visual perception he was mostly talking about the perception and manipulation of inanimate objects. But the same approach can be extended to other kinds of interaction. In particular, the theory of affordances extends quite naturally to social interaction. If you are interacting with another animate being, that being presents an enormous number of social cues (both explicit and implicit, intentional and unintentional) which tell you how best to interact with it. We learn from a very early age how to understand if a pet dog is friendly, or best avoided. Rattlesnakes clearly and effectively indicate that we had ought to stay away. And when we find a conspecific that speaks to us in our language, we can instantly apply a massive and massively sophisticated framework for understanding what they are, how we can interact with them, and what that means both to us and to them.

And when I say massive I do mean massive. Something like 30% of the human brain by volume is devoted to reasoning about social interactions. Arguably (I do argue this, as do others smarter than I, but I wouldn't go so far as saying it's scientific consensus) the combination of changes in brain size and development of sophisticated, prosocial theory of mind in response to evolutionary pressures is what made humans humans. When an object in the world turns out to be a human person you can talk to the presence of that affordance imports a massive neural infrastructure oriented around that precise situation, replete with heuristics, prior beliefs, and a sophisticated ability to simulate the inner state of that person. You know that person has their own goals, and desires. You know that person is perceiving and evaluating you. You know that person sees the world in ways that are roughly similar to the way you see the world. All of this implicit knowledge structures and enables your interaction with that person.

So in a sense "this is a person I can talk to" is the most powerful user interface metaphor of all. If you were to somehow construct a computer program that a user could talk to in a human-like way, then that user would immediately know how to interact with that computer program. They would know that the computer program had a human-like view of the world. That the computer program had its own goals and values. That the computer program would care about the truth, but might be intentionally deceptive. They would know that the computer program could be asked to do something and would try its best, and that if it failed, it would feel bad about it. Just from one simple affordance a vast superstructure of understanding of capabilities would be immediately imported. No carefully laid out windows and folders necessary.

The great UX swindle

Well. Or, as has in fact come to pass, you could construct a computer program that a user could talk to in a human-like way where none of those things are true. A system which effectively deploys this incredibly powerful user metaphor—this is a person you are talking to—one which relies on comprehensions and understandings so central to the human character that many them may be innate rather than learned, but where that metaphor is affirmatively misleading. A system where you have incredibly strong intuitions about what it should, even must be able to do, but where those intuitions are contradicted by the system's behavior over and over again.

This is in many ways a worst case scenario for user experience. An application where clicking "save" deletes your files. An icon where clicking and dragging it makes thousands of copies. A sliding control where every time you move it something different happens. Perhaps the best pre-LLM analogy for the LLM user experience is the browser game QWOP, where something immediately intuitive (in the game's case, running), is rendered dramatically and hilariously unintuitive by the mode of interaction (in the game's case, this is fun).

This mismatch, between this incredibly powerful user metaphor and the actual abilities of these systems, is at the heart of most of the emerging problems with 'AI'. For most people, it is functionally impossible to break the mental connection between "this is a person you talk to" and inferences about internal states and goals. So-called 'AI psychosis', where the funhouse mirror agreeableness of a mindless chatbot sends people into ratiocinatory spirals of delusional ideation, stems from this failed metaphor. If somebody else is agreeing with you and expanding on and elaborating what you're saying, it must make sense, right? They sound like they know what they're talking about.

If you ask an 'AI' search tool for a list of articles, why would it give you a list that was partly true and partly made up? It is not trying to mess with you, it's trying to do its best. It is trying to do a good job. If it failed to do a good job, wouldn't it feel bad? Wouldn’t it notice?

If you tell a model to check its work, it will do that. If it says it has done that, it has done it. If it has not done that, it will disappoint you, and you know it doesn't want to disappoint you. It will not lie to you about this unless it has a reason to.

If you ask the model to write a story, or draw a picture, that story will have interesting things to say. That picture will be enjoyable in the way that a human drawn picture is enjoyable. They will make an artistic statement, have a point of view.

Diverging user stories

Now, I have friends and colleagues who I am certain read the above and said "no! No! None of that is true! That is obviously untrue." They're right! None of those things are true about LLMs, for reasons that are interesting and complex. They are, if you know something about how LLMs work and are trained, obviously untrue. To the friends and colleagues I mentioned, not having these things front of mind when using an LLM at all, ever, for any purpose, is absolute malpractice.

These people—my friends and all the others who are capable of getting reliable, sophisticated work out of LLMs—almost always have a programming background. They are people who have spent years or decades shaping their instructions to a machine to the syntaxes of different programming languages. They have dealt with poorly designed, non-obvious and even actively unhelpful interfaces. They have grown comfortable thinking about the computer and the software running on it in terms of its true capabilities, with the interfaces and interactions presented to the user simply the things you have to bang on to get the underlying system to do what you want. For these people, 'AI' as it exists today is tremendously useful, if tremendously complex, a representational system that can be coerced and structured into doing phenomenally complex feats of automation as long as you are able to describe those feats precisely enough. 'AI' is, in other words, a language in which to program computers, one unprecedented in its flexibility and power. For these people, building elaborate tooling and controlling for unexpected behavior are default modes of interaction, and no interface metaphor is natural enough to disabuse them of the notion that this is how computers must be treated: adversarially, with constant suspicion, and armored with plans and backup plans and alarms to be triggered when things go awry.

And this, I think, is the answer to the question I posed at the start. For the overwhelming majority of people—including the executives pushing the deployment of 'AI'—the story of this technology is one of seemingly infinite capability. For the executives, who are in the business of delegating and assigning work product, this sense is untroubled by complexity in implementation, and so they task those they manage with "figuring out" how to use it most effectively. For those who are forced to try and use it to produce quality work, the vision of infinite capability quickly turns into a mirage, dissolving into endlessly frustrating and inexplicable failures amid massively harmful side effects. For people accustomed to looking behind the curtain, as it were, at the underlying capabilities the user interface is meant to expose and explicate—a set that includes the people at 'AI' companies who develop these systems—the story of this technology is one of absolutely transformative capability across an extraordinary range of domains which will be realized primarily with those who are capable of developing the relevant expertise.

These three user stories, such as they are, represent islands in a vast and unbridgeable gulf. That all three are enabled implicitly by the same user interface should be taken as a warning that most of this technology's promise is right now suspended over an abyss of misunderstanding and failure, and for it to become a socially useful and productive form of automation—dislocating and uncomfortable, yes, and complex in the way that all revolutions of automation are complex—the user experience needs to be reimagined and transformed, from the fallacious and self-serving name all the way down to the use of naturalistic conversational interfaces.