One of the biggest social problems with the rollout of AI has been the way that so many business leaders have reacted to it. Their immediate take on AI was that it would allow them to fire many, or even—in the fevered dreams of AGI proponents—most of their employees. The big labs fed this with breathless statements about how their products will shortly be able to do any work that currently requires people. That nothing like this has yet come to pass does not stop the labs from continuing to promote this line of thinking, and has not stopped industry leaders who buy this argument from initiating waves of poorly thought-out layoffs, in the anticipation of that outcome.

I propose that this phenomenon is downstream of what I'll call the "bad boss fallacy". The idea that AI can replace people outright is based on a mistaken, or at least inadequate, picture of what an employee is and how they exist in an organization, and further that a more holistic and gracious sense of the value humans bring—along with a better understanding of what AI is and is not—is not difficult and would lead to outcomes that were better for all concerned. Regular employees, executives, managers, employees—everybody except maybe Sam Altman.

What do I mean by the bad boss fallacy? Roughly, it is when a manager idealizes their employees as single-purpose replaceable cogs, rather than understanding, deploying, and relying on them as complete people.

One of the books that I've found tremendously influential is Digital Apollo by David Mindell. This book is about the development of the Apollo Guidance Computer, but more broadly its about the interactions between humans and automated systems in early manned spaceflight, and the debates and technical decisions that underlay that. Mindell talks in one of the early chapters about the question of what controls, if any, the early space capsules should have.

This is a famous debate—it was depicted in the book and movie The Right Stuff—which spawned the immortal phrase "spam in a can" as a derisory descriptor of a human astronaut in a capsule who had no actual role in flying their craft. The debate exists in pop culture history as a disagreement between buttoned-up, mathematically oriented rocket engineers, who did not want unpredictable humans sullying their perfect automatic control systems, and test pilots, near-suicidally reckless air cowboys with preternatural flying ability who believed they could pilot any kind of craft with no automation whatsoever, based on little more than sheer, intuitive self-confidence.

The way that Mindell characterizes the debate is very different. For one thing, he notes, all of the participants in this debate—the test pilots and the missile men—were sophisticated, field-leading aeronautircal engineers, many with PhDs, in the business of solving complex quantitative design problems. The debates were not cultural, but technical. The central question was one of engineering, of how humans fit and operated as nodes in a carefully designed complex system, and if the particular capabilities—cognitive and physical flexibility—and weaknesses—relatively slow reaction time, distractibility, susceptibility to g-forces and other conditions—of humans in an engineering context led to them making sense as part of the system. It was not about the feelings or intuitions of the test pilots, or the stubbornness and biases of the missile engineers, but about systems design, and all of the participants in the debate were attacking the same problem, cooperatively.

What stuck with me was the book's description of humans as nodes in an engineered system. The value of the people who were operating the controls and reading the displays and generally interacting with the user interaction mechanisms could be and was defined as precisely as the value of a given sensor or effector mechanism. Having the flexibility of human cognition unlocked different architectural mechanisms than a system that relied purely on digital logic and physical sensing. It was not the case that humans were always a compromise. They were useful in a way that could be quantified, and which differed from the other (also useful) components of the system.

So when I say "bad boss" I don't mean somebody who is mean, or somebody who is too demanding, or somebody who does not communicate clearly. What I mean by "bad boss" is somebody who does not think clearly and coherently about how their employees are people. People are unique, not fungible, and have specific strengths and weaknesses. To a bad boss this uniqueness is an inefficiency to be optimized around. This makes the bad boss bad to work for, certainly, but it also sets the boss and their organization up to fail. There's the famous example of car assembly lines in Japan vs. the United States in the 1970s. In the US, each worker was expected to do their assembly line task and precisely nothing else. Implicitly or explicitly, everything other than the immediate part in front of them was somebody else's problem, and even bringing it up could be cause for disciplining or worse. In Japan, on the other hand, each assembly line worker had a mechanism called an "andon cord", an emergency shutoff for the entire line to deal with a problem. They were encouraged to use their judgment to think about the success of the whole assembly, and to weigh in on that even at the cost of the line's total productivity. This made for Japanese cars that were built to a higher standard of reliability than American cars, and contributed in part to the tremendous success of imported cars against domestic ones in the US even after the gas crises of the 1970s. The Japanese factories used the fact that human workers had judgment—a human characteristic through-and-through—as part of their systems design, where the US factories treated them as tightly circumscribed parts of an automation system responsible for, and capable of, doing only a single function.

The andon cord example makes another point: flexibility, judgment, and sense of responsibility of human employees are often useful even in jobs like assembly-line work that is designed to be as unthinkingly routine as possilbe. Because people have such a unique collection of skills, even heavily Taylorized roles end up implicitly relying on those skills, even if they are not measured or understood by management.

So how does the bad boss fallacy interact with AI? If you think about your employees as carefully fenced-off, narrow parts of an automation system, single purpose and fungible, then the way that mechanical automation works is that you replace one of those parts with a faster and more reliable version. The automated tools do the precise, constrained job you have given to the human to exact specification, and they do it more quickly and with greater precision than a human does. Given a broad enough selection of these widgets—or a sort of omni-widget, capable of being quickly shaped to any specific job type—you can replace humans entirely, at tremendous savings in cost and management hassle.

This kind of thinking is at the heart of many if not most of the non-software applications of AI that have been proposed. A computer game studio has artists to make the thousands of individual graphics that go into a game? Replace that station on the assembly line with a graphics-making robot. An administrative assistant calls for appointments and keeps a schedule? Replace that station on the assembly line with a call-making and scheduling robot. Movie production requires screenwriters who can have competent scriptwork done as quickly as possible? What about a screenwriting robot?

What is missing in all of the ideas are the things that those people bring to their jobs because they are human. Those are the things that, as a bad boss, you miss. The care that the game artists bring to a sense of overall coherence. The relationship the assistant builds with the people they work on scheduling with, and their felt sense of when a schedule is too punishing. The particular view on the world of a screenwriter. The judgment and care all of them attend to their jobs with. To the bad boss, these things are invisible; they do not need to be replaced because they do not exist.

Which is why, when implemented, these kinds of strategies sooner or later fail disastrously. Because even if the management of an organization doesn't understand that the cogs in its machine are actually people, with all the feelings of care and cognitive flexibility and intentionality and prosociality that entails, they get the benefit of that. Without their knowledge or explicit effort, they have created a system which is centrally dependent on the components of the system being actual people.

If that's the kind of failed automation that the bad boss fallacy leads to, what does a good boss approach to automation look like? The key difference is that it remains centered on people; rather than automation tools replacing people—or at least replacing them 1:1—automation tools are used to allow the people within the organization to accomplish more, and more flexibly. This is part of why AI tools for software development have taken off so fast and proven so useful (relatively, there are of course still complexities). The slot they fit into within technical organizations is a technology slot—partially automated tools that allow individual developers to accomplish more in the same time—that more-or-less already existed. Software development has been a process of figuring out how to automate away as much of itself as possible pretty much from the day it was created. High-level languages, IDEs, shell scripting: working synergistically with a specific toolchain that automates the parts of the task you would prefer to have automated is immediately familiar as a desirable goal to anybody who has ever built software.

In any industry, though, from car manufacturing to manned spaceflight, somebody has to think hard about why and how you have humans filling roles in the system, whether in the context of AI or not. AI does not allow you to get around that fundemantal question, the same one the early Apollo researchers debated, but neither does it change the fundamental nature of that question. Organizations that are well-designed around supporting humans with automation are already well-designed around AI.

Because the bad boss fallacy is so endemic, it seems unlikely that bad ideas for the use of AI will go away. The maladaptive idea of 1:1 replacing employees with single-minded, endlessly compliant and congenial robots is one that haunts the dreams of managers who would rather not manage. But it is a fallacious idea, and it will not work. While it seems plausible, or even probable, that people who are good at management—who understand using people as people in business systems—will figure out how to deploy these systems in ways that enhance human productivity as opposed to sapping or subverting it, not just in software but in a broad range of business tasks, that's little comfort to those whose experience of AI has been dislocation, stress and wasted effort, because their bosses misunderstand not just what AI is, but who they are.

The corrolary to the fact that AI can almost never be used to outright replace humans—at least not at scale—in almost any industry is that the set of applications where it could plausibly help humans do better or faster or more satisfying work is vast. But you can't get there unless you start from a perspective of understanding how the system into which AI is to be integrated works, which in turn starts from understanding the value of the humans in that system.