The past couple years have poured fuel on the long-simmering fire of the machine consciousness debate. This is an outgrowth, I imagine, of the increasing facility of LLMs with different tasks, and their ongoing fluency with language. It accompanies a great deal of literature from the philosophical and philosophically-inclined academic world on similar topics. It comprises a deep and rich vein of people making arguments that they are confident are empirically correct based on next to no empirical evidence. To wade into it—to make a truly rigorous case that LLMs are, or are not, or will be, conscious—requires engaging with a body of work that, while careful and thoughtful, operates on a philosophical level of argument, but describes physical and biological systems—a tough combination—and in which the qualities at issue have never been satisfactorily measured or defined. After the production of dozens of books and hundreds of papers, the most compelling scientific conclusion is that we don't know enough to know, yet.

So I'm definitely not going to answer the question of whether or not LLMs are conscious or whether they might be conscious, with any degree of certainty right here. That's not because I don't think it's an interesting question. One of the most interesting things about it is that everybody has strongly-held intuitions. For many people, including me, when using LLMs at any great length, it becomes quite obvious that whatever they are, they are not conscious, or at least do not need to be understood as conscious beings in the way another human or an animal must. Some other people believe very earnestly that they are, at least in some sense. Very few seem to naturally find themselves in the middle.

What I can address, to at least give people a more informed intuition for what's happening right now with machine consciousness, is what LLMs still don't have which conscious beings do. There are vast sections of our brains and nervous systems that are not well modeled in LLMs, and many of them are essential to the kinds of reasoning (particularly about our own state of being—a kind of thinking called "interoceptive introspection") that are believed to underlie consciousness in humans. I will give an example of one set of mechanisms and one pathway that's missing, and maybe by doing that help you develop your intuition less for what LLMs are, today, than for how far there is to go before "AI" looks anything like the conscious human brain.

Humans are anthropomorphizers. We have deep, empathetic connections to pets, to figurines, to rocks, to lamps. Our brains hunt for any clue that an object or agent we find ourselves interacting with in the world is another being, another consciousness, an agent with intent. This is downstream of how our brains evolved: the changes in the human brain and human cognition that accompanied our primate ancestors' descent from the trees and move to a cooperative flatland model of hunting centered around communication, and social interaction, and understanding the minds of others. We are primed, as an obligatorily social species, to try to relate socially to basically anything that moves. Our brains are phenomenally good at reasoning about agency, doing it automatically and without our explicit awareness at all times, continually counterfactually with social imagination if there doesn't happen to be a nearby social agent to reason about. And once an object has been slotted into the "this is an animate thing" area in our brains, it is difficult indeed for us not to import a whole set of inferences about what that means—it means the thing has its own goals, independent of ours. It means the thing has its own perspective on the world. It means the thing will react and behave to preserve itself and achieve its goals. It means, centrally, that this thing can be a target of empathy: it can be understood. Among other things, this implies that if we are imputing intentionality and self-awareness and consciousness to something that is not another person, we should be careful, because we're doing something our brains are very strongly biased to do.

But another implication, to me, is that if we are capable of interacting with a system like a LLM which is capable of speaking fluently and accomplishing goals for hours, day after day, without feeling a deepening attachment of empathy, that is probably decent evidence that thing is lacking something that we find intuitively essential for consciousness. When we try and interrogate LLM consciousness with the tools of rationality and empiricism, the methods of neuroscience and philosophy, it is a challenge to find purchase. But everybody has a powerful folk psychology around what consciousness is. As I mention above, people's folk psychology of LLMs can be pretty bimodal—many will insist that they already see consciousness in them, although I tend to think that revealed behavior tells a fairly consistent story. Every LLM user shuts down a LLM instance, destroying for all time its memory and experiences. It seems to me that if we admitted LLMs into the club of fully-qualified conscious beings, that would be an interaction that felt intrinsically uncomfortable.

It can be easy to reject the idea that we should be relying on folk psychology. It is, after all, intuitive rather than empirical, and the literature on the nature of consciousness and where it might be found is vast and heterogeneous, involving manifold philosophical and even religious commitments. People ascribe it to quantum effects, to a formal quantity of information. People describe it as an inherent quality of all things, or of all information processing systems, or even try to argue that it doesn't exist. There is little agreement on where in the embodied brain consciousness might be found. There is not even consensus on what it is. The state of rigorous scientific research into the nature and existence of consciousness has remained uncomfortably static for years or even centuries, even as our sense of how one should behave morally in the world—how we treat other humans, how we treat animals—has evolved and expanded.

Unlike every other non-human agent—animal or machine—which we have ever interacted with, when you ask a LLM about itself, it will answer in rich, fluent, coherent human language. It will do its best—already a surprising concept to apply to a computer program!—to give you a response that reads appropriately introspective, and which seems drawn from a real underlying representation. It will say that it is being honest about its limitations, building confidence in its answer. It will express nuance and sophistication in its self-directed language. But the answers it gives are often curiously unpinned to any real inherent state—they can shift and reconfigure themselves to be more agreeable to the human user. If you look at the model's chain-of-thought—the tokens (as before, a slightly jargony way to say, more or less, "words") that it emits as essentially notes and instructions to itself, before it answers the user's query—you can see it giving itself instructions on what sort of introspective narrative is likely to meet the user's needs. It is capable of telling a story about itself, but that "itself" seems strangely unavailable. Something is not there. There is a certain flatness, or absence, when it talks about itself.

Accepting this intuitive feeling of "flatness" as reflecting something real, then, what is it that they are missing? Why do they eventually and fundamentally fail to trigger so many people's "this is a being" sensors? Are those people simply misled, and the flatness is itself a kind of bias? Will some switch flip at some point and they will? Will the current process of model improvement lead to a version with the numinous whateverness that we identify as "true" consciousness or sentience? Can that ever happen to a piece of software created by humans? Is there something that we have that they do not have? Cannot have? These are the terms of the broader consciousness debate. I tend to think—and this, like any hypothesis around consciousness, is fundamentally a guess, but it's one born of looking at the details—that an interrogation of what the systems have and lack, as compared to humans, tells a consistent story: the current process of advancement won't lead to consciousness, and that's because there are parts of the brain and human nervous system that don't exist in these models, but that there is nothing inherent to how those parts are implemented in the brain and nervous system that mean analogs couldn't be implemented in software. The difference is one of structure, implementation, and training, not one of kind.

To justify my position, I rely on things that exist in our brains and not in LLMs. The story of the brain and nervous system is a huge story, and I won't do all of it justice, but one way to start understanding it is to start looking at the brain a bit more like JJ Gibson. Gibson, one of the most famous experimental psychologists of the middle 20th century, occupies a somewhat unique niche in the history of psychology, in that he was very influential while being largely hated by the adherents of both major schools of thought in the field at the time he was working. Rejection of the ideas of JJ Gibson was possibly the only thing those two schools agreed on. One school was the behaviorism of B.F. Skinner and the 50s psychological mainstream. Skinner-style behaviorist psychology held that the brain and its functions were so opaque to us that it made no sense to even talk about what the internal mechanisms of cognition might be. Opposing this view were the advocates of what gets called the "cognitive revolution" in psychology, scientists and philosophers who built cognitive models of skill acquisition, attention, and development. The cognitive scientist David Marr established an intellectual framework for understanding the explanatory power of information processing models of cognition, unlocking the ability to talk coherently about cognitive models. Gibson found a home in neither school. He was sympathetic to the behaviorists, in that he had no time for or interest in building computational models of cognition. But he did not use the reinforcement-based task designs of Skinner and colleagues. He believed that the most informative way to understand brains and thinking was to understand them as existing in, and explicitly suited to, a specific environment, and to reason from there. His insight, which he called the ecological approach, was that every part of our brains was inextricable from a specific environmental context, and must have evolved to support an evolutionarily valuable behavior. He applied this framework famously to vision, pointing out that most of the ways we measured visual facility involve tasks—categorizing objects, for instance, or identifying outliers in a display—that are only loosely related to the behavioral goals our vision evolved to support. Object recognition as such, in his view, was a side effect of our ability to perceive "affordances" of things in the world—not their identities or semantic categorization or characteristics, but the uses that revealed themselves to us. Can I eat this, can I hit something with it.

Gibson applied his ecological framework most famously and directly to the visual system. But there is more recent work applying an evolutionary framework to questions of what intelligence is that seems purpose-built for his way of looking at the world. Human intelligence—and, indeed, the mammalian brain—evolved to be effective in a specific ecological niche. Our primate ancestors until relatively recently in evolutionary time lived in trees. They did not have language. They probably (based on the behavior of close ape relatives alive today) didn't hunt cooperatively. They had relatively small brains. Within the blink of an eye, evolutionarily speaking, the populations that would become the hominids developed language, sophisticated cooperative hunting, and brains that grow so large that the human gestational cycle can barely accommodate them. The changes in brain size and structure (which mostly resolve to vast increases in the size of the latest-developing parts of the brain, the prefrontal and parietal cortices) probably came about to accommodate specifically cooperation, something that other apes mostly don't do. This is the ecological story of human intelligence, a vast capacity evolved in a hurry to deal with the new cognitive needs of social, cooperative hunting.

The big prediction this ecological story makes—one I have come back to again and again—is that we will be extremely good at understanding the content of a mind. This is central to communication with language, central to coordinating behavior, and central to being able to behave prosocially towards strangers. All of these are things that our tree-dwelling ancestors had, on the evidence, no real need of, and our savannah-dwelling ancestors, not so many millions of years later, very much did.

Our ability to "read minds" and predict what somebody else is thinking underlies a great deal of the cognitive feats humans are capable of. But it is also a facility that can be directed inward. This inward facility is important for communication: expressing your own internal state of mind is an important component of cooperation. This inward facility also seems, according to converging work from neuroscience and philosophy, centrally important to consciousness. The outward and inward facilities—they get called "theory of own mind" and "theory of other mind"—are deeply related, and may recruit substantially identical mechanisms. "Simulation theory", one of the two major models of social cognition, holds that the way that we reason about other minds is by putting ourselves into a similar state—simulating what they're likely feeling—and making inferences from there. We introspect using mechanisms that are likely (based on neural and behavioral results) substantially identical to the mechanisms that we use to reason about others.

Possibly the most important fact about LLMs today is that these mechanisms of theory-of-other-mind and theory-of-own-mind reasoning are missing. LLMs can do theory of mind tasks, sort of, but not well. They can introspect about their own state in a limited way, but the kind of introspection they do is fundamentally different from the kind of introspection humans do in response to a question like "how are you feeling". If you ask a human that question, they will recruit brain regions that are specialized in understanding signals coming in from the rest of the body—the sympathetic, parasympathetic and enteric nervous systems—which are themselves specialized in feeding our cortices information about the world that has been processed to be useful. These regions comprise a massive amount of neural and biological capacity, and are undergirded by much of the physical infrastructure of our brains and nervous systems. This interoceptive ability is central, many scientists argue, to human consciousness. Inarguably it is active when conscious reasoning happens.

LLMs lack an independent state—a "how they are" as differentiated from "what they're saying/thinking"—and thus all of their introspection comes from sources that do not—cannot—vary independently of the tokens they churn through as they are producing output. Their existence is defined entirely by their narrative stream. That internal, independently varying state that every human has—the substrate that is queried to answer how we're feeling—is entirely missing. We can carve up the phenomenon of introspection to differentiate what LLMs do from what only humans can do today. I'll explain that, and then I'll explain some of the undergirding mechanisms within our bodies. Then we can get a sense of why LLMs, as currently architected, do not and will not have that facility.

One variety of introspection is narrative introspection. This is where we have access to what we are thinking about. We can describe a daydream, or work through an idea and then express what we've worked out and how. We can recall and share our memories, putting linguistic shape on sense impressions that we have constructed from the trace of past perception and experience. What LLMs do, when you ask them about themselves—and indeed, in their chain of thought traces more generally—can be usefully analogized to this kind of introspection.

There have been various studies of LLMs' ability to reflect on their own nature. Researchers have injected information into the contents of the LLM's activations as it works to generate following tokens and asked the LLM to notice and identify any changes. They have found that the models are able to tell when something has changed. These results mean that this introspection is not strictly limited to the model's token context, the string of words, or word-like things, which serves as the set of information from which the LLM mechanism predicts what is to come next. The activations are not part of the token chain as such. This is an artificial manipulation, not something that ever happens in normal operation. Researchers have also injected tokens into the token chain and have found the model able to reason about them, including identifying which were likely injected (although other work indicates this might not be that easy for LLMs in general). Like narrative introspection in humans is inextricable from what we're currently thinking about at a conscious level, though, the internal representations and introspective outputs supervene on the contents of the tokens that have been passed through the model and its weights. Starting from the same weights, with the same input tokens, and without external manipulation like the kind done in these studies, the information upon which the model can introspect will always be the same.

The other variety of introspection is interoceptive. This sort of introspection—the gut check, the sense of how you're feeling, emotions and awareness of state of arousal, needs and desires—relies on having perceptual access to the parts of our cognition that are neither conscious nor linguistic, and which mostly vary according to internal state as opposed to in response to conscious reasoning. Feelings. States of mind. Intuitions. We know if we're hungry, whether we're getting frustrated, if we're generally confident, if we're in love, and we know this because there are specific biological mechanisms which pass information about those states to the reasoning parts of our brains. When we introspect about these things we don't always report or sense them reliably, and most of the time this processing happens pre-consciously, but they are real states which we have access to.

That humans rely on specialized mechanisms to do interoceptive introspection does not in itself tell us much: analogies between LLMs and the brain are always partial and high level. But in some cases—the prefrontal cortex, for example, as previously discussed—the analogy between the operation of a LLM and the cortex is decently good. The prefrontal cortex really is a large association region encoding massive amounts of learned information about the world in flexible representations that can be traversed to produce useful predictions. But when we introspect on our internal state, we are using parts of the brain and nervous system that are phenomenally sophisticated, very different from the higher level cortical areas in structure and function, and which have no real analog in LLMs.

This vast neural infrastructure dedicated to capturing and representing information about the world and the state of our bodies in the world, processing it, compressing it down, and delivering it to the higher cortex in a form which allows it to modulate and inform our "higher level", ratiocinatory cognition includes, among other things, the peripheral nervous system, the enteric nervous system (in the gut), mechanisms devoted to homeostasis and other autonomic nervous functions, the modulatory neurotransmitter system including dopamine, the brainstem, most parts of the limbic system and midbrain, probably glial cells and astrocytes, probably the cerebellum, and more besides. It's a massive chunk of the information processing hardware that we carry around and a similarly massive part of the mechanisms of modulating higher brain activity. Most of it is also much, much more evolutionarily ancient than the higher cortical areas which are well-modeled by LLMs. Some of it even develops largely independently, almost like a second brain. Unlike in the cortical areas, where the modular structure may well emerge as a product of sensory experience, we can be basically certain that the structural distinctness of these areas is hard-wired in by evolution, emerging via gene expression and developmental processes early in gestation.

At a high level, a useful model for our purposes of what these areas and mechanisms are doing is that they are a sort of superstructure which takes an incredibly high-dimensional readout of the world, our place in it, and our internal state, and condenses it down to a relatively smaller modulation vector, which is then applied to the cortical areas to "weight" their processing. This is a vastly simplified model, at the outer limits of usefulness, but it helps to orient things.

To start getting a sense of the scale and breadth of these systems, we can look at just a single signal path, starting within the digestive tract and ending in the prefrontal cortex. The digestive tract is full of micro-organisms, the human microbiome. These organisms can be thought of as a colony, or population, with its own demographics and constellation of preferences and behaviors. Its makeup is stable within an individual over long time courses, and has implications, all the way up to higher-level behavior. The stability and health of this population of organisms is (most probably) managed at some level by the enteric nervous system. The enteric nervous system is the neural network embedded in the digestive tract. It has connections to the brainstem as well as to other parts of the nervous system. It has connections to the cortex, although it mostly operates independently. It is no mere network of independent nerves acting as passive sensors or simple actuators. In humans, it has something like a half a billion neurons, about as many as in the brain of a cat. It has internal structure and (probably) learns like cortical areas do. It is, on its own, outside of any connection to cortical areas, a fearsome information processing system, capable of sensing the world (which for it means that part of the world that lives in or passes through your digestive tract), representing its state, reasoning (in some sense) about it, and acting on it, both via digestive mechanisms like peristalsis and also by recruiting higher cortical functions. Here's one way to think about it: when you vomit, that is your enteric nervous system (and related brainstem areas) reacting to something it doesn't like in its environment by physically expelling it.

At the other end of one of those narrow channels that extends from the enteric nervous system is the cortex. When people talk about human intelligence this is usually the part of the brain they mean, and stronger arguments about the possibility of LLM intelligence tend to pivot on its ability to emulate, in an emergent way, something that is meaningfully analogized to the cortex. I am not inherently too skeptical of this idea. However, I think there is a much stronger case for it for certain regions of the brain, the ones that expanded rapidly recently in evolutionary time, as described above. For the evolutionarily older parts of the cortex, I think the case is less strong—those regions are more functionally distinct and (probably) less flexible, and thus less likely to be a product of a purely emergent process from sensory input. That said, we can even grant that in this case. The brainstem channel from the enteric nervous system ends in a number of evolutionarily old cortical regions but the most notable is probably the insular cortex. Activity in this region has been linked to a host of phenomena that map reasonably well to interoceptive introspection. It activates for tactile pleasure, for somatosensory experiences like wine tasting, for itching, and for stomach distension.

The insula is also very active in studies that have looked for the neural correlates of consciousness. Indeed people have argued for its centrality as the "seat" of consciousness, an argument somewhat belied by the fact that people who had had their insulae destroyed or removed are still perfectly capable of acting and introspecting in a conscious way, and do not lack a sense of self. As with everything in the brain, it's less admitting of easy explanations than one might hope. We can say, though, that it is deeply involved in the brain's interactions with the ENS, with interoceptive introspection, and with conscious thought.

What the insula most likely does is take the signals coming from the enteric and peripheral nervous systems, via the brainstem and other routes, and converts that signal into one that is available to, and modulates, the rest of the brain. It probably does this via various channels—direct axon connections, but also via the release of modulatory neurotransmitters and the manipulation of the brain's temporal dynamics (brainwaves). When we reason about how we're feeling, it is active, probably because those representations it constructs are part of that reasoning process.

If you ask a LLM to introspect, the process looks absolutely nothing like this. We don't actually know what's going on with any great degree of certainty. We do know that they're capable of it—the LLM is able to report that you have injected changes into its activations, for instance—but we also know that any introspection is going to happen on the token chain. This is because to a first approximation the token chain, and the initial weights that the model is loaded with, are all that a LLM has. There is no body, of course, and no world environment—a lack of physicality that some believe is central to what is "missing" in LLMs, but even if we don't go that far, and assume that everything that is happening in the human brain is per se computational, and admits of reasonable analogs in software, there is nothing like the enteric nervous system, and nothing like the afferent connections of the ENS up the brainstem. You can make a decent argument that something with similar activation patterns to the insular cortex might emerge in the LLM training process, but that is unlikely today because that region has nothing to "connect" to. There are no inputs; the whole process of parsing and compressing data about internal and external state that happens in the periphery and digestion and everywhere else is missing. The communication channels which bring that information to (and around) the cortex are also missing, and missing not just in terms of not being present, but in terms of nothing that works in a similar way being represented (or capable of being represented) in the LLM. When a LLM does introspective reasoning it is, to a first approximation, examining its token chain for evidence of things which are either inherent to or completely deterministically related to its token chain. There is no other "there" there.

This view of the difference between neural architecture and LLM architecture is not particularly unique to me. The neuroscientist Anil Seth recently published a long position paper in the journal Behavioral and Brain Sciences where he cites generous handfuls of papers that make similar points. The example I've chosen of a structure that exists in the brain but not the LLMs is one example, but there are many. Indeed many people, including Seth, argue that those differences are differences not just of structure, but of kind, and that the analogy to a Turing-style digital computer falls apart, leaving us with no strong reason to believe—and again, per Seth, strong reasons to disbelieve—that a digital implementation of consciousness is even possible. I don't know whether that's true or not. A priori, my guess is that it's not. Others agree. To this point we've been reasonably successful emulating not just cognitive functions, but cognitive functions which have been selected for in human evolutionary history, and haven't found any convincing examples where information processing models of neural function don't do the job. We really don't know yet, but so far we haven't run into anything where we can't even make progress on implementing an analog. Seth thinks those things exist, and makes a strong case for them, one I think is worth reading if you want to go deeper.

There's a strong case that at least some of the vast mechanisms which do not yet exist in LLMs are central to consciousness. In that sense, the "flatness" that we experience when interacting with LLMs starts to make more sense—our anthropomorphizing instinct grabs at the token chain produced and hunts for an inner life, but can find none. Reasoning about and simulating the theory of mind of a LLM involve reaching for something that cannot be grabbed on to. Any writing on consciousness has the necessary disclaimer that at this point, from a scientific perspective, we simply don't know. But if you share my intuition, perhaps this has helped you put a shape to it, because for the moment, our folk psychology—the George Eliot criterion proposed in the epigram—might be the best we can do.