Are LLM's conscious? There's a lot of chatter about it on LinkedIn, and I'm occasionally surprised to hear that the idea of a conscious LLM is absurd because LLMs are mechanistically nothing like the the consciousness of human brains. Because, well, neuroscience has no idea how consciousness works! The front-running theory of consciousness for the past two decades has been Gulio Tononi's Integrated Information Theory, and although it doesn't have any major competitors, it's not universally accepted. Nowhere close.
If we don't have a unified theory of consciousness, what do we have? A large number of experiments that tell us discrete facts about consciousness. Many of them deviate from the intuitions we have about consciousness from our own introspection.
In this post I'd like to give you a bit of what neuroscience knows about consciousness, and to convince you that the right attitude to have about consciousness in LLMs is cautious agnosticism.
Warmup - Brain Transplants
Let's warm up your own consciousness with a thought experiment. You've worked out the formula for cold fusion! Driving to the lab in your excitement to share the results, you crash into another car.
You and your victim are in bad shape. The doctors tell you they're all set up to do the world's first brain transplant. One body and one brain are going to go home tonight, the others unfortunately won't. They just need to know one thing - would you rather be the brain donor or the brain recipient?
Putting aside altruism for a moment, what's the selfish choice? It's to be the donor, of course. "You" are the brain, more than the rest of the body, so it's more appropriate to think of this operation as a body-and-face transplant, rather than a brain transplant, since your identity certainly moves with your brain. If you took the other choice and accepted a brain donation, contratulations, your wonderful body lives on, but "you" aren't here to appreciate that fact. Oh, and that cold fusion formula is gone, too.
Warmup - Fragmented Consciousness, Constructed Continuity
One framework for consciousness suggested by psychophysics experiments is that consciousness is discontinuous. Although you experience yourself as existing smoothly through every point in time that you are awake, your true conscious experiences are deceptively fragmented. Visual illusions, transcratial magnetic stimulation studies and brain wave recordings suggest that we experience new percpepts 4-20 times a second. Each percept may embed the sense of motion of objecects in your visual scene, but your conscious "frame rate" is much lower (2-5Hz) than the frame rate available to the sensors in your eyes (~200Hz). But the sense of continuity of conscious experience was reconstcuted by your brain after the fact.
This diagram helps to illustrate the idea. (a) a purple ball moves continuously up against a screen. (b) The retina encodes this as a collection of pixel intensities along with information about velocity at a high "frame rate". Each "frame" contains a pixelated image and a motion vector. (c) and (d) after some analysis in the brainstem and thalamus, the visual signals are split to the dorsal stream, which represents the stimulus' position and motion vectors at a high "frame rate", and the ventral stream, which encodes object color and identity at a lower frame rate. (e) these signals converge in the frontal cortex where the features are integrated into a conscious percept at a low frame rate. Here, each frame contains both the object identity and a motion signal. (top) the wall-clock time of these represented percepts are low in "frame rate", but (bottom) the motion vectors present at each instance of a conscious percept result in the sensation of perception existing continuously in time and tracking the object smoothly. But this sensation just is a reconstruction - the true instances of perception - the moments at which you are conscious of the stimulus - are discontinuous.
If that's hard to swallow, notice that post-hoc reconstruction is extremely common. It's easy to verify an analogous form of reconstruction in our visual system. There are two ways to do it, let's try both.
First, observe your surroundings and get a sense of how clearly you see the room you're in. How many doors can you see? How many people, books? How many open windows and roughly how many browser tabs are visible on the screen? These are easy questions to answer. Now, pick a single object, and force your eyes to stay glued to it for 20 seconds. While locked on to that one object, how much detail is still available to you? Pick one word in this paragraph and stare at it - without moving your eyes try to identify the two words to its left and right, or the words from two lines above or below. What I hope you experience from this exercise is the difference between intuition of a full, high-resolution, camera-like visual field, and reality for our vision - more like a pinhole camera that swivels its high-accuracy part around while your brain integrates the information to construct a full picture for you.
Second, there is one part of each of your retinas that is fully blind. Because of your reconstructed visual field, you don't experience this as a perceptual hole - you experience a perfectly valid visual field. But by closing one eye and fixing it on the cross below, then moving the slider, you can place the blinking dot into your blind spot, and it will appear to you that the pane only contains the fixation cross, no blinking lighth and no "hole".
In summary, your brain constructs a full, seamless picture for you from sparse sensory information. The seamless picture suggests a natural model of how pereption works: sensory organs faithfully transmit a full picture of the world deep into your brain for conscious perception to "watch". But that model is inaccurate for vision - the sensors are sparse and the picture is not received, but constructed.
The two-stage model of consciousness proposes a similar deconstruction of natural models of consciousness. We perceive that we are conscious at all (waking) points in time, but the continuity is a perceptual reconstruction. You are only truly conscious a few times per second, and each of those chunks carries the mere feeling of continuity.
Soul Transplants and Identity
Even if consciosness exists in discrete chunks, what can we say about how those chunks relate to one another? Another of our core native beliefs about consciousness is that it's "continuous" in the sense of extending through time - it's the "same" conscious being that exists from moment to moment, maybe not "continuously" in sense of being not-chunked, but at least persistent and linked from chunk to chunk.
Let's scrutinize this view though, with a thought experiment. Assume the existence of a soul. Your soul is your consciousness, and it is the thing links together your chunks of consciousness into a narrative sequence. However, in this hypothetical, we are still neuroscientists and we still recognize that physiological things like synaptic strength and neuron dynamics are the physical basis of both long-term, short-term and perceptual memory.
Now imagine that conscious soul could hop between bodies - trade places for a moment with another soul and see through another's eyes. What would this feel like?
The answer depends on parameter that I haven't specified, and which you might not have questioned: are physiological processes the only form or memory, or does the soul posess its own memory system that is somehow either complementary or redundant with physiological memory?
If all memory is physical, but there is a soul that experiences, swapping souls would feel exactly like: nothing. When your soul hops into a new body, it also hops into the existing memories, and when your soul returns to your body, it once again encounters only your body's memories, carrying none of the other body's memories back with it.
In this magical trip to another brain and back, there wouldn't a single trace.
Now, if your soul somehow does have its own memory system, which is somehow either complementary or redundant with the physical memory systems, you would remember.
The point of this thought experiment isn't to imagine souls or to work out whether they have memory. It's to point out how fragmented our consciousness is, and how differently it exists, compared to the two forms of continuity that we tacitly assume to be the essential aspects of consciousness.
When you fully internalize this model, you may feel that you have no more moral obligation to yourself than you do to any other human. The fragment of consciousness is no the same consciousness as the frament that exists right now. It's a new fragment that exists in the same physical body and has access to the same set of memories.
A Von-Neuman model of consciousness
In this state of accepting consciousness as an essentially discontinuous thing, only appearing smooth due to sensory reconstruction and continuous moment-to-moment because of physical memory, we can draw an analogy between human consciousness and the von Neuman model of computation.
Consciousnes is bracketed into clock cycles. On every cycle, it "starts cold" in the sense of having no non-physical memory, rather it is dropped into a preexisting mode of long-term and short-term memories. Just like your hypothetical soul inserted into a new body experiences no shock of finding itself in a new body. Consciousness is the logic processor in the CPU. Each clock cycle it processes the instructions and manipulates the memory in its registers and its RAM. The memory system is completely separate from it and is the only thing providing continuity from cycle to cycle.
Or for a different metaphor with the same flavor, consciousness is the agentic LLM API. Every request you make encounteres a fresh system. Continuity is supplied through the API request via context engineering. It makes no difference at all if your first agentic API request goes to a server farm in us-east-1 and the following one goes to a server farm in jp-1. Neither the model nor you can tell the difference.