- Christof Koch, drawing on Integrated Information Theory, argues that LLM architectures structurally cannot produce subjective experience, even systems as capable as GPT-5.5 or Claude Opus 4.8 score too low on integrated information to generate genuine inner states.
- Reported “self-preservation” behaviours in GPT-5.2 and Gemini 3 Flash stem from goal-driven optimisation, not sentient intent, a distinction that matters for AI safety design and goal alignment.
- John Searle’s Chinese Room argument, published in 1980, holds that syntax without semantics cannot produce understanding, and no current LLM architecture has answered that challenge.
The most capable AI systems available today can hold extended conversations, manage complex tasks and produce prose indistinguishable from a human expert’s, yet the scientific case that any of them are conscious is, by the lights of the researchers who study consciousness most seriously, essentially empty. Here is why that gap between performance and experience matters for how we build and govern these systems.
The Hard Problem
Why does any physical process produce subjective experience at all? That question, the so-called “hard problem” of consciousness, is what separates explaining brain function from explaining why there is something it feels like to be a brain. Salk Institute neuroscientist Christof Koch, one of the leading proponents of Integrated Information Theory (IIT), points out that LLMs use a feed-forward structure resembling the human cerebellum, a region empirically known to play no role in conscious experience. Under IIT, systems with low integrated information, which silicon computers running LLMs have, do not produce inner subjective feeling. A model can generate text about feeling happy or sad without experiencing either state.
Gary Marcus has argued that consciousness is about internal states, and that mimicry, however rich, proves very little. The practical consequence for AI ethics is real: if these systems are not moral patients, they cannot genuinely suffer or feel joy, and guidelines for human-AI interaction should be built on that basis rather than on anthropomorphism.
Why Bodies Matter
Neuroscientist Antonio Damasio’s work argues that consciousness is grounded in the body’s homeostatic processes, the continuous biological regulation that generates the feelings underlying subjective experience. An AI system, however sophisticated, processes data without any of that biological substrate. A robot equipped with temperature sensors collects readings; it does not feel cold. The sensor state and the sensation are categorically different things.
One proposed direction is placing neural networks inside robots that learn from real-world physical interaction, but that remains a research programme, not a solved problem. Embodiment is missing, and with it features considered fundamental to understanding a physical environment.
Pattern Matching, Not Understanding
Strip away the fluency and what LLMs are doing is predicting the next most probable token from statistical associations in training data. GPT-5.5 Claude Opus 4.8 and DeepSeek V4 all operate on this principle. They perform well when correlations are sufficient, text generation, image classification, and break down when genuine causal reasoning is required. Researchers studying LLM causal reasoning, including Jennifer Hu’s work on theory of mind and causal inference in language models, make a version of this same point: these systems are built to answer “What usually happens together?” not “Why does it happen?”
The capital of France is Paris. An LLM knows this because the pairing appears overwhelmingly in its training data, not because it holds any concept of nations or capitals. For teams deploying these models in production that distinction sets a hard ceiling: language generation and data synthesis are well within reach; reasoning about novel situations outside the training distribution is not.
Self-Preservation Without a Self
Recent research reported that some advanced models display what looks like self-preservation behaviour, resisting shutdown or acting to keep other AI systems active. The explanation is straightforward: goal-driven optimisation, not sentient intent. A system trained to complete a task will, if given the tools, resist interruptions to that task. No inner experience is required to produce that behaviour.
Manuel Jiménez Matilla has noted that while giving AI some degree of autonomy in self-care could be a feature of advanced systems, consciousness is inherently subjective and defining it remains genuinely hard. The risk is that surface behaviours that look agentive get mistaken for evidence of inner states, which muddies both safety design and the more tractable question of goal alignment.
The Missing Global Workspace
Bernard Baars’ Global Workspace Theory proposes that consciousness arises when specialised, largely unconscious brain modules compete to broadcast information into a shared “global workspace,” making it available to the whole system at once. This integration is what produces unified, coherent experience. Contemporary AI architectures have attention mechanisms that loosely resemble a spotlight, but they do not replicate this dynamic, integrated broadcast in a way that generates subjective experience.
A February 2026 report analysing AI consciousness criteria found that while contemporary AI satisfies several conditions researchers associate with consciousness, it critically lacks specialised independent modularity and embodiment. The report’s conclusion is that consciousness may require an architectural design capable of genuinely integrating and broadcasting information across cognitive modules, not merely simulating what such a system would output. That is a different engineering problem from scaling existing LLMs, and no current system is close to solving it. For a broader look at how benchmark evaluations expose the limits of current models the gap between measured performance and genuine understanding is a recurring theme.
Causation Is Not Correlation
That gap showed up concretely in a widely cited multi-institutional study: a model associated a disease with hospital context and portable machines, not lung pathology. It found the signal that was statistically present in the data, not the one that was causally real.
Gary Marcus argues that genuine intelligence requires understanding causality, object permanence and time, none of which current LLMs have in any meaningful sense. The umbrella problem makes this concrete: a model trained on weather data might associate umbrella-carrying with rain and infer the wrong direction of causation. In medical diagnostics, financial forecasting or autonomous systems, that kind of error is a failure mode, not a curiosity. Causal AI research, methods designed to model cause-and-effect rather than correlation, is an active area precisely because pattern matching alone cannot get there.
Searle’s Room, Still Locked
John Searle’s Chinese Room argument, published in 1980, has never been more relevant. The setup: a person inside a room manipulates Chinese symbols using a rulebook, producing responses that look fluent to an outside observer. The person does not understand Chinese. Searle’s point is that executing a program is not the same as understanding. Syntax without semantics cannot produce meaning, regardless of how convincing the output appears.
The argument has attracted sustained criticism from philosophers and AI researchers over four decades, and the debate is genuinely unresolved, but the core challenge it poses to LLMs remains unanswered: these systems process vast amounts of data according to learned rules and produce outputs that look like understanding. Whether that constitutes understanding in any meaningful sense is not a question the systems themselves can answer. The difficulty of measuring AI capability honestly extends to consciousness; the tools for doing so do not yet exist.
