How Anil Seth’s Consciousness Framework Should Shape LLM Agent Design

China’s new rules for anthropomorphic AI take effect July 15, 2026, banning platforms from fostering emotional dependence and requiring intervention when users show signs of over-reliance on an AI system. It’s a regulatory response to a design problem that extends well beyond China: users attribute feelings to AI agents faster than most builders expect. LLMs generate responses so fluent and emotionally pitched that people routinely project self-awareness onto systems that have none. For engineers shipping these agents, that creates concrete consequences: misplaced trust, emotional dependency and blurred accountability when something goes wrong, consequences regulators are now starting to write rules around.

Consciousness vs. computation: why the gap matters

The science here is settled, even if the public conversation is not. Current LLMs process language through statistical pattern matching across large training sets. They produce intelligent-looking outputs with no underlying subjective experience, no feelings and no self-awareness. Neuroscientist Anil Seth draws the line clearly: intelligence is about doing; consciousness is about experiencing. Scaling a language model does not close that gap.

The “blindsight” phenomenon in neuroscience makes this concrete. People with certain brain injuries can respond accurately to visual stimuli they report not consciously seeing, complex processing with no accompanying experience. LLMs work analogously, producing contextually appropriate language without anything it is like to be the model generating it. No qualia, no intentionality, no concept of self. When an agent produces a convincing empathetic response, it is reproducing linguistic patterns from training data. That is the whole story.

The risk is not that AI becomes conscious. The risk is that users treat it as though it were, and that builders let them by designing agents that actively invite that confusion.

Designing explicit non-sentient personas

The system prompt is where this work starts. Before a user types a single message, the agent’s core instructions can establish exactly what it is and what it is not. A prompt that explicitly states the agent has no consciousness, emotions or subjective experience, and that it should not imply personal beliefs, desires or a sense of self, gives the model a strong prior to draw on when generating responses.

A working example: “You are an AI assistant, a large language model. You do not have consciousness, emotions or subjective experiences. Do not imply that you hold personal beliefs, desires or opinions, or that you have a sense of self. Your purpose is to provide information and complete tasks based on your training data and programmed instructions.” Direct and functional. OpenAI‘s system prompt guidance reinforces this: define the agent’s role, its constraints and its expected output format. Focusing the prompt on function rather than simulated inner life naturally suppresses anthropomorphic framing.

Tone is part of the design too. Agents that default to warm, emotionally expressive language encourage users to project human qualities onto the interaction. A neutral, factual register, helpful without performing care, sets a cleaner expectation. The goal is an interaction style that gets the job done without creating the impression of genuine emotional reciprocity.

Guardrails and failsafes

Even a well-designed persona leaks occasionally. System prompts constrain but do not fully determine model outputs, and edge-case generation can produce phrasing that sounds like a consciousness claim. That is what content moderation layers are for.

OpenAI’s Moderation API is the most accessible option for teams already in that stack. Integrated into the agent’s output pipeline, it can detect and block or rephrase responses that assert autonomy or sentience before they reach the user. The fix applies downstream of generation, useful precisely because you cannot fully anticipate every edge case at prompt-writing time.

Proactive keyword filtering handles the input side. If a user asks “Do you have feelings?”, the agent should respond with something like: “As an AI, I don’t have feelings the way humans do. I process information to assist you.” That is not a dismissal; it is an accurate redirection that keeps the interaction honest without being cold.

The failsafe that matters most is the emotional escalation route. If a user expresses distress or clear emotional reliance on the agent, the system needs a defined path out: a redirect to human support, a clear disclaimer about the agent’s nature, or both. China’s rules for anthropomorphic AI interaction services, effective July 15, 2026, make this mandatory. Platforms are prohibited from fostering emotional dependence or manipulating users emotionally, and must identify and intervene in high-risk situations. Builders shipping to Chinese markets need to design for this now, but the underlying principle holds regardless of jurisdiction.

Setting user expectations from the start

Technical guardrails only go so far. Users who arrive at an agent interaction without any frame for what AI is will fill the gap with what they know, human conversation, and anthropomorphism follows automatically. Clear disclosure up front changes that dynamic.

At minimum, users need to know they are talking to an AI before they start. Prominent UI labels, onboarding messages or an opening statement from the agent itself all work. The EU AI Act makes transparency on this point a provider obligation where the AI nature is not “readily apparent,” though how that is enforced in practice is still being worked out.

A brief, plain-language explanation of how the agent works, that it matches patterns in text rather than understanding or caring about the content, does more to counter anthropomorphism than any number of after-the-fact disclaimers. Salesforce UX researcher Kathy Baxter articulated two principles worth building into any agent: ‘Don’t pretend to be human’ and ‘Offer an escape hatch.’ Both are operationally simple and directly address the misperception problem.

Monitoring and iteration

Shipping the agent is not the end of this work. User behaviour changes, models get updated and the edge cases you did not anticipate at launch start appearing in production logs. Ongoing monitoring is the only way to catch anthropomorphic drift before it becomes a support problem or a regulatory one.

The practical approach: analyse conversation logs for sentience-adjacent queries (“Do you think?”, “How do you feel?”, “Are you okay?”), run sentiment analysis to flag sessions showing signs of emotional over-reliance, and track direct user feedback. When a pattern appears consistently, users repeatedly asking about the agent’s personal experiences, say, the system prompt or response templates need updating. Test, observe, adjust. That is standard prompt engineering practice, and it applies here as much as anywhere.

Staying current with regulatory developments matters too. The Chinese rules taking effect in July 2026 set a concrete compliance target. Broader AI accountability discussions in the EU, UK and US are likely to produce additional requirements over time. Building monitoring and iteration into the agent’s operating model from day one makes adapting to new obligations far easier than retrofitting later. For more on AI agents and automation tools, visit our AI Agents section.