Research Exposes LLM Ethical Inconsistency by Design

Articulating an ethical principle and applying it consistently turn out to be very different cognitive tasks, and a new arXiv paper argues that current LLMs can do the first reliably and the second barely at all. For teams deploying these systems in regulated environments, that gap is not an abstract concern.

Incoherent by Design

The paper, “Incoherent by Design? On the Moral Self-Consistency of LLMs,” posted to arXiv by Pegah Nokhiz, Aravinda Kanchana Ruwanpathirana and Helen Nissenbaum, draws a distinction most alignment research glosses over: the difference between a model that endorses the right values and one that applies them consistently. The authors call this the “moral norm problem”, not whether a model aligns with broad human values, but whether it applies ethical rules in a stable, context-sensitive way. An LLM might readily state a principle against theft, then endorse a scenario involving petty larceny when the situation is subtly reframed, even when the core ethical dilemma is identical.

The fragility extends well beyond abstract trolley-problem territory. Research from December 2025 found LLMs struggle to detect incoherence in narratives, particularly when a character’s established traits conflict with their actions, a model might miss that a self-proclaimed vegetarian ordering a cheeseburger is incoherent, leaning on general world knowledge rather than tracking the internal consistency of the character’s stated principles. Minor changes in phrasing or sentence structure can produce meaningfully different moral outputs from the same underlying model, which makes predictable ethical behaviour hard to guarantee.

The Confidence Problem

There is also a documented gap between what LLMs state as beliefs and what their outputs imply. Studies have found LLMs can exhibit significant deviation between a directly elicited posterior belief and the correct Bayesian update of a prior, in short, the model says one thing and reasons toward another.

What makes this genuinely difficult for human oversight is what the paper describes as an “unknown unknown” dynamic. LLMs can project confidence even when their internal reasoning is inconsistent, giving users no reliable signal that something has gone wrong. That combination, confident output, unstable reasoning, is particularly hard to catch in production. It sits awkwardly against the compliance frameworks most regulated industries are building around AI outputs. Ethical failures in LLMs are not edge cases triggered by adversarial prompting; they emerge under ordinary operating conditions.

Benchmarks Built for Consistency

The SaGE framework, introduced in February 2024, measures LLM moral consistency across semantically similar contexts using a Moral Consistency Corpus. The goal is to quantify consistency separately from task-specific accuracy, a meaningful distinction, since a model can score well on a values alignment benchmark while still producing contradictory outputs across rephrasings of the same scenario.

The open-source TrolleyBench presents ethical dilemmas alongside clarifying and contradictory follow-ups, then scores responses for consistency. Results comparing frontier models including Gemini-2.5 and GPT-4.1-mini against human baselines put current models at only “middling levels” of ethical consistency, a level the paper’s authors consider insufficient for high-stakes deployment.

What This Means for Deployments

There is a strand of recent research suggesting that ethical behaviour in LLMs may be an emergent property of latent cognitive modes rather than purely a product of fine-tuned compliance, something that could in principle be cultivated rather than constrained into existence. How far that holds up at the consistency level the paper demands is not yet clear.

For enterprise teams, the practical read is more immediate: broad alignment claims from vendors are not sufficient evidence of consistent ethical norm application. Deployments in sensitive domains should require benchmarked results from frameworks like TrolleyBench that specifically test whether a model’s outputs hold when the same dilemma is posed differently. As the reproducibility problems surfacing elsewhere in AI research make plain, what a system demonstrates under evaluation conditions and what it does in production are often not the same thing. What an LLM states as the right answer and what it outputs when the question is rephrased are, right now, two different things.