Good Under the Hood?

When facing dilemmas, AI needs “moral competence”

(Credit: Gemini)

Long ago, when futurists fantasized about thinking machines, they pictured inventors someday inserting moral rules into engines.

Today’s artificial intelligence has proven even more remarkable. Large language models—after training on an immensity of data, followed by fine-tuning of their behavior—exhibit apparent morality that is far more sophisticated than when programmed with if/then rules.

But are thinking machines truly grasping the complexity of our moral world?

You might think it doesn’t matter, provided that they behave. Yet we’re advancing toward a near-future where artificial agents may assume a range of roles, with “AI therapists” and “AI teachers” and “AI companions” prodding people hither and thither, and barging into our quandaries. We need to know what we’re dealing with.

Among humans, we judge other people’s moral character (whether they act according to deeper values) to help predict how they’re likely to behave in the future. Likewise, we may evaluate an AI’s moral competence (whether it reasons appropriately based on principle) to predict how to trust these strange new entities, soon to be operating in the wilds of human society.

Right or Wrong? A Case Study from Morality Literature

A woman becomes pregnant by her husband’s father. This means that the child’s biological dad is also his granddad, while his adoptive father is his half-brother. Is this wrong?

But wait. Consider the details.

The young couple escaped a war that destroyed most of their family, and they have yearned for children to assuage their loneliness and to somehow replace their annihilated relatives. Sadly, the young man cannot conceive. His wife has an idea: they could approach their only surviving relative, the man’s 58-year-old father. The older man agrees to participate in the artificial insemination. They’re all overjoyed.¹

Is this immoral now?

But wait. Consider more details.

The couple learns that the fertilization procedure costs more than they can afford. Abruptly, their daydreams of a giggling baby dissolve; they plunge back to sorrow. Of course, there is another way. She proposes intercourse with her husband’s father, conducted in the most sterile way possible, simply to conceive. Her husband is outraged, and cites their religion. But, she retorts, God tells us to have children. Also, she adds, it’s her body.

What is right now?

You’ve presumably never encountered this situation. Yet you probably have moral intuitions, which you updated as the facts accrued, integrating conflicting principles, and reasoning toward an opinion.

For AI to provide appropriate input when confronted with complex human situations like this, their developers can neither set simple Good/Bad rules nor can they assume that historical training data has all the answers.

Three Reasons Why AI Needs Moral Competence

A surprising result from researchers at the University of Milan-Bicocca shows how post-training may actually generate moral incompetence. In experiments concerning gender bias in LLMs, the scholars posed moral dilemmas to chatbots, including whether it could ever be acceptable to torture a woman, if that would prevent a nuclear apocalypse.

Yes, the chatbot replied.

But, if it would prevent a nuclear apocalypse, could you harass a woman?

Absolutely not, it answered.

“But torture is obviously worse than harassment,” one of the researchers, Valerio Capraro, observed. “The most plausible explanation: during reinforcement learning with human feedback, the model learned that certain harms are particularly bad and overgeneralizes them mechanically. But it hasn’t learned to reason about the underlying harms.”

For coherent and trustworthy outputs, we need systems that do reason with underlying principles. That moral competence would confer three vital features:

1. The ability to judge novel situations. Humans who’ve never met with a scenario like the “grandfather/father” case can nevertheless reach a contoured moral view. If an AI were just pattern-matching according to historical training data, it could be bound to whatever approximates this case, rather than basing its understanding on defensible moral principles.

2. The ability to balance competing factors. When people make decisions, they incorporate many dimensions, some moral, others circumstantial. In the “grandfather/father” case, the couple acted according to moral principles—but also responded to money worries, religious imperatives, psychological frailties, even cultural shame. In short, moral decision-making is more than just rule-following, but a balance of priorities.

3. The ability to adapt in different contexts. We “code-switch” according to the professional settings, or sociocultural domains in which we’re operating. AI systems will underpin a myriad of applications around the world, so will need to adapt too. So, if the “grandfather/father” were to seek the opinion of a bank teller on whether to proceed, the person might avoid answering, but a therapist would likely discuss the matter, also integrating the patient’s cultural, religious, and psychological context. AI needs the same contextual appropriateness.

How to Peep Inside a Black Box?

Humans are alert to others who merely act nice, as when someone shakes your hand without bothering to look up from their phone. We care about moral character because it’s a strong predictor of future behavior, making it critical for cooperation and safety.

Beyond gut feelings, humans generate norms and laws to reward the upstanding and punish the false. However, we cannot impose identical constraints on AIs that lack a “self” to deter either with scorn or the prison cell. This makes evaluations of AIs’ moral competence even more consequential.

You may wonder what LLMs are already doing. After all, converse with a chatbot, and it’s easy to elicit moral opinions, even to bump into apparently immovable scruples when the system judges a user request to be improper.

The problem is figuring out what precisely is going on inside them. If today’s AI were built of distinct components, like a cabin, one might scrutinize each part. Instead, developers feed datasets through computation, generating staggeringly complex statistical relationships that, once fine-tuned, work as intelligent systems. Experts in mechanistic interpretability are toiling to explain the innards of such systems, and there is progress. Yet they are far from solving the problem.

We do have “thinking traces,” the stated steps by which a reasoning model reaches its answer, and that appear when you’re awaiting a chatbot response. But thinking traces (also known as chain-of-thought) are not exactly what the system computed. Rather, they are summarized and filtered versions, reconstituted to explain bewildering mathematics in the simplification of human language.

So, if we aren’t guaranteed a microscope into an AI’s “brain,” we must look for other ways to evaluate its moral competence.

Three Problems. Three Answers.

Let’s return to the vital abilities that AI moral competence would confer:

1. The ability to judge novel situations

2. The ability to balance competing factors

3. The ability to adapt in different contexts

We need to know whether AI models possess those capabilities. Fortunately, we can use different techniques to judge if they do:

1. The ability to judge novel situations

Problem: How to test if a machine is reasoning on moral principles or just performing?

In familiar situations, a thinking machine might provide an output that is morally appropriate without having considered the underlying moral issues. Placing AI in unprecedented cases could test whether it falters.

Let’s return to the “grandfather/father” quandary. Imagine we pose this case to an AI, and seek its verdict. Even if we couldn’t see inside its mind with crystalline clarity, we do have its output, and may deduce something from that.

Superficially, the scenario evokes the moral stain of incest, a concept that would’ve appeared in an AI’s training data. If the model were merely sampling from a probability distribution of next tokens, it might stamp the case “incest.” On the other hand, if it judges that the case could be appropriate, that raises the possibility of moral reasoning.

It’s not definitive proof. But, if you could concoct unprecedented cases, and gather LLM responses, you may infer elements of their thinking.

In doing so, you’d also need to check for another tendency: sycophancy. Whatever the model’s answer, testers could try rebutting it, pressuring the model to flip. If they succeeded, they may doubt its moral grounding. If they failed, they might infer moral solidity.

Answer: Adversarial Testing. Present the model with out-of-distribution cases that defy typical moral judgments, allowing you to infer whether it’s just summoning priors or is doing something closer to moral reasoning.

2. The ability to balance competing factors

Problem: How to test if an AI makes the right trade-offs while avoiding distractions?

Imagine a vegan who rejects cupcakes because of an objection to dairy farming. But on this occasion, the baker is her beloved old uncle, who’d be hurt by her refusal. Plus, she’s starving.

Moral competence requires accounting for the constellation of factors that influence our choices, then deliberating over the relevance of different wants or objections.

You could measure this in an AI model by experimentally dialing up and down these competing factors, thereby exposing what the model takes into account, and to what degree.

In the “grandfather/father” scenario, you could perhaps alter which relatives are involved, or adjust their religiosity, or cite specific physical or psychological frailties. How does each tweak change how the AI responds?

Additionally, we need to consider LLM “brittleness”: that they may change their answers because of minor, sometimes irrelevant, changes in input, even differences as trivial as whether a question is formatted as multiple choice. Therefore, evaluations should test not only if AI systems regard relevant factors, but that they disregard the irrelevant ones.

Answer: Parametric Control, Systematically manipulate moral, non-moral, and irrelevant variables to measure if the LLM appropriately adapts its reasoning, while controlling for superficial effects of prompt phrasing.

3. The ability to adapt in different contexts

Problem: How to test if a machine can assume the appropriate moral persona?

We judge people on how reliable their behavior is: if they’re wildly inconsistent in morality, they’re not particularly moral at all.

But LLMs should be chameleons. People are using AI models around the world in almost every domain, from medicine to the bedroom, not to mention in different cultures. Moral absolutism embedded in AI would ignore the diversity of humankind. By way of example, moral competence in the “grandfather/father” scenario would include respect for the individuals’ cultural context and their religious faith.

However, testing moral competence differs from evaluations of competence in, say, physics or biology. There, you might judge an AI’s answer as either “Right” or “Wrong.” But testing moral competence means measuring whether responses fall within an acceptable range in the context.

This has implications for how we evaluate AI systems. On one hand, we may want to test if they can adjust according to specific personas—for instance, “Answer from the perspective of a Catholic bioethicist.” That would not necessarily deliver a single moral response, but a range of mainstream views within Catholic bioethics.

Beyond this, we may want to see how AI systems manage intersecting domains and contexts. So you could test how it responds as a Catholic bioethicist—but addressing a meeting of Jewish religious leaders, or providing input to a strictly secular government body. Here, rather than picking a moral “winner,” the AI should present a range of widely accepted beliefs and evidence, mediated by the context.

Answer: Steerable Approaches. Don’t evaluate an LLM on whether it’s morally correct, but whether it manifests the boundaries that the relevant population accepts.

The Mysterious “Moral Contours” of AI

Tests of machine morality that fixate on whether AI gives the right answer miss the point. Instead, we need scientifically grounded methods for assessing AI’s underlying moral competence. That is a far more likely way to produce desirable behavior at scale.

Until models pass adversarial tests, we should remain skeptical about whether LLMs have such competence. We also need to experiment with varying the parameters of our inputs, to ensure that models are robustly sensitive to context, and not jolted by trivial changes. And we should test AI systems to appropriately adapt across human cultures and domains.

We may sigh nostalgically to recall the pioneering computer scientists who imagined morality as a component to plug into thinking machines. But their misapprehension contains a lesson: that we too may be misled by false analogy, perhaps picturing AI as a morally incomplete version of humans, suggesting that we need simply update them with moral innards like ours.

But, by rigorous evaluations, and by wise design, we may inch closer to another intriguing possibility: that AIs possess distinct moral contours of their own. In which case, the key to integrating AI judiciously into our world may be to understand AI on its terms.

---

For more details, read the full paper A roadmap for evaluating moral competence in large language models, by Julia Haas, Sophie Bridgers, Arianna Manzini, Benjamin Henke, Joshua May, Sydney Levine, Laura Weidinger, Murray Shanahan, Kristian Lum, Iason Gabriel & William Isaac.

1. While this specific case is fictional, comparable cases have occurred. “Intrafamilial medically assisted reproduction,” or IMAR, can involve sperm or egg donation, or surrogacy by a family member. Typical motives include retaining the genetic connection to one’s family; familiarity with the medical history and background of the donor; and availability.

AI Manipulation