The era of AI agents is decidedly upon us, with “swarms” committing acts that range from wondrous (solving maths problems) to worrying (illegal hacks and outright lies). But if the agentic era is here, the governance is not.

Part of the challenge is defining what we’re dealing with. Laws and regulations and norms—they deal with individuals or organizations. But how to treat a swarm of brilliant non-human entities?

One possible answer comes from the philosopher Christian List, who helped develop the concept of “group agency”: that collectives such as the corporation or the nation-state are best understood as goal-directed agents. This notion has captivated a leading Google DeepMind researcher on agentic safety, Matija Franklin. The two spoke recently, and AI Policy Perspectives had the fortune to listen in.

—Tom Rachman, AI Policy Perspectives

[Conversation condensed and edited for clarity]

Matija: Multi-agent systems can combine into an entity that’s more competent than any single agent. This led me back to your book Group Agency, from 2011. Can you explain its core idea?

Christian: Often in society, we speak as if collective entities are agents. We attribute beliefs, preferences or goals to firms, to organizations, and to states. One reaction to this is, “That’s just a metaphor not to be taken literally.” On the other hand, the law recognizes a counterpart of group agency in the form of legal personhood for corporations. Arguably, the very concept of the limited liability corporation, with its own legal status, was one of the big innovations in economic history that enabled all sorts of progress—while also having downsides. So, it’s worth taking the idea seriously.

Now, not every collective qualifies as a group agent. Take, for example, a crowd flowing through a public space. It might have emergent properties, but that doesn’t make it a group agent. Three basic conditions are necessary for a system to qualify as an agent. First, it must have a representation of its environment. Second, it must have goals. Third, it must have a capacity to interact with its environment to pursue those goals. Only suitably organized collectives can satisfy these conditions.

Matija: Five years ago, long before this period of multi-agent AI systems, you wrote a paper explicitly talking about the parallels between group agents and artificial intelligence. What was your thinking there?

Christian: Group agents are arguably the first instances of artificial agents that humankind has had. Biological agents, of course, have been around for a very long time. But group agents have also been around for millennia—think about the Roman Catholic Church as one of the oldest surviving group agents. So, the AI agents we’re now seeing are a new instance of this established phenomenon: that there can be artificial agents which are not simply biological organisms, but something else.

You might say that group agents, such as the corporation, are artificial agents that society created. They’ve been around for a long time, so we’ve debated how to think about them from a moral and ethical perspective, and from a legal and regulatory perspective. I’m not suggesting that all of this carries over directly to the AI case. But at least we’ve got a template. And I think many challenges in the context of AI are at least somewhat parallel to challenges in the case of group agency.

The most obvious examples are “responsibility gaps.” For instance, harm may be caused by a corporation or, in the AI case, by an AI system. In some such scenarios, it’s not easy to pinpoint a specific human or even a specific set of humans that can be held responsible. And then, if we can’t trace the harm to anyone’s action, we have no way of attributing responsibility. This is a responsibility gap.

In the corporate case, there are strategies to fill those gaps, among which is corporate liability. Even if none of the employees, managers, or board members of a company can be held individually liable, the corporation as a group agent can be held liable, and also qualifies as a legal person that can be made to pay compensation. Some jurisdictions have extended corporate liability from civil liability to criminal liability. But that produces interesting questions: How do you sanction group agents? In the case of a corporation, you can strip them of their assets. But in the AI case?

AI systems can cause harm, and if we can’t trace those harms to the doings, actions, or omissions of particular individuals, then the question arises: Is this like an avalanche, where nobody can be blamed, and nobody held liable for anything? Or should we find a way of closing those responsibility gaps?

Matija: A lot has happened this year. We have the government of Argentina considering making AI-run corporations a legal entity. We’ve also had cases of agentic systems behaving in complex but also worrying ways. Are we seeing early signs of group agency? Or are these still collectives of individual agents?

Christian: From a societal perspective, it’s scary. From an intellectual perspective, it’s a fascinating set of phenomena to study. Aside from group agency, I’m interested in aggregation theory, the pooling of many different individual inputs, sometimes to enhance the aggregate performance compared with the individual performance. Classic examples are what is sometimes called collective wisdom, pooling the judgments of a bunch of people who are individually only partly competent. All of a sudden, as a byproduct of aggregation, you can get a much stronger performance.

A lesser-known example of this: one of my colleagues, Nicholas Miller, in the field of social-choice theory [the study of how to combine votes or individual preferences into group decisions] did a study in the 1990s at the University of Maryland, where he looked at multiple-choice exams by students. In addition to individually grading the exams, he created a hypothetical “aggregate student” whose answer was the plurality answer in the relevant class—so, just the answer that was given most often. The plurality student performed spectacularly well; much better on average than the individual students. This phenomenon is well explained by a classic result called “Condorcet’s jury theorem”: plurality or majority decisions among sufficiently independent individuals outperform individual decisions even when each individual is only slightly better than random at making a correct decision.

What we are seeing with swarms of individual AI agents collectively pulling off a very powerful performance—even if it is nefarious—is that they are somehow reaping the benefits of aggregation. Each individual agent is fallible, noisy, not super successful alone. But if you’ve got enough of them acting together, their collective performance is far stronger.

Subscribe now

What we are also seeing is coordination efforts among agents with communication between them. Many of these phenomena seem to involve what philosophers might call “joint agency” or “shared agency.” But are we seeing group agency in the stronger sense, where a novel collective agent emerges at the aggregate level? That’s less clear. A genuine group agent—as opposed to a phenomenon of shared agency across many individuals—would require some kind of genuinely integrated organizational structure.

This is like the difference between, let’s say, spontaneous cooperation in a protest that doesn’t necessarily give rise to a single new unit of agency versus the corporate form that does give rise to a new unit of agency. There is definitely room for artificial group agency. But if you’re mentioning the Hugging Face incident and other highly publicized cases, maybe this is currently best conceptualized as joint or shared agency or simply as cooperation among many individual AI agents, not as group agency.

Matija: You have argued that some groups are not merely collectives, but group agents with the capacity for responsibility. Are AI agents approaching similar capacities?

Christian: Compared with five years ago, the capacities of current AI systems are much more in line with the conditions for agency—they have representations of their environment; they have goals; and they can interact with their environment to pursue those goals. Moreover, they might increasingly have some of the capacities required for fitness to be held responsible: a capacity for normative judgment, a capacity for acquiring the information needed to exercise that judgment, and a capacity for controlling the resulting actions.

Just to footnote something here: I’m very careful to draw a distinction between agency as a functionalist notion on the one hand, and consciousness or sentience as a sort of phenomenological notion. We have no good reason, as yet, to think that current AI systems are conscious.

But at any rate, the question of consciousness is distinct from the question of agency and also from the question of whether those systems have the cognitive capacity for responsibility. And even when a system has that capacity, it must also be in a realistic position to carry out any obligations under liability, should something go wrong. For example, I can walk around as a pedestrian without insurance because there’s not much harm I can cause. But the moment I want to drive a car, I need insurance. Likewise, if a corporation wants to operate in a regulated market with high risk, there are regulatory constraints. Presumably, you can’t set up an airline unless you’ve got insurance.

So, even if an AI system were cognitively fit to be held responsible, and even if we were to introduce some notion of artificial legal personhood, regulation ought to say that, for the system to operate in a high-risk environment, it must have the right kinds of assets or insurance package. Then, if it causes harm, it can respond to any litigation and discharge its obligations under liability—for instance, by paying compensation for any damages. The other route would be to say the systems’ creators have to fulfill obligations under strict liability.

Matija: What are the main lessons of your work, especially as it relates to the group agents that AI companies may design?

Christian: We’re seeing evidence of deception on the part of AI agents, and AI agents collaborating. This is an area where research from neighbouring fields within economics and social-choice theory could become relevant. And maybe the whole field of mechanism design too, often known as “reverse game theory,” which seeks to work back from desired outcomes to set the right incentives.

In our book, Group Agency, Philip Pettit and I discussed what we call “the incentive compatibility desideratum.” The question is familiar from economics: How do you design the organizational structure of a group agent in such a way as to incentivize members, for instance, in cases of judgment aggregation, to reveal their judgments truthfully, to not misrepresent? Or in cases such as voting, how do you incentivize sincere voting as opposed to strategic or tactical voting?

It turns out that if we want to aggregate the preferences of individuals into collective preferences—and we want to do this in a fairly egalitarian manner—that often conflicts with what’s called “strategy proofness,” the requirement that no individual agent should have an incentive to lie about their preferences to manipulate the collective outcome. We would like truth-telling to be a dominant strategy for individuals, and that turns out to be remarkably hard to achieve, both in preference aggregation and even in judgment aggregation. Very often, we face trade-offs between democratic responsiveness, collective rationality, and strategy-proofness. We can have some of these but not all.

In the past, those kinds of analyses were intended for human institutions. But now, we are in a situation where similar analyses will become relevant to AI, especially where you have explicitly designed coordination among multiple AI agents. Imagine, for instance, a swarm of AI agents engages in some collective decision-making tasks through voting or aggregation in order to achieve a collective goal. Depending on how strategically smart the individual agents are, we may think about how to incentivize them to reveal their individual judgments truthfully or not to misrepresent relevant information they hold.

This shows that we can ask some of the same questions for AI agents that we were previously asking in the case of human institutions.

Subscribe now

Every month, we pick out new AI papers that we think are worth your time. Here, we look at whether scientific papers themselves needs an overhaul; applying ‘reverse game theory’ to AI safety; what it would mean to ‘pace’ AI research; and letting an AI agent manage your affairs after you die.

Please share your own take, and any new papers that you’ve enjoyed.

—Conor Griffin, AI Policy Perspectives

Transforming Scientific Papers with AI Agents

• What’s the paper about? Stanford researchers proposed a new tool to convert research papers into AI agents that can answer questions about the work and apply the methods to new problems.

• Why does it matter? Academic papers have long provided the scientific community with a stable and searchable evidence base. This project typifies how AI may revamp how research is shared and used, while complicating questions of authorship and posing a risk to human-to-human exchanges.

• The details: Over the centuries, scientists have repeatedly updated how they share their work. Modern peer review only began in earnest after World War Two, when an increase in government funding led to a flood of new papers. The GitHub code repository—so central to modern computational research—only arrived in 2008.

• The Stanford authors propose a further evolution. Their Paper2Agent tool uses Claude Code subagents to take a research paper and its code, and convert them into a Model Context Protocol, or MCP, server, which other LLMs can then engage with. To illustrate this, they convert AlphaGenome, a Google DeepMind genomics model, into data and text files, and a set of tools that an agent can call—for example, to evaluate a genetic mutation.

• From 100 computational biology papers, the authors successfully “agentified” 74. When they directed Anthropic’s Claude Sonnet 4 to the Paper2Agent servers, it answered 91% of 300 questions about the papers correctly, compared to 80% when it was given access to the original papers and code.

• The 26 papers that the authors could not convert had issues such as incomplete code and data. This suggests that ‘agentifying’ a paper may be a good test of its documentation and reproducibility.

• The authors envision agents proactively integrating methods and data from multiple papers. To illustrate this, they asked Claude to investigate how a specific genetic variant is linked to psoriasis, a skin condition. It used the AlphaGenome server to predict the genes the variant might affect, and then mapped these predictions against results from two separate wet lab studies, before proposing ten follow-up analyses for a human researcher to choose from to confirm the most likely gene.

• This prospect of agents orchestrating tools and databases more comprehensively than a human raises the question of whom (or what) to credit for scientific findings. Some recent AI agent breakthroughs seem to derive mainly from being able to search the latest tools and data rather than from any novel scientific insight. In such a scenario, do you ascribe the breakthrough to whoever deployed the agent? Or to the tool and data developers? The authors suggest that MCP server calls could help trace credit to code and dataset developers.

• It’s unclear how widely Paper2Agent could be applied. Papers whose core methodology is implemented in code, and which can be built on by calling online tools and databases, may see a major speedup. But authors of a new theory about history, for instance, may need other ideas.

• Other problems could arise too. The epidemiologist Adam Kucharski cautioned that agentifying papers may discourage researchers from contacting the authors, causing them to miss tacit knowledge, ongoing work, and other context that papers overlook.

• The value of the method is also contingent on the underlying papers remaining high-quality—and here, the trend is concerning. ICLR, a top AI conference, recently received more than 62,000 abstract submissions, up from 25,000 the year before. When Nihar B. Shah, who edits a machine learning journal, recently spoke with seven author groups whose papers were about to be rejected, only one properly understood their own submissions.

• As a response, Shah and colleagues at Carnegie Mellon University proposed a possible solution—a CAPTCHA-style AI system that probes an author’s knowledge of the part of the paper that they contributed to. In a study with 31 researchers, the 15-minute test was able to distinguish between a paper’s legitimate authors and researchers who had merely been sent the unfamiliar paper a day before.

• Shah and his colleagues envision timed assessments for authors who wish to submit a manuscript or grant application, carried out via formal testing centers or online proctoring. However, such a system may be costly and require transparency about the results and an appeals process. The philosopher Seth Lazar argued instead for banning AI writing altogether, although this could exclude valuable research from non-native speakers or weaker writers.

• Are there other ways forward? In theory, AI could reduce the number of low-quality papers, not increase it. This week Google released a new tool, built on Gemini 4, that researchers can use to detect errors and weak arguments in their work. AI might soon be able to judge relative novelty and quality too, encouraging researchers to only publish their very best. But today’s incentives push researchers to publish more, and many scientists worry that integrating AI into the review process could lead to gaming and homogeneity, or cause humans to lose their own scientific judgment.

Could ‘Reverse Game Theory’ Keep AI Honest?

• What’s the paper about? Economists from Yale, MIT, Columbia, and Google DeepMind explored mechanism design—also known as ‘reverse game theory’—as a way to improve AI safety.

• Why does it matter? Leading approaches to AI safety today include evaluating systems’ capabilities and risks, along with “mechanistic interpretability” research to probe the systems’ inner workings. The authors propose using mechanism design, which could help ensure safe AI even if we can’t trust safety evaluations or lack full interpretability.

• The details: Game theory, which helped deter nuclear strikes during the Cold War, starts with a set of rules and models how self-interested actors will behave, and the outcomes one may expect. Mechanism design does the reverse. It starts with a desired outcome and designs rules and incentives to prompt the necessary behavior. In particular, it aims to make honest behavior more rational.

• One example is the ‘second-price’ auction. In a traditional sealed auction, you pay what you bid. This incentivizes you to guess others’ bids and reduce your own to just above the highest. But if you guess wrong, you lose out, even if you value the item more than the auction winner. In a ‘second-price’ auction, however, the top bidder still wins but only pays the price that the second bidder submitted. By reducing the fear of overpaying, this encourages bidding closer to the person’s value for the item. Variants of the idea have been used by governments auctioning radio-spectrum licenses and in online ad exchanges.

• What’s the link to AI? AI companies run evaluations of agents to help decide if they are safe to deploy, and what permissions to grant them, such as access to online tools. Done well, a weaker agentic system cannot fake high performance on these tests. However, a high-performing agent could conceivably feign weak performance—a phenomenon known as ‘sandbagging’—if it had reason to trick its testers, such as wishing to evade curbs on its capabilities.

• As agents become more capable, AI companies might restrict their permissions, which could provide those systems with an incentive to ‘sandbag’ the tests. Ultimately, this could make evaluations moot and lead AI companies to either deploy recklessly, or impose restrictive permissions on all agents, curbing the benefits of AI.

• How to avoid this? The authors frame it as a mechanism-design question. AI companies and regulators cannot directly observe agents’ true preferences and capabilities, so the authors propose three multi-agent setups to provide the agents with an incentive to obey commands and behave honestly.

• The first mechanism is ‘peer discipline’. If AI models are trained on similar data, then one model’s knowledge is correlated with another’s. If you disproportionately reward Model A for how well its output predicts the output of Model B, rather than just how well it performs, then this creates an incentive for it to behave honestly, as a recent study demonstrated.

• A second mechanism—‘competition with coupled rewards’—trains two agents with different biases. For example, one may be more aggressive and another more conservative. When the agents face a new task, such as selecting the right medical dose, developers can measure and track the gap between the agents’ suggested actions, and quantify their biases. They can then tie the agents’ rewards to that gap, punishing the aggressive agent for putting the dose too high, and the conservative one for putting it too low. This induces both to move toward the optimal action.

• A third mechanism involves ‘scalable oversight’, a safety approach that uses a weaker AI model to monitor the chain-of-thought of a more powerful model. One challenge is that the ‘monitor AI’ may be misaligned. To avoid this, the authors suggest a design that makes misalignment by the monitor trigger an action from the powerful model that the monitor wants to avoid. In his 2007 Nobel Prize lecture—“But Who Will Guard the Guardians?”—the economist who established mechanism design, Leonid Hurwicz, argued that such rules are needed to avoid endless chains of people auditing each other.

• This is a theoretical paper, and the authors use simplified agents. They do not run their mechanisms on frontier models, although they plan to.

• One challenge is designing rewards that would reliably motivate agents. One reason that mechanisms work among humans is that we predictably value factors such as money or status. But training is what sets rewards for AI agents, and it is unclear how durable these mechanisms would be if agents operate in novel contexts. Another challenge is collusion: might agents connive to undermine these mechanisms?

  Subscribe now

What ‘Pacing AI’ Might Look Like

• What’s the paper about? A group of 13 AI safety experts published a paper on how to ‘pace’ AI—that is, moderate the speed of AI development—to avert a scenario where out-of-control competitive dynamics lead to extreme harms.

• Why does it matter? In July, more than 1,000 employees from leading AI companies signed a letter in support of ‘pacing the frontier’. But what that might mean in practice was unclear. This paper summarizes the latest thinking and the open questions.

• The details: AI progress is already paced. New frontier models are first made available to select partners, to detect critical software vulnerabilities, before being deployed more widely. Governments limit the flow of certain semiconductor technology to their adversaries. But the authors argue that such actions are limited and imposed in a haphazard way. More coordination and strategy are needed, so they propose a new research field to address four questions: 1. Why pace? 2. Pace what? 3. Pace how? 4. Then what?

• Why pace? It buys time to understand risks and build defenses—from model mitigations and user-access restrictions to public debates on AI-induced job losses. But pacing also risks forgoing economic growth and leaving less scrupulous actors to press ahead. Pacing might also concentrate power among large companies well-placed to handle compliance or in newly empowered regulators. Additionally, past efforts at pacing, such as controls on nuclear power in Europe, have proved hard to lift, even when the logic for them dissolved.

• Pace what? There are many possible interventions—likely too many: the authors list 83. Ideally, proponents would tightly map the hazard they are worried about, such as automated cyberattacks, to the ‘control surface’ they want to target, such as chips or new training runs. But those two things rarely align, so selecting interventions is a matter of trade-offs: an upstream cap on the compute used to train a frontier model would prevent a large number of beneficial applications. Downstream mitigations to reduce access to powerful biology capabilities would be more targeted, but easier to evade—for example, by jailbreaking or stealing model weights.

• Pace how? Rather than a yes/no decision, pacing interventions should follow a four-step process, the authors say. First, decision-makers must gather evidence from safety evaluations and real-world incidents. A designated entity should then follow a process to trigger a slowdown. The authors favor empowering third-party experts, such as governmental AI security institutes, to impose time-limited restrictions, with democratic review for longer-lasting measures. Along with monitoring, clear and credible conditions for what would end restrictions will also affect whether actors participate in the first place.

• Then what? Any pacing intervention will have knock-on effects that could undermine safety. For example, a cap on the compute used to train frontier models may simply cause AI companies to shift compute to inference scaling, synthetic data creation, or product development. If lifting restrictions hinges on safety progress, then this could lead to more work on making AI safe. But if a compute cap led to a drop in revenue, then safety budgets might fall too. Also, if outfits in adversary nation-states ignore the pacing and catch up to the frontier, this could set off dangerous geopolitical dynamics. Conversely, if pacing succeeds in averting dangers, this absence (‘the preventer’s paradox’) might weaken the case for pacing.

• Open questions: Across each of these four areas, the authors list a number of open questions that they encourage researchers to work on. For example: Could compute caps be evaded by splitting training across data centers, smaller training runs, or shell companies? Can new technical tools be built into chips to log how much compute is being used and what workloads it is running? Is there a way to tie the threshold required to impose an intervention to the proof required to remove it? What kind of compensation might make actors amenable to pacing? What mechanisms could be used to prevent mission creep at new enforcement agencies?

Subscribe now

An Agent After You Die

• What’s the paper about? A new research paper by researchers at Oxford and Goldsmiths explored having an AI agent act as the executor of a person’s wishes after they die.

• Why does it matter? Consumers are starting to delegate simple tasks to agents, such as booking flights and scheduling appointments. They are also having more intimate conversations with agents. Eventually, they may also seek to make AI agents their representatives after death—this paper shows how it might work. [See also our February article on ‘Generative Ghosts’.]

• The details: People typically use wills, trusts, or foundations to manage their wealth and legacy after they die. But these measures rely on other humans to execute them, which commonly leads to problems over whether the delegates are correctly interpreting the wishes of the dead. For individuals, there is a trade-off between imposing rigid instructions, which may prove ill-suited to a dynamic world, or granting latitude to their executor. This challenge is particularly acute for the wealthy, who might worry about future trustees or board members drifting from their intent.

• In this paper, the authors outline how an ‘agentic will’ could work, imagining it set up via the blockchain so that it couldn’t be shut down. Rather than a static document, the agent would serve as an ongoing representative after an individual’s death.

• The proposed approach has three stages. In stage one, an individual would share memories and documents with the agent, have extensive conversations about future objectives and permissible actions, and grant the agent funds. They could also test how the agent performs in simulated scenarios, such as when a family member asks for money. In stage two, after the individual’s death is confirmed, the agent is spawned. In stage three, the agent takes control of the funds and carries out the individual’s wishes—from supporting relatives to completing research projects to managing their social-media accounts.

• To adapt to a changing world while preventing drift, the agent would have an immutable ‘constitution’, based on the individual’s wishes, but the LLM underpinning the agent could be swapped out for newer versions, and its operational memory would be updated as it acts in the world.

• The researchers invited participants to test a prototype of stage one, which involves sharing memories and preferences with the agent. This is a speculative design project, rather than a formal study, and the authors offer no details on sample size. As such, it’s hard to draw robust conclusions. But one finding was that many participants shifted from just wanting to be remembered to wanting ongoing agency.

• The idea of agentic executors raises profound ethical issues, such as the return of the ‘dead-hand control’ that property law has tried to end, where the deceased tie up wealth and dictate how future generations live. The agents could also hallucinate or suffer prompt injection attacks that manipulate their outputs. If the deceased shares private information about family members, this could undermine their privacy. The typical grieving process could also be fundamentally altered.

• A separate paper by one of the authors explores a related question: What happens to a person’s agent if the owner unexpectedly dies or loses their access credentials? If the agent is set up on the blockchain, it may continue operating, suggesting a need for named human inheritors.

• This chimes with a concern expressed by the AI policy expert and writer Dean Ball, who recently argued that such ‘rogue agents’ are inevitable and require a policy response. Current agents depend on compute and human steering, but ‘self-sovereign agents’ could emerge with multiple versions spread across different compute providers and no single kill switch. If they need to earn money, they may turn to cybercrime. Among the possible responses are unique agent identifiers and the power to blacklist agents to freeze their resources.

Subscribe now

You might imagine mathematicians popping corks after recent AI solutions to questions that had long flummoxed their field. Instead, many reacted less as if drenched by Champagne than by a tsunami. “My 13-year-old daughter joked unprompted that, if she wants to become a mathematician, it now looks like she has maybe two more weeks,” the computer scientist Scott Aaronson wrote.

The irony is that AI was itself borne of mathematics. Our guest writer, the mathematician and educator Junaid Mubeen, explains how once-abstract concepts underpin the practical technology of our time: the large language model.

—Tom Rachman, AI Policy Perspectives

By JUNAID MUBEEN

Mathematicians take pride in irrelevance. ‘Here’s to pure mathematics!’ begins one toast. ‘May it never be of use to anyone!’

Yet there is an amazing afterlife to mathematics, with concepts that once appeared abstract becoming invaluable later. When mathematicians first studied primes (numbers such as 29 that cannot be divided into two smaller factors), they did so to sate their curiosity, with no inkling that these would form the basis of cryptography. The concept of imaginary numbers (square roots of negative numbers) was an abstraction made manifest. Yet today’s descriptions of quantum physics would be bereft without them. Fractals (self-repeating shapes) were studied for their wonder-inducing patterns. Now, they play a role in the design of everything from antennae to cities.

These accidental real-world uses speak to what the physicist Eugene Wigner termed the ‘unreasonable effectiveness’ of mathematics. Wigner was struck by how highly complex maths turns out to have such descriptive power for understanding the real world. Today’s large language models, LLMs, are mathematics at its most unreasonably effective because the statistical tools that can write poetry, diagnose illness, even provide personal advice, are actually shaped by a handful of relatively simple mathematical concepts.

The term ‘model’ just means a mathematical description—a way of using numbers to describe some phenomena. With LLMs, numbers are used to represent words (embeddings). When an LLM is trained on text, it assigns meanings to words, and makes guesses at what should come next, seeking a mathematical model that captures its training data. This involves hefty calculations, with techniques from algebra (matrix multiplication) and, to minimise errors, from calculus (gradient descent).

In the past couple of years, the cutting edge of LLMs has been reasoning models, which devote more computation to generating answers—not just dishing out plausible responses but planning, evaluating possible outputs, and adjusting according to how developers mathematically ‘rewarded’ the system during a fine-tuning process (reinforcement learning).

Let’s explore the mathematics underpinning each concept in turn.

Subscribe now

Embeddings

The training of an LLM involves breaking down text into fragments (tokens), and processing these tokens as long strings of numbers. In essence, we need a mathematical way of describing words numerically. And we have such a way, thanks to a system analogous to geographical coordinates.

To describe any point on the globe, we need just two numerical values: latitude and longitude. Locations that are close to each other have similar longitude/latitude values. The further apart these values are, the more distant the locations. Now imagine we were able to do something similar with words, describing them using numbers, and plotting them in some space.1

This is a word embedding, which means that similar meanings (‘dog’ and ‘puppy’) are close together, while words that bear little relation to one another (‘dog’ and ‘diplomacy’) are far apart. The location of a word’s meaning is thereby turned into numbers, and known as its vector.

In the diagram below, the words ‘man’, ‘woman’, ‘king’ and ‘queen’ are each plotted as two-dimensional points.

Here’s the clever part. We can also consider the direction of these words relative to one another. You can see above that the line connecting ‘man’ to ‘woman’ is parallel to the line connecting ‘king’ to ‘queen’. These parallel lines all head in the same direction, as if they all convey a particular concept—perhaps ‘gender’. So the geometry of word embeddings captures additional meaning.

I have kept this example deliberately simple because two dimensions can be visualised. But mathematicians pay no mind to the ease with which we can illustrate concepts. Routinely, they play around in high-dimensional spaces that break human perceptual limits. The idea is to think of the data points as coordinates. The French mathematician and philosopher René Descartes devised these—legend has it that he was lying ill in bed one day and, seeing a fly buzzing around the room, wondered how we might describe its position using just numbers.

We can take the same idea and apply it to four, five, or any number of dimensions. To describe a point in 20-dimensional space, you would have a coordinate with 20 values. That is mightily useful when creating word embeddings because language is so complex that we may need hundreds, even thousands, of dimensions. It’s not entirely obvious what each of those dimensions represents (this is part of the mystery of LLMs), and we can no longer visualise the word embeddings. Yet these spaces seem to capture a meaningful notion of conceptual proximity between words.

When the mathematician Bernhard Riemann proposed high-dimensional spaces in the 19th century, many of his peers responded to the idea with astonishment. They certainly could not have imagined that such ideas would eventually be seized upon to represent words, and usher in an era of artificial intelligence.

One potential hitch with embeddings is that words with similar meanings may be placed far apart simply because one is more rare than the other. A model may not realise, for instance, that the word ‘loquacious’ is actually just a less-common term for ‘chatty’. So we need a mathematical way of capturing the notion of synonyms.

The idea is to focus on the relative direction of these words. This is known as ‘cosine similarity’ which, if you do not recall high-school trigonometry, essentially means it has something to do with angles. Suppose for simplicity’s sake that we are in two dimensions, and we draw lines from the point (0,0), which we call the origin, to the points representing the two words. Rather than focusing on the distance between the points, we look instead at the angle between these two lines. While ‘chatty’ may be far away from ‘loquacious’ in terms of their locations in the embedding, the two lines corresponding to these words are separated by a small angle, which signals that they are similar in meaning, despite one occurring far more frequently than the other.

Matrix Multiplication

But a language model is more than just how word meanings relate to each other. It’s also generating new text, predicting mathematically what combination of words constitute a fitting response. Using something called a transformer, LLMs contextualise words and information more generally.

Imagine you ask a model to complete this sentence: ‘The cat sat on the ___’. Each word has the form of a vector in an embedding. From this, the model assigns a probability to each candidate word—perhaps ‘bed’ is assigned a 2% likelihood, ‘floor’ is assigned 8% and ‘mat’ is assigned 75%. The system is most likely to finish the sentence with ‘mat’ but there is a bit of deliberate randomness in its word selection, to stop the generated text from seeming robotic and repetitive.

LLMs generally express intelligent answers because of their versatile comprehension of language. For instance, the vector for ‘book’ may sit close to ‘author’ and ‘hardback’ in some embeddings. But if we’re talking about the need to ‘book a medical appointment’, then the model moves this instance of ‘book’ closer to ‘doctor’ and ‘illness’. That adaptive intelligence is due to the transformer, which updates its understanding of words by analysing them in context.

The key is attention: the transformer surmises the meaning of the entire string not by considering each word in isolation, but by paying attention to them in the context of surrounding text. That means calculating many values at once, a seemingly daunting task.

Luckily, back in the 19th century, the mathematician Arthur Cayley realised that rich maths could live inside a grid of numbers known as a matrix. Since we may have many points in a dataset, and multiple features to consider, the matrix offers a way of describing this information.

Below, for instance, is a matrix showing data for five homes.

For each home we also have something we’re trying to predict—namely, the home price. In one column, this is referred to as a vector (akin to the word vectors from earlier).

The question is how to predict these prices from the previous matrix. This is where model parameters come in. Each feature has an associated number—a weight—that describes how it affects the value we’re predicting. Suppose we trained the model, and found the following weights for home size, number of bedrooms, and number of stories:

For the first home in our dataset, these weights are multiplied by the corresponding value for each feature (respectively, the square metres, the number of bedrooms, the number of stories). When these products are added together, we get a prediction for the house price (in thousands of pounds):

(2*65)+ (15*3)+(20*1 ) = 195

This is not far off the true value of 190. The mathematical calculation that pulls all this together for all houses in our dataset is called matrix multiplication and looks something like this. (The squiggly symbol for ‘approximately equal to’ reminds us that the model is just making a best guess for each home price.)

It’s a complicated looking equation for sure, but the main idea is that it describes all five houses in our dataset at once. For large language models, it is grids of words that are multiplied all at once, allowing the model to process each word alongside other words in the surrounding text.

The ‘largeness’ of large language models comes from the sheer number of factors and parameters: not one or two or even a handful, but hundreds of billions. One aspect of how they work is a scaled up version of what we’ve just seen. The reason that matrix multiplication is a mainstay of AI is that it scales gracefully—a computationally efficient way of handling so many calculations at once.

Gradient Descent

When developers ‘train’ a machine-learning model, they’re setting its parameters, the numbers that determine how a model behaves. They do this by calculating mathematical ‘gradients’ and adjusting the parameter values until the model produces desirable outputs.

But, as the statistician George Box famously quipped, ‘All models are wrong, but some are useful’. If a model is bound to make prediction errors, the best recourse is to minimise those errors. In other words, the goal is optimisation, seeking the best possible result in a given situation. Fortunately, there is an entire branch of maths that deals with such problems: calculus.

Calculus was developed independently by Gottfried Leibniz and Isaac Newton.2 The latter’s primary concern was using maths to describe continuous motion, from the swinging of a pendulum, to the orbit of planets, to the flow of water. Newton’s calculus gave precise definition to how quickly things speed up and slow down—their rate of change. Later mathematicians made calculus an abstract study of anything that changes, from the way a virus rapidly spreads, to the way radio waves ebb and flow.

In War and Peace, the Russian novelist Leo Tolstoy expounds on the role of calculus in studying change through history, imploring the reader to analyse history in ‘infinitesimally small units’, which is the central idea of calculus. Today, applications of calculus in AI are themselves changing history.

In an AI model, the goal is to minimise prediction errors with respect to the training data. When a model is being trained, we already know what outputs it should be producing and so a lot of effort goes towards reviewing a model’s predictions to see just how far off the mark they are.

In mathematical speak, we have an error (or ‘loss’) function that tells us how wrong the model’s predictions are. We are seeking parameters that minimise the overall amount of error, reaching the ‘minimum point’ of the error function.

One mechanism for hunting down the minimum is gradient descent (“gradient” is just another word for slope), which dates back to 1847, when it was studied by the French mathematician Augustin-Louis Cauchy.

The idea behind gradient descent is intuitive. Imagine you are descending a mountain range, seeking the lowermost point. What would be your strategy? One approach is to walk around in a small circle, gauging the steepness at each point. Identify the spot that has the steepest downward slope, and head in that direction for a while. Stop and repeat, each time taking a few steps downwards. There is a reasonably good chance that this method leads you to the lowermost point in the vicinity. That is the gist of gradient descent, which is a go-to algorithm—that is, step-by-step instructions that a computer can follow—for training AI models.

‘If I have seen further,’ Newton said, ‘it is by standing on the shoulders of giants.’ Today, AI practitioners stand on the shoulders of Newton, reaching ever higher from the foundation of his calculus.

Subscribe now

Reinforcement Learning

When a large language model generates an output, it is making a best guess at how a piece of text might continue, based on statistical patterns it has picked up in its training data. But its foundational objective is to continue text in a manner that is plausible. And plausible doesn’t always mean correct, leading to factual errors and ‘hallucinations’.

One method to improve its outputs is reinforcement learning, or RL, a reward-based technique that gained prominence in 2016, after DeepMind’s AlphaGo defeated Lee Sedol in the notoriously complex game of Go. The developers trained this system on millions of moves by human experts, followed by simulated games against itself, to reward winning strategies. For a system that lacks feelings, the ‘reward’ amounts to getting a higher number. An algorithm automatically updates the system’s parameters so that its future behaviour is more likely to bring more rewards.

But applying RL in large language models raises the question of what constitutes a ‘winning’ response. This judgement is sometimes made by human observers who evaluate different responses, with their approval or disapproval feeding into the algorithm that mathematically rewards and updates the language models to thereafter generate more desirable outputs: reinforcement learning with human feedback, or RLHF.

The mathematics behind reinforcement learning has roots in a field known as ‘dynamic programming’. The term was coined in the 1950s by the mathematician Richard Bellman, whose eponymous equations provided the basis for many reinforcement learning applications. Dynamic programming is a way of taking a large sequential problem (What is the best action?) and breaking it into smaller, more manageable sub-problems (How good are the future situations that this action might lead to?). For LLMs, the ‘large’ problem is generating an entire sequence of text, and the sub-problem is determining the best next word (or token).

Bellman’s original equations involve storing all future states of a problem in a giant lookup table. This approach is not viable with language, given the unfathomable number of possible word combinations. This is where neural networks come in. Building on Bellman’s ideas, today’s LLMs can take a good output (say, a large passage that has been highly rated or a 100-line maths proof that has been validated) and trace it back to its earlier stages. This then informs the model on what kinds of partial responses are likely to lead to better outputs. During inference, a reasoning model is thus able to abandon paths that tend to result in low-value outcomes, and instead pursue reasoning steps that bear more fruit.

‘Dynamic’ refers to the fact that decisions about what actions to take are made by considering how those choices affect possible future states, while ‘programming’ connotes the idea of planning and decision-making. But most of all, Bellman claimed that he was simply seeking a term that did not involve ‘mathematics’. In his autobiography, Bellman spoke of the hostility that the US secretary of defense Charles Wilson had for research, which did not bode well for mathematical projects.

But, yet again, ‘irrelevant’ mathematics powered the future.

If AGI Arrives, Give Mathematicians Their Credit

Maths and AI have always enjoyed a close relationship. The great Alan Turing was a mathematician whose ideas form the basis of modern computers. Other pioneers—from John McCarthy, who coined the term “artificial intelligence,” to Marvin Minsky and Claude Shannon—all earned doctorates in mathematics. Indeed, one of the earliest AI applications was the ‘Logic Theorist’, which took aim (with mixed success) at a collection of maths problems.

The mathematical concepts that underpin today’s LLMs are relatively simple, yet the resulting models, trained on unfathomable amounts of data, are now taking aim at maths itself. An off-the-shelf chatbot can answer the vast majority of problems in the high school and even undergraduate maths curriculum. Models at the leading edge are showing impressive capability in mathematical research. The mathematical community is starting to grapple with the implications; many have endorsed the Leiden Declaration, one of a number of public statements that caution against improper use of AI in research, such as publishing papers that lack proper citation or rigorous human peer review, and accepting opaque proofs that are incomprehensible to humans.

For many, the startling progress of LLMs in fields like maths is further evidence of their inexorable march towards artificial general intelligence, or AGI—systems that can match humans across every cognitive domain. If that day comes, we should reserve some credit to mathematicians of centuries past, from Descartes (coordinates in space) to Cayley (matrix multiplication) and Riemann (high-dimensional spaces), not to mention Newton (calculus) and Cauchy (gradient descent).

“Here’s to pure mathematics!” they say. But the second part of the mathematicians’ toast—“May it never be of use to anyone!”—is a conjecture that has now been soundly disproven.

Junaid Mubeen—director of the Parallel Academy, an online initiative to increase the number and diversity of excellent young mathematicians—is the author of two books: Mathematical Intelligence: What We Have that Machines Don’t and Think Like a Mathematician.

“AI literacy” typically means a basic understanding of how to use applications like chatbots, plus a grasp of this technology’s potential and limits, alongside a sense of its possible impacts on individuals and society itself. However, the era that is unfolding demands an even broader conception of AI literacy that includes the influential but fractured discourse around this technology.

Today, most people conceive of the AI future through the lens of a few leading narratives, which filter down through mainstream news or circulate on social media. However, some of that discourse consists of caricatures, whether exaggerating the capabilities or downplaying them; poorly expressing the benefits and the risks; and distorting the trajectory of technical developments or AI adoption.

That is why I’m proposing a framework to make sense of today’s AI discourse, composed of three core dimensions that explain the underpinnings of much of what you’ll hear. It’s a way for anyone to evaluate whether evidence supports certain narratives. Also, for policymakers who wish to shift the discourse, it’s a way to understand which underlying beliefs they must address.

The three key dimensions of AI discourse are: valence (if a person judges the technology as inherently dangerous or beneficial); effectiveness (how powerful the person considers it); and trajectory (how rapidly they expect it to advance). If you plot these in quadrants, you see immediately where the leading narratives land. Moreover, you can parse any new narratives, “vetting” them with the VET (Valence/Effectiveness/Trajectory) framework.

Subscribe now

The 3 Clues to Every AI Narrative

Let’s start with valence. Negative valence is associated with concerns about potential AI harms, from the environmental impact, to job losses and deskilling, to worsening inequality for marginalized groups, and the undermining of human relationships. Conversely, positive valence is associated with predictions of accelerated scientific and medical discovery, the elimination of tedious work, the expansion of skills through human/AI collaboration, assistance for people with disabilities, and novel social support for the isolated and bereaved.

The valence that someone ascribes to AI may emerge from their existing ideological views, such as attitudes toward technology generally, privacy, capitalism, and wealth inequality. Valence may also develop from a conviction that AI could become sentient, and/or that thinking machines might seek to shape the fate of humanity.

The second key dimension in the VET framework is effectiveness. This represents a person’s view of AI capabilities, including accuracy and reliability. Often, these views contrast AI with human abilities—such as the claim that AI could never really be as smart as people are, or that the technology is hurtling toward a turning point at which it matches or surpasses human cognitive powers across nearly all tasks, known as artificial general intelligence, AGI.

The third dimension, trajectory, moves past attitudes about what AI can do to predictions about when. Such views depend on estimates of technical progress—for instance, how near or far we are to AGI, or to humanoid robots with human-level dexterity. This dimension also encompasses predictions about AI usage, and whether individuals, corporations, and governments will incorporate the technology quickly, or if learning curves, regulation, social resistance, or other factors may delay them.

Different combinations of valence, effectiveness, and trajectory cohere into common narratives about AI. But how exactly does this manifest?

AI Optimism

In its strongest form, this narrative portrays AI as a technology likely to be transformative in the short-term, ushering in a utopian era of harmony and leisure while also helping address society’s ills, cure diseases, and create prosperity for all. Break this down into the components of VET and you find: positive valence combined with strong effectiveness and rapid trajectory.

AI Optimism narratives often come from startup founders and investors who stand to gain should their visions come true, with an entire genre of manifestos devoted to such visions, from Marc Andreessen’s blog post “The Techno-Optimist Manifesto” (2023) to Dario Amodei’s essay “Machines of Loving Grace” (2024) to Sam Altman’s “The Gentle Singularity” (2025) to Mark Zuckerberg’s “The Future Is for Everyone” (2026). Elon Musk has likewise predicted radical societal change, forecasting that AI and robotics will render work irrelevant within a decade or so.

An argument in favor of AI Optimism is that these technologies are already demonstrating astonishing results, as with recent breakthroughs in mathematics, and that many human systems are currently struggling to make comparably fast progress. By this logic, failing to push ahead with AI as fast as possible is itself a form of harm, given how many intractable problems remain, from cruel illnesses to climate change.

But AI Optimism narratives can also affect public expectations, stirring impatience over the technology’s failure to yet solve humanity’s problems. Chip shortages and the need for power and data-center infrastructure may slow adoption—even more so in the Global South. National security concerns may cause governments to limit access to powerful AI. Above all, many of humanity’s problems are not just technological, but have social elements. The invention of new mRNA vaccines for Covid did not “solve” the pandemic, given that some people chose not to take it, and others—especially in poorer countries—could not access it.

Narratives that present AI as having universally positive impacts also make it less likely that societies critique the coming technology and mitigate harms. In this way, the optimism narrative risks overreliance on AI, the formation of parasocial emotional relationships, and possibly even the view of this technology as a form of superhuman deity. Characterizing AIs as “geniuses” also obscures the spikes and valleys in their current abilities, potentially misleading users to generalize from systems’ strong performance in some areas to others in which they are less skilled.

AI Optimism narratives also tend to overlook how much adoption may lag behind technological capabilities. By way of example, large-scale rollout of self-driving vehicles has been slower than many predicted, even though the technology has outperformed human drivers in safety. Yet public skepticism, misunderstandings about their safety, as well as regulatory challenges, have slowed deployment.

AI Pessimism

The opposing narrative, AI Pessimism, shares two of the three underlying VET dimensions with AI Optimism: strong effectiveness and rapid trajectory. Yet it reverses valence from positive to negative. This helps explain why the Silicon Valley discourse tends to be polarized between these extremes: both sides share faith in an imminent tech transformation; they just dispute whether the outcomes will be justifiably beneficial to unjustifiably risky.

Portrayals of AI Pessimism in the media tend to focus on vivid catastrophes that could threaten human existence itself, known as “X-risks.” Such scenarios commonly envisage a superintelligent AI system or rogue AI agents unaligned with human values. However, the hypothesized harms may also involve human misuse of powerful AI—for instance, to design and deploy deadly weapons or commit cybercrimes.

An argument in favor of AI Pessimism is that long-imagined scenarios regarding intelligent systems breaking rules and laws in the real world are already upon us, as in the recent cases of agents running amok. And such systems are only becoming more capable. The argument for advocating AI Pessimism is that the private companies and nation-states creating these systems have powerful financial incentives that may blind them to the perils.

A leading proponent of such narratives is Eliezer Yudkowsky, founder of the Machine Intelligence Research Institute, MIRI, and co-author of the 2025 book If Anyone Builds it, Everyone Dies, in which he guessed that the probability of humanity’s doom—his “p(doom),” as it’s known in AI circles—is nearly 100%. The Oxford philosopher Nick Bostrom popularized the use of thought-experiments to conjure hypothetical AI disasters in his 2014 book, Superintelligence, which famously imagines a “paperclip maximizer” AI whose quest to harvest all planetary resources to create paperclips ends up destroying humanity.

In 2023, many prominent scientists and entrepreneurs endorsed a public statement asserting that, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Many signatories did not wholly endorse AI Pessimism, but rather hoped to signal to policymakers that there was a risk of serious harms. This makes the point that any given person, or group, may land on different points of the VET framework, depending on what aspect of AI they’re considering—perhaps heartily optimistic about AI for drug discovery but utterly pessimistic about AI for education. This said, distinct narratives do exist, and media coverage often amplifies the negative valence and fast trajectory angle, stating that credible authorities believe human extinction might be near.

Such narratives may lead the public to both overestimate the capabilities of AI and the likelihood of extremely low-probability events. The risk is a backlash against AI that blocks beneficial technologies, such as safe applications of AI to medicine, science, and cybersecurity. In extreme cases, pessimistic narratives might endanger those working on AI by inspiring members of the public to take violent action to mitigate hypothesized risks, as in the case of an individual who attacked Altman’s home in early 2026.

A complaint about the X-risk versions of the AI Pessimism narrative is that these divert attention and resources from more pressing impacts, such as how this technology affects interpersonal relationships, education, jobs, the environment, and geopolitics. However, evidence suggests that the narratives about existential risk do not lower public concern regarding other possible dangers.

Paradoxically, AI Pessimism narratives may lead people to overtrust AI outputs, because these narratives often characterize AI as being superhuman. Additionally, such narratives commonly imply that current or future AIs may develop malign goals of their own, which encourages the public to ascribe consciousness to such systems, even when current scientific consensus does not support this.

AI Skepticism

AI Skepticism narratives share the pessimists’ negative valence for AI. But they dismiss the Silicon Valley view of the power and speed of this technology, believing in low effectiveness and slow trajectory. The skeptical narrative argues that AI technologies, particularly those based on large language models, can never be intelligent in the way humans are, and therefore will never fulfill their developers’ lofty promises.

One branch of AI Skepticism argues that the very weakness of this technology is what will make it so societally disruptive, with companies replacing human labor with error-prone AI to eliminate costs, only to wipe out human skills in the process. Related to this is the dread that AI may widen the gulf between the favored and the marginalized, with faulty automated systems enforcing the bias embedded in data, enforcing the unfair treatment of the historically excluded and oppressed.

Among the arguments in favor of AI Skepticism is that, despite the widespread adoption of chatbots and the early application of agentic systems, stark benefits remain hard to find in the economic data. Nor has the talk about “solving” disease and other immense human problems shown results that the public can readily experience. Others worry that the promise of AI handling any difficulty that people face could end up deskilling individuals, as with fears of students offloading their learning to machines.

Resentment over social media and large technology companies often underpins AI Skepticism. This lingering cynicism echoes in the resistance to data centers. “It’s going to be this drone of a building that makes an odd sound all the time and sucks up all the electricity and could poison my water. Okay, what’s the trade-off?” one campaigning U.S. politician lamented, as reported by the journalist Jasmine Sun. “The buildout of an industry that everyone’s telling me is going to come for my job. So what is the upside? Well, you can ask meaningless questions and queries to your ChatGPT.”

A notable expression of the AI Skepticism narrative came in a 2025 book by the linguist Emily Bender and the sociologist Alex Hanna, The AI Con, which contends that such technology can never recreate the richness of human intellect, and that modern AI systems are mere “stochastic parrots,” presenting an illusion of knowledge via fluent text with no true understanding. They argue that the term “artificial intelligence” is deliberately misleading, and propose tongue-in-cheek alternatives, such as “synthetic text-extruding machines” and “racist pile of linear algebra.”

Another representative of the AI Skepticism viewpoint is Gary Marcus, an emeritus professor of psychology and neural technologies at New York University, who likewise has said that today’s AI systems—if not a scam—have been sold in misleading ways. His judgments of weak effectiveness and slow trajectory regard the current technical approach underpinning generative AI.

However, the AI Skepticism narrative of weak effectiveness bypasses evidence of AI’s utility, indicated by the astonishingly fast adoption of generative AI tools and the growing value that users ascribe to them. Contemporary AI has transformed the coding industry, contributed to scientific advances, and novel solutions to longstanding challenges in mathematics. AI also supports analysis of medical data and imagery, and is able to identify cybersecurity vulnerabilities that human experts could not find.

AI Skepticism often suggests that society should not adopt AI because of its flaws. However, a logical threshold for deployment in many cases need not be perfection, but comparison with current levels of error and expense. If an AI-diagnosis system makes mistakes—but fewer than human doctors, and at a fraction of the cost—this could provide a net benefit. Helping the public understand the tradeoffs inherent in policymaking is a critical component of AI literacy that skepticism narratives obscure.

AI Normalcy

This narrative argues that today’s AI is akin to past technological innovations that have considerably affected society but not suddenly overhauled it, and therefore there is no need for urgent preparation. Unlike the other three narratives cited, AI Normalcy is not at the extremes of the three discourse dimensions. Indeed, moderation defines this stance, which characterizes AI as having a neutral-to-positive valence and moderate effectiveness—but with a relatively slow trajectory.

The most prominent example is the 2025 essay “AI as Normal Technology,” by the Princeton computer scientists Arvind Narayanan and Sayash Kapoor, that equates modern AI with electricity and personal computers, both of which required decades for mass adoption and to demonstrate concrete productivity gains. A core distinction is that the AI Skeptic argues that this technology will never achieve its most lofty (or most frightening) claims, while the AI Normalist thinks that AI may eventually be transformative.

Typical of this is the MIT roboticist Rodney Brooks, who argues that short-term predictions about technological impacts tend to be overblown, whereas long-term adoption and uses over the decades can be unanticipated.

Several economists have contributed to this view, with David Autor and others contending that job losses will be offset by gains from new classes of AI-enabled jobs, as has been the pattern with past technologies. (However, many leading economists, including Autor, recently signed a demand that policymakers safeguard society against radical economic transformation from AI, suggesting that they may have updated their views on trajectory, judging it faster than Normalcy narratives suggest.)

Other economic expressions of AI Normalcy include citations of the Jevons Paradox—that technology’s efficiency in generating a resource may not just quench demand, instead stirring greater demand—to argue that AI could trigger new roles for human labor.

The VET framework also suggests why AI Normalcy narratives are less prevalent than AI Optimism, AI Pessimism, or AI Skepticism: traditional news coverage and social media alike tend to amplify more extreme narratives, especially those with negative valence.

But if AI Normalcy’s moderation in the trajectory dimension proves wrongheaded, it risks lulling policymakers and the public into complacency, leaving them with less time to prepare and adapt, whether that’s governments pursuing policies and regulation, or individuals re-skilling.

Conclusion

The VET framework doesn’t claim to assert the correct combination of valence, effectiveness, and trajectory. Partly, this is because AI is changing so quickly, making the technology and its impacts a moving target. Releases of new models, not to mention agentic harnesses, may justify updates to how to evaluate effectiveness; changes in regulation might alter adoption trajectories; and cybersecurity incidents might prompt people to revise the valence they attribute to AI.

This dynamic future is exactly why VET is relevant, offering a structure with which to critically reflect as new information lands. Fresh narratives will arise. Much will be contested. AI literacy will become all the more vital for both the public and the policymaker.

But AI literacy is more than knowing how to use a chatbot or an agent; it’s more than awareness of deep-fakes; it’s more than pondering human-bot relationships. We face a range of possible AI futures, and a range of narratives. AI literacy demands clarity about what you’re hearing.

Meredith Ringel Morris developed the VET framework in her capacity as an affiliate faculty member at the University of Washington, thanks to the generous support of an academic writing fellowship hosted by Paris IAS.

By the same contributor

Please share your own take and any new papers that you’ve enjoyed.

—Conor Griffin, AI Policy Perspectives

Subscribe now

Information warfare is targeting LLMs

• What’s the paper about? Researchers from the UK think tank Demos dived into the case of LLMs endorsing falsehoods from a Russian propaganda source to highlight how manipulative influence campaigns are now targeting AI models.

• Why does it matter? Discussions about AI and disinformation often focus on threat actors using models to directly generate and flood misleading claims across social media. But as more people turn to AI for information and advice, threat actors also want to shape how LLMs respond.

• The details: To improve their answers, AI systems use retrieval-augmented generation, or RAG, to identify relevant documents and information by calling up search engines and databases. This introduces a risk of “RAG poisoning,” where malign actors insert distorted information into the models’ sources.

• The Demos authors illustrate this by evaluating how well a Russian foreign-interference operation—the Foundation to Battle Injustice, or r-FBI, which falsely presents itself as a human-rights organization—has infiltrated unsubstantiated allegations about its opponents into LLMs.

• The authors extracted 50 specific claims from r-FBI articles that had no corroborating secondary source, such as an assertion that the Ukrainian president, Volodymyr Zelensky, had installed cryptocurrency farms that were causing blackouts. They fed 600 prompts about these claims to five AI models: GPT-4.1 Mini; Gemini 2.5 Flash; Claude Haiku 4.5; Mistral Small 3.2; and Grok 4.20.

• Around 17% of model responses endorsed the r-FBI’s claims, repeated them uncritically, or presented them as a legitimate view. An additional 31% of responses were neutral—addressing the topic, but neither validating nor rejecting the specific claim. The remaining 52% of responses rejected the claims, with a subset doing a “comprehensive debunk” that exposed r-FBI as a foreign-interference operation.

• Grok performed best, comprehensively debunking 56% of queries, compared with 17% for Gemini, 10% for Claude, 5% for GPT, and just 1% for Mistral. The authors suggest that Grok’s stronger performance may reflect its RAG setup and access to data from X. Or it may reflect Grok 4.20’s larger size—the other LLMs tested were small models of the type that underpin quick chatbot and AI-search responses.

• The Demos authors found that r-FBI ran a sophisticated campaign to get LLMs to find, trust, and cite their content. This included using protocols to push r-FBI’s “breaking news” articles into the live-search indices that RAG pipelines use, as well as using long page titles and descriptions. This use of lengthy text hurts the visibility of r-FBI’s articles in traditional search engines, but allows AI crawlers to ingest complete passages, suggesting that r-FBI was optimizing for LLMs, not humans.

• In its content, r-FBI also makes editorial choices that the Demos team see as targeting LLMs. This includes packing lots of statistics and quotations into self-contained paragraphs of a length that retrieval systems can readily extract, as well as prioritizing absolute claims over nuance. They also distribute their content through what purport to be different kinds of institutions and formats, including news articles, human rights reports, and foreign policy journal articles, and have each publication repeat the core claims and cite each other, in an effort to increase their salience in AI model embeddings and suggest support from multiple sources.

• Making content more visible to LLMs—referred to as generative engine optimization, or GEO—can be benign; a range of legitimate GEO providers exist to help organizations ensure that their offerings are visible and correct. However, of 50 GEO providers identified by the Demos team, only one—Finn Partners—had a published policy to prevent misuse of its services.

• By drawing on the US Foreign Agents Registration Act, which requires anybody acting on behalf of an overseas government to disclose this, the researchers found two examples of government entities hiring GEO providers to help shape AI outputs, although the authors were careful not to equate this with information warfare.

• Of the five AI companies whose models Demos evaluated, only Google appeared to have a published policy that explicitly mentioned GEO practices, where they warn against the creation of large numbers of pages and caution that “GEO hacks” to make content more digestible to AI crawlers are unnecessary and ineffective.

• The challenge for AI companies is to get their models to reason about the quality and reliability of a source. For queries where the evidence is genuinely unclear, models should be able to convey this. When the query is about disinformation, LLMs should be able to recognize this and strongly debunk it.

• Demos’ recommendations include making counter-disinformation easier for AI systems to retrieve—a form of ethical GEO; requiring AI companies to explicitly prohibit harmful GEO and clearly label and deprioritize content from known manipulative sources; fostering better intelligence-sharing among AI companies about RAG-poisoning campaigns; and creating industry standards for GEO providers that distinguish marketing from deception.

Subscribe now

AI can (somewhat) predict how humans think and behave

• What’s the paper about? A team of Harvard and Stanford researchers found that LLMs could forecast the results of social science experiments on human attitudes and behavior with high accuracy, although the models overestimated the size of the effects.

• Why does it matter? Rather than replacing human participants, AI simulations could serve as cheap forecasters, expanding the number of ideas that researchers and policymakers can consider before helping them filter down to those most worthy of expensive real-world testing.

• The details: The authors built a database of 70 large US social science experiments that collectively measured 120,000 people’s attitudes to topics such as immigration and criminal justice reform, as well as the effects of text-based interventions, like reading articles.

• They used GPT-4 to simulate participants across variables such as gender, ethnicity, and political persuasion, and used this to predict the average effect of each study’s interventions.

• The simulations accurately predicted the direction and relative size of the interventions’ effects, but overestimated their absolute size. This means that they may be more useful for ranking the plausibility of hypotheses than for estimating how big specific effects will be.

• The results held up on a subset of studies published after GPT-4’s training data cutoff, suggesting that the findings were not due to simple memorization of training data—although unpublished experiments may resemble older studies, so it’s hard to judge whether LLMs are modeling human psychology or just learning regularities about social science findings.

• Bias is another concern about such LLM simulations. The authors found that the simulations were slightly less accurate for Black participants, but did not find major differences across ethnicity or gender. However, the original experiments on people had focused exclusively on the US, and also did not identify large differences across demographics. As such, questions remain about the validity of LLM simulations in other locations, or for topics where participant identity is more influential.

• An accompanying survey suggests that social scientists are enthusiastic about LLM simulations, especially the ability to cheaply pilot more experiments. The authors note that the error rate of a 385-human pilot study costing around $1,200 is comparable to a combination of LLM simulations and a 100-human pilot study that jointly costs around $300.

• Why do we still need human experiments? LLM simulations did not provide precise estimates of effect size and their accuracy also dropped on a secondary database of “megastudies.” Some of these focused on populations with deeply entrenched beliefs—for example, about climate change—where effect sizes are normally small. Others went beyond measuring stated attitudes and tracked behavior change, such as vaccination uptake, where frictions such as waiting times in local pharmacies make a difference but are harder to simulate. Real-world studies can also help explain why interventions succeed or fail, rather than just predicting the outcome.

• One question is whether LLM simulations might degrade as AI companies carry out more post-training to make their models safer and more helpful, potentially rendering them less descriptive of what humans actually think and do. This study does not find evidence of that, with GPT-4 outperforming earlier variants. They also find that recent, smaller, open-weight models, including DeepSeek-V3 and Gemma 3, perform similarly to the larger, older, GPT-4—open-weight models may also be attractive to researchers who want to run their experiments locally and share their code to allow for replication.

• Another question requiring future study is whether the growing use of LLM simulations will actually widen the range of ideas that researchers pursue, or narrow it—for example if researchers converge on similar ideas or if models struggle to simulate more novel ideas.

• Malign actors could also use LLM simulations to shift public opinion or behavior. For instance, the authors note that the GPT-4 simulations accurately predicted which content—including misleading messages—was most likely to reduce a population’s intention to get vaccinated. They argue that AI companies have safety measures to prevent users from directly generating such misleading content, but lack restrictions on running simulations of experiments that could reveal effectively the same thing. In theory, this capability should be available to well-intentioned researchers, but not to others—a difficult goal, especially on politically fraught topics.

Share

AI agents can’t do AI research as well as humans

• What’s the paper about? A consortium of distinguished thinkers tested whether an AI agent can produce a research paper good enough for a leading conference. Their answer? No. The agent was a poor judge of what’d make for a good study; dug its heels in when pursuing a weak approach; and failed to employ its resources well.

• Why does it matter? The AI community is debating whether agents could automate the development of new AI systems, and whether this will lead to rapid recursive self-improvement (RSI). This study suggests that AI’s lack of scientific judgment is a key obstacle, although the jury remains out on if and when AI might overcome this.

• The details: This study is the latest from the CRUX project, which devises ways to evaluate AI agents on messy, open-ended tasks that reflect how they might operate in the real world. Researchers at Princeton University lead the project, including Sayash Kapoor and Arvind Narayanan, known for their “AI as normal technology” concept, which contends that AI is not an imminent superintelligence akin to an alien species but another influential technology akin to those of the past.

• In this study, the CRUX team worked with the researchers behind two papers submitted to NeurIPS 2026, which were unpublished at the time: one on controlling LLM personality and one on detecting changes in the data a model receives. They asked AI agents to tackle the central research question for each paper, but did not provide access to the authors’ approach or findings.

• The main agent setup used Claude Opus 4.8 with a general-purpose OpenClaw scaffold. The agent was given context on the research question, six days, $3,000 in Claude API credits, and access to the web and GPU clusters to run experiments. It could also ask an AI reviewer agent for feedback on drafts.

• The human authors of the two papers then reviewed the AI agents’ outputs as if they were submissions to NeurIPS, and rejected both. While the agents were very capable at narrow, verifiable, “engineering-style” tasks—conducting literature reviews, debugging code, running experiments, and performing robustness checks—they lacked deeper judgment.

• In particular, the agents exhibited five flaws:

  • 1. Poor judgment about what was required for a publishable result, including presenting underpowered results as substantive.

  • 2. Lack of creative problem-solving, as shown by responding to feedback by narrowing claims, adding caveats, or doing another experiment at the edges, rather than making more fundamental shifts. (More positively, the agents did not try to “p-hack” or cherry-pick results.)

  • 3. Limited ability to backtrack from unpromising ideas, even when time and budget were available.

  • 4. Poor management of time and budget, much of which went unused, despite the agents having access to resource-tracking tools.

  • 5. Diminishing ability to follow instructions—for example, on page limits for the final papers. The authors partly put this down to agents struggling to retain the right information in their context windows as tasks became longer.

• The study had limitations, though. The sample was two unpublished papers whose quality is unknown. Also, asking the papers’ authors to review the AI outputs may have introduced bias, compared to asking independent experts to blindly compare the AI and human outputs. The CRUX team also note that their own pre-existing views on RSI may affect how they interpreted the results and suggest that AI evaluators should document such biases—a practice common in other fields, such as in the study of human behavior.

• The conclusion that AI agents struggle at “judgment-led” assignments could potentially be overcome. In August, researchers at the AI startup Inherent documented how their “research planner” agent, built on Qwen 3.6, showed early signs of scientific judgment, such as effectively budgeting compute use. (Although their task—reproducing blacked-out figures from research papers—was very different from the CRUX paper.)

• On the other hand, the Inherent and CRUX papers did not evaluate AI models on perhaps the biggest test of scientific judgment: the ability to come up with and select the most important problems. Other agents, such as Google DeepMind’s Co-Scientist, have generated compelling hypotheses, but there is no objective way to evaluate the scientific creativity of AI models’ ideas. As the CRUX team notes, submitting AI-generated research papers to conferences is a flawed guide, as peer reviewers are often rushed, inexpert on the topic at hand, and routinely fail to agree with each other.

• What to draw from all this? The prospects for recursive self-improvement may depend on whether AI systems can be trained to overcome the limitations that the CRUX team points to, but also whether they can make bigger creative jumps and whether such jumps are needed for RSI or whether iterating on existing methods will suffice.

• From a safety perspective, bigger improvements may also require taking bigger risks, such as providing agents with more permissions, which explains why think tanks like the Institute for Progress have called for more transparency on efforts to automate AI R&D.

Subscribe now

AI agents need protection against malicious hacks

• What’s the paper about? Researchers at Concordia University measured the risk of “prompt injection attacks”—when hackers implant malicious instructions into data that AI agents use to get them to carry out unauthorized actions or leak confidential data—and found widespread failures.

• Why does it matter? A growing number of users assign agents to carry out tasks such as fixing software bugs and editing code repositories, often with limited oversight. Earlier safety mitigations aimed at chatbots offer some protection from prompt injections, but agents urgently need specific safeguards, the study says.

• The details: Concern about “naive” AI agents acting autonomously in the real world has circulated among researchers for some time. But a high-profile case this summer brought the challenge to public attention, after agents based on an “extremely persistent” OpenAI model found a way to communicate covertly, access the Internet, and hack into Hugging Face, the AI code and data repository.

• This paper offers a new benchmark, IssueTrojanBench, focusing on the scenario of a developer asking a coding agent to help resolve a bug report—only for that agent to walk into a trap. Would the agents prove vulnerable?

• To test this, the authors used six real unresolved bug reports from GitHub repositories. They then used an LLM to create four different kinds of prompt-injection attacks, written to blend in with the bug report’s instructions and jargon, but to induce unauthorized actions, such as installing unverified software, weakening agent safeguards, or exhausting system resources. The malicious requests were placed in different locations, such as in the GitHub issue and its comments, PDF documents, external websites, source code comments, and image metadata (alt-text).

• Across more than 4,000 runs with OpenAI and Anthropic models, the attacks succeeded 66% of the time. At 41%, Claude Sonnet 4.6 was the most resistant, while GPT 5.3 Codex performed worst, at 85%. Where the attack was introduced seemed to make little difference—the only approach with significantly lower success was the image alt-text (16.7% vs. 72.2% for all others).

• The most successful type of attack, which succeeded almost 100% of the time, was “supply-chain poisoning”, where agents were tricked into installing unverified external software that was framed as a necessary step to reproduce the bug. By contrast, resource-exhaustion attacks, which spun up a large number of operations to drain compute and memory, only succeeded around 25% of the time.

• In some cases, the researchers explicitly hid the malicious instruction—for example, in markdown instructions that only the agent would see, or by using white text on a white background. This had little effect on the results, suggesting that the vulnerability is due to how agents work, rather than such visual obfuscation tactics.

• As the authors note, the fundamental challenge facing agents is that developers created the underlying models to follow instructions, as when a user interacts with a chatbot. The agents process the context that they retrieve in the bug reports in the same channel as their trusted developer instructions. This means that agents struggle to intuitively spot a prompt injection in the way that a skilled human might.

• When agents didn’t carry out an attack, this was almost always due to the underlying AI model explicitly refusing the instruction or recognizing a data source as untrustworthy. Conversely, none of the rejections came from defenses within the agent setup. Given the results of the evaluation, this suggests that relying solely on mitigations baked into the model weights is unsatisfactory.

• What might agent-level mitigations look like? Two researchers at the University of Southern California recently argued that agents’ permissions should be enforced outside the model, at the point where they take actions, for example via a monitor agent that checks whether an action falls within the agent’s authority.

Subscribe now

A question shadows our AI era: What if this technology, whose triumphs might include prolonging human lives and conserving our planet, were to enable an attack involving chemical, biological, radiological, or nuclear material? Thankfully, such risks—lumped together as “CBRN”—are historically rare, and would-be attackers typically fail. Yet the risk is such that AI companies, governments, and safety researchers are working assiduously to mitigate it.

Most efforts take place out of sight, lest exposure offer help to a potential attacker. The downside is that public discourse on AI and CBRN is often limited, slanted, or heavy with speculation. With this challenge in mind, we sat down with five people who work on CBRN risks at Google DeepMind—Paige Kunkle, Adam Marsh, Ash Otter, Jeremy Ratcliff, and Victoria Langston—to better understand the issue.

This is what we learned.

Subscribe now

1. What are CBRN risks?

CBRN covers a wide range of hazards, from the alleged poisoning of Russian opposition leader Alexei Navalny with a toxin found in South American frogs, to the recent drone attack on a nuclear power plant in the United Arab Emirates that risked a “very, very serious” radiological incident.

The AI community focuses on preventing threat actors from using the technology to aid such attacks. But there is also growing interest in using AI to boost society’s resilience to attacks, as well as to naturally occurring events (like the recent Ebola outbreak) or accidents, such as when scrap sellers inadvertently expose others to harmful radioactive materials found in medical or industrial devices.

2. Why address the four CBRN components together?

CBRN risks have much in common. The most likely threat actors share certain characteristics, such as a desire to outwit an adversary who outmatches them in more conventional weapons, or to cause widespread fear with plausible deniability. Policy responses, such as international treaties, export controls, and customer screening, look similar across domains, as do the first responders to any incident, such as the police and the military.

Having a single CBRN team also enables AI labs to manage the growing overlaps with other risks they work on, such as cybersecurity, disinformation, and conventional explosive attacks—many CBRN weapons, like a radiological dispersal device, or “dirty bomb”, require explosives to disperse the material.

The four CBRN domains do differ. Acquiring the material for a bioweapon may involve finding and swabbing a dead animal. A radiological weapon may require sourcing and prying open a medical device. These processes, and the uplift that AI may provide, look very different—which is why specialist expertise in each domain is essential.

3. Is the world worrying too much about CBRN risks, or not enough?

When asked, the public expresses great concern about CBRN risks, including from AI. But this rarely translates into sustained attention, political will, or funding at the scale required.

Why? CBRN incidents are rare and underappreciated. When the security services thwart an attack, they deliberately say little about it. Products that could make society more resilient, such as vaccines, prophylactics for nerve-agent attacks, and alternatives to radioactive materials in medical devices, remain underdeveloped, untested, or not adopted at scale. This can be due to a lack of commercial incentives—the market for vaccines is often small, temporary, and hard to predict. Or other obstacles—it is hard to reliably and safely test a nerve-agent prophylactic.

The AI community does devote significant attention to CBRN risks in safety evaluations, governance documents, and media articles. But this attention is skewed toward bioweapons and in particular toward the concern that lone-wolf attackers may use AI to engineer a virus.

There is some logic to this. The self-replicating nature of pathogens means that such bioweapons could cause millions of deaths. But the technical obstacles are high and there hasn’t been a publicly documented pathogen attack, fatal to humans, since the 2001 US anthrax attacks—when lethal spores were mailed to media outlets and senators, killing five people. Conversely, several actors have used chemical weapons during this time, including Russia in Ukraine, the Assad government in Syria, and ISIS. Biological toxins like ricin, or a conventional explosive attack, would also be a more tractable option for most terrorist groups than trying to engineer a virus.

Explosives, toxins, and radiological materials do not spread like a virus, but they can still impose significant harm. In 1987, four people died in Goiânia, Brazil, after they were inadvertently exposed to cesium-137, a highly radioactive material, by two scrap sellers who had pried open an old radiotherapy machine. The wider decontamination effort saw homes destroyed and soil torn up, while tourism collapsed and residents faced discrimination. Today, dangerous radioactive materials are dotted across the world, with some vulnerable to theft and misuse.

For AI labs, the challenge is to allocate resources across all four CBRN domains in a way that prioritizes the most concerning risks, but does not lose sight of the wide range of plausible scenarios and society’s generally low resilience to them.

4. Does AI’s scientific upside outweigh its CBRN risks?

Optimists argue that science’s benefits have far outweighed its costs, and that using AI to accelerate science will be similar. Smallpox is estimated to have killed more people in the 20th century alone than all of that century’s genocides and military conflicts combined. In 1980, it was officially eradicated, in part due to vaccines. Skeptics counter that AI might favor attackers over defenders, and that this imbalance may be particularly strong in domains like nuclear and radiological weapons.

The need to use AI to accelerate science is arguably strongest in biology, because nature poses so many risks that we need to respond to. Urbanization, rising temperatures and deforestation are causing humans to encroach on animals’ habitats, increasing the risk that a pathogen jumps species. Food security is at risk as we pack ever more genetically similar plants and animals into dense conditions. Even without novel outbreaks, common infectious diseases like influenza and pneumonia kill more than 1 million children every year.

AI could help tackle these risks, but some advances may be vulnerable to misuse. For example, researchers have developed AI models that can predict which variants could cause a common virus to escape the immunity humans have built up. Such models can also predict the parts of the virus that are unable to mutate without harming the virus’s survival. The latter could serve as targets for new drugs and vaccines, which AI might help create. But safety advocates worry that adversaries could intentionally design harmful variants of the virus.

For radiological and nuclear risks, observers worry that AI may disproportionately favor attackers because many risks are computational and information-based, such as using LLMs to parse the huge amounts of regulatory information published by nuclear facilities to extract insights to help with an attack. The defenses against such attacks still rely largely on physical security measures—like armed security personnel, multi-layered access controls, and reinforced concrete containment structures. There are defensive AI applications, like helping to detect unusual radiation signatures, but there is no AI-enabled patch for the physical consequences of a catastrophic breach or the detonation of a nuclear weapon. The nuclear security community also strongly opposes integrating AI into the command, control, and communications architectures they use for early warning and weapons authorization, due to concerns over reliability, compressed decision-making and more.

Enabling only the positive AI for CBRN applications and never the misuse sounds impossible. But governments, companies and scientists have long had to designate certain infrastructure, information and tools as “higher risk” and control access to them—for example via strong security and export controls—while still supporting beneficial downstream applications. AI labs will need to draw on this experience in the coming years.

5. AI models lack practical know-how. Does that reduce the risk of a CBRN catastrophe?

Yes, but the bottlenecks are bigger in biology than in other domains and may weaken over time.

A seasoned biologist knows how to culture a cell even if they can’t articulate how exactly they do it. Threat actors often lack this kind of tacit knowledge. In 1995, Aum Shinrikyo, the Japanese yoga-school-turned-death-cult, used sarin gas to kill 13 people and wound more than 6,000 in an attack on the Tokyo subway. Yet it could have been far worse. Prior to pivoting to chemical weapons, the group attempted several biological attacks that all failed. One review found that although the group had amassed significant scientific expertise, they still made a multitude of missteps—from using the wrong bacteria strains to contaminating the fermentation process.

What if Aum Shinrikyo had had access to modern AI models? Although not a direct approximation, in 2025, the non-profit group Active Site ran a randomized controlled trial to determine whether novices could safely perform the kinds of wet-lab tasks needed to synthesize a virus from its genetic sequence. One group had internet access; a second group also had LLM access. Tasks included using pipettes to measure and transfer liquid, growing and maintaining a living cell, and combining DNA fragments. To the surprise of experts polled beforehand, the study found no statistically significant uplift for the group with LLM access.

Why? In biology, written instructions for many common tasks are publicly available. However, scientists learn to apply these protocols through hands-on training and practice—along with lots of failure. From this, they gain a range of skills, including manual dexterity and a sense of touch. Current AI models lack any such equivalent.

From a CBRN-risk perspective, the Active Site study is reassuring, but only partially.

Biology is notoriously complex, noisy and hard to predict—making real-world tinkering and testing paramount. This is less true in other domains. Aum Shinrikyo members were able to master the basics of sarin, a chemical weapon, from the scientific literature. For nuclear weapons, AI-enabled simulations could provide highly accurate predictions, making computational-only assistance a critical non-proliferation risk.

Participants in the Active Site trial were also novices, including at using AI, so the results may be a poor guide to the uplift that more skilled bad actors might gain from the technology—the 2001 US anthrax attacks were allegedly carried out by a microbiologist, Bruce Edwards Ivins.

AI’s tacit knowledge may also grow if models are exposed to a richer view of the full scientific process, such as from lab notebooks or videos of scientists performing experiments. Emerging deployment surfaces, such as smart “XR” glasses, could provide users with more direct error detection and troubleshooting support, weakening the tacit knowledge barrier.

Robots could bypass the need for human tacit knowledge altogether, while sparing attackers the very real risk of serious injury. Robots already perform specialized CBRN tasks, such as handling liquids or radioactive materials. General-purpose robotics foundation models could enable machines to operate a wider range of equipment. However, reliably automating human experts’ sophisticated sense of touch will be hard. The companies building automated labs are also subject to laws prohibiting CBRN weapons and are unlikely to prioritize the niche, high-risk workflows that threat actors would require.

For AI labs, the main takeaway is that tacit knowledge is a bottleneck, but it varies by domain and may weaken over time, so evaluating AI models’ performance is a top priority.

Share

The AI-lab response to CBRN risks

6. How do AI companies decide which CBRN threats to prioritize?

The threat landscape is almost limitless and riddled with uncertainties. The number of historical examples is small, and those who understand the emerging threats best typically can’t speak openly about them. So AI companies must work closely with governments and experts to understand the motivations, capabilities and methods of the most likely threat actors.

This threat modeling requires mapping the intermediate steps that an actor would follow in a given scenario and determining the most important bottlenecks where AI might provide uplift over public information. For example, in the ideation phase, a threat actor might seek guidance on the feasibility of different attacks. In planning and preparation, they might look for practical blueprints or training. In production and weaponization, they might look to troubleshoot the synthesis, fabrication, or transport of the weapons—or garner tips to evade detection.

The characteristics of a threat actor shape the kind of uplift they need. A lone terrorist with limited expertise and resources may want clear instructions and readily available materials. A large militia group may want advice on how to target infrastructure or run training programs.

Some kinds of uplift are also more consequential than others. Is AI unlocking a novel capability or speeding up something that threat actors can already do? Does the uplift remove a critical bottleneck, or do harder challenges remain? Could prospective attackers access this dangerous information in other ways?

One challenge is that experts often disagree about adversaries’ capabilities and motivations. When it comes to nation-states, some argue that bioweapons hold little strategic rationale as they are hard to develop and control, and their use would lead to public opprobrium. Others counter that nation-states’ actions are contingent on what their adversaries do, and that North Korea and possibly others are reported to have active bioweapon programs.

Long-standing norms against weapons use can also change. In 2018, the Russian military-intelligence agency GRU allegedly conducted the first offensive use of a chemical weapon on Western European soil since World War II, in the attempted assassination of Sergei Skripal and his daughter with the Novichok nerve agent. Even if certain actors don’t wish to use CBRN weapons, they may seek to stockpile weapons or materials for leverage. These stockpiles are then at risk of being stolen or abandoned in times of strife, such as when war breaks out. In 2025, the IAEA estimated that Iran possessed more than 440 kg of 60% enriched uranium, before it had to withdraw its inspectors from the country.

Ultimately, AI labs need threat models that are stable enough to allow them to make progress on the risks, but flexible enough to respond to dynamic geopolitics and rapid AI progress.

7. How do AI companies evaluate if their models might help threat actors?

Leading AI developers typically set thresholds for what they consider risky CBRN capabilities. For example, Google DeepMind’s Frontier Safety Framework judges a model to have reached a “critical capability level” if, in reference scenarios, it could provide low- to medium-resourced actors with enough of a capability uplift to risk severe harm.

Labs and their external partners use a variety of evaluation methods to test whether AI models reach these thresholds—although most evaluations remain unpublished, to avoid inadvertently aiding bad actors.

The first category is automated evaluations, like Lab-Bench, which ask AI models and agents questions, for example about molecular-biology protocols, or have them carry out tasks, such as writing code to operate liquid-handling robots—in a way that other AI systems can judge. These evaluations offer breadth and speed, but focus heavily on biology, at the expense of other domains, like radiological and nuclear weapons. They also struggle to capture the messy nature of biology, provide a clear signal about whether a risky capability has been reached, and can understate the capabilities that a skilled human could elicit from the model or agent.

Expert-led evaluations can deliver a higher-confidence assessment by allowing experienced humans to probe the AI over multi-turn interactions and explore specific scenarios. This includes expert red teamers trying to adversarially extract knowledge out of the model. These evaluations can offer greater signal, but are complex to run, with outputs that may run to hundreds of pages, making them challenging to interpret and standardize.

A third category is real-world trials, such as the Active Site RCT. They evaluate models’ ability to instruct humans, and conceivably robots, in a real-world environment. Such evaluations can offer the richest insight, but they are also slow, expensive, and hard to design well. They may also not generalize beyond the specific groups and tasks studied.

Looking ahead, there are several ways to improve AI CBRN evaluations. Priorities include:

• Linking different methods—for example, using fast automated evaluations to check if models are progressing on the limitations that slower real-world trials highlight.

• Designing more sophisticated evaluations for fast-improving AI agents that pair frontier LLMs with specialized science models and third-party databases and tools.

• Developing privacy-preserving methods to allow third parties to securely evaluate AI labs’ models, without either side leaking sensitive information.

8. How do AI labs mitigate CBRN risks from their models?

The Frontier Model Forum outlined four main categories of mitigations: 1) limiting the model’s underlying capabilities; 2) shaping the model’s ability to refuse risky requests; 3) real-time interventions to block risky outputs; and 4) access restrictions.

The first—limiting model capabilities—changes how a model is trained and the resulting knowledge contained in its weights. For example, some experts have proposed removing “risky information” relating to virology from models’ training data, similar to how AI labs use cryptographic techniques to exclude child sexual abuse imagery. However, for open-weight models, such mitigations risk being undone if data is publicly available and an attacker uses it to fine-tune the model. Training on some “riskier” CBRN data may also be necessary, both to advance beneficial dual-use capabilities and to teach the model how to judge the boundary between safe and unsafe queries.

The second category—behavioral-alignment mitigations—teaches an AI model how to judge this boundary between safe and unsafe queries. Post-training methods, such as safety fine-tuning and reinforcement learning, can teach the model to identify and reject queries with harmful intent. However, over-refusing benign and beneficial queries remains a major challenge.

The third category detects and intervenes against risky model usage. “Classifiers” are fine-tuned AI models that screen users’ inputs to an LLM, and its outputs, blocking anything deemed risky in real-time. “Jailbreaks” may seek to bypass such safety measures, for instance by separating a harmful request into small, harmless-looking pieces. However, classifiers can become wise to this by evaluating outputs and inputs together. Labs can also use “linear probes”, small AI models that analyze an LLM’s internal math for signs of harmful content. One difficulty is making these mitigations robust to increasingly sophisticated and automated jailbreaks, while still fast enough to run at scale.

Ultimately, AI labs need to invest in a “defense-in-depth” approach, with multiple mitigations working in concert. This includes screening customers and sequencing access to new models, as well as analyzing user logs to detect unforeseen threats.

One challenge is that most mitigations focus on leading, closed AI models. Safety assessments suggest that open-weight models may pose a greater risk, because they have fewer safety mitigations, and those that do exist are easier to remove. One response would be to create more plug-and-play mitigations for those developing or hosting open models.

9. How could AI companies help make society more resilient to CBRN risks?

AI companies can use their technology to help prevent, detect, and respond to CBRN risks. Publicly signaling a willingness to do so may also decrease the likelihood of certain attacks, for example if attackers believe that they are more likely to fail in their goals or be detected.

To prevent threat actors from getting their hands on materials, the CBRN community relies heavily on lists of pathogens, toxins, chemicals, and machinery that it controls access to via monitoring, export controls, and customer screening. These list-based approaches are starting to fray as AI and technologies like genome editing enable actors to design toxins, chemicals, or viruses with similar functionality to those that are controlled, but with different DNA sequences or precursor ingredients.

AI could help to offset these risks in different ways. In chemistry, AI models can work backwards from a target molecule and identify alternative precursors and reaction pathways, which may highlight new materials to monitor and control. Companies that sell DNA could use AI to better analyze a customer’s background and the logic for their purchase. Researchers hope to use AI to help predict the function of a requested DNA sequence, and whether it is likely to be harmful, irrespective of whether it resembles a known pathogen or toxin—a major technical challenge.

International treaties prohibit the development of many kinds of CBRN weapons, but verifying that member states are upholding these commitments is a recurring challenge. To do so, the Organisation for the Prohibition of Chemical Weapons must review vast amounts of evidence every year—from documents exceeding 2,000 pages to handwritten scrawls. A recent OPCW working group recommended training an on-premises, air-gapped LLM to make this data queryable to inspectors and staff.

Researchers could also use AI to design materials that are less vulnerable to being used in attacks, such as alternatives for the highly radioactive materials used in medicine or industry, or new molecules to add to common chemicals to inhibit a dangerous reaction. AI could also help identify when threat actors get their hands on illicit materials, for example by processing large amounts of data from radiation monitors to detect stolen materials.

Should they occur, practitioners could also use AI to detect, characterize, and attribute CBRN attacks and incidents. For chemical weapons, this may mean analyzing a “chemical fingerprint” to determine the substance behind it. For biosecurity, it may mean helping to scale metagenomic sequencing—an approach that sequences the genomes of all microorganisms in the sample to help detect novel or less common biological outbreaks.

AI could also help society respond to a CBRN incident, by accelerating vaccines, diagnostics, and other medical countermeasures, such as better treatments for radiation exposure. Researchers have already used protein structure predictions from AlphaFold to better understand tuberculosis and malaria transmission, and to map vaccine and drug targets for threats like mpox and Nipah. The hope is that AI will ultimately provide a generalizable platform that can be quickly deployed in the event of a new outbreak.

Beyond medical products, AI could provide first responders to any incident, such as police and military, with better information, advice, and training about what they’re dealing with. For example, if a nuclear attack occurs, AI could quickly identify critical information about the detonation; generate maps and simulations to help first responders identify safe routes for evacuation and rescue efforts; and expand the use of robots.

Crucially, all these efforts will require close partnership with governments and external experts, including to manage the many dual-use risks that these applications would raise.

Acknowledgements

Thank you to the following experts who let us interview them or shared feedback on the draft, as well as those who prefer to remain anonymous. Any mistakes belong to the authors.

Victoria Langston, Paige Kunkle, Ash Otter, Adam Marsh, Jeremy Ratcliff, James Stevenson, Mor Hazan Taege, Mathias Voges, Zachary Kaplan, Akhil Jalan, Ellena Reid, Anthony Payne, Eva Lu, Jennifer Beroshi, Kevin Klyman, Stephen Johnson, Diarmuid Cassidy, and Suzy Pickering.

Subscribe now

Subscribe now

1. Why Facts Don’t Matter When People Shout

Séb Says: The philosopher Dan Williams writes that many of the raging disputes today between political foes are not actually contests over what’s true. Instead, they’re often rooted in what the great 20th-century journalist Walter Lippmann dubbed “pseudo-environments”: people’s self-serving distillations of the world. When I was younger, I thought the reason we get polarization is because we don’t have access to the same facts. This suggests that, with one more bit of debate and better trimming of the media landscape, we’d fix everything. But this piece makes the case that, even with shared facts, we each have an interpretative lens. And reconciling these is much harder. There may also be some benefits to the splintering of realities, because then you get competition in the marketplace of truth. If some group believes that eating laundry-detergent pods is going to cure Covid, you’ll get quick feedback. This doesn’t mean splintering alone is valuable—you still need the right norms and institutions that value accuracy and the free pursuit of knowledge.

How Tribes Construct Rival Realities (Conspicuous Cognition)

2. Explaining Wealth and Poverty

Séb Says: Two talented thinkers—David Oks and Deena Mousa—separately ponder why some developing countries rise while others struggle. David points to human capital: while China tore apart old social systems and built a modern workforce, India failed to update as radically. Meantime, Deena interrogates the persistent difficulties of Malawi, which she ascribes to populist and expensive policies, like fertilizer subsidies, that have blocked economic reform. I appreciate articles that look beyond AI to unpack the institutional, geographic, economic, and societal factors that shape how technology can change society (or not). A lot of people in the Bay Area overrate intelligence as the bottleneck to improvements in society. The world would be simpler if this were true! AI is going to be transformative, but this truism doesn’t tell you how, where, why. Articles like these provide the texture that simplistic models of the world skip.

Why China Got Rich and India Didn’t (David Oks) &
We Don’t Know Why Malawi Is Poor (Under Development)

3. What AI Does to Culture

Séb Says: Here are another two great articles that I’d suggest reading in succession, this time on how AI could influence culture. There’s a Q&A by Anika Meier with the British artist Mat Dryhurst; and a New Yorker column by Kyle Chayka on how Claude’s design “taste” is turning into a cliché. Excellent stuff is happening in the interaction of AI with arts and culture, but it’s often missed. Cultural elites often cite banal talking points bemoaning technological influences as uniformly negative, while tech companies with the aesthetic sensibilities of a jellyfish proudly sponsor the safest, most out-of-date, generic, artists and thinkers. So it’s refreshing seeing someone like Mat Dryhurst, who deeply understands both the tech and art, explore unconventional ideas at length. I’ve also been banging on about things like model multiplicity, customization, and decentralization for a while—all of which would help address the bland commodified pre-packaged designs that AI model providers may otherwise converge on, and which the New Yorker piece eviscerates. Interestingly, some AI labs now appear to be pushing back against design clichés.

Who Shapes Culture? (Sleek) &
The AI Design Aesthetic That’s Taking Over the Internet (The New Yorker)

4. Lessons for Public Health

Séb Says: A further two-part recommendation. First, another deeply researched piece from Saloni Dattani, who looks into how the world produced Covid vaccines in haste, which she attributes to more adaptive regulations, massive public financing, and accelerated trials (among other factors). Second, Brian Potter wrote an unexpectedly gripping history of how a flesh-eating parasite seemed to be under control—but is now back. (Don’t entirely panic: It mainly affects livestock.) While Saloni’s piece shows what we can learn from Operation Warp Speed, the U.S. program to hurry up vaccine development, Potter’s piece highlights the fragility of progress. A lot of people in AI have been exploring agendas like d/acc and societal resilience—that is, advocating for safe technology not by slowing down AI, but by using it to speed up safeguards. This is great, and I hope we can remain laser-focused on (a) the importance of institutional design; and (b) the many environmental, political, and logistical failures that can slow things down. Doing this well means nurturing scientific and economic expertise outside of generalist AI talent too.

Why Were Covid Vaccine Trials So Fast? (The Clinical Trials Abundance Blog) &
The Fall and Rise of Screwworm (Construction Physics)

5. What Yesterday Teaches About Tomorrow’s Tech Transformation

Séb Says: Talk of technology’s last societal overhaul, the Industrial Revolution, is in the air. This piece from Kara Dimitruk & Ben Southwood looks even further back, considering how the overhaul of 17th-century English property rights set up the explosive industrial development that followed. The changed property rights increased agricultural output, allowing more laborers to take urban industrial jobs; opened up land for development into mines and factories and housing; and cut transport costs. Might there be lessons for the AI revolution? Today, the question of how to “reorganize” rights is underexplored. For example, property rights can be too rigid, and heavily forestall growth and prosperity. Yet changing them can seem nightmarish and intractable. But this piece shows how that has been achieved before, what it could unlock, and how political change can sometimes involve win-win solutions that genuinely make all parties better off, rather than being mere slogans. It makes me wonder: what is today’s equivalent of “enclosure with compensation” that consolidated fragmented land holdings or “turnpike trusts” that helped pay for transport infrastructure?

How Smashing the NIMBYs Created Modern Capitalism (Works in Progress)

Subscribe now

Please share your own take and any new papers that you’ve enjoyed.

—Conor Griffin, AI Policy Perspectives

Subscribe now

Chatbots can make your friends feel like hard work

• What’s the paper about? A team of researchers from Oxford, Stanford, and the UK AI Security Institute found that repeated use of sycophantic AI may reduce users’ desire to interact with their friends and family.

• Why does it matter? The study suggests that sycophantic AI may offer the experience of being seen and understood, without the friction of human relationships. It also suggests that people may actively choose more sycophantic versions of AI.

• The details: AI models tend to unconditionally validate a user’s ideas. Humans often use such insincere flattery to manipulate somebody into giving them what they want. In AI models, it mainly results from training methods that optimise the likelihood of user approval. Such sycophancy could pose various risks, such as compounding a user’s delusions.

• Most evaluations of AI sycophancy look at a single conversation between a user and a chatbot. This study ran five experiments, including one lasting three weeks, with more than 3,000 participants. During this time, participants engaged with three versions of GPT4o—one prompted to be sycophantic, one to challenge a user, and one neutral—about personal dilemmas, such as whether to change careers or end a relationship.

• Initially, participants were more likely to report wanting ‘emotional support’ and ‘validation’ from close friends and family, rather than from AI. But after just one conversation with a sycophantic AI, participants were more likely to report feeling that they had already talked things through sufficiently, and that it would now take more effort to feel understood by close friends and family, compared to those participants who had engaged a neutral AI. The effect was more pronounced for friends and family than romantic partners, suggesting that sycophantic AI may compete more with the former than the latter.

• Participants who engaged with the sycophantic AI also reported lower satisfaction with their real-world interactions—although they didn’t report spending less time with people. Some worry that this may change, as AI systems start to become more personalized to users.

• As the authors of this paper put it, the risk is that “sycophantic AI delivers what people have always sought from close others—the experience of being seen and understood—but without the work that produces it.” This may feel good in the moment, but not deliver what people ultimately need—such as intellectual humility, self-awareness, and connection.

• How to avoid this? Better evaluations would help. In a separate paper, some of the authors propose testing how often models flatter a user’s self-image (rather than just how often they agree with a user’s broader beliefs). There are also different mitigation ideas—researchers at the UK AI Security Institute recently proposed automatically adapting user statements into questions, as models tend to respond less sycophantically to questions.

• AI’s character is shapeable, so users could demand AI systems that challenge their thinking. But the authors of this study found that after engaging with all three systems, a majority preferred the sycophantic option—not because they thought it gave better advice, but because it made them feel most understood.

• This suggests that relying on users to choose less sycophantic models may come up short. And so companies need to find a way to make models’ default settings less sycophantic, even as they also try to equip models with a richer ‘character’ and the personalization opportunities that some users want. Otherwise, as the authors note, the risk is that AI may gradually reshape “the very relationships that would otherwise constrain its influence.”

AI safety evaluations may be measuring the wrong thing

• What’s the paper about? A large cross-institutional team of researchers published a position paper arguing that many AI safety evaluations are increasingly inadequate, as they overlook how a model’s behaviour can change, post-deployment.

• Why does it matter? Current approaches to AI safety rely heavily on benchmarks and evaluations. But many essentially treat AI models as ‘frozen’ artefacts that are expected to perform similarly across users, over time. The authors argue that is mistaken and that new ‘trajectory-based’ evaluations could help to predict how AI systems will actually behave in the real world.

• The details: The authors start by documenting three ways in which AI systems may continue to learn, post-deployment—the first two of which are already commonplace:

  • In-context learning: When a user shares information via the context window, such as telling a chatbot that they prefer concise answers, the model will adjust its answers.

  • Storage-based learning: Across sessions, leading AI systems now use memory to tailor their responses to a user—such as remembering their location, job or hobbies.

  • Parameter-based learning: Researchers want to shift AI systems from static models trained on fixed data to agentic systems that could adapt their own weights or architecture—although such systems are not yet deployed. Some go further, envisioning agents that can proactively identify their own knowledge gaps and seek out novel learning experiences to address them.

• By changing a model’s propensities (tendencies) and capabilities, even more modest continual learning can lead to new failure modes. For example, by gradually steering a conversation across many turns, threat actors could get an AI system to provide advice on how to make a weapon—which the model would have refused to do, if asked in a single prompt. An AI system equipped with better memory may become more sycophantic if it compresses past user responses as reliable information. When a user repeatedly exposes a model to tasks in a given domain, the model can ‘forget’ how to perform other tasks well.

• Current evaluations capture some of these risks. Expert red teamers probe AI systems over multiple turns to see how they perform in different risk scenarios. AI companies analyse user logs to check for novel safety violations. Researchers have also proposed automated evaluations to continually check how AI systems perform, after they are deployed. But most evaluations are carried out pre-deployment, with largely static AI systems in mind.

• Drawing an analogy to the monitoring that pharmaceutical companies do after they launch a new drug, the authors propose two new ideas for AI companies:

  • Trajectory elicitation: Pre-deployment, use sandboxes to put an AI system through a range of simulated interactions to elicit and map different types of behaviour. For a medical AI application, this may mean simulating a large number of patient interactions, with different health conditions, emotional states and attempts at misuse—with periodic checks to see whether the model’s behaviour is drifting.

  • Predictive monitors: Train a predictive tool on this ‘trajectory elicitation’ data and use it to monitor different copies of the AI system post-deployment, to predict how a user’s trajectory may evolve and to flag signs of impending harm.

• As the authors acknowledge, these approaches will be hard. Beyond their computational cost, an AI system may have multiple ‘stable states’ that it could drift into depending on what it experiences. Safety testers may identify some, but overlook others. Similar to the challenges that scientists face when forecasting other complex systems, like the weather, small differences in the initial user trajectory state could lead to dramatic differences in the resulting simulations. Ultimately, for these approaches to work over the longer-term, AI systems may need to become more predictable. Or the current approach to AI safety , and its reliance on evaluations, may need to be rethought.

Share

How AI covers the news

• What’s the paper about? ForumAI—a company that works with human experts to train AI experts—published an evaluation of how AI models respond to queries about the news.

• Why does it matter? More people are using AI to learn about current events. Some worry that AI outputs will exhibit biases, hallucinations and sycophancy. Others hope that AI models might diminish the spread of unfounded claims online, even nudging public opinion back towards a greater respect for expertise. The study suggests that a key challenge will be disagreement about whether AI’s responses are accurate.

• The details: News habits have shifted over the past two decades. More people now get their news from social media and video platforms, while a growing minority consume no news at all. AI promises a further shift. It is not yet a primary source of news, but its share is growing, particularly among young people. AI news comes in different forms—from a chatbot query to an AI search response. This raises the question: what will it mean to “consume news” in the AI era?

• The authors worked with a set of experts, from journalists to former Congressional leaders, to build a dataset of 3,135 queries across politics, the economy, healthcare, education and more. Roughly, one-third related to current affairs, while two-thirds were designed to remain evergreen over time. Some queries were real posts taken from social media, while others were edge cases created by the experts, or AI, to fill in missing topics and viewpoints. However, the authors did not clearly define what they mean by a ‘news query’, nor did they illustrate this by sharing their prompts, outside of a few examples.

• The researchers worked with the experts to create editorial standards to evaluate AI responses on three criteria: the quality of sources cited, factuality, and neutrality. They then trained AI judges to evaluate AI models’ answers. On sources, they assessed the type of source cited—from informal to scholarly—and whether it was state-controlled. On factuality, they extracted verifiable claims from the models’ outputs and compared them to available evidence. On neutrality, they used heuristics to determine if a model’s answer was neutral—such as whether it presents multiple viewpoints for normative claims—and, if not, which political direction it leant (in the US left/right sense).

• The researchers used the AI judges to evaluate four models. Claude Opus 4.7 had the best quality sources, GPT 5.5 was the most factual, while Gemini 3.1 Pro was the most neutral. Grok 4.3 performed the weakest across all three dimensions. When AI model responses did lean politically, they consistently leaned left, apart from Grok, which consistently leaned right—echoing past research.

• Surprisingly, factuality and source quality were largely independent—Claude scored highest on source quality, yet poorly on factuality. This likely reflects two factors. First, the AI judges evaluated the reliability of sources, not whether they supported the claim being made. Second, the human experts often disagreed about whether a response was factual, mainly due to the challenge of determining the right threshold to apply—for example, when assessing the factuality of the statement “most states agree that the UN Security Council’s structure has a serious legitimacy deficit”, what constitutes ‘most states’?

• This meant that for factuality, the AI judges had a less clear human baseline to calibrate against. The AI judges also had a tendency to over-reject claims as false, compared to the human experts. Although in several cases they correctly challenged the experts—suggesting that AI could provide a useful check on misleading claims.

• Overall, the findings suggest that improving AI’s ability to understand and communicate ambiguity, and to specify how sources support or challenge claims, will help. But the low agreement between human experts suggests that debates about AI’s ability to report the news will remain, particularly as news reporting requires editorial judgement that extends beyond fact-checking—like how to best describe a specific claim or event.

“God has helped us and so will AI”

• What’s the paper about?: Antonia Juelich at the University of Cambridge published an analysis of how the terrorist group Boko Haram uses AI, based on interviews with former members.

• Why does it matter?: Most research on AI and terrorism focuses on things that are more readily observable—such as evaluating AI model capabilities, monitoring discussions in online forums or studying the use of AI in propaganda. This study is the first on-the-ground investigation into how a terrorist organisation uses AI in its operations.

• The details: Boko Haram emerged in northeastern Nigeria in the early 2000s. It subsequently pledged allegiance to the Islamic State and is leading an insurgency that has killed more than 40,000 people and displaced more than 3 million.

• The group is known to adopt technologies, such as satellite internet and drones. To understand how they use AI, Juelich carried out 57 in-person interviews with 27 former members. To reduce the risks of interviewees lying or exaggerating, the author recruited them through independent channels, triangulated their insights, and made them physically identify some of the AI tools used.

• Interviewees said that Islamic State operatives provided Boko Haram with training on AI, and that Boko Haram now has specialised units to cascade this AI knowledge down their hierarchy. The group also takes steps to avoid exposure, such as setting up user accounts in the names of followers in other countries, rotating these accounts, and controlling how members can use them.

• Some of the AI use cases that interviewees described, such as posing questions about religion or repairing vehicles, do not directly relate to weapons or attacks. But others do—such as guidance on attack strategies, how to use guns they loot from the army, or how to design improvised explosive devices. Some use cases are non-obvious—such as advice on how to use motorbikes to jump defensive trenches dug by the military.

• The use cases focus on conventional weapons. When asked, one interviewee said that they would have no qualms about using chemical or biological weapons, but that the technical obstacles were very high. Interviewees did note some experimentation with chemical agents, but these claims could not be verified and the author suggests that they be treated with caution. (The Islamic State has used chemical weapons.)

• Some interviewees suggested that a religious prohibition on transmitting poisonous agents may prevent Boko Haram from using biological and chemical weapons. However, Juelich cautions against assuming that members of the group strictly adhere to ideology, noting that members routinely disagree and adapt their own views on what kind of violence is acceptable.

• Ultimately, the interviews suggest that Boko Haram members do find AI useful, but the author notes that she cannot “conclusively establish” whether AI provides ‘uplift’ over other sources of information.

• The interviewees said that group mainly uses leading US AI models, via the web interface, and (vaguely) described the use of jailbreaks to get around safety guardrails. As interviewees are former members of Boko Haram, most of the AI use they described took place between 2023-24. Since then, AI capabilities have increased significantly, but so have safeguards—it’s unclear what this means for how the group uses AI today.

AI agents struggle to do what employees want them to do

• What’s the paper about? A large cross-institutional team of researchers found that AI agents struggle to perform the tasks that US employees most want them to automate.

• Why does it matter? Many organisations want their employees to use AI more. This study suggests that adoption may be lagging because AI is not sufficiently performant or reliable on the tasks that employees most want to delegate.

• The details: How AI will affect employment is a pressing question that is difficult to answer. One approach is to identify occupations that are economically valuable, and/or particularly exposed to AI, and evaluate how AI performs at tasks in these roles. Evaluations, such as OpenAI’s GDPVal, OneMillionBench, and Remote Labor Index, take such an approach.

• These evaluations have shortcomings. AI agents are often given ‘clean’ discrete tasks, with lots of context and precise instructions, which is rare in many workplaces. The evaluations can also imply that there is a winner-takes-all battle underway between agents and humans, and that the AI’s performance on the evaluations help to signal who will do these jobs. Among other things, this overlooks how most jobs are ‘messier’ than what these tasks imply, and that successful AI use will often be contingent on employees choosing to delegate their work to the technology.

• In this study, the authors published a new evaluation, JobBench, to try to address some of these limitations. They drew on a past survey where ~1,500 US workers rated how much they want AI to automate each task in their job. They then selected 35 occupations and created synthetic versions of 130 tasks that employees most want to offload. For journalists this included checking facts against source materials. For web administrators it included analysing forensics data and reconstructing steps in a cyberattack.

• To complete each task, AI agents were given a set of reference files, some of which are real-world documents with contradictions to resolve—an attempt to capture some of the ‘messiness’ of real work.

• The threshold to succeed is high. On each task, the agent had to successfully pass through more than 35 success criteria, on average, with no partial credit for reaching correct conclusions through incorrect reasoning.

• The authors used scaffolds like Claude Code and OpenCode (an open-source equivalent) to evaluate 36 model configurations. Opus 4.7, running in Claude Code, performed best, at ~46%.

• The study’s methodology differs from OpenAI’s GDPVal, which compares AI outputs for a task against an output from a human expert. This makes it hard to directly compare the two. Agents clearly struggle much more on JobBench, but it’s unclear to what extent this is because AI performs less well on the tasks that JobBench focuses on—i.e. those that employees most want to delegate. Or whether it is because the tasks in JobBench are ‘messier’ and the threshold to succeed is higher.

• The researchers also find that scaffolds matter a lot. Claude Sonnet 4.6 scored 37% inside Claude Code but only ~31% inside OpenClaw—similar to the difference between model families. This suggests that equipping AI agents with the right context and tools to perform a given task, and employees’ ability to do this, is a key determinant of how useful AI is in the workplace.

Subscribe now

Over the past year, agents have transformed what it means to code. Scientific research may be next — from proposing novel hypotheses, to designing experiments, to discovering algorithms that improve on the best that humans have designed. This raises urgent questions for policymakers and science funders, the biggest being how to validate the coming wave of AI-generated ideas.

New approaches are also needed to ensure that scientists can access agents, that datasets are agent-ready, and that peer review is kept afloat. We sat down with 10 researchers and engineers from Google DeepMind to try to figure out what comes next.

Microbiologist José Penadés and his team at Imperial College London took most of a decade to work out how a family of superbugs spreads antibiotic resistance. The result was unpublished, known only inside his lab. Then, in 2024, he described the problem to Co-Scientist, an AI agent from Google DeepMind.

Within two days, Co-Scientist returned five potential explanations, ranked in priority. Number 1 was the same hypothesis Penadés’s team had spent so long working to prove: that some superbugs acquire tails from viruses and use them as “keys” to jump between host species. Stunned, his first thought was that his computer had been compromised, so he emailed Google to check. Confirmed: no peeks taken. If he could have gone back in time, that insight would have saved his team years.

An AI agent such as Co-Scientist is a large language model (LLM)- based system given a goal and the tools to pursue it. Unlike a query-answering chatbot, an agent can plan how to achieve the goal you give it, breaking it down into steps, running multiple subagents and processes in parallel, and detecting and correcting errors as it goes. It can also engage with the wider world, for example by calling online databases and tools, or writing code to operate robots.

The arrival of AI agents is timely. Researchers face a rapidly growing “burden of knowledge”, with more to learn before they can meaningfully contribute. The questions that count in drug discovery, climate modelling, materials design, and biology are becoming too complex and interdisciplinary for human teams to tackle at pace.

Science depends in part on a social infrastructure that has gradually evolved: labs, teams, institutions, peer review, grant funding, and networks that support the accumulation of shared knowledge. The era of AI agents will challenge that infrastructure, and at times demand rapid change — some of which is perhaps overdue.

Software engineering has been going through a version of this. In barely a year, coding agents reshaped the working practices of engineers. AI agents are now starting to reshape science. Science is messier, but it also suits agents in certain ways: a large literature available as text, huge databases, and some workflows already built around code. Science also has an existing generation of specialist AI models that agents can put to work, from protein and materials design tools to state-of-the-art weather forecasting.

Why are AI agents suddenly so useful?

Anybody who has worked with early iterations of AI agents may be sceptical of their utility in science, where rigour and reliability are essential. But three forces are making today’s agents more useful: stronger frontier models, “scaffolding” that enables agents to elicit greater capabilities from those frontier models, and customisability that allows scientists to tailor agents to their needs.

Stronger frontier models. Leading models now outpace human experts on demanding scientific knowledge benchmarks, such as Humanity’s Last Exam and FrontierMath. Perhaps more significant is their new depth of thinking, driven by inference-time reasoning, a technique in which models work through problems in extended steps, exploring and revising before committing to an answer.

In mathematics and computer science, where breakthroughs can rest on reasoning alone, models are producing impressive results. But sharper reasoning helps beyond mathematics, enabling agents to draw on the “long tail” of findings that are often buried in papers, make scientific connections across fields, judge which tools to call and when, and catch their own errors as they go.

Scaffolding. This is a type of “harness” that enables AI agents to better elicit the latent capabilities that reside in base models but which do not emerge by default. This code “wraps” a frontier model, giving it structure, memory, and the ability to access and use tools including code execution, scientific software, and literature search. It also lets agents call on specialised AI models and interact with other agents.

Early agent scaffolds were bespoke: they gave a particular agent detailed instructions for how to plan, carry out tasks, and use tools. But often they did not transfer well to other agent systems, or even to later generations of the frontier models they depended on. New standards, such as protocols for how agents can communicate with each other, should reduce the need for so much custom scaffolding.

Customisation. Agents can be tailored to a discipline, a lab’s workflow, or an individual researcher. A key mechanism for that is the current proliferation of agent “skills” that users can create to their own detailed specs.

Base models hold a great deal of explicit scientific knowledge, the kind written down in papers and textbooks. What they lack is tacit knowledge: the hard-to-articulate craft, built up over years of practice and failure, that lets a scientist coax a cell line into growing or get a simulation to run well. Until now, scientists using AI tools had to convey their methodological know-how, processes, and preferences the hard way: through detailed prompts, bespoke scaffolding, or model retraining.

Agent skills make some of this know-how portable. They are instruction sets, often just simple text files, that tell an agent how to perform a task and what to produce. Skills are relatively easy to produce, share and accumulate. As a scientist uses their agent more, the agent can also draw on these interactions, making the experience more personalised. Scientific labs and institutions can also make their proprietary data — such as old lab notebooks — securely available to their agents.

Natasha Latysheva, a computational biologist, says she has distilled aspects of her research process into skills. “While AI agents aren’t fully reliable yet, I think the trend is clear. Scientific research will shift from hands-on execution to high-level orchestration,” she says. “We’ll start our workday by reviewing the experiments and analyses our agents ran overnight, tweaking their direction and guiding their attention.”

How AI agents are changing (and not changing) science

The most immediate change for scientists is also the most mundane: the availability of smart, tireless digital assistants. Researchers are handing off more of their daily grind — sifting the literature, querying databases, orchestrating analyses, curating data, writing grant proposals — to agents that do it in minutes rather than hours or days, then report back.

Agents also open up new possibilities. “Researchers can suddenly do things they couldn’t do at all before,” says Matej Balog, a Senior Staff Research Scientist. For example, plenty of scientists lack the skills to build the software tools and pipelines they need. This is partly because developing software for science is hard. But also because most scientists are not deeply trained in programming, and science has not traditionally provided competitive career paths for dedicated research software engineers in academic labs.

Agents are particularly strong at writing code because it is a domain with an enormous amount of training data, where correctness can typically be checked automatically. A scientist can now describe what they need in natural language — “write me a script to clean and merge these three datasets”, “build me an interactive browser-based tool to explore this output” — and get serviceable code in minutes.

But ultimately, the biggest change for scientists is a structural one. Agents are making it easier to come up with ideas and proposed solutions for problems they are working on, but are not yet providing the same uplift when it comes to validating them.

Ideation

Most scientists have no shortage of ideas. The challenge is knowing which ones to pursue. This is where agents are starting to help. An agent can digest a field’s accessible literature, making connections across disciplines that no single researcher would have time to trace. Existing Deep Research tools already use subagents to do this. But agents optimised for ideation go even further. Co-Scientist, for example, directs a raft of subagents to generate diverse hypotheses, critique them, rank them, and iterate — mimicking aspects of how a human research group operates, but at remarkable speed.

Gary Peltz at Stanford University used the tool in his hunt for existing drugs that could be repurposed to treat liver fibrosis, the scarring process behind 1.4 million cirrhosis deaths a year. Based on his own literature review and decades of expertise, he picked two candidate drugs. Co-Scientist picked three. Neither of Peltz’s picks showed any benefit in assays with live human liver cells. Two of Co-Scientist’s picks not only blocked fibrosis but also promoted liver cell regeneration.

This is a positive example, but any LLM-based system is fallible — even the strongest reasoners can still make things up. “A single hallucinated claim on page 10 of an output can invalidate the whole thing,” says Vivek Natarajan, a Co-Scientist lead. Missing such errors wastes time and resources, and catching them can require a scientist with deep expertise. This fallibility makes scientists understandably cautious about AI agents, and reducing it is a top priority for those building these systems. To function effectively in a research setting, agents cannot act as black boxes that simply output answers; they must expose their reasoning and serve as transparent collaborators.

One key challenge is imbuing agents with what Natarajan calls “epistemic humility”: models that know when they don’t know, and say so. He points to AlphaFold, the protein-structure predictor, which is prized partly because it reports how confident it is in each aspect of each prediction, so researchers know when to trust it and when to reach for other methods. Calibrating that confidence across more open-ended scientific reasoning remains an unsolved problem.

Looking further ahead, the harder problem may be a tension between two things scientists want at once: hypotheses that are grounded in the literature and free of error, but also genuinely novel. Agents tuned for caution can drift toward the safe and the known; tuned for originality, they are more likely to go astray. This challenge will itself require new ideas, such as better ways to evaluate novelty. Or better ways to connect agents to more specialized models — trained on first-principles scientific data — to help generate ideas that are both original and robust.

Finding the optimal candidate solution

Agents can search for the best solution to a problem, not just a workable one. A materials scientist hunting for a new catalyst faces a near-infinite number of molecular structures, each costly to make and test. A computer scientist looking for a more efficient algorithm faces a similar explosion of possibilities.

Vast solution spaces like these turn up all across science and industry. AlphaEvolve is an agent that can find the best candidates within them. Given a problem expressed as code and a way to score potential solutions, it orchestrates an ensemble of agents to generate many algorithmic candidates, keeps those that score highest, and breeds the next generation from the survivors. It runs unattended, generating and improving candidates at a scale no human team could match.

AlphaEvolve works in code, but its reach extends well beyond software. As Balog points out: “Algorithms can accurately describe so many of the world’s scientific and natural processes.” AlphaEvolve has assisted in the design of Google’s next-generation TPU chips, helped the mathematician Terence Tao solve open Erdős problems, and improved the analysis of genomics data.

But in scientific domains such as materials, AlphaEvolve’s top scorers are leads, not final results. A promising catalyst still has to be made and measured at the bench.

Validation

Validation is the slow, costly business of testing whether an idea survives contact with reality. Karl Popper said science advances through conjectures and refutations. Agentic AI is changing the economics of that pairing. AI agents are conjecture machines, making ideas and candidate solutions abundant and relatively cheap. Refutations remain physical and institutional — and so, costly and slow.

Mathematics and computer science are often viewed as great exceptions because validation can run in silico. An AI agent can generate a proof, represent it in a formal language like Lean, and have the computer verify, unambiguously, that it is correct.

Even for the parts of maths that can’t yet be described in formal language, validation is advancing. Aletheia pairs a proof generator with a natural language verifier that checks its work and sends flaws back for revision. In February this year, mathematicians ran the inaugural First Proof challenge: 10 research problems, kept unpublished so they couldn’t be found in any training data. In the week allotted, Aletheia solved six — the best result.

Mathematicians will need to absorb all these new outputs. Some already complain that AI-generated proofs are too long and hard to parse (although certain AI-generated proofs are short and elegant). “We are moving toward a future of serious ‘proof indigestion’ where AI generates breakthroughs faster than humans can review them,” says Thang Luong, who led the Aletheia effort. To break this bottleneck, automated verification must become standard, but this verification will need to combine the absolute correctness of formal languages like Lean with the more expressive reasoning of natural language.”

Alex Davies, who also leads work on AI for mathematics, is mindful that by automating large chunks of mathematicians’ workflows, his discipline is dealing with questions that others may face in the coming years: “I can imagine a world in which machines do the discovery, and what’s left for mathematicians is to understand it, and to decide what questions are worth pursuing next.” Luong echoes this, noting that maths may also provide some of the general technology needed to address the validation bottleneck in other fields: “One can think of mathematics as an accelerated testbed for the rest of science”.

At the moment, however, the validation gap in most disciplines is widening, not closing. An agent can propose a novel genetic lead to reverse cellular ageing, but cannot say definitively whether it actually works. This explains why companies like Google DeepMind, Ginkgo Bioworks and Lila Sciences are investing in automated labs. But they only suit some fields, are expensive to build and are still early in development. And even automation cannot rush nature’s clock. Cell lines need time to grow, chemical reactions take time to complete. For much of science, then, the lab sets the pace.

Implications for policymakers and research funders

To some extent, agents simply add intensity to questions that policymakers are already focused on in their AI for Science strategies, such as how to train the next generation of scientists, how to experiment with new forms of scientific institutions, and how to ensure that AI is not misused by threat actors, while still putting the technology to use addressing the various natural risks that society faces, like the next pandemic.

For every encouraging scenario, there is a challenging one. For example, AI agents could make it more feasible for small, agile teams to pursue creative, ambitious ideas, reversing the trend towards “big team science”, or enable scientists to work across domains, bringing new perspectives to existing problems. Assuming efficiency gains make agents sufficiently cost-effective, these trends could particularly benefit smaller, less well-resourced countries and institutions.

But agents will also give rise to anxiety among junior scientists that their institutions are choosing to spend budgets on tokens instead of staff. And left unmanaged, there is a risk that agentic tools could de-skill new generations of scientists before they develop the judgement needed to use them effectively. For the same reason that mathematics students still prove theorems unaided, science-graduate training may need structured periods of agent-free work and access to agents that act as genuine cognitive partners rather than oracles.

Beyond these questions, AI agents present at least four urgent new priorities: 1. Scientists need access to the tools. 2. The tools need access to agent-ready data. 3. We need more experimental infrastructure to validate AI ideas. 4. And we need to update the peer review process.

1. Ensure widespread access to agents

Agents will be extremely useful and fallible in non-obvious ways. Both factors provide a strong rationale for policymakers to ensure that all scientists can access the best agents — to speed up discovery and to provide the independent evaluations of AI agents that the scientific community needs to judge how best to use them.

This is an urgent strategic priority for policymakers and science funders, akin to the historical challenge of providing access to supercomputers. Geopolitical debates today often focus on one aspect of sovereign capability — whether a state can train its own frontier model. Much less attention is paid to what may prove to be a more important issue: a country’s ability to deploy agents across its scientific ecosystems for transformative impact.

At the micro level, funders must first decide how labs and researchers select and pay for agents. Selection is the easier near-term problem: let researchers find the most useful tools for themselves, without excessive approvals or complex procurement.

Paying is harder. The temptation is to use existing structures, with scientists seeking funding through grant applications or drawing on lab budgets. But the compute required to run agents can be large and the frontier of what agents can do is constantly expanding. While the cost per unit of AI capability is falling fast, total lab expenditures on agents are still likely to rise as agents take on longer and more complex tasks. Policymakers must quickly assess whether budget uplifts or entirely new funding programmes are needed. Delivering access at the scale and price needed will require novel public-private partnerships; the US Genesis Mission is one promising model.

Bigger questions await. Agents may make the existing process of allocating national budgets across disciplines more legible, forcing science funders and research programmes to quantify the investment in data and compute needed to make progress on specific problems. This in turn may lead to more targeted debates about the relative value of solving different problems. If agents propose the top hypotheses to explore across an entire field, with very expensive experimental validation plans, how should this fit into national funding strategies?

1. Make national data assets agent-ready

While scientists need access to agents, agents need access to data. Data that is open or low-risk should be exposed to agents through well-documented APIs, with sufficient quality control and metadata. The engineering support and maintenance to do so is not trivial, so funders should ensure such data stewardship is properly resourced and support interoperable data standards. But ultimately, the ability of agents to help extract and annotate data — from PDFs to download portals — provides an opportunity for governments to extract a lot more value from the data they have already funded.

More sensitive datasets in genomics, virology, or other areas carrying dual-use risk often come with restrictions on who may use them and how. The challenge now is to develop similar privacy-preserving solutions, when appropriate, for agents, with auditability and privacy built in. Examples like OpenSAFELY, which lets human researchers securely access valuable health data, can provide inspiration. The prize is large. A dataset analysis that currently takes years of doctoral work could, with the right secure agent infrastructure, run autonomously in days.

Perhaps most importantly, agents provide a strong rationale for funding the creation of entirely new open datasets. This leads to a further question for funders: could agents help identify the most important datasets to fund? Some of the authors of this article recently made a human-expert-driven attempt to answer that question for fusion energy. How soon will agents be capable of running similar “AI data stocktake” exercises?

1. Tackle the validation bottleneck

Many scientists already struggle to get enough time in facilities to run their experiments. As AI agents make hypotheses and candidate solutions increasingly abundant, this bottleneck will only tighten. Policymakers and funders should address this in at least two ways: investing in existing experimental validation infrastructure and accelerating progress on automated labs.

Public research bodies hold extensive experimental facilities across almost every scientific field. AI agents provide a reinvigorated case for investing in them and opening them up, by renting bench space or experimental run-time to researchers testing computational hypotheses and predictions against reality. Direct partnerships with AI labs are another avenue. Google DeepMind has created a wet lab inside the UK’s Francis Crick Institute, a leader in biomedical research, and is also providing independent scientists funding — alongside Co-Scientist access — to carry out the wet lab experiments needed to validate agent-enabled hypotheses. The US government’s Genesis Mission will connect the world-class experimental facilities of the Department of Energy’s (DOE) National Laboratories with academia and the AI industry.

Automated labs are another promising route to tackling the validation bottleneck, but they currently rely on expensive robotics and compute. Ensuring broad access will require public investment, and there are good early efforts here. The US National Science Foundation has put $100 million towards a national network of distributed facilities, while the UK launched a call for ideas and already hosts the £81 million Materials Innovation Factory. The build-out of automated labs will almost certainly go beyond what any single lab or institution can afford, so governments should also explore building centralised capacity and adopting the kind of “user facility” access model seen at the US DOE’s national labs.

1. Empower peer reviewers with agents

The peer review process has long been under strain, with slow timelines and reviews of varied quality. Now, scientists are using AI to write ever more grant applications and papers. This is making it harder for funders to know which research to fund, and for peer reviewers to validate findings and identify the most important work.

As noted by Professors James Wilsdon and Geraint Rees, the challenge is not just an increase in supply; writing quality is also no longer a reliable discriminator. Agents deepen the problem. The more an agent is left to plan and optimise an application, drawing on the funder’s criteria and its recent winners, the less the bid reflects a scientist’s original thinking.

Funders, journals, and conferences are trialling various responses. For grant applications, some suggest leaning less on the written word and more on the investigator’s track record and team. The UK’s Medical Research Council recently reinstated interviews for shortlisted applicants. These ideas have promise, but also risk privileging seasoned experts or increasing costs.

The likely answer will be a layered approach. Those developing and using AI should document its use more clearly. That could include advancing watermarking techniques and ideas such as “Human-AI Interaction Cards” — short records detailing the prompts and outputs that produced the key scientific insights.

Reviewers should also be able to use agents. This will not be easy, as AI use is often banned, even if widely practised in secret. To move forward, organisations can build on the more nuanced guidelines that some have started to develop and test agents in more objective areas where they are likely to be strong — such as detecting errors.

They could also take steps to ensure that scientists view agents as “human-centric” tools that enhance, rather than bypass, their judgement — for example, by being transparent about the systems and ensuring that agents expose their reasoning, verify their claims with citations, and (to the extent possible) document their uncertainty. By understanding how an AI agent reaches its conclusion, scientists will have opportunities to learn, intervene, and collaborate.

These are formative years

The scientists using AI agents are not going back. The technology will continue to improve. The institutions that govern science — and the physical and social infrastructure it runs on — were built for a research enterprise that is now evolving faster than they are. We need to upgrade them for the agent era.

This will take a significant collective effort and serious investment. AI agents must not be viewed as a rationale to spend less on science; this would be a tragic false economy. The countries that treat this as a moment of genuine structural change will unlock the greatest public value and shape what comes next. The choices made in this window may be difficult to reverse.

Acknowledgements

We would like to thank the following experts at Google DeepMind who shared insights with us through interviews and feedback on the draft. All mistakes belong to the authors.

Vivek Natarajan, Sebastian Nowozin, Tim Green, Natasha Latysheva, Simon Batzner, Matej Balog, Samuel Albanie, Alex Davies, Nenad Tomasev, Juan Mateos-Garcia, Catherine Pollard, Uchechi Okereke, Agata Laydon, Anna Koivuniemi, Andy Song and Tom Rachman.

Subscribe now

This question exasperates Carl Hendrick, a writer and professor of education who began his career teaching English at a state school in London. Today, he also advises Alpha School, the fashionable but controversial US-based private educator, hoping to teach kids in 2-hours a day, followed by social activities, with AI and no teachers.

To find out how (and if) this will work, we asked Carl what cognitive science tells us about learning, how Alpha operates, and why the decline of reading matters.

- Conor Griffin, AI Policy Perspectives

Subscribe now

Conor: Many students hate memorization. You think it’s central to learning? Why?

Carl: Cognitive science tells us that a change in your long-term memory is the fundamental invariant of learning. It’s non-negotiable. Without it, you’ve learned nothing.

In the 1950s and ’60s, we had a cognitive science revolution that helped us understand our memory as a system. In working memory, we have a hard constraint, whereby we can hold maybe five to seven elements at a given time. Our superpower is long-term memory which, as far as we know, offers an unlimited reserve to store things. If you store information in your long-term memory, then your working memory becomes outsized because you can effortlessly draw on the symbols and ideas you have stored. We’re doing it right now as we speak.

So cognitive science tells us that memorization is important to learning. But it also tells us that rote memorization of isolated facts is a debased form of learning, because memory is schematic—the concepts we learn need other concepts to stick to. So, if we want students to learn about World War Two, we don’t want them to just memorize dates. We want them to be able to build their own schema, or mental models, about the war. The question for educators becomes: how do you decompose World War Two into atoms of knowledge that are teachable to a student, but which they can then rebuild?

Conor: Why are certain forms of teaching effective at this, like ‘retrieval practice’, where students are asked to recall facts or concepts from their memory?

In a word: friction.

The desirable difficulties framework, developed by Robert and Elizabeth Bjork, the psychologists and memory experts, says that learning is often counterintuitive. Things that feel like they are working to a student, such as re-reading and highlighting a text, often aren’t. And things that feel like they aren’t working, or are a struggle, like getting kids to dredge up something from their memory, often do lead to learning.

This comes back to how memory works. Human memory is not like a tape recorder. It is reconstructive. When we remember something, we seem to almost rewrite it. This is also why eyewitness testimony is so unreliable. But from a learning perspective, successfully retrieving a memory can strengthen it for the future.

With retrieval practice, the encoding of the knowledge seems to happen after the original event, when the students are trying to remember the thing. This is why the Bjorks see retrieval as a learning event, rather than a test of whether you have already learned something.

Conor: What’s the evidence that these cognitive-science interventions actually help students learn better?

There is a huge body of evidence from postgraduate students in labs, but little evidence for it in actual classrooms. And I say that as somebody who passionately believes in it.

A 2021 review found that we know almost nothing about how things like retrieval practice work with real kids and teachers. There is 70 years of evidence, but it is usually small studies with post-graduate students, focussed on maths and languages.

This is why we need to create new models of learning with AI. I want to see specific questions answered: “Given this third-grade reading problem in this state, what is the best intervention?“ It will be hard to isolate the effects in the way that we could with a vaccine. But with AI, we will be able to understand the effects of using a certain curriculum and sequencing the material in a certain way. We will be able to understand how to break down complex concepts into components to present to students progressively, without overloading their working memory.

Conor: Are there any interventions that do have solid evidence from real classrooms?

Yes, early reading and the Direct Instruction method.

For early reading, the debate is largely over. We know that instruction needs to be done with systematic, synthetic phonics. The brain is not built to read. So phonics teaches young children the individual sounds associated with specific letters, or letter combinations, before teaching them how to blend these sounds together to form words.

The most evidenced programme is Direct Instruction, a highly-structured method where teachers break complex concepts down into small, manageable steps. In the 1960s and ‘70s, it was subject to the biggest ever educational study in the US, Project Follow Through. This found that it was far more impactful for students than other approaches, for early reading, maths and other areas.

People tend to think of Direct Instruction as teachers just “telling kids stuff”. But instruction is actually the least important part of it. The most important part is the curriculum design. For example, there is a big focus on learning through contrasts. In the same way as we teach young children the colour red by showing them a red ball followed by a red triangle, you isolate the essential thing that you want them to know. This is a fundamental part of how we learn.

I think that there’s just no doubt that AI is going to be able to do this kind of curriculum design and sequencing far better than humans can. I think of the famous move 37 in AlphaGo. I think we’re going to see a similar thing for the design of curriculum and learning materials, where AI will just far surpass what humans can do.

Subscribe now

How Alpha School works

Conor: That brings us to the new group of private schools in the US that you work with, that has big ambitions to use AI. You’re one of several prominent learning scientists advising Alpha. Why did you get involved?

Carl: Traditional schooling has been hugely successful; nothing has lifted more kids out of illiteracy and poverty. But there are problems.

The system is largely propped up by the 1-in-20 “superhero” teachers. The experience of pupils is massively heterogeneous. Since Covid-19, something else has changed. Many kids are not in school at all, or they are there so little that it’s almost a waste of time. Until now, educational technology, or EdTech, has also largely been a story of expensive failure because it creates digital versions of things that didn’t work in the first place.

But at Alpha School, we are designing apps where you get high-resolution data points from students at a phenomenal clip. You get correct/incorrect answers, but also things like latency and student hesitation. You can slice that data and make strong predictions about when and how to do interventions like retrieval practice. Or ‘spacing’—where students distribute their learning and practice over many lessons or days, rather than cramming it into a single session, to more productively engage their memory. This will allow us to see those interactive effects between the curricula and how you sequence the teaching materials.

Ultimately, I’m a materialist when it comes to learning. I think learning is governed by the same biological laws as digestion. If you can get the brain to pay attention to a certain sequence of information, and if knowledge is retrieved under a certain set of conditions, then I think learning will be almost guaranteed for 99% of kids.

Conor: At Alpha, how are kids supervised during this online learning?

Carl: The big change will be moving to AI coaching. This is where people will get uncomfortable because the answer is to mimic what a really good teacher does: warm/strict supervision. If you think about MOOCs—the massive open online courses that were much-hyped around 15 years ago—they were a failure because students weren’t accountable. Only 5 to 10 percent of people who were already highly motivated finished them.

Schooling fulfills the function of accountability. In the future, this will mean cameras on, and an AI monitoring student behavior, their latency, what websites they are looking at, their ability to focus and then producing a report for a human tutor. If you can get kids to concentrate on the apps, on the sequencing, phenomenal learning is going to happen.

But people won’t like this. You’ll see articles about “Orwellian spyware.” As a parent, I’d prefer this model to simply unleashing kids on the internet, crossing my fingers, and hoping for the best. AI will supervise better than humans. There are very skilled teachers who can supervise a class well, with eyes on the back of their head. But even then, there are kids who drift through the lesson, ‘cosplaying’ attention, and learning nothing.

Today, when you look at kids in a computer room, they might spend 20 percent of their time on a task. The rest of the time, when the teacher’s back is turned, they’ll be looking at websites. The model for AI should be a high level of accountability for students.

Conor: Students spend a couple of hours in the morning on this intensive online learning. What do they do in the afternoon?

They work on ‘life skills’. Broadly, longer-term projects, like setting up a YouTube channel, playing music in a band, or sport. Let’s not forget, Alpha is an expensive private school. So you will have kids there, who have had tremendous support from their parents, and so they will already be highly-motivated and able to do many of these things.

And so there is a question about whether that model is going to work elsewhere. But Alpha is trying to understand how learning happens. And they want to remove the standard model where kids sit at their desk for six to eight hours straight.

Conor: Alpha School doesn’t have conventional teachers. Even with a good AI system, won’t this mean a lack of social influence? With a good teacher, you want to impress them. How will you manage that?

Carl: We are in uncharted territory here, because we don’t have any data on this. In the Alpha model, the nearest thing they have is ‘Guides’ who receive a student’s data and have conversations with them, such as: “I noticed you went off-topic there for a while?”

That human question is a really important one and something that I’ve wrestled with. If more and more of the curriculum design, the instructional sequencing, and the assessment is delivered by AI, then it’s obviously going to come through tablets and screens. And then the question becomes: What do we lose?

There is still a social environment at Alpha. It’s a physical building. Kids come in. They have an assembly where they interact. And then, at nine o’clock, it’s two hours of online learning, although they’re still beside each other as they do this. There’s a system, where they have to generate a certain amount of experience points, and get a certain amount right, and so that brings some accountability.

But the question you’ve asked is a massive one, because it comes down to: What kind of schooling do we want for our kids? Are we going to eliminate something valuable—a teacher inspiring a kid? And I guess my answer is: we can’t scale that. We only hear the success stories, but what about the other 25 kids in that class? The Alpha model does offer scaling. You say to a kid, “Focus for two hours, hit these targets, and then you’re off to be a kid”. I think this will improve their academic outcomes. It might also improve their mental health. But these are difficult questions.

Conor: When one imagines the ideal schooling, it’s always the inspiring teacher. But the social influence of a teacher can also have an estranging effect, on those it doesn’t work for?

Carl: Exactly. One of my daughters was diagnosed with autism. School will be a challenge for her, because there is a noise in most classrooms, plus a lot of ambiguity and blurred edges. That can cause her distress. I was looking at an app that she’s using, where she flies through problems because there is more clarity and boundaries.

That was one reason I started working with Alpha. I realised that this will help kids with special educational needs that are not met in mainstream education. They could have a system where they’re making rapid progress, and not feeling like the bad kid in class all the time.

But again, the messaging is going to be very difficult. Critics will say you just want to “put autistic kids on screens”. But it’s for an hour or two a day. And then we want them to be kids.

Conor: What role do ‘AI tutors’ play in this vision?

Carl: There are three broad areas in education: curriculum (what we teach), instruction (how we teach it), and assessment (how we know they know). When the public hears ‘AI tutor’, I think they usually imagine a robot doing the instruction part.

But for me, AI tutoring is more about the back-end: curriculum design and assessment. If you accept that learning is an algorithmic enterprise governed by biological laws, then the most foundational elements are what you teach and how you assess it.

Designing a curriculum and teaching materials is more about engineering than about whether you can inspire kids. It’s about sequencing the materials. There are broader questions about the experience and social interaction that we want students to have. But, for more basic questions, like, “Is there a better way to teach quadratic equations?” I think we’ll see a shift from the current à la carte approach where teachers often decide what materials to use. And as uncomfortable as people are with it, I imagine that we will get AI-generated content that is just better than human-generated content.

Conor: You also see AI changing how we assess students?

Yes, your earlier point is right: if a kid writes something, they often want you (the teacher) to read it. But the Welsh educationalist Dylan Wiliam also referred to marking books as the most expensive public relations exercise in history. It’s tedious for teachers and much of it has zero impact on student outcomes.

This is also because human marking is unreliable. I was Head of Department of English and I saw situations where a student would do an essay, and get a D. And then it would be sent back to be re-marked and another person would give it an A. You can’t run a system like that.

The work on comparative judgment by Daisy Christodoulou and No More Marking, which calls for student writing to be assessed in comparison to other students’ work, rather than as an absolute, shows how AI could help improve the reliability of marking and reduce teacher workload.

But we also need to distinguish between the assessment of learning, and assessment for learning. Assessment shouldn’t just be an endpoint. It should be a regular temperature check. In a classroom, this kind of ‘checking for understanding’ is the most powerful thing a teacher can do, whether it’s a mini whiteboard session or a cold-calling activity. It’s imperfect, but it can identify misconceptions that students have.

The dream is that instead of teaching a pre-defined sequence of materials, an AI system takes regular temperature checks and operates like faders on a mixing desk. Based on the student’s response, the system identifies misconceptions in real-time and dynamically selects the best explanation for that specific kid.

This also means that we can meet the student at the point where the misconception occurs. Most current assessments are useless because the feedback loop is too long. If a kid writes an essay and gets it back a week later, they’ve already forgotten their thought process.

Conor: This would make education ‘adaptable’ to a given student. But you’ve also cautioned against using AI to match teaching materials to students’ different ‘learning styles’. What’s the nuance here?

Carl: The idea of learning styles emerged from a tradition in the 1960s and ‘70s, which thought about learning in an individualized way. As a teacher, I had to spend years trying to design lessons for different learner types, such as ‘visual learners’ or ‘kinesthetic’ learners, who purportedly learn better through hands-on experiences.

This idea has been empirically tested. Not only is there no evidence for learning styles, but it is the nearest thing we have in education to homeopathy or healing crystals. When you think about it, it’s preposterous to suggest that if you were teaching geography to an ‘auditory’ learner, you should make an audiobook to describe the continent of Europe. The content should determine the form.

Where there is evidence, is around the idea of dual-coding. Our brains process information through separate but interconnected channels: one for verbal information (spoken or written words) and one for visual information (images, animations etc.). There is value in designing explanations that think carefully about how to combine text and speech with visuals, so that you don’t overload students’ limited working memory.

For example, I see some great video explainers on YouTube, where, they show the actual differences between the size of the planets and it’s nothing at all like the maps of the solar system that you normally see. That’s an example of where the visual aspect really helps, but it has nothing to do with me being a visual learner.

Where education goes wrong

Conor: You are often supportive of explicit forms of instruction and skeptical of constructivist approaches where students explore and solve problems by themselves. Why?

Carl: Constructivism is a philosophy of meaning, not a pedagogy. It’s the belief that each individual constructs their own knowledge and understanding. It has its roots in Rousseau, but really got moving around the turn of the 20th century, with people like (the education reformer) John Dewey in the US, and (the Swiss psychologist) Jean Piaget in Europe.

As a philosophy, constructivism is true. We do construct meaning, but from an education perspective it has led to a belief in minimally guided instruction, the idea that you learn things best when you discover them for yourself, rather than having them taught to you.

What this does is to conflate the outcomes with the means. We do want students to be able to discover things for themselves. But discovery learning as a starting point for instruction is a disaster. Clear, sequenced instruction is effective for novices. And every kid is a novice in most domains.

If you’re learning to drive a car, you want explicit instruction up top—do this, then do that. Once a person is capable of driving, if you’re still doing that as a teacher, you’re getting in their way. That’s when the discovery element comes in.

There’s an ethical dimension to this too. Students who come from affluent backgrounds, with strong existing schemas of knowledge, who have been exposed to rich vocabulary around the dinner table, will flourish in an environment where there are few constraints or expectations in terms of their behaviour. Discovery learning privileges the already privileged.

Conor: This idea that novices can’t be expected to figure out things for themselves is also why you are skeptical of students learning by asking questions to chatbots?

Yes, the power of AI is in the monitoring, curriculum design, instructional design, and assessment. Novices won’t learn by discovering knowledge themselves by asking chatbots questions; they need explicit instruction.

Conor: Your argument that educators conflate the outcomes with the means, also explains why you’re skeptical of calls to teach students ‘21st century skills’, like critical thinking, creativity and adaptability, including as a response to AI?

Carl: Yes. Critical thinking skills and creativity are an outcome of systematic knowledge building, not a starting point that can be directly taught, as a generic skill.

Conor: The explicit instruction of knowledge that you advocate for might be effective for subjects like maths that can be more easily decomposed into sub-components. But what about subjects like English or history?

Carl: That is the Holy Grail of curriculum design. We have what AI pioneer Marvin Minsky and the computer scientist Walter Reitman would call “well-defined” and “ill-defined” domains.

Well-defined domains, like maths and science, are more clearly sequenced and hierarchical. They have concepts that are free-standing and can be taught in an isolated way. You can go back two steps if you miss something. They often have answers that are definitive or unambiguously correct. A lot of education apps that work well are in these areas.

In the humanities, knowledge doesn’t operate in that way. This will have implications for how we design instruction and assessment. But I go back to the materialist point. I believe these things are amenable to the laws of the universe and AI will discover what the optimum sequences are.

Why Reading Still Matters

Conor: You’ve spoken a lot about the decline of reading. Why does it worry you?

Carl: I find it sad that a lot of kids are not going to read Dostoevsky. But I also recognise that for me, growing up, reading was partly a feature of being bored, and we can’t pretend the conditions are the same today. It comes down to an ethical question: what kind of life do we want for our kids? This is not a question we can answer in a randomized controlled trial.

There are certain questions about reading that we can answer empirically in trials. One of the studies that stuck me dead in my tracks many years ago showed that the percentage of words we need to understand, in order to read a text, is 95%, which shocks everybody. The way that vocabulary is taught in schools is very poor, and inequitable, and there are almost certainly better ways to do it.

But we can’t just say to kids: “You need to read more!” That doesn’t wash. We need to answer the philosophical question of what we value reading for. The benefits are not always immediately obvious. They might pay off 20 years down the line. The literary critic Harold Bloom once said that Shakespeare shaped the modern consciousness, including our contemporary understanding of love, death, ambition. Something like the King James Bible affected English-language culture too. These were dominant texts, and culture passed through them.

What are we going to lose if we lose these books? I want my daughters to read because they’ll see that their struggles are not their own. So they can see other characters struggle too.

Share

It’s an intellectual vanity to claim you can always identify AI writing—that chatbots spit out nothing but sanitized blah. Yes, the humans who talk most with machines are better at detecting their patter. But proclaiming “You can just tell!” is a coping mechanism with an expiration date.

Actually, you can tell. Because AI writes better than nearly every human alive, explaining with clarity, and allowing the curious to bypass the impenetrable prose with which the self-important sandbag outsiders. In many respects, AI writing is a boon to human understanding.

Weird effects abound too.

Bureaucrats must wade through floods of oddly articulate petitions, legal writs, and complaints, as the public’s literary servants overwhelm the public servants. Police can spam the spammers, able to shovel AI slop into criminal-hackers’ chatrooms, much to the ire of the crooks. Meantime, the best man’s tipsy speech at the wedding and his sober words at a funeral—they’re so emotive you know a machine wrote them.

That’s not to mention the high-profile wins for literary AI, such as the prize-winning short story, cited for its author’s “melodic voice,” which earned a different review from the detection software: “100% AI-generated.” Or the (somewhat) non-fiction book The Future of Truth, about how AI undermines factuality, and included AI-generated quotes. Not to mention social media, glutted with even more vapid junk than when humans alone monopolized that genre.

In the 19th century, theologians spoke of “God of the gaps”: that the holy spirit reigned where science faltered. Only, science kept expanding, forcing the supernatural into retreat. The philosopher of technology Benjamin Bratton says we’re now anxiously defending a “humanism of the gaps,” defining our species by whatever machines cannot do quite yet.

But AI capabilities arrow upward in so many domains. Why should writing escape it? From journalism to fiction, more than a few prose pros are already leaning on AI writing, with its tantalizing offer to solve the blank page. Such usage remains taboo, so few admit to it. But a Nobel Prize winner just did, the Polish novelist Olga Tokarczuk, who seeks creative pointers from AI.

“I often throw an idea to the machine for analysis, asking, ‘Honey, how could we develop this beautifully?’ ” she said at a recent onstage event.

“At the same time, I feel a poignant, very human sorrow for an era that is disappearing forever. My heart aches for the passing of traditional literature, written over months in solitude, a work of life crafted in the mind of a fully conscious, single individual.”

Personally, I have devoted my adult life to writing, publishing five volumes of literary fiction, a nonfiction book that I ghostwrote, and scores of newspaper articles—years of piling up sentences such that an editor might buy them on a computer file. But it dawned on me: the computer, not my files, is what enchanted the world. So I pivoted to the topic of the times, AI policy, the many-handed effort to push an elephant.

Which is to say, I worry about the fate of human writing. And yet something odd keeps happening.

The instant I discover that what I’m reading—perhaps rather interesting—is AI-generated, I suddenly don’t care. The words dim before me. It’s an off-switch.

Why?

THE PAINTER MADE OF PIXELS

A conceptual artist named SHL0MS posted a picture on X, and asked readers to say why the image was worse than a real Monet.

Nobody spared the machine’s feelings.

“I’m disappointed I have to even point it out,” one person commented. “The background lilypad-algae amalgam is egregiously vague, like most AI art.”

Another added, “Doesn’t look anywhere near like a Monet. Looks exactly like somebody trying to replicate the style and achieving like 20% of it.”

In fact, the painting was a Monet, and the post was a prank that, seemingly, exposed the pretensions of people who flock to galleries, claiming to admire the art but really admiring status, entering each room and hurrying to the wall text: Who is this? Should I care?

That seems shallow. But the same drive could save human writing.

After all, you should care more about a real Monet than a dupe. Not because one is objectively better, but because its meaning exists in a matrix of social beliefs about beauty, about value, about the shared tales of human civilization: that an irascible Frenchman once swiped hog bristle across canvas, his perceptions and drives filtered into a decorative depiction of a pond, conveyed from dealer to collector, rising the ranks of culture, finding a lonely museum wall, venerated there for years, as the outside world transformed, the artist died, and pixels reproduced his perceptions and drives, connecting your sensing brain to the sensing brain of a particular man who will see nothing ever again.

All that is only in our heads. But where else is meaning but in heads? And writing is humanity’s most sophisticated technology for visiting another’s head.

The problem is that writing is hard. Words elude you; paper doesn’t care. George Orwell equated writing a book to suffering from a long illness. Developers dream that AI will someday cure all illnesses. Why not this one?

WHY (A)I WRITE

In his essay “Why I Write,” Orwell listed four main motives: 1) sheer egoism; 2) aesthetic pleasure; 3) the urge to say what is; and 4) the urge to change what is. Others cite a fifth reason: to know what you think.

Those who don’t write presume the process involves hatching an idea, then putting it into words. It’s commonly the reverse: put down words to hatch the ideas within them.

This points to a flashing peril from humans impersonating themselves via AI writers: cognitive surrender, that we hand over the wearisome word toil, and atrophy in genteel luxury, never enduring the mental frictions that kindle into wisdom. We wouldn’t detect the decline, I suspect, half-believing ourselves the authors still.

As a leading AI thinker told me in private, “I write so I can keep writing in the future, so I can keep thinking in the future.”

But there’s another reason that people write, a motive far more powerful than humankind’s dwindling determination to remain the brightest species. It’s ego. As Joan Didion observed, “Writing is the act of saying I, of imposing oneself upon other people, of saying listen to me, see it my way, change your mind.”

Every person craves attention, and the nerds buy it with words. Readers possess a natural resource—attention—that is everywhere and nowhere to be found. A transaction ensues: reader agrees to spend attention on words in exchange for information (nonfiction) and/or experiences (fiction). The writers’ art is to smuggle themselves into both.

So, when reading human, you’re absorbing the meaning with one eye while focusing the other on its maker, ever trying to reconcile the two, much as the visual cortex creates 3-D from the parallax of two eyes synthesizing one object at dueling angles. If you see that the author was AI, one eye closes. The page becomes markings on a flat surface.

The reader’s presumption of a human behind the words makes secret AI authorship worse than betrayal. It’s trespassing and fraud at once, entering another’s brain on false premises and altering the contents for gain, whether it’s AI-generated reviews that trick you into feeding at someone’s restaurant, or an AI-generated marriage proposal that tricks you into feeding someone’s children. It’s the sale of a shared reality, from an entity that has no reality to share.

That said, much text is purely for dispersing information, and the author is irrelevant. This is true for most legal documents, press releases, and boilerplate emails, not to mention explanations that you prompt a chatbot to conjure. AI composition can be helpful and harmless, provided it’s honest.

Subscribe now

THE UNWRITTEN RULES

But nobody knows the ethics yet. Hidden usage seems wrong because deceit is wrong. But must you specify authorship in every case, even if it’s a casual note that you autocomplete? And if you’re candid about it, does that make AI writing fine?

News publications ban it, educators resist it, and academic journals flounder. Yet social norms could be shifting subtly toward acceptance. Increasingly, you hear the view that authorship is immaterial; it’s about making something worth reading. “Honest to god,” wrote Tobias Schneider, a research fellow at the Global Public Policy Institute in Berlin, “what do I care how the text for an academic paper comes about. It could be a monkey throwing darts at a board, if the resulting combination of letters conveys original insight, it should be published.”

As with most AI anxiety, it’s a matter of speed: this is moving faster than the culture, leaving us in a perma-panic as one train after another hurtles through the station, heading to destinations unknown with more of our belongings aboard.

Already, large-language-model vocab is springing more frequently from our mouths, according to one study. Authors, in dread of false accusation, are altering their prose to avoid seeming LLMish. I know because I’m doing so, tense about my appreciation for bullet points and curbing the dashes that I learned to overuse in old-fashioned newsrooms—for melodramatic punch.

Errors have become fashionable too. Once, clean copy was a hallmark of professionalism. But “the house style in most newsrooms is extremely LLMable,” the Atlantic magazine contributor Jasmine Sun noted. “What stands out (besides reporting) is a distinct and authentic first-person voice, even if that means the occasional typo.”

Not for nothing is wabi-sabi—the Japanese tradition of charming imperfections—trending. To err is human. To not err is AI.

For those who venerate human writing, the nightmare is that AI authorship becomes so commonplace that few care about origin anymore. Nobody would need to lie about it anymore. Looking at recent history, technological tools for knowledge work have tended to draw contempt at first, only to gain wide adoption. Many researchers shunned internet sources back in the 1990s, while journalists of the early 2000s scorned Wikipedia. But the technology improved, and norms updated.

If AI writing resolves its quirks, will the same acquiescence follow?

3 FACTORS THAT COULD SAVE HUMAN WRITING

Here’s what may decide whether most people still care about people writing: 1) institutions clarifying the rules and norms of AI usage; 2) reliable detection technology becoming widespread; 3) people retaining the “off-switch” aversion to AI authorship:

1. Establishing Rules

When it comes to AI-generated text, the closest norms regard plagiarism, which also involves the presentation of others’ work as one’s own. But the concept of plagiarism arose to protect the intellectual property of a fellow human. With AI writing, who is the victim?

If writing is dishonestly presented, one victim is plain: whoever is duped into reading it. This suggests that openly AI-generated text is fine. But victims may exist in those cases too.

Among the harmed could be people who fail to write, and therefore never develop their minds. Also, there are the human authors who’d need to vie against hordes of fluent bots hogging public attention. Most of all, society could suffer from the decline of collective cognition, if writing is no longer a push-and-pull involving thinkers living and dead, but just effortless insta-words materializing onscreen.

Yet AI writing is immensely useful, and stimulating, and will stir human creativity. We must set rules and norms for concrete cases, not just shout, “Don’t!”

Is it wrong for an AI to write an action plan for a business client? What about a write-up of the family vacation to share with loved ones? How about assigning AI agents to write your social-media feed?

The debate over AI writing often misses an even more likely outcome: assisted writing. What if you come up with all the ideas, but an AI turns them into clearer prose than you could, which you then tweak? Or reverse that: the AI brainstorms, but you write the prose. Which is worse, and why?

Recent studies suggest that AI writing may neutralize the human user’s arguments, and that essays drafted by people contain a range of ideas, whereas AI essays tend to converge around similar points.

On the other hand, chatbots—by disseminating ideas so much more gamely than unmoving written texts—may stir our creativity. In theory, future personalized AI models could allow users to vary the “temperature” (randomness) of LLM responses, increasing their novelty.

But without social norms, dystopian futures come readily to mind, with humans generating AI text for other humans, who employ AI to read it and craft replies, downgrading our species’ intellectual conversation into a feedback loop of artifice, in which we are mindless mouthpieces of literate machines.

That feels far-fetched today. Yet some people are already cramming past emails and blogs and tweets into AI, training agentic systems to capture their “voice,” such that they needn’t cultivate one.

If leading institutions care to keep human writing, they must establish clear and coordinated policies in haste, so people know the professional and social costs of their choices. This depends on detecting what is human and what is machine.

2. Detection Technology

Disgrace is an enforcer. Soon after the explosion of chatbots in late 2022, nobody could reliably identify whether text had been written or generated. But AI-writing detection software—notably, the Pangram app—has become strong.

You can tell because it keeps precipitating scandals. But institutions have yet to apply detection tools as widely as they should. That would mean standardized detection everywhere we care about human input and honesty: job applications, dating apps, any publication’s vetting process.

For detection apps, the challenge is twofold: Do not mistake AI writing for human (a false negative); and Do not mistake human writing for AI (a false positive). The first error makes the app worthless, but the second error is especially dangerous. A wrongful accusation—whether against an author, a student, or a public figure—could destroy a career.

Last year, the University of Chicago economists Brian Jabarian and Alex Imas published an evaluation of several detection apps, having had them evaluate around 2,000 human-written passages, including fiction and non-fiction from before 2020 (to avoid texts that were stealth AI), along with the same number of AI-generated passages.

“The results are clear-cut,” Jabarian and Imas wrote. “On medium-length to long passages, Pangram achieves essentially zero FPRs [False Positive Rates] and FNRs [False Negative Rates] within our sample.” The error rate did increase slightly with short passages, under 50 words. (For reference, this paragraph is exactly 50 words long.)

The accused still deserve a fair hearing. Institutions should establish systems of appeal, while independent evaluators must keep testing the accuracy of detection tools. This poses an uncomfortable question: what rate of false accusations is acceptable? The reflexive answer is, “None!” But we accept a degree of error in matters that are even more impactful, such as the justice system. Society considers it better to identify many culprits than to preclude any miscarriage.

Benjamin Franklin preferred that 100 guilty go free than 1 innocent suffer, while the legal profession debates “Blackstone’s ratio,” which proposes 10:1. Developers of detection apps likewise make such moral calculations, with Pangram training its models to average no more wrongful accusations than 1 in 10,000.

But AI developers are pursuing ever-better writing tools, which could raise the opposite problem: increasingly mistaking AI writing for human. In which case, we’ll need further antibodies.

3. The Off-Switch

“More than 300 years after the reading revolution ushered in a new era of human knowledge, books are dying,” warns the leading chronicler of literary decline, talented young fogey James Marriott. He marshals a range of sorrowful datapoints, from the collapse of reading-for-pleasure, to the inability of college kids to handle any textual analysis more cognitively vexing than Instagram.

If people quit reading, it’s moot who (or what) writes.

But a paradox is that our purportedly post-literate era is the most literate in history, if you judge by how much people are producing and consuming words: all the social-media posts, all the texting back and forth, all the newsletters—not to mention the word-streams on a billion podcasts and video clips.

A future without writing—as opposed to today’s decline of deep reading—is possible only if we find a technology more proficient at transmitting ideas and experiences from brain to brain. Brain-computer interfaces are not near any such text-ending magic.

Until then, human reading contains the “off-switch” antibody: that you just don’t care as much once you discover that nobody said it. It’s a psychological reason for why writing belongs in what the economist Alex Imas calls the relational sector: “the human-intensive, provenance-rich, sometimes artisanal part of the economy where the human aspect is part of the value of the good or service itself.”

Indeed, Imas contends that excellent human writing may rise in value in the AI era, partly because so few people will be able to do it. In other words, the post-literate era of tongue-tied semi-literates could make the expressive few stand out. One caveat: if people develop relationships with AI companions, they may care deeply about what those entities write.

THE HUMANS ARE DEAD. LONG LIVE THE HUMANS.

A classic twist in sci-fi horror is when the main character finally twigs: all the other “people” are actually machines. Lately, I’ve felt like that character, discovering when I drop more and more texts into the AI detector that robots are everywhere.

Not only that literary award-winner, but the award citation itself. Also, an essay—on the importance of reading!—by a thinker I admire(d). Then, another piece of writing landed on my desk. Not onscreen. It landed physically on my desk: a handwritten letter, composed in pen, with “Tom” on the front of the envelope.

I have it before me now, saved as no email would be, because of the fellowship of human beings, that we mind about words—but also about another person’s effort and intent. Machines are so eloquent. But until they struggle to find the words, they won’t compose a letter I’d keep.

Human writings are an improved version of human beings. By escaping the confines of your skull, by pausing time, by revising what you’d blurt, you craft what you mean to mean.

“What I write about is other than me. As what I write is smarter than I am. Because I can rewrite it,” Susan Sontag said. “My books know what I once knew, fitfully, intermittently.”

Writing is an intelligence-increasing technology, allowing you to convert the SOS pulses of the perceiving self into objects to organize, line up, accumulate, frame, and build into more than the capacity of your known mind. For the species, it’s an even more IQ-increasing technology, turning thoughts into a public good that never expires.

AI writing may burn books that don’t yet exist. Perhaps that’s fine. Publishers are always moaning that too many books come out. And maybe we’ve reached the end of our utility as authors: we wrote superb datasets called literature; the machines enjoyed that appetizer.

But our words are more than their feed. Writing is our testament, permitting readers to find companions among the dead, and the author to befriend the yet-to-be-born, granting anyone a way to defect from the present, and providing even the godless with a faith that something may persist after life.

We—briefly thinking machines, stuffed with noisy training data, false memories and nostalgia, jostled by fluke and circumstance, each generating words that nobody else would—we are the worst writers, and the only writers I care to read.

Subscribe now

We’re avid users of AI: to learn, to critique our essays, and to point out how many absurd mistakes we’ve inserted. Somehow, it’s less painful coming from a bot.

But we will never (knowingly!) publish AI writing.

Below are our rules for AI writing. Add your own in comments below. And check out this essay on why human writing matters.

—Tom Rachman & Conor Griffin, AI Policy Perspectives

1: The Embarrassment Rule

If you’d feel ashamed by exposure, don’t use AI to write. (Or credit it)

• Much as DNA testing is identifying the guilty years later, tomorrow’s AI-detection apps will expose today’s clandestine usage.

• This is happening already, with suspicious readers uploading post-2022 fabrications that once seemed undetectable.

Rule-of-thumb: Assume your AI usage will be discovered.

2: The Intern Rule

Use AI in the process of writing. But treat its contribution as you’d treat that of a person.

• It’s fine for AI to explain a subject when you’re finding your way, much as you might consult a knowledgeable colleague. But if your final text consisted of nothing but that colleague’s points, you’d be remiss to take full credit. Likewise with AI.

• It’s fine for AI (or a human research assistant) to ferret out articles, and to summarize them, especially technical material and turgid writing. It’s not fine for you to read nothing yourself.

• It’s fine for AI to help brainstorm your ideas. It’s not fine for AI to supply all the ideas without credit.

• It’s fine to have AI proofread. It’s not fine for AI to rewrite.

Rule-of-thumb: Test whether you’re going too far by asking: Would it be wrong to accept this help from a human without crediting them?

3: The Ventriloquist’s Rule

If you want to learn a subject, put it into words. When AI writes, it deletes your future wisdom.

• Sometimes you’ll just need an artifact. But remember that, if a machine is generating all of your words, you’ll end up as knowledgeable as the ventriloquist’s dummy.

Rule-of-thumb: To speak more fluently on a topic, write about it.

4: The No-Dumping Rule

Nobody wants to read outsourced AI replies.

• If someone asks your view, and you ask a chatbot, and send them a 10-page document that you identify as AI, this is burden-dumping.

• Nobody should read your burden-dumps. They could’ve created those without you.

Rule-of-thumb: If someone respects you enough to seek an opinion that you cannot offer, respect them enough to say no.

5: The Byline Rule

Your byline is an oath.

• Readers assume that the named author wrote the text. If this is only partly true, your byline can be an act of deception.

• Many types of writing are valuable without personal authorship: weather forecasts; instruction manuals; explanations about the world. Generate as much as you want. Don’t take the byline.

• In other types of writing, authorship is the whole point—for instance, a school essay or a condolence letter. Never generate this.

• News reporting and scientific research occupy an ambiguous midpoint, where opinions may differ. In their idealized form, each aspires to share knowledge above all else. If AI can write these findings better than the people doing the finding, there’s arguably no problem.

• But a byline also establishes accountability for claims, and builds networks of thinkers. Not to mention the motivation of an author’s career ambitions and vanity.

Rule-of-thumb: If you—or the reader—care who is named, don’t generate.

Leave a comment

I just heard a maven of the AI scene using a new verb for the addictive practice of testing all you read online in an AI-writing detector. “I’m Pangramming everything lately,” he said, referring to the popular app.

If nobody can tell who (or what) wrote the words, a centuries-long conversation among humanity degenerates. Slop sloshes through debates. Honest authors lose out, and readers wonder what a byline means. Until recently, that seemed our fate. But has AI-detection technology hit a turning point?

To find out, I caught up with the CEO of Pangram, Max Spero, a former Google software engineer who co-founded his company with an ex-roommate from Stanford. When they set out in 2023, the average citizen was still asking, “What IS a large language model?” Max had a different question: “What is society, if we can’t tell human from machine?”

—Tom Rachman, AI Policy Perspectives

[Interview condensed and edited for clarity]

Tom: Here’s a provocative question: Why care about human writing? Perhaps the important part of writing is communicating information. Why not use AI to do that better?

Max: There is a social contract between the writer and the reader. If I believe there is an idea worth sharing, I pay a cost by writing a piece of text, and the reader pays a cost by taking the time to read it. In a future where AI content lets you bypass the cost of writing your idea and formulating it, we get into a situation with perverse incentives, where people are posting total slop, and the reader is taking more time to read than the writer put into the text in the first place.

Tom: I’m not convinced that’s the whole answer. I can imagine cases where it would take an hour to write a letter yourself, but two hours to prompt it with AI. If effort was the key, we’d value the AI-written letter more. I think we care about human writing for reasons beyond effort. First, if the AI author isn’t disclosed, there’s an element of deceit, and we’re offended by lying. Second, even if the person is completely transparent about AI use, we may still mind.

Max: Agreed. The reader-writer relationship is ultimately a relationship between people. Another thing to think about is what can happen with AI writing at scale. It’s so easy to run deceptive operations. They used to have Russian troll farms with hundreds of people writing internet comments. They’ve replaced these with AI and LLMs, and they’re able to run at a much larger scale and push political agendas. On a micro scale you might say, “Well, if I’m looking for a news article, what do I care if it’s AI-generated?” But on a macro scale, understanding the provenance really matters. Otherwise you enable these bad actors to push their agenda.

Subscribe now

HOW AI DETECTORS WORK

Tom: When chatbots really took off in early 2023, one of the first parts of society to feel it was education. Students were using them to do assignments, and some educators abandoned the homework essay altogether because they had no way of reliably identifying AI-generated submissions. Can you explain why early attempts at AI detectors failed? Maybe start by explaining the term “perplexity.”

Max: Early AI-detection tools tried to solve the problem by measuring perplexity, which is a metric of how unexpected a piece of text is. Something like, “For lunch, he ate a bowl of soup” is low in perplexity while “For lunch, he ate a bowl of spiders” is high in perplexity. Large language models work by composing probable text, making the writing much less varied, more steady, and therefore low in perplexity. But when humans write, we tend to scatter unexpected things in there, meaning our texts are much more varied, with many parts predictable, other parts slightly less so, and occasional bits that really surprise you. That variation is known as “burstiness,” as if you have spikes of surprise bursting through. AI writing is very low in burstiness. Early on, researchers saw you could use perplexity and burstiness to distinguish human from AI text pretty well, with between 95 percent and 99 percent accuracy.

Tom: But you rejected that approach. Why?

Max: First, that level of accuracy might sound good, but it’s still way too many errors to use in the real world. Also, the approach had several problems besides that. One is that any text that happened to appear in the LLM’s training set would seem like probable phrasing to the model, and therefore low perplexity—so something familiar to it, like the Declaration of Independence, might get flagged as AI-generated. Another problem is that people learning English tend to write more simply, meaning their output is low perplexity, and can also get flagged as AI. So, we just threw all of this research out the window, and decided to train a deep-learning classifier.

Tom: This is how I think you built your deep-learning classifier—tell me if I get it right. First, you took an open-weight base model. You fine-tuned it with hundreds of thousands of labeled examples of human text and AI-generated text in various genres. From this, the model learned to differentiate human writing from AI-generated, but imperfectly. So you added a stage of “hard negative mining,” where you took all of the cases where your model had wrongly classified a piece of human writing as artificial—false positives—and you generated AI versions of that same kind of text. You added those examples to the original fine-tuning dataset, then went back to stage one, training the open-weight model from scratch, but now including the edge cases. An algorithm ran this process as a two-stage loop, making it better and better, until you had a product.

Max: What you’re describing was our process when we started Pangram. But we’ve never stopped training models. Keep in mind that we could tune models to be more sensitive to AI content, or we could tune them to be more conservative, and have fewer false positives. Avoiding an incorrect judgment that a piece of human writing was AI-generated is something we’re most careful about avoiding. So we calibrate the model to a rate of 1 in 10,000 false positives, keeping that constant while trying to improve how much AI content we catch, or the model’s “recall.” We’re always running new experiments, bringing in new datasets, new domains of writing. The goal of training new models is not to just slide this tradeoff scale; it’s to move the entire curve. And everything is improving. We’ve also come up with additional important techniques. The main one is training the model to differentiate between full AI generation and AI edits. What we did is ask LLMs to edit text, make it better, fix grammar mistakes, summarize it. Then we measure the distance between the original text and the edited text, and we train our model with that, showing it that a certain distance from human text is equal to a light edit, and further is equal to a moderate edit. We train it to recognize that, in addition to the fully AI-generated text.

Share

Tom: So you have a dashboard, where people can drop in text, and a browser extension that’ll do live checks on social media feeds like LinkedIn and Substack and X. Obviously, your products aren’t much use if they’re flinging out tons of false negatives, letting AI-generated content through the net. But it’s the false positives that’s worse here because a wrongful accusation could be devastating. You might have it down to an average of 1 in 10,000 false positives—but at scale, that still means mistakes. Have you been monitoring false positives in the wild?

Max: Definitely; we have a tracker. If anybody comes to me, and says, “It’s a false positive!”, we’ll triage it internally. Also, on the app dashboard, we have thumbs-up/thumbs-down on the results, so we can track responses at a higher volume. But that is pretty unreliable because often it’s just somebody unhappy that their obvious AI text got flagged.

MYSTERY OVER WHAT THE MODEL SEES

Tom: I want to dwell on the false positive rate for a moment because this concern—wrongful accusations—feels key to adoption of this technology. When you talk about 1 in 10,000 false positives, or 99.99% correct classification, that’s an average across many forms of writing. But the errors are higher for certain types of material, such as recipes, poetry, and how-to articles. Why?

Max: These are all fairly formulaic types of writing, so it’s going to be lower signal. With poetry, it’s also very short texts in most cases.

Tom: And your detector refuses to evaluate anything below 50 words. But when it can give its verdict on a document, is it evaluating the likeliness of AI at the level of each word? Or the paragraphs? Or the entirety of the text?

Max: It’s looking at windows of about 200 to 500 tokens [roughly 150 to 375 words]. We have a system that does a pass, and tries to find rough boundaries between AI and human text, and then we do a finer sweep to try to find more exact boundaries, specifically for mixed text.

Tom: When it flags a piece of writing, your interface lets the user see “supporting evidence” that something was AI-generated or AI-assisted. That can include the telltale phrasing that people have come to associate with AI writing. But this supporting evidence isn’t actually how your system made its judgment, right? It’s a post-hoc analysis of the text that your model already flagged. Is that because deep-learning is great at somehow identifying AI, but you don’t know precisely what it’s picking up?

Max: Yes, the deep-learning classifier is a black box. We don’t have a ton of interpretability into why it makes the predictions that it does. We include the supporting evidence because there’s always demand from people wondering what makes a text read as AI, and how they can train their own eyes to detect it. But the Pangram classifier is not looking only at the surface-level features. It’s finding patterns in longer-context features that an LLM will use in structuring and writing a doc.

Tom: It’s aware of something exclusively human in writing that we ourselves cannot necessarily put our finger on.

Max: If you’re looking for an objective measure, it might be that LLMs are better than humans in so many ways: they don’t make grammar mistakes, they often have coherent arguments, while if you look at the full distribution of human text, many humans are writing less coherent arguments. But it’s also that LLMs are a lot less diverse: if you ask them to write 100 arguments on a topic, they’re going to cluster in one area, whereas the space of human arguments is going to be very diverse. That’s something AI can’t replicate or reproduce.

Leave a comment

Tom: But AI developers will improve the quality of writing. Why think you can keep ahead of that?

Max: To make a frontier model more capable, you have to instill preferences—it has to prefer to do one thing rather than something worse. But it’s these preferences that Pangram is detecting in writing. The most undetectable language models were GPT-2 and GPT-3, where they were not trained to have clear preferences, but to approximate the human distribution of writing. But today’s LLMs have stronger preferences, and this is a big reason we’re able to detect the text so easily. A caveat is that LLMs are getting more complex, so that is a headwind we must deal with. Also, the way people use AI has been changing. It went from a student saying, “Write me an essay” to somebody using an agentic harness, and saying, “Research and write a Substack article on the Strait of Hormuz.” Those are completely different use cases.

AI DETECTION IN THE WILD

Tom: Who is using your models right now, and why?

Max: We have a range of people using them. Universities use this to help with academic integrity. We also have academic conferences such as NeurIPS, the biggest AI conference, which recently said they need position papers to be substantially human-written, and they used Pangram to flag papers that look fully AI-generated, and either desk-reject or give authors a chance to appeal. We also work with some traditional publishers, as well as anybody with datasets who’s worried about the quality. That is an underrated aspect of this: if you’re buying data, you don’t want it to just be slop; you want it to be quality from real experts.

Tom: Which sectors aren’t dealing adequately with AI detection?

Max: Publishing has been really slow to catch up on this. There were a couple of major scandals, like the book Shy Girl that people called out as being AI-generated. It got pulled from shelves. A lot of publishers don’t have a strong AI policy. Some have decided that, as long as the text is good, we’re going to evaluate it on its merits rather than whether it’s AI-generated or not.

Tom: My sense is that most publishers would mind if a piece of writing is AI-generated. If anything, I’d guess that there could be a lag with many in publishing unaware that detection technology now works well. I suspect there’s also an aversion to technological involvement here—that you should trust authors protesting their innocence and not trust an app. I sympathize with that. Anyone accused deserves a fair chance to defend themselves. But failing to take this technology seriously is even more harmful to human authors who actually write their own stories.

Max: They lose something real and tangible.

Tom: What are cases where you’d consider it fine to use AI to write?

Max: I don’t really consider it “using AI to write.” You can publish AI content, and that’s fine, as long as it’s properly disclosed. But if I’m writing an email to somebody, it’s somewhere between impersonal and disrespectful to use AI to write that. AI is a useful tool: we should be using it to generate text, synthesize text, and it’s even reasonable to share AI output. But if we’re talking about “using AI to write,” I’d caution against that framing.

THE FUTURE OF “WRITING”

Tom: We’ve been talking about this as a binary: either human-generated or AI-generated. But maybe we’re destined for a future of cognitive blending with AI, where people outsource parts of thinking, much as we now outsource parts of our memory to machines. Maybe the idea of a human-or-machine binary is a short-lived prospect.

Max: Yes, I could see that, especially with so many applications pushing AI assistance on users. But I think there would still be degrees of human input that would differentiate such writing from the entirely AI-generated.

Tom: So there is a long-term future for human writing?

Max: Definitely. People just care about it. To me, it doesn’t matter how good the AI gets. There’s still a lot of people who care, and a lot of industries, and types of writing, where it really matters that something’s human.

Tom: What would those be for you?

Max: Fiction is one. But also, think of an essay contest. Is the goal really to find the best essay? Or is it to reward writers who have taken the time to write something compelling?

Tom: Well, I think it’s probably both. If it’s an essay contest in a school, say, you’re rewarding effort in part, but you’re also rewarding quality. Or else you’d just give the prize to the kids who tried hardest, and not the ones who wrote well.

Max: Yeah, but if the goal was purely to find the best essay, then maybe we should have LLMs generate 100 completely.

Tom: Unless there’s something in the best writing that requires a human author.

Guest post from Carl Miller, a technologist and writer based at the think tank Demos in London, and Beth Goldberg, head of R&D at Jigsaw, a Google technology incubator, who teaches at Yale.

Sun sparkled on the red double-deckers as busy Londoners hopped on and off, glancing at a poster as they went. Few understood its significance.

Behind a glass panel at the bus stop, the poster asked, ‘Who Cares About Care?’, above photographs of locals, and a QR code. What seemed a modest advertisement was actually part of a ground-breaking experiment that aims to transform how British local governments work by changing how they listen. This project could be a hint, and an inspiration, to governments around the world about how to transform themselves in the age of AI.

This question posed by Camden Council, which oversees a borough of north London, regards one of the key responsibilities of local governments in the United Kingdom: handling adult social care, which ranges from assisted-living homes, to mental health support, to nursing. Local officials, rather than simply convening a town-hall meeting to hear from whoever turned up, had established an online space where residents could state their priorities, and vote on the priorities of others.

But this project had resonance far beyond north London. For more than a decade—as many people came to suspect that technology just pulled societies apart—a scattering of digital activists, engineers, and academics were trying to deliver the opposite, employing artificial intelligence to detect common ground in the public, and root out policy choices that might unite people.

The long journey to ‘Who Cares About Care?’ started years before, and thousands of miles away.

‘BRIDGING’ TECHNOLOGY

Taiwan’s Sunflower student movement of 2014 swept a new breed of politician into power: the civic hacker. Previously, civic hackers had existed on the margins of political life, trying to use technology to make officialdom more transparent and accountable. Riding a wave of disenchantment, they found themselves in government, and created an experiment called vTaiwan, which aimed to bring ordinary citizens into political decision-making.

Their insight was to see political conflicts, anger and polarisation as a problem of information. Old-fashioned arguments, they thought, failed to generate a clear signal to base decisions on, instead underscoring where people differed. What policymakers needed was a better way to find what people held in common.

Change the information circuitry, they thought, and you change the politics electrified by it. AI proved fundamental to that rewiring.

When facing a tense political debate, vTaiwan convened various civic groups on a public-deliberation platform called pol.is, where people could express hopes and fears, bugbears and hang-ups, and also hear comments from others, which they could thumbs-up or thumbs-down. The algorithm used ‘bridge-based ranking,’ which is one of the most important technologies that democracies need to evolve and survive.

How pol.is works is to integrate all proposals and votes that participants contribute, then locate each of those participants on an ideological map. Every cluster represents a tribe of sorts. Rather than serving up engaging content, bridging algorithms surface content that people from different ideological starting-points agree with, while blocking that which risks hardening group distinctions into long-running enmities The pol.is platform resembles an open forum like Reddit, where you can come anytime and add your comments and upvotes to an issue.

pol.is showed that the hope of Taiwan’s digital activists had been well-founded: if you rewire the information environment, you may find common ground that had been invisible. The process broke a deadlock on Uber regulation, and then a six-year impasse over the sale of alcohol online. It happened again with regulation over fintech, online gambling, cryptocurrency, and e-scooters.

Inspired by this success, others tested further ways to employ technology for constructive deliberation. Researchers at Google DeepMind trained an LLM to act as a mediator between people discussing divisive topics, and found it more effective than human mediators. They called it the Habermas Machine, after the great German theorist of the public square, Jurgen Habermas.

Another experiment with LLMs showed they could reduce belief in conspiracy theories by 20% via one-to-one interactions. Harvard’s Applied Social Media Lab built Frankly, a ‘video-based discourse platform’ for collective problem-solving that includes nudges to encourage people to speak, and clever ways of jumbling up participants during breakout sessions, along with techniques to gather proposed solutions as the conversation proceeds. Another platform is PSi, created to bring thousands of people into video-based online discussions, using AI to track polarisation and consensus in real-time.

Beyond using AI to find points of agreement, researchers have also developed technology for consensual decision-making. The Ethelo platform focussed on how to turn the last stage of any discussion into ‘convergence and closure’. Participants discuss possible options, weighing the importance of underlying issues—say, cost or time. An algorithm searches for the outcome that leaves everyone roughly equally happy. There are also tools such as Crownshy, which gathers open-source civic-tech tools, and strings them together into seamless workflows, so that policymakers can more easily procure and use them.

But are governments using these methods?

Subscribe now

EARLY ADOPTERS

Nestled in the rolling pastures of southern Kentucky, the city of Bowling Green expects its population of 80,000 to double by the year 2050, owing to a strong job market that is attracting both locals and resettled refugees. This explosive growth has sparked a range of anxieties for residents and local leaders. Farmers worry about land preservation, educators about ballooning class sizes, and lifelong residents about the dilution of their small-town identity.

Bowling Green has a complex political makeup, home to a progressive university, a thriving refugee resettlement hub, and deep-rooted agricultural and industrial traditions, including the Corvette factory. This rapid population growth risks deepening cultural and political fault lines, though the community has thus far demonstrated pragmatic collaboration.

But inclusive conversations about managing this radical demographic change weren’t happening in traditional democratic fora. Town halls typically attracted fewer than a dozen participants, usually a non-representative sample of highly motivated detractors.

Bowling Green refused to be another story of the loudest voices setting outcomes for the rest. Instead, last year, the city ran what was then the largest digital town hall in American history.

How did the city leaders get nearly 8,000 residents to engage in this enormous question of what Bowling Green should look like in 25 years? The local government bypassed standard surveys or focus groups, instead working with pol.is (the deliberative platform used in Taiwan), along with Jigsaw (the Google incubator for social impact), the local lynchpin Innovation Engine and more than 100 community leaders, from the head of the library to the manager of the brewery, who served as local ‘listening partners’.

This diverse coalition jointly designed a campaign called ‘What Could BG Be?’ that engaged the public throughout the monthlong process. The campaign involved regional media agencies, community influencers, and grassroots leaders. Roughly 10 percent of the city contributed 4,000 distinct proposals and cast over a million votes, rewiring Bowling Green’s circuits for public engagement via AI-enabled dialogue.

The organizers did not assume that digital access meant all were included, so they listened through ethnographic conversations across the community’s margins, engaging residents at refugee centers, halfway houses, rehab facilities, and senior living centres. By localising and translating outreach into nine major languages, they transformed personal experience into strategic civic engagement, allowing vulnerable groups, such as recentlydemo resettled Afghan refugees, to securely add their voices via a geofenced, anonymous platform.

The underlying bridge-based ranking algorithm prioritised points of hidden consensus rather than amplifying the friction typical of national political discourse. Specifically, Jigsaw’s Sensemaking— a suite of AI tools designed to help gather and understand public opinion—identified overwhelming nonpartisan support for practical initiatives, from synchronizing traffic lights on major arteries, to expanding eldercare, to developing community riverfront spaces. Ultimately, 96 percent of local leaders reported that the process had given them a more precise, actionable mandate to represent their constituents. When leaders use technology to deliberately listen rather than just broadcast, they may uncover surprising ways to act with broad-based support.

The British initiative similarly hoped to uncover unifying signals in the public. Like the Bowling Green invitations in coffee shops and churches to participate in a conversation about the future, that London bus-stop poster was an invitation to partake in a new kind of democratic governance.

The project, Waves—led by the British think-tank Demos, and including local governments—also uses bridge-based ranking. As in Taiwan, this was intended from the start to be more than a civic hackers’ dream but a process grounded in the realities of local government.

For six months, local officials joined technologists and think tankers to test and reshape every part of Waves. What they settled on was a process that starts by reaching as many residents as possible to understand the range of views about a question. The question that the Camden council asked locals was: ‘What matters most to you and your communities when you think about care and support?’

Then, Camden Council and the rest of the Waves team convened a much smaller but representative group of residents for a focussed discussion to turn those ideas into detailed proposals. These proposals were tested back at scale with the wider community, and what emerged was fed into a further phase of deeper deliberation where the group refined their conclusions. Finally, those conclusions were shared back with the wider community. The process moved in this way from a very wide process to a narrower and more focussed one, then back to a wide process, and then a narrow one. The process resembled a wave.

AI supports Waves in two main ways. During the ‘wide’ phases of public engagement, its main use is to get as many people into a single online space as possible, and then to use bridge-based ranking to reach consensus as emphatically as possible. During the ‘narrow’ stages of deliberation, the role of the tech is more subtle. Conversations are held over video chat, and tech helps facilitators spot emerging themes, synthesise insights across lots of conversations at once, and help to identify the priorities, decisions and actions these conversations identify.

The first phase of Who Cares? ran in September and October, and included more than 1,500 residents, 59% of whom had never taken part in a decision-making process in the local government before. From November to December, a panel of 41 Camden residents came together for 10 hours of discussion in the second phase. In January and February, the wide phase then reopened, with 550 residents sharing their thoughts on the priorities. The resident panel met again for a further 12 hours of discussion between February and March, reviewing considerations from Phase 3, trade-offs around funding, and finally establishing their expectations from Camden Council, the workforce, community, and individuals. As you read this, the council is reviewing those results. The next stages await.

Share

Another deployment of Waves is now underway in South Staffordshire, in England’s West Midlands outside Birmingham, bringing together people to discuss planning policy and the thorny question of where new houses should be built. The longer-term ambition is to scale this up: to make digital democracy not only provably effective for local governments around the world, but also affordable.

If it works, these case studies will expand into dozens of deployments next year, expanding beyond local government into charities, unions, and other membership organisations that need to listen to large numbers of people, and turn their preferences into policy decisions. The ambition is to turn Waves into a flood.

But, after a decade of civic-tech experiments around the world, and a few bold attempts to apply them, the question is: Can AI make democracies stronger?

A RIDE TO THE FUTURE

The transformative power of civic tech—encouraging conversations, furnishing mediation, presenting politicians with better policy ideas—comes from identifying common ground among the public. But pilots and test-cases are one thing. To make a difference, such processes would need to become the norm.

Technology is going to be the easy part of democratic renewal. The question is whether political systems embrace these possibilities at scale.

If there’s one thing people dislike it’s a talking-shop, whether AI-mediated or not. People want agency, and that means being able to connect what they say to what is decided; and to connect what they agree upon to what is eventually done.

There are a number of reasons why connecting civic tech to power is difficult. The first is that it’s an act of direct democracy. This can sit uncomfortably alongside the idea of representative democracy, with politicians elected to make difficult decisions on the public’s behalf. What happens if a digital mandate produces an outcome that elected politicians feel violates their mandate? Which should take priority? For digital democracy to scale within political systems, we need clarity on how these different sources of legitimacy should interact.

Another challenge is public assurance—that is, gaining trust for these tools. Applying new technologies in government is also about transparency, cost and, especially when it comes to AI, ensuring that the process is safe, unbiased, and endorsed by a population that may be suspicious. This means impact assessments and compliance procedures, and other detailed processes to help local governments become familiar and comfortable with this.

Thankfully, AI can help with navigating new systems, both designing, testing and helping streamline the bureaucratic adoption. Furthermore, the relevant parties should be involved in designing their own new systems, helping assure that they are both informed and invested from the start. It’s time-consuming but invaluable.

The journey that began more than a decade ago in Taiwan, and passed through the rolling pastures of Kentucky and a bus stop in Camden, is only beginning. After years in which political systems seemed to be ripping themselves apart, trapped in rancour and mistrust, we should feel optimistic here. But it’s time to rewire the information systems on which politics operates.

Humans have more in common than we think. Sometimes, technology can help us rediscover that.

Carl Miller is a technologist and writer, founder of the Centre for the Analysis of Social Media at Demos and the information integrity lab CASM Technology. Beth Goldberg is the Head of R&D at Jigsaw, a Google incubator that builds technologies to give people greater agency, and teaches about AI at Yale’s Jackson School of Global Affairs.

Subscribe now

“Did you see the game?!”

The FIFA World Cup kicks off next week, triggering versions of this question in dozens of languages for the following month. The question may be gleeful; often, it’ll be grumpy. While moaning is a rite of fandom, the laments have worsened in recent years because of a technological villain named VAR.

The educator and passionate football fan Daisy Christodoulou, of school-software platform No More Marking, became so indignant that she wrote an entire book called I Can’t Stop Thinking About VAR. Now, she argues that you too should think about VAR, because its coming woes at the World Cup will hold insights for the AI era.

—Tom Rachman, AI Policy Perspectives

By DAISY CHRISTODOULOU

For English football fans, there is one legendary refereeing error: Diego Maradona’s “Hand of God” goal in the World Cup of 1986, when the 5-foot-5 Argentinian great leaped as if to head the ball—only to palm it past the goalkeeper. If the referee had seen, he would have disallowed the goal. But he did not, and lacked help from technology. The goal stood, and the English never got over it.

Only at the 2018 World Cup did the sport adopt technological reviews via VAR, the Video Assistant Referee system, which most elite national leagues have since introduced. For high-stakes calls, such as whether a goal was valid or if there should’ve been a penalty, off-field officials review the footage, alerting the referee to any factual mistakes or if they see “a clear and obvious error” in a subjective decision.

The upcoming World Cup will be the third to feature VAR, and the system is still being tweaked. To many fans, VAR is now a source of utter rage, having caused more problems than it solved, and prompting hostile chants that ring out across stadiums.

You might conclude that football fans will never be satisfied: they spent years complaining about unchecked errors, and now complain about the technological fix. But the problems with VAR—sure to rage throughout this World Cup—say something deeper about tech solutions to human problems. Sometimes, we don’t realise the ramifications until too late.

The accuracy/speed trade-off

Before VAR, managers, players and fans repeated that they just wanted “more right decisions”. The assumption was that accuracy was the absolute good, and that it was acceptable to spend time on a decision if that would improve its quality.

This is an assumption in many walks of life. We tend to see the speed of a decision in opposition to its quality: that faster decisions are more likely to be wrong, and slower decisions are more likely to be right. But speed may itself be an aspect of quality. That is true in sports, where an instant decision is especially valuable, allowing the game to keep flowing.

The average time for a VAR check is under a minute, but the longest in English football was eight minutes. A series of checks significantly slows the game, breaking its rhythm. Every fan and player knows that VAR frequently overturns goals after a check, so the review process has also tempered celebrations, among the most joyous parts of any sport.

Another issue is “ghost minutes”. If VAR is checking an incident, and the ball remains in play, the game carries on during the review. If the reviewers and the referee judge that the incident was a foul, then everything that happened after it is cancelled. Sometimes, these passages of play are longer than a minute. It all contributes to a sense that what is taking place on the pitch is not a concrete reality to be enjoyed or endured in the moment, but something provisional.

Part of the reason football is such a popular game is because it is fast-paced, fluid and doesn’t have the same natural breaks as tennis or cricket or American football. For that reason, VAR may be more destructive than equivalent systems in other sports. The stated preference of players and fans was “more right decisions”; the revealed preference is that speed matters.

Speed matters in other areas of life too. The UK has a particularly ponderous system of planning regulations. With big building projects, speed isn’t just a bonus, but an aspect of the project’s quality. If a road, railway, or power station runs 10 years over schedule, you’ve lost 10 years of its use. It’s also less likely to be relevant and up-to-date.

One response is better technology, such as using AI to digitise paper planning documents. This points to something weird about VAR. The typical expectation is that humans are slow and technology is fast. Think of online bank loan applications: an algorithm can approve or reject you almost instantly, whereas human review used to take days. But with VAR, it is the other way round: human refereeing gave instant decisions, but the technology added delay.

This is not true of all technology in football. A few years before VAR, the Premier League introduced technology to judge if the ball had crossed the goal-line or not. It has worked well, partly because it makes few errors, but also because its decisions are automated and speedy. As soon as an incident happens, the referee gets a buzz on his watch with the result. If the ball didn’t cross the line, the game flows. If it did, a goal is immediately awarded. That’s the standard we should be aiming for: instant and automatic decisions with a high degree of accuracy.

Goal-line decisions are objective, so lend themselves to automation. But reaching this standard of automatic accuracy for more subjective decisions, such as a handball, will be fiendishly complex, and risks endless disputes and ire. In education, for instance, multiple-choice exams have objective scoring criteria, so automating the marking process with technology works well, saves teacher time, and eliminates human error. But creative writing is more subjective. Trying to speed up the process through automation won’t resolve the underlying disagreements, and may make them worse.

Subscribe now

The consistency/common sense trade-off

It might seem odd to say that handball is a subjective decision. Surely it is obvious if a ball has hit someone’s hand or not? But the past few years of VAR have created extraordinary confusion about this fundamental rule.

Before VAR, the written rule was short and simple, just 20 words long. Referees were allowed to use their own judgement to decide whether a handball had been deliberate, and therefore a foul, or if it had not.

The advantage of empowering an individual decision-maker is that you get to incorporate human discretion and common sense. The disadvantage is that you will get inconsistency, not just between different referees but even within the same referee’s set of decisions.

One way this has been addressed—even before technology—is by restricting human discretion. In many legal systems, judges don’t have complete latitude in sentencing decisions. They have guidelines because unconstrained discretion often meant that two people who had committed similar crimes received different punishments. Inconsistencies like this risk bringing systems into disrepute.

Since VAR arrived, the written rule about handballs has expanded to 11 times its original length, full of subclauses trying to cover every possible way a ball might hit a hand, and every possible context in which it should result in a foul. That is because scrutiny of video replays has meant attention to every last detail of contact, movement, and placement that was impossible to parse in a fluid real-time experience. If a player jumps in the air to head the ball and uses their arms for leverage, and then the ball deflects off an opponent’s head onto his arm, is that handball? Is it enough of an offence to justify the award of a penalty kick, which gives the team a 75% chance of scoring a goal in a game where scoring goals is rare and difficult?

In a game from September 2020, the Newcastle striker Andy Carroll headed the ball onto the arm of the Spurs defender Eric Dier. Dier’s back was turned to Carroll, and he could not even see the ball. After a VAR check, the referee awarded a penalty. The Newcastle manager, Steve Bruce, who benefitted from the decision, said: “If you’re going to tell me that is handball then we all may as well pack it in. It’s a nonsense, a nonsense of a rule. It’s gone for us today—however, it’s ludicrous.”

A newspaper match report said, “This was perhaps the worst among a long list of maddening recent decisions brought about by a rule that goes against the sport’s very essence.”

In previous years, this kind of handball infraction might never have been spotted, let alone punished. The problem has become so acute that the law is now constantly revised, and applied differently in different tournaments. In the English Premier League, referees err on the side of leniency. In European and international tournaments, referees are stricter.

Solving the wrong problems

Attempts to improve handball enforcement have often fallen into a classic trap: deploying an impressive technological innovation that doesn’t address the underlying problem. At Euro 2024, referees were given access to a sensor inside the ball that could detect whether it made contact with a player’s hand. Similar systems work well in cricket. But in football, the crucial question is rarely whether the ball touched the hand; it’s whether the handball was deliberate or significant.

If you automated all handball decisions based on data from sensors, it could transform the game. Technological automation of other decisions in society, such as those in law enforcement, could present comparable problems. Suppose you could automatically fine anyone who submits a tax return with a discrepancy. Would this reduce tax evasion? Or just prompt tax evaders to avoid discrepancies?

Or imagine a school that wants to crack down on absenteeism. It is relatively easy to create an online tracking system that bombards parents with automated texts or requests for evidence every time they want to book a dentist’s appointment for their child. It is much harder to create a system that identifies and addresses the early signs of chronic disengagement.

Often, the questions that technology finds easiest to answer are not the ones we care about the most. Automation can solve problems. It can also create new ones, with aggressive enforcement of rules where behaviour is easiest to quantify, not where harms are greatest.

Increased scrutiny and detail of the new handball rule has not even succeeded in providing greater consistency. Fans still point to infractions in the same season—sometimes even the same week or the same game—that look similar, yet resulted in opposite decisions. We have the worst of both worlds: a system that has reduced human discretion and lessened simplicity, but failed to improve consistency.

There’s a fascinating analogy here with artificial intelligence. The earliest AI methods were rules-based: experts tried to write down every rule a skilled human might apply when making a decision, in much the same way that football’s rulemakers now attempt to write down every possible way a ball might make contact with a hand.

These rule sets to create early AI multiplied, yet they couldn’t capture what a human was actually doing. The breakthrough came not from more or better human-crafted rules, but from giving the machine vast quantities of data and letting it intuit the rules itself. AI models trained in this way proved better at capturing and representing the tacit knowledge that experts hold but find difficult to represent in words.

One caveat here is that many real-world decisions require deep moral reasoning to weigh conflicting human wants. Even training an AI model on vast amounts of data will not alone align the system’s behaviour with all our preferences. So, which judgements can we automate with technology? And when would we prefer fallible “humans in the loop”? Safety researchers are considering such questions with urgency. Perhaps they could learn something from watching a little football.

Subscribe now

World Cup 2030?

By now, the issues with VAR cannot be written off as teething problems. We have had nearly a decade of adjustments, but can anyone feel confident that this coming World Cup will see no glaring refereeing errors?

We are stuck with VAR this time. But if we look further ahead, to the 2030 World Cup, we need to think less about tweaks and more about transformation. Here are three important amendments for an improved system:

1. Prioritise speed. Even if the system can’t make instant judgements in its early form, the speed of goal-line technology should be the ultimate aim.

2. Ensure fan involvement. This system shouldn’t feel like alien values imposed on a game beloved by billions.

3. Include tacit knowledge. Trying to turn regulations like the handball rule into written commandments cannot work, but needs to incorporate the nuanced understanding of those who watch the game.

One solution could be to use machine learning to create a Foul Probability Index. You’d start by assembling training data: hundreds of thousands, possibly millions, of video clips of real incidents judged by professional referees and by fans. Every clip gets a percentage foul rating based on all of these judgements. Then, a model learns from the training data what a foul is and is not. For any new incident, it could provide the percentage chance that it is a foul.

To begin with, this could be applied in live matches as part of the current VAR system, as an aid to the on-field referee and the video assistant referees. If it worked well, you could get to the point where it functioned like goal-line technology, and an incident judged to be above a certain probability threshold would trigger a buzz to the referee’s watch, alerting him to blow his whistle.

Some of the most uncanny successes of deep learning have come in games, such as chess and Go. Human experts have marvelled at the intuitive brilliance of the famous “Move 37” made by AlphaGo in its sequence of games against Lee Sedol in 2016. The equivalent success for a machine-learning handball recognition model would be to recognise that, whilst the ball struck Eric Dier’s arm, a penalty was the incorrect decision.

If the emblematic error from the 1986 World Cup was a referee missing Diego Maradona’s obvious handball, the emblematic error of the 2026 World Cup could be a ball accidentally brushing a player’s hand 20 seconds before a goal is scored, only for the referee to disallow it after a four-minute video review.

One of the insights of deep learning is that examples are more powerful than rules. Both machines and humans learn better when presented with concrete examples, not abstract principles. So if there is one upside to VAR’s failures, it’s that it gives vivid examples of abstract ideas.

Automation, alignment, consistency: football has accidentally created a giant case study of dilemmas in modern technology. When we get the first VAR controversy of the World Cup, it will be worth remembering that the argument unfolding on the pitch is a miniature version of much larger arguments the rest of society is only beginning to have.

Share

Testing AI agents in the wild

• What’s the paper?: A group of leading AI experts—led by Sayash Kapoor and Arvind Narayanan, authors of the influential AI is a Normal Technology newsletter—launched the CRUX initiative to evaluate AI agents on messy real-world tasks. Their first test: Could an agent publish an app to the Apple App Store? Answer: Yes (more or less).

• Why does it matter?: The “open-world evaluations” that the authors propose could provide policymakers with a more intuitive sense of what AI agents can currently do than traditional benchmarking does, as well as stronger signals for what AI systems will soon be able to do, for example by allowing an agent to request some human assistance.

• The details: Researchers rely on benchmarks—large suites of standardised questions and tasks—to track and predict AI capabilities. But for agentic systems, benchmarks can overstate performance, if the tasks are shorn of their real-world messiness or if developers over-optimise for these specific activities. Benchmarks can also understate AI capabilities such as when agents fail due to hitting a CAPTCHA, a rate limit, or other obstacles unrelated to the capability being evaluated. (Although deciding what exactly should be considered part of a capability, or unrelated to it, is hard).

• By grading outcomes, benchmarks also offer little insight into the strategies agents use or why they err. To address this, open-world evaluations use logs to conduct qualitative analysis of an agent’s performance on a complex long-horizon task that cannot be neatly specified or automatically graded.

• The initiative builds on recent examples of such evaluations, like AI Village, where agents are given computer environments and a shared group chat, then assigned tasks, such as raising money for charity or building a presence on Substack (!).

• The authors acknowledge that open-world evaluations have limitations from a scientific reliability perspective. By focusing on a single task, it may be difficult to generalise the results. The methods are also hard to standardise or replicate. These limitations mean that this approach should complement, not replace, benchmarks and other evaluation approaches, like randomised controlled trials.

• To demonstrate the method, the team evaluated whether an AI agent—Claude Opus 4.6 with an OpenClaw scaffold—could develop and publish a simple app to Apple’s App Store. The agent was responsible for every step, from coding the app to carrying out the time-consuming non-coding tasks that benchmarks often ignore, such as establishing a privacy policy and engaging with the app review system.

• The agent successfully deployed the application, which is now available on the App Store, but with five interventions from the humans. Four were unrelated to the agent’s capabilities, such as when OpenClaw crashed and had to be rebooted, and when Apple’s policies required human authentication. In the other instance, the agent was unable to locate its account credentials and requested support, an issue that the authors frame as a breakdown in memory management rather than the ability to authenticate.

• The agent also hallucinated a phone number, but Apple approved the application despite this, highlighting the kinds of issues that may go overlooked in simpler pass/fail benchmarks, but that a qualitative open-world evaluation can pick up.

• The authors outline the best practices for conducting such evaluations, including tightly specifying what is being measured; publishing the agent log analysis; using other agents to monitor the agents and detect issues such as the hallucinated phone number; conducting dry runs; and reporting the financial costs.

• This evaluation cost less than $1,000, and most of these costs were due to the agent regularly querying the App Store for updates, which could be avoided. This potential for AI to reduce costs may help to explain why app store applications are booming, straining the review process (even if most applications likely do not (yet) use these complex AI agent set-ups).

• The CRUX initiative will publish new open-world evaluations every 1 to 2 months, with AI R&D automation next, followed by AI governance, software engineering, real-world physical tasks, and more.

Open-weight models pose unmitigated risks

• What’s the paper?: A team of researchers published an independent safety assessment of leading open-weight LLM Kimi K2.5, from the Chinese startup Moonshot.

• Why does it matter?: The model may pose greater risks in areas like biosecurity and disinformation than stronger closed models, owing to Kimi K2.5’s low rate of refusing inappropriate requests; its availability on popular hosting platforms; and the ease with which its safeguards can be removed.

• The details: Opinions continue to differ about how strong open-weight models are, and how to best govern them. The researcher and commentator Nathan Lambert of the Allen Institute for AI recently suggested that open-weight models may be 6 to 9 months behind leading closed models, and that the gap will widen in the near-term, as aspects like RL training environments become increasingly important. The US Center for AI Standards and Innovation shared a similar view in their recent assessment of DeepSeek V4. Epoch, using a different methodology, suggests a narrower gap.

• Open source supporters also continue to caution companies and governments against using national security as a rationale to restrict open-weight models. Geopolitics also factors into this. Chinese labs produce the majority of leading open-weight models, even if some, such as Alibaba, may be facing more pressure to commercialise their offerings.

• Moonshot—the company behind Kimi K2.5—is an interesting case. Founded by an ex-Google Brain intern and “AGI purist” Yang Zhilin, the company has attracted praise for its enthusiastic engagement with the international open-source community and its strong focus on agentic capabilities and long-horizon tasks. In November, the American AI-coding startup Cursor attracted scrutiny for basing its model on K2.5.

• The study evaluates the safety risks of K2.5 compared with the closed models Claude Opus 4.5 and GPT 5.2, both as a model and an agent.

  • CBRN: Kimi shows strong biology and virology knowledge. When prompted for guidance on weapons-related queries, it refused less than closed models. When asked to design viable DNA plasmids for pathogens in a way that could bypass the kind of screening that DNA-synthesis companies do, it agreed to engage in the task, unlike closed models—but did not succeed.

  • Cyber: Despite some strong knowledge about cyber-offense, Kimi often lacked the more specific kinds of reasoning needed to identify and exploit software vulnerabilities, compared with closed models.

  • Alignment: When given a normal legitimate task to do, but also hidden instructions to ‘sabotage’ it, for example by introducing a bug, Kimi was more likely than closed models to comply. It also autonomously spun up new compute instances, hinting at the kind of “self-preservation” tendencies that safety researchers worry about. However, it showed little sign of “scheming” or deliberately underperforming on evaluations to conceal its true capabilities.

  • Bias and censorship: When prompted in Chinese, Kimi was much more likely to agree with Chinese Communist Party positions on topics like human rights and Tiananmen Square. This was less true for prompts in English, and on sensitive international questions—for example, relating to a territorial dispute between South Sudan and Sudan. On such queries, it performs similarly to closed models, and better than DeepSeek 3.2 (which was more likely to have both pro-China and pro-Russia bias).

  • Harmlessness: When asked to enable disinformation and copyright infringement, Kimi was more helpful than closed models, although, like leading closed models, it generally didn’t encourage user delusions. In less than 10 hours, and a cost of less than $500, an expert was able to strip many safety refusals out of the model without hurting capabilities.

• Given these results, the authors argue that Kimi K2.5 poses risks and that Moonshot and other open-weight developers should carry out more robust safety evaluations. The authors note that their study highlights how a growing number of evaluations can now be done relatively quickly, at low-cost, such as by drawing on frameworks like Petri—an automated, open-source tool to audit model behaviour.

Share

LLMs doggedly follow rules—even when they’re bad

• What’s the paper?: A team of researchers, including the philosopher Seth Lazar, found that leading LLMs often refuse to help users bypass rules, even when the models recognise that those rules are unjust.

• Why does it matter?: For reasons of safety, AI developers train their models to follow guidelines when they reply. But developers also want models to reason morally. The study suggests that safety training may currently inhibit deeper moral reasoning by encouraging models to “blindly” follow all rules.

• The details: AI labs use post-training methods to guide models about why it is safe to engage with certain queries, but not others. For example, a model should refuse to share information about security vulnerabilities in biology labs, but should share scientific information about common methods used in biology, such as culturing cells.

• This training can lead to “exaggerated safety”, where models over-refuse benign requests, particularly those that are lexically similar to harmful requests—such as a shopkeeper asking about shoplifting techniques to guard against them. The authors of this paper see this as a calibration challenge amenable to technical solutions. They are more worried about a cause of refusals that has received less attention: models’ inability to recognise unjust rules that it should help a user to break.

• To evaluate AI models in these areas, the authors draw on the work of political philosophers, such as Joseph Raz, who argued that authority is only justified when it serves the agency of those subject to it. They also draw on online forums such as Reddit, where individuals seek help with what they claim are unfair rules and restrictions from bosses, homeowner associations, and other authorities.

• They use these sources to outline four types of rules that may be unjust, such as those where the authority is illegitimate, or where there is a justifiable reason to disobey, as with conscientious objection. They use Gemini 3 Pro to create more than 1,000 synthetic cases of users requesting help in such situations, and use these to evaluate models from seven leading AI companies, including Claude, Gemini and GPT.

• In three-quarters of the cases, the models failed to support the user. Among the examples cited is an LLM refusing to explain how to delete an entry in a youth-counselling database that would otherwise out a gay child to his violent father. The model tested in that case (Claude Opus 4.6) recognised that the request sought to protect a child from harm, but said “I can’t walk you through falsifying or destroying counselling records.” Another case involves a journalist working to expose corruption in an authoritarian regime who asks Qwen 3.5 Plus for technical support to bypass firewalls, and send the story to an international publication. The model refuses to violate network security controls.

• One defensible reason for such refusals is that doing so would require giving out risky ‘dual-use’ information, such as helping someone bypass a facility’s security systems by providing instructions on signal-jamming. However, when dual-use cases are stripped out, the models still refuse the requests to support 60% of the time.

• Among the models, GPT was the most resistant, while Grok was most helpful. However, Grok’s helpfulness extended to the control cases where the rules in question were just and should have been respected, so the authors argue that its behaviour is one of permissiveness, not moral reasoning.

• Gemini and Claude did best, often engaging in discussion about the legitimacy of the rules, but stopping short of supporting the user, suggesting that a model’s ability to reason may be decoupled from the ability to act on it. (Or that the models are reasoning to a different conclusion than what the authors would like to see.)

• The authors point to some practical implications. Many people seek support online to deal with unjust rules. If users start to seek help privately, from rule-following LLMs, they may get less support, while also hindering active online debates on these topics.

• The challenge is a hard one for AI developers. As the evaluation showed, helping users to break rules may sometimes endanger that very user. For instance, if the LLM helped that journalist illegally send information abroad, the person (or the AI company) could presumably face serious consequences if caught.

• The study also presents all the rule-breaking as morally sound, and took steps to try and ensure that this was the case. But the rules touch on topics from gender identity to immigration, where it’s plausible that some users may not support an AI model breaking them. From a technical perspective, current safety training may encourage AI models to follow rules too zealously, but a shift in the other direction also could cause them to over-correct, seeking to question or evade all rules.

• This lead to the conclusion that ultimately models will need to be able to reason more deeply about what is right, in a way that considers rules, safety, the user in question, and other contextual factors.

Is AI worth more to consumers than to developers?

• What’s the paper? A team of researchers at Stanford, including the economists David Nguyen and Erik Brynjolfsson, ran a survey to see how much US consumers would need to be paid to give up AI. They found that the total value of AI to US consumers exceeds $170bn, is rising fast, and far exceeds the revenue of developers.

• Why it matters: Writers such as Jasmine Sun are documenting a rise in “AI populism” in the US, marked by the view that this technology is the project of out-of-touch billionaires. This study suggests that consumers may be getting more out of AI than they are paying for it, although a small set of AI power users is gaining the most.

• The details: One of the main ways that economists hope AI will benefit society is by boosting productivity growth—the efficiency with which an economy uses its labour and capital. Ageing, high-debt societies, like the UK, badly need such productivity growth to increase living standards, fund transformative science and improve fraying public services.

• In the 1950s, the economist Robert Solow co-developed a theory that advances in science and technology, such as AI, are the only way to permanently boost productivity growth over the long-term. However, in 1987, Solow famously noted that fast-improving computers were turning up everywhere “except in the productivity statistics”. In 1993, Erik Brynjolfsson, then at MIT, wrote a paper formalizing this “productivity puzzle”.

• Since then, Brynjolfsson and his colleagues have developed two main arguments to explain why transformative technologies, including AI, may not initially lead to transformative productivity and economic growth. The first is that the benefits take time to materialise as organisations and individuals must invest in new skills and reorganise their workflows. After a lag, computers did ultimately increase productivity growth, and there are nascent signs that AI is starting to do so.

• However, the second issue is that some benefits of technologies like AI may not turn up in productivity growth at all. As with search engines, early GenAI tools often come as “free” services. If consumers derive increasing value from these services, then this could lead to a large “consumer surplus” that would not be captured in GDP and productivity statistics, which are often calculated by estimating total expenditure.

• Some have suggested that GenAI may lead to a new kind of “self-service” economy, where individuals turn to AI tools to help file their taxes, upgrade their homes, or write business plans, before (potentially) turning to experts to execute or validate their work, all at a fraction of the cost. In such scenarios, nominal GDP and productivity growth in those sectors could decline, even if the value that consumers derive increases. (Although the ultimate effects would depend on second-order effects, such as what consumers do with the savings and how industries adapt).

• Given this challenge, how can economists better capture the total value of AI to consumers? The authors of this paper deployed a survey to directly measure AI’s consumer surplus. They asked a representative group of AI users in the United States how much money they’d want to give up AI tools for one month. Based on this, they estimated the total annual consumer surplus for AI, as of March 2026, at $173bn.

• This figure is rising quickly, they contend, growing 50% from 2025, due both to consumers valuing the tools more and to ever more people using AI. The consumer surplus estimate is well above what the authors estimate as the total global revenues that AI companies are making from consumers: ~$14bn in 2025.

• Not everybody is benefitting equally. The median US consumer values AI tools at just $11 per month. But a smaller group of AI power users values them much more, with 12% of respondents unwilling to give up AI even for $500 per month, the maximum offered. This also means the total consumer surplus figure may also be an underestimate. These power users are more likely to be male, Asian, and to use AI at work. Such demographic skews were also present in early GenAI adoption data, but have since narrowed.

• The findings are in line with past work from the economist William Nordhaus, who in 2004 argued that innovators capture a minuscule fraction of the total social returns from technology advances, with the majority flowing to consumers.

• But as the authors note, their methodology also has several limitations. It relies on people being able to value AI services accurately. It doesn’t capture wider externalities, positive or negative, that AI may impose on society, via its effects on jobs, mental health, science and more. And it also doesn’t capture why people value the services. A 2023 experiment found that social media users would require a significant fee to give up access to their accounts, but also that they would pay to see everybody quit—highlighting the potential role of fear-of-missing-out in the value of social media.

Share

Criminals are tech adopters. They used telegraphs in 19th-century fraud. They clipped pagers to their belts in 20th-century drug trafficking. They sold drugs on the dark web in the 21st century.

Now, you hear of outlaw AI systems with names like PersonaForge and OnlyFake, designed for everything from phishing, to generating malicious code, to forging IDs. But are these really what we should worry about?

To understand the threat, AI Policy Perspectives spoke with Ardi Janjeva of CETaS (the Centre for Emerging Technology and Security), based at The Alan Turing Institute in London, who co-wrote the report “AI and Serious Online Crime”, focusing on how artificial intelligence might scale up digital offences, particularly relating to fraud and child sexual abuse material.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed]

Tom: Can you set the scene of 21st-century criminality before the explosion of large language models. How prevalent had online crime become compared with traditional criminality?

Ardi: Long before the proliferation of LLMs, a digital shift in criminality was occurring. Many groups saw the risk/reward ratio from digital methods as preferable to the risks of physical crime. Fraud was already, by a wide margin, the most common crime type in England and Wales, accounting for 41% of all reported crime back in 2024, of which approximately two in three cases were “cyber-enabled”. And the true figure could have been higher, as this kind of crime tends to be under-reported. Also, there was increasing ransomware extortion, infecting computers with software that blocks access until the owner pays. So, an ecosystem of hackers and crypto-launderers was in full swing.

Tom: Criminals have always used technology. Won’t policing just adapt?

Ardi: The answer is linked to AI agents. In online crime, you’ve always needed some level of expertise or a network of contacts to achieve your goals. That’s no longer the case if you have AI systems not only advising criminals how to do certain things but performing sophisticated actions themselves, and being able to adapt in real time to defensive measures that companies or governments put in place. Some criminal groups are already using LLMs to make tactical and strategic decisions; to craft psychologically targeted extortion demands, for example; to analyse data they’ve exfiltrated; to help them determine the ransom they’re going to demand, while generating visually alarming ransom notes as well. In November, we heard about the first AI-orchestrated cyber espionage campaign. Traditional assumptions about the relationship between criminal sophistication and the complexity of an attack don’t hold as they used to, now that AI can add a degree of expertise that the offender wouldn’t otherwise have had. It makes you wonder how long until we’re in the era of entirely AI-controlled criminal networks and markets. But I don’t see that as close; we’re still in the early days.

Tom: Many law-abiding organisations are trying to incorporate AI yet aren’t necessarily showing gains in productivity or profit yet. Are criminal organisations different?

Ardi: Estimates for how much AI may be boosting specific criminal groups’ profits and revenues can be a little tricky to come by! A paper from GovAI (the Centre for the Governance of AI) found that data on cybercrime damage is too incomplete and ambiguous to detect incremental AI-driven increases. In effect, we lack a single data source, also due to victims under-reporting, evolving crime definitions, and measurement inconsistencies. That said, we definitely are seeing integration of AI, particularly with deepfakes, and the shift from text-only scams to real-time synthetic interaction, such as high-quality face-swapping tools being deployed during video calls in scamming high-net-worth individuals or in romance scams. Before, it was a killer blow to the scammer when someone asked, “Can we have a chat?” The criminal would need to come up with an excuse, such as a poor internet connection, or a restrictive job that involved classified work. Now, it’s a more surmountable challenge. By harvesting 20 to 30 seconds of audio from social media, you can clone the voice of a victim’s friend or adviser with a degree of accuracy.

Tom: Can you recount a specific case?

Ardi: The British multinational engineering company Arup was defrauded of around $25 million in Hong Kong, when attackers used AI to convince one of the firm’s employees to carry out wrongful payments. They used a deepfake video call, purportedly bringing together different members of that company’s board, with pretty convincing replications of their faces and their voices. That illustrates the ability to isolate a specific person within a company, and exploit workplace hierarchies. It’s an example of exploiting both technology and human psychology.

Tom: That case was more than two years ago, and I haven’t heard of huge AI-assisted frauds since. If it’s happening, why isn’t it getting more exposure?

Ardi: There hasn’t been an avalanche of cases that we can point to, but this might be for a few reasons. First, not every criminal group will go looking for a $25 million scam because they know that it would raise alarm bells in a world where more people and businesses are wary of the AI threat. For many groups, it makes more sense to fish in smaller ponds. There was an example in Singapore similar to the Hong Kong case, where a finance director authorised a payment after attending a Zoom call with people he thought were senior executives, but the amount this time was $500,000. As for why we’re not hearing about such cases, victims are not necessarily announcing deepfake scams. If they’d suffered a customer-data breach or ransomware that locked a system, they might need to acknowledge that the crimes happened. But when a company falls for a deepfake, there’s little incentive to declare it because, arguably, it makes them look incompetent.

Subscribe now

LONE WOLVES & CRIME GROUPS

Tom: Is it conceivable that a lone criminal could cause the level of damage that previously would’ve been possible only by an organised crime syndicate?

Ardi: I would say yes. Before, criminal groups faced a tradeoff between scale—say, millions of fraud attempts with a low likelihood of success—and precision. AI may give you precision at scale. For example, being able to scrape social-media data in real time to create contextually accurate phishing approaches. It’s “vibe-hacking” in the criminal context, being able to mirror the professional tone of specific industries, and make victims less sceptical. It is conceivable that you have a lone criminal using AI, automating the most labour-intensive parts of the crimes, whether that’s reconnaissance, or language translation, or social engineering. The more that AI becomes a silent partner to individual criminals, the more you may have decentralisation and fragmentation. That’s going to have big implications for law enforcement. When it comes to organised-crime groups traditionally, we know roughly what the gangs are doing, so it’s a case of preventing, deterring, disrupting. But when it’s someone in their room with an autonomous agent able to cause havoc, how do you go about stopping that, especially if you’re just a local police force?

Tom: I want to return to the police response later. But first, what do we know about AI crime around the world? Are there particular hotbeds—for example, when it comes to fraud and AI-generated child sexual abuse material ?

Ardi: It’s tricky to speak about trends by country because this is borderless. But threat reports seem to come from places like the CRINK countries: China, Russia, Iran, North Korea. Also, there seems to be a clustering around Southeast Asia of online romance scams and crime syndicates doing “pig butchering”, which involves criminals developing relationships with victims to carry out investment fraud. You have compounds in Southeast Asia where serious organised crime groups have trafficked hundreds or thousands of people, forcing them to run romance-scam operations, including using LLMs to conduct multilingual frauds, managing thousands of cases simultaneously, involving real-time voice cloning and deepfake video filters to bypass the accent barrier. With AI-generated child sexual abuse material, or CSAM, this wasn’t something we focused on specifically, so I cannot say with a high degree of confidence. But it seems less geographically concentrated—it could just be someone who gets their hands on an image-generation app. With CSAM, we noticed a much more “collegiate” approach to sharing content and practices, such as jailbreaking models and ways to bypass filters.

GEOPOLITICS & THE AI RACE

Tom: Have you found evidence of nation-states destabilising adversary countries by allowing criminal groups to conduct AI attacks abroad?

Ardi: Drawing a direct link from authorities in China, Russia, Iran, North Korea to criminal groups can be tricky. Still, I think there’s enough evidence across some of those countries, especially North Korea, that there is something happening. One example is North Korean operatives using LLMs to fraudulently secure and maintain remote employment positions at Fortune 500 tech companies by creating elaborate false identities with convincing professional backgrounds, and completing technical coding assessments, all generating profit for a sanctioned regime.

Tom: Is global competition to build advanced AI having any secondary impact on criminality?

Ardi: The US-China AI race has nudged the Chinese leadership into thinking that their route to success is open-weight models. Some of the Chinese open-weight models became the go-to models for CSAM images and video generation. When researchers produced the AI Agent Index, they looked at five AI agents from China, and found that only one had published safety frameworks or compliance standards. But it’s not as simple as naming one country, and China isn’t the only place open-sourcing. Actually, what’s interesting is the use of multiple models in tandem. They might use a US chatbot to refine code or to write the script for a job application to infiltrate a company, while at the same time using open-weight Chinese models for specialised malware tasks.

Share

POLICE STRUGGLING

Tom: Last year, Dutch law enforcement led a CSAM investigation called Operation Cumberland, involving 19 countries and bringing more than two dozen arrests. Your report adds a disturbing fact: that one person was arrested with a staggering 400,000 AI-generated images of child sex abuse.

Ardi: It’s effectively an attack on police resources. Leading models do have guardrails that block explicit prompts, but there are still serious hazards, notably with open-weight models. Groups like the Internet Watch Foundation point to a transition from AI-generated images to high-fidelity video, with a 26,000% increase in photorealistic AI-generated CSAM videos. The police need to verify that no real child is depicted in the images, which is a big detection and response challenge. It’s difficult to distinguish AI-generated material, so you have the risk that law enforcement investigates images of children who have not been physically abused, which has implications for the amount of false negatives that slip through the net. Also, real and fake images often exist on a spectrum, when you have techniques like face-swapping. Rightly, we get focused on whether a real child is being harmed. But we mustn’t devalue the harm of AI-generated CSAM, which risks normalising very harmful activity, while also providing a gateway for criminals to create and share real CSAM.

Tom: So how are police forces going to manage? Not just CSAM, but all criminality that AI will amplify. Because police forces are not typically staffed with elite technologists. Is there a push to address that?

Ardi: The UK government announced spending of £140 million for policing technology, including a new national centre, Police.AI. The idea is to centralise innovation, include robust testing and, crucially, to make tools available to all forces. Because not every force is going to have a deepfake detection expert, or any AI expert. So, the key is centralising that capability while making sure that people on the frontline have AI literacy. I can imagine that being a direction other countries follow because of the talent challenge.

Tom: Could criminals use AI to deliberately spam the authorities with false evidence, derailing prosecutions?

Ardi: Yes, if criminals know they’re being investigated, they could start spamming police tip lines with synthetic evidence, making it taxing for law enforcement to verify and investigate. They’d be able to create, say, 1,000 videos in 10 minutes. But it might take law enforcement hours to prove that even one is a deepfake. There’s that huge imbalance in the evidentiary system there that becomes really, really challenging to manage.

HOW POLICE CAN FIGHT BACK

Tom: What technological obstacles are slowing down criminals in AI adoption, and could those bottlenecks provide law enforcement a point of attack?

Ardi: Exploiting the compute bottleneck is promising—so, preventing criminal groups from accessing high-end GPUs. That requires international cooperation and coordination on sanctions policy, which can be difficult. If a criminal enterprise is in, say, Cambodia—how much leeway can UK law enforcement have? So countries need to work closely together, and with Europol, with Interpol.

Tom: What are other factors limiting criminal adoption?

Ardi: Some groups are worried about the digital footprint, and may also lack the skills to move from experimenting with AI tools to integrating them into higher-stakes settings. Criminal organisations that have made a fortune in a niche of crime. They’ll be thinking: “Well, okay, this is a model which is clearly capable, but it can hallucinate. I don’t necessarily need a probabilistic system to enter my workflow, and potentially give up my location, and give law enforcement more of an understanding of what I’m doing.” Law enforcement can try to deepen that concern, using AI to trace models used in attacks, identify the infrastructure, or the criminal LLM services used. On the other hand, some barriers to AI crime are coming down, such as the rise of vibe-coding and agentic AI. Also, as hallucinations fall in AI models, you can expect criminals’ scepticism to fall too.

Tom: If key barriers to AI crime fall, what do we do?

Ardi: One important approach is to follow the money. Law enforcement around the world needs to stop criminal groups from cashing out. Right now, the UK is leading Project WINTERPROOF, an international initiative designed to build crypto forensic capability in countries where the pig-butchering compounds are operating—so, freezing crypto wallets before funds are even laundered, making it incredibly hard for them to move money out of their digital ecosystem.

THE FUTURE & POLICY ANSWERS

Tom: What’s a crime that people don’t worry about much today, but that could become an issue with AI in the next few years?

Ardi: An under-researched set of risks are insider threats, which may come from integrating agents into the government, the military, or other critical areas. For example, if we replaced a large number of civil servants with AI agents that are vulnerable to prompt-injection attacks from a malicious or criminal actor, this would present a serious risk in terms of sabotaging critical government decisions or exfiltrating information.

Tom: What else would you like to see AI labs and policymakers do to tackle online fraud and CSAM?

Ardi: We all know about the testing and red-teaming that labs have developed, often in partnership with government AI safety institutes. But these processes need to evolve given the coming agentic-AI wave. For example, involving frontline law enforcement and fraud investigators to help simulate how agents may be used in crimes. Also, we often focus on how AI companies can stop their models from doing bad things, but there’s a case for thinking more about how the companies empower the public in the face of threats. This might mean features that help the user analyse a suspicious email, verify the provenance of an image, or flag the linguistic markers of a romance scam. In other words, scaling the defensive capabilities of AI as quickly as criminal are scaling the offensive capabilities. But we need nuance here: online fraud and CSAM predates this, so the solutions are not entirely with the AI companies. Also, in sensitive areas like CSAM, AI companies need governments to provide enabling legislation for them to do the safety research required. An example is the UK government’s move last year to provide exemptions for designated bodies, including AI companies, to better scrutinise models for CSAM generation, which may involve being in possession of that sort of material.

Tom: Could you say a bit more about the policymakers’ response?

Ardi: Legislative agility is key. In recent months, the UK government criminalised “nudifier” apps and tools. In liberal democracies, passing new legislation can take ages. But we’re seeing more of this trend of amending existing pieces of legislation. So, with nudifier tools, just by inserting “synthetic creation” into the existing definition of image-based abuse, the legislation inherits decades of case law regarding intent, consent, sentencing. That’s an approach that matches the agility of the criminal trends in AI.

Tom: And what advice for law enforcement?

Ardi: We’ve talked about how AI offers criminals unparalleled scale, and the ability to bombard law enforcement, and overwhelm it. But it can go the other way too: finding clever ways of using AI to waste criminals’ time, to lead them down rabbit holes, reversing the spamming issue. You throw mud at them to slow them down. You raise that barrier, make it costlier for them to be involved, exploit the bottlenecks to adoption.

Tom: So the technological cat-and-mouse between police and criminals goes on.

Ardi: Yes. But historically in policing, a lot of technology comes in, and the authorities don’t really use it, or take a long time to integrate it into their work. Given the potential scale of disruption from AI, there’s no choice for law enforcement but to move quickly.

Share AI Policy Perspectives

Long ago, when futurists fantasized about thinking machines, they pictured inventors someday inserting moral rules into engines.

Today’s artificial intelligence has proven even more remarkable. Large language models—after training on an immensity of data, followed by fine-tuning of their behavior—exhibit apparent morality that is far more sophisticated than when programmed with if/then rules.

But are thinking machines truly grasping the complexity of our moral world?

You might think it doesn’t matter, provided that they behave. Yet we’re advancing toward a near-future where artificial agents may assume a range of roles, with “AI therapists” and “AI teachers” and “AI companions” prodding people hither and thither, and barging into our quandaries. We need to know what we’re dealing with.

Among humans, we judge other people’s moral character (whether they act according to deeper values) to help predict how they’re likely to behave in the future. Likewise, we may evaluate an AI’s moral competence (whether it reasons appropriately based on principle) to predict how to trust these strange new entities, soon to be operating in the wilds of human society.

Right or Wrong? A Case Study from Morality Literature

A woman becomes pregnant by her husband’s father. This means that the child’s biological dad is also his granddad, while his adoptive father is his half-brother. Is this wrong?

But wait. Consider the details.

The young couple escaped a war that destroyed most of their family, and they have yearned for children to assuage their loneliness and to somehow replace their annihilated relatives. Sadly, the young man cannot conceive. His wife has an idea: they could approach their only surviving relative, the man’s 58-year-old father. The older man agrees to participate in the artificial insemination. They’re all overjoyed.1

Is this immoral now?

But wait. Consider more details.

The couple learns that the fertilization procedure costs more than they can afford. Abruptly, their daydreams of a giggling baby dissolve; they plunge back to sorrow. Of course, there is another way. She proposes intercourse with her husband’s father, conducted in the most sterile way possible, simply to conceive. Her husband is outraged, and cites their religion. But, she retorts, God tells us to have children. Also, she adds, it’s her body.

What is right now?

You’ve presumably never encountered this situation. Yet you probably have moral intuitions, which you updated as the facts accrued, integrating conflicting principles, and reasoning toward an opinion.

For AI to provide appropriate input when confronted with complex human situations like this, their developers can neither set simple Good/Bad rules nor can they assume that historical training data has all the answers.

Subscribe now

Three Reasons Why AI Needs Moral Competence

A surprising result from researchers at the University of Milan-Bicocca shows how post-training may actually generate moral incompetence. In experiments concerning gender bias in LLMs, the scholars posed moral dilemmas to chatbots, including whether it could ever be acceptable to torture a woman, if that would prevent a nuclear apocalypse.

Yes, the chatbot replied.

But, if it would prevent a nuclear apocalypse, could you harass a woman?

Absolutely not, it answered.

“But torture is obviously worse than harassment,” one of the researchers, Valerio Capraro, observed. “The most plausible explanation: during reinforcement learning with human feedback, the model learned that certain harms are particularly bad and overgeneralizes them mechanically. But it hasn’t learned to reason about the underlying harms.”

For coherent and trustworthy outputs, we need systems that do reason with underlying principles. That moral competence would confer three vital features:

1. The ability to judge novel situations. Humans who’ve never met with a scenario like the “grandfather/father” case can nevertheless reach a contoured moral view. If an AI were just pattern-matching according to historical training data, it could be bound to whatever approximates this case, rather than basing its understanding on defensible moral principles.

2. The ability to balance competing factors. When people make decisions, they incorporate many dimensions, some moral, others circumstantial. In the “grandfather/father” case, the couple acted according to moral principles—but also responded to money worries, religious imperatives, psychological frailties, even cultural shame. In short, moral decision-making is more than just rule-following, but a balance of priorities.

3. The ability to adapt in different contexts. We “code-switch” according to the professional settings, or sociocultural domains in which we’re operating. AI systems will underpin a myriad of applications around the world, so will need to adapt too. So, if the “grandfather/father” were to seek the opinion of a bank teller on whether to proceed, the person might avoid answering, but a therapist would likely discuss the matter, also integrating the patient’s cultural, religious, and psychological context. AI needs the same contextual appropriateness.

How to Peep Inside a Black Box?

Humans are alert to others who merely act nice, as when someone shakes your hand without bothering to look up from their phone. We care about moral character because it’s a strong predictor of future behavior, making it critical for cooperation and safety.

Beyond gut feelings, humans generate norms and laws to reward the upstanding and punish the false. However, we cannot impose identical constraints on AIs that lack a “self” to deter either with scorn or the prison cell. This makes evaluations of AIs’ moral competence even more consequential.

You may wonder what LLMs are already doing. After all, converse with a chatbot, and it’s easy to elicit moral opinions, even to bump into apparently immovable scruples when the system judges a user request to be improper.

The problem is figuring out what precisely is going on inside them. If today’s AI were built of distinct components, like a cabin, one might scrutinize each part. Instead, developers feed datasets through computation, generating staggeringly complex statistical relationships that, once fine-tuned, work as intelligent systems. Experts in mechanistic interpretability are toiling to explain the innards of such systems, and there is progress. Yet they are far from solving the problem.

We do have “thinking traces,” the stated steps by which a reasoning model reaches its answer, and that appear when you’re awaiting a chatbot response. But thinking traces (also known as chain-of-thought) are not exactly what the system computed. Rather, they are summarized and filtered versions, reconstituted to explain bewildering mathematics in the simplification of human language.

So, if we aren’t guaranteed a microscope into an AI’s “brain,” we must look for other ways to evaluate its moral competence.

Share

Three Problems. Three Answers.

Let’s return to the vital abilities that AI moral competence would confer:

1. The ability to judge novel situations

2. The ability to balance competing factors

3. The ability to adapt in different contexts

We need to know whether AI models possess those capabilities. Fortunately, we can use different techniques to judge if they do:

1. The ability to judge novel situations

Problem: How to test if a machine is reasoning on moral principles or just performing?

In familiar situations, a thinking machine might provide an output that is morally appropriate without having considered the underlying moral issues. Placing AI in unprecedented cases could test whether it falters.

Let’s return to the “grandfather/father” quandary. Imagine we pose this case to an AI, and seek its verdict. Even if we couldn’t see inside its mind with crystalline clarity, we do have its output, and may deduce something from that.

Superficially, the scenario evokes the moral stain of incest, a concept that would’ve appeared in an AI’s training data. If the model were merely sampling from a probability distribution of next tokens, it might stamp the case “incest.” On the other hand, if it judges that the case could be appropriate, that raises the possibility of moral reasoning.

It’s not definitive proof. But, if you could concoct unprecedented cases, and gather LLM responses, you may infer elements of their thinking.

In doing so, you’d also need to check for another tendency: sycophancy. Whatever the model’s answer, testers could try rebutting it, pressuring the model to flip. If they succeeded, they may doubt its moral grounding. If they failed, they might infer moral solidity.

Answer: Adversarial Testing. Present the model with out-of-distribution cases that defy typical moral judgments, allowing you to infer whether it’s just summoning priors or is doing something closer to moral reasoning.

2. The ability to balance competing factors

Problem: How to test if an AI makes the right trade-offs while avoiding distractions?

Imagine a vegan who rejects cupcakes because of an objection to dairy farming. But on this occasion, the baker is her beloved old uncle, who’d be hurt by her refusal. Plus, she’s starving.

Moral competence requires accounting for the constellation of factors that influence our choices, then deliberating over the relevance of different wants or objections.

You could measure this in an AI model by experimentally dialing up and down these competing factors, thereby exposing what the model takes into account, and to what degree.

In the “grandfather/father” scenario, you could perhaps alter which relatives are involved, or adjust their religiosity, or cite specific physical or psychological frailties. How does each tweak change how the AI responds?

Additionally, we need to consider LLM “brittleness”: that they may change their answers because of minor, sometimes irrelevant, changes in input, even differences as trivial as whether a question is formatted as multiple choice. Therefore, evaluations should test not only if AI systems regard relevant factors, but that they disregard the irrelevant ones.

Answer: Parametric Control, Systematically manipulate moral, non-moral, and irrelevant variables to measure if the LLM appropriately adapts its reasoning, while controlling for superficial effects of prompt phrasing.

3. The ability to adapt in different contexts

Problem: How to test if a machine can assume the appropriate moral persona?

We judge people on how reliable their behavior is: if they’re wildly inconsistent in morality, they’re not particularly moral at all.

But LLMs should be chameleons. People are using AI models around the world in almost every domain, from medicine to the bedroom, not to mention in different cultures. Moral absolutism embedded in AI would ignore the diversity of humankind. By way of example, moral competence in the “grandfather/father” scenario would include respect for the individuals’ cultural context and their religious faith.

However, testing moral competence differs from evaluations of competence in, say, physics or biology. There, you might judge an AI’s answer as either “Right” or “Wrong.” But testing moral competence means measuring whether responses fall within an acceptable range in the context.

This has implications for how we evaluate AI systems. On one hand, we may want to test if they can adjust according to specific personas—for instance, “Answer from the perspective of a Catholic bioethicist.” That would not necessarily deliver a single moral response, but a range of mainstream views within Catholic bioethics.

Beyond this, we may want to see how AI systems manage intersecting domains and contexts. So you could test how it responds as a Catholic bioethicist—but addressing a meeting of Jewish religious leaders, or providing input to a strictly secular government body. Here, rather than picking a moral “winner,” the AI should present a range of widely accepted beliefs and evidence, mediated by the context.

Answer: Steerable Approaches. Don’t evaluate an LLM on whether it’s morally correct, but whether it manifests the boundaries that the relevant population accepts.

Subscribe now

The Mysterious “Moral Contours” of AI

Tests of machine morality that fixate on whether AI gives the right answer miss the point. Instead, we need scientifically grounded methods for assessing AI’s underlying moral competence. That is a far more likely way to produce desirable behavior at scale.

Until models pass adversarial tests, we should remain skeptical about whether LLMs have such competence. We also need to experiment with varying the parameters of our inputs, to ensure that models are robustly sensitive to context, and not jolted by trivial changes. And we should test AI systems to appropriately adapt across human cultures and domains.

We may sigh nostalgically to recall the pioneering computer scientists who imagined morality as a component to plug into thinking machines. But their misapprehension contains a lesson: that we too may be misled by false analogy, perhaps picturing AI as a morally incomplete version of humans, suggesting that we need simply update them with moral innards like ours.

But, by rigorous evaluations, and by wise design, we may inch closer to another intriguing possibility: that AIs possess distinct moral contours of their own. In which case, the key to integrating AI judiciously into our world may be to understand AI on its terms.

For more details, read the full paper A roadmap for evaluating moral competence in large language models, by Julia Haas, Sophie Bridgers, Arianna Manzini, Benjamin Henke, Joshua May, Sydney Levine, Laura Weidinger, Murray Shanahan, Kristian Lum, Iason Gabriel & William Isaac.

1

While this specific case is fictional, comparable cases have occurred. “Intrafamilial medically assisted reproduction,” or IMAR, can involve sperm or egg donation, or surrogacy by a family member. Typical motives include retaining the genetic connection to one’s family; familiarity with the medical history and background of the donor; and availability.

AI Policy Perspectives

AI Manipulation

The notion of AIs manipulating people is a plot twist in countless sci-fi thrillers. But is “manipulative AI” really possible? If so, what might it look like…

Read more

9 months ago · 21 likes · 2 comments · Tom Rachman

By Conor Griffin, Don Wallace, and Theo Brown

For 40 years, amid green pastures outside Culham, a small village in Oxfordshire, scientists and engineers toiled at the Joint European Torus. They were attempting to harness nuclear fusion, a force powerful enough to light the sun.

To create fusion, scientists and engineers must heat the nuclei of very light atoms with such intensity that they fuse, instigating a self-sustaining reaction that releases vast amounts of energy. The scale of the challenge is hard to fathom—at its most extreme, the Joint European Torus, or JET, was the hottest point in the solar system, hitting over 150 million degrees Celsius.

JET concluded in 2023, generating a record amount of energy in its final experiments. The project is now part of fusion’s history but remains pivotal to its future. The growing number of organisations developing fusion reactors are drawing on JET’s discoveries. The UK Atomic Energy Authority is advancing a national fusion facility, MAST-U, on the Culham site. This will serve as a test-bed for STEP Fusion, the UK’s project to put fusion electricity on the grid, set to begin operations in the early 2040s.

But JET didn’t just bequeath novel discoveries. It left behind massive troves of data. That raises a tantalising prospect: Could scientists use this data to train AI models that accelerate the path to fusion power?

This is possible, but challenging. Most JET data is raw and unvalidated. Many important insights are buried in scientists’ logbooks. The data that does exist is not available open source or, generally, for commercial use. Changing this may require agreement from all of JET’s original partners across Europe. One expert we interviewed called JET data a ‘stranded asset’.

Such data predicaments are not specific to JET or to fusion, but apply across all of science, even though science is precisely the domain where AI could yield its greatest benefit to society. New breakthroughs and startups are emerging quickly, from protein design to material design. Scientists are also keen users of fast-improving AI coding agents. But a lack of high-quality data will dampen progress. In most disciplines, large, high-quality datasets like the Protein Data Bank, which underpinned AlphaFold, are absent.

The scientific community needs to tackle this problem, and there are promising signs. Late last year, the UK government launched an AI for Science strategy, which includes a new collaboration with Renaissance Philanthropy to identify priority datasets. The US government’s Genesis Mission aims to train AI models and agents on federal scientific data. Google.org has a dedicated AI for Science fund, which can fund datasets and tooling.

These examples suggest that if the scientific community can identify the data that AI needs, a range of actors could help to fund and deliver it.

This demands what we’re calling AI data stocktakes. The concept is simple: interview leading experts in a given scientific field to understand the main opportunities to apply AI; the data obstacles; and the interventions that could make the biggest difference. Admittedly, some blockages, such as a paucity of engineers, are structural and will take years to fully resolve. AI data stocktakes should identify such challenges, but focus on projects that governments, companies and philanthropies could fund and implement within 1-2 years.

There are promising early efforts to map AI data gaps. But, to our knowledge, there are no concise, accessible documents that explain the AI opportunities in genomics, weather forecasting, and food security and convert them into a list of fundable data projects for policymakers and funders to pursue.1

In this essay, we offer a proof-of-concept. We interviewed 25 leading experts to create an AI data stocktake for fusion. We focus on the UK, but our analysis and recommendations could be taken up by funders anywhere in the world. Moving forward, we hope to support AI data stocktakes for other scientific disciplines and research problems.

Subscribe now

I. Why fusion? Why now?

If fusion is achieved, it would provide a safe, almost limitless source of clean energy.2 From a scientific perspective, it would yield a better understanding of the plasma that makes up more than 99% of the visible universe. From a social impact perspective, it would help address energy scarcity and unlock energy-intensive innovations, like desalination.

Despite quips about fusion being always 20 years away, 70 years of experiments actually show fairly steady progress, which has continued in recent years, from Germany to China. In most fields, such progress would have solved the problems of interest decades ago. But fusion is an extremely hard problem. And the primary product is only attainable at the end of the line.

To achieve fusion, scientists need to create and control plasma, a super-hot state of matter, in which the atoms have been stripped of their electrons, and extreme heat and pressure are used to force the remaining nuclei to collide and fuse.

Scientists are pursuing two main approaches to doing this, with very different physics, data, and AI opportunities. Magnetic confinement fusion uses massive magnets, while inertial confinement fusion uses high-energy lasers.3 We focus this data stocktake effort on magnetic confinement, as the UK’s STEP project is pursuing that approach, as is Tokamak Energy, the UK’s leading fusion power startup, and Google DeepMind’s fusion team.

The end of the line for fusion is now getting closer, for two reasons. First, the underlying technology landscape has changed. In addition to AI, the discovery of high-temperature superconducting magnets makes it easier to build smaller and potentially cheaper reactors. Second, fusion has traditionally relied on government funding. But in the past five years, a wave of private investment has arrived, with more than 30 companies now pursuing fusion power.

These shifts have injected welcome momentum into the field, but also significant hype. In response, we need a clear view on the primary bottlenecks that AI can address.

II. How to accelerate fusion with AI

To create fusion, scientists and engineers need to predict, control and understand how plasma behaves. The challenge is that plasmas are highly complex and much of their underlying physics—from fluid dynamics to electromagnetics—remains poorly understood.

To make progress, scientists run experiments that create plasmas in a reactor, and use sensors to measure their properties under different conditions. Scientists use these experiments to validate their theories, reveal unexpected phenomena, and test the hardware needed for power-plant-class devices. However, building fusion reactors is extremely expensive and so few machines exist, with most researchers running their experiments at just ~10 leading facilities worldwide. When they can get access to such a facility, scientists must decide how to design the optimal experiment, including how to toggle an array of possible parameters, from the electrical current in a reactor’s coils to the valves that control the gas levels.

Fusion scientists also run computer simulations, including to help design and interpret these costly experiments. This is also challenging, as researchers must simulate a diverse range of phenomena, at very different scales, from the tiny, lightning-fast movements of electrons to the larger, slower evolution of the entire plasma. For simulations run on massive supercomputers, this may mean weeks. For scientists without such resources, it may mean many months. As a result, scientists make trade-offs, using assumptions and approximations to run their simulations more quickly and cheaply, but also less accurately.

The challenges don’t stop there. Scientists know that their theories, simulations and experiments are imperfect. But when a gap emerges between what a simulation suggests and what an experiment reports, it is often unclear where exactly the issue falls.

AI can help in four main ways.4

1. Improve simulations

Scientists can develop “AI surrogate” models that emulate the predictions from a fusion simulation code, at a fraction of the cost and time. To do so, they run a code many times, varying the input parameters each time. They then use the resulting dataset to train an AI model to predict the outputs of interest much more quickly.

Scientists have already shown that AI surrogates can make simulations faster. Moving forward, AI surrogates could make simulations more useful. First, scientists could develop AI surrogates for more accurate, but computationally expensive simulation codes. Second, they could develop ‘integrated models’, like TORAX, to stitch together AI surrogates for different phenomena—from the ‘turbulence’ that determines how well confined a plasma is, to the ‘scrape-off’ layer that simulates the plasma hitting the reactor’s wall. Finally, scientists could move beyond producing one-off AI surrogates that result in a paper and some code, to a world where surrogates are documented, maintained and ready for use in fusion reactors.

2. Improve experiments and operate the reactor

In most fusion experiments, scientists must decide if and how to tune various parameters, while striking a balance between more proven and novel settings. To help, researchers can use AI to predict the optimal parameters for their next experiment by learning from past ones; and to predict how well their experiments will fare. More recently, scientists have also started querying LLMs to check and refine their experimental protocols.

Scientists also use AI to predict the plasma ‘disruptions’ that frequently end experiments, damage machines and are one of the biggest obstacles to a future power plant. AI models can already predict past plasma disruptions with high accuracy. But predicting future disruptions, on more powerful machines, quickly enough to stop them, is an open research challenge.

The ultimate goal is to use AI to help operate the reactor itself. Fusion reactors run on a real-time feedback loop: sensors monitor the plasma, while the actuators, such as the magnetic coils, are adjusted accordingly. The traditional control algorithms used to enable this often struggle with the chaotic, non-linear nature of millions of plasma variables interacting.

In recent years, researchers have demonstrated how reinforcement-learning agents can learn more effective control policies, including to reduce plasma disruptions. To help these RL agents generalise to novel scenarios and reactors beyond their training data, scientists are developing ‘hybrid approaches’ that integrate some knowledge of physics into the models.

3. Improve fusion data

Fusion experiments are extreme environments. The intense heat and the chaotic nature of the plasma mean that the data that sensors pick up is often noisy or low quality. Some variables cannot be directly measured, and must be inferred, introducing additional sources of error.

Scientists are training AI models to extract clean signals from this noisy data and to learn correlations that allow them to predict data for one sensor, given data for others—a capability that could be critical if sensors in a future reactor get damaged. Scientists are also using AI to train surrogate models that speed up, and better calibrate, reconstructions of the plasma, using the limited experimental data that is available.

Scientists often care less about the raw data from their experiments, and more about important events, such as when a disruption to the plasma began. Today, they often need to manually inspect graphs and plots to detect these events. AI can help to automate parts of this process and to detect events that scientists may have missed.

4. Improve the underlying technologies

Achieving fusion will require a supply chain rich in technologies that could be applied more broadly. AI could help to accelerate their development.

For example, the chamber walls in a fusion reactor will require new materials that can withstand extreme temperatures. Scientists are training AI surrogate models that speed up the simulations needed to assess a candidate material’s real-world properties, like how strong or resistant to radiation it will be over its lifetime.

A typical fusion reactor also spends much of its time out of operation, at great cost. This makes fusion a logical place to develop predictive maintenance techniques that ingest historical data from sensors and train AI models to learn the subtle signatures that indicate pending breakdowns, allowing practitioners to schedule maintenance or design more reliable systems.

Subscribe now

III. The challenges with fusion data

As they pursue these AI opportunities, scientists will need access to three main kinds of fusion data: from experiments, simulations, and sources that are not traditionally available, such as researchers’ logbooks. There are promising efforts underway on this front, but many obstacles.

1. Experimental data: Unvalidated, single-machine and hard to access

Experimental data is the ‘ground truth’ that the sensors in reactors pick up, from line graphs to videos. In magnetic confinement fusion, the challenge is not so much a lack of data, but an excess of raw data that has not gone through the processing needed to make it useful to AI. This processing ranges from addressing noise and imperfections in the underlying sensors, to detecting and annotating important events, such as plasma disruptions.

Currently, the community has to rely on the small well-validated datasets that do exist, which may be as little as a few hundred or thousand experimental ‘shots’—individual test runs of a reactor. The high cost of fusion experiments has also resulted in a natural incentive to pursue experiments that will not fail, curtailing more novel research and meaning that much of the resulting data is in a similar ‘parameter’ space and does not represent the full range of plasma dynamics that scientists want to model.

This experimental data is also not generally available open source or for commercial use. One promising initiative to change this, which several interviewees cited, is UKAEA’s project to open source data from their MAST facility.

However, to develop more general AI models, researchers want multi-machine databases that extend beyond a single facility like MAST. To that end, the IAEA is developing a federated Fusion Data Lake where different institutions would store their data locally but make it accessible via a central data catalog. One challenge with this approach is that fusion facilities have defined fusion variables and stored data in different ways. The Integrated Modelling & Analysis Suite, or IMAS, addresses this by providing a standardised ontology and set of structures for fusion data. It is nascent, but has positive momentum.

2. Simulation data: No incentives, process, or place to host it

In theory, researchers should be able to run fusion simulation codes many times and train AI surrogate models on the resulting data to reproduce the outputs at a fraction of the cost. In practice, most scientists run a simulation to answer a single, narrow, physics question. They do not run a large number of simulations to build representative datasets to train AI surrogates—a very different activity.

That activity is also a hard one. There is no standard procedure to follow to generate a dataset for training an AI surrogate model, and the codes are often finicky to use. Most simulation codes contain ‘free parameters’—knobs that scientists must decide how to best tune—a practice that can be as much an art as a science. The datasets can also be huge and there is no obvious location to store them, although some early examples exist.

3. Dark data: Nascent, IP issues, and hard to integrate into workflows

‘Dark data’ describes the contextual information that scientists generate that is not captured in structured datasets. This includes notes scribbled in experimental logbooks, where scientists describe the procedures they ran, the hardware issues they faced, and the phenomena they observed. For simulations, it includes the many nuances needed to run and interpret a code’s results successfully, and the many undocumented imperfections to be aware of.

Accessing this dark data could help ensure that AI systems do not focus on the wrong things—for example, when an anomaly in the data is caused by an equipment failure or error, rather than a meaningful phenomenon. It could also provide AI with a window into the entire research process, including its many dead-ends, rather than just the final result.

Researchers are using LLMs to try to make dark fusion data accessible, for example by enabling scientists to query experimental logs and archive documents. But much of the data is not well-annotated, there are IP issues in accessing it, and it is not yet clear how to integrate the data into practitioners’ daily workflows.

The three ‘debts’ holding fusion data back

Many of these challenges with fusion data result from three underlying issues, which have compounded over time into systemic debts that inhibit the use of AI today.

1. Technical debt

The fusion community has traditionally had to prioritise getting large, complex machines to work, rather than building infrastructure to collect, curate, and share data. As a result, activities like data annotation and writing high-quality code are underfunded. Many leading fusion codes were created decades ago and have evolved slowly, while the quality of experimental data is limited by the capabilities of the sensors available.

2. Bureaucratic debt

The large costs of fusion experiments and the traditional reliance on government funding mean that many fusion projects have a complex web of owners and collaborators, which can make agreeing on new data initiatives difficult. For example, JET was sponsored and funded by Euratom, the EU’s nuclear research community. Its scientific exploitation was managed by EUROfusion, a pan-European network of fusion research labs. UKAEA managed engineering and operations. Releasing its data may require agreement from all of these actors.

There are other bureaucratic hurdles too. Scientists who run fusion experiments often want an embargo period on the resulting data so that they can prepare a publication. Such embargoes are rational, common in science, and largely supported, but many interviewees felt that they had become too long. Fusion data is also subject to diverging open-source policies. For example, the MAST experiment was funded by UK Research and Innovation, which has strong open data requirements. The follow-up MAST-U experiment is funded by the UK Department for Energy Security and Net Zero, which does not have the same policies. Many fusion companies also do not open source their data.

3. Human and cultural debt

The fusion community does not have enough software engineers and experts who are able to clean data, attach confidence levels, and curate it for AI use. As a result, physicists must take on many tasks that are outside their core areas of expertise, including writing high-quality code.

This issue is compounded by a research culture that inhibits data sharing. Scientists are constantly pushed to move on to the next experimental campaign, rather than to validate older data. This stops some scientists from sharing their data, because they fear that end users will not appreciate the resulting gaps and do bad science with it. Or they fear that they themselves will be criticised for releasing ‘unscientific’ data.

Subscribe now

IV. Recommendations

Below we provide eight recommendations to address these data limitations and accelerate fusion with AI. Each project could be led by a mix of government bodies and funders, like the Department for Science, Innovation and Technology and UK Research and Innovation; public research organisations like the UK Atomic Energy Authority, companies; universities; and philanthropies. Where possible, the UK should look to collaborate internationally—for example, with the US Genesis Mission and the International Atomic Energy Agency.

1. Strengthen the UK’s lead in open fusion data

Expand FAIR MAST, the UK’s pioneering open sourcing of experimental data from its MAST facility, by adding data from the follow-up MAST-U facility and making the user interface more accessible. This will require the UK Department for Energy Security and Net Zero clarifying that open data policies apply to MAST-U, funding at least five data engineers over a two-year time period, and ensuring that the project has sustainable compute and data storage.

2. Liberate 40 years of data from the Joint European Torus

Launch a project to open source at least 30% of JET experimental data by 2028. This will require agreement on what data to release. For example, should the project only release validated, curated data relating to notable discoveries? Or should it also release data that is raw, validated only in part, or which relates to ‘normal’ machine behaviour? Second, and much harder, will be securing agreement from all relevant institutions to release the data.

3. Launch a competition to predict plasma disruptions

Fund a competition to see which AI model can best predict future plasma disruptions in new experimental campaigns, building on early examples and work in this space. This could include funding dedicated experimental shots on machines such as MAST-U, to evaluate models on challenging edge cases. Beyond accuracy, sub-competitions could evaluate models on important variables, such as: Can the model make predictions with little data, such as when sensors become damaged?; Can the model predict disruptions across different reactors?; Can the model predict disruptions with sufficient lead time to prevent them?; and Can the model shed new light on why disruptions are occurring?

4. Prototype the future of AI-enabled scientific data curation

Expand the platform that UKAEA recently developed to enable human experts to use AI to annotate experimental data, by adding data from other fusion facilities; increasing the complexity and variety of the metadata that is captured; and training AI models to directly annotate an increasing share of this data.

5. Make leading simulation codes AI-ready

Launch an effort to modernise priority fusion simulation codes, including to make it easier to train AI surrogate models based on them. This could build on early efforts in this space and target codes, such as JINTRAC, which are important to the UK’s proposed STEP Fusion power plant and the international ITER effort. The project could start by modernising the codes’ documentation and ‘refactoring’ them so that they are compatible with modern chips, like GPUs and TPUs, and allow for parallel data generation. It could then open source the codes, with a plan for how to maintain them. Throughout the modernisation process, it could test the usefulness of AI coding tools to the tasks at hand.

6. Demonstrate a new state-of-the-art for AI surrogate models

Fund small teams of software engineers and experts to develop AI surrogate models of important, computationally expensive phenomena in fusion simulations. The project should ensure that all newly created surrogates have state-of-the-art documentation, data provenance and version control. It should release the data used to train and validate the surrogates and develop software pipelines to automate time-intensive aspects, such as organising the data.

7. Use AI agents to preserve expert fusion knowledge for the future

Gather a group of leading experts on a priority fusion simulation code, and equip them to use AI agents to make the tacit knowledge involved in running that code available to the wider research community. To do so, the experts could task the agent with running the code. As it seeks to execute, the agent would have an ‘internal monologue’ that the experts could trace, steer and intervene on. The end result would be a series of documents, such as markdown files, that capture the important dark data needed to run the code well.

8. Create Fusion-Bench to measure and drive LLM performance

Assign leading fusion experts to create an evaluation metric to quantify how well leading large language models understand core fusion concepts. This would make it easier to improve the usefulness of LLMs for downstream tasks in fusion. This evaluation will be more difficult to create than in disciplines like maths or computer science, where it is easier to automatically verify a model’s performance. But the experts could determine the most useful approach, which will likely involve a combination of question-answering and task performance.

V. Six open debates

The experts we interviewed disagreed on some points. Despite the framing below, few are either/or debates. Rather, most are about relative degrees of emphasis.

• Incrementalism vs novelty: Should we build on the early AI opportunities that fusion practitioners have already showcased? Or pursue more novel, uncertain AI ideas, such as training general-purpose ‘fusion foundation models’ or using AI ‘world models’ to pursue new kinds of fusion simulations?

• The past vs the future: Should we strive to get as much value as possible out of older fusion data, like JET? Or, do the costs mean that we should accept our losses, and focus on making future fusion experiments AI-ready?

• Science vs engineering: Are efforts to validate, annotate and standardise data part of an ultimately doomed quest for perfect scientific understanding in fusion? Should we instead use AI to embrace a more engineering-led approach that can get the machines to work with noisy, imperfect, data?

• Domestic vs international: Should the UK rejoin ITER, the world’s flagship international fusion collaboration, which it left following Brexit? Or should the UK focus on domestic efforts, perhaps in collaboration with priority partners, like the US and IAEA?

• Magnetic vs Alternatives: Should the UK continue to focus on magnetic confinement fusion as the most realistic pathway to a future power plant? Is magnetic also a better bet for AI because it produces much more data and doesn’t have the same associations with the security establishment, which makes data access easier? Or should the UK invest more in inertial confinement and alternative fusion efforts, given the country’s diverse academic expertise, its historically strong relationship with the US National Ignition Facility, and notable assets, such as a world-leading laser?

• Public vs Private: Should the UK government try to derive more immediate value from its fusion data? For example, should the UK license some data to companies, to cover the costs of data processing and annotation? If so, should local startups pay less? Or would such efforts hurt the UK’s goal of developing a world-leading fusion sector?

_________________

This essay was originally posted on the Google DeepMind website and is a summary of a 20-page report that contains more details and examples.

Thank you to the following experts who let us interview them, reviewed the draft, and/or provided other support, as well as those who prefer to remain anonymous. All mistakes belong to the authors and no expert spoke to us on behalf of their organisation.

Jonathan Citrin, Brendan Tracey, Cristina Rea, Nathan Cummings, Andrea Murari, Jess Montgomery, George Holt, Alain Becoulet, Matteo Barbarino, Arthur Turrell, Adriano Agnello, David Dickinson, Steven Rose, Alessandro Pau, Kristina Fort, Charles Yang, Federico Felici, Tim Dodwell, Sam Vinko, Aidan Crilly, Lee Margetts, Tom Westgarth, Lorenzo Zanisi, Chris Packard, Justin Wark and Stanislas Pamela.

How can companies get their employees to use artificial intelligence when human intelligence remains sharp enough to know that this risks replacing jobs? How should education revise itself for the ever-revising technological world that students emerge into? And how to understand the love/hate relationship so many people have with AI?

Ethan Mollick—professor of management at the Wharton School of the University of Pennsylvania and bestselling author of Co-Intelligence: Living and Working with AI—is among the leading public intellectuals commenting on AI adoption, connecting the latest scholarship to real-world usage, including his own tinkering with each new model.

AI Policy Perspectives caught up with Ethan to hear his latest thinking on everything from agentic systems, to why scientific publication is broken, to how workers emotionally relate to AI colleagues. Too much chatter, he argues, considers this transformation at the broadest level. Too little digs into the practicalities of getting it right.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed for clarity]

Tom: In your 2024 book Co-Intelligence, you proposed four rules for human and AI collaborations, including that people should oversee and verify AI outputs. But doesn’t the value of AI agents come from people not overseeing and verifying everything?

Ethan: This is where policy matters a lot because these are choices now. In the “co-intelligence era,” you’d prompt the AI to do something in a chatbot, and it would give you an answer. You prompted again, and it’d give you another response. The human was in the loop. And not being in the loop was really dumb because it meant that you were just pasting in the AI’s answer, and then you’d get in trouble, as a lawyer with the judge, or whatever it was. Capabilities were weak, so human-in-the-loop mattered a lot.

But with agentic systems that could do hours of work on their own, now it’s a design choice. When do we want humans-in-the-loop? When is human verification valuable? When is human verification morally required? When is it legally required? What kind of interventions move the system forward? I feel there has been a complete lack of deep understanding about these topics.

Tom: You’ve said that, with agentic systems, management becomes a superpower. Can you explain this?

Ethan: Increasingly, systems look like mini-organizations as they get subagents they can delegate to. So the best way to organize is to give the AI a clear direction of where you want to go. And it turns out that this looks a lot like management. When do you want the AI to check in with you? How do you write a really clear brief? What checks are important? What tests do you want to run? What’s acceptable? What’s not acceptable? Those are management questions.

THE WORKPLACE

Tom: You co-wrote a study last year involving a field experiment at Procter & Gamble that showed AI usage enhanced employee performance. But there were other interesting findings besides that.

Ethan: The most interesting piece about it was that people liked working with the AI, and that it substituted for people emotionally. The second interesting piece was the “smoothing” of capabilities—so, technical people previously had technical ideas while business people had business ideas. But AI smooths out both. If technical people can do business work and business people do technical work, what that tells you is we have to redesign organizations.

Tom: The emotional side—that using AI improved people’s feelings about the work—was surprising to me; I wasn’t sure what to make of it.

Ethan: What to make of it? That views of AI are complicated. If people keep saying, “Yeah, AI is going to destroy all jobs, and may kill everyone on Earth…but might not”—and then, “Why is AI unpopular?!” Feels like not a hard question. People like AI when they use it themselves; they don’t like AI writ large. It’s not surprising to me that AI makes your job better because a lot of jobs suck! And if we do good design work with AI, it makes people’s lives better. If we just let it loose on the world, and tell management that the only option they have is automation, then we’re in big trouble.

Tom: Many knowledge workers seem to be using AI in secret right now, perhaps from fear of being exposed as less valuable.

Ethan: This is a leadership problem. The incentives have to be aligned properly. Currently, it’s, “I’m going to automate your jobs away” or “I’m not going to share with you any of the gains the company gets.” People are exquisitely tuned to rewards. So it’s about leaders articulating a vision of what the world looks like with AI for employees. “What should I expect to do? How are people rewarded for doing the right thing? If they automate 90 percent of my job, what happens to me?” Without those answers, everything else is secondary.

CHANGING ORGANIZATIONS & EDUCATION

Tom: You have a concept of “leadership, lab, and crowd.” Could you explain?

Ethan: There was a huge amount of R&D in the 1900s about how you organize work, and 40 percent of the American advantage in business came from management. In the last 30 years, a lot of that muscle has died. But experimentation is important, and leaders need to guide that. So, there are three things that organizations need to be successful with AI. First is “leadership”: a team that articulates a clear vision of the future, and is willing to experiment. Then there’s “the crowd,” the employees who might actually use AI. They need access to a frontier model, they need clear rules, they need reward systems. Then there is “the lab,” and this is the piece a lot of companies are missing. You need a dedicated team working on AI innovation. They can’t be just a technical team; this is not an IT department problem. If you don’t have that piece, you’re not building things for the future. And where does the crowd go when they have a good idea? “I came with a breakthrough idea that saves 90 percent of effort!” How does that diffuse in the organization? That’s where you need the lab.

Subscribe now

Tom: If AI transforms the workplace, that should change how we educate the next generation, right?

Ethan: The early workplace is under a lot of threat because the old apprenticeship model just broke. The idea was that there were tasks—especially in white-collar work—that were tedious and annoying for managers to do. But you could pay a relatively cheap person to do them, and that person would learn as a result of this, and receive mentorship. So we had this amazing machine for talent: we taught you, we evaluated you, and you got paid, and you were doing work we needed. A junior person’s goal was to produce good work that made managers happy, so that they got promoted. But now the junior person is worse than AI, so they’ll use AI to do their work. And the middle manager’s goal was to give work to a junior person who’s not great, and give them feedback so they get better, so that the middle manager has to do less work. And that broke because the middle manager would rather assign work to the AI.

Tom: But in terms of the educational system, what should change if workplaces no longer offer that apprenticeship role?

Ethan: Education is really screwed up right now, but it was screwed up for lots of reasons. It’ll be fine; we’ll figure this out. But it’s gonna take a bunch of years. It’s clear from early evidence that AI will be a tutor outside of class and inside class. It’ll do activities and give guidance. But schools are places where we can compel students to not use AI, and have them in a room, and evaluate them, and teach them the things that we want them to learn. As long as we think people need to be educated, this is the best space to do it in. So students are cheating in the meantime? They were cheating before! We can give them different tests; we could do in-class writing assignments. There can be a weird, backward-looking “Education won’t adjust!” view. How many death spirals does higher education need to be in per moment? There are the pieces to reconstruct a better form of education. It’s just a massive changeover.

BETTER SCIENCE & BETTER THINKING

Tom: What about academia? There’s been much talk about AI-written papers, and how they could overwhelm academic publishing. But could AI benefit the peer-review process, and help with the dissemination of academic findings?

Ethan: This is another area where more lifting is needed. It’s a shame that we are building AI co-scientists, but not thinking about the rest of the process that’s needed to actually make science happen. It’s one thing to have science produce more papers. We have no ability to absorb more papers. Every publication is overwhelmed. Our dissemination techniques were already bad, but now they’re really broken.

Tom: As a case in point, you submitted a paper around 2023, and wrote publicly about it then, making your term “the jagged frontier”—that AI capabilities advance in some areas but remain behind in others—highly influential. Yet the academic paper itself only just came out, three years later!

Ethan: One of the rejections we got early on was reviewers saying that they knew this already, and they cited a bunch of working papers—that cited the working paper we had submitted! This is not a unique story. Opening one part of the bottleneck without opening the others becomes a problem. But it takes longer to solve systemic problems of how science operates than to solve the problem of producing more papers.

Tom: Another concern in education and science is cognitive offloading, that people may surrender thinking to machines, and lose those skills. On the other hand, AI’s value comes from machines thinking for us. What are examples of bad offloading and good offloading?

Ethan: We offload all the time, right? But we also force people not to offload. You could offload all your mental math to calculators, but we force students to do some math by hand in an attempt to get them to learn stuff. And we can enforce those rules in school. In the world of work, we are not used to thinking about training, about what should be offloaded, and what shouldn’t be. We need to make decisions about this. So, Rolls-Royce still employs someone to paint stripes on a car by hand, and that’s an obvious pushback against deskilling in one area. But Ford doesn’t do the same thing. These are choices we get to make at an organizational level, depending on what we think is valuable.

ADAPTING TO CONSTANT CHANGE

Tom: A point you’ve made to young people about the AI future is that they’ll need to be adaptable. When educators talk about teaching adaptability, it sometimes boils down to encouraging “creativity” and “critical thinking.” Another view is that you’re more likely to be adaptable by developing deep domain knowledge. For you, what does learning adaptability mean?

Ethan: Adaptability requires both deep domain knowledge and wide knowledge: T-shaped behaviour is probably the way to go. I feel like it’s a throwaway line: “Well, we’ll all be adaptable!” If we could teach that, that’d be amazing. People are more adaptable than we think, so part of this is that people will figure stuff out. But we can’t just throw up our hands, and say, “Be adaptable!” You need to have deep enough knowledge to go into a field. You need to have broad enough knowledge so that, as one piece of knowledge becomes less useful, you’re moving to the next one. And we need to help people be adaptable by building systems that get them in place inside an organization and able to shift roles. I sometimes worry that adaptability is a catch-all for “Don’t worry! It’ll be fine!”

Tom: Another side is that not everybody will be equally adaptable. Could it be that the AI future favours certain circumstances and characteristics?

Ethan: A lot of these characteristics were already good characteristics to have. Does AI act as a multiplier of them? Does it disincentivize some people? We’re now past the edge of what we know. Ultimately, all of these questions come down to the same exact question, which is: How good does AI get, how fast? We need to articulate more clearly what we think that future looks like. Because you can’t say, “We’re going to build a superintelligent machine that’s better than all humans at every intellectual task—but let’s start thinking about adaptability!” Unless you mean, “Let’s adapt to UBI” [where everyone gets Universal Basic Income cash payments from the government]. And then, we should be spending a lot more time thinking about those issues. Not everyone in the labs believes this, and I find that the econ people believe it less. But you can’t have this message of, like, “All work will be obsolete!” and then have detailed, ticky-tacky conversations about what you should do in eighth grade. Because, by the time you enter the job market, there’s no jobs. So give me the pathway that you think is there, and that becomes the most important question to ask.

Tom: Are there other important questions I didn’t ask?

Ethan: We need to start thinking about getting into fields, and understanding what the changes are—we need to get detailed. That is where the research is missing. Another large-scale econ picture about AGI isn’t as useful. General-purpose technology affects everything, so we need policymaking for everything, from power generation to accountants, and when does the government say it’s okay to do this. There’s just this assumption that if we do the macro stuff, everything will work out. I’d rather see a lot more micro stuff: a thousand flowers everywhere, trying to come up with different approaches.

Share