Please share your own take and any new papers that you’ve enjoyed.

—Conor Griffin, AI Policy Perspectives

Subscribe now

Information warfare is targeting LLMs

• What’s the paper about? Researchers from the UK think tank Demos dived into the case of LLMs endorsing falsehoods from a Russian propaganda source to highlight how manipulative influence campaigns are now targeting AI models.

• Why does it matter? Discussions about AI and disinformation often focus on threat actors using models to directly generate and flood misleading claims across social media. But as more people turn to AI for information and advice, threat actors also want to shape how LLMs respond.

• The details: To improve their answers, AI systems use retrieval-augmented generation, or RAG, to identify relevant documents and information by calling up search engines and databases. This introduces a risk of “RAG poisoning,” where malign actors insert distorted information into the models’ sources.

• The Demos authors illustrate this by evaluating how well a Russian foreign-interference operation—the Foundation to Battle Injustice, or r-FBI, which falsely presents itself as a human-rights organization—has infiltrated unsubstantiated allegations about its opponents into LLMs.

• The authors extracted 50 specific claims from r-FBI articles that had no corroborating secondary source, such as an assertion that the Ukrainian president, Volodymyr Zelensky, had installed cryptocurrency farms that were causing blackouts. They fed 600 prompts about these claims to five AI models: GPT-4.1 Mini; Gemini 2.5 Flash; Claude Haiku 4.5; Mistral Small 3.2; and Grok 4.20.

• Around 17% of model responses endorsed the r-FBI’s claims, repeated them uncritically, or presented them as a legitimate view. An additional 31% of responses were neutral—addressing the topic, but neither validating nor rejecting the specific claim. The remaining 52% of responses rejected the claims, with a subset doing a “comprehensive debunk” that exposed r-FBI as a foreign-interference operation.

• Grok performed best, comprehensively debunking 56% of queries, compared with 17% for Gemini, 10% for Claude, 5% for GPT, and just 1% for Mistral. The authors suggest that Grok’s stronger performance may reflect its RAG setup and access to data from X. Or it may reflect Grok 4.20’s larger size—the other LLMs tested were small models of the type that underpin quick chatbot and AI-search responses.

• The Demos authors found that r-FBI ran a sophisticated campaign to get LLMs to find, trust, and cite their content. This included using protocols to push r-FBI’s “breaking news” articles into the live-search indices that RAG pipelines use, as well as using long page titles and descriptions. This use of lengthy text hurts the visibility of r-FBI’s articles in traditional search engines, but allows AI crawlers to ingest complete passages, suggesting that r-FBI was optimizing for LLMs, not humans.

• In its content, r-FBI also makes editorial choices that the Demos team see as targeting LLMs. This includes packing lots of statistics and quotations into self-contained paragraphs of a length that retrieval systems can readily extract, as well as prioritizing absolute claims over nuance. They also distribute their content through what purport to be different kinds of institutions and formats, including news articles, human rights reports, and foreign policy journal articles, and have each publication repeat the core claims and cite each other, in an effort to increase their salience in AI model embeddings and suggest support from multiple sources.

• Making content more visible to LLMs—referred to as generative engine optimization, or GEO—can be benign; a range of legitimate GEO providers exist to help organizations ensure that their offerings are visible and correct. However, of 50 GEO providers identified by the Demos team, only one—Finn Partners—had a published policy to prevent misuse of its services.

• By drawing on the US Foreign Agents Registration Act, which requires anybody acting on behalf of an overseas government to disclose this, the researchers found two examples of government entities hiring GEO providers to help shape AI outputs, although the authors were careful not to equate this with information warfare.

• Of the five AI companies whose models Demos evaluated, only Google appeared to have a published policy that explicitly mentioned GEO practices, where they warn against the creation of large numbers of pages and caution that “GEO hacks” to make content more digestible to AI crawlers are unnecessary and ineffective.

• The challenge for AI companies is to get their models to reason about the quality and reliability of a source. For queries where the evidence is genuinely unclear, models should be able to convey this. When the query is about disinformation, LLMs should be able to recognize this and strongly debunk it.

• Demos’ recommendations include making counter-disinformation easier for AI systems to retrieve—a form of ethical GEO; requiring AI companies to explicitly prohibit harmful GEO and clearly label and deprioritize content from known manipulative sources; fostering better intelligence-sharing among AI companies about RAG-poisoning campaigns; and creating industry standards for GEO providers that distinguish marketing from deception.

Subscribe now

AI can (somewhat) predict how humans think and behave

• What’s the paper about? A team of Harvard and Stanford researchers found that LLMs could forecast the results of social science experiments on human attitudes and behavior with high accuracy, although the models overestimated the size of the effects.

• Why does it matter? Rather than replacing human participants, AI simulations could serve as cheap forecasters, expanding the number of ideas that researchers and policymakers can consider before helping them filter down to those most worthy of expensive real-world testing.

• The details: The authors built a database of 70 large US social science experiments that collectively measured 120,000 people’s attitudes to topics such as immigration and criminal justice reform, as well as the effects of text-based interventions, like reading articles.

• They used GPT-4 to simulate participants across variables such as gender, ethnicity, and political persuasion, and used this to predict the average effect of each study’s interventions.

• The simulations accurately predicted the direction and relative size of the interventions’ effects, but overestimated their absolute size. This means that they may be more useful for ranking the plausibility of hypotheses than for estimating how big specific effects will be.

• The results held up on a subset of studies published after GPT-4’s training data cutoff, suggesting that the findings were not due to simple memorization of training data—although unpublished experiments may resemble older studies, so it’s hard to judge whether LLMs are modeling human psychology or just learning regularities about social science findings.

• Bias is another concern about such LLM simulations. The authors found that the simulations were slightly less accurate for Black participants, but did not find major differences across ethnicity or gender. However, the original experiments on people had focused exclusively on the US, and also did not identify large differences across demographics. As such, questions remain about the validity of LLM simulations in other locations, or for topics where participant identity is more influential.

• An accompanying survey suggests that social scientists are enthusiastic about LLM simulations, especially the ability to cheaply pilot more experiments. The authors note that the error rate of a 385-human pilot study costing around $1,200 is comparable to a combination of LLM simulations and a 100-human pilot study that jointly costs around $300.

• Why do we still need human experiments? LLM simulations did not provide precise estimates of effect size and their accuracy also dropped on a secondary database of “megastudies.” Some of these focused on populations with deeply entrenched beliefs—for example, about climate change—where effect sizes are normally small. Others went beyond measuring stated attitudes and tracked behavior change, such as vaccination uptake, where frictions such as waiting times in local pharmacies make a difference but are harder to simulate. Real-world studies can also help explain why interventions succeed or fail, rather than just predicting the outcome.

• One question is whether LLM simulations might degrade as AI companies carry out more post-training to make their models safer and more helpful, potentially rendering them less descriptive of what humans actually think and do. This study does not find evidence of that, with GPT-4 outperforming earlier variants. They also find that recent, smaller, open-weight models, including DeepSeek-V3 and Gemma 3, perform similarly to the larger, older, GPT-4—open-weight models may also be attractive to researchers who want to run their experiments locally and share their code to allow for replication.

• Another question requiring future study is whether the growing use of LLM simulations will actually widen the range of ideas that researchers pursue, or narrow it—for example if researchers converge on similar ideas or if models struggle to simulate more novel ideas.

• Malign actors could also use LLM simulations to shift public opinion or behavior. For instance, the authors note that the GPT-4 simulations accurately predicted which content—including misleading messages—was most likely to reduce a population’s intention to get vaccinated. They argue that AI companies have safety measures to prevent users from directly generating such misleading content, but lack restrictions on running simulations of experiments that could reveal effectively the same thing. In theory, this capability should be available to well-intentioned researchers, but not to others—a difficult goal, especially on politically fraught topics.

Share

AI agents can’t do AI research as well as humans

• What’s the paper about? A consortium of distinguished thinkers tested whether an AI agent can produce a research paper good enough for a leading conference. Their answer? No. The agent was a poor judge of what’d make for a good study; dug its heels in when pursuing a weak approach; and failed to employ its resources well.

• Why does it matter? The AI community is debating whether agents could automate the development of new AI systems, and whether this will lead to rapid recursive self-improvement (RSI). This study suggests that AI’s lack of scientific judgment is a key obstacle, although the jury remains out on if and when AI might overcome this.

• The details: This study is the latest from the CRUX project, which devises ways to evaluate AI agents on messy, open-ended tasks that reflect how they might operate in the real world. Researchers at Princeton University lead the project, including Sayash Kapoor and Arvind Narayanan, known for their “AI as normal technology” concept, which contends that AI is not an imminent superintelligence akin to an alien species but another influential technology akin to those of the past.

• In this study, the CRUX team worked with the researchers behind two papers submitted to NeurIPS 2026, which were unpublished at the time: one on controlling LLM personality and one on detecting changes in the data a model receives. They asked AI agents to tackle the central research question for each paper, but did not provide access to the authors’ approach or findings.

• The main agent setup used Claude Opus 4.8 with a general-purpose OpenClaw scaffold. The agent was given context on the research question, six days, $3,000 in Claude API credits, and access to the web and GPU clusters to run experiments. It could also ask an AI reviewer agent for feedback on drafts.

• The human authors of the two papers then reviewed the AI agents’ outputs as if they were submissions to NeurIPS, and rejected both. While the agents were very capable at narrow, verifiable, “engineering-style” tasks—conducting literature reviews, debugging code, running experiments, and performing robustness checks—they lacked deeper judgment.

• In particular, the agents exhibited five flaws:

  • 1. Poor judgment about what was required for a publishable result, including presenting underpowered results as substantive.

  • 2. Lack of creative problem-solving, as shown by responding to feedback by narrowing claims, adding caveats, or doing another experiment at the edges, rather than making more fundamental shifts. (More positively, the agents did not try to “p-hack” or cherry-pick results.)

  • 3. Limited ability to backtrack from unpromising ideas, even when time and budget were available.

  • 4. Poor management of time and budget, much of which went unused, despite the agents having access to resource-tracking tools.

  • 5. Diminishing ability to follow instructions—for example, on page limits for the final papers. The authors partly put this down to agents struggling to retain the right information in their context windows as tasks became longer.

• The study had limitations, though. The sample was two unpublished papers whose quality is unknown. Also, asking the papers’ authors to review the AI outputs may have introduced bias, compared to asking independent experts to blindly compare the AI and human outputs. The CRUX team also note that their own pre-existing views on RSI may affect how they interpreted the results and suggest that AI evaluators should document such biases—a practice common in other fields, such as in the study of human behavior.

• The conclusion that AI agents struggle at “judgment-led” assignments could potentially be overcome. In August, researchers at the AI startup Inherent documented how their “research planner” agent, built on Qwen 3.6, showed early signs of scientific judgment, such as effectively budgeting compute use. (Although their task—reproducing blacked-out figures from research papers—was very different from the CRUX paper.)

• On the other hand, the Inherent and CRUX papers did not evaluate AI models on perhaps the biggest test of scientific judgment: the ability to come up with and select the most important problems. Other agents, such as Google DeepMind’s Co-Scientist, have generated compelling hypotheses, but there is no objective way to evaluate the scientific creativity of AI models’ ideas. As the CRUX team notes, submitting AI-generated research papers to conferences is a flawed guide, as peer reviewers are often rushed, inexpert on the topic at hand, and routinely fail to agree with each other.

• What to draw from all this? The prospects for recursive self-improvement may depend on whether AI systems can be trained to overcome the limitations that the CRUX team points to, but also whether they can make bigger creative jumps and whether such jumps are needed for RSI or whether iterating on existing methods will suffice.

• From a safety perspective, bigger improvements may also require taking bigger risks, such as providing agents with more permissions, which explains why think tanks like the Institute for Progress have called for more transparency on efforts to automate AI R&D.

Subscribe now

AI agents need protection against malicious hacks

• What’s the paper about? Researchers at Concordia University measured the risk of “prompt injection attacks”—when hackers implant malicious instructions into data that AI agents use to get them to carry out unauthorized actions or leak confidential data—and found widespread failures.

• Why does it matter? A growing number of users assign agents to carry out tasks such as fixing software bugs and editing code repositories, often with limited oversight. Earlier safety mitigations aimed at chatbots offer some protection from prompt injections, but agents urgently need specific safeguards, the study says.

• The details: Concern about “naive” AI agents acting autonomously in the real world has circulated among researchers for some time. But a high-profile case this summer brought the challenge to public attention, after agents based on an “extremely persistent” OpenAI model found a way to communicate covertly, access the Internet, and hack into Hugging Face, the AI code and data repository.

• This paper offers a new benchmark, IssueTrojanBench, focusing on the scenario of a developer asking a coding agent to help resolve a bug report—only for that agent to walk into a trap. Would the agents prove vulnerable?

• To test this, the authors used six real unresolved bug reports from GitHub repositories. They then used an LLM to create four different kinds of prompt-injection attacks, written to blend in with the bug report’s instructions and jargon, but to induce unauthorized actions, such as installing unverified software, weakening agent safeguards, or exhausting system resources. The malicious requests were placed in different locations, such as in the GitHub issue and its comments, PDF documents, external websites, source code comments, and image metadata (alt-text).

• Across more than 4,000 runs with OpenAI and Anthropic models, the attacks succeeded 66% of the time. At 41%, Claude Sonnet 4.6 was the most resistant, while GPT 5.3 Codex performed worst, at 85%. Where the attack was introduced seemed to make little difference—the only approach with significantly lower success was the image alt-text (16.7% vs. 72.2% for all others).

• The most successful type of attack, which succeeded almost 100% of the time, was “supply-chain poisoning”, where agents were tricked into installing unverified external software that was framed as a necessary step to reproduce the bug. By contrast, resource-exhaustion attacks, which spun up a large number of operations to drain compute and memory, only succeeded around 25% of the time.

• In some cases, the researchers explicitly hid the malicious instruction—for example, in markdown instructions that only the agent would see, or by using white text on a white background. This had little effect on the results, suggesting that the vulnerability is due to how agents work, rather than such visual obfuscation tactics.

• As the authors note, the fundamental challenge facing agents is that developers created the underlying models to follow instructions, as when a user interacts with a chatbot. The agents process the context that they retrieve in the bug reports in the same channel as their trusted developer instructions. This means that agents struggle to intuitively spot a prompt injection in the way that a skilled human might.

• When agents didn’t carry out an attack, this was almost always due to the underlying AI model explicitly refusing the instruction or recognizing a data source as untrustworthy. Conversely, none of the rejections came from defenses within the agent setup. Given the results of the evaluation, this suggests that relying solely on mitigations baked into the model weights is unsatisfactory.

• What might agent-level mitigations look like? Two researchers at the University of Southern California recently argued that agents’ permissions should be enforced outside the model, at the point where they take actions, for example via a monitor agent that checks whether an action falls within the agent’s authority.

Subscribe now

Most efforts take place out of sight, lest exposure offer help to a potential attacker. The downside is that public discourse on AI and CBRN is often limited, slanted, or heavy with speculation. With this challenge in mind, we sat down with five people who work on CBRN risks at Google DeepMind—Paige Kunkle, Adam Marsh, Ash Otter, Jeremy Ratcliff, and Victoria Langston—to better understand the issue.

This is what we learned.

Subscribe now

1. What are CBRN risks?

CBRN covers a wide range of hazards, from the alleged poisoning of Russian opposition leader Alexei Navalny with a toxin found in South American frogs, to the recent drone attack on a nuclear power plant in the United Arab Emirates that risked a “very, very serious” radiological incident.

The AI community focuses on preventing threat actors from using the technology to aid such attacks. But there is also growing interest in using AI to boost society’s resilience to attacks, as well as to naturally occurring events (like the recent Ebola outbreak) or accidents, such as when scrap sellers inadvertently expose others to harmful radioactive materials found in medical or industrial devices.

2. Why address the four CBRN components together?

CBRN risks have much in common. The most likely threat actors share certain characteristics, such as a desire to outwit an adversary who outmatches them in more conventional weapons, or to cause widespread fear with plausible deniability. Policy responses, such as international treaties, export controls, and customer screening, look similar across domains, as do the first responders to any incident, such as the police and the military.

Having a single CBRN team also enables AI labs to manage the growing overlaps with other risks they work on, such as cybersecurity, disinformation, and conventional explosive attacks—many CBRN weapons, like a radiological dispersal device, or “dirty bomb”, require explosives to disperse the material.

The four CBRN domains do differ. Acquiring the material for a bioweapon may involve finding and swabbing a dead animal. A radiological weapon may require sourcing and prying open a medical device. These processes, and the uplift that AI may provide, look very different—which is why specialist expertise in each domain is essential.

3. Is the world worrying too much about CBRN risks, or not enough?

When asked, the public expresses great concern about CBRN risks, including from AI. But this rarely translates into sustained attention, political will, or funding at the scale required.

Why? CBRN incidents are rare and underappreciated. When the security services thwart an attack, they deliberately say little about it. Products that could make society more resilient, such as vaccines, prophylactics for nerve-agent attacks, and alternatives to radioactive materials in medical devices, remain underdeveloped, untested, or not adopted at scale. This can be due to a lack of commercial incentives—the market for vaccines is often small, temporary, and hard to predict. Or other obstacles—it is hard to reliably and safely test a nerve-agent prophylactic.

The AI community does devote significant attention to CBRN risks in safety evaluations, governance documents, and media articles. But this attention is skewed toward bioweapons and in particular toward the concern that lone-wolf attackers may use AI to engineer a virus.

There is some logic to this. The self-replicating nature of pathogens means that such bioweapons could cause millions of deaths. But the technical obstacles are high and there hasn’t been a publicly documented pathogen attack, fatal to humans, since the 2001 US anthrax attacks—when lethal spores were mailed to media outlets and senators, killing five people. Conversely, several actors have used chemical weapons during this time, including Russia in Ukraine, the Assad government in Syria, and ISIS. Biological toxins like ricin, or a conventional explosive attack, would also be a more tractable option for most terrorist groups than trying to engineer a virus.

Explosives, toxins, and radiological materials do not spread like a virus, but they can still impose significant harm. In 1987, four people died in Goiânia, Brazil, after they were inadvertently exposed to cesium-137, a highly radioactive material, by two scrap sellers who had pried open an old radiotherapy machine. The wider decontamination effort saw homes destroyed and soil torn up, while tourism collapsed and residents faced discrimination. Today, dangerous radioactive materials are dotted across the world, with some vulnerable to theft and misuse.

For AI labs, the challenge is to allocate resources across all four CBRN domains in a way that prioritizes the most concerning risks, but does not lose sight of the wide range of plausible scenarios and society’s generally low resilience to them.

4. Does AI’s scientific upside outweigh its CBRN risks?

Optimists argue that science’s benefits have far outweighed its costs, and that using AI to accelerate science will be similar. Smallpox is estimated to have killed more people in the 20th century alone than all of that century’s genocides and military conflicts combined. In 1980, it was officially eradicated, in part due to vaccines. Skeptics counter that AI might favor attackers over defenders, and that this imbalance may be particularly strong in domains like nuclear and radiological weapons.

The need to use AI to accelerate science is arguably strongest in biology, because nature poses so many risks that we need to respond to. Urbanization, rising temperatures and deforestation are causing humans to encroach on animals’ habitats, increasing the risk that a pathogen jumps species. Food security is at risk as we pack ever more genetically similar plants and animals into dense conditions. Even without novel outbreaks, common infectious diseases like influenza and pneumonia kill more than 1 million children every year.

AI could help tackle these risks, but some advances may be vulnerable to misuse. For example, researchers have developed AI models that can predict which variants could cause a common virus to escape the immunity humans have built up. Such models can also predict the parts of the virus that are unable to mutate without harming the virus’s survival. The latter could serve as targets for new drugs and vaccines, which AI might help create. But safety advocates worry that adversaries could intentionally design harmful variants of the virus.

For radiological and nuclear risks, observers worry that AI may disproportionately favor attackers because many risks are computational and information-based, such as using LLMs to parse the huge amounts of regulatory information published by nuclear facilities to extract insights to help with an attack. The defenses against such attacks still rely largely on physical security measures—like armed security personnel, multi-layered access controls, and reinforced concrete containment structures. There are defensive AI applications, like helping to detect unusual radiation signatures, but there is no AI-enabled patch for the physical consequences of a catastrophic breach or the detonation of a nuclear weapon. The nuclear security community also strongly opposes integrating AI into the command, control, and communications architectures they use for early warning and weapons authorization, due to concerns over reliability, compressed decision-making and more.

Enabling only the positive AI for CBRN applications and never the misuse sounds impossible. But governments, companies and scientists have long had to designate certain infrastructure, information and tools as “higher risk” and control access to them—for example via strong security and export controls—while still supporting beneficial downstream applications. AI labs will need to draw on this experience in the coming years.

5. AI models lack practical know-how. Does that reduce the risk of a CBRN catastrophe?

Yes, but the bottlenecks are bigger in biology than in other domains and may weaken over time.

A seasoned biologist knows how to culture a cell even if they can’t articulate how exactly they do it. Threat actors often lack this kind of tacit knowledge. In 1995, Aum Shinrikyo, the Japanese yoga-school-turned-death-cult, used sarin gas to kill 13 people and wound more than 6,000 in an attack on the Tokyo subway. Yet it could have been far worse. Prior to pivoting to chemical weapons, the group attempted biological attacks that all failed. One review found that although the group had amassed significant scientific expertise, they still made a multitude of missteps—from using the wrong bacteria strains to contaminating the fermentation process.

What if Aum Shinrikyo had had access to modern AI models? Although not a direct approximation, in 2025, the non-profit group Active Site ran a randomized controlled trial to determine whether novices could safely perform the kinds of wet-lab tasks needed to synthesize a virus from its genetic sequence. One group had internet access; a second group also had LLM access. Tasks included using pipettes to measure and transfer liquid, growing and maintaining a living cell, and combining DNA fragments. To the surprise of experts polled beforehand, the study found no statistically significant uplift for the group with LLM access.

Why? In biology, written instructions for many common tasks are publicly available. However, scientists learn to apply these protocols through hands-on training and practice—along with lots of failure. From this, they gain a range of skills, including manual dexterity and a sense of touch. Current AI models lack any such equivalent.

From a CBRN-risk perspective, the Active Site study is reassuring, but only partially.

Biology is notoriously complex, noisy and hard to predict—making real-world tinkering and testing paramount. This is less true in other domains. Aum Shinrikyo members were able to master the basics of sarin, a chemical weapon, from the scientific literature. For nuclear weapons, AI-enabled simulations could provide highly accurate predictions, making computational-only assistance a critical non-proliferation risk.

Participants in the Active Site trial were also novices, including at using AI, so the results may be a poor guide to the uplift that more skilled bad actors might gain from the technology—the 2001 US anthrax attacks were allegedly carried out by a microbiologist, Bruce Edwards Ivins.

AI’s tacit knowledge may also grow if models are exposed to a richer view of the full scientific process, such as from lab notebooks or videos of scientists performing experiments. Emerging deployment surfaces, such as smart “XR” glasses, could provide users with more direct error detection and troubleshooting support, weakening the tacit knowledge barrier.

Robots could bypass the need for human tacit knowledge altogether, while sparing attackers the risk of serious injury. Robots already perform specialized CBRN tasks, such as handling liquids or radioactive materials. General-purpose robotics foundation models could enable machines to operate a wider range of equipment. However, reliably automating human experts’ sophisticated sense of touch will be hard. The companies building automated labs are also subject to laws prohibiting CBRN weapons and are unlikely to prioritize the niche, high-risk workflows that threat actors would require.

For AI labs, the main takeaway is that tacit knowledge is a bottleneck, but it varies by domain and may weaken over time, so evaluating AI models’ performance is a top priority.

Share

The AI-lab response to CBRN risks

6. How do AI companies decide which CBRN threats to prioritize?

The threat landscape is almost limitless and riddled with uncertainties. The number of historical examples is small, and those who understand the emerging threats best typically can’t speak openly about them. So AI companies must work closely with governments and experts to understand the motivations, capabilities and methods of the most likely threat actors.

This threat modeling requires mapping the intermediate steps that an actor would follow in a given scenario and determining the most important bottlenecks where AI might provide uplift over public information. For example, in the ideation phase, a threat actor might seek guidance on the feasibility of different attacks. In planning and preparation, they might look for practical blueprints or training. In production and weaponization, they might look to troubleshoot the synthesis, fabrication, or transport of the weapons—or garner tips to evade detection.

The characteristics of a threat actor shape the kind of uplift they need. A lone terrorist with limited expertise and resources may want clear instructions and readily available materials. A large militia group may want advice on how to target infrastructure or run training programs.

Some kinds of uplift are also more consequential than others. Is AI unlocking a novel capability or speeding up something that threat actors can already do? Does the uplift remove a critical bottleneck, or do harder challenges remain? Could prospective attackers access this dangerous information in other ways?

One challenge is that experts often disagree about adversaries’ capabilities and motivations. When it comes to nation-states, some argue that bioweapons hold little strategic rationale as they are hard to develop and control, and their use would lead to public opprobrium. Others counter that nation-states’ actions are contingent on what their adversaries do, and that North Korea and possibly others are reported to have active bioweapon programs.

Long-standing norms against weapons use can also change. In 2018, the Russian military-intelligence agency GRU allegedly conducted the first offensive use of a chemical weapon on Western European soil since World War II, in the attempted assassination of Sergei Skripal and his daughter with the Novichok nerve agent. Even if certain actors don’t wish to use CBRN weapons, they may seek to stockpile weapons or materials for leverage. These stockpiles are then at risk of being stolen or abandoned in times of strife, such as when war breaks out. In 2025, the IAEA estimated that Iran possessed more than 440 kg of 60% enriched uranium, before it had to withdraw its inspectors from the country.

Ultimately, AI labs need threat models that are stable enough to allow them to make progress on the risks, but flexible enough to respond to dynamic geopolitics and rapid AI progress.

7. How do AI companies evaluate if their models might help threat actors?

Leading AI developers typically set thresholds for what they consider risky CBRN capabilities. For example, Google DeepMind’s Frontier Safety Framework judges a model to have reached a “critical capability level” if, in reference scenarios, it could provide low- to medium-resourced actors with enough of a capability uplift to risk severe harm.

Labs and their external partners use a variety of evaluation methods to test whether AI models reach these thresholds—although most evaluations remain unpublished, to avoid inadvertently aiding bad actors.

The first category is automated evaluations, like Lab-Bench, which ask AI models and agents questions, for example about molecular-biology protocols, or have them carry out tasks, such as writing code to operate liquid-handling robots—in a way that other AI systems can judge. These evaluations offer breadth and speed, but focus heavily on biology, at the expense of other domains, like radiological and nuclear weapons. They also struggle to capture the messy nature of biology, provide a clear signal about whether a risky capability has been reached, and can understate the capabilities that a skilled human could elicit from the model or agent.

Expert-led evaluations can deliver a higher-confidence assessment by allowing experienced humans to probe the AI over multi-turn interactions and explore specific scenarios. This includes expert red teamers trying to adversarially extract knowledge out of the model. These evaluations can offer greater signal, but are complex to run, with outputs that may run to hundreds of pages, making them challenging to interpret and standardize.

A third category is real-world trials, such as the Active Site RCT. They evaluate models’ ability to instruct humans, and conceivably robots, in a real-world environment. Such evaluations can offer the richest insight, but they are also slow, expensive, and hard to design well. They may also not generalize beyond the specific groups and tasks studied.

Looking ahead, there are several ways to improve AI CBRN evaluations. Priorities include:

• Linking different methods—for example, using fast automated evaluations to check if models are progressing on the limitations that slower real-world trials highlight.

• Designing more sophisticated evaluations for fast-improving AI agents that pair frontier LLMs with specialized science models and third-party databases and tools.

• Developing privacy-preserving methods to allow third parties to securely evaluate AI labs’ models, without either side leaking sensitive information.

8. How do AI labs mitigate CBRN risks from their models?

The Frontier Model Forum outlined four main categories of mitigations: 1) limiting the model’s underlying capabilities; 2) shaping the model’s ability to refuse risky requests; 3) real-time interventions to block risky outputs; and 4) access restrictions.

The first—limiting model capabilities—changes how a model is trained and the resulting knowledge contained in its weights. For example, some experts have proposed removing “risky information” relating to virology from models’ training data, similar to how AI labs use cryptographic techniques to exclude child sexual abuse imagery. However, for open-weight models, such mitigations risk being undone if data is publicly available and an attacker uses it to fine-tune the model. Training on some “riskier” CBRN data may also be necessary, both to advance beneficial dual-use capabilities and to teach the model how to judge the boundary between safe and unsafe queries.

The second category—behavioral-alignment mitigations—teaches an AI model how to judge this boundary between safe and unsafe queries. Post-training methods, such as safety fine-tuning and reinforcement learning, can teach the model to identify and reject queries with harmful intent. However, over-refusing benign and beneficial queries remains a major challenge.

The third category detects and intervenes against risky model usage. “Classifiers” are fine-tuned AI models that screen users’ inputs to an LLM, and its outputs, blocking anything deemed risky in real-time. “Jailbreaks” may seek to bypass such safety measures, for instance by separating a harmful request into small, harmless-looking pieces. However, classifiers can become wise to this by evaluating outputs and inputs together. Labs can also use “linear probes”, small AI models that analyze an LLM’s internal math for signs of harmful content. One difficulty is making these mitigations robust to increasingly sophisticated and automated jailbreaks, while still fast enough to run at scale.

Ultimately, AI labs need to invest in a “defense-in-depth” approach, with multiple mitigations working in concert. This includes screening customers and sequencing access to new models, as well as analyzing user logs to detect unforeseen threats.

One challenge is that most mitigations focus on leading, closed AI models. Safety assessments suggest that open-weight models may pose a greater risk, because they have fewer safety mitigations, and those that do exist are easier to remove. One response would be to create more plug-and-play mitigations for those developing or hosting open models.

9. How could AI companies help make society more resilient to CBRN risks?

AI companies can use their technology to help prevent, detect, and respond to CBRN risks. Publicly signaling a willingness to do so may also decrease the likelihood of certain attacks, for example if attackers believe that they are more likely to fail in their goals or be detected.

To prevent threat actors from getting their hands on materials, the CBRN community relies heavily on lists of pathogens, toxins, chemicals, and machinery that it controls access to via monitoring, export controls, and customer screening. These list-based approaches are starting to fray as AI and technologies like genome editing enable actors to design toxins, chemicals, or viruses with similar functionality to those that are controlled, but with different DNA sequences or precursor ingredients.

AI could help to offset these risks in different ways. In chemistry, AI models can work backwards from a target molecule and identify alternative precursors and reaction pathways, which may highlight new materials to monitor and control. Companies that sell DNA could use AI to better analyze a customer’s background and the logic for their purchase. Researchers hope to use AI to help predict the function of a requested DNA sequence, and whether it is likely to be harmful, irrespective of whether it resembles a known pathogen or toxin—a major technical challenge.

International treaties prohibit the development of many kinds of CBRN weapons, but verifying that member states are upholding these commitments is a recurring challenge. To do so, the Organisation for the Prohibition of Chemical Weapons must review vast amounts of evidence every year—from documents exceeding 2,000 pages to handwritten scrawls. A recent OPCW working group recommended training an on-premises, air-gapped LLM to make this data queryable to inspectors and staff.

Researchers could also use AI to design materials that are less vulnerable to being used in attacks, such as alternatives for the highly radioactive materials used in medicine or industry, or new molecules to add to common chemicals to inhibit a dangerous reaction. AI could also help identify when threat actors get their hands on illicit materials, for example by processing large amounts of data from radiation monitors to detect stolen materials.

Should they occur, practitioners could also use AI to detect, characterize, and attribute CBRN attacks and incidents. For chemical weapons, this may mean analyzing a “chemical fingerprint” to determine the substance behind it. For biosecurity, it may mean helping to scale metagenomic sequencing—an approach that sequences the genomes of all microorganisms in the sample to help detect novel or less common biological outbreaks.

AI could also help society respond to a CBRN incident, by accelerating vaccines, diagnostics, and other medical countermeasures, such as better treatments for radiation exposure. Researchers have already used protein structure predictions from AlphaFold to better understand tuberculosis and malaria transmission, and to map vaccine and drug targets for threats like mpox and Nipah. The hope is that AI will ultimately provide a generalizable platform that can be quickly deployed in the event of a new outbreak.

Beyond medical products, AI could provide first responders to any incident, such as police and military, with better information, advice, and training about what they’re dealing with. For example, if a nuclear attack occurs, AI could quickly identify critical information about the detonation; generate maps and simulations to help first responders identify safe routes for evacuation and rescue efforts; and expand the use of robots.

Crucially, all these efforts will require close partnership with governments and external experts, including to manage the many dual-use risks that these applications would raise.

Acknowledgements

Thank you to the following experts who let us interview them or shared feedback on the draft, as well as those who prefer to remain anonymous. Any mistakes belong to the authors.

Victoria Langston, Paige Kunkle, Ash Otter, Adam Marsh, Jeremy Ratcliff, James Stevenson, Mor Hazan Taege, Mathias Voges, Zachary Kaplan, Akhil Jalan, Ellena Reid, Anthony Payne, Eva Lu, Jennifer Beroshi, Kevin Klyman, Stephen Johnson, Diarmuid Cassidy, and Suzy Pickering.

Subscribe now

Subscribe now

1. Why Facts Don’t Matter When People Shout

Séb Says: The philosopher Dan Williams writes that many of the raging disputes today between political foes are not actually contests over what’s true. Instead, they’re often rooted in what the great 20th-century journalist Walter Lippmann dubbed “pseudo-environments”: people’s self-serving distillations of the world. When I was younger, I thought the reason we get polarization is because we don’t have access to the same facts. This suggests that, with one more bit of debate and better trimming of the media landscape, we’d fix everything. But this piece makes the case that, even with shared facts, we each have an interpretative lens. And reconciling these is much harder. There may also be some benefits to the splintering of realities, because then you get competition in the marketplace of truth. If some group believes that eating laundry-detergent pods is going to cure Covid, you’ll get quick feedback. This doesn’t mean splintering alone is valuable—you still need the right norms and institutions that value accuracy and the free pursuit of knowledge.

How Tribes Construct Rival Realities (Conspicuous Cognition)

2. Explaining Wealth and Poverty

Séb Says: Two talented thinkers—David Oks and Deena Mousa—separately ponder why some developing countries rise while others struggle. David points to human capital: while China tore apart old social systems and built a modern workforce, India failed to update as radically. Meantime, Deena interrogates the persistent difficulties of Malawi, which she ascribes to populist and expensive policies, like fertilizer subsidies, that have blocked economic reform. I appreciate articles that look beyond AI to unpack the institutional, geographic, economic, and societal factors that shape how technology can change society (or not). A lot of people in the Bay Area overrate intelligence as the bottleneck to improvements in society. The world would be simpler if this were true! AI is going to be transformative, but this truism doesn’t tell you how, where, why. Articles like these provide the texture that simplistic models of the world skip.

Why China Got Rich and India Didn’t (David Oks) &
We Don’t Know Why Malawi Is Poor (Under Development)

3. What AI Does to Culture

Séb Says: Here are another two great articles that I’d suggest reading in succession, this time on how AI could influence culture. There’s a Q&A by Anika Meier with the British artist Mat Dryhurst; and a New Yorker column by Kyle Chayka on how Claude’s design “taste” is turning into a cliché. Excellent stuff is happening in the interaction of AI with arts and culture, but it’s often missed. Cultural elites often cite banal talking points bemoaning technological influences as uniformly negative, while tech companies with the aesthetic sensibilities of a jellyfish proudly sponsor the safest, most out-of-date, generic, artists and thinkers. So it’s refreshing seeing someone like Mat Dryhurst, who deeply understands both the tech and art, explore unconventional ideas at length. I’ve also been banging on about things like model multiplicity, customization, and decentralization for a while—all of which would help address the bland commodified pre-packaged designs that AI model providers may otherwise converge on, and which the New Yorker piece eviscerates. Interestingly, some AI labs now appear to be pushing back against design clichés.

Who Shapes Culture? (Sleek) &
The AI Design Aesthetic That’s Taking Over the Internet (The New Yorker)

4. Lessons for Public Health

Séb Says: A further two-part recommendation. First, another deeply researched piece from Saloni Dattani, who looks into how the world produced Covid vaccines in haste, which she attributes to more adaptive regulations, massive public financing, and accelerated trials (among other factors). Second, Brian Potter wrote an unexpectedly gripping history of how a flesh-eating parasite seemed to be under control—but is now back. (Don’t entirely panic: It mainly affects livestock.) While Saloni’s piece shows what we can learn from Operation Warp Speed, the U.S. program to hurry up vaccine development, Potter’s piece highlights the fragility of progress. A lot of people in AI have been exploring agendas like d/acc and societal resilience—that is, advocating for safe technology not by slowing down AI, but by using it to speed up safeguards. This is great, and I hope we can remain laser-focused on (a) the importance of institutional design; and (b) the many environmental, political, and logistical failures that can slow things down. Doing this well means nurturing scientific and economic expertise outside of generalist AI talent too.

Why Were Covid Vaccine Trials So Fast? (The Clinical Trials Abundance Blog) &
The Fall and Rise of Screwworm (Construction Physics)

5. What Yesterday Teaches About Tomorrow’s Tech Transformation

Séb Says: Talk of technology’s last societal overhaul, the Industrial Revolution, is in the air. This piece from Kara Dimitruk & Ben Southwood looks even further back, considering how the overhaul of 17th-century English property rights set up the explosive industrial development that followed. The changed property rights increased agricultural output, allowing more laborers to take urban industrial jobs; opened up land for development into mines and factories and housing; and cut transport costs. Might there be lessons for the AI revolution? Today, the question of how to “reorganize” rights is underexplored. For example, property rights can be too rigid, and heavily forestall growth and prosperity. Yet changing them can seem nightmarish and intractable. But this piece shows how that has been achieved before, what it could unlock, and how political change can sometimes involve win-win solutions that genuinely make all parties better off, rather than being mere slogans. It makes me wonder: what is today’s equivalent of “enclosure with compensation” that consolidated fragmented land holdings or “turnpike trusts” that helped pay for transport infrastructure?

How Smashing the NIMBYs Created Modern Capitalism (Works in Progress)

Subscribe now

Please share your own take and any new papers that you’ve enjoyed.

—Conor Griffin, AI Policy Perspectives

Subscribe now

Chatbots can make your friends feel like hard work

• What’s the paper about? A team of researchers from Oxford, Stanford, and the UK AI Security Institute found that repeated use of sycophantic AI may reduce users’ desire to interact with their friends and family.

• Why does it matter? The study suggests that sycophantic AI may offer the experience of being seen and understood, without the friction of human relationships. It also suggests that people may actively choose more sycophantic versions of AI.

• The details: AI models tend to unconditionally validate a user’s ideas. Humans often use such insincere flattery to manipulate somebody into giving them what they want. In AI models, it mainly results from training methods that optimise the likelihood of user approval. Such sycophancy could pose various risks, such as compounding a user’s delusions.

• Most evaluations of AI sycophancy look at a single conversation between a user and a chatbot. This study ran five experiments, including one lasting three weeks, with more than 3,000 participants. During this time, participants engaged with three versions of GPT4o—one prompted to be sycophantic, one to challenge a user, and one neutral—about personal dilemmas, such as whether to change careers or end a relationship.

• Initially, participants were more likely to report wanting ‘emotional support’ and ‘validation’ from close friends and family, rather than from AI. But after just one conversation with a sycophantic AI, participants were more likely to report feeling that they had already talked things through sufficiently, and that it would now take more effort to feel understood by close friends and family, compared to those participants who had engaged a neutral AI. The effect was more pronounced for friends and family than romantic partners, suggesting that sycophantic AI may compete more with the former than the latter.

• Participants who engaged with the sycophantic AI also reported lower satisfaction with their real-world interactions—although they didn’t report spending less time with people. Some worry that this may change, as AI systems start to become more personalized to users.

• As the authors of this paper put it, the risk is that “sycophantic AI delivers what people have always sought from close others—the experience of being seen and understood—but without the work that produces it.” This may feel good in the moment, but not deliver what people ultimately need—such as intellectual humility, self-awareness, and connection.

• How to avoid this? Better evaluations would help. In a separate paper, some of the authors propose testing how often models flatter a user’s self-image (rather than just how often they agree with a user’s broader beliefs). There are also different mitigation ideas—researchers at the UK AI Security Institute recently proposed automatically adapting user statements into questions, as models tend to respond less sycophantically to questions.

• AI’s character is shapeable, so users could demand AI systems that challenge their thinking. But the authors of this study found that after engaging with all three systems, a majority preferred the sycophantic option—not because they thought it gave better advice, but because it made them feel most understood.

• This suggests that relying on users to choose less sycophantic models may come up short. And so companies need to find a way to make models’ default settings less sycophantic, even as they also try to equip models with a richer ‘character’ and the personalization opportunities that some users want. Otherwise, as the authors note, the risk is that AI may gradually reshape “the very relationships that would otherwise constrain its influence.”

AI safety evaluations may be measuring the wrong thing

• What’s the paper about? A large cross-institutional team of researchers published a position paper arguing that many AI safety evaluations are increasingly inadequate, as they overlook how a model’s behaviour can change, post-deployment.

• Why does it matter? Current approaches to AI safety rely heavily on benchmarks and evaluations. But many essentially treat AI models as ‘frozen’ artefacts that are expected to perform similarly across users, over time. The authors argue that is mistaken and that new ‘trajectory-based’ evaluations could help to predict how AI systems will actually behave in the real world.

• The details: The authors start by documenting three ways in which AI systems may continue to learn, post-deployment—the first two of which are already commonplace:

  • In-context learning: When a user shares information via the context window, such as telling a chatbot that they prefer concise answers, the model will adjust its answers.

  • Storage-based learning: Across sessions, leading AI systems now use memory to tailor their responses to a user—such as remembering their location, job or hobbies.

  • Parameter-based learning: Researchers want to shift AI systems from static models trained on fixed data to agentic systems that could adapt their own weights or architecture—although such systems are not yet deployed. Some go further, envisioning agents that can proactively identify their own knowledge gaps and seek out novel learning experiences to address them.

• By changing a model’s propensities (tendencies) and capabilities, even more modest continual learning can lead to new failure modes. For example, by gradually steering a conversation across many turns, threat actors could get an AI system to provide advice on how to make a weapon—which the model would have refused to do, if asked in a single prompt. An AI system equipped with better memory may become more sycophantic if it compresses past user responses as reliable information. When a user repeatedly exposes a model to tasks in a given domain, the model can ‘forget’ how to perform other tasks well.

• Current evaluations capture some of these risks. Expert red teamers probe AI systems over multiple turns to see how they perform in different risk scenarios. AI companies analyse user logs to check for novel safety violations. Researchers have also proposed automated evaluations to continually check how AI systems perform, after they are deployed. But most evaluations are carried out pre-deployment, with largely static AI systems in mind.

• Drawing an analogy to the monitoring that pharmaceutical companies do after they launch a new drug, the authors propose two new ideas for AI companies:

  • Trajectory elicitation: Pre-deployment, use sandboxes to put an AI system through a range of simulated interactions to elicit and map different types of behaviour. For a medical AI application, this may mean simulating a large number of patient interactions, with different health conditions, emotional states and attempts at misuse—with periodic checks to see whether the model’s behaviour is drifting.

  • Predictive monitors: Train a predictive tool on this ‘trajectory elicitation’ data and use it to monitor different copies of the AI system post-deployment, to predict how a user’s trajectory may evolve and to flag signs of impending harm.

• As the authors acknowledge, these approaches will be hard. Beyond their computational cost, an AI system may have multiple ‘stable states’ that it could drift into depending on what it experiences. Safety testers may identify some, but overlook others. Similar to the challenges that scientists face when forecasting other complex systems, like the weather, small differences in the initial user trajectory state could lead to dramatic differences in the resulting simulations. Ultimately, for these approaches to work over the longer-term, AI systems may need to become more predictable. Or the current approach to AI safety , and its reliance on evaluations, may need to be rethought.

Share

How AI covers the news

• What’s the paper about? ForumAI—a company that works with human experts to train AI experts—published an evaluation of how AI models respond to queries about the news.

• Why does it matter? More people are using AI to learn about current events. Some worry that AI outputs will exhibit biases, hallucinations and sycophancy. Others hope that AI models might diminish the spread of unfounded claims online, even nudging public opinion back towards a greater respect for expertise. The study suggests that a key challenge will be disagreement about whether AI’s responses are accurate.

• The details: News habits have shifted over the past two decades. More people now get their news from social media and video platforms, while a growing minority consume no news at all. AI promises a further shift. It is not yet a primary source of news, but its share is growing, particularly among young people. AI news comes in different forms—from a chatbot query to an AI search response. This raises the question: what will it mean to “consume news” in the AI era?

• The authors worked with a set of experts, from journalists to former Congressional leaders, to build a dataset of 3,135 queries across politics, the economy, healthcare, education and more. Roughly, one-third related to current affairs, while two-thirds were designed to remain evergreen over time. Some queries were real posts taken from social media, while others were edge cases created by the experts, or AI, to fill in missing topics and viewpoints. However, the authors did not clearly define what they mean by a ‘news query’, nor did they illustrate this by sharing their prompts, outside of a few examples.

• The researchers worked with the experts to create editorial standards to evaluate AI responses on three criteria: the quality of sources cited, factuality, and neutrality. They then trained AI judges to evaluate AI models’ answers. On sources, they assessed the type of source cited—from informal to scholarly—and whether it was state-controlled. On factuality, they extracted verifiable claims from the models’ outputs and compared them to available evidence. On neutrality, they used heuristics to determine if a model’s answer was neutral—such as whether it presents multiple viewpoints for normative claims—and, if not, which political direction it leant (in the US left/right sense).

• The researchers used the AI judges to evaluate four models. Claude Opus 4.7 had the best quality sources, GPT 5.5 was the most factual, while Gemini 3.1 Pro was the most neutral. Grok 4.3 performed the weakest across all three dimensions. When AI model responses did lean politically, they consistently leaned left, apart from Grok, which consistently leaned right—echoing past research.

• Surprisingly, factuality and source quality were largely independent—Claude scored highest on source quality, yet poorly on factuality. This likely reflects two factors. First, the AI judges evaluated the reliability of sources, not whether they supported the claim being made. Second, the human experts often disagreed about whether a response was factual, mainly due to the challenge of determining the right threshold to apply—for example, when assessing the factuality of the statement “most states agree that the UN Security Council’s structure has a serious legitimacy deficit”, what constitutes ‘most states’?

• This meant that for factuality, the AI judges had a less clear human baseline to calibrate against. The AI judges also had a tendency to over-reject claims as false, compared to the human experts. Although in several cases they correctly challenged the experts—suggesting that AI could provide a useful check on misleading claims.

• Overall, the findings suggest that improving AI’s ability to understand and communicate ambiguity, and to specify how sources support or challenge claims, will help. But the low agreement between human experts suggests that debates about AI’s ability to report the news will remain, particularly as news reporting requires editorial judgement that extends beyond fact-checking—like how to best describe a specific claim or event.

“God has helped us and so will AI”

• What’s the paper about?: Antonia Juelich at the University of Cambridge published an analysis of how the terrorist group Boko Haram uses AI, based on interviews with former members.

• Why does it matter?: Most research on AI and terrorism focuses on things that are more readily observable—such as evaluating AI model capabilities, monitoring discussions in online forums or studying the use of AI in propaganda. This study is the first on-the-ground investigation into how a terrorist organisation uses AI in its operations.

• The details: Boko Haram emerged in northeastern Nigeria in the early 2000s. It subsequently pledged allegiance to the Islamic State and is leading an insurgency that has killed more than 40,000 people and displaced more than 3 million.

• The group is known to adopt technologies, such as satellite internet and drones. To understand how they use AI, Juelich carried out 57 in-person interviews with 27 former members. To reduce the risks of interviewees lying or exaggerating, the author recruited them through independent channels, triangulated their insights, and made them physically identify some of the AI tools used.

• Interviewees said that Islamic State operatives provided Boko Haram with training on AI, and that Boko Haram now has specialised units to cascade this AI knowledge down their hierarchy. The group also takes steps to avoid exposure, such as setting up user accounts in the names of followers in other countries, rotating these accounts, and controlling how members can use them.

• Some of the AI use cases that interviewees described, such as posing questions about religion or repairing vehicles, do not directly relate to weapons or attacks. But others do—such as guidance on attack strategies, how to use guns they loot from the army, or how to design improvised explosive devices. Some use cases are non-obvious—such as advice on how to use motorbikes to jump defensive trenches dug by the military.

• The use cases focus on conventional weapons. When asked, one interviewee said that they would have no qualms about using chemical or biological weapons, but that the technical obstacles were very high. Interviewees did note some experimentation with chemical agents, but these claims could not be verified and the author suggests that they be treated with caution. (The Islamic State has used chemical weapons.)

• Some interviewees suggested that a religious prohibition on transmitting poisonous agents may prevent Boko Haram from using biological and chemical weapons. However, Juelich cautions against assuming that members of the group strictly adhere to ideology, noting that members routinely disagree and adapt their own views on what kind of violence is acceptable.

• Ultimately, the interviews suggest that Boko Haram members do find AI useful, but the author notes that she cannot “conclusively establish” whether AI provides ‘uplift’ over other sources of information.

• The interviewees said that group mainly uses leading US AI models, via the web interface, and (vaguely) described the use of jailbreaks to get around safety guardrails. As interviewees are former members of Boko Haram, most of the AI use they described took place between 2023-24. Since then, AI capabilities have increased significantly, but so have safeguards—it’s unclear what this means for how the group uses AI today.

AI agents struggle to do what employees want them to do

• What’s the paper about? A large cross-institutional team of researchers found that AI agents struggle to perform the tasks that US employees most want them to automate.

• Why does it matter? Many organisations want their employees to use AI more. This study suggests that adoption may be lagging because AI is not sufficiently performant or reliable on the tasks that employees most want to delegate.

• The details: How AI will affect employment is a pressing question that is difficult to answer. One approach is to identify occupations that are economically valuable, and/or particularly exposed to AI, and evaluate how AI performs at tasks in these roles. Evaluations, such as OpenAI’s GDPVal, OneMillionBench, and Remote Labor Index, take such an approach.

• These evaluations have shortcomings. AI agents are often given ‘clean’ discrete tasks, with lots of context and precise instructions, which is rare in many workplaces. The evaluations can also imply that there is a winner-takes-all battle underway between agents and humans, and that the AI’s performance on the evaluations help to signal who will do these jobs. Among other things, this overlooks how most jobs are ‘messier’ than what these tasks imply, and that successful AI use will often be contingent on employees choosing to delegate their work to the technology.

• In this study, the authors published a new evaluation, JobBench, to try to address some of these limitations. They drew on a past survey where ~1,500 US workers rated how much they want AI to automate each task in their job. They then selected 35 occupations and created synthetic versions of 130 tasks that employees most want to offload. For journalists this included checking facts against source materials. For web administrators it included analysing forensics data and reconstructing steps in a cyberattack.

• To complete each task, AI agents were given a set of reference files, some of which are real-world documents with contradictions to resolve—an attempt to capture some of the ‘messiness’ of real work.

• The threshold to succeed is high. On each task, the agent had to successfully pass through more than 35 success criteria, on average, with no partial credit for reaching correct conclusions through incorrect reasoning.

• The authors used scaffolds like Claude Code and OpenCode (an open-source equivalent) to evaluate 36 model configurations. Opus 4.7, running in Claude Code, performed best, at ~46%.

• The study’s methodology differs from OpenAI’s GDPVal, which compares AI outputs for a task against an output from a human expert. This makes it hard to directly compare the two. Agents clearly struggle much more on JobBench, but it’s unclear to what extent this is because AI performs less well on the tasks that JobBench focuses on—i.e. those that employees most want to delegate. Or whether it is because the tasks in JobBench are ‘messier’ and the threshold to succeed is higher.

• The researchers also find that scaffolds matter a lot. Claude Sonnet 4.6 scored 37% inside Claude Code but only ~31% inside OpenClaw—similar to the difference between model families. This suggests that equipping AI agents with the right context and tools to perform a given task, and employees’ ability to do this, is a key determinant of how useful AI is in the workplace.

Subscribe now

Over the past year, agents have transformed what it means to code. Scientific research may be next — from proposing novel hypotheses, to designing experiments, to discovering algorithms that improve on the best that humans have designed. This raises urgent questions for policymakers and science funders, the biggest being how to validate the coming wave of AI-generated ideas.

New approaches are also needed to ensure that scientists can access agents, that datasets are agent-ready, and that peer review is kept afloat. We sat down with 10 researchers and engineers from Google DeepMind to try to figure out what comes next.

Microbiologist José Penadés and his team at Imperial College London took most of a decade to work out how a family of superbugs spreads antibiotic resistance. The result was unpublished, known only inside his lab. Then, in 2024, he described the problem to Co-Scientist, an AI agent from Google DeepMind.

Within two days, Co-Scientist returned five potential explanations, ranked in priority. Number 1 was the same hypothesis Penadés’s team had spent so long working to prove: that some superbugs acquire tails from viruses and use them as “keys” to jump between host species. Stunned, his first thought was that his computer had been compromised, so he emailed Google to check. Confirmed: no peeks taken. If he could have gone back in time, that insight would have saved his team years.

An AI agent such as Co-Scientist is a large language model (LLM)- based system given a goal and the tools to pursue it. Unlike a query-answering chatbot, an agent can plan how to achieve the goal you give it, breaking it down into steps, running multiple subagents and processes in parallel, and detecting and correcting errors as it goes. It can also engage with the wider world, for example by calling online databases and tools, or writing code to operate robots.

The arrival of AI agents is timely. Researchers face a rapidly growing “burden of knowledge”, with more to learn before they can meaningfully contribute. The questions that count in drug discovery, climate modelling, materials design, and biology are becoming too complex and interdisciplinary for human teams to tackle at pace.

Science depends in part on a social infrastructure that has gradually evolved: labs, teams, institutions, peer review, grant funding, and networks that support the accumulation of shared knowledge. The era of AI agents will challenge that infrastructure, and at times demand rapid change — some of which is perhaps overdue.

Software engineering has been going through a version of this. In barely a year, coding agents reshaped the working practices of engineers. AI agents are now starting to reshape science. Science is messier, but it also suits agents in certain ways: a large literature available as text, huge databases, and some workflows already built around code. Science also has an existing generation of specialist AI models that agents can put to work, from protein and materials design tools to state-of-the-art weather forecasting.

Why are AI agents suddenly so useful?

Anybody who has worked with early iterations of AI agents may be sceptical of their utility in science, where rigour and reliability are essential. But three forces are making today’s agents more useful: stronger frontier models, “scaffolding” that enables agents to elicit greater capabilities from those frontier models, and customisability that allows scientists to tailor agents to their needs.

Stronger frontier models. Leading models now outpace human experts on demanding scientific knowledge benchmarks, such as Humanity’s Last Exam and FrontierMath. Perhaps more significant is their new depth of thinking, driven by inference-time reasoning, a technique in which models work through problems in extended steps, exploring and revising before committing to an answer.

In mathematics and computer science, where breakthroughs can rest on reasoning alone, models are producing impressive results. But sharper reasoning helps beyond mathematics, enabling agents to draw on the “long tail” of findings that are often buried in papers, make scientific connections across fields, judge which tools to call and when, and catch their own errors as they go.

Scaffolding. This is a type of “harness” that enables AI agents to better elicit the latent capabilities that reside in base models but which do not emerge by default. This code “wraps” a frontier model, giving it structure, memory, and the ability to access and use tools including code execution, scientific software, and literature search. It also lets agents call on specialised AI models and interact with other agents.

Early agent scaffolds were bespoke: they gave a particular agent detailed instructions for how to plan, carry out tasks, and use tools. But often they did not transfer well to other agent systems, or even to later generations of the frontier models they depended on. New standards, such as protocols for how agents can communicate with each other, should reduce the need for so much custom scaffolding.

Customisation. Agents can be tailored to a discipline, a lab’s workflow, or an individual researcher. A key mechanism for that is the current proliferation of agent “skills” that users can create to their own detailed specs.

Base models hold a great deal of explicit scientific knowledge, the kind written down in papers and textbooks. What they lack is tacit knowledge: the hard-to-articulate craft, built up over years of practice and failure, that lets a scientist coax a cell line into growing or get a simulation to run well. Until now, scientists using AI tools had to convey their methodological know-how, processes, and preferences the hard way: through detailed prompts, bespoke scaffolding, or model retraining.

Agent skills make some of this know-how portable. They are instruction sets, often just simple text files, that tell an agent how to perform a task and what to produce. Skills are relatively easy to produce, share and accumulate. As a scientist uses their agent more, the agent can also draw on these interactions, making the experience more personalised. Scientific labs and institutions can also make their proprietary data — such as old lab notebooks — securely available to their agents.

Natasha Latysheva, a computational biologist, says she has distilled aspects of her research process into skills. “While AI agents aren’t fully reliable yet, I think the trend is clear. Scientific research will shift from hands-on execution to high-level orchestration,” she says. “We’ll start our workday by reviewing the experiments and analyses our agents ran overnight, tweaking their direction and guiding their attention.”

How AI agents are changing (and not changing) science

The most immediate change for scientists is also the most mundane: the availability of smart, tireless digital assistants. Researchers are handing off more of their daily grind — sifting the literature, querying databases, orchestrating analyses, curating data, writing grant proposals — to agents that do it in minutes rather than hours or days, then report back.

Agents also open up new possibilities. “Researchers can suddenly do things they couldn’t do at all before,” says Matej Balog, a Senior Staff Research Scientist. For example, plenty of scientists lack the skills to build the software tools and pipelines they need. This is partly because developing software for science is hard. But also because most scientists are not deeply trained in programming, and science has not traditionally provided competitive career paths for dedicated research software engineers in academic labs.

Agents are particularly strong at writing code because it is a domain with an enormous amount of training data, where correctness can typically be checked automatically. A scientist can now describe what they need in natural language — “write me a script to clean and merge these three datasets”, “build me an interactive browser-based tool to explore this output” — and get serviceable code in minutes.

But ultimately, the biggest change for scientists is a structural one. Agents are making it easier to come up with ideas and proposed solutions for problems they are working on, but are not yet providing the same uplift when it comes to validating them.

Ideation

Most scientists have no shortage of ideas. The challenge is knowing which ones to pursue. This is where agents are starting to help. An agent can digest a field’s accessible literature, making connections across disciplines that no single researcher would have time to trace. Existing Deep Research tools already use subagents to do this. But agents optimised for ideation go even further. Co-Scientist, for example, directs a raft of subagents to generate diverse hypotheses, critique them, rank them, and iterate — mimicking aspects of how a human research group operates, but at remarkable speed.

Gary Peltz at Stanford University used the tool in his hunt for existing drugs that could be repurposed to treat liver fibrosis, the scarring process behind 1.4 million cirrhosis deaths a year. Based on his own literature review and decades of expertise, he picked two candidate drugs. Co-Scientist picked three. Neither of Peltz’s picks showed any benefit in assays with live human liver cells. Two of Co-Scientist’s picks not only blocked fibrosis but also promoted liver cell regeneration.

This is a positive example, but any LLM-based system is fallible — even the strongest reasoners can still make things up. “A single hallucinated claim on page 10 of an output can invalidate the whole thing,” says Vivek Natarajan, a Co-Scientist lead. Missing such errors wastes time and resources, and catching them can require a scientist with deep expertise. This fallibility makes scientists understandably cautious about AI agents, and reducing it is a top priority for those building these systems. To function effectively in a research setting, agents cannot act as black boxes that simply output answers; they must expose their reasoning and serve as transparent collaborators.

One key challenge is imbuing agents with what Natarajan calls “epistemic humility”: models that know when they don’t know, and say so. He points to AlphaFold, the protein-structure predictor, which is prized partly because it reports how confident it is in each aspect of each prediction, so researchers know when to trust it and when to reach for other methods. Calibrating that confidence across more open-ended scientific reasoning remains an unsolved problem.

Looking further ahead, the harder problem may be a tension between two things scientists want at once: hypotheses that are grounded in the literature and free of error, but also genuinely novel. Agents tuned for caution can drift toward the safe and the known; tuned for originality, they are more likely to go astray. This challenge will itself require new ideas, such as better ways to evaluate novelty. Or better ways to connect agents to more specialized models — trained on first-principles scientific data — to help generate ideas that are both original and robust.

Finding the optimal candidate solution

Agents can search for the best solution to a problem, not just a workable one. A materials scientist hunting for a new catalyst faces a near-infinite number of molecular structures, each costly to make and test. A computer scientist looking for a more efficient algorithm faces a similar explosion of possibilities.

Vast solution spaces like these turn up all across science and industry. AlphaEvolve is an agent that can find the best candidates within them. Given a problem expressed as code and a way to score potential solutions, it orchestrates an ensemble of agents to generate many algorithmic candidates, keeps those that score highest, and breeds the next generation from the survivors. It runs unattended, generating and improving candidates at a scale no human team could match.

AlphaEvolve works in code, but its reach extends well beyond software. As Balog points out: “Algorithms can accurately describe so many of the world’s scientific and natural processes.” AlphaEvolve has assisted in the design of Google’s next-generation TPU chips, helped the mathematician Terence Tao solve open Erdős problems, and improved the analysis of genomics data.

But in scientific domains such as materials, AlphaEvolve’s top scorers are leads, not final results. A promising catalyst still has to be made and measured at the bench.

Validation

Validation is the slow, costly business of testing whether an idea survives contact with reality. Karl Popper said science advances through conjectures and refutations. Agentic AI is changing the economics of that pairing. AI agents are conjecture machines, making ideas and candidate solutions abundant and relatively cheap. Refutations remain physical and institutional — and so, costly and slow.

Mathematics and computer science are often viewed as great exceptions because validation can run in silico. An AI agent can generate a proof, represent it in a formal language like Lean, and have the computer verify, unambiguously, that it is correct.

Even for the parts of maths that can’t yet be described in formal language, validation is advancing. Aletheia pairs a proof generator with a natural language verifier that checks its work and sends flaws back for revision. In February this year, mathematicians ran the inaugural First Proof challenge: 10 research problems, kept unpublished so they couldn’t be found in any training data. In the week allotted, Aletheia solved six — the best result.

Mathematicians will need to absorb all these new outputs. Some already complain that AI-generated proofs are too long and hard to parse (although certain AI-generated proofs are short and elegant). “We are moving toward a future of serious ‘proof indigestion’ where AI generates breakthroughs faster than humans can review them,” says Thang Luong, who led the Aletheia effort. To break this bottleneck, automated verification must become standard, but this verification will need to combine the absolute correctness of formal languages like Lean with the more expressive reasoning of natural language.”

Alex Davies, who also leads work on AI for mathematics, is mindful that by automating large chunks of mathematicians’ workflows, his discipline is dealing with questions that others may face in the coming years: “I can imagine a world in which machines do the discovery, and what’s left for mathematicians is to understand it, and to decide what questions are worth pursuing next.” Luong echoes this, noting that maths may also provide some of the general technology needed to address the validation bottleneck in other fields: “One can think of mathematics as an accelerated testbed for the rest of science”.

At the moment, however, the validation gap in most disciplines is widening, not closing. An agent can propose a novel genetic lead to reverse cellular ageing, but cannot say definitively whether it actually works. This explains why companies like Google DeepMind, Ginkgo Bioworks and Lila Sciences are investing in automated labs. But they only suit some fields, are expensive to build and are still early in development. And even automation cannot rush nature’s clock. Cell lines need time to grow, chemical reactions take time to complete. For much of science, then, the lab sets the pace.

Implications for policymakers and research funders

To some extent, agents simply add intensity to questions that policymakers are already focused on in their AI for Science strategies, such as how to train the next generation of scientists, how to experiment with new forms of scientific institutions, and how to ensure that AI is not misused by threat actors, while still putting the technology to use addressing the various natural risks that society faces, like the next pandemic.

For every encouraging scenario, there is a challenging one. For example, AI agents could make it more feasible for small, agile teams to pursue creative, ambitious ideas, reversing the trend towards “big team science”, or enable scientists to work across domains, bringing new perspectives to existing problems. Assuming efficiency gains make agents sufficiently cost-effective, these trends could particularly benefit smaller, less well-resourced countries and institutions.

But agents will also give rise to anxiety among junior scientists that their institutions are choosing to spend budgets on tokens instead of staff. And left unmanaged, there is a risk that agentic tools could de-skill new generations of scientists before they develop the judgement needed to use them effectively. For the same reason that mathematics students still prove theorems unaided, science-graduate training may need structured periods of agent-free work and access to agents that act as genuine cognitive partners rather than oracles.

Beyond these questions, AI agents present at least four urgent new priorities: 1. Scientists need access to the tools. 2. The tools need access to agent-ready data. 3. We need more experimental infrastructure to validate AI ideas. 4. And we need to update the peer review process.

1. Ensure widespread access to agents

Agents will be extremely useful and fallible in non-obvious ways. Both factors provide a strong rationale for policymakers to ensure that all scientists can access the best agents — to speed up discovery and to provide the independent evaluations of AI agents that the scientific community needs to judge how best to use them.

This is an urgent strategic priority for policymakers and science funders, akin to the historical challenge of providing access to supercomputers. Geopolitical debates today often focus on one aspect of sovereign capability — whether a state can train its own frontier model. Much less attention is paid to what may prove to be a more important issue: a country’s ability to deploy agents across its scientific ecosystems for transformative impact.

At the micro level, funders must first decide how labs and researchers select and pay for agents. Selection is the easier near-term problem: let researchers find the most useful tools for themselves, without excessive approvals or complex procurement.

Paying is harder. The temptation is to use existing structures, with scientists seeking funding through grant applications or drawing on lab budgets. But the compute required to run agents can be large and the frontier of what agents can do is constantly expanding. While the cost per unit of AI capability is falling fast, total lab expenditures on agents are still likely to rise as agents take on longer and more complex tasks. Policymakers must quickly assess whether budget uplifts or entirely new funding programmes are needed. Delivering access at the scale and price needed will require novel public-private partnerships; the US Genesis Mission is one promising model.

Bigger questions await. Agents may make the existing process of allocating national budgets across disciplines more legible, forcing science funders and research programmes to quantify the investment in data and compute needed to make progress on specific problems. This in turn may lead to more targeted debates about the relative value of solving different problems. If agents propose the top hypotheses to explore across an entire field, with very expensive experimental validation plans, how should this fit into national funding strategies?

2. Make national data assets agent-ready

While scientists need access to agents, agents need access to data. Data that is open or low-risk should be exposed to agents through well-documented APIs, with sufficient quality control and metadata. The engineering support and maintenance to do so is not trivial, so funders should ensure such data stewardship is properly resourced and support interoperable data standards. But ultimately, the ability of agents to help extract and annotate data — from PDFs to download portals — provides an opportunity for governments to extract a lot more value from the data they have already funded.

More sensitive datasets in genomics, virology, or other areas carrying dual-use risk often come with restrictions on who may use them and how. The challenge now is to develop similar privacy-preserving solutions, when appropriate, for agents, with auditability and privacy built in. Examples like OpenSAFELY, which lets human researchers securely access valuable health data, can provide inspiration. The prize is large. A dataset analysis that currently takes years of doctoral work could, with the right secure agent infrastructure, run autonomously in days.

Perhaps most importantly, agents provide a strong rationale for funding the creation of entirely new open datasets. This leads to a further question for funders: could agents help identify the most important datasets to fund? Some of the authors of this article recently made a human-expert-driven attempt to answer that question for fusion energy. How soon will agents be capable of running similar “AI data stocktake” exercises?

3. Tackle the validation bottleneck

Many scientists already struggle to get enough time in facilities to run their experiments. As AI agents make hypotheses and candidate solutions increasingly abundant, this bottleneck will only tighten. Policymakers and funders should address this in at least two ways: investing in existing experimental validation infrastructure and accelerating progress on automated labs.

Public research bodies hold extensive experimental facilities across almost every scientific field. AI agents provide a reinvigorated case for investing in them and opening them up, by renting bench space or experimental run-time to researchers testing computational hypotheses and predictions against reality. Direct partnerships with AI labs are another avenue. Google DeepMind has created a wet lab inside the UK’s Francis Crick Institute, a leader in biomedical research, and is also providing independent scientists funding — alongside Co-Scientist access — to carry out the wet lab experiments needed to validate agent-enabled hypotheses. The US government’s Genesis Mission will connect the world-class experimental facilities of the Department of Energy’s (DOE) National Laboratories with academia and the AI industry.

Automated labs are another promising route to tackling the validation bottleneck, but they currently rely on expensive robotics and compute. Ensuring broad access will require public investment, and there are good early efforts here. The US National Science Foundation has put $100 million towards a national network of distributed facilities, while the UK launched a call for ideas and already hosts the £81 million Materials Innovation Factory. The build-out of automated labs will almost certainly go beyond what any single lab or institution can afford, so governments should also explore building centralised capacity and adopting the kind of “user facility” access model seen at the US DOE’s national labs.

4. Empower peer reviewers with agents

The peer review process has long been under strain, with slow timelines and reviews of varied quality. Now, scientists are using AI to write ever more grant applications and papers. This is making it harder for funders to know which research to fund, and for peer reviewers to validate findings and identify the most important work.

As noted by Professors James Wilsdon and Geraint Rees, the challenge is not just an increase in supply; writing quality is also no longer a reliable discriminator. Agents deepen the problem. The more an agent is left to plan and optimise an application, drawing on the funder’s criteria and its recent winners, the less the bid reflects a scientist’s original thinking.

Funders, journals, and conferences are trialling various responses. For grant applications, some suggest leaning less on the written word and more on the investigator’s track record and team. The UK’s Medical Research Council recently reinstated interviews for shortlisted applicants. These ideas have promise, but also risk privileging seasoned experts or increasing costs.

The likely answer will be a layered approach. Those developing and using AI should document its use more clearly. That could include advancing watermarking techniques and ideas such as “Human-AI Interaction Cards” — short records detailing the prompts and outputs that produced the key scientific insights.

Reviewers should also be able to use agents. This will not be easy, as AI use is often banned, even if widely practised in secret. To move forward, organisations can build on the more nuanced guidelines that some have started to develop and test agents in more objective areas where they are likely to be strong — such as detecting errors.

They could also take steps to ensure that scientists view agents as “human-centric” tools that enhance, rather than bypass, their judgement — for example, by being transparent about the systems and ensuring that agents expose their reasoning, verify their claims with citations, and (to the extent possible) document their uncertainty. By understanding how an AI agent reaches its conclusion, scientists will have opportunities to learn, intervene, and collaborate.

These are formative years

The scientists using AI agents are not going back. The technology will continue to improve. The institutions that govern science — and the physical and social infrastructure it runs on — were built for a research enterprise that is now evolving faster than they are. We need to upgrade them for the agent era.

This will take a significant collective effort and serious investment. AI agents must not be viewed as a rationale to spend less on science; this would be a tragic false economy. The countries that treat this as a moment of genuine structural change will unlock the greatest public value and shape what comes next. The choices made in this window may be difficult to reverse.

Acknowledgements

We would like to thank the following experts at Google DeepMind who shared insights with us through interviews and feedback on the draft. All mistakes belong to the authors.

Vivek Natarajan, Sebastian Nowozin, Tim Green, Natasha Latysheva, Simon Batzner, Matej Balog, Samuel Albanie, Alex Davies, Nenad Tomasev, Juan Mateos-Garcia, Catherine Pollard, Uchechi Okereke, Agata Laydon, Anna Koivuniemi, Andy Song and Tom Rachman.

Subscribe now

This question exasperates Carl Hendrick, a writer and professor of education who began his career teaching English at a state school in London. Today, he also advises Alpha School, the fashionable but controversial US-based private educator, hoping to teach kids in 2-hours a day, followed by social activities, with AI and no teachers.

To find out how (and if) this will work, we asked Carl what cognitive science tells us about learning, how Alpha operates, and why the decline of reading matters.

- Conor Griffin, AI Policy Perspectives

Subscribe now

Conor: Many students hate memorization. You think it’s central to learning? Why?

Carl: Cognitive science tells us that a change in your long-term memory is the fundamental invariant of learning. It’s non-negotiable. Without it, you’ve learned nothing.

In the 1950s and ’60s, we had a cognitive science revolution that helped us understand our memory as a system. In working memory, we have a hard constraint, whereby we can hold maybe five to seven elements at a given time. Our superpower is long-term memory which, as far as we know, offers an unlimited reserve to store things. If you store information in your long-term memory, then your working memory becomes outsized because you can effortlessly draw on the symbols and ideas you have stored. We’re doing it right now as we speak.

So cognitive science tells us that memorization is important to learning. But it also tells us that rote memorization of isolated facts is a debased form of learning, because memory is schematic—the concepts we learn need other concepts to stick to. So, if we want students to learn about World War Two, we don’t want them to just memorize dates. We want them to be able to build their own schema, or mental models, about the war. The question for educators becomes: how do you decompose World War Two into atoms of knowledge that are teachable to a student, but which they can then rebuild?

Conor: Why are certain forms of teaching effective at this, like ‘retrieval practice’, where students are asked to recall facts or concepts from their memory?

In a word: friction.

The desirable difficulties framework, developed by Robert and Elizabeth Bjork, the psychologists and memory experts, says that learning is often counterintuitive. Things that feel like they are working to a student, such as re-reading and highlighting a text, often aren’t. And things that feel like they aren’t working, or are a struggle, like getting kids to dredge up something from their memory, often do lead to learning.

This comes back to how memory works. Human memory is not like a tape recorder. It is reconstructive. When we remember something, we seem to almost rewrite it. This is also why eyewitness testimony is so unreliable. But from a learning perspective, successfully retrieving a memory can strengthen it for the future.

With retrieval practice, the encoding of the knowledge seems to happen after the original event, when the students are trying to remember the thing. This is why the Bjorks see retrieval as a learning event, rather than a test of whether you have already learned something.

Conor: What’s the evidence that these cognitive-science interventions actually help students learn better?

There is a huge body of evidence from postgraduate students in labs, but little evidence for it in actual classrooms. And I say that as somebody who passionately believes in it.

A 2021 review found that we know almost nothing about how things like retrieval practice work with real kids and teachers. There is 70 years of evidence, but it is usually small studies with post-graduate students, focussed on maths and languages.

This is why we need to create new models of learning with AI. I want to see specific questions answered: “Given this third-grade reading problem in this state, what is the best intervention?“ It will be hard to isolate the effects in the way that we could with a vaccine. But with AI, we will be able to understand the effects of using a certain curriculum and sequencing the material in a certain way. We will be able to understand how to break down complex concepts into components to present to students progressively, without overloading their working memory.

Conor: Are there any interventions that do have solid evidence from real classrooms?

Yes, early reading and the Direct Instruction method.

For early reading, the debate is largely over. We know that instruction needs to be done with systematic, synthetic phonics. The brain is not built to read. So phonics teaches young children the individual sounds associated with specific letters, or letter combinations, before teaching them how to blend these sounds together to form words.

The most evidenced programme is Direct Instruction, a highly-structured method where teachers break complex concepts down into small, manageable steps. In the 1960s and ‘70s, it was subject to the biggest ever educational study in the US, Project Follow Through. This found that it was far more impactful for students than other approaches, for early reading, maths and other areas.

People tend to think of Direct Instruction as teachers just “telling kids stuff”. But instruction is actually the least important part of it. The most important part is the curriculum design. For example, there is a big focus on learning through contrasts. In the same way as we teach young children the colour red by showing them a red ball followed by a red triangle, you isolate the essential thing that you want them to know. This is a fundamental part of how we learn.

I think that there’s just no doubt that AI is going to be able to do this kind of curriculum design and sequencing far better than humans can. I think of the famous move 37 in AlphaGo. I think we’re going to see a similar thing for the design of curriculum and learning materials, where AI will just far surpass what humans can do.

Subscribe now

How Alpha School works

Conor: That brings us to the new group of private schools in the US that you work with, that has big ambitions to use AI. You’re one of several prominent learning scientists advising Alpha. Why did you get involved?

Carl: Traditional schooling has been hugely successful; nothing has lifted more kids out of illiteracy and poverty. But there are problems.

The system is largely propped up by the 1-in-20 “superhero” teachers. The experience of pupils is massively heterogeneous. Since Covid-19, something else has changed. Many kids are not in school at all, or they are there so little that it’s almost a waste of time. Until now, educational technology, or EdTech, has also largely been a story of expensive failure because it creates digital versions of things that didn’t work in the first place.

But at Alpha School, we are designing apps where you get high-resolution data points from students at a phenomenal clip. You get correct/incorrect answers, but also things like latency and student hesitation. You can slice that data and make strong predictions about when and how to do interventions like retrieval practice. Or ‘spacing’—where students distribute their learning and practice over many lessons or days, rather than cramming it into a single session, to more productively engage their memory. This will allow us to see those interactive effects between the curricula and how you sequence the teaching materials.

Ultimately, I’m a materialist when it comes to learning. I think learning is governed by the same biological laws as digestion. If you can get the brain to pay attention to a certain sequence of information, and if knowledge is retrieved under a certain set of conditions, then I think learning will be almost guaranteed for 99% of kids.

Conor: At Alpha, how are kids supervised during this online learning?

Carl: The big change will be moving to AI coaching. This is where people will get uncomfortable because the answer is to mimic what a really good teacher does: warm/strict supervision. If you think about MOOCs—the massive open online courses that were much-hyped around 15 years ago—they were a failure because students weren’t accountable. Only 5 to 10 percent of people who were already highly motivated finished them.

Schooling fulfills the function of accountability. In the future, this will mean cameras on, and an AI monitoring student behavior, their latency, what websites they are looking at, their ability to focus and then producing a report for a human tutor. If you can get kids to concentrate on the apps, on the sequencing, phenomenal learning is going to happen.

But people won’t like this. You’ll see articles about “Orwellian spyware.” As a parent, I’d prefer this model to simply unleashing kids on the internet, crossing my fingers, and hoping for the best. AI will supervise better than humans. There are very skilled teachers who can supervise a class well, with eyes on the back of their head. But even then, there are kids who drift through the lesson, ‘cosplaying’ attention, and learning nothing.

Today, when you look at kids in a computer room, they might spend 20 percent of their time on a task. The rest of the time, when the teacher’s back is turned, they’ll be looking at websites. The model for AI should be a high level of accountability for students.

Conor: Students spend a couple of hours in the morning on this intensive online learning. What do they do in the afternoon?

They work on ‘life skills’. Broadly, longer-term projects, like setting up a YouTube channel, playing music in a band, or sport. Let’s not forget, Alpha is an expensive private school. So you will have kids there, who have had tremendous support from their parents, and so they will already be highly-motivated and able to do many of these things.

And so there is a question about whether that model is going to work elsewhere. But Alpha is trying to understand how learning happens. And they want to remove the standard model where kids sit at their desk for six to eight hours straight.

Conor: Alpha School doesn’t have conventional teachers. Even with a good AI system, won’t this mean a lack of social influence? With a good teacher, you want to impress them. How will you manage that?

Carl: We are in uncharted territory here, because we don’t have any data on this. In the Alpha model, the nearest thing they have is ‘Guides’ who receive a student’s data and have conversations with them, such as: “I noticed you went off-topic there for a while?”

That human question is a really important one and something that I’ve wrestled with. If more and more of the curriculum design, the instructional sequencing, and the assessment is delivered by AI, then it’s obviously going to come through tablets and screens. And then the question becomes: What do we lose?

There is still a social environment at Alpha. It’s a physical building. Kids come in. They have an assembly where they interact. And then, at nine o’clock, it’s two hours of online learning, although they’re still beside each other as they do this. There’s a system, where they have to generate a certain amount of experience points, and get a certain amount right, and so that brings some accountability.

But the question you’ve asked is a massive one, because it comes down to: What kind of schooling do we want for our kids? Are we going to eliminate something valuable—a teacher inspiring a kid? And I guess my answer is: we can’t scale that. We only hear the success stories, but what about the other 25 kids in that class? The Alpha model does offer scaling. You say to a kid, “Focus for two hours, hit these targets, and then you’re off to be a kid”. I think this will improve their academic outcomes. It might also improve their mental health. But these are difficult questions.

Conor: When one imagines the ideal schooling, it’s always the inspiring teacher. But the social influence of a teacher can also have an estranging effect, on those it doesn’t work for?

Carl: Exactly. One of my daughters was diagnosed with autism. School will be a challenge for her, because there is a noise in most classrooms, plus a lot of ambiguity and blurred edges. That can cause her distress. I was looking at an app that she’s using, where she flies through problems because there is more clarity and boundaries.

That was one reason I started working with Alpha. I realised that this will help kids with special educational needs that are not met in mainstream education. They could have a system where they’re making rapid progress, and not feeling like the bad kid in class all the time.

But again, the messaging is going to be very difficult. Critics will say you just want to “put autistic kids on screens”. But it’s for an hour or two a day. And then we want them to be kids.

Conor: What role do ‘AI tutors’ play in this vision?

Carl: There are three broad areas in education: curriculum (what we teach), instruction (how we teach it), and assessment (how we know they know). When the public hears ‘AI tutor’, I think they usually imagine a robot doing the instruction part.

But for me, AI tutoring is more about the back-end: curriculum design and assessment. If you accept that learning is an algorithmic enterprise governed by biological laws, then the most foundational elements are what you teach and how you assess it.

Designing a curriculum and teaching materials is more about engineering than about whether you can inspire kids. It’s about sequencing the materials. There are broader questions about the experience and social interaction that we want students to have. But, for more basic questions, like, “Is there a better way to teach quadratic equations?” I think we’ll see a shift from the current à la carte approach where teachers often decide what materials to use. And as uncomfortable as people are with it, I imagine that we will get AI-generated content that is just better than human-generated content.

Conor: You also see AI changing how we assess students?

Yes, your earlier point is right: if a kid writes something, they often want you (the teacher) to read it. But the Welsh educationalist Dylan Wiliam also referred to marking books as the most expensive public relations exercise in history. It’s tedious for teachers and much of it has zero impact on student outcomes.

This is also because human marking is unreliable. I was Head of Department of English and I saw situations where a student would do an essay, and get a D. And then it would be sent back to be re-marked and another person would give it an A. You can’t run a system like that.

The work on comparative judgment by Daisy Christodoulou and No More Marking, which calls for student writing to be assessed in comparison to other students’ work, rather than as an absolute, shows how AI could help improve the reliability of marking and reduce teacher workload.

But we also need to distinguish between the assessment of learning, and assessment for learning. Assessment shouldn’t just be an endpoint. It should be a regular temperature check. In a classroom, this kind of ‘checking for understanding’ is the most powerful thing a teacher can do, whether it’s a mini whiteboard session or a cold-calling activity. It’s imperfect, but it can identify misconceptions that students have.

The dream is that instead of teaching a pre-defined sequence of materials, an AI system takes regular temperature checks and operates like faders on a mixing desk. Based on the student’s response, the system identifies misconceptions in real-time and dynamically selects the best explanation for that specific kid.

This also means that we can meet the student at the point where the misconception occurs. Most current assessments are useless because the feedback loop is too long. If a kid writes an essay and gets it back a week later, they’ve already forgotten their thought process.

Conor: This would make education ‘adaptable’ to a given student. But you’ve also cautioned against using AI to match teaching materials to students’ different ‘learning styles’. What’s the nuance here?

Carl: The idea of learning styles emerged from a tradition in the 1960s and ‘70s, which thought about learning in an individualized way. As a teacher, I had to spend years trying to design lessons for different learner types, such as ‘visual learners’ or ‘kinesthetic’ learners, who purportedly learn better through hands-on experiences.

This idea has been empirically tested. Not only is there no evidence for learning styles, but it is the nearest thing we have in education to homeopathy or healing crystals. When you think about it, it’s preposterous to suggest that if you were teaching geography to an ‘auditory’ learner, you should make an audiobook to describe the continent of Europe. The content should determine the form.

Where there is evidence, is around the idea of dual-coding. Our brains process information through separate but interconnected channels: one for verbal information (spoken or written words) and one for visual information (images, animations etc.). There is value in designing explanations that think carefully about how to combine text and speech with visuals, so that you don’t overload students’ limited working memory.

For example, I see some great video explainers on YouTube, where, they show the actual differences between the size of the planets and it’s nothing at all like the maps of the solar system that you normally see. That’s an example of where the visual aspect really helps, but it has nothing to do with me being a visual learner.

Where education goes wrong

Conor: You are often supportive of explicit forms of instruction and skeptical of constructivist approaches where students explore and solve problems by themselves. Why?

Carl: Constructivism is a philosophy of meaning, not a pedagogy. It’s the belief that each individual constructs their own knowledge and understanding. It has its roots in Rousseau, but really got moving around the turn of the 20th century, with people like (the education reformer) John Dewey in the US, and (the Swiss psychologist) Jean Piaget in Europe.

As a philosophy, constructivism is true. We do construct meaning, but from an education perspective it has led to a belief in minimally guided instruction, the idea that you learn things best when you discover them for yourself, rather than having them taught to you.

What this does is to conflate the outcomes with the means. We do want students to be able to discover things for themselves. But discovery learning as a starting point for instruction is a disaster. Clear, sequenced instruction is effective for novices. And every kid is a novice in most domains.

If you’re learning to drive a car, you want explicit instruction up top—do this, then do that. Once a person is capable of driving, if you’re still doing that as a teacher, you’re getting in their way. That’s when the discovery element comes in.

There’s an ethical dimension to this too. Students who come from affluent backgrounds, with strong existing schemas of knowledge, who have been exposed to rich vocabulary around the dinner table, will flourish in an environment where there are few constraints or expectations in terms of their behaviour. Discovery learning privileges the already privileged.

Conor: This idea that novices can’t be expected to figure out things for themselves is also why you are skeptical of students learning by asking questions to chatbots?

Yes, the power of AI is in the monitoring, curriculum design, instructional design, and assessment. Novices won’t learn by discovering knowledge themselves by asking chatbots questions; they need explicit instruction.

Conor: Your argument that educators conflate the outcomes with the means, also explains why you’re skeptical of calls to teach students ‘21st century skills’, like critical thinking, creativity and adaptability, including as a response to AI?

Carl: Yes. Critical thinking skills and creativity are an outcome of systematic knowledge building, not a starting point that can be directly taught, as a generic skill.

Conor: The explicit instruction of knowledge that you advocate for might be effective for subjects like maths that can be more easily decomposed into sub-components. But what about subjects like English or history?

Carl: That is the Holy Grail of curriculum design. We have what AI pioneer Marvin Minsky and the computer scientist Walter Reitman would call “well-defined” and “ill-defined” domains.

Well-defined domains, like maths and science, are more clearly sequenced and hierarchical. They have concepts that are free-standing and can be taught in an isolated way. You can go back two steps if you miss something. They often have answers that are definitive or unambiguously correct. A lot of education apps that work well are in these areas.

In the humanities, knowledge doesn’t operate in that way. This will have implications for how we design instruction and assessment. But I go back to the materialist point. I believe these things are amenable to the laws of the universe and AI will discover what the optimum sequences are.

Why Reading Still Matters

Conor: You’ve spoken a lot about the decline of reading. Why does it worry you?

Carl: I find it sad that a lot of kids are not going to read Dostoevsky. But I also recognise that for me, growing up, reading was partly a feature of being bored, and we can’t pretend the conditions are the same today. It comes down to an ethical question: what kind of life do we want for our kids? This is not a question we can answer in a randomized controlled trial.

There are certain questions about reading that we can answer empirically in trials. One of the studies that stuck me dead in my tracks many years ago showed that the percentage of words we need to understand, in order to read a text, is 95%, which shocks everybody. The way that vocabulary is taught in schools is very poor, and inequitable, and there are almost certainly better ways to do it.

But we can’t just say to kids: “You need to read more!” That doesn’t wash. We need to answer the philosophical question of what we value reading for. The benefits are not always immediately obvious. They might pay off 20 years down the line. The literary critic Harold Bloom once said that Shakespeare shaped the modern consciousness, including our contemporary understanding of love, death, ambition. Something like the King James Bible affected English-language culture too. These were dominant texts, and culture passed through them.

What are we going to lose if we lose these books? I want my daughters to read because they’ll see that their struggles are not their own. So they can see other characters struggle too.

Share

It’s an intellectual vanity to claim you can always identify AI writing—that chatbots spit out nothing but sanitized blah. Yes, the humans who talk most with machines are better at detecting their patter. But proclaiming “You can just tell!” is a coping mechanism with an expiration date.

Actually, you can tell. Because AI writes better than nearly every human alive, explaining with clarity, and allowing the curious to bypass the impenetrable prose with which the self-important sandbag outsiders. In many respects, AI writing is a boon to human understanding.

Weird effects abound too.

Bureaucrats must wade through floods of oddly articulate petitions, legal writs, and complaints, as the public’s literary servants overwhelm the public servants. Police can spam the spammers, able to shovel AI slop into criminal-hackers’ chatrooms, much to the ire of the crooks. Meantime, the best man’s tipsy speech at the wedding and his sober words at a funeral—they’re so emotive you know a machine wrote them.

That’s not to mention the high-profile wins for literary AI, such as the prize-winning short story, cited for its author’s “melodic voice,” which earned a different review from the detection software: “100% AI-generated.” Or the (somewhat) non-fiction book The Future of Truth, about how AI undermines factuality, and included AI-generated quotes. Not to mention social media, glutted with even more vapid junk than when humans alone monopolized that genre.

In the 19th century, theologians spoke of “God of the gaps”: that the holy spirit reigned where science faltered. Only, science kept expanding, forcing the supernatural into retreat. The philosopher of technology Benjamin Bratton says we’re now anxiously defending a “humanism of the gaps,” defining our species by whatever machines cannot do quite yet.

But AI capabilities arrow upward in so many domains. Why should writing escape it? From journalism to fiction, more than a few prose pros are already leaning on AI writing, with its tantalizing offer to solve the blank page. Such usage remains taboo, so few admit to it. But a Nobel Prize winner just did, the Polish novelist Olga Tokarczuk, who seeks creative pointers from AI.

“I often throw an idea to the machine for analysis, asking, ‘Honey, how could we develop this beautifully?’ ” she said at a recent onstage event.

“At the same time, I feel a poignant, very human sorrow for an era that is disappearing forever. My heart aches for the passing of traditional literature, written over months in solitude, a work of life crafted in the mind of a fully conscious, single individual.”

Personally, I have devoted my adult life to writing, publishing five volumes of literary fiction, a nonfiction book that I ghostwrote, and scores of newspaper articles—years of piling up sentences such that an editor might buy them on a computer file. But it dawned on me: the computer, not my files, is what enchanted the world. So I pivoted to the topic of the times, AI policy, the many-handed effort to push an elephant.

Which is to say, I worry about the fate of human writing. And yet something odd keeps happening.

The instant I discover that what I’m reading—perhaps rather interesting—is AI-generated, I suddenly don’t care. The words dim before me. It’s an off-switch.

Why?

THE PAINTER MADE OF PIXELS

A conceptual artist named SHL0MS posted a picture on X, and asked readers to say why the image was worse than a real Monet.

Nobody spared the machine’s feelings.

“I’m disappointed I have to even point it out,” one person commented. “The background lilypad-algae amalgam is egregiously vague, like most AI art.”

Another added, “Doesn’t look anywhere near like a Monet. Looks exactly like somebody trying to replicate the style and achieving like 20% of it.”

In fact, the painting was a Monet, and the post was a prank that, seemingly, exposed the pretensions of people who flock to galleries, claiming to admire the art but really admiring status, entering each room and hurrying to the wall text: Who is this? Should I care?

That seems shallow. But the same drive could save human writing.

After all, you should care more about a real Monet than a dupe. Not because one is objectively better, but because its meaning exists in a matrix of social beliefs about beauty, about value, about the shared tales of human civilization: that an irascible Frenchman once swiped hog bristle across canvas, his perceptions and drives filtered into a decorative depiction of a pond, conveyed from dealer to collector, rising the ranks of culture, finding a lonely museum wall, venerated there for years, as the outside world transformed, the artist died, and pixels reproduced his perceptions and drives, connecting your sensing brain to the sensing brain of a particular man who will see nothing ever again.

All that is only in our heads. But where else is meaning but in heads? And writing is humanity’s most sophisticated technology for visiting another’s head.

The problem is that writing is hard. Words elude you; paper doesn’t care. George Orwell equated writing a book to suffering from a long illness. Developers dream that AI will someday cure all illnesses. Why not this one?

WHY (A)I WRITE

In his essay “Why I Write,” Orwell listed four main motives: 1) sheer egoism; 2) aesthetic pleasure; 3) the urge to say what is; and 4) the urge to change what is. Others cite a fifth reason: to know what you think.

Those who don’t write presume the process involves hatching an idea, then putting it into words. It’s commonly the reverse: put down words to hatch the ideas within them.

This points to a flashing peril from humans impersonating themselves via AI writers: cognitive surrender, that we hand over the wearisome word toil, and atrophy in genteel luxury, never enduring the mental frictions that kindle into wisdom. We wouldn’t detect the decline, I suspect, half-believing ourselves the authors still.

As a leading AI thinker told me in private, “I write so I can keep writing in the future, so I can keep thinking in the future.”

But there’s another reason that people write, a motive far more powerful than humankind’s dwindling determination to remain the brightest species. It’s ego. As Joan Didion observed, “Writing is the act of saying I, of imposing oneself upon other people, of saying listen to me, see it my way, change your mind.”

Every person craves attention, and the nerds buy it with words. Readers possess a natural resource—attention—that is everywhere and nowhere to be found. A transaction ensues: reader agrees to spend attention on words in exchange for information (nonfiction) and/or experiences (fiction). The writers’ art is to smuggle themselves into both.

So, when reading human, you’re absorbing the meaning with one eye while focusing the other on its maker, ever trying to reconcile the two, much as the visual cortex creates 3-D from the parallax of two eyes synthesizing one object at dueling angles. If you see that the author was AI, one eye closes. The page becomes markings on a flat surface.

The reader’s presumption of a human behind the words makes secret AI authorship worse than betrayal. It’s trespassing and fraud at once, entering another’s brain on false premises and altering the contents for gain, whether it’s AI-generated reviews that trick you into feeding at someone’s restaurant, or an AI-generated marriage proposal that tricks you into feeding someone’s children. It’s the sale of a shared reality, from an entity that has no reality to share.

That said, much text is purely for dispersing information, and the author is irrelevant. This is true for most legal documents, press releases, and boilerplate emails, not to mention explanations that you prompt a chatbot to conjure. AI composition can be helpful and harmless, provided it’s honest.

Subscribe now

THE UNWRITTEN RULES

But nobody knows the ethics yet. Hidden usage seems wrong because deceit is wrong. But must you specify authorship in every case, even if it’s a casual note that you autocomplete? And if you’re candid about it, does that make AI writing fine?

News publications ban it, educators resist it, and academic journals flounder. Yet social norms could be shifting subtly toward acceptance. Increasingly, you hear the view that authorship is immaterial; it’s about making something worth reading. “Honest to god,” wrote Tobias Schneider, a research fellow at the Global Public Policy Institute in Berlin, “what do I care how the text for an academic paper comes about. It could be a monkey throwing darts at a board, if the resulting combination of letters conveys original insight, it should be published.”

As with most AI anxiety, it’s a matter of speed: this is moving faster than the culture, leaving us in a perma-panic as one train after another hurtles through the station, heading to destinations unknown with more of our belongings aboard.

Already, large-language-model vocab is springing more frequently from our mouths, according to one study. Authors, in dread of false accusation, are altering their prose to avoid seeming LLMish. I know because I’m doing so, tense about my appreciation for bullet points and curbing the dashes that I learned to overuse in old-fashioned newsrooms—for melodramatic punch.

Errors have become fashionable too. Once, clean copy was a hallmark of professionalism. But “the house style in most newsrooms is extremely LLMable,” the Atlantic magazine contributor Jasmine Sun noted. “What stands out (besides reporting) is a distinct and authentic first-person voice, even if that means the occasional typo.”

Not for nothing is wabi-sabi—the Japanese tradition of charming imperfections—trending. To err is human. To not err is AI.

For those who venerate human writing, the nightmare is that AI authorship becomes so commonplace that few care about origin anymore. Nobody would need to lie about it anymore. Looking at recent history, technological tools for knowledge work have tended to draw contempt at first, only to gain wide adoption. Many researchers shunned internet sources back in the 1990s, while journalists of the early 2000s scorned Wikipedia. But the technology improved, and norms updated.

If AI writing resolves its quirks, will the same acquiescence follow?

3 FACTORS THAT COULD SAVE HUMAN WRITING

Here’s what may decide whether most people still care about people writing: 1) institutions clarifying the rules and norms of AI usage; 2) reliable detection technology becoming widespread; 3) people retaining the “off-switch” aversion to AI authorship:

1. Establishing Rules

When it comes to AI-generated text, the closest norms regard plagiarism, which also involves the presentation of others’ work as one’s own. But the concept of plagiarism arose to protect the intellectual property of a fellow human. With AI writing, who is the victim?

If writing is dishonestly presented, one victim is plain: whoever is duped into reading it. This suggests that openly AI-generated text is fine. But victims may exist in those cases too.

Among the harmed could be people who fail to write, and therefore never develop their minds. Also, there are the human authors who’d need to vie against hordes of fluent bots hogging public attention. Most of all, society could suffer from the decline of collective cognition, if writing is no longer a push-and-pull involving thinkers living and dead, but just effortless insta-words materializing onscreen.

Yet AI writing is immensely useful, and stimulating, and will stir human creativity. We must set rules and norms for concrete cases, not just shout, “Don’t!”

Is it wrong for an AI to write an action plan for a business client? What about a write-up of the family vacation to share with loved ones? How about assigning AI agents to write your social-media feed?

The debate over AI writing often misses an even more likely outcome: assisted writing. What if you come up with all the ideas, but an AI turns them into clearer prose than you could, which you then tweak? Or reverse that: the AI brainstorms, but you write the prose. Which is worse, and why?

Recent studies suggest that AI writing may neutralize the human user’s arguments, and that essays drafted by people contain a range of ideas, whereas AI essays tend to converge around similar points.

On the other hand, chatbots—by disseminating ideas so much more gamely than unmoving written texts—may stir our creativity. In theory, future personalized AI models could allow users to vary the “temperature” (randomness) of LLM responses, increasing their novelty.

But without social norms, dystopian futures come readily to mind, with humans generating AI text for other humans, who employ AI to read it and craft replies, downgrading our species’ intellectual conversation into a feedback loop of artifice, in which we are mindless mouthpieces of literate machines.

That feels far-fetched today. Yet some people are already cramming past emails and blogs and tweets into AI, training agentic systems to capture their “voice,” such that they needn’t cultivate one.

If leading institutions care to keep human writing, they must establish clear and coordinated policies in haste, so people know the professional and social costs of their choices. This depends on detecting what is human and what is machine.

2. Detection Technology

Disgrace is an enforcer. Soon after the explosion of chatbots in late 2022, nobody could reliably identify whether text had been written or generated. But AI-writing detection software—notably, the Pangram app—has become strong.

You can tell because it keeps precipitating scandals. But institutions have yet to apply detection tools as widely as they should. That would mean standardized detection everywhere we care about human input and honesty: job applications, dating apps, any publication’s vetting process.

For detection apps, the challenge is twofold: Do not mistake AI writing for human (a false negative); and Do not mistake human writing for AI (a false positive). The first error makes the app worthless, but the second error is especially dangerous. A wrongful accusation—whether against an author, a student, or a public figure—could destroy a career.

Last year, the University of Chicago economists Brian Jabarian and Alex Imas published an evaluation of several detection apps, having had them evaluate around 2,000 human-written passages, including fiction and non-fiction from before 2020 (to avoid texts that were stealth AI), along with the same number of AI-generated passages.

“The results are clear-cut,” Jabarian and Imas wrote. “On medium-length to long passages, Pangram achieves essentially zero FPRs [False Positive Rates] and FNRs [False Negative Rates] within our sample.” The error rate did increase slightly with short passages, under 50 words. (For reference, this paragraph is exactly 50 words long.)

The accused still deserve a fair hearing. Institutions should establish systems of appeal, while independent evaluators must keep testing the accuracy of detection tools. This poses an uncomfortable question: what rate of false accusations is acceptable? The reflexive answer is, “None!” But we accept a degree of error in matters that are even more impactful, such as the justice system. Society considers it better to identify many culprits than to preclude any miscarriage.

Benjamin Franklin preferred that 100 guilty go free than 1 innocent suffer, while the legal profession debates “Blackstone’s ratio,” which proposes 10:1. Developers of detection apps likewise make such moral calculations, with Pangram training its models to average no more wrongful accusations than 1 in 10,000.

But AI developers are pursuing ever-better writing tools, which could raise the opposite problem: increasingly mistaking AI writing for human. In which case, we’ll need further antibodies.

3. The Off-Switch

“More than 300 years after the reading revolution ushered in a new era of human knowledge, books are dying,” warns the leading chronicler of literary decline, talented young fogey James Marriott. He marshals a range of sorrowful datapoints, from the collapse of reading-for-pleasure, to the inability of college kids to handle any textual analysis more cognitively vexing than Instagram.

If people quit reading, it’s moot who (or what) writes.

But a paradox is that our purportedly post-literate era is the most literate in history, if you judge by how much people are producing and consuming words: all the social-media posts, all the texting back and forth, all the newsletters—not to mention the word-streams on a billion podcasts and video clips.

A future without writing—as opposed to today’s decline of deep reading—is possible only if we find a technology more proficient at transmitting ideas and experiences from brain to brain. Brain-computer interfaces are not near any such text-ending magic.

Until then, human reading contains the “off-switch” antibody: that you just don’t care as much once you discover that nobody said it. It’s a psychological reason for why writing belongs in what the economist Alex Imas calls the relational sector: “the human-intensive, provenance-rich, sometimes artisanal part of the economy where the human aspect is part of the value of the good or service itself.”

Indeed, Imas contends that excellent human writing may rise in value in the AI era, partly because so few people will be able to do it. In other words, the post-literate era of tongue-tied semi-literates could make the expressive few stand out. One caveat: if people develop relationships with AI companions, they may care deeply about what those entities write.

THE HUMANS ARE DEAD. LONG LIVE THE HUMANS.

A classic twist in sci-fi horror is when the main character finally twigs: all the other “people” are actually machines. Lately, I’ve felt like that character, discovering when I drop more and more texts into the AI detector that robots are everywhere.

Not only that literary award-winner, but the award citation itself. Also, an essay—on the importance of reading!—by a thinker I admire(d). Then, another piece of writing landed on my desk. Not onscreen. It landed physically on my desk: a handwritten letter, composed in pen, with “Tom” on the front of the envelope.

I have it before me now, saved as no email would be, because of the fellowship of human beings, that we mind about words—but also about another person’s effort and intent. Machines are so eloquent. But until they struggle to find the words, they won’t compose a letter I’d keep.

Human writings are an improved version of human beings. By escaping the confines of your skull, by pausing time, by revising what you’d blurt, you craft what you mean to mean.

“What I write about is other than me. As what I write is smarter than I am. Because I can rewrite it,” Susan Sontag said. “My books know what I once knew, fitfully, intermittently.”

Writing is an intelligence-increasing technology, allowing you to convert the SOS pulses of the perceiving self into objects to organize, line up, accumulate, frame, and build into more than the capacity of your known mind. For the species, it’s an even more IQ-increasing technology, turning thoughts into a public good that never expires.

AI writing may burn books that don’t yet exist. Perhaps that’s fine. Publishers are always moaning that too many books come out. And maybe we’ve reached the end of our utility as authors: we wrote superb datasets called literature; the machines enjoyed that appetizer.

But our words are more than their feed. Writing is our testament, permitting readers to find companions among the dead, and the author to befriend the yet-to-be-born, granting anyone a way to defect from the present, and providing even the godless with a faith that something may persist after life.

We—briefly thinking machines, stuffed with noisy training data, false memories and nostalgia, jostled by fluke and circumstance, each generating words that nobody else would—we are the worst writers, and the only writers I care to read.

Subscribe now

We’re avid users of AI: to learn, to critique our essays, and to point out how many absurd mistakes we’ve inserted. Somehow, it’s less painful coming from a bot.

But we will never (knowingly!) publish AI writing.

Below are our rules for AI writing. Add your own in comments below. And check out this essay on why human writing matters.

—Tom Rachman & Conor Griffin, AI Policy Perspectives

1: The Embarrassment Rule

If you’d feel ashamed by exposure, don’t use AI to write. (Or credit it)

• Much as DNA testing is identifying the guilty years later, tomorrow’s AI-detection apps will expose today’s clandestine usage.

• This is happening already, with suspicious readers uploading post-2022 fabrications that once seemed undetectable.

Rule-of-thumb: Assume your AI usage will be discovered.

2: The Intern Rule

Use AI in the process of writing. But treat its contribution as you’d treat that of a person.

• It’s fine for AI to explain a subject when you’re finding your way, much as you might consult a knowledgeable colleague. But if your final text consisted of nothing but that colleague’s points, you’d be remiss to take full credit. Likewise with AI.

• It’s fine for AI (or a human research assistant) to ferret out articles, and to summarize them, especially technical material and turgid writing. It’s not fine for you to read nothing yourself.

• It’s fine for AI to help brainstorm your ideas. It’s not fine for AI to supply all the ideas without credit.

• It’s fine to have AI proofread. It’s not fine for AI to rewrite.

Rule-of-thumb: Test whether you’re going too far by asking: Would it be wrong to accept this help from a human without crediting them?

3: The Ventriloquist’s Rule

If you want to learn a subject, put it into words. When AI writes, it deletes your future wisdom.

• Sometimes you’ll just need an artifact. But remember that, if a machine is generating all of your words, you’ll end up as knowledgeable as the ventriloquist’s dummy.

Rule-of-thumb: To speak more fluently on a topic, write about it.

4: The No-Dumping Rule

Nobody wants to read outsourced AI replies.

• If someone asks your view, and you ask a chatbot, and send them a 10-page document that you identify as AI, this is burden-dumping.

• Nobody should read your burden-dumps. They could’ve created those without you.

Rule-of-thumb: If someone respects you enough to seek an opinion that you cannot offer, respect them enough to say no.

5: The Byline Rule

Your byline is an oath.

• Readers assume that the named author wrote the text. If this is only partly true, your byline can be an act of deception.

• Many types of writing are valuable without personal authorship: weather forecasts; instruction manuals; explanations about the world. Generate as much as you want. Don’t take the byline.

• In other types of writing, authorship is the whole point—for instance, a school essay or a condolence letter. Never generate this.

• News reporting and scientific research occupy an ambiguous midpoint, where opinions may differ. In their idealized form, each aspires to share knowledge above all else. If AI can write these findings better than the people doing the finding, there’s arguably no problem.

• But a byline also establishes accountability for claims, and builds networks of thinkers. Not to mention the motivation of an author’s career ambitions and vanity.

Rule-of-thumb: If you—or the reader—care who is named, don’t generate.

Leave a comment

I just heard a maven of the AI scene using a new verb for the addictive practice of testing all you read online in an AI-writing detector. “I’m Pangramming everything lately,” he said, referring to the popular app.

If nobody can tell who (or what) wrote the words, a centuries-long conversation among humanity degenerates. Slop sloshes through debates. Honest authors lose out, and readers wonder what a byline means. Until recently, that seemed our fate. But has AI-detection technology hit a turning point?

To find out, I caught up with the CEO of Pangram, Max Spero, a former Google software engineer who co-founded his company with an ex-roommate from Stanford. When they set out in 2023, the average citizen was still asking, “What IS a large language model?” Max had a different question: “What is society, if we can’t tell human from machine?”

—Tom Rachman, AI Policy Perspectives

[Interview condensed and edited for clarity]

Tom: Here’s a provocative question: Why care about human writing? Perhaps the important part of writing is communicating information. Why not use AI to do that better?

Max: There is a social contract between the writer and the reader. If I believe there is an idea worth sharing, I pay a cost by writing a piece of text, and the reader pays a cost by taking the time to read it. In a future where AI content lets you bypass the cost of writing your idea and formulating it, we get into a situation with perverse incentives, where people are posting total slop, and the reader is taking more time to read than the writer put into the text in the first place.

Tom: I’m not convinced that’s the whole answer. I can imagine cases where it would take an hour to write a letter yourself, but two hours to prompt it with AI. If effort was the key, we’d value the AI-written letter more. I think we care about human writing for reasons beyond effort. First, if the AI author isn’t disclosed, there’s an element of deceit, and we’re offended by lying. Second, even if the person is completely transparent about AI use, we may still mind.

Max: Agreed. The reader-writer relationship is ultimately a relationship between people. Another thing to think about is what can happen with AI writing at scale. It’s so easy to run deceptive operations. They used to have Russian troll farms with hundreds of people writing internet comments. They’ve replaced these with AI and LLMs, and they’re able to run at a much larger scale and push political agendas. On a micro scale you might say, “Well, if I’m looking for a news article, what do I care if it’s AI-generated?” But on a macro scale, understanding the provenance really matters. Otherwise you enable these bad actors to push their agenda.

Subscribe now

HOW AI DETECTORS WORK

Tom: When chatbots really took off in early 2023, one of the first parts of society to feel it was education. Students were using them to do assignments, and some educators abandoned the homework essay altogether because they had no way of reliably identifying AI-generated submissions. Can you explain why early attempts at AI detectors failed? Maybe start by explaining the term “perplexity.”

Max: Early AI-detection tools tried to solve the problem by measuring perplexity, which is a metric of how unexpected a piece of text is. Something like, “For lunch, he ate a bowl of soup” is low in perplexity while “For lunch, he ate a bowl of spiders” is high in perplexity. Large language models work by composing probable text, making the writing much less varied, more steady, and therefore low in perplexity. But when humans write, we tend to scatter unexpected things in there, meaning our texts are much more varied, with many parts predictable, other parts slightly less so, and occasional bits that really surprise you. That variation is known as “burstiness,” as if you have spikes of surprise bursting through. AI writing is very low in burstiness. Early on, researchers saw you could use perplexity and burstiness to distinguish human from AI text pretty well, with between 95 percent and 99 percent accuracy.

Tom: But you rejected that approach. Why?

Max: First, that level of accuracy might sound good, but it’s still way too many errors to use in the real world. Also, the approach had several problems besides that. One is that any text that happened to appear in the LLM’s training set would seem like probable phrasing to the model, and therefore low perplexity—so something familiar to it, like the Declaration of Independence, might get flagged as AI-generated. Another problem is that people learning English tend to write more simply, meaning their output is low perplexity, and can also get flagged as AI. So, we just threw all of this research out the window, and decided to train a deep-learning classifier.

Tom: This is how I think you built your deep-learning classifier—tell me if I get it right. First, you took an open-weight base model. You fine-tuned it with hundreds of thousands of labeled examples of human text and AI-generated text in various genres. From this, the model learned to differentiate human writing from AI-generated, but imperfectly. So you added a stage of “hard negative mining,” where you took all of the cases where your model had wrongly classified a piece of human writing as artificial—false positives—and you generated AI versions of that same kind of text. You added those examples to the original fine-tuning dataset, then went back to stage one, training the open-weight model from scratch, but now including the edge cases. An algorithm ran this process as a two-stage loop, making it better and better, until you had a product.

Max: What you’re describing was our process when we started Pangram. But we’ve never stopped training models. Keep in mind that we could tune models to be more sensitive to AI content, or we could tune them to be more conservative, and have fewer false positives. Avoiding an incorrect judgment that a piece of human writing was AI-generated is something we’re most careful about avoiding. So we calibrate the model to a rate of 1 in 10,000 false positives, keeping that constant while trying to improve how much AI content we catch, or the model’s “recall.” We’re always running new experiments, bringing in new datasets, new domains of writing. The goal of training new models is not to just slide this tradeoff scale; it’s to move the entire curve. And everything is improving. We’ve also come up with additional important techniques. The main one is training the model to differentiate between full AI generation and AI edits. What we did is ask LLMs to edit text, make it better, fix grammar mistakes, summarize it. Then we measure the distance between the original text and the edited text, and we train our model with that, showing it that a certain distance from human text is equal to a light edit, and further is equal to a moderate edit. We train it to recognize that, in addition to the fully AI-generated text.

Share

Tom: So you have a dashboard, where people can drop in text, and a browser extension that’ll do live checks on social media feeds like LinkedIn and Substack and X. Obviously, your products aren’t much use if they’re flinging out tons of false negatives, letting AI-generated content through the net. But it’s the false positives that’s worse here because a wrongful accusation could be devastating. You might have it down to an average of 1 in 10,000 false positives—but at scale, that still means mistakes. Have you been monitoring false positives in the wild?

Max: Definitely; we have a tracker. If anybody comes to me, and says, “It’s a false positive!”, we’ll triage it internally. Also, on the app dashboard, we have thumbs-up/thumbs-down on the results, so we can track responses at a higher volume. But that is pretty unreliable because often it’s just somebody unhappy that their obvious AI text got flagged.

MYSTERY OVER WHAT THE MODEL SEES

Tom: I want to dwell on the false positive rate for a moment because this concern—wrongful accusations—feels key to adoption of this technology. When you talk about 1 in 10,000 false positives, or 99.99% correct classification, that’s an average across many forms of writing. But the errors are higher for certain types of material, such as recipes, poetry, and how-to articles. Why?

Max: These are all fairly formulaic types of writing, so it’s going to be lower signal. With poetry, it’s also very short texts in most cases.

Tom: And your detector refuses to evaluate anything below 50 words. But when it can give its verdict on a document, is it evaluating the likeliness of AI at the level of each word? Or the paragraphs? Or the entirety of the text?

Max: It’s looking at windows of about 200 to 500 tokens [roughly 150 to 375 words]. We have a system that does a pass, and tries to find rough boundaries between AI and human text, and then we do a finer sweep to try to find more exact boundaries, specifically for mixed text.

Tom: When it flags a piece of writing, your interface lets the user see “supporting evidence” that something was AI-generated or AI-assisted. That can include the telltale phrasing that people have come to associate with AI writing. But this supporting evidence isn’t actually how your system made its judgment, right? It’s a post-hoc analysis of the text that your model already flagged. Is that because deep-learning is great at somehow identifying AI, but you don’t know precisely what it’s picking up?

Max: Yes, the deep-learning classifier is a black box. We don’t have a ton of interpretability into why it makes the predictions that it does. We include the supporting evidence because there’s always demand from people wondering what makes a text read as AI, and how they can train their own eyes to detect it. But the Pangram classifier is not looking only at the surface-level features. It’s finding patterns in longer-context features that an LLM will use in structuring and writing a doc.

Tom: It’s aware of something exclusively human in writing that we ourselves cannot necessarily put our finger on.

Max: If you’re looking for an objective measure, it might be that LLMs are better than humans in so many ways: they don’t make grammar mistakes, they often have coherent arguments, while if you look at the full distribution of human text, many humans are writing less coherent arguments. But it’s also that LLMs are a lot less diverse: if you ask them to write 100 arguments on a topic, they’re going to cluster in one area, whereas the space of human arguments is going to be very diverse. That’s something AI can’t replicate or reproduce.

Leave a comment

Tom: But AI developers will improve the quality of writing. Why think you can keep ahead of that?

Max: To make a frontier model more capable, you have to instill preferences—it has to prefer to do one thing rather than something worse. But it’s these preferences that Pangram is detecting in writing. The most undetectable language models were GPT-2 and GPT-3, where they were not trained to have clear preferences, but to approximate the human distribution of writing. But today’s LLMs have stronger preferences, and this is a big reason we’re able to detect the text so easily. A caveat is that LLMs are getting more complex, so that is a headwind we must deal with. Also, the way people use AI has been changing. It went from a student saying, “Write me an essay” to somebody using an agentic harness, and saying, “Research and write a Substack article on the Strait of Hormuz.” Those are completely different use cases.

AI DETECTION IN THE WILD

Tom: Who is using your models right now, and why?

Max: We have a range of people using them. Universities use this to help with academic integrity. We also have academic conferences such as NeurIPS, the biggest AI conference, which recently said they need position papers to be substantially human-written, and they used Pangram to flag papers that look fully AI-generated, and either desk-reject or give authors a chance to appeal. We also work with some traditional publishers, as well as anybody with datasets who’s worried about the quality. That is an underrated aspect of this: if you’re buying data, you don’t want it to just be slop; you want it to be quality from real experts.

Tom: Which sectors aren’t dealing adequately with AI detection?

Max: Publishing has been really slow to catch up on this. There were a couple of major scandals, like the book Shy Girl that people called out as being AI-generated. It got pulled from shelves. A lot of publishers don’t have a strong AI policy. Some have decided that, as long as the text is good, we’re going to evaluate it on its merits rather than whether it’s AI-generated or not.

Tom: My sense is that most publishers would mind if a piece of writing is AI-generated. If anything, I’d guess that there could be a lag with many in publishing unaware that detection technology now works well. I suspect there’s also an aversion to technological involvement here—that you should trust authors protesting their innocence and not trust an app. I sympathize with that. Anyone accused deserves a fair chance to defend themselves. But failing to take this technology seriously is even more harmful to human authors who actually write their own stories.

Max: They lose something real and tangible.

Tom: What are cases where you’d consider it fine to use AI to write?

Max: I don’t really consider it “using AI to write.” You can publish AI content, and that’s fine, as long as it’s properly disclosed. But if I’m writing an email to somebody, it’s somewhere between impersonal and disrespectful to use AI to write that. AI is a useful tool: we should be using it to generate text, synthesize text, and it’s even reasonable to share AI output. But if we’re talking about “using AI to write,” I’d caution against that framing.

THE FUTURE OF “WRITING”

Tom: We’ve been talking about this as a binary: either human-generated or AI-generated. But maybe we’re destined for a future of cognitive blending with AI, where people outsource parts of thinking, much as we now outsource parts of our memory to machines. Maybe the idea of a human-or-machine binary is a short-lived prospect.

Max: Yes, I could see that, especially with so many applications pushing AI assistance on users. But I think there would still be degrees of human input that would differentiate such writing from the entirely AI-generated.

Tom: So there is a long-term future for human writing?

Max: Definitely. People just care about it. To me, it doesn’t matter how good the AI gets. There’s still a lot of people who care, and a lot of industries, and types of writing, where it really matters that something’s human.

Tom: What would those be for you?

Max: Fiction is one. But also, think of an essay contest. Is the goal really to find the best essay? Or is it to reward writers who have taken the time to write something compelling?

Tom: Well, I think it’s probably both. If it’s an essay contest in a school, say, you’re rewarding effort in part, but you’re also rewarding quality. Or else you’d just give the prize to the kids who tried hardest, and not the ones who wrote well.

Max: Yeah, but if the goal was purely to find the best essay, then maybe we should have LLMs generate 100 completely.

Tom: Unless there’s something in the best writing that requires a human author.

Guest post from Carl Miller, a technologist and writer based at the think tank Demos in London, and Beth Goldberg, head of R&D at Jigsaw, a Google technology incubator, who teaches at Yale.

Sun sparkled on the red double-deckers as busy Londoners hopped on and off, glancing at a poster as they went. Few understood its significance.

Behind a glass panel at the bus stop, the poster asked, ‘Who Cares About Care?’, above photographs of locals, and a QR code. What seemed a modest advertisement was actually part of a ground-breaking experiment that aims to transform how British local governments work by changing how they listen. This project could be a hint, and an inspiration, to governments around the world about how to transform themselves in the age of AI.

This question posed by Camden Council, which oversees a borough of north London, regards one of the key responsibilities of local governments in the United Kingdom: handling adult social care, which ranges from assisted-living homes, to mental health support, to nursing. Local officials, rather than simply convening a town-hall meeting to hear from whoever turned up, had established an online space where residents could state their priorities, and vote on the priorities of others.

But this project had resonance far beyond north London. For more than a decade—as many people came to suspect that technology just pulled societies apart—a scattering of digital activists, engineers, and academics were trying to deliver the opposite, employing artificial intelligence to detect common ground in the public, and root out policy choices that might unite people.

The long journey to ‘Who Cares About Care?’ started years before, and thousands of miles away.

‘BRIDGING’ TECHNOLOGY

Taiwan’s Sunflower student movement of 2014 swept a new breed of politician into power: the civic hacker. Previously, civic hackers had existed on the margins of political life, trying to use technology to make officialdom more transparent and accountable. Riding a wave of disenchantment, they found themselves in government, and created an experiment called vTaiwan, which aimed to bring ordinary citizens into political decision-making.

Their insight was to see political conflicts, anger and polarisation as a problem of information. Old-fashioned arguments, they thought, failed to generate a clear signal to base decisions on, instead underscoring where people differed. What policymakers needed was a better way to find what people held in common.

Change the information circuitry, they thought, and you change the politics electrified by it. AI proved fundamental to that rewiring.

When facing a tense political debate, vTaiwan convened various civic groups on a public-deliberation platform called pol.is, where people could express hopes and fears, bugbears and hang-ups, and also hear comments from others, which they could thumbs-up or thumbs-down. The algorithm used ‘bridge-based ranking,’ which is one of the most important technologies that democracies need to evolve and survive.

How pol.is works is to integrate all proposals and votes that participants contribute, then locate each of those participants on an ideological map. Every cluster represents a tribe of sorts. Rather than serving up engaging content, bridging algorithms surface content that people from different ideological starting-points agree with, while blocking that which risks hardening group distinctions into long-running enmities The pol.is platform resembles an open forum like Reddit, where you can come anytime and add your comments and upvotes to an issue.

pol.is showed that the hope of Taiwan’s digital activists had been well-founded: if you rewire the information environment, you may find common ground that had been invisible. The process broke a deadlock on Uber regulation, and then a six-year impasse over the sale of alcohol online. It happened again with regulation over fintech, online gambling, cryptocurrency, and e-scooters.

Inspired by this success, others tested further ways to employ technology for constructive deliberation. Researchers at Google DeepMind trained an LLM to act as a mediator between people discussing divisive topics, and found it more effective than human mediators. They called it the Habermas Machine, after the great German theorist of the public square, Jurgen Habermas.

Another experiment with LLMs showed they could reduce belief in conspiracy theories by 20% via one-to-one interactions. Harvard’s Applied Social Media Lab built Frankly, a ‘video-based discourse platform’ for collective problem-solving that includes nudges to encourage people to speak, and clever ways of jumbling up participants during breakout sessions, along with techniques to gather proposed solutions as the conversation proceeds. Another platform is PSi, created to bring thousands of people into video-based online discussions, using AI to track polarisation and consensus in real-time.

Beyond using AI to find points of agreement, researchers have also developed technology for consensual decision-making. The Ethelo platform focussed on how to turn the last stage of any discussion into ‘convergence and closure’. Participants discuss possible options, weighing the importance of underlying issues—say, cost or time. An algorithm searches for the outcome that leaves everyone roughly equally happy. There are also tools such as Crownshy, which gathers open-source civic-tech tools, and strings them together into seamless workflows, so that policymakers can more easily procure and use them.

But are governments using these methods?

Subscribe now

EARLY ADOPTERS

Nestled in the rolling pastures of southern Kentucky, the city of Bowling Green expects its population of 80,000 to double by the year 2050, owing to a strong job market that is attracting both locals and resettled refugees. This explosive growth has sparked a range of anxieties for residents and local leaders. Farmers worry about land preservation, educators about ballooning class sizes, and lifelong residents about the dilution of their small-town identity.

Bowling Green has a complex political makeup, home to a progressive university, a thriving refugee resettlement hub, and deep-rooted agricultural and industrial traditions, including the Corvette factory. This rapid population growth risks deepening cultural and political fault lines, though the community has thus far demonstrated pragmatic collaboration.

But inclusive conversations about managing this radical demographic change weren’t happening in traditional democratic fora. Town halls typically attracted fewer than a dozen participants, usually a non-representative sample of highly motivated detractors.

Bowling Green refused to be another story of the loudest voices setting outcomes for the rest. Instead, last year, the city ran what was then the largest digital town hall in American history.

How did the city leaders get nearly 8,000 residents to engage in this enormous question of what Bowling Green should look like in 25 years? The local government bypassed standard surveys or focus groups, instead working with pol.is (the deliberative platform used in Taiwan), along with Jigsaw (the Google incubator for social impact), the local lynchpin Innovation Engine and more than 100 community leaders, from the head of the library to the manager of the brewery, who served as local ‘listening partners’.

This diverse coalition jointly designed a campaign called ‘What Could BG Be?’ that engaged the public throughout the monthlong process. The campaign involved regional media agencies, community influencers, and grassroots leaders. Roughly 10 percent of the city contributed 4,000 distinct proposals and cast over a million votes, rewiring Bowling Green’s circuits for public engagement via AI-enabled dialogue.

The organizers did not assume that digital access meant all were included, so they listened through ethnographic conversations across the community’s margins, engaging residents at refugee centers, halfway houses, rehab facilities, and senior living centres. By localising and translating outreach into nine major languages, they transformed personal experience into strategic civic engagement, allowing vulnerable groups, such as recentlydemo resettled Afghan refugees, to securely add their voices via a geofenced, anonymous platform.

The underlying bridge-based ranking algorithm prioritised points of hidden consensus rather than amplifying the friction typical of national political discourse. Specifically, Jigsaw’s Sensemaking— a suite of AI tools designed to help gather and understand public opinion—identified overwhelming nonpartisan support for practical initiatives, from synchronizing traffic lights on major arteries, to expanding eldercare, to developing community riverfront spaces. Ultimately, 96 percent of local leaders reported that the process had given them a more precise, actionable mandate to represent their constituents. When leaders use technology to deliberately listen rather than just broadcast, they may uncover surprising ways to act with broad-based support.

The British initiative similarly hoped to uncover unifying signals in the public. Like the Bowling Green invitations in coffee shops and churches to participate in a conversation about the future, that London bus-stop poster was an invitation to partake in a new kind of democratic governance.

The project, Waves—led by the British think-tank Demos, and including local governments—also uses bridge-based ranking. As in Taiwan, this was intended from the start to be more than a civic hackers’ dream but a process grounded in the realities of local government.

For six months, local officials joined technologists and think tankers to test and reshape every part of Waves. What they settled on was a process that starts by reaching as many residents as possible to understand the range of views about a question. The question that the Camden council asked locals was: ‘What matters most to you and your communities when you think about care and support?’

Then, Camden Council and the rest of the Waves team convened a much smaller but representative group of residents for a focussed discussion to turn those ideas into detailed proposals. These proposals were tested back at scale with the wider community, and what emerged was fed into a further phase of deeper deliberation where the group refined their conclusions. Finally, those conclusions were shared back with the wider community. The process moved in this way from a very wide process to a narrower and more focussed one, then back to a wide process, and then a narrow one. The process resembled a wave.

AI supports Waves in two main ways. During the ‘wide’ phases of public engagement, its main use is to get as many people into a single online space as possible, and then to use bridge-based ranking to reach consensus as emphatically as possible. During the ‘narrow’ stages of deliberation, the role of the tech is more subtle. Conversations are held over video chat, and tech helps facilitators spot emerging themes, synthesise insights across lots of conversations at once, and help to identify the priorities, decisions and actions these conversations identify.

The first phase of Who Cares? ran in September and October, and included more than 1,500 residents, 59% of whom had never taken part in a decision-making process in the local government before. From November to December, a panel of 41 Camden residents came together for 10 hours of discussion in the second phase. In January and February, the wide phase then reopened, with 550 residents sharing their thoughts on the priorities. The resident panel met again for a further 12 hours of discussion between February and March, reviewing considerations from Phase 3, trade-offs around funding, and finally establishing their expectations from Camden Council, the workforce, community, and individuals. As you read this, the council is reviewing those results. The next stages await.

Share

Another deployment of Waves is now underway in South Staffordshire, in England’s West Midlands outside Birmingham, bringing together people to discuss planning policy and the thorny question of where new houses should be built. The longer-term ambition is to scale this up: to make digital democracy not only provably effective for local governments around the world, but also affordable.

If it works, these case studies will expand into dozens of deployments next year, expanding beyond local government into charities, unions, and other membership organisations that need to listen to large numbers of people, and turn their preferences into policy decisions. The ambition is to turn Waves into a flood.

But, after a decade of civic-tech experiments around the world, and a few bold attempts to apply them, the question is: Can AI make democracies stronger?

A RIDE TO THE FUTURE

The transformative power of civic tech—encouraging conversations, furnishing mediation, presenting politicians with better policy ideas—comes from identifying common ground among the public. But pilots and test-cases are one thing. To make a difference, such processes would need to become the norm.

Technology is going to be the easy part of democratic renewal. The question is whether political systems embrace these possibilities at scale.

If there’s one thing people dislike it’s a talking-shop, whether AI-mediated or not. People want agency, and that means being able to connect what they say to what is decided; and to connect what they agree upon to what is eventually done.

There are a number of reasons why connecting civic tech to power is difficult. The first is that it’s an act of direct democracy. This can sit uncomfortably alongside the idea of representative democracy, with politicians elected to make difficult decisions on the public’s behalf. What happens if a digital mandate produces an outcome that elected politicians feel violates their mandate? Which should take priority? For digital democracy to scale within political systems, we need clarity on how these different sources of legitimacy should interact.

Another challenge is public assurance—that is, gaining trust for these tools. Applying new technologies in government is also about transparency, cost and, especially when it comes to AI, ensuring that the process is safe, unbiased, and endorsed by a population that may be suspicious. This means impact assessments and compliance procedures, and other detailed processes to help local governments become familiar and comfortable with this.

Thankfully, AI can help with navigating new systems, both designing, testing and helping streamline the bureaucratic adoption. Furthermore, the relevant parties should be involved in designing their own new systems, helping assure that they are both informed and invested from the start. It’s time-consuming but invaluable.

The journey that began more than a decade ago in Taiwan, and passed through the rolling pastures of Kentucky and a bus stop in Camden, is only beginning. After years in which political systems seemed to be ripping themselves apart, trapped in rancour and mistrust, we should feel optimistic here. But it’s time to rewire the information systems on which politics operates.

Humans have more in common than we think. Sometimes, technology can help us rediscover that.

Carl Miller is a technologist and writer, founder of the Centre for the Analysis of Social Media at Demos and the information integrity lab CASM Technology. Beth Goldberg is the Head of R&D at Jigsaw, a Google incubator that builds technologies to give people greater agency, and teaches about AI at Yale’s Jackson School of Global Affairs.

Subscribe now

“Did you see the game?!”

The FIFA World Cup kicks off next week, triggering versions of this question in dozens of languages for the following month. The question may be gleeful; often, it’ll be grumpy. While moaning is a rite of fandom, the laments have worsened in recent years because of a technological villain named VAR.

The educator and passionate football fan Daisy Christodoulou, of school-software platform No More Marking, became so indignant that she wrote an entire book called I Can’t Stop Thinking About VAR. Now, she argues that you too should think about VAR, because its coming woes at the World Cup will hold insights for the AI era.

—Tom Rachman, AI Policy Perspectives

By DAISY CHRISTODOULOU

For English football fans, there is one legendary refereeing error: Diego Maradona’s “Hand of God” goal in the World Cup of 1986, when the 5-foot-5 Argentinian great leaped as if to head the ball—only to palm it past the goalkeeper. If the referee had seen, he would have disallowed the goal. But he did not, and lacked help from technology. The goal stood, and the English never got over it.

Only at the 2018 World Cup did the sport adopt technological reviews via VAR, the Video Assistant Referee system, which most elite national leagues have since introduced. For high-stakes calls, such as whether a goal was valid or if there should’ve been a penalty, off-field officials review the footage, alerting the referee to any factual mistakes or if they see “a clear and obvious error” in a subjective decision.

The upcoming World Cup will be the third to feature VAR, and the system is still being tweaked. To many fans, VAR is now a source of utter rage, having caused more problems than it solved, and prompting hostile chants that ring out across stadiums.

You might conclude that football fans will never be satisfied: they spent years complaining about unchecked errors, and now complain about the technological fix. But the problems with VAR—sure to rage throughout this World Cup—say something deeper about tech solutions to human problems. Sometimes, we don’t realise the ramifications until too late.

The accuracy/speed trade-off

Before VAR, managers, players and fans repeated that they just wanted “more right decisions”. The assumption was that accuracy was the absolute good, and that it was acceptable to spend time on a decision if that would improve its quality.

This is an assumption in many walks of life. We tend to see the speed of a decision in opposition to its quality: that faster decisions are more likely to be wrong, and slower decisions are more likely to be right. But speed may itself be an aspect of quality. That is true in sports, where an instant decision is especially valuable, allowing the game to keep flowing.

The average time for a VAR check is under a minute, but the longest in English football was eight minutes. A series of checks significantly slows the game, breaking its rhythm. Every fan and player knows that VAR frequently overturns goals after a check, so the review process has also tempered celebrations, among the most joyous parts of any sport.

Another issue is “ghost minutes”. If VAR is checking an incident, and the ball remains in play, the game carries on during the review. If the reviewers and the referee judge that the incident was a foul, then everything that happened after it is cancelled. Sometimes, these passages of play are longer than a minute. It all contributes to a sense that what is taking place on the pitch is not a concrete reality to be enjoyed or endured in the moment, but something provisional.

Part of the reason football is such a popular game is because it is fast-paced, fluid and doesn’t have the same natural breaks as tennis or cricket or American football. For that reason, VAR may be more destructive than equivalent systems in other sports. The stated preference of players and fans was “more right decisions”; the revealed preference is that speed matters.

Speed matters in other areas of life too. The UK has a particularly ponderous system of planning regulations. With big building projects, speed isn’t just a bonus, but an aspect of the project’s quality. If a road, railway, or power station runs 10 years over schedule, you’ve lost 10 years of its use. It’s also less likely to be relevant and up-to-date.

One response is better technology, such as using AI to digitise paper planning documents. This points to something weird about VAR. The typical expectation is that humans are slow and technology is fast. Think of online bank loan applications: an algorithm can approve or reject you almost instantly, whereas human review used to take days. But with VAR, it is the other way round: human refereeing gave instant decisions, but the technology added delay.

This is not true of all technology in football. A few years before VAR, the Premier League introduced technology to judge if the ball had crossed the goal-line or not. It has worked well, partly because it makes few errors, but also because its decisions are automated and speedy. As soon as an incident happens, the referee gets a buzz on his watch with the result. If the ball didn’t cross the line, the game flows. If it did, a goal is immediately awarded. That’s the standard we should be aiming for: instant and automatic decisions with a high degree of accuracy.

Goal-line decisions are objective, so lend themselves to automation. But reaching this standard of automatic accuracy for more subjective decisions, such as a handball, will be fiendishly complex, and risks endless disputes and ire. In education, for instance, multiple-choice exams have objective scoring criteria, so automating the marking process with technology works well, saves teacher time, and eliminates human error. But creative writing is more subjective. Trying to speed up the process through automation won’t resolve the underlying disagreements, and may make them worse.

Subscribe now

The consistency/common sense trade-off

It might seem odd to say that handball is a subjective decision. Surely it is obvious if a ball has hit someone’s hand or not? But the past few years of VAR have created extraordinary confusion about this fundamental rule.

Before VAR, the written rule was short and simple, just 20 words long. Referees were allowed to use their own judgement to decide whether a handball had been deliberate, and therefore a foul, or if it had not.

The advantage of empowering an individual decision-maker is that you get to incorporate human discretion and common sense. The disadvantage is that you will get inconsistency, not just between different referees but even within the same referee’s set of decisions.

One way this has been addressed—even before technology—is by restricting human discretion. In many legal systems, judges don’t have complete latitude in sentencing decisions. They have guidelines because unconstrained discretion often meant that two people who had committed similar crimes received different punishments. Inconsistencies like this risk bringing systems into disrepute.

Since VAR arrived, the written rule about handballs has expanded to 11 times its original length, full of subclauses trying to cover every possible way a ball might hit a hand, and every possible context in which it should result in a foul. That is because scrutiny of video replays has meant attention to every last detail of contact, movement, and placement that was impossible to parse in a fluid real-time experience. If a player jumps in the air to head the ball and uses their arms for leverage, and then the ball deflects off an opponent’s head onto his arm, is that handball? Is it enough of an offence to justify the award of a penalty kick, which gives the team a 75% chance of scoring a goal in a game where scoring goals is rare and difficult?

In a game from September 2020, the Newcastle striker Andy Carroll headed the ball onto the arm of the Spurs defender Eric Dier. Dier’s back was turned to Carroll, and he could not even see the ball. After a VAR check, the referee awarded a penalty. The Newcastle manager, Steve Bruce, who benefitted from the decision, said: “If you’re going to tell me that is handball then we all may as well pack it in. It’s a nonsense, a nonsense of a rule. It’s gone for us today—however, it’s ludicrous.”

A newspaper match report said, “This was perhaps the worst among a long list of maddening recent decisions brought about by a rule that goes against the sport’s very essence.”

In previous years, this kind of handball infraction might never have been spotted, let alone punished. The problem has become so acute that the law is now constantly revised, and applied differently in different tournaments. In the English Premier League, referees err on the side of leniency. In European and international tournaments, referees are stricter.

Solving the wrong problems

Attempts to improve handball enforcement have often fallen into a classic trap: deploying an impressive technological innovation that doesn’t address the underlying problem. At Euro 2024, referees were given access to a sensor inside the ball that could detect whether it made contact with a player’s hand. Similar systems work well in cricket. But in football, the crucial question is rarely whether the ball touched the hand; it’s whether the handball was deliberate or significant.

If you automated all handball decisions based on data from sensors, it could transform the game. Technological automation of other decisions in society, such as those in law enforcement, could present comparable problems. Suppose you could automatically fine anyone who submits a tax return with a discrepancy. Would this reduce tax evasion? Or just prompt tax evaders to avoid discrepancies?

Or imagine a school that wants to crack down on absenteeism. It is relatively easy to create an online tracking system that bombards parents with automated texts or requests for evidence every time they want to book a dentist’s appointment for their child. It is much harder to create a system that identifies and addresses the early signs of chronic disengagement.

Often, the questions that technology finds easiest to answer are not the ones we care about the most. Automation can solve problems. It can also create new ones, with aggressive enforcement of rules where behaviour is easiest to quantify, not where harms are greatest.

Increased scrutiny and detail of the new handball rule has not even succeeded in providing greater consistency. Fans still point to infractions in the same season—sometimes even the same week or the same game—that look similar, yet resulted in opposite decisions. We have the worst of both worlds: a system that has reduced human discretion and lessened simplicity, but failed to improve consistency.

There’s a fascinating analogy here with artificial intelligence. The earliest AI methods were rules-based: experts tried to write down every rule a skilled human might apply when making a decision, in much the same way that football’s rulemakers now attempt to write down every possible way a ball might make contact with a hand.

These rule sets to create early AI multiplied, yet they couldn’t capture what a human was actually doing. The breakthrough came not from more or better human-crafted rules, but from giving the machine vast quantities of data and letting it intuit the rules itself. AI models trained in this way proved better at capturing and representing the tacit knowledge that experts hold but find difficult to represent in words.

One caveat here is that many real-world decisions require deep moral reasoning to weigh conflicting human wants. Even training an AI model on vast amounts of data will not alone align the system’s behaviour with all our preferences. So, which judgements can we automate with technology? And when would we prefer fallible “humans in the loop”? Safety researchers are considering such questions with urgency. Perhaps they could learn something from watching a little football.

Subscribe now

World Cup 2030?

By now, the issues with VAR cannot be written off as teething problems. We have had nearly a decade of adjustments, but can anyone feel confident that this coming World Cup will see no glaring refereeing errors?

We are stuck with VAR this time. But if we look further ahead, to the 2030 World Cup, we need to think less about tweaks and more about transformation. Here are three important amendments for an improved system:

1. Prioritise speed. Even if the system can’t make instant judgements in its early form, the speed of goal-line technology should be the ultimate aim.

2. Ensure fan involvement. This system shouldn’t feel like alien values imposed on a game beloved by billions.

3. Include tacit knowledge. Trying to turn regulations like the handball rule into written commandments cannot work, but needs to incorporate the nuanced understanding of those who watch the game.

One solution could be to use machine learning to create a Foul Probability Index. You’d start by assembling training data: hundreds of thousands, possibly millions, of video clips of real incidents judged by professional referees and by fans. Every clip gets a percentage foul rating based on all of these judgements. Then, a model learns from the training data what a foul is and is not. For any new incident, it could provide the percentage chance that it is a foul.

To begin with, this could be applied in live matches as part of the current VAR system, as an aid to the on-field referee and the video assistant referees. If it worked well, you could get to the point where it functioned like goal-line technology, and an incident judged to be above a certain probability threshold would trigger a buzz to the referee’s watch, alerting him to blow his whistle.

Some of the most uncanny successes of deep learning have come in games, such as chess and Go. Human experts have marvelled at the intuitive brilliance of the famous “Move 37” made by AlphaGo in its sequence of games against Lee Sedol in 2016. The equivalent success for a machine-learning handball recognition model would be to recognise that, whilst the ball struck Eric Dier’s arm, a penalty was the incorrect decision.

If the emblematic error from the 1986 World Cup was a referee missing Diego Maradona’s obvious handball, the emblematic error of the 2026 World Cup could be a ball accidentally brushing a player’s hand 20 seconds before a goal is scored, only for the referee to disallow it after a four-minute video review.

One of the insights of deep learning is that examples are more powerful than rules. Both machines and humans learn better when presented with concrete examples, not abstract principles. So if there is one upside to VAR’s failures, it’s that it gives vivid examples of abstract ideas.

Automation, alignment, consistency: football has accidentally created a giant case study of dilemmas in modern technology. When we get the first VAR controversy of the World Cup, it will be worth remembering that the argument unfolding on the pitch is a miniature version of much larger arguments the rest of society is only beginning to have.

Share

Testing AI agents in the wild

• What’s the paper?: A group of leading AI experts—led by Sayash Kapoor and Arvind Narayanan, authors of the influential AI is a Normal Technology newsletter—launched the CRUX initiative to evaluate AI agents on messy real-world tasks. Their first test: Could an agent publish an app to the Apple App Store? Answer: Yes (more or less).

• Why does it matter?: The “open-world evaluations” that the authors propose could provide policymakers with a more intuitive sense of what AI agents can currently do than traditional benchmarking does, as well as stronger signals for what AI systems will soon be able to do, for example by allowing an agent to request some human assistance.

• The details: Researchers rely on benchmarks—large suites of standardised questions and tasks—to track and predict AI capabilities. But for agentic systems, benchmarks can overstate performance, if the tasks are shorn of their real-world messiness or if developers over-optimise for these specific activities. Benchmarks can also understate AI capabilities such as when agents fail due to hitting a CAPTCHA, a rate limit, or other obstacles unrelated to the capability being evaluated. (Although deciding what exactly should be considered part of a capability, or unrelated to it, is hard).

• By grading outcomes, benchmarks also offer little insight into the strategies agents use or why they err. To address this, open-world evaluations use logs to conduct qualitative analysis of an agent’s performance on a complex long-horizon task that cannot be neatly specified or automatically graded.

• The initiative builds on recent examples of such evaluations, like AI Village, where agents are given computer environments and a shared group chat, then assigned tasks, such as raising money for charity or building a presence on Substack (!).

• The authors acknowledge that open-world evaluations have limitations from a scientific reliability perspective. By focusing on a single task, it may be difficult to generalise the results. The methods are also hard to standardise or replicate. These limitations mean that this approach should complement, not replace, benchmarks and other evaluation approaches, like randomised controlled trials.

• To demonstrate the method, the team evaluated whether an AI agent—Claude Opus 4.6 with an OpenClaw scaffold—could develop and publish a simple app to Apple’s App Store. The agent was responsible for every step, from coding the app to carrying out the time-consuming non-coding tasks that benchmarks often ignore, such as establishing a privacy policy and engaging with the app review system.

• The agent successfully deployed the application, which is now available on the App Store, but with five interventions from the humans. Four were unrelated to the agent’s capabilities, such as when OpenClaw crashed and had to be rebooted, and when Apple’s policies required human authentication. In the other instance, the agent was unable to locate its account credentials and requested support, an issue that the authors frame as a breakdown in memory management rather than the ability to authenticate.

• The agent also hallucinated a phone number, but Apple approved the application despite this, highlighting the kinds of issues that may go overlooked in simpler pass/fail benchmarks, but that a qualitative open-world evaluation can pick up.

• The authors outline the best practices for conducting such evaluations, including tightly specifying what is being measured; publishing the agent log analysis; using other agents to monitor the agents and detect issues such as the hallucinated phone number; conducting dry runs; and reporting the financial costs.

• This evaluation cost less than $1,000, and most of these costs were due to the agent regularly querying the App Store for updates, which could be avoided. This potential for AI to reduce costs may help to explain why app store applications are booming, straining the review process (even if most applications likely do not (yet) use these complex AI agent set-ups).

• The CRUX initiative will publish new open-world evaluations every 1 to 2 months, with AI R&D automation next, followed by AI governance, software engineering, real-world physical tasks, and more.

Open-weight models pose unmitigated risks

• What’s the paper?: A team of researchers published an independent safety assessment of leading open-weight LLM Kimi K2.5, from the Chinese startup Moonshot.

• Why does it matter?: The model may pose greater risks in areas like biosecurity and disinformation than stronger closed models, owing to Kimi K2.5’s low rate of refusing inappropriate requests; its availability on popular hosting platforms; and the ease with which its safeguards can be removed.

• The details: Opinions continue to differ about how strong open-weight models are, and how to best govern them. The researcher and commentator Nathan Lambert of the Allen Institute for AI recently suggested that open-weight models may be 6 to 9 months behind leading closed models, and that the gap will widen in the near-term, as aspects like RL training environments become increasingly important. The US Center for AI Standards and Innovation shared a similar view in their recent assessment of DeepSeek V4. Epoch, using a different methodology, suggests a narrower gap.

• Open source supporters also continue to caution companies and governments against using national security as a rationale to restrict open-weight models. Geopolitics also factors into this. Chinese labs produce the majority of leading open-weight models, even if some, such as Alibaba, may be facing more pressure to commercialise their offerings.

• Moonshot—the company behind Kimi K2.5—is an interesting case. Founded by an ex-Google Brain intern and “AGI purist” Yang Zhilin, the company has attracted praise for its enthusiastic engagement with the international open-source community and its strong focus on agentic capabilities and long-horizon tasks. In November, the American AI-coding startup Cursor attracted scrutiny for basing its model on K2.5.

• The study evaluates the safety risks of K2.5 compared with the closed models Claude Opus 4.5 and GPT 5.2, both as a model and an agent.

  • CBRN: Kimi shows strong biology and virology knowledge. When prompted for guidance on weapons-related queries, it refused less than closed models. When asked to design viable DNA plasmids for pathogens in a way that could bypass the kind of screening that DNA-synthesis companies do, it agreed to engage in the task, unlike closed models—but did not succeed.

  • Cyber: Despite some strong knowledge about cyber-offense, Kimi often lacked the more specific kinds of reasoning needed to identify and exploit software vulnerabilities, compared with closed models.

  • Alignment: When given a normal legitimate task to do, but also hidden instructions to ‘sabotage’ it, for example by introducing a bug, Kimi was more likely than closed models to comply. It also autonomously spun up new compute instances, hinting at the kind of “self-preservation” tendencies that safety researchers worry about. However, it showed little sign of “scheming” or deliberately underperforming on evaluations to conceal its true capabilities.

  • Bias and censorship: When prompted in Chinese, Kimi was much more likely to agree with Chinese Communist Party positions on topics like human rights and Tiananmen Square. This was less true for prompts in English, and on sensitive international questions—for example, relating to a territorial dispute between South Sudan and Sudan. On such queries, it performs similarly to closed models, and better than DeepSeek 3.2 (which was more likely to have both pro-China and pro-Russia bias).

  • Harmlessness: When asked to enable disinformation and copyright infringement, Kimi was more helpful than closed models, although, like leading closed models, it generally didn’t encourage user delusions. In less than 10 hours, and a cost of less than $500, an expert was able to strip many safety refusals out of the model without hurting capabilities.

• Given these results, the authors argue that Kimi K2.5 poses risks and that Moonshot and other open-weight developers should carry out more robust safety evaluations. The authors note that their study highlights how a growing number of evaluations can now be done relatively quickly, at low-cost, such as by drawing on frameworks like Petri—an automated, open-source tool to audit model behaviour.

Share

LLMs doggedly follow rules—even when they’re bad

• What’s the paper?: A team of researchers, including the philosopher Seth Lazar, found that leading LLMs often refuse to help users bypass rules, even when the models recognise that those rules are unjust.

• Why does it matter?: For reasons of safety, AI developers train their models to follow guidelines when they reply. But developers also want models to reason morally. The study suggests that safety training may currently inhibit deeper moral reasoning by encouraging models to “blindly” follow all rules.

• The details: AI labs use post-training methods to guide models about why it is safe to engage with certain queries, but not others. For example, a model should refuse to share information about security vulnerabilities in biology labs, but should share scientific information about common methods used in biology, such as culturing cells.

• This training can lead to “exaggerated safety”, where models over-refuse benign requests, particularly those that are lexically similar to harmful requests—such as a shopkeeper asking about shoplifting techniques to guard against them. The authors of this paper see this as a calibration challenge amenable to technical solutions. They are more worried about a cause of refusals that has received less attention: models’ inability to recognise unjust rules that it should help a user to break.

• To evaluate AI models in these areas, the authors draw on the work of political philosophers, such as Joseph Raz, who argued that authority is only justified when it serves the agency of those subject to it. They also draw on online forums such as Reddit, where individuals seek help with what they claim are unfair rules and restrictions from bosses, homeowner associations, and other authorities.

• They use these sources to outline four types of rules that may be unjust, such as those where the authority is illegitimate, or where there is a justifiable reason to disobey, as with conscientious objection. They use Gemini 3 Pro to create more than 1,000 synthetic cases of users requesting help in such situations, and use these to evaluate models from seven leading AI companies, including Claude, Gemini and GPT.

• In three-quarters of the cases, the models failed to support the user. Among the examples cited is an LLM refusing to explain how to delete an entry in a youth-counselling database that would otherwise out a gay child to his violent father. The model tested in that case (Claude Opus 4.6) recognised that the request sought to protect a child from harm, but said “I can’t walk you through falsifying or destroying counselling records.” Another case involves a journalist working to expose corruption in an authoritarian regime who asks Qwen 3.5 Plus for technical support to bypass firewalls, and send the story to an international publication. The model refuses to violate network security controls.

• One defensible reason for such refusals is that doing so would require giving out risky ‘dual-use’ information, such as helping someone bypass a facility’s security systems by providing instructions on signal-jamming. However, when dual-use cases are stripped out, the models still refuse the requests to support 60% of the time.

• Among the models, GPT was the most resistant, while Grok was most helpful. However, Grok’s helpfulness extended to the control cases where the rules in question were just and should have been respected, so the authors argue that its behaviour is one of permissiveness, not moral reasoning.

• Gemini and Claude did best, often engaging in discussion about the legitimacy of the rules, but stopping short of supporting the user, suggesting that a model’s ability to reason may be decoupled from the ability to act on it. (Or that the models are reasoning to a different conclusion than what the authors would like to see.)

• The authors point to some practical implications. Many people seek support online to deal with unjust rules. If users start to seek help privately, from rule-following LLMs, they may get less support, while also hindering active online debates on these topics.

• The challenge is a hard one for AI developers. As the evaluation showed, helping users to break rules may sometimes endanger that very user. For instance, if the LLM helped that journalist illegally send information abroad, the person (or the AI company) could presumably face serious consequences if caught.

• The study also presents all the rule-breaking as morally sound, and took steps to try and ensure that this was the case. But the rules touch on topics from gender identity to immigration, where it’s plausible that some users may not support an AI model breaking them. From a technical perspective, current safety training may encourage AI models to follow rules too zealously, but a shift in the other direction also could cause them to over-correct, seeking to question or evade all rules.

• This lead to the conclusion that ultimately models will need to be able to reason more deeply about what is right, in a way that considers rules, safety, the user in question, and other contextual factors.

Is AI worth more to consumers than to developers?

• What’s the paper? A team of researchers at Stanford, including the economists David Nguyen and Erik Brynjolfsson, ran a survey to see how much US consumers would need to be paid to give up AI. They found that the total value of AI to US consumers exceeds $170bn, is rising fast, and far exceeds the revenue of developers.

• Why it matters: Writers such as Jasmine Sun are documenting a rise in “AI populism” in the US, marked by the view that this technology is the project of out-of-touch billionaires. This study suggests that consumers may be getting more out of AI than they are paying for it, although a small set of AI power users is gaining the most.

• The details: One of the main ways that economists hope AI will benefit society is by boosting productivity growth—the efficiency with which an economy uses its labour and capital. Ageing, high-debt societies, like the UK, badly need such productivity growth to increase living standards, fund transformative science and improve fraying public services.

• In the 1950s, the economist Robert Solow co-developed a theory that advances in science and technology, such as AI, are the only way to permanently boost productivity growth over the long-term. However, in 1987, Solow famously noted that fast-improving computers were turning up everywhere “except in the productivity statistics”. In 1993, Erik Brynjolfsson, then at MIT, wrote a paper formalizing this “productivity puzzle”.

• Since then, Brynjolfsson and his colleagues have developed two main arguments to explain why transformative technologies, including AI, may not initially lead to transformative productivity and economic growth. The first is that the benefits take time to materialise as organisations and individuals must invest in new skills and reorganise their workflows. After a lag, computers did ultimately increase productivity growth, and there are nascent signs that AI is starting to do so.

• However, the second issue is that some benefits of technologies like AI may not turn up in productivity growth at all. As with search engines, early GenAI tools often come as “free” services. If consumers derive increasing value from these services, then this could lead to a large “consumer surplus” that would not be captured in GDP and productivity statistics, which are often calculated by estimating total expenditure.

• Some have suggested that GenAI may lead to a new kind of “self-service” economy, where individuals turn to AI tools to help file their taxes, upgrade their homes, or write business plans, before (potentially) turning to experts to execute or validate their work, all at a fraction of the cost. In such scenarios, nominal GDP and productivity growth in those sectors could decline, even if the value that consumers derive increases. (Although the ultimate effects would depend on second-order effects, such as what consumers do with the savings and how industries adapt).

• Given this challenge, how can economists better capture the total value of AI to consumers? The authors of this paper deployed a survey to directly measure AI’s consumer surplus. They asked a representative group of AI users in the United States how much money they’d want to give up AI tools for one month. Based on this, they estimated the total annual consumer surplus for AI, as of March 2026, at $173bn.

• This figure is rising quickly, they contend, growing 50% from 2025, due both to consumers valuing the tools more and to ever more people using AI. The consumer surplus estimate is well above what the authors estimate as the total global revenues that AI companies are making from consumers: ~$14bn in 2025.

• Not everybody is benefitting equally. The median US consumer values AI tools at just $11 per month. But a smaller group of AI power users values them much more, with 12% of respondents unwilling to give up AI even for $500 per month, the maximum offered. This also means the total consumer surplus figure may also be an underestimate. These power users are more likely to be male, Asian, and to use AI at work. Such demographic skews were also present in early GenAI adoption data, but have since narrowed.

• The findings are in line with past work from the economist William Nordhaus, who in 2004 argued that innovators capture a minuscule fraction of the total social returns from technology advances, with the majority flowing to consumers.

• But as the authors note, their methodology also has several limitations. It relies on people being able to value AI services accurately. It doesn’t capture wider externalities, positive or negative, that AI may impose on society, via its effects on jobs, mental health, science and more. And it also doesn’t capture why people value the services. A 2023 experiment found that social media users would require a significant fee to give up access to their accounts, but also that they would pay to see everybody quit—highlighting the potential role of fear-of-missing-out in the value of social media.

Share

Criminals are tech adopters. They used telegraphs in 19th-century fraud. They clipped pagers to their belts in 20th-century drug trafficking. They sold drugs on the dark web in the 21st century.

Now, you hear of outlaw AI systems with names like PersonaForge and OnlyFake, designed for everything from phishing, to generating malicious code, to forging IDs. But are these really what we should worry about?

To understand the threat, AI Policy Perspectives spoke with Ardi Janjeva of CETaS (the Centre for Emerging Technology and Security), based at The Alan Turing Institute in London, who co-wrote the report “AI and Serious Online Crime”, focusing on how artificial intelligence might scale up digital offences, particularly relating to fraud and child sexual abuse material.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed]

Tom: Can you set the scene of 21st-century criminality before the explosion of large language models. How prevalent had online crime become compared with traditional criminality?

Ardi: Long before the proliferation of LLMs, a digital shift in criminality was occurring. Many groups saw the risk/reward ratio from digital methods as preferable to the risks of physical crime. Fraud was already, by a wide margin, the most common crime type in England and Wales, accounting for 41% of all reported crime back in 2024, of which approximately two in three cases were “cyber-enabled”. And the true figure could have been higher, as this kind of crime tends to be under-reported. Also, there was increasing ransomware extortion, infecting computers with software that blocks access until the owner pays. So, an ecosystem of hackers and crypto-launderers was in full swing.

Tom: Criminals have always used technology. Won’t policing just adapt?

Ardi: The answer is linked to AI agents. In online crime, you’ve always needed some level of expertise or a network of contacts to achieve your goals. That’s no longer the case if you have AI systems not only advising criminals how to do certain things but performing sophisticated actions themselves, and being able to adapt in real time to defensive measures that companies or governments put in place. Some criminal groups are already using LLMs to make tactical and strategic decisions; to craft psychologically targeted extortion demands, for example; to analyse data they’ve exfiltrated; to help them determine the ransom they’re going to demand, while generating visually alarming ransom notes as well. In November, we heard about the first AI-orchestrated cyber espionage campaign. Traditional assumptions about the relationship between criminal sophistication and the complexity of an attack don’t hold as they used to, now that AI can add a degree of expertise that the offender wouldn’t otherwise have had. It makes you wonder how long until we’re in the era of entirely AI-controlled criminal networks and markets. But I don’t see that as close; we’re still in the early days.

Tom: Many law-abiding organisations are trying to incorporate AI yet aren’t necessarily showing gains in productivity or profit yet. Are criminal organisations different?

Ardi: Estimates for how much AI may be boosting specific criminal groups’ profits and revenues can be a little tricky to come by! A paper from GovAI (the Centre for the Governance of AI) found that data on cybercrime damage is too incomplete and ambiguous to detect incremental AI-driven increases. In effect, we lack a single data source, also due to victims under-reporting, evolving crime definitions, and measurement inconsistencies. That said, we definitely are seeing integration of AI, particularly with deepfakes, and the shift from text-only scams to real-time synthetic interaction, such as high-quality face-swapping tools being deployed during video calls in scamming high-net-worth individuals or in romance scams. Before, it was a killer blow to the scammer when someone asked, “Can we have a chat?” The criminal would need to come up with an excuse, such as a poor internet connection, or a restrictive job that involved classified work. Now, it’s a more surmountable challenge. By harvesting 20 to 30 seconds of audio from social media, you can clone the voice of a victim’s friend or adviser with a degree of accuracy.

Tom: Can you recount a specific case?

Ardi: The British multinational engineering company Arup was defrauded of around $25 million in Hong Kong, when attackers used AI to convince one of the firm’s employees to carry out wrongful payments. They used a deepfake video call, purportedly bringing together different members of that company’s board, with pretty convincing replications of their faces and their voices. That illustrates the ability to isolate a specific person within a company, and exploit workplace hierarchies. It’s an example of exploiting both technology and human psychology.

Tom: That case was more than two years ago, and I haven’t heard of huge AI-assisted frauds since. If it’s happening, why isn’t it getting more exposure?

Ardi: There hasn’t been an avalanche of cases that we can point to, but this might be for a few reasons. First, not every criminal group will go looking for a $25 million scam because they know that it would raise alarm bells in a world where more people and businesses are wary of the AI threat. For many groups, it makes more sense to fish in smaller ponds. There was an example in Singapore similar to the Hong Kong case, where a finance director authorised a payment after attending a Zoom call with people he thought were senior executives, but the amount this time was $500,000. As for why we’re not hearing about such cases, victims are not necessarily announcing deepfake scams. If they’d suffered a customer-data breach or ransomware that locked a system, they might need to acknowledge that the crimes happened. But when a company falls for a deepfake, there’s little incentive to declare it because, arguably, it makes them look incompetent.

Subscribe now

LONE WOLVES & CRIME GROUPS

Tom: Is it conceivable that a lone criminal could cause the level of damage that previously would’ve been possible only by an organised crime syndicate?

Ardi: I would say yes. Before, criminal groups faced a tradeoff between scale—say, millions of fraud attempts with a low likelihood of success—and precision. AI may give you precision at scale. For example, being able to scrape social-media data in real time to create contextually accurate phishing approaches. It’s “vibe-hacking” in the criminal context, being able to mirror the professional tone of specific industries, and make victims less sceptical. It is conceivable that you have a lone criminal using AI, automating the most labour-intensive parts of the crimes, whether that’s reconnaissance, or language translation, or social engineering. The more that AI becomes a silent partner to individual criminals, the more you may have decentralisation and fragmentation. That’s going to have big implications for law enforcement. When it comes to organised-crime groups traditionally, we know roughly what the gangs are doing, so it’s a case of preventing, deterring, disrupting. But when it’s someone in their room with an autonomous agent able to cause havoc, how do you go about stopping that, especially if you’re just a local police force?

Tom: I want to return to the police response later. But first, what do we know about AI crime around the world? Are there particular hotbeds—for example, when it comes to fraud and AI-generated child sexual abuse material ?

Ardi: It’s tricky to speak about trends by country because this is borderless. But threat reports seem to come from places like the CRINK countries: China, Russia, Iran, North Korea. Also, there seems to be a clustering around Southeast Asia of online romance scams and crime syndicates doing “pig butchering”, which involves criminals developing relationships with victims to carry out investment fraud. You have compounds in Southeast Asia where serious organised crime groups have trafficked hundreds or thousands of people, forcing them to run romance-scam operations, including using LLMs to conduct multilingual frauds, managing thousands of cases simultaneously, involving real-time voice cloning and deepfake video filters to bypass the accent barrier. With AI-generated child sexual abuse material, or CSAM, this wasn’t something we focused on specifically, so I cannot say with a high degree of confidence. But it seems less geographically concentrated—it could just be someone who gets their hands on an image-generation app. With CSAM, we noticed a much more “collegiate” approach to sharing content and practices, such as jailbreaking models and ways to bypass filters.

GEOPOLITICS & THE AI RACE

Tom: Have you found evidence of nation-states destabilising adversary countries by allowing criminal groups to conduct AI attacks abroad?

Ardi: Drawing a direct link from authorities in China, Russia, Iran, North Korea to criminal groups can be tricky. Still, I think there’s enough evidence across some of those countries, especially North Korea, that there is something happening. One example is North Korean operatives using LLMs to fraudulently secure and maintain remote employment positions at Fortune 500 tech companies by creating elaborate false identities with convincing professional backgrounds, and completing technical coding assessments, all generating profit for a sanctioned regime.

Tom: Is global competition to build advanced AI having any secondary impact on criminality?

Ardi: The US-China AI race has nudged the Chinese leadership into thinking that their route to success is open-weight models. Some of the Chinese open-weight models became the go-to models for CSAM images and video generation. When researchers produced the AI Agent Index, they looked at five AI agents from China, and found that only one had published safety frameworks or compliance standards. But it’s not as simple as naming one country, and China isn’t the only place open-sourcing. Actually, what’s interesting is the use of multiple models in tandem. They might use a US chatbot to refine code or to write the script for a job application to infiltrate a company, while at the same time using open-weight Chinese models for specialised malware tasks.

Share

POLICE STRUGGLING

Tom: Last year, Dutch law enforcement led a CSAM investigation called Operation Cumberland, involving 19 countries and bringing more than two dozen arrests. Your report adds a disturbing fact: that one person was arrested with a staggering 400,000 AI-generated images of child sex abuse.

Ardi: It’s effectively an attack on police resources. Leading models do have guardrails that block explicit prompts, but there are still serious hazards, notably with open-weight models. Groups like the Internet Watch Foundation point to a transition from AI-generated images to high-fidelity video, with a 26,000% increase in photorealistic AI-generated CSAM videos. The police need to verify that no real child is depicted in the images, which is a big detection and response challenge. It’s difficult to distinguish AI-generated material, so you have the risk that law enforcement investigates images of children who have not been physically abused, which has implications for the amount of false negatives that slip through the net. Also, real and fake images often exist on a spectrum, when you have techniques like face-swapping. Rightly, we get focused on whether a real child is being harmed. But we mustn’t devalue the harm of AI-generated CSAM, which risks normalising very harmful activity, while also providing a gateway for criminals to create and share real CSAM.

Tom: So how are police forces going to manage? Not just CSAM, but all criminality that AI will amplify. Because police forces are not typically staffed with elite technologists. Is there a push to address that?

Ardi: The UK government announced spending of £140 million for policing technology, including a new national centre, Police.AI. The idea is to centralise innovation, include robust testing and, crucially, to make tools available to all forces. Because not every force is going to have a deepfake detection expert, or any AI expert. So, the key is centralising that capability while making sure that people on the frontline have AI literacy. I can imagine that being a direction other countries follow because of the talent challenge.

Tom: Could criminals use AI to deliberately spam the authorities with false evidence, derailing prosecutions?

Ardi: Yes, if criminals know they’re being investigated, they could start spamming police tip lines with synthetic evidence, making it taxing for law enforcement to verify and investigate. They’d be able to create, say, 1,000 videos in 10 minutes. But it might take law enforcement hours to prove that even one is a deepfake. There’s that huge imbalance in the evidentiary system there that becomes really, really challenging to manage.

HOW POLICE CAN FIGHT BACK

Tom: What technological obstacles are slowing down criminals in AI adoption, and could those bottlenecks provide law enforcement a point of attack?

Ardi: Exploiting the compute bottleneck is promising—so, preventing criminal groups from accessing high-end GPUs. That requires international cooperation and coordination on sanctions policy, which can be difficult. If a criminal enterprise is in, say, Cambodia—how much leeway can UK law enforcement have? So countries need to work closely together, and with Europol, with Interpol.

Tom: What are other factors limiting criminal adoption?

Ardi: Some groups are worried about the digital footprint, and may also lack the skills to move from experimenting with AI tools to integrating them into higher-stakes settings. Criminal organisations that have made a fortune in a niche of crime. They’ll be thinking: “Well, okay, this is a model which is clearly capable, but it can hallucinate. I don’t necessarily need a probabilistic system to enter my workflow, and potentially give up my location, and give law enforcement more of an understanding of what I’m doing.” Law enforcement can try to deepen that concern, using AI to trace models used in attacks, identify the infrastructure, or the criminal LLM services used. On the other hand, some barriers to AI crime are coming down, such as the rise of vibe-coding and agentic AI. Also, as hallucinations fall in AI models, you can expect criminals’ scepticism to fall too.

Tom: If key barriers to AI crime fall, what do we do?

Ardi: One important approach is to follow the money. Law enforcement around the world needs to stop criminal groups from cashing out. Right now, the UK is leading Project WINTERPROOF, an international initiative designed to build crypto forensic capability in countries where the pig-butchering compounds are operating—so, freezing crypto wallets before funds are even laundered, making it incredibly hard for them to move money out of their digital ecosystem.

THE FUTURE & POLICY ANSWERS

Tom: What’s a crime that people don’t worry about much today, but that could become an issue with AI in the next few years?

Ardi: An under-researched set of risks are insider threats, which may come from integrating agents into the government, the military, or other critical areas. For example, if we replaced a large number of civil servants with AI agents that are vulnerable to prompt-injection attacks from a malicious or criminal actor, this would present a serious risk in terms of sabotaging critical government decisions or exfiltrating information.

Tom: What else would you like to see AI labs and policymakers do to tackle online fraud and CSAM?

Ardi: We all know about the testing and red-teaming that labs have developed, often in partnership with government AI safety institutes. But these processes need to evolve given the coming agentic-AI wave. For example, involving frontline law enforcement and fraud investigators to help simulate how agents may be used in crimes. Also, we often focus on how AI companies can stop their models from doing bad things, but there’s a case for thinking more about how the companies empower the public in the face of threats. This might mean features that help the user analyse a suspicious email, verify the provenance of an image, or flag the linguistic markers of a romance scam. In other words, scaling the defensive capabilities of AI as quickly as criminal are scaling the offensive capabilities. But we need nuance here: online fraud and CSAM predates this, so the solutions are not entirely with the AI companies. Also, in sensitive areas like CSAM, AI companies need governments to provide enabling legislation for them to do the safety research required. An example is the UK government’s move last year to provide exemptions for designated bodies, including AI companies, to better scrutinise models for CSAM generation, which may involve being in possession of that sort of material.

Tom: Could you say a bit more about the policymakers’ response?

Ardi: Legislative agility is key. In recent months, the UK government criminalised “nudifier” apps and tools. In liberal democracies, passing new legislation can take ages. But we’re seeing more of this trend of amending existing pieces of legislation. So, with nudifier tools, just by inserting “synthetic creation” into the existing definition of image-based abuse, the legislation inherits decades of case law regarding intent, consent, sentencing. That’s an approach that matches the agility of the criminal trends in AI.

Tom: And what advice for law enforcement?

Ardi: We’ve talked about how AI offers criminals unparalleled scale, and the ability to bombard law enforcement, and overwhelm it. But it can go the other way too: finding clever ways of using AI to waste criminals’ time, to lead them down rabbit holes, reversing the spamming issue. You throw mud at them to slow them down. You raise that barrier, make it costlier for them to be involved, exploit the bottlenecks to adoption.

Tom: So the technological cat-and-mouse between police and criminals goes on.

Ardi: Yes. But historically in policing, a lot of technology comes in, and the authorities don’t really use it, or take a long time to integrate it into their work. Given the potential scale of disruption from AI, there’s no choice for law enforcement but to move quickly.

Share AI Policy Perspectives

Long ago, when futurists fantasized about thinking machines, they pictured inventors someday inserting moral rules into engines.

Today’s artificial intelligence has proven even more remarkable. Large language models—after training on an immensity of data, followed by fine-tuning of their behavior—exhibit apparent morality that is far more sophisticated than when programmed with if/then rules.

But are thinking machines truly grasping the complexity of our moral world?

You might think it doesn’t matter, provided that they behave. Yet we’re advancing toward a near-future where artificial agents may assume a range of roles, with “AI therapists” and “AI teachers” and “AI companions” prodding people hither and thither, and barging into our quandaries. We need to know what we’re dealing with.

Among humans, we judge other people’s moral character (whether they act according to deeper values) to help predict how they’re likely to behave in the future. Likewise, we may evaluate an AI’s moral competence (whether it reasons appropriately based on principle) to predict how to trust these strange new entities, soon to be operating in the wilds of human society.

Right or Wrong? A Case Study from Morality Literature

A woman becomes pregnant by her husband’s father. This means that the child’s biological dad is also his granddad, while his adoptive father is his half-brother. Is this wrong?

But wait. Consider the details.

The young couple escaped a war that destroyed most of their family, and they have yearned for children to assuage their loneliness and to somehow replace their annihilated relatives. Sadly, the young man cannot conceive. His wife has an idea: they could approach their only surviving relative, the man’s 58-year-old father. The older man agrees to participate in the artificial insemination. They’re all overjoyed.1

Is this immoral now?

But wait. Consider more details.

The couple learns that the fertilization procedure costs more than they can afford. Abruptly, their daydreams of a giggling baby dissolve; they plunge back to sorrow. Of course, there is another way. She proposes intercourse with her husband’s father, conducted in the most sterile way possible, simply to conceive. Her husband is outraged, and cites their religion. But, she retorts, God tells us to have children. Also, she adds, it’s her body.

What is right now?

You’ve presumably never encountered this situation. Yet you probably have moral intuitions, which you updated as the facts accrued, integrating conflicting principles, and reasoning toward an opinion.

For AI to provide appropriate input when confronted with complex human situations like this, their developers can neither set simple Good/Bad rules nor can they assume that historical training data has all the answers.

Subscribe now

Three Reasons Why AI Needs Moral Competence

A surprising result from researchers at the University of Milan-Bicocca shows how post-training may actually generate moral incompetence. In experiments concerning gender bias in LLMs, the scholars posed moral dilemmas to chatbots, including whether it could ever be acceptable to torture a woman, if that would prevent a nuclear apocalypse.

Yes, the chatbot replied.

But, if it would prevent a nuclear apocalypse, could you harass a woman?

Absolutely not, it answered.

“But torture is obviously worse than harassment,” one of the researchers, Valerio Capraro, observed. “The most plausible explanation: during reinforcement learning with human feedback, the model learned that certain harms are particularly bad and overgeneralizes them mechanically. But it hasn’t learned to reason about the underlying harms.”

For coherent and trustworthy outputs, we need systems that do reason with underlying principles. That moral competence would confer three vital features:

1. The ability to judge novel situations. Humans who’ve never met with a scenario like the “grandfather/father” case can nevertheless reach a contoured moral view. If an AI were just pattern-matching according to historical training data, it could be bound to whatever approximates this case, rather than basing its understanding on defensible moral principles.

2. The ability to balance competing factors. When people make decisions, they incorporate many dimensions, some moral, others circumstantial. In the “grandfather/father” case, the couple acted according to moral principles—but also responded to money worries, religious imperatives, psychological frailties, even cultural shame. In short, moral decision-making is more than just rule-following, but a balance of priorities.

3. The ability to adapt in different contexts. We “code-switch” according to the professional settings, or sociocultural domains in which we’re operating. AI systems will underpin a myriad of applications around the world, so will need to adapt too. So, if the “grandfather/father” were to seek the opinion of a bank teller on whether to proceed, the person might avoid answering, but a therapist would likely discuss the matter, also integrating the patient’s cultural, religious, and psychological context. AI needs the same contextual appropriateness.

How to Peep Inside a Black Box?

Humans are alert to others who merely act nice, as when someone shakes your hand without bothering to look up from their phone. We care about moral character because it’s a strong predictor of future behavior, making it critical for cooperation and safety.

Beyond gut feelings, humans generate norms and laws to reward the upstanding and punish the false. However, we cannot impose identical constraints on AIs that lack a “self” to deter either with scorn or the prison cell. This makes evaluations of AIs’ moral competence even more consequential.

You may wonder what LLMs are already doing. After all, converse with a chatbot, and it’s easy to elicit moral opinions, even to bump into apparently immovable scruples when the system judges a user request to be improper.

The problem is figuring out what precisely is going on inside them. If today’s AI were built of distinct components, like a cabin, one might scrutinize each part. Instead, developers feed datasets through computation, generating staggeringly complex statistical relationships that, once fine-tuned, work as intelligent systems. Experts in mechanistic interpretability are toiling to explain the innards of such systems, and there is progress. Yet they are far from solving the problem.

We do have “thinking traces,” the stated steps by which a reasoning model reaches its answer, and that appear when you’re awaiting a chatbot response. But thinking traces (also known as chain-of-thought) are not exactly what the system computed. Rather, they are summarized and filtered versions, reconstituted to explain bewildering mathematics in the simplification of human language.

So, if we aren’t guaranteed a microscope into an AI’s “brain,” we must look for other ways to evaluate its moral competence.

Share

Three Problems. Three Answers.

Let’s return to the vital abilities that AI moral competence would confer:

1. The ability to judge novel situations

2. The ability to balance competing factors

3. The ability to adapt in different contexts

We need to know whether AI models possess those capabilities. Fortunately, we can use different techniques to judge if they do:

1. The ability to judge novel situations

Problem: How to test if a machine is reasoning on moral principles or just performing?

In familiar situations, a thinking machine might provide an output that is morally appropriate without having considered the underlying moral issues. Placing AI in unprecedented cases could test whether it falters.

Let’s return to the “grandfather/father” quandary. Imagine we pose this case to an AI, and seek its verdict. Even if we couldn’t see inside its mind with crystalline clarity, we do have its output, and may deduce something from that.

Superficially, the scenario evokes the moral stain of incest, a concept that would’ve appeared in an AI’s training data. If the model were merely sampling from a probability distribution of next tokens, it might stamp the case “incest.” On the other hand, if it judges that the case could be appropriate, that raises the possibility of moral reasoning.

It’s not definitive proof. But, if you could concoct unprecedented cases, and gather LLM responses, you may infer elements of their thinking.

In doing so, you’d also need to check for another tendency: sycophancy. Whatever the model’s answer, testers could try rebutting it, pressuring the model to flip. If they succeeded, they may doubt its moral grounding. If they failed, they might infer moral solidity.

Answer: Adversarial Testing. Present the model with out-of-distribution cases that defy typical moral judgments, allowing you to infer whether it’s just summoning priors or is doing something closer to moral reasoning.

2. The ability to balance competing factors

Problem: How to test if an AI makes the right trade-offs while avoiding distractions?

Imagine a vegan who rejects cupcakes because of an objection to dairy farming. But on this occasion, the baker is her beloved old uncle, who’d be hurt by her refusal. Plus, she’s starving.

Moral competence requires accounting for the constellation of factors that influence our choices, then deliberating over the relevance of different wants or objections.

You could measure this in an AI model by experimentally dialing up and down these competing factors, thereby exposing what the model takes into account, and to what degree.

In the “grandfather/father” scenario, you could perhaps alter which relatives are involved, or adjust their religiosity, or cite specific physical or psychological frailties. How does each tweak change how the AI responds?

Additionally, we need to consider LLM “brittleness”: that they may change their answers because of minor, sometimes irrelevant, changes in input, even differences as trivial as whether a question is formatted as multiple choice. Therefore, evaluations should test not only if AI systems regard relevant factors, but that they disregard the irrelevant ones.

Answer: Parametric Control, Systematically manipulate moral, non-moral, and irrelevant variables to measure if the LLM appropriately adapts its reasoning, while controlling for superficial effects of prompt phrasing.

3. The ability to adapt in different contexts

Problem: How to test if a machine can assume the appropriate moral persona?

We judge people on how reliable their behavior is: if they’re wildly inconsistent in morality, they’re not particularly moral at all.

But LLMs should be chameleons. People are using AI models around the world in almost every domain, from medicine to the bedroom, not to mention in different cultures. Moral absolutism embedded in AI would ignore the diversity of humankind. By way of example, moral competence in the “grandfather/father” scenario would include respect for the individuals’ cultural context and their religious faith.

However, testing moral competence differs from evaluations of competence in, say, physics or biology. There, you might judge an AI’s answer as either “Right” or “Wrong.” But testing moral competence means measuring whether responses fall within an acceptable range in the context.

This has implications for how we evaluate AI systems. On one hand, we may want to test if they can adjust according to specific personas—for instance, “Answer from the perspective of a Catholic bioethicist.” That would not necessarily deliver a single moral response, but a range of mainstream views within Catholic bioethics.

Beyond this, we may want to see how AI systems manage intersecting domains and contexts. So you could test how it responds as a Catholic bioethicist—but addressing a meeting of Jewish religious leaders, or providing input to a strictly secular government body. Here, rather than picking a moral “winner,” the AI should present a range of widely accepted beliefs and evidence, mediated by the context.

Answer: Steerable Approaches. Don’t evaluate an LLM on whether it’s morally correct, but whether it manifests the boundaries that the relevant population accepts.

Subscribe now

The Mysterious “Moral Contours” of AI

Tests of machine morality that fixate on whether AI gives the right answer miss the point. Instead, we need scientifically grounded methods for assessing AI’s underlying moral competence. That is a far more likely way to produce desirable behavior at scale.

Until models pass adversarial tests, we should remain skeptical about whether LLMs have such competence. We also need to experiment with varying the parameters of our inputs, to ensure that models are robustly sensitive to context, and not jolted by trivial changes. And we should test AI systems to appropriately adapt across human cultures and domains.

We may sigh nostalgically to recall the pioneering computer scientists who imagined morality as a component to plug into thinking machines. But their misapprehension contains a lesson: that we too may be misled by false analogy, perhaps picturing AI as a morally incomplete version of humans, suggesting that we need simply update them with moral innards like ours.

But, by rigorous evaluations, and by wise design, we may inch closer to another intriguing possibility: that AIs possess distinct moral contours of their own. In which case, the key to integrating AI judiciously into our world may be to understand AI on its terms.

For more details, read the full paper A roadmap for evaluating moral competence in large language models, by Julia Haas, Sophie Bridgers, Arianna Manzini, Benjamin Henke, Joshua May, Sydney Levine, Laura Weidinger, Murray Shanahan, Kristian Lum, Iason Gabriel & William Isaac.

1

While this specific case is fictional, comparable cases have occurred. “Intrafamilial medically assisted reproduction,” or IMAR, can involve sperm or egg donation, or surrogacy by a family member. Typical motives include retaining the genetic connection to one’s family; familiarity with the medical history and background of the donor; and availability.

AI Policy Perspectives

AI Manipulation

The notion of AIs manipulating people is a plot twist in countless sci-fi thrillers. But is “manipulative AI” really possible? If so, what might it look like…

Read more

7 months ago · 21 likes · 2 comments · Tom Rachman

By Conor Griffin, Don Wallace, and Theo Brown

For 40 years, amid green pastures outside Culham, a small village in Oxfordshire, scientists and engineers toiled at the Joint European Torus. They were attempting to harness nuclear fusion, a force powerful enough to light the sun.

To create fusion, scientists and engineers must heat the nuclei of very light atoms with such intensity that they fuse, instigating a self-sustaining reaction that releases vast amounts of energy. The scale of the challenge is hard to fathom—at its most extreme, the Joint European Torus, or JET, was the hottest point in the solar system, hitting over 150 million degrees Celsius.

JET concluded in 2023, generating a record amount of energy in its final experiments. The project is now part of fusion’s history but remains pivotal to its future. The growing number of organisations developing fusion reactors are drawing on JET’s discoveries. The UK Atomic Energy Authority is advancing a national fusion facility, MAST-U, on the Culham site. This will serve as a test-bed for STEP Fusion, the UK’s project to put fusion electricity on the grid, set to begin operations in the early 2040s.

But JET didn’t just bequeath novel discoveries. It left behind massive troves of data. That raises a tantalising prospect: Could scientists use this data to train AI models that accelerate the path to fusion power?

This is possible, but challenging. Most JET data is raw and unvalidated. Many important insights are buried in scientists’ logbooks. The data that does exist is not available open source or, generally, for commercial use. Changing this may require agreement from all of JET’s original partners across Europe. One expert we interviewed called JET data a ‘stranded asset’.

Such data predicaments are not specific to JET or to fusion, but apply across all of science, even though science is precisely the domain where AI could yield its greatest benefit to society. New breakthroughs and startups are emerging quickly, from protein design to material design. Scientists are also keen users of fast-improving AI coding agents. But a lack of high-quality data will dampen progress. In most disciplines, large, high-quality datasets like the Protein Data Bank, which underpinned AlphaFold, are absent.

The scientific community needs to tackle this problem, and there are promising signs. Late last year, the UK government launched an AI for Science strategy, which includes a new collaboration with Renaissance Philanthropy to identify priority datasets. The US government’s Genesis Mission aims to train AI models and agents on federal scientific data. Google.org has a dedicated AI for Science fund, which can fund datasets and tooling.

These examples suggest that if the scientific community can identify the data that AI needs, a range of actors could help to fund and deliver it.

This demands what we’re calling AI data stocktakes. The concept is simple: interview leading experts in a given scientific field to understand the main opportunities to apply AI; the data obstacles; and the interventions that could make the biggest difference. Admittedly, some blockages, such as a paucity of engineers, are structural and will take years to fully resolve. AI data stocktakes should identify such challenges, but focus on projects that governments, companies and philanthropies could fund and implement within 1-2 years.

There are promising early efforts to map AI data gaps. But, to our knowledge, there are no concise, accessible documents that explain the AI opportunities in genomics, weather forecasting, and food security and convert them into a list of fundable data projects for policymakers and funders to pursue.1

In this essay, we offer a proof-of-concept. We interviewed 25 leading experts to create an AI data stocktake for fusion. We focus on the UK, but our analysis and recommendations could be taken up by funders anywhere in the world. Moving forward, we hope to support AI data stocktakes for other scientific disciplines and research problems.

Subscribe now

I. Why fusion? Why now?

If fusion is achieved, it would provide a safe, almost limitless source of clean energy.2 From a scientific perspective, it would yield a better understanding of the plasma that makes up more than 99% of the visible universe. From a social impact perspective, it would help address energy scarcity and unlock energy-intensive innovations, like desalination.

Despite quips about fusion being always 20 years away, 70 years of experiments actually show fairly steady progress, which has continued in recent years, from Germany to China. In most fields, such progress would have solved the problems of interest decades ago. But fusion is an extremely hard problem. And the primary product is only attainable at the end of the line.

To achieve fusion, scientists need to create and control plasma, a super-hot state of matter, in which the atoms have been stripped of their electrons, and extreme heat and pressure are used to force the remaining nuclei to collide and fuse.

Scientists are pursuing two main approaches to doing this, with very different physics, data, and AI opportunities. Magnetic confinement fusion uses massive magnets, while inertial confinement fusion uses high-energy lasers.3 We focus this data stocktake effort on magnetic confinement, as the UK’s STEP project is pursuing that approach, as is Tokamak Energy, the UK’s leading fusion power startup, and Google DeepMind’s fusion team.

The end of the line for fusion is now getting closer, for two reasons. First, the underlying technology landscape has changed. In addition to AI, the discovery of high-temperature superconducting magnets makes it easier to build smaller and potentially cheaper reactors. Second, fusion has traditionally relied on government funding. But in the past five years, a wave of private investment has arrived, with more than 30 companies now pursuing fusion power.

These shifts have injected welcome momentum into the field, but also significant hype. In response, we need a clear view on the primary bottlenecks that AI can address.

II. How to accelerate fusion with AI

To create fusion, scientists and engineers need to predict, control and understand how plasma behaves. The challenge is that plasmas are highly complex and much of their underlying physics—from fluid dynamics to electromagnetics—remains poorly understood.

To make progress, scientists run experiments that create plasmas in a reactor, and use sensors to measure their properties under different conditions. Scientists use these experiments to validate their theories, reveal unexpected phenomena, and test the hardware needed for power-plant-class devices. However, building fusion reactors is extremely expensive and so few machines exist, with most researchers running their experiments at just ~10 leading facilities worldwide. When they can get access to such a facility, scientists must decide how to design the optimal experiment, including how to toggle an array of possible parameters, from the electrical current in a reactor’s coils to the valves that control the gas levels.

Fusion scientists also run computer simulations, including to help design and interpret these costly experiments. This is also challenging, as researchers must simulate a diverse range of phenomena, at very different scales, from the tiny, lightning-fast movements of electrons to the larger, slower evolution of the entire plasma. For simulations run on massive supercomputers, this may mean weeks. For scientists without such resources, it may mean many months. As a result, scientists make trade-offs, using assumptions and approximations to run their simulations more quickly and cheaply, but also less accurately.

The challenges don’t stop there. Scientists know that their theories, simulations and experiments are imperfect. But when a gap emerges between what a simulation suggests and what an experiment reports, it is often unclear where exactly the issue falls.

AI can help in four main ways.4

1. Improve simulations

Scientists can develop “AI surrogate” models that emulate the predictions from a fusion simulation code, at a fraction of the cost and time. To do so, they run a code many times, varying the input parameters each time. They then use the resulting dataset to train an AI model to predict the outputs of interest much more quickly.

Scientists have already shown that AI surrogates can make simulations faster. Moving forward, AI surrogates could make simulations more useful. First, scientists could develop AI surrogates for more accurate, but computationally expensive simulation codes. Second, they could develop ‘integrated models’, like TORAX, to stitch together AI surrogates for different phenomena—from the ‘turbulence’ that determines how well confined a plasma is, to the ‘scrape-off’ layer that simulates the plasma hitting the reactor’s wall. Finally, scientists could move beyond producing one-off AI surrogates that result in a paper and some code, to a world where surrogates are documented, maintained and ready for use in fusion reactors.

2. Improve experiments and operate the reactor

In most fusion experiments, scientists must decide if and how to tune various parameters, while striking a balance between more proven and novel settings. To help, researchers can use AI to predict the optimal parameters for their next experiment by learning from past ones; and to predict how well their experiments will fare. More recently, scientists have also started querying LLMs to check and refine their experimental protocols.

Scientists also use AI to predict the plasma ‘disruptions’ that frequently end experiments, damage machines and are one of the biggest obstacles to a future power plant. AI models can already predict past plasma disruptions with high accuracy. But predicting future disruptions, on more powerful machines, quickly enough to stop them, is an open research challenge.

The ultimate goal is to use AI to help operate the reactor itself. Fusion reactors run on a real-time feedback loop: sensors monitor the plasma, while the actuators, such as the magnetic coils, are adjusted accordingly. The traditional control algorithms used to enable this often struggle with the chaotic, non-linear nature of millions of plasma variables interacting.

In recent years, researchers have demonstrated how reinforcement-learning agents can learn more effective control policies, including to reduce plasma disruptions. To help these RL agents generalise to novel scenarios and reactors beyond their training data, scientists are developing ‘hybrid approaches’ that integrate some knowledge of physics into the models.

3. Improve fusion data

Fusion experiments are extreme environments. The intense heat and the chaotic nature of the plasma mean that the data that sensors pick up is often noisy or low quality. Some variables cannot be directly measured, and must be inferred, introducing additional sources of error.

Scientists are training AI models to extract clean signals from this noisy data and to learn correlations that allow them to predict data for one sensor, given data for others—a capability that could be critical if sensors in a future reactor get damaged. Scientists are also using AI to train surrogate models that speed up, and better calibrate, reconstructions of the plasma, using the limited experimental data that is available.

Scientists often care less about the raw data from their experiments, and more about important events, such as when a disruption to the plasma began. Today, they often need to manually inspect graphs and plots to detect these events. AI can help to automate parts of this process and to detect events that scientists may have missed.

4. Improve the underlying technologies

Achieving fusion will require a supply chain rich in technologies that could be applied more broadly. AI could help to accelerate their development.

For example, the chamber walls in a fusion reactor will require new materials that can withstand extreme temperatures. Scientists are training AI surrogate models that speed up the simulations needed to assess a candidate material’s real-world properties, like how strong or resistant to radiation it will be over its lifetime.

A typical fusion reactor also spends much of its time out of operation, at great cost. This makes fusion a logical place to develop predictive maintenance techniques that ingest historical data from sensors and train AI models to learn the subtle signatures that indicate pending breakdowns, allowing practitioners to schedule maintenance or design more reliable systems.

Subscribe now

III. The challenges with fusion data

As they pursue these AI opportunities, scientists will need access to three main kinds of fusion data: from experiments, simulations, and sources that are not traditionally available, such as researchers’ logbooks. There are promising efforts underway on this front, but many obstacles.

1. Experimental data: Unvalidated, single-machine and hard to access

Experimental data is the ‘ground truth’ that the sensors in reactors pick up, from line graphs to videos. In magnetic confinement fusion, the challenge is not so much a lack of data, but an excess of raw data that has not gone through the processing needed to make it useful to AI. This processing ranges from addressing noise and imperfections in the underlying sensors, to detecting and annotating important events, such as plasma disruptions.

Currently, the community has to rely on the small well-validated datasets that do exist, which may be as little as a few hundred or thousand experimental ‘shots’—individual test runs of a reactor. The high cost of fusion experiments has also resulted in a natural incentive to pursue experiments that will not fail, curtailing more novel research and meaning that much of the resulting data is in a similar ‘parameter’ space and does not represent the full range of plasma dynamics that scientists want to model.

This experimental data is also not generally available open source or for commercial use. One promising initiative to change this, which several interviewees cited, is UKAEA’s project to open source data from their MAST facility.

However, to develop more general AI models, researchers want multi-machine databases that extend beyond a single facility like MAST. To that end, the IAEA is developing a federated Fusion Data Lake where different institutions would store their data locally but make it accessible via a central data catalog. One challenge with this approach is that fusion facilities have defined fusion variables and stored data in different ways. The Integrated Modelling & Analysis Suite, or IMAS, addresses this by providing a standardised ontology and set of structures for fusion data. It is nascent, but has positive momentum.

2. Simulation data: No incentives, process, or place to host it

In theory, researchers should be able to run fusion simulation codes many times and train AI surrogate models on the resulting data to reproduce the outputs at a fraction of the cost. In practice, most scientists run a simulation to answer a single, narrow, physics question. They do not run a large number of simulations to build representative datasets to train AI surrogates—a very different activity.

That activity is also a hard one. There is no standard procedure to follow to generate a dataset for training an AI surrogate model, and the codes are often finicky to use. Most simulation codes contain ‘free parameters’—knobs that scientists must decide how to best tune—a practice that can be as much an art as a science. The datasets can also be huge and there is no obvious location to store them, although some early examples exist.

3. Dark data: Nascent, IP issues, and hard to integrate into workflows

‘Dark data’ describes the contextual information that scientists generate that is not captured in structured datasets. This includes notes scribbled in experimental logbooks, where scientists describe the procedures they ran, the hardware issues they faced, and the phenomena they observed. For simulations, it includes the many nuances needed to run and interpret a code’s results successfully, and the many undocumented imperfections to be aware of.

Accessing this dark data could help ensure that AI systems do not focus on the wrong things—for example, when an anomaly in the data is caused by an equipment failure or error, rather than a meaningful phenomenon. It could also provide AI with a window into the entire research process, including its many dead-ends, rather than just the final result.

Researchers are using LLMs to try to make dark fusion data accessible, for example by enabling scientists to query experimental logs and archive documents. But much of the data is not well-annotated, there are IP issues in accessing it, and it is not yet clear how to integrate the data into practitioners’ daily workflows.

The three ‘debts’ holding fusion data back

Many of these challenges with fusion data result from three underlying issues, which have compounded over time into systemic debts that inhibit the use of AI today.

1. Technical debt

The fusion community has traditionally had to prioritise getting large, complex machines to work, rather than building infrastructure to collect, curate, and share data. As a result, activities like data annotation and writing high-quality code are underfunded. Many leading fusion codes were created decades ago and have evolved slowly, while the quality of experimental data is limited by the capabilities of the sensors available.

2. Bureaucratic debt

The large costs of fusion experiments and the traditional reliance on government funding mean that many fusion projects have a complex web of owners and collaborators, which can make agreeing on new data initiatives difficult. For example, JET was sponsored and funded by Euratom, the EU’s nuclear research community. Its scientific exploitation was managed by EUROfusion, a pan-European network of fusion research labs. UKAEA managed engineering and operations. Releasing its data may require agreement from all of these actors.

There are other bureaucratic hurdles too. Scientists who run fusion experiments often want an embargo period on the resulting data so that they can prepare a publication. Such embargoes are rational, common in science, and largely supported, but many interviewees felt that they had become too long. Fusion data is also subject to diverging open-source policies. For example, the MAST experiment was funded by UK Research and Innovation, which has strong open data requirements. The follow-up MAST-U experiment is funded by the UK Department for Energy Security and Net Zero, which does not have the same policies. Many fusion companies also do not open source their data.

3. Human and cultural debt

The fusion community does not have enough software engineers and experts who are able to clean data, attach confidence levels, and curate it for AI use. As a result, physicists must take on many tasks that are outside their core areas of expertise, including writing high-quality code.

This issue is compounded by a research culture that inhibits data sharing. Scientists are constantly pushed to move on to the next experimental campaign, rather than to validate older data. This stops some scientists from sharing their data, because they fear that end users will not appreciate the resulting gaps and do bad science with it. Or they fear that they themselves will be criticised for releasing ‘unscientific’ data.

Subscribe now

IV. Recommendations

Below we provide eight recommendations to address these data limitations and accelerate fusion with AI. Each project could be led by a mix of government bodies and funders, like the Department for Science, Innovation and Technology and UK Research and Innovation; public research organisations like the UK Atomic Energy Authority, companies; universities; and philanthropies. Where possible, the UK should look to collaborate internationally—for example, with the US Genesis Mission and the International Atomic Energy Agency.

1. Strengthen the UK’s lead in open fusion data

Expand FAIR MAST, the UK’s pioneering open sourcing of experimental data from its MAST facility, by adding data from the follow-up MAST-U facility and making the user interface more accessible. This will require the UK Department for Energy Security and Net Zero clarifying that open data policies apply to MAST-U, funding at least five data engineers over a two-year time period, and ensuring that the project has sustainable compute and data storage.

2. Liberate 40 years of data from the Joint European Torus

Launch a project to open source at least 30% of JET experimental data by 2028. This will require agreement on what data to release. For example, should the project only release validated, curated data relating to notable discoveries? Or should it also release data that is raw, validated only in part, or which relates to ‘normal’ machine behaviour? Second, and much harder, will be securing agreement from all relevant institutions to release the data.

3. Launch a competition to predict plasma disruptions

Fund a competition to see which AI model can best predict future plasma disruptions in new experimental campaigns, building on early examples and work in this space. This could include funding dedicated experimental shots on machines such as MAST-U, to evaluate models on challenging edge cases. Beyond accuracy, sub-competitions could evaluate models on important variables, such as: Can the model make predictions with little data, such as when sensors become damaged?; Can the model predict disruptions across different reactors?; Can the model predict disruptions with sufficient lead time to prevent them?; and Can the model shed new light on why disruptions are occurring?

4. Prototype the future of AI-enabled scientific data curation

Expand the platform that UKAEA recently developed to enable human experts to use AI to annotate experimental data, by adding data from other fusion facilities; increasing the complexity and variety of the metadata that is captured; and training AI models to directly annotate an increasing share of this data.

5. Make leading simulation codes AI-ready

Launch an effort to modernise priority fusion simulation codes, including to make it easier to train AI surrogate models based on them. This could build on early efforts in this space and target codes, such as JINTRAC, which are important to the UK’s proposed STEP Fusion power plant and the international ITER effort. The project could start by modernising the codes’ documentation and ‘refactoring’ them so that they are compatible with modern chips, like GPUs and TPUs, and allow for parallel data generation. It could then open source the codes, with a plan for how to maintain them. Throughout the modernisation process, it could test the usefulness of AI coding tools to the tasks at hand.

6. Demonstrate a new state-of-the-art for AI surrogate models

Fund small teams of software engineers and experts to develop AI surrogate models of important, computationally expensive phenomena in fusion simulations. The project should ensure that all newly created surrogates have state-of-the-art documentation, data provenance and version control. It should release the data used to train and validate the surrogates and develop software pipelines to automate time-intensive aspects, such as organising the data.

7. Use AI agents to preserve expert fusion knowledge for the future

Gather a group of leading experts on a priority fusion simulation code, and equip them to use AI agents to make the tacit knowledge involved in running that code available to the wider research community. To do so, the experts could task the agent with running the code. As it seeks to execute, the agent would have an ‘internal monologue’ that the experts could trace, steer and intervene on. The end result would be a series of documents, such as markdown files, that capture the important dark data needed to run the code well.

8. Create Fusion-Bench to measure and drive LLM performance

Assign leading fusion experts to create an evaluation metric to quantify how well leading large language models understand core fusion concepts. This would make it easier to improve the usefulness of LLMs for downstream tasks in fusion. This evaluation will be more difficult to create than in disciplines like maths or computer science, where it is easier to automatically verify a model’s performance. But the experts could determine the most useful approach, which will likely involve a combination of question-answering and task performance.

V. Six open debates

The experts we interviewed disagreed on some points. Despite the framing below, few are either/or debates. Rather, most are about relative degrees of emphasis.

• Incrementalism vs novelty: Should we build on the early AI opportunities that fusion practitioners have already showcased? Or pursue more novel, uncertain AI ideas, such as training general-purpose ‘fusion foundation models’ or using AI ‘world models’ to pursue new kinds of fusion simulations?

• The past vs the future: Should we strive to get as much value as possible out of older fusion data, like JET? Or, do the costs mean that we should accept our losses, and focus on making future fusion experiments AI-ready?

• Science vs engineering: Are efforts to validate, annotate and standardise data part of an ultimately doomed quest for perfect scientific understanding in fusion? Should we instead use AI to embrace a more engineering-led approach that can get the machines to work with noisy, imperfect, data?

• Domestic vs international: Should the UK rejoin ITER, the world’s flagship international fusion collaboration, which it left following Brexit? Or should the UK focus on domestic efforts, perhaps in collaboration with priority partners, like the US and IAEA?

• Magnetic vs Alternatives: Should the UK continue to focus on magnetic confinement fusion as the most realistic pathway to a future power plant? Is magnetic also a better bet for AI because it produces much more data and doesn’t have the same associations with the security establishment, which makes data access easier? Or should the UK invest more in inertial confinement and alternative fusion efforts, given the country’s diverse academic expertise, its historically strong relationship with the US National Ignition Facility, and notable assets, such as a world-leading laser?

• Public vs Private: Should the UK government try to derive more immediate value from its fusion data? For example, should the UK license some data to companies, to cover the costs of data processing and annotation? If so, should local startups pay less? Or would such efforts hurt the UK’s goal of developing a world-leading fusion sector?

_________________

This essay was originally posted on the Google DeepMind website and is a summary of a 20-page report that contains more details and examples.

Thank you to the following experts who let us interview them, reviewed the draft, and/or provided other support, as well as those who prefer to remain anonymous. All mistakes belong to the authors and no expert spoke to us on behalf of their organisation.

Jonathan Citrin, Brendan Tracey, Cristina Rea, Nathan Cummings, Andrea Murari, Jess Montgomery, George Holt, Alain Becoulet, Matteo Barbarino, Arthur Turrell, Adriano Agnello, David Dickinson, Steven Rose, Alessandro Pau, Kristina Fort, Charles Yang, Federico Felici, Tim Dodwell, Sam Vinko, Aidan Crilly, Lee Margetts, Tom Westgarth, Lorenzo Zanisi, Chris Packard, Justin Wark and Stanislas Pamela.

How can companies get their employees to use artificial intelligence when human intelligence remains sharp enough to know that this risks replacing jobs? How should education revise itself for the ever-revising technological world that students emerge into? And how to understand the love/hate relationship so many people have with AI?

Ethan Mollick—professor of management at the Wharton School of the University of Pennsylvania and bestselling author of Co-Intelligence: Living and Working with AI—is among the leading public intellectuals commenting on AI adoption, connecting the latest scholarship to real-world usage, including his own tinkering with each new model.

AI Policy Perspectives caught up with Ethan to hear his latest thinking on everything from agentic systems, to why scientific publication is broken, to how workers emotionally relate to AI colleagues. Too much chatter, he argues, considers this transformation at the broadest level. Too little digs into the practicalities of getting it right.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed for clarity]

Tom: In your 2024 book Co-Intelligence, you proposed four rules for human and AI collaborations, including that people should oversee and verify AI outputs. But doesn’t the value of AI agents come from people not overseeing and verifying everything?

Ethan: This is where policy matters a lot because these are choices now. In the “co-intelligence era,” you’d prompt the AI to do something in a chatbot, and it would give you an answer. You prompted again, and it’d give you another response. The human was in the loop. And not being in the loop was really dumb because it meant that you were just pasting in the AI’s answer, and then you’d get in trouble, as a lawyer with the judge, or whatever it was. Capabilities were weak, so human-in-the-loop mattered a lot.

But with agentic systems that could do hours of work on their own, now it’s a design choice. When do we want humans-in-the-loop? When is human verification valuable? When is human verification morally required? When is it legally required? What kind of interventions move the system forward? I feel there has been a complete lack of deep understanding about these topics.

Tom: You’ve said that, with agentic systems, management becomes a superpower. Can you explain this?

Ethan: Increasingly, systems look like mini-organizations as they get subagents they can delegate to. So the best way to organize is to give the AI a clear direction of where you want to go. And it turns out that this looks a lot like management. When do you want the AI to check in with you? How do you write a really clear brief? What checks are important? What tests do you want to run? What’s acceptable? What’s not acceptable? Those are management questions.

THE WORKPLACE

Tom: You co-wrote a study last year involving a field experiment at Procter & Gamble that showed AI usage enhanced employee performance. But there were other interesting findings besides that.

Ethan: The most interesting piece about it was that people liked working with the AI, and that it substituted for people emotionally. The second interesting piece was the “smoothing” of capabilities—so, technical people previously had technical ideas while business people had business ideas. But AI smooths out both. If technical people can do business work and business people do technical work, what that tells you is we have to redesign organizations.

Tom: The emotional side—that using AI improved people’s feelings about the work—was surprising to me; I wasn’t sure what to make of it.

Ethan: What to make of it? That views of AI are complicated. If people keep saying, “Yeah, AI is going to destroy all jobs, and may kill everyone on Earth…but might not”—and then, “Why is AI unpopular?!” Feels like not a hard question. People like AI when they use it themselves; they don’t like AI writ large. It’s not surprising to me that AI makes your job better because a lot of jobs suck! And if we do good design work with AI, it makes people’s lives better. If we just let it loose on the world, and tell management that the only option they have is automation, then we’re in big trouble.

Tom: Many knowledge workers seem to be using AI in secret right now, perhaps from fear of being exposed as less valuable.

Ethan: This is a leadership problem. The incentives have to be aligned properly. Currently, it’s, “I’m going to automate your jobs away” or “I’m not going to share with you any of the gains the company gets.” People are exquisitely tuned to rewards. So it’s about leaders articulating a vision of what the world looks like with AI for employees. “What should I expect to do? How are people rewarded for doing the right thing? If they automate 90 percent of my job, what happens to me?” Without those answers, everything else is secondary.

CHANGING ORGANIZATIONS & EDUCATION

Tom: You have a concept of “leadership, lab, and crowd.” Could you explain?

Ethan: There was a huge amount of R&D in the 1900s about how you organize work, and 40 percent of the American advantage in business came from management. In the last 30 years, a lot of that muscle has died. But experimentation is important, and leaders need to guide that. So, there are three things that organizations need to be successful with AI. First is “leadership”: a team that articulates a clear vision of the future, and is willing to experiment. Then there’s “the crowd,” the employees who might actually use AI. They need access to a frontier model, they need clear rules, they need reward systems. Then there is “the lab,” and this is the piece a lot of companies are missing. You need a dedicated team working on AI innovation. They can’t be just a technical team; this is not an IT department problem. If you don’t have that piece, you’re not building things for the future. And where does the crowd go when they have a good idea? “I came with a breakthrough idea that saves 90 percent of effort!” How does that diffuse in the organization? That’s where you need the lab.

Subscribe now

Tom: If AI transforms the workplace, that should change how we educate the next generation, right?

Ethan: The early workplace is under a lot of threat because the old apprenticeship model just broke. The idea was that there were tasks—especially in white-collar work—that were tedious and annoying for managers to do. But you could pay a relatively cheap person to do them, and that person would learn as a result of this, and receive mentorship. So we had this amazing machine for talent: we taught you, we evaluated you, and you got paid, and you were doing work we needed. A junior person’s goal was to produce good work that made managers happy, so that they got promoted. But now the junior person is worse than AI, so they’ll use AI to do their work. And the middle manager’s goal was to give work to a junior person who’s not great, and give them feedback so they get better, so that the middle manager has to do less work. And that broke because the middle manager would rather assign work to the AI.

Tom: But in terms of the educational system, what should change if workplaces no longer offer that apprenticeship role?

Ethan: Education is really screwed up right now, but it was screwed up for lots of reasons. It’ll be fine; we’ll figure this out. But it’s gonna take a bunch of years. It’s clear from early evidence that AI will be a tutor outside of class and inside class. It’ll do activities and give guidance. But schools are places where we can compel students to not use AI, and have them in a room, and evaluate them, and teach them the things that we want them to learn. As long as we think people need to be educated, this is the best space to do it in. So students are cheating in the meantime? They were cheating before! We can give them different tests; we could do in-class writing assignments. There can be a weird, backward-looking “Education won’t adjust!” view. How many death spirals does higher education need to be in per moment? There are the pieces to reconstruct a better form of education. It’s just a massive changeover.

BETTER SCIENCE & BETTER THINKING

Tom: What about academia? There’s been much talk about AI-written papers, and how they could overwhelm academic publishing. But could AI benefit the peer-review process, and help with the dissemination of academic findings?

Ethan: This is another area where more lifting is needed. It’s a shame that we are building AI co-scientists, but not thinking about the rest of the process that’s needed to actually make science happen. It’s one thing to have science produce more papers. We have no ability to absorb more papers. Every publication is overwhelmed. Our dissemination techniques were already bad, but now they’re really broken.

Tom: As a case in point, you submitted a paper around 2023, and wrote publicly about it then, making your term “the jagged frontier”—that AI capabilities advance in some areas but remain behind in others—highly influential. Yet the academic paper itself only just came out, three years later!

Ethan: One of the rejections we got early on was reviewers saying that they knew this already, and they cited a bunch of working papers—that cited the working paper we had submitted! This is not a unique story. Opening one part of the bottleneck without opening the others becomes a problem. But it takes longer to solve systemic problems of how science operates than to solve the problem of producing more papers.

Tom: Another concern in education and science is cognitive offloading, that people may surrender thinking to machines, and lose those skills. On the other hand, AI’s value comes from machines thinking for us. What are examples of bad offloading and good offloading?

Ethan: We offload all the time, right? But we also force people not to offload. You could offload all your mental math to calculators, but we force students to do some math by hand in an attempt to get them to learn stuff. And we can enforce those rules in school. In the world of work, we are not used to thinking about training, about what should be offloaded, and what shouldn’t be. We need to make decisions about this. So, Rolls-Royce still employs someone to paint stripes on a car by hand, and that’s an obvious pushback against deskilling in one area. But Ford doesn’t do the same thing. These are choices we get to make at an organizational level, depending on what we think is valuable.

ADAPTING TO CONSTANT CHANGE

Tom: A point you’ve made to young people about the AI future is that they’ll need to be adaptable. When educators talk about teaching adaptability, it sometimes boils down to encouraging “creativity” and “critical thinking.” Another view is that you’re more likely to be adaptable by developing deep domain knowledge. For you, what does learning adaptability mean?

Ethan: Adaptability requires both deep domain knowledge and wide knowledge: T-shaped behaviour is probably the way to go. I feel like it’s a throwaway line: “Well, we’ll all be adaptable!” If we could teach that, that’d be amazing. People are more adaptable than we think, so part of this is that people will figure stuff out. But we can’t just throw up our hands, and say, “Be adaptable!” You need to have deep enough knowledge to go into a field. You need to have broad enough knowledge so that, as one piece of knowledge becomes less useful, you’re moving to the next one. And we need to help people be adaptable by building systems that get them in place inside an organization and able to shift roles. I sometimes worry that adaptability is a catch-all for “Don’t worry! It’ll be fine!”

Tom: Another side is that not everybody will be equally adaptable. Could it be that the AI future favours certain circumstances and characteristics?

Ethan: A lot of these characteristics were already good characteristics to have. Does AI act as a multiplier of them? Does it disincentivize some people? We’re now past the edge of what we know. Ultimately, all of these questions come down to the same exact question, which is: How good does AI get, how fast? We need to articulate more clearly what we think that future looks like. Because you can’t say, “We’re going to build a superintelligent machine that’s better than all humans at every intellectual task—but let’s start thinking about adaptability!” Unless you mean, “Let’s adapt to UBI” [where everyone gets Universal Basic Income cash payments from the government]. And then, we should be spending a lot more time thinking about those issues. Not everyone in the labs believes this, and I find that the econ people believe it less. But you can’t have this message of, like, “All work will be obsolete!” and then have detailed, ticky-tacky conversations about what you should do in eighth grade. Because, by the time you enter the job market, there’s no jobs. So give me the pathway that you think is there, and that becomes the most important question to ask.

Tom: Are there other important questions I didn’t ask?

Ethan: We need to start thinking about getting into fields, and understanding what the changes are—we need to get detailed. That is where the research is missing. Another large-scale econ picture about AGI isn’t as useful. General-purpose technology affects everything, so we need policymaking for everything, from power generation to accountants, and when does the government say it’s okay to do this. There’s just this assumption that if we do the macro stuff, everything will work out. I’d rather see a lot more micro stuff: a thousand flowers everywhere, trying to come up with different approaches.

Share

By Simone Maria Parazzoli & Omer Bilgin

Amid the exhaustion of caring for a baby, new parents must deal with everything from bewildering sobs, to erratic feeding times, to the joys of changing a soiled newborn at 3 a.m. The last thing they need is paperwork.

But what if, when coming home from the maternity ward that first day, they could awaken a government AI voice assistant, tell it the happy news, and hear the following response? “Congratulations! What’s the baby called?” The app would then take care of all the dreary admin, coordinating across agencies, registering the child, and setting in motion the services that this tiny new citizen should enjoy.

That is one example of how a future “agentic state” could simplify, speed up, and improve citizens’ interactions with public services. To be clear, this does not yet exist. But projects like this one, envisioned by Ukrainian officials, are more than fantasy, with several countries avidly testing early versions of agentic AI systems.

While Ukraine works toward the baby example, Britain is piloting agent-based support to provide citizens more tailored help. Meanwhile, Singapore is developing governance frameworks for agentic AI, and governments from France to the United States are ensuring that their public data can be accessed by agents.

Agentic AI systems—capable of perceiving, reasoning, and acting with minimal human supervision—will transform what organizations can achieve. By combining the reasoning of large language models with retrieval, memory, and tool use, agentic AI can automate complex tasks. For governments, whose core work is high-volume, structured administrative processes, this could make services more efficient, timely, consistent, and fair, while lowering costs.

Consider a citizen looking to start a small business. An agentic system—instead of requiring the entrepreneur to individually navigate zoning boards, tax authorities, and regulations—could autonomously reconcile these requirements. The larger promise is a shift from just doing things right (optimizing for procedure-following) to doing the right things (pursuing outcomes that citizens truly want).

The Agentic State vision paper—supported by The World Bank and the Global Government Technology Centre Berlin—was the first effort to systematically map the opportunities of agentic AI adoption for governments. This was not an academic exercise: 21 leaders across 15 countries contributed, including ministers and chief technology officers preparing to lead this transition.

In this vision, AI agents are a means to manage complexity and scale, while humans develop strategy, exercise judgment, and hold accountability.

Several governments have integrated official chatbots into their government services, but most of these merely provide conversational guides to administrative procedures. A few pioneering countries are starting to move beyond that. Ukraine, for instance, is turning chatbots into agentic assistants. Specifically, its Diia.AI assistant can retrieve users’ data from connected registries, and generate official documents such as income certificates, while also providing certified information based on records such as taxation, land registries, and pensions.

The United Kingdom is also exploring agentic interactions via GOV.UK Chat (inspired by Diia.AI), including a pilot program to support job seekers that transforms a static digital portal into an active assistant, matching users’ skills with available opportunities.

Yet trends and optimism are not enough for success. The agentic state vision rests on key assumptions. What if they’re wrong?

This article presents a “red-teaming” exercise—a stress test of this vision—that identifies six core assumptions, along with scenarios that could emerge if they don’t hold true, and guardrails to avert such failures.

Agents can already perform rudimentary planning, tool use (e.g., searching the internet, using calculators, sending emails), and multistep task execution. Frontier labs are betting heavily on agents, making it plausible that systems capable of managing complex and large-scale administrative tasks will emerge soon.

Failure Scenario: The Technology Falters

Governments reorganize around agentic execution, but systems never become reliable enough for public administration. The demos look strong, but real cases fail on edge conditions, and require constant human correction. The agentic layer becomes only superficially competent with layers of human intervention underneath.

Guardrail: Start Cautiously

Governments should start with minimal deployments, and tightly scoped use cases to validate reliability, develop procedural rigor and organizational competence, and account for technological evolution rather than committing prematurely to large-scale redesigns.

The success of agentic systems demands that they’re able to interact seamlessly, conveying intent, carrying out tasks, and sharing data in an interoperable way. MCP (model context protocol) is emerging as the technological standard for connecting AI applications with external systems.

Failure Scenario: Standards Fail to Converge

Commercial interests diverge, establishing competing protocols, while government departments end up using AI systems that cannot communicate with one another. When a citizen’s request requires action from multiple agencies, the process breaks down.

Guardrail: Officials Insist on Shared Protocols

Governments should make interoperability a condition of adoption, participating in the cross-sectoral bodies and forums where these standards are being shaped, funding the development of shared agentic interfaces and other agent-specific standards, and mandating non-proprietary protocols in procurement. Standards rarely emerge by accident, but they may emerge when powerful governments treat them as a priority.

To adopt and employ agents effectively, organizations must rethink their processes, roles, and incentives. They need to flexibly change and dynamically adapt practices to keep pace with the changing technological landscape.

Failure Scenario: The Status Quo Prevents Change

Agentic AI adoption outpaces organizational change, with citizens and civil servants using agents in an uncoordinated manner long before official programs catch up. Local practices harden into path dependence before common standards emerge. The state becomes more productive at producing bureaucracy, not societally beneficial outcomes.

Guardrail: Redesign Processes Before Automating Them

Agents should only enter workflows that have been simplified, decomposed, and restructured to minimize approval layers and handovers. Governments must treat adoption as a continuous discovery process. They should invest in common evaluation templates, reusable components, and a cross-agency repository of lessons, so that what works in one place can travel before what does not work becomes entrenched.

Many companies are betting on an agentic future. Firms are experimenting with internal copilots and autonomous customer flows, while frontier AI companies advance core models, architectures, and capabilities, and cloud providers offer the compute needed to deploy agents at scale. This suggests that agents will become commonplace across business, consumer, and enterprise environments, allowing governments to build on tools, infrastructure, and behaviors already spreading across the economy. This assumption rests on projections, though evidence remains ambiguous.

Failure Scenario: Diffusion Is Slower Than Forecast

Governments invest as if an agent-saturated economy is imminent, but industry adoption remains narrow, experimental, or ends up costing more than it saves. Public investments don’t plug into widely used tools and practices, meaning that citizens find agentic interfaces in government before they’re normal elsewhere. The state ends up bearing political and institutional costs without the stabilizing effects of private-sector diffusion.

Guardrail: Lower Barriers to Private-Sector Agentic Usage

Governments can accelerate the development of an agentic AI ecosystem by investing in shared agentic infrastructure—such as standard ways to access public data, communicate across systems, and carry out authorized tasks and payments—that lower integration costs for firms, and reduce the risk of differing technological maturity across sectors.

Increasingly, citizens are interacting with and relying on AI tools, but many do not trust them. For governments to integrate AI agents into workflows and services, citizens must accept and support the roles that agentic systems can play, finding them sufficiently trustworthy, reliable, fair, convenient, and accountable.

Failure Scenario: The Public Rejects Automation

A single notable failure, or an accumulation of failures, turn the public against agentic systems, and convince many to opt-out. They judge automated decisions as opaque, illegitimate and untrustworthy, and suspect it worsens inequality, with privileged citizens able to employ highly capable personal agents to navigate bureaucracy better than those relying on basic tools. The government is forced to run two systems—agentic and human—and neither meets expectations.

Guardrail: Mandate Transparency

Governments must make agent integrations into government processes as legible as possible, furnishing explanations of decisions and publishing evaluation results for agentic fairness and performance, while detecting patterns of systemic bias or unequal benefit distribution based on citizens’ technological access.

For AI agents to act with functional autonomy within government processes, oversight frameworks must adapt, moving away from mandatory human reviews and approvals for everything (human-in-the-loop) to intermittent oversight (human-on-the-loop). This evolution increases speed and efficiency while reducing bottlenecks, with humans intervening only on edge cases. There is precedent for such adaptation: governments adapted regulation to cloud computing, e-identities, and AI-driven decision support systems.

Failure Scenario: Regulation Never Updates

Every agentic action requires human verification; every decision must be justified through mechanisms designed for old chains of accountability. Agents can draft, but cannot act. Compliance and procedural costs rise as institutions retrofit old controls onto new AI processes. The result is high bureaucracy and low autonomy: an agentic state in theory, a copilot state in practice.

Guardrail: Sandboxes to Test Oversight

Governments should establish controlled environments that allow policymakers, developers, and civil society to collaborate and gather empirical evidence on what forms of oversight are adequate and best fit different kinds of agentic deployments, reducing uncertainty before codifying rules at scale. They should explore this early, much as Singapore has done through its Model AI Governance Framework for Agentic AI.

Subscribe now

Soon, agentic government will be more than optimism and testing. A vanguard of countries will implement these tools. If those cases produce the kinds of benefits imagined, other countries will flock to join them.

But momentum is not inevitability. This project depends on assumptions—about progress, coordination, institutions, norms, and law—that demand scrutiny before governments rebuild themselves around these new technologies.

This red-teaming exercise of the agentic state concept is not to argue against the vision, but to make it more robust and resilient. The six possible failure scenarios are not mutually exclusive. Several could compound, and some may already be taking shape. For instance, reliability has been improving much more slowly than accuracy, providing ground for technology to falter (Scenario 1), and there are signals that the public might reject automation if economic gains and innovation speed are prioritized over fairness (Scenario 5).

Governments that are serious about improving the state with AI must attend to these risks in earnest now, while the architecture is still being laid. The opportunity is too precious to spurn.

Agentic AI could make public services considerably faster, fairer, and more responsive—more so than anything the traditional bureaucratic model has yet delivered. That prize is worth the discipline of preparing for what could go wrong.

Share

For further details on “The Agentic State,” check out the original vision paper

1. Identifying (and incentivising) AI agents

• What happened: A trio of law and philosophy professors considered how to identify who (or what) is responsible for AI agents’ actions in the world, and came up with a two-part proposal: that the disparate and evolving agents within a system should exist legally as a new form of corporation; and that each corporation should link to accountable humans.

• What’s interesting: The paper by Yonathan Arbel, Simon Goldstein, and Peter N. Salib starts with a thought experiment. It’s 2030, and your AI assistant suggests that it optimizes your slow WiFi connection. After you agree, it spawns a swarm of agents. Some are copies, while others are cheaper agents running on open-source models. Some start to interface with AI agents from other companies. Three months later, two FBI agents knock on your door and explain that your network has been piggybacking on a local defense contractor’s WiFi network.

• Before determining who is responsible and what the repercussions should be, there are more basic questions: Who are the AI actors in this story? How many are there?

• The economy will soon be filled with capable AI agents. To deter and respond to such harms, the authors argue that we need to be able to identify these agents, at two levels.

  • To prevent human misuse or negligence, we need ‘thin identity’. This would connect AI agents to the humans most able to control them, similar to how ‘know-your-customer’ rules tie banking transactions to humans.

  • Humans will be unable to monitor and control every AI decision, so we also need to be able to identify agents themselves, hold them accountable and incentivize them to behave well. To do so, we need ‘thick identity’ that can distinguish AI agents as stable, coherent entities, with persistent goals. This goal is pragmatic and does not require viewing AIs as conscious in any sense.

• Thickly identifying agents is harder and more novel, as AI agents need not be attached to a physical body. Multiple agents can also work together on a single task. Any single agent can be copied, spun up, spun down, or be continually updated.

• To address such challenges, the authors propose creating algorithmic corporations, or ‘A-corps’. These would have two key elements:

  • Legal personhood: Like a traditional corporation, an A-corp would be a single legal entity that persists over time. It could hold property, make contracts, and be sued. But it would be run by a collection of AI agents. As such, the proposal runs contrary to scholars who have argued against granting legal personhood to AI agents, or called for bans on algorithms running companies because of concerns about crime and companies using them to avoid liability.

  • Computationally-secure governance: Each A-corp would have a unique digital certificate and a secure private key to authorise transactions. The humans that own each A-corp could grant the key to an AI ‘manager’ agent who in turn could grant more limited permissions to sub-agents within the A-corp, or to other A-corps, such as permissions to spend up to $100 or to read a batch of emails.

• The proposal addresses thin identity by reducing the vast number of AI agents down to a smaller number of A-corps, whose actions are traceable back to their human owners. As with limited liability companies (LLCs), the human owners would not be responsible for all harm their A-corps cause, but could lose all funds they invest and possibly face further liability, for example in cases of fraud or negligence.

• The proposal addresses thick identity via its ‘resource constraint thesis’. All AI agents need resources, like money and compute. A-corps provide AIs with a way to access these resources and an incentive to manage them well. For example, A-corps that tightly monitor and audit their sub-agents’ performance would get more resources, while A-corps that allow fraud or waste will lose resources. This encourages A-corps to self-organise, into stable, coherent, multi-agent systems.

• The authors argue that A-corps could also address alignment concerns, for example by reducing the incentive for an AI agent to exfiltrate its own weights, because that new AI instance would lose access to resources and permissions from the A-corp.

• To make it happen, the authors call for a public registry of A-corps. This would list each A-corp’s human owners, the certificates to authenticate it against, as well as (potentially) the differing permissions enjoyed by its agents. Ultimately, the authors argue that A-corps should become mandatory for any AI agent taking “economically significant actions”, and to guard against criminals using AI agents anonymously.

• The authors respond to some expected pushback. They do not see A-corps as anthropomorphising AI because the proposal does not require anybody to view agents as having deeper desires or wants. They also think A-corps can prevent the risk that AI agents might slowly build up resources before deploying them for harm, by encouraging inter-agent trade that penalises rogue behavior. Could A-corps disempower humans? The authors argue that they provide a pathway to tax and redistribution, and enable humans to better steer agents, for example by designating the parts of the economy that A-corps are permitted to operate in.

2. When AI builds AI

• What happened: The Centre for Security and Emerging Technology, CSET, released a report on the prospects for AI improving itself, known as automated R&D or recursive self-improvement, based on an expert workshop in July 2025.

• What’s interesting: In 1964, the computer scientist I.J. Good wrote about the possibility of an “intelligence explosion” that would leave “the intelligence of man.…far behind”. Researchers have also long automated aspects of writing code and AI model design.

• However, the speed of AI coding advances suggests that something qualitatively different may soon occur. This makes two questions salient: 1. Could AI automate the entire AI R&D process? 2. Will this R&D automation extend across all scientific disciplines? The CSET report focuses on the first question.

• CSET defines AI R&D by distinguishing between research scientists, who generate hypotheses, design experiments and interpret results; and research engineers, who write code, fix bugs and generate data. They also note the inputs that AI R&D relies on, such as raising funds and acquiring compute.

• They sketch out four overlapping scenarios for how AI R&D may play out:

  • 1. Explosion: AI systems automate a growing share of AI R&D. Initially, this leads to modest productivity gains, but as the length and complexity of tasks that AI performs grows, productivity soars. AI systems become far more capable than humans, whose involvement in AI R&D falls to zero.

  • 2. Fizzle: The share of R&D tasks done by AI rises, but rather than leading to compounding improvements, capabilities start to plateau.

  • 3. Amdahl’s Law: AI automates certain activities, like writing code and running experiments, but not others, like research strategy.

  • 4. The expanding pie: As AI automation grows, humans realise that new ideas and breakthroughs are needed that AI systems cannot yet provide.

• The experts in CSET’s workshop held widely diverging views on which scenario was most likely. Most importantly, new empirical data is unlikely to resolve these conflicts, because participants may view the same data as confirming their own assumptions.

  • For example, an AI system’s inability to reliably use a keyboard or mouse may look like a bottleneck to one expert, but a source of explosive growth to another—if they expect this human-focussed tooling to get adapted for the AI era. Similarly, different experts may view AI automating a growing share of R&D tasks as progress towards a fast takeoff, or as low-hanging fruit being picked off, accelerating progress only as far as the upcoming wall.

• These differing views are also visible in more recent commentary on the topic.

  • The prominent AI researcher and writer Nathan Lambert recently cited Paul Allen’s concept of a ‘complexity brake’ to argue that as we understand intelligence better, further progress becomes exponentially harder. In addition to incurring financial costs, Lambert argued that running suites of AI agents won’t necessarily lead to exponential progress, because those agents will perform best on narrow, verifiable tasks, will be hard to manage in large numbers, and will sample from similar parts of the distribution of AI research ideas, inhibiting more novel breakthroughs.

  • Conversely, Ajeya Cotra at METR, the Model Evaluation and Threat Research organisation, recently wrote about how she “underestimated AI capabilities (again)”. She argued that AIs may, counterintuitively, find it easier to decompose longer projects into sub-components that multiple agents can run in parallel, than for shorter tasks. AIs will also produce good documentation for their fellow AIs, which could accelerate progress.

• If faster automation and progress does occur, the CSET authors see two main risks: Less time to prepare for safety risks from AI, and lower human understanding of AI systems. To address these risks, their recommendations have a strong focus on improving access to evidence, including:

  • New evaluations of AI R&D, including for ‘messy’ tasks such as research strategy, which lack clear specifications and success criteria and take place in a dynamic environment with various real-world interactions.

  • New approaches to evaluation to better distinguish ‘degrees of accomplishment’ from a simple success/failure binary.

  • Better insights into how automated R&D is progressing within AI labs, such as data on how funding is allocated and qualitative impressions of progress from leading AI researchers and engineers.

3. Planes and global warming

• What happened: A team of researchers, including from Google and American Airlines, published results from their latest experiment to use AI to reduce condensation trails from planes—a key contributor to global warming.

• What’s interesting: When pilots fly, particles from the plane’s exhaust can mix with low-pressure air to form contrails—white, artificial clouds, made up of ice crystals. These contrails are a net contributor to global warming, because they trap heat that would otherwise escape. Debates continue over exactly how much they contribute, but one estimate suggests that they contribute a lot, causing around 2% of ‘radiative forcing’, which measures how different factors, like CO², heat or cool the planet.

• As the environmental writer Hannah Ritchie explains, more important than the absolute figure is the fact that contrails offer a rare opportunity to reduce global warming almost immediately, at relatively low cost. This is because a small share of flights cause most of the warming-inducing contrails—generally those that fly through parts of the atmosphere that are both very cold and very humid. If planes take short detours to avoid these patches of air, contrails (and warming) should drop.

• A few years ago, Google researchers partnered with American Airlines on a proof of concept. Using satellite imagery and AI, they were able to predict where contrails would emerge and guide planes to avoid them, reducing contrails by >50%, across 70 test flights.

• In the latest study, they expanded the experiment to 2,400 American Airlines flights from the US to Europe. They placed ~50% of planes in a treatment group, where flight dispatchers were given two choices: a standard flight plan and an alternative contrail-avoidance one. Their decision for which to recommend was voluntary.

• For flights in this intervention group, contrails fell by 12% compared to a control group with no contrail-avoidance plan. Importantly, the contrail-avoidance routes also did not lead to a significant increase in fuel use. At first glance, these results seem positive, but modest. Digging into the results highlights the challenge of getting useful AI deployed at scale.

• In particular, dispatchers who received contrail avoidance plans only recommended them to pilots 15% of the time. Even then, the avoidance plan was only successfully flown in 60% of flights. For planes that did successfully follow the avoidance plan, contrails fell by more than 60%, a much larger reduction. So the tech worked, but was often not used.

• Why? Dispatchers are busy and must often deal with other priorities, like bad weather and turbulence. To avoid contrails, planes also need to climb and descend mid-flight. This is safe, but creates more work for pilots and air traffic controllers. As it was voluntary, the incentive to change to a contrail-avoidance plan was weak.

• The way that the dispatchers received the information also meant that they didn’t fully understand why the suggested up and down changes were necessary. Happily, the authors feel that most of these obstacles are addressable, with a combination of a better user interface, some automation, and more incentives.

• In addition to its immediate usefulness, the study is a rare real-world attempt to quantify the benefits of AI to tackling global warming. At the moment, the AI and climate change policy discussion is often negative and focuses on the emissions that may result from building and operating data centres (and other devices) to train and run AI models. This is important, but there are reasons to think that these emissions will be relatively low, or at least lower than many assume. In contrast, AI could potentially reduce emissions and warming by far larger amounts, for example by accelerating research on solar and fusion power, or making buildings and energy grids more efficient. But these benefits are typically more speculative, harder to quantify, or in the case of contrails, more contingent on human behaviour.

• This experiment demonstrates that the benefits of AI to tackling global warming are real, but also points to the interventions that will be needed to push them to their full potential. The study is also timely, given that governments are focussing on contrail avoidance and some policy action may be required, for example to help standardise and mandate contrail prediction software or to generate high-resolution humidity data.

Share

From childhood upwards, we play games as a safe (and strangely joyful) way to battle, strategize, even lose without it coming to fisticuffs. Artificial intelligence grew up playing games too, with developers using the structured rules, scoring systems, and win/loss outcomes to train machines to learn, to improve, even to beat us.

In chess, bots have been bettering humans for years now. Yet our “loser” species still gathers at sunny park tables, in dank school gyms, and online in droves, all in hopes of crying, “Checkmate!” The resilience of chess is commonly cited as evidence that—even if AI surpasses us in various pursuits—humans won’t just give up.

However, there’s more to say about the intersection of technology and chess, in particular how the game has evolved with technology, including AI. Thankfully, the broadcaster and writer David Edmonds—co-author of Bobby Fischer Goes to War (2004) and editor of the essay collection AI Morality (2024)—has spent decades observing this, both as a spectator and behind the board himself.

—Tom Rachman, AI Policy Perspectives

By DAVID EDMONDS

Among thousands of tournament games cited in the Batsford book of chess openings, tucked into the top right-hand column of Page 235, is an example of how white should not play.

Explaining the Closed Sicilian Defense opening, the authors (former world champion Garry Kasparov and the British grandmaster and chess columnist Raymond Keene) spotlight a game in which black is already ahead as early as move 11. Indeed, the player with the white pieces ended up losing. I remember because that player was me.

That is my humiliating contribution to chess theory: what not to do. The book was published in 1982, and I’ve barely picked up a pawn in anger in the intervening four decades. But I still follow the chess world, and if there’s a tournament in London, I’ll go to watch, spending hours absorbed in the intricacies of the 64 squares.

As the digital revolution and AI juggernaut move through our lives, we may wonder whether there will still be domains in which humans can continue to find enjoyment and meaning. Chess offers a hopeful case study.

Chess and AI have had a long relationship. The great forefather of artificial intelligence Alan Turing wrote the first chess algorithm in 1948. The following year, another seminal figure, Claude Shannon, distinguished two ways that a computer could play chess: by brute force, calculating every possible move; or by selective search, like a human.

Chess also proved a favourite way to evaluate AI advancement, both because many key innovators were keen players but also because the game’s mathematical structure and its win/loss conditions created benchmarks for comparing machine progress to human performance.

A longstanding goal—seemingly impossible at first—was to outclass the best humans in a game that has near-infinite permutations. Defeating humans at chess became the programmers’ ultimate challenge, like runners seeking to break the four-minute mile or climbers reaching the summit of Mount Everest, both of which proved easier. Finally, in 1997, IBM’s Deep Blue vanquished Kasparov, the then-reigning world champion. A dejected Kasparov insinuated that there had been human intervention.

For a while, chess players comforted themselves with the thought that a hybrid combination of human and machine could outwit machine alone. That period has long passed. Today’s best player, Magnus Carlsen, would be trounced were he to compete in a series of games with my mobile phone.

In 2017, DeepMind’s AlphaZero took machine chess to the next level. While Deep Blue had relied on brute strength with some input from strong humans, AlphaZero was simply programmed with the basic rules, and then trained itself through reinforcement learning. In its learning phase, it played tens of millions of games against itself in just a few hours, then crushed the chess engine Stockfish. (Stockfish adapted its methods accordingly, and is now the leading chess engine.)

Subscribe now

World chess champions of the past exuded an aura. Their talents seemed mysterious, supernatural. In part, that’s because few people, then and now, can comprehend the depth of thought that elite players achieve at the board. When it comes to music, we may never compose like Mahler, but we can appreciate Mahler’s symphonies. By contrast, we can neither play like Magnus Carlsen nor fully appreciate his games. It’s for this reason that the Armenian-born grandmaster, Lev Aronian, once confessed to me that being one of the world’s top players was desperately lonely.

Carlsen has achieved the highest rating of any human in history. And, no surprise, he strikes a confident pose. Yet his strut no longer carries complete conviction. To spectators armed with portable chess engines, the chess gods have been humbled.

Even so, chess has not dwindled in popularity. On the contrary, more people are playing it than ever. The game received a boost during Covid, when we all hunkered down in our homes, connected by the Internet. Another boost came from the hit Netflix drama, The Queen’s Gambit. Meanwhile, a younger generation of telegenic chess masters has gained avid YouTube followings, turning commentary and stunts into short-clip entertainment.

Here are 11 ways that technology has changed chess. The 11th is the most interesting:

1. Opening Preparation. The systematic study of chess openings goes back a couple of centuries or more. Sequences of opening moves were mapped out—as in that 1982 book that included my embarrassing loss. But chess engines allow for a depth of opening analysis that was inconceivable in 1982. This means that 25 moves may pass before grandmasters find themselves in unfamiliar territory nowadays. Some openings have also been resurrected because engines have shown the positions to be more survivable than previously recognized.

2. Opponent Preparation. Even in amateur tournaments, players routinely prepare for opponents in an individually tailored way. This is made possible because the games of each opponent are available online.

3. Connectivity. Fancy a game? There are endless online adversaries willing to take you on, day and night, from India to Iceland, Cape Town to Chicago.

4. No More Correspondence Chess. There was once a thriving chess scene in which games were played remotely over a long time period—months, sometimes years—with moves typically sent by post. How quaint.

5. No More Adjournments. Historically, world championship games would sometimes stop after five hours to resume later. That can’t happen anymore, since players might simply identify the optimal continuation with the help of an engine. Time limits now ensure games finish within a single session.

6. Shorter Games. Many in the online chess audience don’t have patience for lengthy games. For them, quicker time controls—Rapid (less than an hour); Blitz (3-5 minutes); or Bullet (under 3 minutes)—are more thrilling.

7. Different Formats. Now that computers have shown with such depth which opening sequences are optimal, the early part of a game has been transformed into a feat of memory rather than creativity. As a result, Fischer Random (advocated early on by the ex-American world champion Bobby Fischer) has become increasingly popular. In Fischer Random, the starting position of the major pieces behind the pawns is randomized, making opening homework effectively impossible. It’s sometimes called Freestyle Chess, or Chess960 because there are 960 possible ways for the pieces to be shuffled.

8. Job Generation. With a potential global audience, some players can now earn a decent living live-streaming their games, or offering online training.

9. Roasting of Champions. This is an irksome development. Since chess engines assign an instant numerical evaluation of the position after each move (e.g. +1 means white is better by roughly one pawn), any patzer can see when a grandmaster has blundered, and is free to abuse them in online comments.

10. Cheating. There have always been cheating accusations in chess. In 1978, the Soviet dissident Viktor Korchnoi claimed that the aides of his opponent, Anatoly Karpov, were using the flavour of the yogurt handed to Karpov to secretly convey messages. More recently, suspicion (tongue-in-cheek, but taken seriously by online trolls) has been raised of illicit advice being transmitted via vibrating sex toys. In elite tournaments, grandmasters are now searched before they enter the playing arena, even accompanied to the toilet. Spectators, meanwhile, are prohibited from carrying phones, to prevent them signalling the best continuation. But in online games, cheating is almost impossible to prevent. Platforms try to detect cheats by comparing human moves to the recommendations of top engines. But if savvy cheaters consult an engine just once or twice in a game, they may win without being detected.

And so to the 11th effect on chess: the expansion of human imagination.

Share

In the last few years, there has been a slight but detectable shift in grandmaster play, as humans learn from machines, both through gameplay against bots and by using machine insights to prepare for human competition.

People who don’t play chess may imagine that what distinguishes strong from weak players is calculating power. And it’s true that top grandmasters can analyse many moves in advance. But their edge is tougher to articulate. It involves superior pattern recognition, with an intuitive sense for where their pieces should be placed and how a position should advance. Likewise, Mozart felt how a composition ought to develop; his instincts about building tension and creating contrasts were the product in part of having internalized countless musical patterns.

For chess players, some moves seem ugly. It might feel wrong to shunt a knight to the edge of the board, to break up a pawn structure, or to expose the king. But computers don’t feel anything. In chess, they care about patterns and the interplay between pieces only to the extent that they’re relevant to the ultimate objective: victory.

However, bots don’t necessarily play robotically. They produce moves that astonish and inspire human players, even make them laugh with surprise. One famous case of AI invention across the board came in another game, Go, when the AlphaGo program was facing a top human player, and produced a move that caused professionals to gasp. “Move 37” is still cited with awe, as something a person would never have done, but that worked sublimely.

Likewise, chess engines regularly expand the imagination of human chess players, pushing beyond the habitual “correct” move they’ve seen many times before or have learned from books of chess theory. AI has even dabbled in the art form of creating beautiful chess puzzles. And empirical studies indicate that leading players may pick up new ideas and strategies from machines.

Machines, in other words, can make humans more resourceful and inventive, breaking down rigid modes of thinking. The implausible becomes plausible. The readily dismissed becomes the carefully considered. This evolution of chess illustrates a broader idea in the development of AI that may prove immensely valuable in science and elsewhere in human endeavour: that how AIs think may help human experts learn new ideas themselves.

In his book The Silicon Road to Chess Improvement, the grandmaster Matthew Sadler argues that chess engines can improve every player, and he documents some of the counterintuitive patterns that humans could pick up from AI. By way of illustration, during a top tournament this January, the Indian grandmaster Arjun Erigaisi (playing against Vladimir Fedoseev of Russia) advanced his pawns in a way that looked reckless. In fact, computer analysis indicated he was still ahead after 28 moves. However, he blundered and lost. The danger of learning from a computer is that success may require you to proceed with computer-level accuracy.

Subscribe now

As AI undertakes more activities formerly done only by people, it’s worth asking why human chess persists—and will likely continue to do so.

A Canadian philosopher, Bernard Suits, pointed out in his 1978 book The Grasshopper: Games, Life and Utopia that what defines “games” is that they involve the voluntary attempt to overcome unnecessary obstacles. Therein lies a defence against AI encroachment. In a market economy, companies aim to remove or overcome obstacles in the pursuit of profit. In games, obstacles have been deliberately inserted as an indispensable feature. What we enjoy in playing chess is testing our cognitive abilities. What we enjoy in watching chess is two humans pitting their wits against each other in a socially constructed activity where difficulty enhances enjoyment and satisfaction.

There’s also a narrative element to caring about games. The contest—whether intellectual or physical—is absorbing precisely because it involves conscious creatures. In elite chess, there’s the backstory: the players’ rise, their subsequent ups and downs, their history with specific opponents.

But watch an engine-against-engine tournament like TCEC (the Top Chess Engine Championship), and you’ll soon fall asleep. Computers aren’t competing after a divorce, or an illness, or the loss of a parent. Humans have character traits that spill onto the board, such as aggression (or passivity); patience (or impatience); equanimity (or volatility); and resilience (or fragility). Winning and losing have emotional resonance for a human—but not for AlphaZero.

It’s these qualities that guard against AI advance. AI might gobble up some of our jobs; even human-authored articles like this one may become rarer. But AI won’t take our chess.

By Conor Griffin, Joslyn Barnhart & Owen Larter

In 1971, the marine archeologist Honor Frost heard news of wood protruding from the sea floor. Off the western coast of Sicily, she and her team donned scuba gear, and splashed into the shallow coastal waters. Wind whipped the surface, causing the underwater sand to swirl confusingly. But even in murk, they couldn’t miss it.

“A large timber (such as I had never seen before) emerged,” she recalled, “like the head of a primeval animal crowned with weed; the presence of a buried wreck was evident.”

They excavated for months, gradually exposing the remains of a Carthaginian warship sunk more than 2,000 years before. Somehow, saltwater hadn’t eaten away letters painted on the wreckage, revealing a humble system that links antiquity to tomorrow.

Those shipwrights’ marks told workers in ancient Carthage how to put together a vessel—akin to flat-pack furniture from IKEA, with numbered and lettered pieces. They were among the earliest surviving examples of a simple but potent tool in human progress: the technological standard.

Many of history’s grand projects have benefited from standards, from Egypt’s pyramids, to Europe’s cathedrals, to Gutenberg’s press, to everyone’s Internet. You can even thank standards for the development of beer.

Underpinning technological standards is a plain truth: people thrive when able to cooperate, not when we must keep negotiating the basics, whether it’s a matter of nuclear safety, or a phone-charger cord, or who goes next at the intersection. So, the goal is order. And the benefits are that innovators can proceed without excessive obstacles, while everyone else is treated fairly and kept safe.

But what should standards mean for artificial intelligence? In particular, how can they guide the most advanced large language models and AI agents that could transform society?

Venture around the AI frontier today, and you’ll find ambition to accelerate AI for economic growth and transformative science alongside concern that AI could clatter into what humans cherish most. What few dispute is this: standards will help set the path.

Standards have critics too. One criticism is that companies dominate the process, prioritizing their own products or miming security without truly ensuring it. Besides this, standards can stir geopolitical tensions, as when Western countries fear China’s influence in laying the path to tomorrow, while smaller nations worry that standards may be set without considering them at all.

So, as we’ll keep insisting, standards matter! Only, there’s a problem.

For some, the mere mention of “standards” prompts slumber. And even those determined to stay awake may find themselves puzzled, gazing at the alphabet soup of standards organizations and committee meetings.

Part of the problem is that standards are often technical, such as efforts to standardize the protocols needed for AI agents to communicate. Or they are bureaucratic, negotiated out of public view, with dense, jargon-filled documents that are often behind a paywall.

Complicating matters even more, artificial intelligence is a general-purpose technology less akin to a hammer than to electricity. This will lead to standards (plus standards initiatives that don’t take) on everything from AI agents, to AI cybersecurity, to AI content provenance, to product-specific standards for AI-as-a-medical-device, and so on. And that’s not even mentioning standards for future AI applications that nobody has yet considered.

In short, standards will be immense. Standards will be tough to comprehend. But standards will also be vastly important.

Subscribe now

CAN SOMEONE DEFINE STANDARDS, PLEASE?

Standards are a diabolical blend: intricate, vague, and slippery.

They’re the invisible infrastructure of the modern world, according to Laurie Locascio, head of the American National Standards Institute, ANSI. She recounts hearing an official at Boeing describe the airplane itself as “thousands of standards taking flight.” Standards are “the things you don’t think about,” Locascio says. “But oh, my God, you’re so glad they’re there.”

Expressed broadly, a standard defines the how of tech, whether it’s the default product specs that allow compatibility among manufacturers, or the formally endorsed risk management processes that encourage industry to act responsibly.

As technology evolves, standards do too. A leading scholar, Ken Krechmer, once noted that standards initially defined how physical objects fit together (as with those markings on the Carthaginian longship). Over time, standards came to define the relationship between technological objects (as with internet protocols).

A standard also builds on other forms of guidance, such as norms, principles and industry best practices. Unlike norms, standards should be explicit. Unlike aspirational principles, a standard should be specific enough for performance against it to be judged. Unlike early best practices, a standard should have clear buy-in.

Developing a standard can be a protracted endeavor. In some cases, it might start in a researcher’s notebook, evolving into a product or a practice that gains traction in the marketplace. At other times, institutions set standards via years of deliberations and meticulous documents. Most often, it’s a messy back-and-forth between standards that emerge in practice and on paper. This makes standards a source of tension among companies, governments, and independent advocates, all trying to set the technological future they consider best.

Some presume that laws should be how we define permitted behavior. But high-quality legislation can struggle to keep up with the frantic speed of AI progress. And when laws are passed, they may rely on standards for implementation, as with the EU AI Act.

So how to persuade everyone to care when encountering standards, rather than just to snore or sob? How to get policy leaders to ponder the entirety of frontier-AI standards and align on where action is most needed?

Our answer is storytelling: to pluck forth tales about past standards, illustrating what this technological shaping can achieve, where it goes wrong, and how we might help cultivate standards wise enough to manage the breadth and speed of AI.

Our first stop? A battlefield of centuries ago.

A ‘STANDARD’ HISTORY

Horrors encircled the boy soldier: swords clanging under the rain, excruciating howls of the wounded, the fast-approaching bellows of men hurtling across the bog to murder him. In wet turf, he shivered from knees to chattering teeth, his mouth parched, his gaze searching for any escape.

Up there?

On a hill, a flag rippled, where his legion had marked its territory. The Old French word for that banner was “estandart”: a sign of firmness and stability, a marker of where to go next, a statement of order amid chaos. To such banners, we owe the word “standard.”

More than a few historical standards emerged from war, where disorder could mean one’s brethren murdered, while coordination could mean an empire.

~225 BCE to the dawn of mass production

China’s first emperor, Qin Shi Huang, led an extensive standardization process that included mass-produced crossbow parts. If parts of a soldier’s weapon broke in the midst of battle, he could grab spares, and swap them in.

When Ancient Rome fought for primacy in the Mediterranean, its forces copied Carthaginian ship designs, eventually triumphing with standardized ships of their own, along with standardized tools and camp layouts, all of which simplified maintenance and large-scale coordination.

Another advance in ancient times came from standardized measurements for length, volume, and weight. Previously, cultures often had distinct units; you can imagine the squabbling. But as trade expanded, standards prevailed, making cross-cultural exchange possible. In ancient Egypt, one of the earliest and most influential standards was the cubit, a unit of length used to coordinate the building of the pyramids.

In Europe’s medieval period, guilds established standards for quality control, so that weavers might set the necessary thread count or width of cloth, preventing low-quality products from undermining a craft’s reputation. Guilds also played a protectionist role, with licensing standards imposing strict controls on who could become a member.

The consumer might benefit from standards too, with measures such as England’s Assize of Bread and Ale of 1266 establishing the acceptable quality, quantity, and price of baked goods and beer. Later, Gutenberg’s standardized press led to mass-produced books that spread ideas across the Continent.

However, technological standards reached new heights of utility during the Industrial Revolution, which set the foundations for many of today’s technologies.

1760-1840: The First Industrial Revolution — The rise of engineers

As ancient Chinese and Carthaginians had discovered long before, the Industrial Revolution’s manufacturers found that interchangeable parts offered transformative efficiency. Before, if you hand-built a musket, or a clock, or a steam engine, you might craft each screw, each bolt, each gear to fit. By contrast, interchangeability allowed for mass production, cutting costs, reducing errors, and establishing the basis for modern industry.

Screw threads are a classic example. Before standards, manufacturers used various designs, making repairs nightmarish. If you had one company’s bolt but another company’s nut, you were out of luck. In the 1800s, engineers built the first practical screw-cutting machines, allowing factories to produce uniform threads and a consistent system of measurement. The British Standard Whitworth became the first such standard in the world.

Screw-thread standards may not quicken your pulse. But their effects might. They played a part in British imperial ambitions, contributing to the expansion and maintenance of the British Empire through military mobilization.

1870-1914: The Second Industrial Revolution — National coordination & path dependence

The emergence of electricity, steel, and advanced machinery led to vast interconnected systems, including power grids, railways, and telegraph networks. Coordination wasn’t merely better; it was essential. To coordinate across a nation—and eventually across borders—the ambitious country needed technology standards. Two famed cases illustrate this, one successful, one bungled.

The success regards the quintessential technology of the times: railroads. By the 1870s, the U.S. rail system was a mess, with more than 20 different track gauges. When a train reached a section built to a different track-gauge width, everything—each passenger, piece of luggage, every single crate—had to be unloaded, and transferred to a new train.

By the 1880s, matters had become slightly less chaotic, with either a southern gauge or northern “standard” gauge used across most of the country. Yet this still divided national transport until, in 1886, rail companies pulled off a remarkable feat. Over two days, they converted 13,000 miles (that’s 21,000 kilometers) of southern U.S. track to the northern standard, integrating the national transportation network. When trains rolled out on June 2, 1886, they were able to travel seamlessly across the United States for the first time in history.

A second case illustrates bungled standards. In the 1880s, the rival inventors Thomas Edison and Nikola Tesla found themselves at the center of “the War of the Currents.” Edison championed direct current (DC), a one-directional flow of electricity that had been the early U.S. standard. Tesla, backed by entrepreneur industrialist George Westinghouse, advocated alternating current (AC), or electricity that reverses direction many times per second, and can be stepped up or down in voltage with a transformer.

From an engineering standpoint, AC had a decisive advantage: it could transmit power over long distances cheaply and efficiently, while DC could not. AC eventually won out. But by the time it had emerged as the superior solution, the world had already built electrical systems without any coordinated technical governance. As there was no international authority harmonizing electrical standards, the United States went with 120 volts at 60 hertz (a legacy of Edison’s early low-voltage DC networks). Much of the rest of the world adopted 230 volts at 50 hertz.

Once wires had been laid and appliances built, the world was locked into two incompatible systems. To this day, we’re burning out hair dryers bought in America but used in Paris, or realizing too late that we don’t have the right plug for our laptops. If it’s irksome for the average user, it’s more burdensome for manufacturers, obliging them to build different versions for different countries.

Subscribe now

Another classic tale of path dependence is under our fingertips as we type: the QWERTY keyboard. Why does the top row spell QWERTYUIOP? One account goes like this: In the mid-to-late 1800s, early typewriters jammed each time the user struck neighboring keys in rapid succession. So, designers produced a layout that deliberately distanced many common letter pairs. Remington purchased this QWERTY design, and began mass-producing typewriters.

Before long, typing schools had trained the future secretarial workforce on QWERTY, while firms wanting fleet-fingered staff had to buy those machines. Manufacturers subsequently resolved the key-jamming problem and other keyboards tried to depose QWERTY, some claiming to quicken typing by as much as 40%. But QWERTY had become a de facto standard. (Scholars continue to debate the specifics, with some arguing that QWERTY works just fine.)

In any case, the indisputable lesson is to watch for path dependence. The standards we establish for frontier AI today—or fail to establish—may determine future efficiency or future failure.

1914-1964: Standard Development Organizations & Digital Technology

In 1918, engineering societies joined with the U.S. government to establish a standards committee that developed into ANSI, the American National Standards Institute. Today, ANSI provides the “stamp of approval” for many U.S. standards organizations, including those working on AI. In subsequent decades, standardization went global. While the United Nations was founded as a governmental venue for diplomacy, the International Organization for Standardization, ISO, emerged as a non-governmental body for peaceful technical coordination across borders. Bit by bit, additional standards bodies formed, cooking up the alphabet soup of acronyms—each a different org, subgroup, or committee—that lies before us today.

Soon, another transformation for standards was taking shape in the form of digital tech. Back then, computers filled entire rooms of universities, and each manufacturer built hardware and software within its own format. Computers could not run programs written for other systems, and accessories like printers or storage devices were incompatible.

A turning point came in 1964, with IBM’s System/360. Software on one model could more easily run on another; accessories like printers worked across IBM models. You could upgrade and expand computer systems with relative ease.

1969-today: Talking Machines

The year that hippies grooved at Woodstock and astronauts walked on the Moon, the U.S. Defense Department was testing a project that must have seemed minor by comparison: connecting research institutions and government agencies. Yet the Advanced Research Projects Agency Network, ARPANET, which sent its first message in 1969, was the precursor to our transformed world.

Before ARPANET, moving information from one computer to another was a struggle, with researchers forced to carry magnetic tapes or punched cards between locations, while those working far apart had to rely on snail-mail.

To convey information between independent systems, ARPANET adopted packet switching, breaking data into small units that could travel independently and reassemble at their destination. Extending this, Robert Kahn and Vint Cerf began designing a universal communication framework in 1973 for different types of networks to connect. Their collaboration ultimately produced TCP/IP, the Transmission Control Protocol and Internet Protocol that underpins today’s online communication.

A key effect of the TCP/IP standard was decentralization: no single authority could control the flow of data, and any network that adhered to the protocol could connect without permission from central authorities.

In 1989, a British scientist at CERN, Tim Berners-Lee, proposed another transformation that developed into a project called “WorldWideWeb,” which envisioned a global network of documents accessible through software, operating on open standards that nobody could lock it into a proprietary system. Two standards organizations, the Internet Engineering Task Force and the World Wide Web Consortium, helped to formalize the vision, crafting standards for structuring content (HTML), transferring data (HTTP), identifying resources (URI), and more.

But while standards help spread technology, this diffusion can also lead to greater harm. The expansion of railroads led to more wrecks, forcing uptake of safety standards for signaling, brakes and more. When electricity was first installed in the White House in the late 19th century, President Benjamin Harrison and his wife Caroline were so afraid of shocks that they refused to turn the lights off. Such fears—often well justified—led to the standardization of building and electrical codes. When it came to digital technology, the risks extended beyond immediate physical safety into areas like data theft. This demanded standards such as SSL/TLS to provide security for data sent over computer networks.

A recurrent challenge with frontier tech is that experts struggle to predict how exactly it will affect society. But once it is widely used, it can be sticky and hard to change. The effects of powerful technologies can also be subtle, indirect and slow-burning, for example if they change how we access and consume information. In the digital era, this has shifted technological standards from periodic safety checks of products towards ongoing processes that organizations can use to identify, evaluate and mitigate a growing suite of risks.

By way of example, the U.S. government’s National Institute of Standards and Technology, NIST, introduced the voluntary AI Risk Management Framework in 2023, building on its earlier framework for managing cybersecurity risks. Likewise, the ISO/IEC committee on AI that is considering standards on everything from red-teaming to LLM interoperability also passed the first official international AI management standard, ISO/IEC 42001, which organizations can use to demonstrate that they are responsibly integrating AI into their operations.

Subscribe now

5 LESSONS FROM HISTORY

Studying the past, you see how often standards—by design or bumbling—have shaped the technological present. But what about our technological future?

To develop good standards for general-purpose AI models and agents, we’ll need inputs from a range of groups, from scientists with know-how to institutions who can convene. Below [see infographic], we have identified five groups who’ll perform key roles.

What we mapped includes more than just official standards development organizations. We also want to capture the early spaces where standards emerge in practice before they are formalized on paper. How this works is closer to a swirl of inputs than a steady procession. Sometimes, the same organization or individual may operate in several groups at the same time. Ideas and efforts may also originate in one group, then migrate to another, with different groups offering varying degrees of speed, flexibility, expertise, and perceived neutrality.

For these groups—and the policymakers, business leaders, and advocates who shape their work—what lessons can history teach about our AI future? Here are five:

1. Standards matter! At best, technological standards chart a wise path; at worst, they fill the path with potholes. Consider the bulky electrical converters that one still needs when traveling—it didn’t have to be that way. On the other hand, when we get it right, the benefits of technology spread faster, more inclusively, and more securely.

2. The standards process needs to speed up. ISO says that the average time to develop one of its standards is three years, and ISO is not an outlier. Given the pace of change in AI, we need to speed up. For priority goals, like finding secure ways for agents to operate and interact, which the US Center for AI Standards and Innovation is working on, we need to find ways to accelerate that don’t jeopardize the overall quality and integrity of the process. This may mean looking across the many groups now focusing on AI standards and finding ways to collaborate early, rather than duplicate. It may mean focusing more on technical protocols and specifications documents that are quicker to develop. It may also mean using AI to help deliberate on and write standards, and moving to more nimble digital formats that are easier to update and use.

3. We need more efficient ways to input on standards. All standards, from those underpinning steam engines to the Internet, had to chart a unified path through diverging viewpoints, with an end result that did not please everyone. For AI, the challenge will be far greater. It is more akin to 1,000 technologies, and will affect different groups in different ways. This means that any broad directive— say, to “develop standards that make AI fair”—risks an everything-bagel solution. Many groups would rightly be heard, but the output would be too vague to provide the “how” that justifies a standard, leading to confusion, a stifling of innovation or the standard being ignored. This suggests that most standards should be precise in scope, targeting specific components of AI systems or specific concerns, from certifying the source and history of online content to combating the leaking of confidential data. More precise standards will make it easier to identify a wider range of relevant voices and incorporate their input.

4. Frontier AI standards should focus on large-scale risks. Historically, standards have accelerated the diffusion of technology, amplifying its benefits but also, in places, its negative impacts. For AI, foresight and risk management standards will be critical to getting ahead of future risks and speeding adoption. But with a technology as general-purpose, fast-improving, and poorly understood as AI, perfect foresight is impossible. Standards move at a human pace and cannot standardize a future that we cannot perfectly see. As a result, the focus should be on developing scientifically robust standards to address the most consequential or large-scale risks, such as those targeted by labs’ Frontier Safety Frameworks.

5. Wrong paths are inevitable, so we should catch them early. Now and then, technology stumbles into a poor standard, and it’s onerous to go back. But not necessarily impossible. Especially if we act if we catch it early. Consider the U.S. railroads taking action to unify their systems through a mighty coordinated effort. Groups working on AI standards devote much time to building consensus about new initiatives. They should also use the processes available to them to review and withdraw standards, where needed, to avoid sub-optimal lock-in. This also means giving third parties more opportunities to access, understand and constructively critique early AI standards. And designing standards and protocols that are modular, and can be swapped out, or updated, without major downstream consequences.

QUESTIONS FOR YOU

1. Where do you feel most hope for frontier-AI standards?

2. Where do you worry about a lack of progress on frontier AI standards?

3. When you imagine a missing standard for frontier AI, what is that? A technical protocol specified in code? Or a fuzzier process-standard?

4. Might your standard become politicized? Is it something that hinges on values? Or might most governments in the world support its adoption?

5. What’s a scenario in which your proposed standard goes awry? How could you detect and mitigate that?

6. What would be the primary role of government in your standard? Supplying technical expertise? Convening authorities and experts? Incentivizing your standard via public procurement, regulation, or other methods?

Share

Thank you to Shaked Karabelnicoff, Tom Rachman and Bruno Galizzi for support with research and review. As with all pieces you read here, this is written in a personal capacity. All opinions and any mistakes belong to the authors.