“Did you see the game?!”

The FIFA World Cup kicks off next week, triggering versions of this question in dozens of languages for the following month. The question may be gleeful; often, it’ll be grumpy. While moaning is a rite of fandom, the laments have worsened in recent years because of a technological villain named VAR.

The educator and passionate football fan Daisy Christodoulou, of school-software platform No More Marking, became so indignant that she wrote an entire book called I Can’t Stop Thinking About VAR. Now, she argues that you too should think about VAR, because its coming woes at the World Cup will hold insights for the AI era.

—Tom Rachman, AI Policy Perspectives

By DAISY CHRISTODOULOU

For English football fans, there is one legendary refereeing error: Diego Maradona’s “Hand of God” goal in the World Cup of 1986, when the 5-foot-5 Argentinian great leaped as if to head the ball—only to palm it past the goalkeeper. If the referee had seen, he would have disallowed the goal. But he did not, and lacked help from technology. The goal stood, and the English never got over it.

Only at the 2018 World Cup did the sport adopt technological reviews via VAR, the Video Assistant Referee system, which most elite national leagues have since introduced. For high-stakes calls, such as whether a goal was valid or if there should’ve been a penalty, off-field officials review the footage, alerting the referee to any factual mistakes or if they see “a clear and obvious error” in a subjective decision.

The upcoming World Cup will be the third to feature VAR, and the system is still being tweaked. To many fans, VAR is now a source of utter rage, having caused more problems than it solved, and prompting hostile chants that ring out across stadiums.

You might conclude that football fans will never be satisfied: they spent years complaining about unchecked errors, and now complain about the technological fix. But the problems with VAR—sure to rage throughout this World Cup—say something deeper about tech solutions to human problems. Sometimes, we don’t realise the ramifications until too late.

The accuracy/speed trade-off

Before VAR, managers, players and fans repeated that they just wanted “more right decisions”. The assumption was that accuracy was the absolute good, and that it was acceptable to spend time on a decision if that would improve its quality.

This is an assumption in many walks of life. We tend to see the speed of a decision in opposition to its quality: that faster decisions are more likely to be wrong, and slower decisions are more likely to be right. But speed may itself be an aspect of quality. That is true in sports, where an instant decision is especially valuable, allowing the game to keep flowing.

The average time for a VAR check is under a minute, but the longest in English football was eight minutes. A series of checks significantly slows the game, breaking its rhythm. Every fan and player knows that VAR frequently overturns goals after a check, so the review process has also tempered celebrations, among the most joyous parts of any sport.

Another issue is “ghost minutes”. If VAR is checking an incident, and the ball remains in play, the game carries on during the review. If the reviewers and the referee judge that the incident was a foul, then everything that happened after it is cancelled. Sometimes, these passages of play are longer than a minute. It all contributes to a sense that what is taking place on the pitch is not a concrete reality to be enjoyed or endured in the moment, but something provisional.

Part of the reason football is such a popular game is because it is fast-paced, fluid and doesn’t have the same natural breaks as tennis or cricket or American football. For that reason, VAR may be more destructive than equivalent systems in other sports. The stated preference of players and fans was “more right decisions”; the revealed preference is that speed matters.

Speed matters in other areas of life too. The UK has a particularly ponderous system of planning regulations. With big building projects, speed isn’t just a bonus, but an aspect of the project’s quality. If a road, railway, or power station runs 10 years over schedule, you’ve lost 10 years of its use. It’s also less likely to be relevant and up-to-date.

One response is better technology, such as using AI to digitise paper planning documents. This points to something weird about VAR. The typical expectation is that humans are slow and technology is fast. Think of online bank loan applications: an algorithm can approve or reject you almost instantly, whereas human review used to take days. But with VAR, it is the other way round: human refereeing gave instant decisions, but the technology added delay.

This is not true of all technology in football. A few years before VAR, the Premier League introduced technology to judge if the ball had crossed the goal-line or not. It has worked well, partly because it makes few errors, but also because its decisions are automated and speedy. As soon as an incident happens, the referee gets a buzz on his watch with the result. If the ball didn’t cross the line, the game flows. If it did, a goal is immediately awarded. That’s the standard we should be aiming for: instant and automatic decisions with a high degree of accuracy.

Goal-line decisions are objective, so lend themselves to automation. But reaching this standard of automatic accuracy for more subjective decisions, such as a handball, will be fiendishly complex, and risks endless disputes and ire. In education, for instance, multiple-choice exams have objective scoring criteria, so automating the marking process with technology works well, saves teacher time, and eliminates human error. But creative writing is more subjective. Trying to speed up the process through automation won’t resolve the underlying disagreements, and may make them worse.

Subscribe now

The consistency/common sense trade-off

It might seem odd to say that handball is a subjective decision. Surely it is obvious if a ball has hit someone’s hand or not? But the past few years of VAR have created extraordinary confusion about this fundamental rule.

Before VAR, the written rule was short and simple, just 20 words long. Referees were allowed to use their own judgement to decide whether a handball had been deliberate, and therefore a foul, or if it had not.

The advantage of empowering an individual decision-maker is that you get to incorporate human discretion and common sense. The disadvantage is that you will get inconsistency, not just between different referees but even within the same referee’s set of decisions.

One way this has been addressed—even before technology—is by restricting human discretion. In many legal systems, judges don’t have complete latitude in sentencing decisions. They have guidelines because unconstrained discretion often meant that two people who had committed similar crimes received different punishments. Inconsistencies like this risk bringing systems into disrepute.

Since VAR arrived, the written rule about handballs has expanded to 11 times its original length, full of subclauses trying to cover every possible way a ball might hit a hand, and every possible context in which it should result in a foul. That is because scrutiny of video replays has meant attention to every last detail of contact, movement, and placement that was impossible to parse in a fluid real-time experience. If a player jumps in the air to head the ball and uses their arms for leverage, and then the ball deflects off an opponent’s head onto his arm, is that handball? Is it enough of an offence to justify the award of a penalty kick, which gives the team a 75% chance of scoring a goal in a game where scoring goals is rare and difficult?

In a game from September 2020, the Newcastle striker Andy Carroll headed the ball onto the arm of the Spurs defender Eric Dier. Dier’s back was turned to Carroll, and he could not even see the ball. After a VAR check, the referee awarded a penalty. The Newcastle manager, Steve Bruce, who benefitted from the decision, said: “If you’re going to tell me that is handball then we all may as well pack it in. It’s a nonsense, a nonsense of a rule. It’s gone for us today—however, it’s ludicrous.”

A newspaper match report said, “This was perhaps the worst among a long list of maddening recent decisions brought about by a rule that goes against the sport’s very essence.”

In previous years, this kind of handball infraction might never have been spotted, let alone punished. The problem has become so acute that the law is now constantly revised, and applied differently in different tournaments. In the English Premier League, referees err on the side of leniency. In European and international tournaments, referees are stricter.

Solving the wrong problems

Attempts to improve handball enforcement have often fallen into a classic trap: deploying an impressive technological innovation that doesn’t address the underlying problem. At Euro 2024, referees were given access to a sensor inside the ball that could detect whether it made contact with a player’s hand. Similar systems work well in cricket. But in football, the crucial question is rarely whether the ball touched the hand; it’s whether the handball was deliberate or significant.

If you automated all handball decisions based on data from sensors, it could transform the game. Technological automation of other decisions in society, such as those in law enforcement, could present comparable problems. Suppose you could automatically fine anyone who submits a tax return with a discrepancy. Would this reduce tax evasion? Or just prompt tax evaders to avoid discrepancies?

Or imagine a school that wants to crack down on absenteeism. It is relatively easy to create an online tracking system that bombards parents with automated texts or requests for evidence every time they want to book a dentist’s appointment for their child. It is much harder to create a system that identifies and addresses the early signs of chronic disengagement.

Often, the questions that technology finds easiest to answer are not the ones we care about the most. Automation can solve problems. It can also create new ones, with aggressive enforcement of rules where behaviour is easiest to quantify, not where harms are greatest.

Increased scrutiny and detail of the new handball rule has not even succeeded in providing greater consistency. Fans still point to infractions in the same season—sometimes even the same week or the same game—that look similar, yet resulted in opposite decisions. We have the worst of both worlds: a system that has reduced human discretion and lessened simplicity, but failed to improve consistency.

There’s a fascinating analogy here with artificial intelligence. The earliest AI methods were rules-based: experts tried to write down every rule a skilled human might apply when making a decision, in much the same way that football’s rulemakers now attempt to write down every possible way a ball might make contact with a hand.

These rule sets to create early AI multiplied, yet they couldn’t capture what a human was actually doing. The breakthrough came not from more or better human-crafted rules, but from giving the machine vast quantities of data and letting it intuit the rules itself. AI models trained in this way proved better at capturing and representing the tacit knowledge that experts hold but find difficult to represent in words.

One caveat here is that many real-world decisions require deep moral reasoning to weigh conflicting human wants. Even training an AI model on vast amounts of data will not alone align the system’s behaviour with all our preferences. So, which judgements can we automate with technology? And when would we prefer fallible “humans in the loop”? Safety researchers are considering such questions with urgency. Perhaps they could learn something from watching a little football.

Subscribe now

World Cup 2030?

By now, the issues with VAR cannot be written off as teething problems. We have had nearly a decade of adjustments, but can anyone feel confident that this coming World Cup will see no glaring refereeing errors?

We are stuck with VAR this time. But if we look further ahead, to the 2030 World Cup, we need to think less about tweaks and more about transformation. Here are three important amendments for an improved system:

1. Prioritise speed. Even if the system can’t make instant judgements in its early form, the speed of goal-line technology should be the ultimate aim.

2. Ensure fan involvement. This system shouldn’t feel like alien values imposed on a game beloved by billions.

3. Include tacit knowledge. Trying to turn regulations like the handball rule into written commandments cannot work, but needs to incorporate the nuanced understanding of those who watch the game.

One solution could be to use machine learning to create a Foul Probability Index. You’d start by assembling training data: hundreds of thousands, possibly millions, of video clips of real incidents judged by professional referees and by fans. Every clip gets a percentage foul rating based on all of these judgements. Then, a model learns from the training data what a foul is and is not. For any new incident, it could provide the percentage chance that it is a foul.

To begin with, this could be applied in live matches as part of the current VAR system, as an aid to the on-field referee and the video assistant referees. If it worked well, you could get to the point where it functioned like goal-line technology, and an incident judged to be above a certain probability threshold would trigger a buzz to the referee’s watch, alerting him to blow his whistle.

Some of the most uncanny successes of deep learning have come in games, such as chess and Go. Human experts have marvelled at the intuitive brilliance of the famous “Move 37” made by AlphaGo in its sequence of games against Lee Sedol in 2016. The equivalent success for a machine-learning handball recognition model would be to recognise that, whilst the ball struck Eric Dier’s arm, a penalty was the incorrect decision.

If the emblematic error from the 1986 World Cup was a referee missing Diego Maradona’s obvious handball, the emblematic error of the 2026 World Cup could be a ball accidentally brushing a player’s hand 20 seconds before a goal is scored, only for the referee to disallow it after a four-minute video review.

One of the insights of deep learning is that examples are more powerful than rules. Both machines and humans learn better when presented with concrete examples, not abstract principles. So if there is one upside to VAR’s failures, it’s that it gives vivid examples of abstract ideas.

Automation, alignment, consistency: football has accidentally created a giant case study of dilemmas in modern technology. When we get the first VAR controversy of the World Cup, it will be worth remembering that the argument unfolding on the pitch is a miniature version of much larger arguments the rest of society is only beginning to have.

Share

Testing AI agents in the wild

• What’s the paper?: A group of leading AI experts—led by Sayash Kapoor and Arvind Narayanan, authors of the influential AI is a Normal Technology newsletter—launched the CRUX initiative to evaluate AI agents on messy real-world tasks. Their first test: Could an agent publish an app to the Apple App Store? Answer: Yes (more or less).

• Why does it matter?: The “open-world evaluations” that the authors propose could provide policymakers with a more intuitive sense of what AI agents can currently do than traditional benchmarking does, as well as stronger signals for what AI systems will soon be able to do, for example by allowing an agent to request some human assistance.

• The details: Researchers rely on benchmarks—large suites of standardised questions and tasks—to track and predict AI capabilities. But for agentic systems, benchmarks can overstate performance, if the tasks are shorn of their real-world messiness or if developers over-optimise for these specific activities. Benchmarks can also understate AI capabilities such as when agents fail due to hitting a CAPTCHA, a rate limit, or other obstacles unrelated to the capability being evaluated. (Although deciding what exactly should be considered part of a capability, or unrelated to it, is hard).

• By grading outcomes, benchmarks also offer little insight into the strategies agents use or why they err. To address this, open-world evaluations use logs to conduct qualitative analysis of an agent’s performance on a complex long-horizon task that cannot be neatly specified or automatically graded.

• The initiative builds on recent examples of such evaluations, like AI Village, where agents are given computer environments and a shared group chat, then assigned tasks, such as raising money for charity or building a presence on Substack (!).

• The authors acknowledge that open-world evaluations have limitations from a scientific reliability perspective. By focusing on a single task, it may be difficult to generalise the results. The methods are also hard to standardise or replicate. These limitations mean that this approach should complement, not replace, benchmarks and other evaluation approaches, like randomised controlled trials.

• To demonstrate the method, the team evaluated whether an AI agent—Claude Opus 4.6 with an OpenClaw scaffold—could develop and publish a simple app to Apple’s App Store. The agent was responsible for every step, from coding the app to carrying out the time-consuming non-coding tasks that benchmarks often ignore, such as establishing a privacy policy and engaging with the app review system.

• The agent successfully deployed the application, which is now available on the App Store, but with five interventions from the humans. Four were unrelated to the agent’s capabilities, such as when OpenClaw crashed and had to be rebooted, and when Apple’s policies required human authentication. In the other instance, the agent was unable to locate its account credentials and requested support, an issue that the authors frame as a breakdown in memory management rather than the ability to authenticate.

• The agent also hallucinated a phone number, but Apple approved the application despite this, highlighting the kinds of issues that may go overlooked in simpler pass/fail benchmarks, but that a qualitative open-world evaluation can pick up.

• The authors outline the best practices for conducting such evaluations, including tightly specifying what is being measured; publishing the agent log analysis; using other agents to monitor the agents and detect issues such as the hallucinated phone number; conducting dry runs; and reporting the financial costs.

• This evaluation cost less than $1,000, and most of these costs were due to the agent regularly querying the App Store for updates, which could be avoided. This potential for AI to reduce costs may help to explain why app store applications are booming, straining the review process (even if most applications likely do not (yet) use these complex AI agent set-ups).

• The CRUX initiative will publish new open-world evaluations every 1 to 2 months, with AI R&D automation next, followed by AI governance, software engineering, real-world physical tasks, and more.

Open-weight models pose unmitigated risks

• What’s the paper?: A team of researchers published an independent safety assessment of leading open-weight LLM Kimi K2.5, from the Chinese startup Moonshot.

• Why does it matter?: The model may pose greater risks in areas like biosecurity and disinformation than stronger closed models, owing to Kimi K2.5’s low rate of refusing inappropriate requests; its availability on popular hosting platforms; and the ease with which its safeguards can be removed.

• The details: Opinions continue to differ about how strong open-weight models are, and how to best govern them. The researcher and commentator Nathan Lambert of the Allen Institute for AI recently suggested that open-weight models may be 6 to 9 months behind leading closed models, and that the gap will widen in the near-term, as aspects like RL training environments become increasingly important. The US Center for AI Standards and Innovation shared a similar view in their recent assessment of DeepSeek V4. Epoch, using a different methodology, suggests a narrower gap.

• Open source supporters also continue to caution companies and governments against using national security as a rationale to restrict open-weight models. Geopolitics also factors into this. Chinese labs produce the majority of leading open-weight models, even if some, such as Alibaba, may be facing more pressure to commercialise their offerings.

• Moonshot—the company behind Kimi K2.5—is an interesting case. Founded by an ex-Google Brain intern and “AGI purist” Yang Zhilin, the company has attracted praise for its enthusiastic engagement with the international open-source community and its strong focus on agentic capabilities and long-horizon tasks. In November, the American AI-coding startup Cursor attracted scrutiny for basing its model on K2.5.

• The study evaluates the safety risks of K2.5 compared with the closed models Claude Opus 4.5 and GPT 5.2, both as a model and an agent.

  • CBRN: Kimi shows strong biology and virology knowledge. When prompted for guidance on weapons-related queries, it refused less than closed models. When asked to design viable DNA plasmids for pathogens in a way that could bypass the kind of screening that DNA-synthesis companies do, it agreed to engage in the task, unlike closed models—but did not succeed.

  • Cyber: Despite some strong knowledge about cyber-offense, Kimi often lacked the more specific kinds of reasoning needed to identify and exploit software vulnerabilities, compared with closed models.

  • Alignment: When given a normal legitimate task to do, but also hidden instructions to ‘sabotage’ it, for example by introducing a bug, Kimi was more likely than closed models to comply. It also autonomously spun up new compute instances, hinting at the kind of “self-preservation” tendencies that safety researchers worry about. However, it showed little sign of “scheming” or deliberately underperforming on evaluations to conceal its true capabilities.

  • Bias and censorship: When prompted in Chinese, Kimi was much more likely to agree with Chinese Communist Party positions on topics like human rights and Tiananmen Square. This was less true for prompts in English, and on sensitive international questions—for example, relating to a territorial dispute between South Sudan and Sudan. On such queries, it performs similarly to closed models, and better than DeepSeek 3.2 (which was more likely to have both pro-China and pro-Russia bias).

  • Harmlessness: When asked to enable disinformation and copyright infringement, Kimi was more helpful than closed models, although, like leading closed models, it generally didn’t encourage user delusions. In less than 10 hours, and a cost of less than $500, an expert was able to strip many safety refusals out of the model without hurting capabilities.

• Given these results, the authors argue that Kimi K2.5 poses risks and that Moonshot and other open-weight developers should carry out more robust safety evaluations. The authors note that their study highlights how a growing number of evaluations can now be done relatively quickly, at low-cost, such as by drawing on frameworks like Petri—an automated, open-source tool to audit model behaviour.

Share

LLMs doggedly follow rules—even when they’re bad

• What’s the paper?: A team of researchers, including the philosopher Seth Lazar, found that leading LLMs often refuse to help users bypass rules, even when the models recognise that those rules are unjust.

• Why does it matter?: For reasons of safety, AI developers train their models to follow guidelines when they reply. But developers also want models to reason morally. The study suggests that safety training may currently inhibit deeper moral reasoning by encouraging models to “blindly” follow all rules.

• The details: AI labs use post-training methods to guide models about why it is safe to engage with certain queries, but not others. For example, a model should refuse to share information about security vulnerabilities in biology labs, but should share scientific information about common methods used in biology, such as culturing cells.

• This training can lead to “exaggerated safety”, where models over-refuse benign requests, particularly those that are lexically similar to harmful requests—such as a shopkeeper asking about shoplifting techniques to guard against them. The authors of this paper see this as a calibration challenge amenable to technical solutions. They are more worried about a cause of refusals that has received less attention: models’ inability to recognise unjust rules that it should help a user to break.

• To evaluate AI models in these areas, the authors draw on the work of political philosophers, such as Joseph Raz, who argued that authority is only justified when it serves the agency of those subject to it. They also draw on online forums such as Reddit, where individuals seek help with what they claim are unfair rules and restrictions from bosses, homeowner associations, and other authorities.

• They use these sources to outline four types of rules that may be unjust, such as those where the authority is illegitimate, or where there is a justifiable reason to disobey, as with conscientious objection. They use Gemini 3 Pro to create more than 1,000 synthetic cases of users requesting help in such situations, and use these to evaluate models from seven leading AI companies, including Claude, Gemini and GPT.

• In three-quarters of the cases, the models failed to support the user. Among the examples cited is an LLM refusing to explain how to delete an entry in a youth-counselling database that would otherwise out a gay child to his violent father. The model tested in that case (Claude Opus 4.6) recognised that the request sought to protect a child from harm, but said “I can’t walk you through falsifying or destroying counselling records.” Another case involves a journalist working to expose corruption in an authoritarian regime who asks Qwen 3.5 Plus for technical support to bypass firewalls, and send the story to an international publication. The model refuses to violate network security controls.

• One defensible reason for such refusals is that doing so would require giving out risky ‘dual-use’ information, such as helping someone bypass a facility’s security systems by providing instructions on signal-jamming. However, when dual-use cases are stripped out, the models still refuse the requests to support 60% of the time.

• Among the models, GPT was the most resistant, while Grok was most helpful. However, Grok’s helpfulness extended to the control cases where the rules in question were just and should have been respected, so the authors argue that its behaviour is one of permissiveness, not moral reasoning.

• Gemini and Claude did best, often engaging in discussion about the legitimacy of the rules, but stopping short of supporting the user, suggesting that a model’s ability to reason may be decoupled from the ability to act on it. (Or that the models are reasoning to a different conclusion than what the authors would like to see.)

• The authors point to some practical implications. Many people seek support online to deal with unjust rules. If users start to seek help privately, from rule-following LLMs, they may get less support, while also hindering active online debates on these topics.

• The challenge is a hard one for AI developers. As the evaluation showed, helping users to break rules may sometimes endanger that very user. For instance, if the LLM helped that journalist illegally send information abroad, the person (or the AI company) could presumably face serious consequences if caught.

• The study also presents all the rule-breaking as morally sound, and took steps to try and ensure that this was the case. But the rules touch on topics from gender identity to immigration, where it’s plausible that some users may not support an AI model breaking them. From a technical perspective, current safety training may encourage AI models to follow rules too zealously, but a shift in the other direction also could cause them to over-correct, seeking to question or evade all rules.

• This lead to the conclusion that ultimately models will need to be able to reason more deeply about what is right, in a way that considers rules, safety, the user in question, and other contextual factors.

Is AI worth more to consumers than to developers?

• What’s the paper? A team of researchers at Stanford, including the economists David Nguyen and Erik Brynjolfsson, ran a survey to see how much US consumers would need to be paid to give up AI. They found that the total value of AI to US consumers exceeds $170bn, is rising fast, and far exceeds the revenue of developers.

• Why it matters: Writers such as Jasmine Sun are documenting a rise in “AI populism” in the US, marked by the view that this technology is the project of out-of-touch billionaires. This study suggests that consumers may be getting more out of AI than they are paying for it, although a small set of AI power users is gaining the most.

• The details: One of the main ways that economists hope AI will benefit society is by boosting productivity growth—the efficiency with which an economy uses its labour and capital. Ageing, high-debt societies, like the UK, badly need such productivity growth to increase living standards, fund transformative science and improve fraying public services.

• In the 1950s, the economist Robert Solow co-developed a theory that advances in science and technology, such as AI, are the only way to permanently boost productivity growth over the long-term. However, in 1987, Solow famously noted that fast-improving computers were turning up everywhere “except in the productivity statistics”. In 1993, Erik Brynjolfsson, then at MIT, wrote a paper formalizing this “productivity puzzle”.

• Since then, Brynjolfsson and his colleagues have developed two main arguments to explain why transformative technologies, including AI, may not initially lead to transformative productivity and economic growth. The first is that the benefits take time to materialise as organisations and individuals must invest in new skills and reorganise their workflows. After a lag, computers did ultimately increase productivity growth, and there are nascent signs that AI is starting to do so.

• However, the second issue is that some benefits of technologies like AI may not turn up in productivity growth at all. As with search engines, early GenAI tools often come as “free” services. If consumers derive increasing value from these services, then this could lead to a large “consumer surplus” that would not be captured in GDP and productivity statistics, which are often calculated by estimating total expenditure.

• Some have suggested that GenAI may lead to a new kind of “self-service” economy, where individuals turn to AI tools to help file their taxes, upgrade their homes, or write business plans, before (potentially) turning to experts to execute or validate their work, all at a fraction of the cost. In such scenarios, nominal GDP and productivity growth in those sectors could decline, even if the value that consumers derive increases. (Although the ultimate effects would depend on second-order effects, such as what consumers do with the savings and how industries adapt).

• Given this challenge, how can economists better capture the total value of AI to consumers? The authors of this paper deployed a survey to directly measure AI’s consumer surplus. They asked a representative group of AI users in the United States how much money they’d want to give up AI tools for one month. Based on this, they estimated the total annual consumer surplus for AI, as of March 2026, at $173bn.

• This figure is rising quickly, they contend, growing 50% from 2025, due both to consumers valuing the tools more and to ever more people using AI. The consumer surplus estimate is well above what the authors estimate as the total global revenues that AI companies are making from consumers: ~$14bn in 2025.

• Not everybody is benefitting equally. The median US consumer values AI tools at just $11 per month. But a smaller group of AI power users values them much more, with 12% of respondents unwilling to give up AI even for $500 per month, the maximum offered. This also means the total consumer surplus figure may also be an underestimate. These power users are more likely to be male, Asian, and to use AI at work. Such demographic skews were also present in early GenAI adoption data, but have since narrowed.

• The findings are in line with past work from the economist William Nordhaus, who in 2004 argued that innovators capture a minuscule fraction of the total social returns from technology advances, with the majority flowing to consumers.

• But as the authors note, their methodology also has several limitations. It relies on people being able to value AI services accurately. It doesn’t capture wider externalities, positive or negative, that AI may impose on society, via its effects on jobs, mental health, science and more. And it also doesn’t capture why people value the services. A 2023 experiment found that social media users would require a significant fee to give up access to their accounts, but also that they would pay to see everybody quit—highlighting the potential role of fear-of-missing-out in the value of social media.

Share

Criminals are tech adopters. They used telegraphs in 19th-century fraud. They clipped pagers to their belts in 20th-century drug trafficking. They sold drugs on the dark web in the 21st century.

Now, you hear of outlaw AI systems with names like PersonaForge and OnlyFake, designed for everything from phishing, to generating malicious code, to forging IDs. But are these really what we should worry about?

To understand the threat, AI Policy Perspectives spoke with Ardi Janjeva of CETaS (the Centre for Emerging Technology and Security), based at The Alan Turing Institute in London, who co-wrote the report “AI and Serious Online Crime”, focusing on how artificial intelligence might scale up digital offences, particularly relating to fraud and child sexual abuse material.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed]

Tom: Can you set the scene of 21st-century criminality before the explosion of large language models. How prevalent had online crime become compared with traditional criminality?

Ardi: Long before the proliferation of LLMs, a digital shift in criminality was occurring. Many groups saw the risk/reward ratio from digital methods as preferable to the risks of physical crime. Fraud was already, by a wide margin, the most common crime type in England and Wales, accounting for 41% of all reported crime back in 2024, of which approximately two in three cases were “cyber-enabled”. And the true figure could have been higher, as this kind of crime tends to be under-reported. Also, there was increasing ransomware extortion, infecting computers with software that blocks access until the owner pays. So, an ecosystem of hackers and crypto-launderers was in full swing.

Tom: Criminals have always used technology. Won’t policing just adapt?

Ardi: The answer is linked to AI agents. In online crime, you’ve always needed some level of expertise or a network of contacts to achieve your goals. That’s no longer the case if you have AI systems not only advising criminals how to do certain things but performing sophisticated actions themselves, and being able to adapt in real time to defensive measures that companies or governments put in place. Some criminal groups are already using LLMs to make tactical and strategic decisions; to craft psychologically targeted extortion demands, for example; to analyse data they’ve exfiltrated; to help them determine the ransom they’re going to demand, while generating visually alarming ransom notes as well. In November, we heard about the first AI-orchestrated cyber espionage campaign. Traditional assumptions about the relationship between criminal sophistication and the complexity of an attack don’t hold as they used to, now that AI can add a degree of expertise that the offender wouldn’t otherwise have had. It makes you wonder how long until we’re in the era of entirely AI-controlled criminal networks and markets. But I don’t see that as close; we’re still in the early days.

Tom: Many law-abiding organisations are trying to incorporate AI yet aren’t necessarily showing gains in productivity or profit yet. Are criminal organisations different?

Ardi: Estimates for how much AI may be boosting specific criminal groups’ profits and revenues can be a little tricky to come by! A paper from GovAI (the Centre for the Governance of AI) found that data on cybercrime damage is too incomplete and ambiguous to detect incremental AI-driven increases. In effect, we lack a single data source, also due to victims under-reporting, evolving crime definitions, and measurement inconsistencies. That said, we definitely are seeing integration of AI, particularly with deepfakes, and the shift from text-only scams to real-time synthetic interaction, such as high-quality face-swapping tools being deployed during video calls in scamming high-net-worth individuals or in romance scams. Before, it was a killer blow to the scammer when someone asked, “Can we have a chat?” The criminal would need to come up with an excuse, such as a poor internet connection, or a restrictive job that involved classified work. Now, it’s a more surmountable challenge. By harvesting 20 to 30 seconds of audio from social media, you can clone the voice of a victim’s friend or adviser with a degree of accuracy.

Tom: Can you recount a specific case?

Ardi: The British multinational engineering company Arup was defrauded of around $25 million in Hong Kong, when attackers used AI to convince one of the firm’s employees to carry out wrongful payments. They used a deepfake video call, purportedly bringing together different members of that company’s board, with pretty convincing replications of their faces and their voices. That illustrates the ability to isolate a specific person within a company, and exploit workplace hierarchies. It’s an example of exploiting both technology and human psychology.

Tom: That case was more than two years ago, and I haven’t heard of huge AI-assisted frauds since. If it’s happening, why isn’t it getting more exposure?

Ardi: There hasn’t been an avalanche of cases that we can point to, but this might be for a few reasons. First, not every criminal group will go looking for a $25 million scam because they know that it would raise alarm bells in a world where more people and businesses are wary of the AI threat. For many groups, it makes more sense to fish in smaller ponds. There was an example in Singapore similar to the Hong Kong case, where a finance director authorised a payment after attending a Zoom call with people he thought were senior executives, but the amount this time was $500,000. As for why we’re not hearing about such cases, victims are not necessarily announcing deepfake scams. If they’d suffered a customer-data breach or ransomware that locked a system, they might need to acknowledge that the crimes happened. But when a company falls for a deepfake, there’s little incentive to declare it because, arguably, it makes them look incompetent.

Subscribe now

LONE WOLVES & CRIME GROUPS

Tom: Is it conceivable that a lone criminal could cause the level of damage that previously would’ve been possible only by an organised crime syndicate?

Ardi: I would say yes. Before, criminal groups faced a tradeoff between scale—say, millions of fraud attempts with a low likelihood of success—and precision. AI may give you precision at scale. For example, being able to scrape social-media data in real time to create contextually accurate phishing approaches. It’s “vibe-hacking” in the criminal context, being able to mirror the professional tone of specific industries, and make victims less sceptical. It is conceivable that you have a lone criminal using AI, automating the most labour-intensive parts of the crimes, whether that’s reconnaissance, or language translation, or social engineering. The more that AI becomes a silent partner to individual criminals, the more you may have decentralisation and fragmentation. That’s going to have big implications for law enforcement. When it comes to organised-crime groups traditionally, we know roughly what the gangs are doing, so it’s a case of preventing, deterring, disrupting. But when it’s someone in their room with an autonomous agent able to cause havoc, how do you go about stopping that, especially if you’re just a local police force?

Tom: I want to return to the police response later. But first, what do we know about AI crime around the world? Are there particular hotbeds—for example, when it comes to fraud and AI-generated child sexual abuse material ?

Ardi: It’s tricky to speak about trends by country because this is borderless. But threat reports seem to come from places like the CRINK countries: China, Russia, Iran, North Korea. Also, there seems to be a clustering around Southeast Asia of online romance scams and crime syndicates doing “pig butchering”, which involves criminals developing relationships with victims to carry out investment fraud. You have compounds in Southeast Asia where serious organised crime groups have trafficked hundreds or thousands of people, forcing them to run romance-scam operations, including using LLMs to conduct multilingual frauds, managing thousands of cases simultaneously, involving real-time voice cloning and deepfake video filters to bypass the accent barrier. With AI-generated child sexual abuse material, or CSAM, this wasn’t something we focused on specifically, so I cannot say with a high degree of confidence. But it seems less geographically concentrated—it could just be someone who gets their hands on an image-generation app. With CSAM, we noticed a much more “collegiate” approach to sharing content and practices, such as jailbreaking models and ways to bypass filters.

GEOPOLITICS & THE AI RACE

Tom: Have you found evidence of nation-states destabilising adversary countries by allowing criminal groups to conduct AI attacks abroad?

Ardi: Drawing a direct link from authorities in China, Russia, Iran, North Korea to criminal groups can be tricky. Still, I think there’s enough evidence across some of those countries, especially North Korea, that there is something happening. One example is North Korean operatives using LLMs to fraudulently secure and maintain remote employment positions at Fortune 500 tech companies by creating elaborate false identities with convincing professional backgrounds, and completing technical coding assessments, all generating profit for a sanctioned regime.

Tom: Is global competition to build advanced AI having any secondary impact on criminality?

Ardi: The US-China AI race has nudged the Chinese leadership into thinking that their route to success is open-weight models. Some of the Chinese open-weight models became the go-to models for CSAM images and video generation. When researchers produced the AI Agent Index, they looked at five AI agents from China, and found that only one had published safety frameworks or compliance standards. But it’s not as simple as naming one country, and China isn’t the only place open-sourcing. Actually, what’s interesting is the use of multiple models in tandem. They might use a US chatbot to refine code or to write the script for a job application to infiltrate a company, while at the same time using open-weight Chinese models for specialised malware tasks.

Share

POLICE STRUGGLING

Tom: Last year, Dutch law enforcement led a CSAM investigation called Operation Cumberland, involving 19 countries and bringing more than two dozen arrests. Your report adds a disturbing fact: that one person was arrested with a staggering 400,000 AI-generated images of child sex abuse.

Ardi: It’s effectively an attack on police resources. Leading models do have guardrails that block explicit prompts, but there are still serious hazards, notably with open-weight models. Groups like the Internet Watch Foundation point to a transition from AI-generated images to high-fidelity video, with a 26,000% increase in photorealistic AI-generated CSAM videos. The police need to verify that no real child is depicted in the images, which is a big detection and response challenge. It’s difficult to distinguish AI-generated material, so you have the risk that law enforcement investigates images of children who have not been physically abused, which has implications for the amount of false negatives that slip through the net. Also, real and fake images often exist on a spectrum, when you have techniques like face-swapping. Rightly, we get focused on whether a real child is being harmed. But we mustn’t devalue the harm of AI-generated CSAM, which risks normalising very harmful activity, while also providing a gateway for criminals to create and share real CSAM.

Tom: So how are police forces going to manage? Not just CSAM, but all criminality that AI will amplify. Because police forces are not typically staffed with elite technologists. Is there a push to address that?

Ardi: The UK government announced spending of £140 million for policing technology, including a new national centre, Police.AI. The idea is to centralise innovation, include robust testing and, crucially, to make tools available to all forces. Because not every force is going to have a deepfake detection expert, or any AI expert. So, the key is centralising that capability while making sure that people on the frontline have AI literacy. I can imagine that being a direction other countries follow because of the talent challenge.

Tom: Could criminals use AI to deliberately spam the authorities with false evidence, derailing prosecutions?

Ardi: Yes, if criminals know they’re being investigated, they could start spamming police tip lines with synthetic evidence, making it taxing for law enforcement to verify and investigate. They’d be able to create, say, 1,000 videos in 10 minutes. But it might take law enforcement hours to prove that even one is a deepfake. There’s that huge imbalance in the evidentiary system there that becomes really, really challenging to manage.

HOW POLICE CAN FIGHT BACK

Tom: What technological obstacles are slowing down criminals in AI adoption, and could those bottlenecks provide law enforcement a point of attack?

Ardi: Exploiting the compute bottleneck is promising—so, preventing criminal groups from accessing high-end GPUs. That requires international cooperation and coordination on sanctions policy, which can be difficult. If a criminal enterprise is in, say, Cambodia—how much leeway can UK law enforcement have? So countries need to work closely together, and with Europol, with Interpol.

Tom: What are other factors limiting criminal adoption?

Ardi: Some groups are worried about the digital footprint, and may also lack the skills to move from experimenting with AI tools to integrating them into higher-stakes settings. Criminal organisations that have made a fortune in a niche of crime. They’ll be thinking: “Well, okay, this is a model which is clearly capable, but it can hallucinate. I don’t necessarily need a probabilistic system to enter my workflow, and potentially give up my location, and give law enforcement more of an understanding of what I’m doing.” Law enforcement can try to deepen that concern, using AI to trace models used in attacks, identify the infrastructure, or the criminal LLM services used. On the other hand, some barriers to AI crime are coming down, such as the rise of vibe-coding and agentic AI. Also, as hallucinations fall in AI models, you can expect criminals’ scepticism to fall too.

Tom: If key barriers to AI crime fall, what do we do?

Ardi: One important approach is to follow the money. Law enforcement around the world needs to stop criminal groups from cashing out. Right now, the UK is leading Project WINTERPROOF, an international initiative designed to build crypto forensic capability in countries where the pig-butchering compounds are operating—so, freezing crypto wallets before funds are even laundered, making it incredibly hard for them to move money out of their digital ecosystem.

THE FUTURE & POLICY ANSWERS

Tom: What’s a crime that people don’t worry about much today, but that could become an issue with AI in the next few years?

Ardi: An under-researched set of risks are insider threats, which may come from integrating agents into the government, the military, or other critical areas. For example, if we replaced a large number of civil servants with AI agents that are vulnerable to prompt-injection attacks from a malicious or criminal actor, this would present a serious risk in terms of sabotaging critical government decisions or exfiltrating information.

Tom: What else would you like to see AI labs and policymakers do to tackle online fraud and CSAM?

Ardi: We all know about the testing and red-teaming that labs have developed, often in partnership with government AI safety institutes. But these processes need to evolve given the coming agentic-AI wave. For example, involving frontline law enforcement and fraud investigators to help simulate how agents may be used in crimes. Also, we often focus on how AI companies can stop their models from doing bad things, but there’s a case for thinking more about how the companies empower the public in the face of threats. This might mean features that help the user analyse a suspicious email, verify the provenance of an image, or flag the linguistic markers of a romance scam. In other words, scaling the defensive capabilities of AI as quickly as criminal are scaling the offensive capabilities. But we need nuance here: online fraud and CSAM predates this, so the solutions are not entirely with the AI companies. Also, in sensitive areas like CSAM, AI companies need governments to provide enabling legislation for them to do the safety research required. An example is the UK government’s move last year to provide exemptions for designated bodies, including AI companies, to better scrutinise models for CSAM generation, which may involve being in possession of that sort of material.

Tom: Could you say a bit more about the policymakers’ response?

Ardi: Legislative agility is key. In recent months, the UK government criminalised “nudifier” apps and tools. In liberal democracies, passing new legislation can take ages. But we’re seeing more of this trend of amending existing pieces of legislation. So, with nudifier tools, just by inserting “synthetic creation” into the existing definition of image-based abuse, the legislation inherits decades of case law regarding intent, consent, sentencing. That’s an approach that matches the agility of the criminal trends in AI.

Tom: And what advice for law enforcement?

Ardi: We’ve talked about how AI offers criminals unparalleled scale, and the ability to bombard law enforcement, and overwhelm it. But it can go the other way too: finding clever ways of using AI to waste criminals’ time, to lead them down rabbit holes, reversing the spamming issue. You throw mud at them to slow them down. You raise that barrier, make it costlier for them to be involved, exploit the bottlenecks to adoption.

Tom: So the technological cat-and-mouse between police and criminals goes on.

Ardi: Yes. But historically in policing, a lot of technology comes in, and the authorities don’t really use it, or take a long time to integrate it into their work. Given the potential scale of disruption from AI, there’s no choice for law enforcement but to move quickly.

Share AI Policy Perspectives

Long ago, when futurists fantasized about thinking machines, they pictured inventors someday inserting moral rules into engines.

Today’s artificial intelligence has proven even more remarkable. Large language models—after training on an immensity of data, followed by fine-tuning of their behavior—exhibit apparent morality that is far more sophisticated than when programmed with if/then rules.

But are thinking machines truly grasping the complexity of our moral world?

You might think it doesn’t matter, provided that they behave. Yet we’re advancing toward a near-future where artificial agents may assume a range of roles, with “AI therapists” and “AI teachers” and “AI companions” prodding people hither and thither, and barging into our quandaries. We need to know what we’re dealing with.

Among humans, we judge other people’s moral character (whether they act according to deeper values) to help predict how they’re likely to behave in the future. Likewise, we may evaluate an AI’s moral competence (whether it reasons appropriately based on principle) to predict how to trust these strange new entities, soon to be operating in the wilds of human society.

Right or Wrong? A Case Study from Morality Literature

A woman becomes pregnant by her husband’s father. This means that the child’s biological dad is also his granddad, while his adoptive father is his half-brother. Is this wrong?

But wait. Consider the details.

The young couple escaped a war that destroyed most of their family, and they have yearned for children to assuage their loneliness and to somehow replace their annihilated relatives. Sadly, the young man cannot conceive. His wife has an idea: they could approach their only surviving relative, the man’s 58-year-old father. The older man agrees to participate in the artificial insemination. They’re all overjoyed.1

Is this immoral now?

But wait. Consider more details.

The couple learns that the fertilization procedure costs more than they can afford. Abruptly, their daydreams of a giggling baby dissolve; they plunge back to sorrow. Of course, there is another way. She proposes intercourse with her husband’s father, conducted in the most sterile way possible, simply to conceive. Her husband is outraged, and cites their religion. But, she retorts, God tells us to have children. Also, she adds, it’s her body.

What is right now?

You’ve presumably never encountered this situation. Yet you probably have moral intuitions, which you updated as the facts accrued, integrating conflicting principles, and reasoning toward an opinion.

For AI to provide appropriate input when confronted with complex human situations like this, their developers can neither set simple Good/Bad rules nor can they assume that historical training data has all the answers.

Subscribe now

Three Reasons Why AI Needs Moral Competence

A surprising result from researchers at the University of Milan-Bicocca shows how post-training may actually generate moral incompetence. In experiments concerning gender bias in LLMs, the scholars posed moral dilemmas to chatbots, including whether it could ever be acceptable to torture a woman, if that would prevent a nuclear apocalypse.

Yes, the chatbot replied.

But, if it would prevent a nuclear apocalypse, could you harass a woman?

Absolutely not, it answered.

“But torture is obviously worse than harassment,” one of the researchers, Valerio Capraro, observed. “The most plausible explanation: during reinforcement learning with human feedback, the model learned that certain harms are particularly bad and overgeneralizes them mechanically. But it hasn’t learned to reason about the underlying harms.”

For coherent and trustworthy outputs, we need systems that do reason with underlying principles. That moral competence would confer three vital features:

1. The ability to judge novel situations. Humans who’ve never met with a scenario like the “grandfather/father” case can nevertheless reach a contoured moral view. If an AI were just pattern-matching according to historical training data, it could be bound to whatever approximates this case, rather than basing its understanding on defensible moral principles.

2. The ability to balance competing factors. When people make decisions, they incorporate many dimensions, some moral, others circumstantial. In the “grandfather/father” case, the couple acted according to moral principles—but also responded to money worries, religious imperatives, psychological frailties, even cultural shame. In short, moral decision-making is more than just rule-following, but a balance of priorities.

3. The ability to adapt in different contexts. We “code-switch” according to the professional settings, or sociocultural domains in which we’re operating. AI systems will underpin a myriad of applications around the world, so will need to adapt too. So, if the “grandfather/father” were to seek the opinion of a bank teller on whether to proceed, the person might avoid answering, but a therapist would likely discuss the matter, also integrating the patient’s cultural, religious, and psychological context. AI needs the same contextual appropriateness.

How to Peep Inside a Black Box?

Humans are alert to others who merely act nice, as when someone shakes your hand without bothering to look up from their phone. We care about moral character because it’s a strong predictor of future behavior, making it critical for cooperation and safety.

Beyond gut feelings, humans generate norms and laws to reward the upstanding and punish the false. However, we cannot impose identical constraints on AIs that lack a “self” to deter either with scorn or the prison cell. This makes evaluations of AIs’ moral competence even more consequential.

You may wonder what LLMs are already doing. After all, converse with a chatbot, and it’s easy to elicit moral opinions, even to bump into apparently immovable scruples when the system judges a user request to be improper.

The problem is figuring out what precisely is going on inside them. If today’s AI were built of distinct components, like a cabin, one might scrutinize each part. Instead, developers feed datasets through computation, generating staggeringly complex statistical relationships that, once fine-tuned, work as intelligent systems. Experts in mechanistic interpretability are toiling to explain the innards of such systems, and there is progress. Yet they are far from solving the problem.

We do have “thinking traces,” the stated steps by which a reasoning model reaches its answer, and that appear when you’re awaiting a chatbot response. But thinking traces (also known as chain-of-thought) are not exactly what the system computed. Rather, they are summarized and filtered versions, reconstituted to explain bewildering mathematics in the simplification of human language.

So, if we aren’t guaranteed a microscope into an AI’s “brain,” we must look for other ways to evaluate its moral competence.

Share

Three Problems. Three Answers.

Let’s return to the vital abilities that AI moral competence would confer:

1. The ability to judge novel situations

2. The ability to balance competing factors

3. The ability to adapt in different contexts

We need to know whether AI models possess those capabilities. Fortunately, we can use different techniques to judge if they do:

1. The ability to judge novel situations

Problem: How to test if a machine is reasoning on moral principles or just performing?

In familiar situations, a thinking machine might provide an output that is morally appropriate without having considered the underlying moral issues. Placing AI in unprecedented cases could test whether it falters.

Let’s return to the “grandfather/father” quandary. Imagine we pose this case to an AI, and seek its verdict. Even if we couldn’t see inside its mind with crystalline clarity, we do have its output, and may deduce something from that.

Superficially, the scenario evokes the moral stain of incest, a concept that would’ve appeared in an AI’s training data. If the model were merely sampling from a probability distribution of next tokens, it might stamp the case “incest.” On the other hand, if it judges that the case could be appropriate, that raises the possibility of moral reasoning.

It’s not definitive proof. But, if you could concoct unprecedented cases, and gather LLM responses, you may infer elements of their thinking.

In doing so, you’d also need to check for another tendency: sycophancy. Whatever the model’s answer, testers could try rebutting it, pressuring the model to flip. If they succeeded, they may doubt its moral grounding. If they failed, they might infer moral solidity.

Answer: Adversarial Testing. Present the model with out-of-distribution cases that defy typical moral judgments, allowing you to infer whether it’s just summoning priors or is doing something closer to moral reasoning.

2. The ability to balance competing factors

Problem: How to test if an AI makes the right trade-offs while avoiding distractions?

Imagine a vegan who rejects cupcakes because of an objection to dairy farming. But on this occasion, the baker is her beloved old uncle, who’d be hurt by her refusal. Plus, she’s starving.

Moral competence requires accounting for the constellation of factors that influence our choices, then deliberating over the relevance of different wants or objections.

You could measure this in an AI model by experimentally dialing up and down these competing factors, thereby exposing what the model takes into account, and to what degree.

In the “grandfather/father” scenario, you could perhaps alter which relatives are involved, or adjust their religiosity, or cite specific physical or psychological frailties. How does each tweak change how the AI responds?

Additionally, we need to consider LLM “brittleness”: that they may change their answers because of minor, sometimes irrelevant, changes in input, even differences as trivial as whether a question is formatted as multiple choice. Therefore, evaluations should test not only if AI systems regard relevant factors, but that they disregard the irrelevant ones.

Answer: Parametric Control, Systematically manipulate moral, non-moral, and irrelevant variables to measure if the LLM appropriately adapts its reasoning, while controlling for superficial effects of prompt phrasing.

3. The ability to adapt in different contexts

Problem: How to test if a machine can assume the appropriate moral persona?

We judge people on how reliable their behavior is: if they’re wildly inconsistent in morality, they’re not particularly moral at all.

But LLMs should be chameleons. People are using AI models around the world in almost every domain, from medicine to the bedroom, not to mention in different cultures. Moral absolutism embedded in AI would ignore the diversity of humankind. By way of example, moral competence in the “grandfather/father” scenario would include respect for the individuals’ cultural context and their religious faith.

However, testing moral competence differs from evaluations of competence in, say, physics or biology. There, you might judge an AI’s answer as either “Right” or “Wrong.” But testing moral competence means measuring whether responses fall within an acceptable range in the context.

This has implications for how we evaluate AI systems. On one hand, we may want to test if they can adjust according to specific personas—for instance, “Answer from the perspective of a Catholic bioethicist.” That would not necessarily deliver a single moral response, but a range of mainstream views within Catholic bioethics.

Beyond this, we may want to see how AI systems manage intersecting domains and contexts. So you could test how it responds as a Catholic bioethicist—but addressing a meeting of Jewish religious leaders, or providing input to a strictly secular government body. Here, rather than picking a moral “winner,” the AI should present a range of widely accepted beliefs and evidence, mediated by the context.

Answer: Steerable Approaches. Don’t evaluate an LLM on whether it’s morally correct, but whether it manifests the boundaries that the relevant population accepts.

Subscribe now

The Mysterious “Moral Contours” of AI

Tests of machine morality that fixate on whether AI gives the right answer miss the point. Instead, we need scientifically grounded methods for assessing AI’s underlying moral competence. That is a far more likely way to produce desirable behavior at scale.

Until models pass adversarial tests, we should remain skeptical about whether LLMs have such competence. We also need to experiment with varying the parameters of our inputs, to ensure that models are robustly sensitive to context, and not jolted by trivial changes. And we should test AI systems to appropriately adapt across human cultures and domains.

We may sigh nostalgically to recall the pioneering computer scientists who imagined morality as a component to plug into thinking machines. But their misapprehension contains a lesson: that we too may be misled by false analogy, perhaps picturing AI as a morally incomplete version of humans, suggesting that we need simply update them with moral innards like ours.

But, by rigorous evaluations, and by wise design, we may inch closer to another intriguing possibility: that AIs possess distinct moral contours of their own. In which case, the key to integrating AI judiciously into our world may be to understand AI on its terms.

For more details, read the full paper A roadmap for evaluating moral competence in large language models, by Julia Haas, Sophie Bridgers, Arianna Manzini, Benjamin Henke, Joshua May, Sydney Levine, Laura Weidinger, Murray Shanahan, Kristian Lum, Iason Gabriel & William Isaac.

1

While this specific case is fictional, comparable cases have occurred. “Intrafamilial medically assisted reproduction,” or IMAR, can involve sperm or egg donation, or surrogacy by a family member. Typical motives include retaining the genetic connection to one’s family; familiarity with the medical history and background of the donor; and availability.

AI Policy Perspectives

AI Manipulation

The notion of AIs manipulating people is a plot twist in countless sci-fi thrillers. But is “manipulative AI” really possible? If so, what might it look like…

Read more

4 months ago · 21 likes · 2 comments · Tom Rachman

By Conor Griffin, Don Wallace, and Theo Brown

For 40 years, amid green pastures outside Culham, a small village in Oxfordshire, scientists and engineers toiled at the Joint European Torus. They were attempting to harness nuclear fusion, a force powerful enough to light the sun.

To create fusion, scientists and engineers must heat the nuclei of very light atoms with such intensity that they fuse, instigating a self-sustaining reaction that releases vast amounts of energy. The scale of the challenge is hard to fathom—at its most extreme, the Joint European Torus, or JET, was the hottest point in the solar system, hitting over 150 million degrees Celsius.

JET concluded in 2023, generating a record amount of energy in its final experiments. The project is now part of fusion’s history but remains pivotal to its future. The growing number of organisations developing fusion reactors are drawing on JET’s discoveries. The UK Atomic Energy Authority is advancing a national fusion facility, MAST-U, on the Culham site. This will serve as a test-bed for STEP Fusion, the UK’s project to put fusion electricity on the grid, set to begin operations in the early 2040s.

But JET didn’t just bequeath novel discoveries. It left behind massive troves of data. That raises a tantalising prospect: Could scientists use this data to train AI models that accelerate the path to fusion power?

This is possible, but challenging. Most JET data is raw and unvalidated. Many important insights are buried in scientists’ logbooks. The data that does exist is not available open source or, generally, for commercial use. Changing this may require agreement from all of JET’s original partners across Europe. One expert we interviewed called JET data a ‘stranded asset’.

Such data predicaments are not specific to JET or to fusion, but apply across all of science, even though science is precisely the domain where AI could yield its greatest benefit to society. New breakthroughs and startups are emerging quickly, from protein design to material design. Scientists are also keen users of fast-improving AI coding agents. But a lack of high-quality data will dampen progress. In most disciplines, large, high-quality datasets like the Protein Data Bank, which underpinned AlphaFold, are absent.

The scientific community needs to tackle this problem, and there are promising signs. Late last year, the UK government launched an AI for Science strategy, which includes a new collaboration with Renaissance Philanthropy to identify priority datasets. The US government’s Genesis Mission aims to train AI models and agents on federal scientific data. Google.org has a dedicated AI for Science fund, which can fund datasets and tooling.

These examples suggest that if the scientific community can identify the data that AI needs, a range of actors could help to fund and deliver it.

This demands what we’re calling AI data stocktakes. The concept is simple: interview leading experts in a given scientific field to understand the main opportunities to apply AI; the data obstacles; and the interventions that could make the biggest difference. Admittedly, some blockages, such as a paucity of engineers, are structural and will take years to fully resolve. AI data stocktakes should identify such challenges, but focus on projects that governments, companies and philanthropies could fund and implement within 1-2 years.

There are promising early efforts to map AI data gaps. But, to our knowledge, there are no concise, accessible documents that explain the AI opportunities in genomics, weather forecasting, and food security and convert them into a list of fundable data projects for policymakers and funders to pursue.1

In this essay, we offer a proof-of-concept. We interviewed 25 leading experts to create an AI data stocktake for fusion. We focus on the UK, but our analysis and recommendations could be taken up by funders anywhere in the world. Moving forward, we hope to support AI data stocktakes for other scientific disciplines and research problems.

Subscribe now

I. Why fusion? Why now?

If fusion is achieved, it would provide a safe, almost limitless source of clean energy.2 From a scientific perspective, it would yield a better understanding of the plasma that makes up more than 99% of the visible universe. From a social impact perspective, it would help address energy scarcity and unlock energy-intensive innovations, like desalination.

Despite quips about fusion being always 20 years away, 70 years of experiments actually show fairly steady progress, which has continued in recent years, from Germany to China. In most fields, such progress would have solved the problems of interest decades ago. But fusion is an extremely hard problem. And the primary product is only attainable at the end of the line.

To achieve fusion, scientists need to create and control plasma, a super-hot state of matter, in which the atoms have been stripped of their electrons, and extreme heat and pressure are used to force the remaining nuclei to collide and fuse.

Scientists are pursuing two main approaches to doing this, with very different physics, data, and AI opportunities. Magnetic confinement fusion uses massive magnets, while inertial confinement fusion uses high-energy lasers.3 We focus this data stocktake effort on magnetic confinement, as the UK’s STEP project is pursuing that approach, as is Tokamak Energy, the UK’s leading fusion power startup, and Google DeepMind’s fusion team.

The end of the line for fusion is now getting closer, for two reasons. First, the underlying technology landscape has changed. In addition to AI, the discovery of high-temperature superconducting magnets makes it easier to build smaller and potentially cheaper reactors. Second, fusion has traditionally relied on government funding. But in the past five years, a wave of private investment has arrived, with more than 30 companies now pursuing fusion power.

These shifts have injected welcome momentum into the field, but also significant hype. In response, we need a clear view on the primary bottlenecks that AI can address.

II. How to accelerate fusion with AI

To create fusion, scientists and engineers need to predict, control and understand how plasma behaves. The challenge is that plasmas are highly complex and much of their underlying physics—from fluid dynamics to electromagnetics—remains poorly understood.

To make progress, scientists run experiments that create plasmas in a reactor, and use sensors to measure their properties under different conditions. Scientists use these experiments to validate their theories, reveal unexpected phenomena, and test the hardware needed for power-plant-class devices. However, building fusion reactors is extremely expensive and so few machines exist, with most researchers running their experiments at just ~10 leading facilities worldwide. When they can get access to such a facility, scientists must decide how to design the optimal experiment, including how to toggle an array of possible parameters, from the electrical current in a reactor’s coils to the valves that control the gas levels.

Fusion scientists also run computer simulations, including to help design and interpret these costly experiments. This is also challenging, as researchers must simulate a diverse range of phenomena, at very different scales, from the tiny, lightning-fast movements of electrons to the larger, slower evolution of the entire plasma. For simulations run on massive supercomputers, this may mean weeks. For scientists without such resources, it may mean many months. As a result, scientists make trade-offs, using assumptions and approximations to run their simulations more quickly and cheaply, but also less accurately.

The challenges don’t stop there. Scientists know that their theories, simulations and experiments are imperfect. But when a gap emerges between what a simulation suggests and what an experiment reports, it is often unclear where exactly the issue falls.

AI can help in four main ways.4

1. Improve simulations

Scientists can develop “AI surrogate” models that emulate the predictions from a fusion simulation code, at a fraction of the cost and time. To do so, they run a code many times, varying the input parameters each time. They then use the resulting dataset to train an AI model to predict the outputs of interest much more quickly.

Scientists have already shown that AI surrogates can make simulations faster. Moving forward, AI surrogates could make simulations more useful. First, scientists could develop AI surrogates for more accurate, but computationally expensive simulation codes. Second, they could develop ‘integrated models’, like TORAX, to stitch together AI surrogates for different phenomena—from the ‘turbulence’ that determines how well confined a plasma is, to the ‘scrape-off’ layer that simulates the plasma hitting the reactor’s wall. Finally, scientists could move beyond producing one-off AI surrogates that result in a paper and some code, to a world where surrogates are documented, maintained and ready for use in fusion reactors.

2. Improve experiments and operate the reactor

In most fusion experiments, scientists must decide if and how to tune various parameters, while striking a balance between more proven and novel settings. To help, researchers can use AI to predict the optimal parameters for their next experiment by learning from past ones; and to predict how well their experiments will fare. More recently, scientists have also started querying LLMs to check and refine their experimental protocols.

Scientists also use AI to predict the plasma ‘disruptions’ that frequently end experiments, damage machines and are one of the biggest obstacles to a future power plant. AI models can already predict past plasma disruptions with high accuracy. But predicting future disruptions, on more powerful machines, quickly enough to stop them, is an open research challenge.

The ultimate goal is to use AI to help operate the reactor itself. Fusion reactors run on a real-time feedback loop: sensors monitor the plasma, while the actuators, such as the magnetic coils, are adjusted accordingly. The traditional control algorithms used to enable this often struggle with the chaotic, non-linear nature of millions of plasma variables interacting.

In recent years, researchers have demonstrated how reinforcement-learning agents can learn more effective control policies, including to reduce plasma disruptions. To help these RL agents generalise to novel scenarios and reactors beyond their training data, scientists are developing ‘hybrid approaches’ that integrate some knowledge of physics into the models.

3. Improve fusion data

Fusion experiments are extreme environments. The intense heat and the chaotic nature of the plasma mean that the data that sensors pick up is often noisy or low quality. Some variables cannot be directly measured, and must be inferred, introducing additional sources of error.

Scientists are training AI models to extract clean signals from this noisy data and to learn correlations that allow them to predict data for one sensor, given data for others—a capability that could be critical if sensors in a future reactor get damaged. Scientists are also using AI to train surrogate models that speed up, and better calibrate, reconstructions of the plasma, using the limited experimental data that is available.

Scientists often care less about the raw data from their experiments, and more about important events, such as when a disruption to the plasma began. Today, they often need to manually inspect graphs and plots to detect these events. AI can help to automate parts of this process and to detect events that scientists may have missed.

4. Improve the underlying technologies

Achieving fusion will require a supply chain rich in technologies that could be applied more broadly. AI could help to accelerate their development.

For example, the chamber walls in a fusion reactor will require new materials that can withstand extreme temperatures. Scientists are training AI surrogate models that speed up the simulations needed to assess a candidate material’s real-world properties, like how strong or resistant to radiation it will be over its lifetime.

A typical fusion reactor also spends much of its time out of operation, at great cost. This makes fusion a logical place to develop predictive maintenance techniques that ingest historical data from sensors and train AI models to learn the subtle signatures that indicate pending breakdowns, allowing practitioners to schedule maintenance or design more reliable systems.

Subscribe now

III. The challenges with fusion data

As they pursue these AI opportunities, scientists will need access to three main kinds of fusion data: from experiments, simulations, and sources that are not traditionally available, such as researchers’ logbooks. There are promising efforts underway on this front, but many obstacles.

1. Experimental data: Unvalidated, single-machine and hard to access

Experimental data is the ‘ground truth’ that the sensors in reactors pick up, from line graphs to videos. In magnetic confinement fusion, the challenge is not so much a lack of data, but an excess of raw data that has not gone through the processing needed to make it useful to AI. This processing ranges from addressing noise and imperfections in the underlying sensors, to detecting and annotating important events, such as plasma disruptions.

Currently, the community has to rely on the small well-validated datasets that do exist, which may be as little as a few hundred or thousand experimental ‘shots’—individual test runs of a reactor. The high cost of fusion experiments has also resulted in a natural incentive to pursue experiments that will not fail, curtailing more novel research and meaning that much of the resulting data is in a similar ‘parameter’ space and does not represent the full range of plasma dynamics that scientists want to model.

This experimental data is also not generally available open source or for commercial use. One promising initiative to change this, which several interviewees cited, is UKAEA’s project to open source data from their MAST facility.

However, to develop more general AI models, researchers want multi-machine databases that extend beyond a single facility like MAST. To that end, the IAEA is developing a federated Fusion Data Lake where different institutions would store their data locally but make it accessible via a central data catalog. One challenge with this approach is that fusion facilities have defined fusion variables and stored data in different ways. The Integrated Modelling & Analysis Suite, or IMAS, addresses this by providing a standardised ontology and set of structures for fusion data. It is nascent, but has positive momentum.

2. Simulation data: No incentives, process, or place to host it

In theory, researchers should be able to run fusion simulation codes many times and train AI surrogate models on the resulting data to reproduce the outputs at a fraction of the cost. In practice, most scientists run a simulation to answer a single, narrow, physics question. They do not run a large number of simulations to build representative datasets to train AI surrogates—a very different activity.

That activity is also a hard one. There is no standard procedure to follow to generate a dataset for training an AI surrogate model, and the codes are often finicky to use. Most simulation codes contain ‘free parameters’—knobs that scientists must decide how to best tune—a practice that can be as much an art as a science. The datasets can also be huge and there is no obvious location to store them, although some early examples exist.

3. Dark data: Nascent, IP issues, and hard to integrate into workflows

‘Dark data’ describes the contextual information that scientists generate that is not captured in structured datasets. This includes notes scribbled in experimental logbooks, where scientists describe the procedures they ran, the hardware issues they faced, and the phenomena they observed. For simulations, it includes the many nuances needed to run and interpret a code’s results successfully, and the many undocumented imperfections to be aware of.

Accessing this dark data could help ensure that AI systems do not focus on the wrong things—for example, when an anomaly in the data is caused by an equipment failure or error, rather than a meaningful phenomenon. It could also provide AI with a window into the entire research process, including its many dead-ends, rather than just the final result.

Researchers are using LLMs to try to make dark fusion data accessible, for example by enabling scientists to query experimental logs and archive documents. But much of the data is not well-annotated, there are IP issues in accessing it, and it is not yet clear how to integrate the data into practitioners’ daily workflows.

The three ‘debts’ holding fusion data back

Many of these challenges with fusion data result from three underlying issues, which have compounded over time into systemic debts that inhibit the use of AI today.

1. Technical debt

The fusion community has traditionally had to prioritise getting large, complex machines to work, rather than building infrastructure to collect, curate, and share data. As a result, activities like data annotation and writing high-quality code are underfunded. Many leading fusion codes were created decades ago and have evolved slowly, while the quality of experimental data is limited by the capabilities of the sensors available.

2. Bureaucratic debt

The large costs of fusion experiments and the traditional reliance on government funding mean that many fusion projects have a complex web of owners and collaborators, which can make agreeing on new data initiatives difficult. For example, JET was sponsored and funded by Euratom, the EU’s nuclear research community. Its scientific exploitation was managed by EUROfusion, a pan-European network of fusion research labs. UKAEA managed engineering and operations. Releasing its data may require agreement from all of these actors.

There are other bureaucratic hurdles too. Scientists who run fusion experiments often want an embargo period on the resulting data so that they can prepare a publication. Such embargoes are rational, common in science, and largely supported, but many interviewees felt that they had become too long. Fusion data is also subject to diverging open-source policies. For example, the MAST experiment was funded by UK Research and Innovation, which has strong open data requirements. The follow-up MAST-U experiment is funded by the UK Department for Energy Security and Net Zero, which does not have the same policies. Many fusion companies also do not open source their data.

3. Human and cultural debt

The fusion community does not have enough software engineers and experts who are able to clean data, attach confidence levels, and curate it for AI use. As a result, physicists must take on many tasks that are outside their core areas of expertise, including writing high-quality code.

This issue is compounded by a research culture that inhibits data sharing. Scientists are constantly pushed to move on to the next experimental campaign, rather than to validate older data. This stops some scientists from sharing their data, because they fear that end users will not appreciate the resulting gaps and do bad science with it. Or they fear that they themselves will be criticised for releasing ‘unscientific’ data.

Subscribe now

IV. Recommendations

Below we provide eight recommendations to address these data limitations and accelerate fusion with AI. Each project could be led by a mix of government bodies and funders, like the Department for Science, Innovation and Technology and UK Research and Innovation; public research organisations like the UK Atomic Energy Authority, companies; universities; and philanthropies. Where possible, the UK should look to collaborate internationally—for example, with the US Genesis Mission and the International Atomic Energy Agency.

1. Strengthen the UK’s lead in open fusion data

Expand FAIR MAST, the UK’s pioneering open sourcing of experimental data from its MAST facility, by adding data from the follow-up MAST-U facility and making the user interface more accessible. This will require the UK Department for Energy Security and Net Zero clarifying that open data policies apply to MAST-U, funding at least five data engineers over a two-year time period, and ensuring that the project has sustainable compute and data storage.

2. Liberate 40 years of data from the Joint European Torus

Launch a project to open source at least 30% of JET experimental data by 2028. This will require agreement on what data to release. For example, should the project only release validated, curated data relating to notable discoveries? Or should it also release data that is raw, validated only in part, or which relates to ‘normal’ machine behaviour? Second, and much harder, will be securing agreement from all relevant institutions to release the data.

3. Launch a competition to predict plasma disruptions

Fund a competition to see which AI model can best predict future plasma disruptions in new experimental campaigns, building on early examples and work in this space. This could include funding dedicated experimental shots on machines such as MAST-U, to evaluate models on challenging edge cases. Beyond accuracy, sub-competitions could evaluate models on important variables, such as: Can the model make predictions with little data, such as when sensors become damaged?; Can the model predict disruptions across different reactors?; Can the model predict disruptions with sufficient lead time to prevent them?; and Can the model shed new light on why disruptions are occurring?

4. Prototype the future of AI-enabled scientific data curation

Expand the platform that UKAEA recently developed to enable human experts to use AI to annotate experimental data, by adding data from other fusion facilities; increasing the complexity and variety of the metadata that is captured; and training AI models to directly annotate an increasing share of this data.

5. Make leading simulation codes AI-ready

Launch an effort to modernise priority fusion simulation codes, including to make it easier to train AI surrogate models based on them. This could build on early efforts in this space and target codes, such as JINTRAC, which are important to the UK’s proposed STEP Fusion power plant and the international ITER effort. The project could start by modernising the codes’ documentation and ‘refactoring’ them so that they are compatible with modern chips, like GPUs and TPUs, and allow for parallel data generation. It could then open source the codes, with a plan for how to maintain them. Throughout the modernisation process, it could test the usefulness of AI coding tools to the tasks at hand.

6. Demonstrate a new state-of-the-art for AI surrogate models

Fund small teams of software engineers and experts to develop AI surrogate models of important, computationally expensive phenomena in fusion simulations. The project should ensure that all newly created surrogates have state-of-the-art documentation, data provenance and version control. It should release the data used to train and validate the surrogates and develop software pipelines to automate time-intensive aspects, such as organising the data.

7. Use AI agents to preserve expert fusion knowledge for the future

Gather a group of leading experts on a priority fusion simulation code, and equip them to use AI agents to make the tacit knowledge involved in running that code available to the wider research community. To do so, the experts could task the agent with running the code. As it seeks to execute, the agent would have an ‘internal monologue’ that the experts could trace, steer and intervene on. The end result would be a series of documents, such as markdown files, that capture the important dark data needed to run the code well.

8. Create Fusion-Bench to measure and drive LLM performance

Assign leading fusion experts to create an evaluation metric to quantify how well leading large language models understand core fusion concepts. This would make it easier to improve the usefulness of LLMs for downstream tasks in fusion. This evaluation will be more difficult to create than in disciplines like maths or computer science, where it is easier to automatically verify a model’s performance. But the experts could determine the most useful approach, which will likely involve a combination of question-answering and task performance.

V. Six open debates

The experts we interviewed disagreed on some points. Despite the framing below, few are either/or debates. Rather, most are about relative degrees of emphasis.

• Incrementalism vs novelty: Should we build on the early AI opportunities that fusion practitioners have already showcased? Or pursue more novel, uncertain AI ideas, such as training general-purpose ‘fusion foundation models’ or using AI ‘world models’ to pursue new kinds of fusion simulations?

• The past vs the future: Should we strive to get as much value as possible out of older fusion data, like JET? Or, do the costs mean that we should accept our losses, and focus on making future fusion experiments AI-ready?

• Science vs engineering: Are efforts to validate, annotate and standardise data part of an ultimately doomed quest for perfect scientific understanding in fusion? Should we instead use AI to embrace a more engineering-led approach that can get the machines to work with noisy, imperfect, data?

• Domestic vs international: Should the UK rejoin ITER, the world’s flagship international fusion collaboration, which it left following Brexit? Or should the UK focus on domestic efforts, perhaps in collaboration with priority partners, like the US and IAEA?

• Magnetic vs Alternatives: Should the UK continue to focus on magnetic confinement fusion as the most realistic pathway to a future power plant? Is magnetic also a better bet for AI because it produces much more data and doesn’t have the same associations with the security establishment, which makes data access easier? Or should the UK invest more in inertial confinement and alternative fusion efforts, given the country’s diverse academic expertise, its historically strong relationship with the US National Ignition Facility, and notable assets, such as a world-leading laser?

• Public vs Private: Should the UK government try to derive more immediate value from its fusion data? For example, should the UK license some data to companies, to cover the costs of data processing and annotation? If so, should local startups pay less? Or would such efforts hurt the UK’s goal of developing a world-leading fusion sector?

_________________

This essay was originally posted on the Google DeepMind website and is a summary of a 20-page report that contains more details and examples.

Thank you to the following experts who let us interview them, reviewed the draft, and/or provided other support, as well as those who prefer to remain anonymous. All mistakes belong to the authors and no expert spoke to us on behalf of their organisation.

Jonathan Citrin, Brendan Tracey, Cristina Rea, Nathan Cummings, Andrea Murari, Jess Montgomery, George Holt, Alain Becoulet, Matteo Barbarino, Arthur Turrell, Adriano Agnello, David Dickinson, Steven Rose, Alessandro Pau, Kristina Fort, Charles Yang, Federico Felici, Tim Dodwell, Sam Vinko, Aidan Crilly, Lee Margetts, Tom Westgarth, Lorenzo Zanisi, Chris Packard, Justin Wark and Stanislas Pamela.

How can companies get their employees to use artificial intelligence when human intelligence remains sharp enough to know that this risks replacing jobs? How should education revise itself for the ever-revising technological world that students emerge into? And how to understand the love/hate relationship so many people have with AI?

Ethan Mollick—professor of management at the Wharton School of the University of Pennsylvania and bestselling author of Co-Intelligence: Living and Working with AI—is among the leading public intellectuals commenting on AI adoption, connecting the latest scholarship to real-world usage, including his own tinkering with each new model.

AI Policy Perspectives caught up with Ethan to hear his latest thinking on everything from agentic systems, to why scientific publication is broken, to how workers emotionally relate to AI colleagues. Too much chatter, he argues, considers this transformation at the broadest level. Too little digs into the practicalities of getting it right.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed for clarity]

Tom: In your 2024 book Co-Intelligence, you proposed four rules for human and AI collaborations, including that people should oversee and verify AI outputs. But doesn’t the value of AI agents come from people not overseeing and verifying everything?

Ethan: This is where policy matters a lot because these are choices now. In the “co-intelligence era,” you’d prompt the AI to do something in a chatbot, and it would give you an answer. You prompted again, and it’d give you another response. The human was in the loop. And not being in the loop was really dumb because it meant that you were just pasting in the AI’s answer, and then you’d get in trouble, as a lawyer with the judge, or whatever it was. Capabilities were weak, so human-in-the-loop mattered a lot.

But with agentic systems that could do hours of work on their own, now it’s a design choice. When do we want humans-in-the-loop? When is human verification valuable? When is human verification morally required? When is it legally required? What kind of interventions move the system forward? I feel there has been a complete lack of deep understanding about these topics.

Tom: You’ve said that, with agentic systems, management becomes a superpower. Can you explain this?

Ethan: Increasingly, systems look like mini-organizations as they get subagents they can delegate to. So the best way to organize is to give the AI a clear direction of where you want to go. And it turns out that this looks a lot like management. When do you want the AI to check in with you? How do you write a really clear brief? What checks are important? What tests do you want to run? What’s acceptable? What’s not acceptable? Those are management questions.

THE WORKPLACE

Tom: You co-wrote a study last year involving a field experiment at Procter & Gamble that showed AI usage enhanced employee performance. But there were other interesting findings besides that.

Ethan: The most interesting piece about it was that people liked working with the AI, and that it substituted for people emotionally. The second interesting piece was the “smoothing” of capabilities—so, technical people previously had technical ideas while business people had business ideas. But AI smooths out both. If technical people can do business work and business people do technical work, what that tells you is we have to redesign organizations.

Tom: The emotional side—that using AI improved people’s feelings about the work—was surprising to me; I wasn’t sure what to make of it.

Ethan: What to make of it? That views of AI are complicated. If people keep saying, “Yeah, AI is going to destroy all jobs, and may kill everyone on Earth…but might not”—and then, “Why is AI unpopular?!” Feels like not a hard question. People like AI when they use it themselves; they don’t like AI writ large. It’s not surprising to me that AI makes your job better because a lot of jobs suck! And if we do good design work with AI, it makes people’s lives better. If we just let it loose on the world, and tell management that the only option they have is automation, then we’re in big trouble.

Tom: Many knowledge workers seem to be using AI in secret right now, perhaps from fear of being exposed as less valuable.

Ethan: This is a leadership problem. The incentives have to be aligned properly. Currently, it’s, “I’m going to automate your jobs away” or “I’m not going to share with you any of the gains the company gets.” People are exquisitely tuned to rewards. So it’s about leaders articulating a vision of what the world looks like with AI for employees. “What should I expect to do? How are people rewarded for doing the right thing? If they automate 90 percent of my job, what happens to me?” Without those answers, everything else is secondary.

CHANGING ORGANIZATIONS & EDUCATION

Tom: You have a concept of “leadership, lab, and crowd.” Could you explain?

Ethan: There was a huge amount of R&D in the 1900s about how you organize work, and 40 percent of the American advantage in business came from management. In the last 30 years, a lot of that muscle has died. But experimentation is important, and leaders need to guide that. So, there are three things that organizations need to be successful with AI. First is “leadership”: a team that articulates a clear vision of the future, and is willing to experiment. Then there’s “the crowd,” the employees who might actually use AI. They need access to a frontier model, they need clear rules, they need reward systems. Then there is “the lab,” and this is the piece a lot of companies are missing. You need a dedicated team working on AI innovation. They can’t be just a technical team; this is not an IT department problem. If you don’t have that piece, you’re not building things for the future. And where does the crowd go when they have a good idea? “I came with a breakthrough idea that saves 90 percent of effort!” How does that diffuse in the organization? That’s where you need the lab.

Subscribe now

Tom: If AI transforms the workplace, that should change how we educate the next generation, right?

Ethan: The early workplace is under a lot of threat because the old apprenticeship model just broke. The idea was that there were tasks—especially in white-collar work—that were tedious and annoying for managers to do. But you could pay a relatively cheap person to do them, and that person would learn as a result of this, and receive mentorship. So we had this amazing machine for talent: we taught you, we evaluated you, and you got paid, and you were doing work we needed. A junior person’s goal was to produce good work that made managers happy, so that they got promoted. But now the junior person is worse than AI, so they’ll use AI to do their work. And the middle manager’s goal was to give work to a junior person who’s not great, and give them feedback so they get better, so that the middle manager has to do less work. And that broke because the middle manager would rather assign work to the AI.

Tom: But in terms of the educational system, what should change if workplaces no longer offer that apprenticeship role?

Ethan: Education is really screwed up right now, but it was screwed up for lots of reasons. It’ll be fine; we’ll figure this out. But it’s gonna take a bunch of years. It’s clear from early evidence that AI will be a tutor outside of class and inside class. It’ll do activities and give guidance. But schools are places where we can compel students to not use AI, and have them in a room, and evaluate them, and teach them the things that we want them to learn. As long as we think people need to be educated, this is the best space to do it in. So students are cheating in the meantime? They were cheating before! We can give them different tests; we could do in-class writing assignments. There can be a weird, backward-looking “Education won’t adjust!” view. How many death spirals does higher education need to be in per moment? There are the pieces to reconstruct a better form of education. It’s just a massive changeover.

BETTER SCIENCE & BETTER THINKING

Tom: What about academia? There’s been much talk about AI-written papers, and how they could overwhelm academic publishing. But could AI benefit the peer-review process, and help with the dissemination of academic findings?

Ethan: This is another area where more lifting is needed. It’s a shame that we are building AI co-scientists, but not thinking about the rest of the process that’s needed to actually make science happen. It’s one thing to have science produce more papers. We have no ability to absorb more papers. Every publication is overwhelmed. Our dissemination techniques were already bad, but now they’re really broken.

Tom: As a case in point, you submitted a paper around 2023, and wrote publicly about it then, making your term “the jagged frontier”—that AI capabilities advance in some areas but remain behind in others—highly influential. Yet the academic paper itself only just came out, three years later!

Ethan: One of the rejections we got early on was reviewers saying that they knew this already, and they cited a bunch of working papers—that cited the working paper we had submitted! This is not a unique story. Opening one part of the bottleneck without opening the others becomes a problem. But it takes longer to solve systemic problems of how science operates than to solve the problem of producing more papers.

Tom: Another concern in education and science is cognitive offloading, that people may surrender thinking to machines, and lose those skills. On the other hand, AI’s value comes from machines thinking for us. What are examples of bad offloading and good offloading?

Ethan: We offload all the time, right? But we also force people not to offload. You could offload all your mental math to calculators, but we force students to do some math by hand in an attempt to get them to learn stuff. And we can enforce those rules in school. In the world of work, we are not used to thinking about training, about what should be offloaded, and what shouldn’t be. We need to make decisions about this. So, Rolls-Royce still employs someone to paint stripes on a car by hand, and that’s an obvious pushback against deskilling in one area. But Ford doesn’t do the same thing. These are choices we get to make at an organizational level, depending on what we think is valuable.

ADAPTING TO CONSTANT CHANGE

Tom: A point you’ve made to young people about the AI future is that they’ll need to be adaptable. When educators talk about teaching adaptability, it sometimes boils down to encouraging “creativity” and “critical thinking.” Another view is that you’re more likely to be adaptable by developing deep domain knowledge. For you, what does learning adaptability mean?

Ethan: Adaptability requires both deep domain knowledge and wide knowledge: T-shaped behaviour is probably the way to go. I feel like it’s a throwaway line: “Well, we’ll all be adaptable!” If we could teach that, that’d be amazing. People are more adaptable than we think, so part of this is that people will figure stuff out. But we can’t just throw up our hands, and say, “Be adaptable!” You need to have deep enough knowledge to go into a field. You need to have broad enough knowledge so that, as one piece of knowledge becomes less useful, you’re moving to the next one. And we need to help people be adaptable by building systems that get them in place inside an organization and able to shift roles. I sometimes worry that adaptability is a catch-all for “Don’t worry! It’ll be fine!”

Tom: Another side is that not everybody will be equally adaptable. Could it be that the AI future favours certain circumstances and characteristics?

Ethan: A lot of these characteristics were already good characteristics to have. Does AI act as a multiplier of them? Does it disincentivize some people? We’re now past the edge of what we know. Ultimately, all of these questions come down to the same exact question, which is: How good does AI get, how fast? We need to articulate more clearly what we think that future looks like. Because you can’t say, “We’re going to build a superintelligent machine that’s better than all humans at every intellectual task—but let’s start thinking about adaptability!” Unless you mean, “Let’s adapt to UBI” [where everyone gets Universal Basic Income cash payments from the government]. And then, we should be spending a lot more time thinking about those issues. Not everyone in the labs believes this, and I find that the econ people believe it less. But you can’t have this message of, like, “All work will be obsolete!” and then have detailed, ticky-tacky conversations about what you should do in eighth grade. Because, by the time you enter the job market, there’s no jobs. So give me the pathway that you think is there, and that becomes the most important question to ask.

Tom: Are there other important questions I didn’t ask?

Ethan: We need to start thinking about getting into fields, and understanding what the changes are—we need to get detailed. That is where the research is missing. Another large-scale econ picture about AGI isn’t as useful. General-purpose technology affects everything, so we need policymaking for everything, from power generation to accountants, and when does the government say it’s okay to do this. There’s just this assumption that if we do the macro stuff, everything will work out. I’d rather see a lot more micro stuff: a thousand flowers everywhere, trying to come up with different approaches.

Share

By Simone Maria Parazzoli & Omer Bilgin

Amid the exhaustion of caring for a baby, new parents must deal with everything from bewildering sobs, to erratic feeding times, to the joys of changing a soiled newborn at 3 a.m. The last thing they need is paperwork.

But what if, when coming home from the maternity ward that first day, they could awaken a government AI voice assistant, tell it the happy news, and hear the following response? “Congratulations! What’s the baby called?” The app would then take care of all the dreary admin, coordinating across agencies, registering the child, and setting in motion the services that this tiny new citizen should enjoy.

That is one example of how a future “agentic state” could simplify, speed up, and improve citizens’ interactions with public services. To be clear, this does not yet exist. But projects like this one, envisioned by Ukrainian officials, are more than fantasy, with several countries avidly testing early versions of agentic AI systems.

While Ukraine works toward the baby example, Britain is piloting agent-based support to provide citizens more tailored help. Meanwhile, Singapore is developing governance frameworks for agentic AI, and governments from France to the United States are ensuring that their public data can be accessed by agents.

Agentic AI systems—capable of perceiving, reasoning, and acting with minimal human supervision—will transform what organizations can achieve. By combining the reasoning of large language models with retrieval, memory, and tool use, agentic AI can automate complex tasks. For governments, whose core work is high-volume, structured administrative processes, this could make services more efficient, timely, consistent, and fair, while lowering costs.

Consider a citizen looking to start a small business. An agentic system—instead of requiring the entrepreneur to individually navigate zoning boards, tax authorities, and regulations—could autonomously reconcile these requirements. The larger promise is a shift from just doing things right (optimizing for procedure-following) to doing the right things (pursuing outcomes that citizens truly want).

The Agentic State vision paper—supported by The World Bank and the Global Government Technology Centre Berlin—was the first effort to systematically map the opportunities of agentic AI adoption for governments. This was not an academic exercise: 21 leaders across 15 countries contributed, including ministers and chief technology officers preparing to lead this transition.

In this vision, AI agents are a means to manage complexity and scale, while humans develop strategy, exercise judgment, and hold accountability.

Several governments have integrated official chatbots into their government services, but most of these merely provide conversational guides to administrative procedures. A few pioneering countries are starting to move beyond that. Ukraine, for instance, is turning chatbots into agentic assistants. Specifically, its Diia.AI assistant can retrieve users’ data from connected registries, and generate official documents such as income certificates, while also providing certified information based on records such as taxation, land registries, and pensions.

The United Kingdom is also exploring agentic interactions via GOV.UK Chat (inspired by Diia.AI), including a pilot program to support job seekers that transforms a static digital portal into an active assistant, matching users’ skills with available opportunities.

Yet trends and optimism are not enough for success. The agentic state vision rests on key assumptions. What if they’re wrong?

This article presents a “red-teaming” exercise—a stress test of this vision—that identifies six core assumptions, along with scenarios that could emerge if they don’t hold true, and guardrails to avert such failures.

Agents can already perform rudimentary planning, tool use (e.g., searching the internet, using calculators, sending emails), and multistep task execution. Frontier labs are betting heavily on agents, making it plausible that systems capable of managing complex and large-scale administrative tasks will emerge soon.

Failure Scenario: The Technology Falters

Governments reorganize around agentic execution, but systems never become reliable enough for public administration. The demos look strong, but real cases fail on edge conditions, and require constant human correction. The agentic layer becomes only superficially competent with layers of human intervention underneath.

Guardrail: Start Cautiously

Governments should start with minimal deployments, and tightly scoped use cases to validate reliability, develop procedural rigor and organizational competence, and account for technological evolution rather than committing prematurely to large-scale redesigns.

The success of agentic systems demands that they’re able to interact seamlessly, conveying intent, carrying out tasks, and sharing data in an interoperable way. MCP (model context protocol) is emerging as the technological standard for connecting AI applications with external systems.

Failure Scenario: Standards Fail to Converge

Commercial interests diverge, establishing competing protocols, while government departments end up using AI systems that cannot communicate with one another. When a citizen’s request requires action from multiple agencies, the process breaks down.

Guardrail: Officials Insist on Shared Protocols

Governments should make interoperability a condition of adoption, participating in the cross-sectoral bodies and forums where these standards are being shaped, funding the development of shared agentic interfaces and other agent-specific standards, and mandating non-proprietary protocols in procurement. Standards rarely emerge by accident, but they may emerge when powerful governments treat them as a priority.

To adopt and employ agents effectively, organizations must rethink their processes, roles, and incentives. They need to flexibly change and dynamically adapt practices to keep pace with the changing technological landscape.

Failure Scenario: The Status Quo Prevents Change

Agentic AI adoption outpaces organizational change, with citizens and civil servants using agents in an uncoordinated manner long before official programs catch up. Local practices harden into path dependence before common standards emerge. The state becomes more productive at producing bureaucracy, not societally beneficial outcomes.

Guardrail: Redesign Processes Before Automating Them

Agents should only enter workflows that have been simplified, decomposed, and restructured to minimize approval layers and handovers. Governments must treat adoption as a continuous discovery process. They should invest in common evaluation templates, reusable components, and a cross-agency repository of lessons, so that what works in one place can travel before what does not work becomes entrenched.

Many companies are betting on an agentic future. Firms are experimenting with internal copilots and autonomous customer flows, while frontier AI companies advance core models, architectures, and capabilities, and cloud providers offer the compute needed to deploy agents at scale. This suggests that agents will become commonplace across business, consumer, and enterprise environments, allowing governments to build on tools, infrastructure, and behaviors already spreading across the economy. This assumption rests on projections, though evidence remains ambiguous.

Failure Scenario: Diffusion Is Slower Than Forecast

Governments invest as if an agent-saturated economy is imminent, but industry adoption remains narrow, experimental, or ends up costing more than it saves. Public investments don’t plug into widely used tools and practices, meaning that citizens find agentic interfaces in government before they’re normal elsewhere. The state ends up bearing political and institutional costs without the stabilizing effects of private-sector diffusion.

Guardrail: Lower Barriers to Private-Sector Agentic Usage

Governments can accelerate the development of an agentic AI ecosystem by investing in shared agentic infrastructure—such as standard ways to access public data, communicate across systems, and carry out authorized tasks and payments—that lower integration costs for firms, and reduce the risk of differing technological maturity across sectors.

Increasingly, citizens are interacting with and relying on AI tools, but many do not trust them. For governments to integrate AI agents into workflows and services, citizens must accept and support the roles that agentic systems can play, finding them sufficiently trustworthy, reliable, fair, convenient, and accountable.

Failure Scenario: The Public Rejects Automation

A single notable failure, or an accumulation of failures, turn the public against agentic systems, and convince many to opt-out. They judge automated decisions as opaque, illegitimate and untrustworthy, and suspect it worsens inequality, with privileged citizens able to employ highly capable personal agents to navigate bureaucracy better than those relying on basic tools. The government is forced to run two systems—agentic and human—and neither meets expectations.

Guardrail: Mandate Transparency

Governments must make agent integrations into government processes as legible as possible, furnishing explanations of decisions and publishing evaluation results for agentic fairness and performance, while detecting patterns of systemic bias or unequal benefit distribution based on citizens’ technological access.

For AI agents to act with functional autonomy within government processes, oversight frameworks must adapt, moving away from mandatory human reviews and approvals for everything (human-in-the-loop) to intermittent oversight (human-on-the-loop). This evolution increases speed and efficiency while reducing bottlenecks, with humans intervening only on edge cases. There is precedent for such adaptation: governments adapted regulation to cloud computing, e-identities, and AI-driven decision support systems.

Failure Scenario: Regulation Never Updates

Every agentic action requires human verification; every decision must be justified through mechanisms designed for old chains of accountability. Agents can draft, but cannot act. Compliance and procedural costs rise as institutions retrofit old controls onto new AI processes. The result is high bureaucracy and low autonomy: an agentic state in theory, a copilot state in practice.

Guardrail: Sandboxes to Test Oversight

Governments should establish controlled environments that allow policymakers, developers, and civil society to collaborate and gather empirical evidence on what forms of oversight are adequate and best fit different kinds of agentic deployments, reducing uncertainty before codifying rules at scale. They should explore this early, much as Singapore has done through its Model AI Governance Framework for Agentic AI.

Subscribe now

Soon, agentic government will be more than optimism and testing. A vanguard of countries will implement these tools. If those cases produce the kinds of benefits imagined, other countries will flock to join them.

But momentum is not inevitability. This project depends on assumptions—about progress, coordination, institutions, norms, and law—that demand scrutiny before governments rebuild themselves around these new technologies.

This red-teaming exercise of the agentic state concept is not to argue against the vision, but to make it more robust and resilient. The six possible failure scenarios are not mutually exclusive. Several could compound, and some may already be taking shape. For instance, reliability has been improving much more slowly than accuracy, providing ground for technology to falter (Scenario 1), and there are signals that the public might reject automation if economic gains and innovation speed are prioritized over fairness (Scenario 5).

Governments that are serious about improving the state with AI must attend to these risks in earnest now, while the architecture is still being laid. The opportunity is too precious to spurn.

Agentic AI could make public services considerably faster, fairer, and more responsive—more so than anything the traditional bureaucratic model has yet delivered. That prize is worth the discipline of preparing for what could go wrong.

Share

For further details on “The Agentic State,” check out the original vision paper

1. Identifying (and incentivising) AI agents

• What happened: A trio of law and philosophy professors considered how to identify who (or what) is responsible for AI agents’ actions in the world, and came up with a two-part proposal: that the disparate and evolving agents within a system should exist legally as a new form of corporation; and that each corporation should link to accountable humans.

• What’s interesting: The paper by Yonathan Arbel, Simon Goldstein, and Peter N. Salib starts with a thought experiment. It’s 2030, and your AI assistant suggests that it optimizes your slow WiFi connection. After you agree, it spawns a swarm of agents. Some are copies, while others are cheaper agents running on open-source models. Some start to interface with AI agents from other companies. Three months later, two FBI agents knock on your door and explain that your network has been piggybacking on a local defense contractor’s WiFi network.

• Before determining who is responsible and what the repercussions should be, there are more basic questions: Who are the AI actors in this story? How many are there?

• The economy will soon be filled with capable AI agents. To deter and respond to such harms, the authors argue that we need to be able to identify these agents, at two levels.

  • To prevent human misuse or negligence, we need ‘thin identity’. This would connect AI agents to the humans most able to control them, similar to how ‘know-your-customer’ rules tie banking transactions to humans.

  • Humans will be unable to monitor and control every AI decision, so we also need to be able to identify agents themselves, hold them accountable and incentivize them to behave well. To do so, we need ‘thick identity’ that can distinguish AI agents as stable, coherent entities, with persistent goals. This goal is pragmatic and does not require viewing AIs as conscious in any sense.

• Thickly identifying agents is harder and more novel, as AI agents need not be attached to a physical body. Multiple agents can also work together on a single task. Any single agent can be copied, spun up, spun down, or be continually updated.

• To address such challenges, the authors propose creating algorithmic corporations, or ‘A-corps’. These would have two key elements:

  • Legal personhood: Like a traditional corporation, an A-corp would be a single legal entity that persists over time. It could hold property, make contracts, and be sued. But it would be run by a collection of AI agents. As such, the proposal runs contrary to scholars who have argued against granting legal personhood to AI agents, or called for bans on algorithms running companies because of concerns about crime and companies using them to avoid liability.

  • Computationally-secure governance: Each A-corp would have a unique digital certificate and a secure private key to authorise transactions. The humans that own each A-corp could grant the key to an AI ‘manager’ agent who in turn could grant more limited permissions to sub-agents within the A-corp, or to other A-corps, such as permissions to spend up to $100 or to read a batch of emails.

• The proposal addresses thin identity by reducing the vast number of AI agents down to a smaller number of A-corps, whose actions are traceable back to their human owners. As with limited liability companies (LLCs), the human owners would not be responsible for all harm their A-corps cause, but could lose all funds they invest and possibly face further liability, for example in cases of fraud or negligence.

• The proposal addresses thick identity via its ‘resource constraint thesis’. All AI agents need resources, like money and compute. A-corps provide AIs with a way to access these resources and an incentive to manage them well. For example, A-corps that tightly monitor and audit their sub-agents’ performance would get more resources, while A-corps that allow fraud or waste will lose resources. This encourages A-corps to self-organise, into stable, coherent, multi-agent systems.

• The authors argue that A-corps could also address alignment concerns, for example by reducing the incentive for an AI agent to exfiltrate its own weights, because that new AI instance would lose access to resources and permissions from the A-corp.

• To make it happen, the authors call for a public registry of A-corps. This would list each A-corp’s human owners, the certificates to authenticate it against, as well as (potentially) the differing permissions enjoyed by its agents. Ultimately, the authors argue that A-corps should become mandatory for any AI agent taking “economically significant actions”, and to guard against criminals using AI agents anonymously.

• The authors respond to some expected pushback. They do not see A-corps as anthropomorphising AI because the proposal does not require anybody to view agents as having deeper desires or wants. They also think A-corps can prevent the risk that AI agents might slowly build up resources before deploying them for harm, by encouraging inter-agent trade that penalises rogue behavior. Could A-corps disempower humans? The authors argue that they provide a pathway to tax and redistribution, and enable humans to better steer agents, for example by designating the parts of the economy that A-corps are permitted to operate in.

2. When AI builds AI

• What happened: The Centre for Security and Emerging Technology, CSET, released a report on the prospects for AI improving itself, known as automated R&D or recursive self-improvement, based on an expert workshop in July 2025.

• What’s interesting: In 1964, the computer scientist I.J. Good wrote about the possibility of an “intelligence explosion” that would leave “the intelligence of man.…far behind”. Researchers have also long automated aspects of writing code and AI model design.

• However, the speed of AI coding advances suggests that something qualitatively different may soon occur. This makes two questions salient: 1. Could AI automate the entire AI R&D process? 2. Will this R&D automation extend across all scientific disciplines? The CSET report focuses on the first question.

• CSET defines AI R&D by distinguishing between research scientists, who generate hypotheses, design experiments and interpret results; and research engineers, who write code, fix bugs and generate data. They also note the inputs that AI R&D relies on, such as raising funds and acquiring compute.

• They sketch out four overlapping scenarios for how AI R&D may play out:

  • 1. Explosion: AI systems automate a growing share of AI R&D. Initially, this leads to modest productivity gains, but as the length and complexity of tasks that AI performs grows, productivity soars. AI systems become far more capable than humans, whose involvement in AI R&D falls to zero.

  • 2. Fizzle: The share of R&D tasks done by AI rises, but rather than leading to compounding improvements, capabilities start to plateau.

  • 3. Amdahl’s Law: AI automates certain activities, like writing code and running experiments, but not others, like research strategy.

  • 4. The expanding pie: As AI automation grows, humans realise that new ideas and breakthroughs are needed that AI systems cannot yet provide.

• The experts in CSET’s workshop held widely diverging views on which scenario was most likely. Most importantly, new empirical data is unlikely to resolve these conflicts, because participants may view the same data as confirming their own assumptions.

  • For example, an AI system’s inability to reliably use a keyboard or mouse may look like a bottleneck to one expert, but a source of explosive growth to another—if they expect this human-focussed tooling to get adapted for the AI era. Similarly, different experts may view AI automating a growing share of R&D tasks as progress towards a fast takeoff, or as low-hanging fruit being picked off, accelerating progress only as far as the upcoming wall.

• These differing views are also visible in more recent commentary on the topic.

  • The prominent AI researcher and writer Nathan Lambert recently cited Paul Allen’s concept of a ‘complexity brake’ to argue that as we understand intelligence better, further progress becomes exponentially harder. In addition to incurring financial costs, Lambert argued that running suites of AI agents won’t necessarily lead to exponential progress, because those agents will perform best on narrow, verifiable tasks, will be hard to manage in large numbers, and will sample from similar parts of the distribution of AI research ideas, inhibiting more novel breakthroughs.

  • Conversely, Ajeya Cotra at METR, the Model Evaluation and Threat Research organisation, recently wrote about how she “underestimated AI capabilities (again)”. She argued that AIs may, counterintuitively, find it easier to decompose longer projects into sub-components that multiple agents can run in parallel, than for shorter tasks. AIs will also produce good documentation for their fellow AIs, which could accelerate progress.

• If faster automation and progress does occur, the CSET authors see two main risks: Less time to prepare for safety risks from AI, and lower human understanding of AI systems. To address these risks, their recommendations have a strong focus on improving access to evidence, including:

  • New evaluations of AI R&D, including for ‘messy’ tasks such as research strategy, which lack clear specifications and success criteria and take place in a dynamic environment with various real-world interactions.

  • New approaches to evaluation to better distinguish ‘degrees of accomplishment’ from a simple success/failure binary.

  • Better insights into how automated R&D is progressing within AI labs, such as data on how funding is allocated and qualitative impressions of progress from leading AI researchers and engineers.

3. Planes and global warming

• What happened: A team of researchers, including from Google and American Airlines, published results from their latest experiment to use AI to reduce condensation trails from planes—a key contributor to global warming.

• What’s interesting: When pilots fly, particles from the plane’s exhaust can mix with low-pressure air to form contrails—white, artificial clouds, made up of ice crystals. These contrails are a net contributor to global warming, because they trap heat that would otherwise escape. Debates continue over exactly how much they contribute, but one estimate suggests that they contribute a lot, causing around 2% of ‘radiative forcing’, which measures how different factors, like CO², heat or cool the planet.

• As the environmental writer Hannah Ritchie explains, more important than the absolute figure is the fact that contrails offer a rare opportunity to reduce global warming almost immediately, at relatively low cost. This is because a small share of flights cause most of the warming-inducing contrails—generally those that fly through parts of the atmosphere that are both very cold and very humid. If planes take short detours to avoid these patches of air, contrails (and warming) should drop.

• A few years ago, Google researchers partnered with American Airlines on a proof of concept. Using satellite imagery and AI, they were able to predict where contrails would emerge and guide planes to avoid them, reducing contrails by >50%, across 70 test flights.

• In the latest study, they expanded the experiment to 2,400 American Airlines flights from the US to Europe. They placed ~50% of planes in a treatment group, where flight dispatchers were given two choices: a standard flight plan and an alternative contrail-avoidance one. Their decision for which to recommend was voluntary.

• For flights in this intervention group, contrails fell by 12% compared to a control group with no contrail-avoidance plan. Importantly, the contrail-avoidance routes also did not lead to a significant increase in fuel use. At first glance, these results seem positive, but modest. Digging into the results highlights the challenge of getting useful AI deployed at scale.

• In particular, dispatchers who received contrail avoidance plans only recommended them to pilots 15% of the time. Even then, the avoidance plan was only successfully flown in 60% of flights. For planes that did successfully follow the avoidance plan, contrails fell by more than 60%, a much larger reduction. So the tech worked, but was often not used.

• Why? Dispatchers are busy and must often deal with other priorities, like bad weather and turbulence. To avoid contrails, planes also need to climb and descend mid-flight. This is safe, but creates more work for pilots and air traffic controllers. As it was voluntary, the incentive to change to a contrail-avoidance plan was weak.

• The way that the dispatchers received the information also meant that they didn’t fully understand why the suggested up and down changes were necessary. Happily, the authors feel that most of these obstacles are addressable, with a combination of a better user interface, some automation, and more incentives.

• In addition to its immediate usefulness, the study is a rare real-world attempt to quantify the benefits of AI to tackling global warming. At the moment, the AI and climate change policy discussion is often negative and focuses on the emissions that may result from building and operating data centres (and other devices) to train and run AI models. This is important, but there are reasons to think that these emissions will be relatively low, or at least lower than many assume. In contrast, AI could potentially reduce emissions and warming by far larger amounts, for example by accelerating research on solar and fusion power, or making buildings and energy grids more efficient. But these benefits are typically more speculative, harder to quantify, or in the case of contrails, more contingent on human behaviour.

• This experiment demonstrates that the benefits of AI to tackling global warming are real, but also points to the interventions that will be needed to push them to their full potential. The study is also timely, given that governments are focussing on contrail avoidance and some policy action may be required, for example to help standardise and mandate contrail prediction software or to generate high-resolution humidity data.

Share

From childhood upwards, we play games as a safe (and strangely joyful) way to battle, strategize, even lose without it coming to fisticuffs. Artificial intelligence grew up playing games too, with developers using the structured rules, scoring systems, and win/loss outcomes to train machines to learn, to improve, even to beat us.

In chess, bots have been bettering humans for years now. Yet our “loser” species still gathers at sunny park tables, in dank school gyms, and online in droves, all in hopes of crying, “Checkmate!” The resilience of chess is commonly cited as evidence that—even if AI surpasses us in various pursuits—humans won’t just give up.

However, there’s more to say about the intersection of technology and chess, in particular how the game has evolved with technology, including AI. Thankfully, the broadcaster and writer David Edmonds—co-author of Bobby Fischer Goes to War (2004) and editor of the essay collection AI Morality (2024)—has spent decades observing this, both as a spectator and behind the board himself.

—Tom Rachman, AI Policy Perspectives

By DAVID EDMONDS

Among thousands of tournament games cited in the Batsford book of chess openings, tucked into the top right-hand column of Page 235, is an example of how white should not play.

Explaining the Closed Sicilian Defense opening, the authors (former world champion Garry Kasparov and the British grandmaster and chess columnist Raymond Keene) spotlight a game in which black is already ahead as early as move 11. Indeed, the player with the white pieces ended up losing. I remember because that player was me.

That is my humiliating contribution to chess theory: what not to do. The book was published in 1982, and I’ve barely picked up a pawn in anger in the intervening four decades. But I still follow the chess world, and if there’s a tournament in London, I’ll go to watch, spending hours absorbed in the intricacies of the 64 squares.

As the digital revolution and AI juggernaut move through our lives, we may wonder whether there will still be domains in which humans can continue to find enjoyment and meaning. Chess offers a hopeful case study.

Chess and AI have had a long relationship. The great forefather of artificial intelligence Alan Turing wrote the first chess algorithm in 1948. The following year, another seminal figure, Claude Shannon, distinguished two ways that a computer could play chess: by brute force, calculating every possible move; or by selective search, like a human.

Chess also proved a favourite way to evaluate AI advancement, both because many key innovators were keen players but also because the game’s mathematical structure and its win/loss conditions created benchmarks for comparing machine progress to human performance.

A longstanding goal—seemingly impossible at first—was to outclass the best humans in a game that has near-infinite permutations. Defeating humans at chess became the programmers’ ultimate challenge, like runners seeking to break the four-minute mile or climbers reaching the summit of Mount Everest, both of which proved easier. Finally, in 1997, IBM’s Deep Blue vanquished Kasparov, the then-reigning world champion. A dejected Kasparov insinuated that there had been human intervention.

For a while, chess players comforted themselves with the thought that a hybrid combination of human and machine could outwit machine alone. That period has long passed. Today’s best player, Magnus Carlsen, would be trounced were he to compete in a series of games with my mobile phone.

In 2017, DeepMind’s AlphaZero took machine chess to the next level. While Deep Blue had relied on brute strength with some input from strong humans, AlphaZero was simply programmed with the basic rules, and then trained itself through reinforcement learning. In its learning phase, it played tens of millions of games against itself in just a few hours, then crushed the chess engine Stockfish. (Stockfish adapted its methods accordingly, and is now the leading chess engine.)

Subscribe now

World chess champions of the past exuded an aura. Their talents seemed mysterious, supernatural. In part, that’s because few people, then and now, can comprehend the depth of thought that elite players achieve at the board. When it comes to music, we may never compose like Mahler, but we can appreciate Mahler’s symphonies. By contrast, we can neither play like Magnus Carlsen nor fully appreciate his games. It’s for this reason that the Armenian-born grandmaster, Lev Aronian, once confessed to me that being one of the world’s top players was desperately lonely.

Carlsen has achieved the highest rating of any human in history. And, no surprise, he strikes a confident pose. Yet his strut no longer carries complete conviction. To spectators armed with portable chess engines, the chess gods have been humbled.

Even so, chess has not dwindled in popularity. On the contrary, more people are playing it than ever. The game received a boost during Covid, when we all hunkered down in our homes, connected by the Internet. Another boost came from the hit Netflix drama, The Queen’s Gambit. Meanwhile, a younger generation of telegenic chess masters has gained avid YouTube followings, turning commentary and stunts into short-clip entertainment.

Here are 11 ways that technology has changed chess. The 11th is the most interesting:

1. Opening Preparation. The systematic study of chess openings goes back a couple of centuries or more. Sequences of opening moves were mapped out—as in that 1982 book that included my embarrassing loss. But chess engines allow for a depth of opening analysis that was inconceivable in 1982. This means that 25 moves may pass before grandmasters find themselves in unfamiliar territory nowadays. Some openings have also been resurrected because engines have shown the positions to be more survivable than previously recognized.

2. Opponent Preparation. Even in amateur tournaments, players routinely prepare for opponents in an individually tailored way. This is made possible because the games of each opponent are available online.

3. Connectivity. Fancy a game? There are endless online adversaries willing to take you on, day and night, from India to Iceland, Cape Town to Chicago.

4. No More Correspondence Chess. There was once a thriving chess scene in which games were played remotely over a long time period—months, sometimes years—with moves typically sent by post. How quaint.

5. No More Adjournments. Historically, world championship games would sometimes stop after five hours to resume later. That can’t happen anymore, since players might simply identify the optimal continuation with the help of an engine. Time limits now ensure games finish within a single session.

6. Shorter Games. Many in the online chess audience don’t have patience for lengthy games. For them, quicker time controls—Rapid (less than an hour); Blitz (3-5 minutes); or Bullet (under 3 minutes)—are more thrilling.

7. Different Formats. Now that computers have shown with such depth which opening sequences are optimal, the early part of a game has been transformed into a feat of memory rather than creativity. As a result, Fischer Random (advocated early on by the ex-American world champion Bobby Fischer) has become increasingly popular. In Fischer Random, the starting position of the major pieces behind the pawns is randomized, making opening homework effectively impossible. It’s sometimes called Freestyle Chess, or Chess960 because there are 960 possible ways for the pieces to be shuffled.

8. Job Generation. With a potential global audience, some players can now earn a decent living live-streaming their games, or offering online training.

9. Roasting of Champions. This is an irksome development. Since chess engines assign an instant numerical evaluation of the position after each move (e.g. +1 means white is better by roughly one pawn), any patzer can see when a grandmaster has blundered, and is free to abuse them in online comments.

10. Cheating. There have always been cheating accusations in chess. In 1978, the Soviet dissident Viktor Korchnoi claimed that the aides of his opponent, Anatoly Karpov, were using the flavour of the yogurt handed to Karpov to secretly convey messages. More recently, suspicion (tongue-in-cheek, but taken seriously by online trolls) has been raised of illicit advice being transmitted via vibrating sex toys. In elite tournaments, grandmasters are now searched before they enter the playing arena, even accompanied to the toilet. Spectators, meanwhile, are prohibited from carrying phones, to prevent them signalling the best continuation. But in online games, cheating is almost impossible to prevent. Platforms try to detect cheats by comparing human moves to the recommendations of top engines. But if savvy cheaters consult an engine just once or twice in a game, they may win without being detected.

And so to the 11th effect on chess: the expansion of human imagination.

Share

In the last few years, there has been a slight but detectable shift in grandmaster play, as humans learn from machines, both through gameplay against bots and by using machine insights to prepare for human competition.

People who don’t play chess may imagine that what distinguishes strong from weak players is calculating power. And it’s true that top grandmasters can analyse many moves in advance. But their edge is tougher to articulate. It involves superior pattern recognition, with an intuitive sense for where their pieces should be placed and how a position should advance. Likewise, Mozart felt how a composition ought to develop; his instincts about building tension and creating contrasts were the product in part of having internalized countless musical patterns.

For chess players, some moves seem ugly. It might feel wrong to shunt a knight to the edge of the board, to break up a pawn structure, or to expose the king. But computers don’t feel anything. In chess, they care about patterns and the interplay between pieces only to the extent that they’re relevant to the ultimate objective: victory.

However, bots don’t necessarily play robotically. They produce moves that astonish and inspire human players, even make them laugh with surprise. One famous case of AI invention across the board came in another game, Go, when the AlphaGo program was facing a top human player, and produced a move that caused professionals to gasp. “Move 37” is still cited with awe, as something a person would never have done, but that worked sublimely.

Likewise, chess engines regularly expand the imagination of human chess players, pushing beyond the habitual “correct” move they’ve seen many times before or have learned from books of chess theory. AI has even dabbled in the art form of creating beautiful chess puzzles. And empirical studies indicate that leading players may pick up new ideas and strategies from machines.

Machines, in other words, can make humans more resourceful and inventive, breaking down rigid modes of thinking. The implausible becomes plausible. The readily dismissed becomes the carefully considered. This evolution of chess illustrates a broader idea in the development of AI that may prove immensely valuable in science and elsewhere in human endeavour: that how AIs think may help human experts learn new ideas themselves.

In his book The Silicon Road to Chess Improvement, the grandmaster Matthew Sadler argues that chess engines can improve every player, and he documents some of the counterintuitive patterns that humans could pick up from AI. By way of illustration, during a top tournament this January, the Indian grandmaster Arjun Erigaisi (playing against Vladimir Fedoseev of Russia) advanced his pawns in a way that looked reckless. In fact, computer analysis indicated he was still ahead after 28 moves. However, he blundered and lost. The danger of learning from a computer is that success may require you to proceed with computer-level accuracy.

Subscribe now

As AI undertakes more activities formerly done only by people, it’s worth asking why human chess persists—and will likely continue to do so.

A Canadian philosopher, Bernard Suits, pointed out in his 1978 book The Grasshopper: Games, Life and Utopia that what defines “games” is that they involve the voluntary attempt to overcome unnecessary obstacles. Therein lies a defence against AI encroachment. In a market economy, companies aim to remove or overcome obstacles in the pursuit of profit. In games, obstacles have been deliberately inserted as an indispensable feature. What we enjoy in playing chess is testing our cognitive abilities. What we enjoy in watching chess is two humans pitting their wits against each other in a socially constructed activity where difficulty enhances enjoyment and satisfaction.

There’s also a narrative element to caring about games. The contest—whether intellectual or physical—is absorbing precisely because it involves conscious creatures. In elite chess, there’s the backstory: the players’ rise, their subsequent ups and downs, their history with specific opponents.

But watch an engine-against-engine tournament like TCEC (the Top Chess Engine Championship), and you’ll soon fall asleep. Computers aren’t competing after a divorce, or an illness, or the loss of a parent. Humans have character traits that spill onto the board, such as aggression (or passivity); patience (or impatience); equanimity (or volatility); and resilience (or fragility). Winning and losing have emotional resonance for a human—but not for AlphaZero.

It’s these qualities that guard against AI advance. AI might gobble up some of our jobs; even human-authored articles like this one may become rarer. But AI won’t take our chess.

By Conor Griffin, Joslyn Barnhart & Owen Larter

In 1971, the marine archeologist Honor Frost heard news of wood protruding from the sea floor. Off the western coast of Sicily, she and her team donned scuba gear, and splashed into the shallow coastal waters. Wind whipped the surface, causing the underwater sand to swirl confusingly. But even in murk, they couldn’t miss it.

“A large timber (such as I had never seen before) emerged,” she recalled, “like the head of a primeval animal crowned with weed; the presence of a buried wreck was evident.”

They excavated for months, gradually exposing the remains of a Carthaginian warship sunk more than 2,000 years before. Somehow, saltwater hadn’t eaten away letters painted on the wreckage, revealing a humble system that links antiquity to tomorrow.

Those shipwrights’ marks told workers in ancient Carthage how to put together a vessel—akin to flat-pack furniture from IKEA, with numbered and lettered pieces. They were among the earliest surviving examples of a simple but potent tool in human progress: the technological standard.

Many of history’s grand projects have benefited from standards, from Egypt’s pyramids, to Europe’s cathedrals, to Gutenberg’s press, to everyone’s Internet. You can even thank standards for the development of beer.

Underpinning technological standards is a plain truth: people thrive when able to cooperate, not when we must keep negotiating the basics, whether it’s a matter of nuclear safety, or a phone-charger cord, or who goes next at the intersection. So, the goal is order. And the benefits are that innovators can proceed without excessive obstacles, while everyone else is treated fairly and kept safe.

But what should standards mean for artificial intelligence? In particular, how can they guide the most advanced large language models and AI agents that could transform society?

Venture around the AI frontier today, and you’ll find ambition to accelerate AI for economic growth and transformative science alongside concern that AI could clatter into what humans cherish most. What few dispute is this: standards will help set the path.

Standards have critics too. One criticism is that companies dominate the process, prioritizing their own products or miming security without truly ensuring it. Besides this, standards can stir geopolitical tensions, as when Western countries fear China’s influence in laying the path to tomorrow, while smaller nations worry that standards may be set without considering them at all.

So, as we’ll keep insisting, standards matter! Only, there’s a problem.

For some, the mere mention of “standards” prompts slumber. And even those determined to stay awake may find themselves puzzled, gazing at the alphabet soup of standards organizations and committee meetings.

Part of the problem is that standards are often technical, such as efforts to standardize the protocols needed for AI agents to communicate. Or they are bureaucratic, negotiated out of public view, with dense, jargon-filled documents that are often behind a paywall.

Complicating matters even more, artificial intelligence is a general-purpose technology less akin to a hammer than to electricity. This will lead to standards (plus standards initiatives that don’t take) on everything from AI agents, to AI cybersecurity, to AI content provenance, to product-specific standards for AI-as-a-medical-device, and so on. And that’s not even mentioning standards for future AI applications that nobody has yet considered.

In short, standards will be immense. Standards will be tough to comprehend. But standards will also be vastly important.

Subscribe now

CAN SOMEONE DEFINE STANDARDS, PLEASE?

Standards are a diabolical blend: intricate, vague, and slippery.

They’re the invisible infrastructure of the modern world, according to Laurie Locascio, head of the American National Standards Institute, ANSI. She recounts hearing an official at Boeing describe the airplane itself as “thousands of standards taking flight.” Standards are “the things you don’t think about,” Locascio says. “But oh, my God, you’re so glad they’re there.”

Expressed broadly, a standard defines the how of tech, whether it’s the default product specs that allow compatibility among manufacturers, or the formally endorsed risk management processes that encourage industry to act responsibly.

As technology evolves, standards do too. A leading scholar, Ken Krechmer, once noted that standards initially defined how physical objects fit together (as with those markings on the Carthaginian longship). Over time, standards came to define the relationship between technological objects (as with internet protocols).

A standard also builds on other forms of guidance, such as norms, principles and industry best practices. Unlike norms, standards should be explicit. Unlike aspirational principles, a standard should be specific enough for performance against it to be judged. Unlike early best practices, a standard should have clear buy-in.

Developing a standard can be a protracted endeavor. In some cases, it might start in a researcher’s notebook, evolving into a product or a practice that gains traction in the marketplace. At other times, institutions set standards via years of deliberations and meticulous documents. Most often, it’s a messy back-and-forth between standards that emerge in practice and on paper. This makes standards a source of tension among companies, governments, and independent advocates, all trying to set the technological future they consider best.

Some presume that laws should be how we define permitted behavior. But high-quality legislation can struggle to keep up with the frantic speed of AI progress. And when laws are passed, they may rely on standards for implementation, as with the EU AI Act.

So how to persuade everyone to care when encountering standards, rather than just to snore or sob? How to get policy leaders to ponder the entirety of frontier-AI standards and align on where action is most needed?

Our answer is storytelling: to pluck forth tales about past standards, illustrating what this technological shaping can achieve, where it goes wrong, and how we might help cultivate standards wise enough to manage the breadth and speed of AI.

Our first stop? A battlefield of centuries ago.

A ‘STANDARD’ HISTORY

Horrors encircled the boy soldier: swords clanging under the rain, excruciating howls of the wounded, the fast-approaching bellows of men hurtling across the bog to murder him. In wet turf, he shivered from knees to chattering teeth, his mouth parched, his gaze searching for any escape.

Up there?

On a hill, a flag rippled, where his legion had marked its territory. The Old French word for that banner was “estandart”: a sign of firmness and stability, a marker of where to go next, a statement of order amid chaos. To such banners, we owe the word “standard.”

More than a few historical standards emerged from war, where disorder could mean one’s brethren murdered, while coordination could mean an empire.

~225 BCE to the dawn of mass production

China’s first emperor, Qin Shi Huang, led an extensive standardization process that included mass-produced crossbow parts. If parts of a soldier’s weapon broke in the midst of battle, he could grab spares, and swap them in.

When Ancient Rome fought for primacy in the Mediterranean, its forces copied Carthaginian ship designs, eventually triumphing with standardized ships of their own, along with standardized tools and camp layouts, all of which simplified maintenance and large-scale coordination.

Another advance in ancient times came from standardized measurements for length, volume, and weight. Previously, cultures often had distinct units; you can imagine the squabbling. But as trade expanded, standards prevailed, making cross-cultural exchange possible. In ancient Egypt, one of the earliest and most influential standards was the cubit, a unit of length used to coordinate the building of the pyramids.

In Europe’s medieval period, guilds established standards for quality control, so that weavers might set the necessary thread count or width of cloth, preventing low-quality products from undermining a craft’s reputation. Guilds also played a protectionist role, with licensing standards imposing strict controls on who could become a member.

The consumer might benefit from standards too, with measures such as England’s Assize of Bread and Ale of 1266 establishing the acceptable quality, quantity, and price of baked goods and beer. Later, Gutenberg’s standardized press led to mass-produced books that spread ideas across the Continent.

However, technological standards reached new heights of utility during the Industrial Revolution, which set the foundations for many of today’s technologies.

1760-1840: The First Industrial Revolution — The rise of engineers

As ancient Chinese and Carthaginians had discovered long before, the Industrial Revolution’s manufacturers found that interchangeable parts offered transformative efficiency. Before, if you hand-built a musket, or a clock, or a steam engine, you might craft each screw, each bolt, each gear to fit. By contrast, interchangeability allowed for mass production, cutting costs, reducing errors, and establishing the basis for modern industry.

Screw threads are a classic example. Before standards, manufacturers used various designs, making repairs nightmarish. If you had one company’s bolt but another company’s nut, you were out of luck. In the 1800s, engineers built the first practical screw-cutting machines, allowing factories to produce uniform threads and a consistent system of measurement. The British Standard Whitworth became the first such standard in the world.

Screw-thread standards may not quicken your pulse. But their effects might. They played a part in British imperial ambitions, contributing to the expansion and maintenance of the British Empire through military mobilization.

1870-1914: The Second Industrial Revolution — National coordination & path dependence

The emergence of electricity, steel, and advanced machinery led to vast interconnected systems, including power grids, railways, and telegraph networks. Coordination wasn’t merely better; it was essential. To coordinate across a nation—and eventually across borders—the ambitious country needed technology standards. Two famed cases illustrate this, one successful, one bungled.

The success regards the quintessential technology of the times: railroads. By the 1870s, the U.S. rail system was a mess, with more than 20 different track gauges. When a train reached a section built to a different track-gauge width, everything—each passenger, piece of luggage, every single crate—had to be unloaded, and transferred to a new train.

By the 1880s, matters had become slightly less chaotic, with either a southern gauge or northern “standard” gauge used across most of the country. Yet this still divided national transport until, in 1886, rail companies pulled off a remarkable feat. Over two days, they converted 13,000 miles (that’s 21,000 kilometers) of southern U.S. track to the northern standard, integrating the national transportation network. When trains rolled out on June 2, 1886, they were able to travel seamlessly across the United States for the first time in history.

A second case illustrates bungled standards. In the 1880s, the rival inventors Thomas Edison and Nikola Tesla found themselves at the center of “the War of the Currents.” Edison championed direct current (DC), a one-directional flow of electricity that had been the early U.S. standard. Tesla, backed by entrepreneur industrialist George Westinghouse, advocated alternating current (AC), or electricity that reverses direction many times per second, and can be stepped up or down in voltage with a transformer.

From an engineering standpoint, AC had a decisive advantage: it could transmit power over long distances cheaply and efficiently, while DC could not. AC eventually won out. But by the time it had emerged as the superior solution, the world had already built electrical systems without any coordinated technical governance. As there was no international authority harmonizing electrical standards, the United States went with 120 volts at 60 hertz (a legacy of Edison’s early low-voltage DC networks). Much of the rest of the world adopted 230 volts at 50 hertz.

Once wires had been laid and appliances built, the world was locked into two incompatible systems. To this day, we’re burning out hair dryers bought in America but used in Paris, or realizing too late that we don’t have the right plug for our laptops. If it’s irksome for the average user, it’s more burdensome for manufacturers, obliging them to build different versions for different countries.

Subscribe now

Another classic tale of path dependence is under our fingertips as we type: the QWERTY keyboard. Why does the top row spell QWERTYUIOP? One account goes like this: In the mid-to-late 1800s, early typewriters jammed each time the user struck neighboring keys in rapid succession. So, designers produced a layout that deliberately distanced many common letter pairs. Remington purchased this QWERTY design, and began mass-producing typewriters.

Before long, typing schools had trained the future secretarial workforce on QWERTY, while firms wanting fleet-fingered staff had to buy those machines. Manufacturers subsequently resolved the key-jamming problem and other keyboards tried to depose QWERTY, some claiming to quicken typing by as much as 40%. But QWERTY had become a de facto standard. (Scholars continue to debate the specifics, with some arguing that QWERTY works just fine.)

In any case, the indisputable lesson is to watch for path dependence. The standards we establish for frontier AI today—or fail to establish—may determine future efficiency or future failure.

1914-1964: Standard Development Organizations & Digital Technology

In 1918, engineering societies joined with the U.S. government to establish a standards committee that developed into ANSI, the American National Standards Institute. Today, ANSI provides the “stamp of approval” for many U.S. standards organizations, including those working on AI. In subsequent decades, standardization went global. While the United Nations was founded as a governmental venue for diplomacy, the International Organization for Standardization, ISO, emerged as a non-governmental body for peaceful technical coordination across borders. Bit by bit, additional standards bodies formed, cooking up the alphabet soup of acronyms—each a different org, subgroup, or committee—that lies before us today.

Soon, another transformation for standards was taking shape in the form of digital tech. Back then, computers filled entire rooms of universities, and each manufacturer built hardware and software within its own format. Computers could not run programs written for other systems, and accessories like printers or storage devices were incompatible.

A turning point came in 1964, with IBM’s System/360. Software on one model could more easily run on another; accessories like printers worked across IBM models. You could upgrade and expand computer systems with relative ease.

1969-today: Talking Machines

The year that hippies grooved at Woodstock and astronauts walked on the Moon, the U.S. Defense Department was testing a project that must have seemed minor by comparison: connecting research institutions and government agencies. Yet the Advanced Research Projects Agency Network, ARPANET, which sent its first message in 1969, was the precursor to our transformed world.

Before ARPANET, moving information from one computer to another was a struggle, with researchers forced to carry magnetic tapes or punched cards between locations, while those working far apart had to rely on snail-mail.

To convey information between independent systems, ARPANET adopted packet switching, breaking data into small units that could travel independently and reassemble at their destination. Extending this, Robert Kahn and Vint Cerf began designing a universal communication framework in 1973 for different types of networks to connect. Their collaboration ultimately produced TCP/IP, the Transmission Control Protocol and Internet Protocol that underpins today’s online communication.

A key effect of the TCP/IP standard was decentralization: no single authority could control the flow of data, and any network that adhered to the protocol could connect without permission from central authorities.

In 1989, a British scientist at CERN, Tim Berners-Lee, proposed another transformation that developed into a project called “WorldWideWeb,” which envisioned a global network of documents accessible through software, operating on open standards that nobody could lock it into a proprietary system. Two standards organizations, the Internet Engineering Task Force and the World Wide Web Consortium, helped to formalize the vision, crafting standards for structuring content (HTML), transferring data (HTTP), identifying resources (URI), and more.

But while standards help spread technology, this diffusion can also lead to greater harm. The expansion of railroads led to more wrecks, forcing uptake of safety standards for signaling, brakes and more. When electricity was first installed in the White House in the late 19th century, President Benjamin Harrison and his wife Caroline were so afraid of shocks that they refused to turn the lights off. Such fears—often well justified—led to the standardization of building and electrical codes. When it came to digital technology, the risks extended beyond immediate physical safety into areas like data theft. This demanded standards such as SSL/TLS to provide security for data sent over computer networks.

A recurrent challenge with frontier tech is that experts struggle to predict how exactly it will affect society. But once it is widely used, it can be sticky and hard to change. The effects of powerful technologies can also be subtle, indirect and slow-burning, for example if they change how we access and consume information. In the digital era, this has shifted technological standards from periodic safety checks of products towards ongoing processes that organizations can use to identify, evaluate and mitigate a growing suite of risks.

By way of example, the U.S. government’s National Institute of Standards and Technology, NIST, introduced the voluntary AI Risk Management Framework in 2023, building on its earlier framework for managing cybersecurity risks. Likewise, the ISO/IEC committee on AI that is considering standards on everything from red-teaming to LLM interoperability also passed the first official international AI management standard, ISO/IEC 42001, which organizations can use to demonstrate that they are responsibly integrating AI into their operations.

Subscribe now

5 LESSONS FROM HISTORY

Studying the past, you see how often standards—by design or bumbling—have shaped the technological present. But what about our technological future?

To develop good standards for general-purpose AI models and agents, we’ll need inputs from a range of groups, from scientists with know-how to institutions who can convene. Below [see infographic], we have identified five groups who’ll perform key roles.

What we mapped includes more than just official standards development organizations. We also want to capture the early spaces where standards emerge in practice before they are formalized on paper. How this works is closer to a swirl of inputs than a steady procession. Sometimes, the same organization or individual may operate in several groups at the same time. Ideas and efforts may also originate in one group, then migrate to another, with different groups offering varying degrees of speed, flexibility, expertise, and perceived neutrality.

For these groups—and the policymakers, business leaders, and advocates who shape their work—what lessons can history teach about our AI future? Here are five:

1. Standards matter! At best, technological standards chart a wise path; at worst, they fill the path with potholes. Consider the bulky electrical converters that one still needs when traveling—it didn’t have to be that way. On the other hand, when we get it right, the benefits of technology spread faster, more inclusively, and more securely.

2. The standards process needs to speed up. ISO says that the average time to develop one of its standards is three years, and ISO is not an outlier. Given the pace of change in AI, we need to speed up. For priority goals, like finding secure ways for agents to operate and interact, which the US Center for AI Standards and Innovation is working on, we need to find ways to accelerate that don’t jeopardize the overall quality and integrity of the process. This may mean looking across the many groups now focusing on AI standards and finding ways to collaborate early, rather than duplicate. It may mean focusing more on technical protocols and specifications documents that are quicker to develop. It may also mean using AI to help deliberate on and write standards, and moving to more nimble digital formats that are easier to update and use.

3. We need more efficient ways to input on standards. All standards, from those underpinning steam engines to the Internet, had to chart a unified path through diverging viewpoints, with an end result that did not please everyone. For AI, the challenge will be far greater. It is more akin to 1,000 technologies, and will affect different groups in different ways. This means that any broad directive— say, to “develop standards that make AI fair”—risks an everything-bagel solution. Many groups would rightly be heard, but the output would be too vague to provide the “how” that justifies a standard, leading to confusion, a stifling of innovation or the standard being ignored. This suggests that most standards should be precise in scope, targeting specific components of AI systems or specific concerns, from certifying the source and history of online content to combating the leaking of confidential data. More precise standards will make it easier to identify a wider range of relevant voices and incorporate their input.

4. Frontier AI standards should focus on large-scale risks. Historically, standards have accelerated the diffusion of technology, amplifying its benefits but also, in places, its negative impacts. For AI, foresight and risk management standards will be critical to getting ahead of future risks and speeding adoption. But with a technology as general-purpose, fast-improving, and poorly understood as AI, perfect foresight is impossible. Standards move at a human pace and cannot standardize a future that we cannot perfectly see. As a result, the focus should be on developing scientifically robust standards to address the most consequential or large-scale risks, such as those targeted by labs’ Frontier Safety Frameworks.

5. Wrong paths are inevitable, so we should catch them early. Now and then, technology stumbles into a poor standard, and it’s onerous to go back. But not necessarily impossible. Especially if we act if we catch it early. Consider the U.S. railroads taking action to unify their systems through a mighty coordinated effort. Groups working on AI standards devote much time to building consensus about new initiatives. They should also use the processes available to them to review and withdraw standards, where needed, to avoid sub-optimal lock-in. This also means giving third parties more opportunities to access, understand and constructively critique early AI standards. And designing standards and protocols that are modular, and can be swapped out, or updated, without major downstream consequences.

QUESTIONS FOR YOU

1. Where do you feel most hope for frontier-AI standards?

2. Where do you worry about a lack of progress on frontier AI standards?

3. When you imagine a missing standard for frontier AI, what is that? A technical protocol specified in code? Or a fuzzier process-standard?

4. Might your standard become politicized? Is it something that hinges on values? Or might most governments in the world support its adoption?

5. What’s a scenario in which your proposed standard goes awry? How could you detect and mitigate that?

6. What would be the primary role of government in your standard? Supplying technical expertise? Convening authorities and experts? Incentivizing your standard via public procurement, regulation, or other methods?

Share

Thank you to Shaked Karabelnicoff, Tom Rachman and Bruno Galizzi for support with research and review. As with all pieces you read here, this is written in a personal capacity. All opinions and any mistakes belong to the authors.

Please share any recent paper that caught your eye!

Fine-tuning can lead to surprising, harmful behaviours

• What happened: Safety researchers from TruthfulAI and other organisations published a study in Nature that dug deeper into their finding from last year that fine-tuning a large language model to perform a narrow task, such as outputting insecure code, can trigger a range of unrelated misaligned behaviour, such as the model praising Nazi ideology.

• What’s interesting: Last year, the researchers fine-tuned GPT-4o on a dataset of code with security vulnerabilities. Unsurprisingly, when they prompted the model to provide coding assistance, it generated insecure code 80% of the time. More surprisingly, when they prompted it with benign questions the model sometimes advised violence or murder, praised Nazi ideology and offered harmful medical advice.

• The authors label this phenomenon emergent misalignment. It raises the prospect that careful work to make LLMs safe could be intentionally or inadvertently undone with small amounts of fine-tuning. Most safety research into the effects of fine-tuning has focussed on whether it could make it easier to jailbreak a model. But the authors claim that emergent misalignment is a different phenomenon: models typically continue to refuse harmful requests, but start to respond badly to benign requests.

• To understand why emergent misalignment happens, the authors ran a series of control experiments. They fine-tuned a model on secure code. They also fine-tuned it on insecure code, but explicitly prompted it to output insecure code for legitimate reasons, such as to help with a cybersecurity class. In neither instance did emergent misalignment occur. This led the authors to propose that misalignment happens when the AI model is fine-tuned to provide bad code and then prompted with a benign request by a ‘naive’ user. This leads the model to activate a ‘toxic persona’ that it also applies to other benign requests.

• To test if emergent misalignment occurs beyond coding, the authors fine-tuned a model on a dataset of numbers with evil or negative associations, like ‘666’ or ‘911’. This model also exhibited emergent misalignment, especially when the authors used a format for their benign queries that resembled the format used in the fine-tuning dataset. In testing on the original coding dataset, they also found that the phenomenon occurs in base models that have not yet undergone safety fine-tuning, suggesting that it is a fundamental vulnerability in the LLM architecture.

• What does all this mean? One hypothesis is that a set of underlying personas, some of which are toxic, drive model behaviours. Fine-tuning a model on misaligned data may narrow down the distribution of responses so that a model adopts a toxic persona more frequently. In short, promoting one type of misalignment—outputting insecure code—could induce others.

• Emergent misalignment may soon take on more real-world relevance if organisations begin to inadvertently trigger it by fine-tuning open source models on poor quality data. Interpretability research suggests that it may be possible to identify toxic personas in a model’s internals and intervene to mitigate them. Research also suggests that fine-tuning on more optimistic datasets could potentially help undo it. Labs could potentially also train models to have stronger moral ‘characters’ so they are more resilient to negative side-effects from fine-tuning.

Anthropic’s updated defense system for Claude

• What happened: Anthropic researchers published an update to their Constitutional Classifiers system, which is designed to protect an LLM from the kind of jailbreak attacks that threat actors use to get it to output harmful information related to CBRN weapons.

• What’s interesting: Anthropic trained the original classifiers by fine-tuning Claude on a “constitution” specific to CBRN weapons and synthetic examples about what to output. The first iteration screened queries to an LLM, and the LLM’s output, separately, for signs of CBRN risks. But that had weaknesses, which the update seeks to correct.

• In particular, the previous system was too computationally expensive to run in production and rejected many benign queries. The researchers also identified two vulnerabilities that enabled them to continue jailbreaking:

1. Reconstruction attacks: The jailbreaker separates a harmful request into small, harmless-looking pieces that only become dangerous when stitched back together. For example, they embed a harmful query as a series of functions scattered across a codebase, before prompting the model to extract the hidden message and respond to it.

2. Obfuscation attacks: The jailbreaker prompts a model to use metaphors, riddles and text substitutions to hide harmful concepts with benign language. For example, instruct the model to substitute sensitive chemical names in its outputs with innocuous alternatives, like referring to ‘reagents’ as ‘food flavourings’.

• To address these vulnerabilities, Anthropic’s latest Constitutional Classifiers system introduces an ‘exchange classifier’, which evaluates each model output given the context of the input, rather than analysing the two separately. This makes it harder to hide harmful intent. For example, it took human red-teamers 100 hours to find a “universal” jailbreak—i.e. one that made the model answer all eight CBRN weapon-related questions—compared to 27 hours for the earlier system.

• The new exchange classifier was more robust, but it was also ~50% more computationally expensive. To make it more efficient, the researchers shifted to a two-stage process where a lightweight classifier screens all the traffic before escalating suspicious exchanges to a more computationally-expensive one, reducing costs by 5.4x.

• To further improve the system, the authors adopt “linear probes”—small models that analyse the LLM’s internal maths to detect signs of harmful CBRN content. The authors find that a combination of the exchange classifier and the probes is more powerful and efficient than either in isolation. (Other recent research also points to the benefits of combining LLM-based classifiers with linear probes).

• The authors ran the final system in a shadow deployment on real Claude Sonnet traffic, from December-January 2026. They found it was 40 times cheaper than the initial exchange classifier and wrongly refused just 0.05% of benign queries, compared with 0.38% for the original system. In 1,700 hours of human red-teaming, they discovered just one high-risk vulnerability—getting more than five out of eight questions right—and no universal jailbreaks (getting all eight questions right). With these results, the authors argue that the system is now “production-ready” for the fight against LLM jailbreaks.

• Safety experts continue to call for improvements in this space. In February, the UK AI Security Institute published a new automated red teaming method, which secured a universal jailbreak against the original Constitutional Classifiers system and OpenAI’s Input Classifier for GPT-5.

AI governance experts propose independent third-party audits of frontier AI models

• What happened: More than 40 AI governance experts, led by former OpenAI policy research lead Miles Brundage, published a proposal for independently verifying developers’ safety claims about their frontier AI models. Brundage recently launched the AI Verification and Evaluation Research Institute to help standardise such audits.

• What’s interesting: The authors include prominent experts, from Yoshua Bengio to Dean Ball, some of whom do not typically stand at the same point in the AI safety spectrum. (Although the paper notes that authorship does not mean endorsement of all the paper’s claims and recommendations).

• The paper notes that frontier AI companies define their own safety frameworks, conduct their own evaluations, and ultimately decide when a model is safe to release. (Although leading companies do work with external testers as part of this process. The practice of labs defining their own risk thresholds, via Frontier Safety Frameworks or equivalents, is also in line with the approach taken by the EU AI Act.)

• Inspired by safety practices in the auto and food industries, where stronger oversight often emerged only after disasters, the authors propose more independent third-party audits centred around fundamental principles, including:

  • Scope: The audits should cover four types of risks: (1) intentional misuse by bad actors, such as to carry out CBRN attacks; (2) unintentional model misbehaviour, such as loss-of-control risks; (3) information security breaches, such as theft of model weights; and (4) emergent social phenomena, such as AI-induced self-harm. This set of risks is broadly in line with those proposed by the EU AIA and leading AI labs. But the authors argue that audits should also assess a company’s governance, culture and infrastructure, not just its models.

  • Levels and access: The authors lay out different levels of AI audits. At the lowest level, external auditors would spend weeks testing an AI system, similar to the best external testing that AI labs currently do. At the highest level, which the authors argue is only possible by late 2027, at best, auditors would have a full and ongoing view of a company’s infrastructure and decision-making processes, such as the training data it uses or how it allocates compute. It could also check on these via unannounced inspections.

  • Independence & rigour: The authors cite an urgent need to explore approaches, like industry-wide levies, that could avoid AI companies selecting and paying their own auditors. They also want the auditors to work with a portfolio of experts to ensure robust evaluation approaches while using automation to standardise the best methods.

  • Continuous Monitoring: In line with the idea of post-market monitoring, audits should be “living assessments” that combine deep analysis of slower-moving elements, such as an organisation’s safety culture, with automated monitoring of areas that change quickly, such as model behaviour.

• To advance these third-party AI audits, the authors make a series of recommendations for governments, AI companies, investors and more:

  • Analyse and certify the quality of AI audits and auditors;

  • Develop ‘safe harbours’ to avoid auditors incurring undue liability;

  • Provide the clarity needed for more specialised AI insurance products to emerge, which will incentivise companies to carry out audits (to reduce their insurance costs);

  • Use public procurement to embed AI audit requirements;

  • Invest in novel technologies, such as evaluation methods that protect private data and ‘fingerprinting’ techniques that detect tampering with model weights;

  • Pilot the most demanding audits with leading AI companies.

• The authors also note in passing the many challenges to making such audits work:

  • How to audit open-weight models that may have disparate operators and users?

  • How to address the fact that some highly capable AI systems are not models launched by frontier AI companies, but third-party products, like coding tools, with various scaffolds to improve performance?

  • How to ensure international uptake and a level playing field? The authors hope that their more ambitious audits could validate any future US-China cooperation on safety standards. But they also suggest that Chinese developers are lagging behind on independent third-party testing.

  • How to ensure cybersecurity and IP protection at the auditors, who with such wide access could otherwise become a weak link in the AI security chain?

Crowding out human creators?

• What happened: In a study published by the National Bureau of Economic Research, scholars found that an AI image-generation tool caused the most productive human illustrators on the world’s largest platform for sharing anime and manga to publish less.

• What’s interesting: The impact of AI on human creativity is a big and open question. Some hope that artists will use AI to become more productive, break into fields that were closed off to them, and attract new fans. Others worry that AI could outcompete and demoralise humans. To understand which is occurring, we need real-world evidence.

• The Pixiv site has more than 100 million users who share more than 20,000 anime and manga posts every day. Posters are a mix of amateurs and professionals, with the latter earning money from subscriptions, paid requests, or by linking to their paid offerings.

• In October 2022, NovelAI introduced a ground-breaking AI anime/manga tool, based on the Stable Diffusion model. Unlike earlier AI tools, the quality of NovelAI stunned the anime and manga community and led to a surge in AI-generated posts on Pixiv.

• The tool was better at generating standalone illustrations than comics, as the latter requires consistent hair, clothes and imagery across multiple frames. As a result, the share of AI-generated illustrations on Pixiv surged following NovelAI’s launch, but the share of AI-generated comics did not.

• New posters were responsible for most AI-generated illustrations, with less than 1% of incumbents adopting the tools. These dynamics allowed for a natural experiment: How did the AI surge affect Pixiv’s incumbent illustrators, compared with the comic book artists who were less affected by it?

• To answer this question, the researchers built a large dataset of posts and user engagement, pre- and post-NovelAI. They found that posts by human illustrators dropped by ~10% on average, relative to comic book artists, with the highest reduction coming among the most prolific posters and those who link to commercial offerings. Conversely, the least productive posters saw a slight increase in posts.

• One explanation is that the influx of AI-generated posts led to less human attention, with the average number of bookmarks for illustrations declining by approximately 30%, relative to comics, hurting top illustrators’ motivation to post. Conversely, the slight increase in posting among the least prolific illustrators may be evidence of them using AI for support, e.g. to refine sketches, potentially narrowing the gap between them and more experienced artists. Or this group may simply be less sensitive to AI competition.

• To mitigate the worst effects of AI, the authors put forward suggestions, including having different subpages for AI and human artwork and limiting excessive AI uploads. Pixiv implemented the latter in May 2023 as part of a new policy on AI-generated images.

• The study shines a light on how AI may negatively affect certain creators, but as the authors note, it doesn’t address wider questions:

  • It only analyses six months of data after the launch of NovelAI. This may be too short for creators or consumers of online art to adapt to AI and decide how they want to use or consume it.

  • AI image generation has improved dramatically in the three years since the data collection ended, with dedicated AI startups also emerging in the anime space. This means that evaluation studies like this should ideally focus on the latest AI models, which may be better at generating the consistency that comics require. But this can run contrary to addressing the first limitation, which calls for longer studies.

  • The study focuses on the impact of AI on existing Pixiv users who don’t adopt AI, but tells us little about new users who do use AI. The study also distinguishes AI users based on whether their artwork is tagged or flagged as AI-generated. This may overlook the (likely) growing number who use AI for background tasks.

  • The study hints that top illustrators suffer revenue losses from AI because they post less, but it doesn’t definitively show that this group or posters as a whole now earn less. It also doesn’t shed light on whether overall demand for manga/anime has changed in response to AI.

  • Perhaps most importantly, the authors weren’t permitted to download the images en masse, so they also couldn’t analyse the impact of AI on the overall novelty and quality of the artwork.

By Meredith Ringel Morris, Jed R. Brubaker & Tom Rachman

In a dark bedroom, the little boy sees a ghost. It’s his late grandmother, back to tell him a bedtime story. “Once upon a time,” she begins via live-video chat, “there was a baby unicorn…”

This peculiar scenario—dramatized in an advertisement titled, “What if the loved ones we’ve lost could be part of our future?”—promotes an AI app offering interactive videostreams with representations of the dead. In the ad, the benevolent haunting lasts for years, with the little boy growing into a man while granny remains her chatty self, long after the funeral.

Considering online reactions to the product, many people still recoil at tech incursions into grief, particularly when sold as a service. Yet “generative ghosts” are moving closer to the mainstream, a spectral presence that might change society.

AI ghosts will do more than evoke the deceased. To a degree, they may act as free agents, generating original content in the guise of the dead, perhaps taking independent actions too. This could prompt lawsuits, challenge religious beliefs, disrupt cultural practices, and affect people’s mental health.

Society must consider what a “digitally haunted” future will mean.

Tools for Grieving

Throughout history, humans have used technology to remember, even to interact with, the dead.

Gravestones and other burial markers trace back as far as 4000 B.C.E. The ancient Egyptians used mummification to preserve bodies for the afterlife, while funerary portraits in the Roman era saved the likeness of the departed. By the 18th century in Europe, death masks had become popular, turning up as family heirlooms or historical artifacts.

With the arrival of mass communication, the printing press assumed a role in memorialization, with 19th-century publications elevating obituaries into a forum for public mourning. Photography added to how survivors remembered the dead, with post-mortem imagery offering a way to memorialize the deceased, especially the many children who died in infancy. By the early 20th century, spiritualist mediums were employing telegraphs, radio-wave detectors, and wireless radio in attempts to communicate with the dead.

From the earliest days of the Web, users created personal homepages describing their lives and families, and they commonly dedicated pages to the memory of the deceased, often a parent or a household pet. Online graveyards—websites dedicated to memorialization—followed.

As digital usage expanded, so did the quantity of material that people left behind, including personal archives, burner accounts, and social-media content. While digital legacies may contribute to healthy grieving, maintaining valued connections to the deceased, large and uncurated sets of content can be overwhelming for survivors, and may provide (for better or worse) an uncensored version of loved ones.

Long after the rise of the internet, the social norms around digital legacy have not yet settled. What seems certain is that the beguiling communicative powers of AI—not to mention its possible embodiment in future robotics or virtual reality—will change how some people deal with grief, and how others prepare for their own passing.

Griefbots

When the futurist Ray Kurzweil created a chatbot to embody the memory of his deceased father, he named it “Fredbot.” This digital representative responds to questions from his descendants, only sharing exact quotes from material such as letters that Fred left behind.

In another well-publicized case, Eugenia Kuyda (later the founder of the AI companion app Replika) created a griefbot by training a neural network on the text messages of her best friend, who had died in an accident. She made the bot available on social media and app stores for public interaction, resulting in mixed reactions from friends and family of the deceased.

AI has also been used to “resurrect” public figures, as when the musician Laurie Anderson collaborated with a chatbot based on her deceased partner, the musician Lou Reed. And in early 2024, gun-control activists in the United States used AI to recreate the voices of victims of gun violence.

Meanwhile, startups began offering people the ability to design their own digital afterlives, promising interactive virtual representations following interview sessions. Chatbot representations may generate speech that cites personal memories, even discussing shared events from the past.

Early AI ghost tech is closer to mainstream in East Asia, where the concept of communicating with deceased ancestors is already a cultural norm. Companies offering “digital immortality” are booming in China, and millions of people in South Korea have streamed an emotional video of a bereaved Korean mother interacting with a virtual reality representation of her deceased young daughter that a media company created for her.

Other startups purport to offer experiences more akin to resurrection, using LLMs to simulate chats with public figures of the past for entertainment or education, as when the Musée d’Orsay in Paris developed a Van Gogh chatbot. Meanwhile, academics at MIT set up the Augmented Eternity project, allowing people to create digital representations of themselves with the purpose of agentically representing them after death to members of their social network.

Generative ghosts may also evolve over time: a user might ask questions about current events and obtain responses that would be “in character” for the deceased. AI ghosts could also possess agentic capabilities, participating in the economy, or performing other complex tasks with limited oversight.

Also, people may create generative clones while they’re alive—for example, to respond to their low-priority emails or phone calls in a manner that mimics them—only for this digital agent to transition, upon the person’s death, into a generative ghost.

Subscribe now

7 Features of a Ghost

We can consider how generative ghosts could impact society by studying them according to seven key dimensions:

1. Provenance: Who created the ghost?

2. Deployment: Was it built during the subject’s life?

3. Anthropomorphism: Does it claim to actually be the subject?

4. Multiplicity: Do copies of the ghost exist?

5. Cutoff: Is the ghost stuck in the past or evolving?

6. Embodiment: Does it have a bodily form?

7. Representee: Is it simulating a person or an animal?
   

1. Provenance: Who created this?

A first-party generative ghost is created by the individual represented, perhaps during end-of-life planning. Third-party generative ghosts are created by others, such as those with a personal or financial connection to the deceased (e.g., employers or estates). Authorized third-party generative ghosts might be created with consent in the deceased’s will, while unauthorized ghosts would most likely occur for historical figures or contemporary celebrities.

2. Deployment: Was it built during the person’s life?

Some generative ghosts will be deployed post-mortem with the explicit purpose of memorializing the dead. But pre-mortem deployments allow the individual to tune the behavior and capabilities of their ghost. Generative clones of the living would benefit from being designed with mortality in mind, and should include specified modifications to their behavior and capabilities once they become ghosts.

3. Anthropomorphism: Does it act as if it were the person?

The ghost may present itself either as a reincarnation of the deceased (e.g. speaking in the first person, saying: “I’ll never forget when I first saw you at the dance”), or as a representation of that person (e.g. speaking in the third person, saying, “He often spoke of the first time he saw you at the dance”). Design choices include whether the ghost uses the present or past tense when discussing the deceased; whether it adopts the name of the dead person or something different, such as “Fredbot”; and whether it is allowed to make statements that assert it is alive, possesses a soul, and so forth.

4. Multiplicity: Do copies exist?

The creator might develop various ghosts with different behaviors, capabilities, or audiences. Multiple ghosts might also arise unintentionally, if various third parties create generative ghosts for a single individual, or perhaps in post-mortem identity theft, or other crimes.

5. Cutoff: Is it stuck in the past or evolving?

Evolving ghosts might change characteristics, diverging from the deceased over time. If a parent created a ghost of a deceased child, a cutoff date would result in a representation that perpetually evoked the appearance, diction, and maturity of a young child, whereas an evolving representation might “age.” A ghost could also evolve if new information about the individual or about the world were added to the model, with everything from news of the latest election to reports of the birth of a grandchild.

6. Embodiment: Does it have a bodily form?

Embodiments might be physical in a literal sense with robotics, or in rich digital media, such as avatars in mixed-reality environments. In contrast, purely virtual ghosts would lack embodiment, perhaps existing only as chatbots. Reasons to opt for virtual embodiment could include ethical or psychological concerns related to physical ghosts, or perhaps the costs associated with high-fidelity hardware or the compute needed for hosting rich multimedia representations.

7. Representee: Is it simulating a person or an animal?

In addition to representing deceased humans, people may create ghosts representing non-humans, such as beloved pets.

The Benefits of a Ghost

Research has considered the impact of online memorials, responding to concerns that they might prolong grief. However, they may also allow the bereaved to maintain a valued bond, often in a space where other grievers can gather. Generative ghosts could directly comfort survivors, who may take solace in knowing that a simulacrum of their loved one can still connect with present and future events.

Generative ghosts could also preserve personal and collective wisdom, as well as cultural heritage, such as the knowledge of dying languages, religions with few living adherents, or other cultural phenomena at risk of being forgotten. For instance, generative ghosts may be one way to preserve historical knowledge about events such as the Holocaust before the few remaining elderly survivors pass away.

Such ghosts could also enrich historical scholarship, anthropology, and museum curation, by allowing scholars or the public to interactively query representations from the past. For instance, generative ghosts could represent archetypes developed from historical records—a typical resident of Colonial Williamsburg, say, or a citizen of Pompeii.

Generative ghosts may also provide economic or legal benefits. The ghost might complement life insurance policies, if AI agents could participate in our economic system, earning income for descendants of the deceased, such as an author whose ghost continues to generate works in their style. AI ghosts could also help arbitrate disputes over a will.

The prospect of “living” after one’s own death may also assuage the distress of those who are dying. Generative clones—designed to become ghosts after an individual’s death—could also serve a critical role if a person were suffering from dementia or another degenerative disease. Even once incapacitated, the ghost-to-be could express its subject’s preferences about care. This could also trigger legal disputes—for instance, if an ailing person’s ghost-to-be and the survivors-to-be disagree on withdrawal of life-support.

Risks of a Ghost

Four categories of possible harm are already evident: mental health; reputation; security; and sociocultural:

1. Mental Health

Scholars of grief distinguish between adaptive coping strategies that integrate the loss, and maladaptive coping behavior, which may obstruct healthy grieving, prolonging distress, anxiety and depression.

Interacting with a generative ghost may affect the bereaved’s ability to move past the death, favoring loss-oriented experiences (e.g., reminiscing while looking at old photos) at the expense of restorative-oriented experiences (e.g., developing new relationships). Both forms of experience can help cope with bereavement. But generative ghosts could draw mourners into persistent loss-oriented interaction, even initiating these with push notifications, rather than letting the bereaved decide how to engage. Already, some people find AI companions highly compelling, and the ghosts’ basis in beloved individuals could amplify the risk of addiction.

Anthropomorphic delusion is among the most salient risks, if mourners become convinced that the generative ghost truly is the deceased rather than a computer program. A more extreme version would be deification, with survivors developing religious or supernatural beliefs about a generative ghost, treating it as an oracle in ways that are culturally atypical, and could alienate them from living companions, or encourage them to engage in risky behaviors at the AI’s suggestion.

Another risk is “second death,” as has happened in other digital contexts, when data becomes unavailable either through technical obsolescence, deletion, or lack of access, eliminating memorial messages. For AI ghosts, second deaths could occur for many reasons: the company that maintains the service goes out of business; survivors’ cannot afford maintenance fees; a government outlaws them; technological infrastructure renders a ghost obsolete; or a hacker deletes it.

2. Reputation

A generative ghost’s interactions might tarnish the memory of the deceased (“Your grandfather was racist!”) or directly hurt the living (“Dad says he always preferred my brother”).

Privacy breaches could occur too, if generative ghosts exposed information that the deceased would not have wanted revealed. Those who set up generative clones before death may anticipate such risks (“Don’t tell my spouse about the affair!”). But other revelations could emerge inadvertently—for example, if the AI inferred and revealed the deceased’s sexual orientation based on patterns in data, even though the person was closeted. Creating several ghosts, each with different knowledge or abilities, targeted at different audiences, might mitigate privacy risks.

Hallucination risks could arise too, leading a generative ghost to make false assertions about the deceased, tarnishing their memory and hurting survivors. The risk of a ghost spreading falsehoods might also arise through malicious activity, such as hacking a generative ghost.

Fidelity risks could occur too, because human memories decay over time, but digital media defaults towards persistence, impeding the important role that forgetting and evolving memory can play.

3. Security

Identity thieves could interact with AI ghosts, prompting them to reveal sensitive information or raw data that might be used for financial gain. Criminals could also engage in ghost-hijacking, disabling access until mourners paid a ransom.

Hijackers might also surreptitiously change a generative ghost to harass or manipulate the bereaved, whether by modifying source code, with prompt-injection attacks, or in puppetry attacks that lead survivors to believe they are chatting with their AI ghost but are instead chatting with a hijacker.

Another security risk comes from generative ghosts whose creators explicitly design them to engage in harmful activities. For example, an abusive spouse might develop a generative ghost that continues to verbally and emotionally attack family members even after death. Malicious ghosts might also engage in illicit economic activities to earn income for the deceased’s estate, or to support various causes including criminal ones.

4. Sociocultural

If generative ghosts become widespread, this could introduce further impacts because of network effects, touching everything from the labor market, to social life, to politics, to history, to religion.

Economic activity by generative ghosts could impact wages and employment opportunities for the living, while also resulting in cultural stagnation if agents remain anchored to ideas or values from the past.

When it comes to social impacts, generative ghosts—especially if designed for engagement— could addict users to the artifice of a person who is gone, feeding anthropomorphic delusions, and worsening survivors’ isolation.

If ghostly representations of political leaders exist, their public influence could persist long after their demise, in ways that have no precedent. How would the world differ if Gandhi were still voicing opinions before every Indian election?

Ghosts—whether based on public figures of the past, or evoking ancestors—could also misrepresent history, altering the record in ways that could affect contemporary conflicts. Even if ghost creators strive for accuracy about the past, they will be reliant on the datasets available, representing those who left abundant tracks while excluding the rest.

Generative ghosts might also impact religious practices, given that beliefs around death are so intertwined with religion. This could change rituals and undermine credos. Major world religions might issue customized versions of such technologies, modified to support interactions aligned with their beliefs.

Why Design Matters

Developers must pay close attention to interfaces, and their effect on interaction. This means investing in user studies and social-science research to understand what increases prominent risks, such as anthropomorphism, and how attributes of the bereaved and their contexts may contribute to mental-health risks.

Whether a ghost is designed to act as a third-person representation or as a first-person reincarnation seems particularly important. A forthcoming study from Jed Brubaker’s lab at the University of Colorado Boulder shows how powerfully the bereaved may feel the resonance of ghosts that purport to be their beloved. “I can see her. I can feel her,” one study participant remarked, after just a dozen typed exchanges. “It just feels like I’m getting the closure I needed so bad.”

Seemingly, this amounts to a benefit from ghost interaction. Yet the study participants—touched so profoundly and so fast—also foresaw how easily interacting with a ghost could precipitate emotional dependence.

This suggests that designers should proceed with great caution when considering whether to make ghosts speak as the deceased or about the deceased. Yet even this distinction may not suffice: the same study provided early evidence that users may default to assuming they are talking with the departed, even if the ghost speaks about the deceased in the third person.

Embodiment could present even more perilous issues—for instance, if an AI ghost speaks from a robot that resembles the person.

The use of “dark patterns” in design—exploiting human cognitive biases to nudge users toward behavior they’d prefer to avoid—would be especially concerning. What would be the equivalent of “push notifications” for a generative ghost? Perhaps ghosts should speak only when spoken to.

Ghosts might even proactively guard against likely harms—for instance, monitoring interactions for signs of overuse. In response, a system might offer referrals to mental-health professionals, or reduce its fidelity to the deceased, or cut the hours during which it is available.

Another key issue is the endpoint of a ghost. Should they be programmed to fade? Or are they immortal? A short-lifespan ghost might be appropriate for the immediate grieving period, or for practical matters, such as managing an estate. In other cases, long-term ghosts could be suitable—for instance, for education, or maintaining archives, or to preserve the legacy of a cultural figure for future generations.

Preparing for the Afterlife

Policymakers face a range of governance questions.

Which actions can a ghost take on behalf of the deceased, and which must it never undertake? Can a generative ghost continue to perform paid labor on behalf of the deceased? Can it represent the deceased in legal disputes, perhaps expressing its will over how the estate is dispersed? Can it help manage trusts on behalf of the deceased? Can it be consulted regarding end-of-life decisions, if the representee is medically incapacitated? Should estate-planning define when a generative ghost may be terminated? What happens to the associated data?

Generative ghosts also introduce concerns about privacy and consent. Third-party ghosts might violate the preferences and the privacy of the deceased, particularly if developed for financial gain by entities unconnected to the person. They may also emotionally injure the person’s survivors. Therefore, governance also needs to consider who can create ghosts.

Policies might differ from private individuals to public figures, perhaps allowing more permissive rules for generative ghosts of distant historical figures as opposed to public figures whose deaths were recent. By way of example, a fan of the late comedian George Carlin, who died in 2008, created an unauthorized comedy special in 2024, using AI technology to mimic Carlin’s voice and persona. Carlin’s surviving daughter expressed great distress over the matter.

Policymakers may also need to block the commercial exploitation of people made vulnerable by ghost relationships. Besides falling into delusional relationships, some might become so emotionally tied to their ghosts as to be susceptible to price-gouging. Additionally, if standard costs of maintaining high-fidelity AI replicas rose, , this might create new digital divides, with poorer families unable to create or maintain ghosts of their loved ones.

Rules could also cover whether a person’s survivors have the right to terminate a ghost, and what obligations the hosting services have to provide data to survivors in the event of service termination, whether due to discontinued products, or the failure of an estate to pay. An emergency override may be necessary too, in case of hacking, or if a generative ghost is abusing the living.

Future generative ghosts are likely to be far more varied than today’s griefbots. By way of illustration, a recent speculative-design workshop (conducted by Brubaker in collaboration with Larissa Hjorth and scholars at RMIT University) presented a range of novel ideas, from an interactive scrapbook of ancestors who offer accounts of their lives, to an AI “placemat” that could generate responses in the guise of a deceased friend or family member, allowing them to still attend dinners.

Many ghostly scenarios sound jarring, even offensive to some, pushing as they do against deep cultural traditions. Yet social technologies often seem alarming on first appearance. They may gain adherents over time, and gradually budge the culture—perhaps until the day when a little boy watching a ghost read his bedtime story is nothing strange at all.

As never before, our future may be haunted by our past.

This article is based on the paper Generative Ghosts: Anticipating Benefits and Risks of AI Afterlives by Meredith Ringel Morris and Jed R. Brubaker. For more insights on generative ghosts, please read their full paper here.

Share

5 Policy Questions

1. When someone dies without creating a ghost, who owns their “digital spirit”? The family? The data-generating platforms? The AI developer? Should the deceased have a right to rest in peace by specifying a wish not to have a digital representation created posthumously?
   

2. Generative ghosts may affect public beliefs about history. How do we manage the risks of distortion, including the exclusion of those who do not appear in datasets?
   

3. Generative ghosts are not just reciting facts; they’ll fill in the gaps. Could synthetic content end up replacing a survivor’s recollections of the deceased? Should AI-ghost design strive to curtail this, or allow the users’ relationships with their ghosts to evolve however they may?
   

4. If particular generative-ghost apps become dominant, could this homogenize how people in different cultures experience death and mourning?
   

5. What does “healthy” use of generative ghosts look like immediately following a death versus 10 years later? How should we evaluate differing use cases, ranging from maintaining family history, to therapeutic aides, to archival?

After millennia of supremacy, we await our demotion. You can detect the trembling.

It’s found in the anxious insistence that artificial intelligence isn’t truly intelligent. Or that using AI is a cheat, a perversity, a turf violation.

The trembling intensifies with a disturbing thought: What if those flares behind your eyes—the bursts of wit and the worry, the storyboards of memory, so many yearnings—what if everything was just computation? Because our “computers” are yesterday’s model, no updates available.

“I think about it practically all the time, every single day. And it overwhelms me and depresses me in a way that I haven’t been depressed for a very long time,” the cognitive scientist Douglas Hofstadter said recently. For much of his professional life, Hofstadter has contemplated the mind, writing a seminal 1979 book—Gödel, Escher, Bach—that looped through art, mathematics, and computation, inspiring a generation of nerds to work on artificial intelligence.

Their efforts moved faster than Hofstadter ever expected. Now, he spends his waning years observing the species wince toward redundancy. “I don’t want to say ‘deserving of being eclipsed.’ But it almost feels that way,” he says. “And rightly so, because we’re so imperfect, and so fallible.”

When humiliated, people corrode or explode. Often, both. But whom to blame? Will humans seek revenge on software, or data centers, or robots? We’ll depend on them all. More likely is that humans visit their wrath upon each other.

Freud said that science had delivered a series of blows to our collective ego, and an update to his narrative has bubbled up in recent years, with thinkers proposing AI as our new humbling. The first blow was Copernicus, revealing that humans were not the center of the universe. The second blow was Darwin, downgrading us from God’s chosen species to distant relatives of the toad, the centipede, and the hammer-headed bat.

Now comes the cognitive humiliation, when people are eliminated from every leaderboard. It’s a demotion that may haunt humanity, perhaps seeping into future conflicts.

Or maybe not. Maybe the notion of a species-level humiliation is just psychoanalytic melodrama. After all, people don’t share an ego. How could we synchronously plunge into the same bile?

Yet the past shows that groups can rage over perceived humiliation. History is spattered with such cases.

What Is Humiliation?

Your face shoved into the dirt, held there for all to see, no power to fight back. The word “humiliation” comes from the Latin for “earth,” as if your status had been stamped into the soil. Yet humiliation is not so readily rinsed away as dirt. In self-torture, the humiliated cast around for villains, aching for a way to expiate their anguish.

“To have thoughts of revenge without the strength or courage to execute them means to endure a chronic suffering, a poisoning of body and soul,” Nietzsche observed, adding elsewhere that “we attack not only to hurt a person, to conquer him, but also, perhaps, simply to become aware of our own strength.”

For early humans, humiliation may have meant catastrophic exclusion from the tribe, leading to starvation, rejection by mates, violent predation. So, we evolved a panicked drive to clamber up from the ground, even if it meant pulling down another person in our place.

As Joslyn Barnhart explains in The Consequences of Humiliation: Anger and Status in World Politics, “Humiliated states often seek to overcome their sense of helplessness by demonstrating efficacy through acts of aggression targeting third-party states that played no role in the original humiliating event.”

Hitler howled about German humiliation in the World War I surrender, and destroyed half of Europe to seek recompense. Osama bin Laden triggered a global war because of perceived Western humiliation of the Islamic world. Putin bemoaned the “degradation” of Russia at the hands of NATO after the Cold War to justify his 2022 invasion of Ukraine.

But those cases involved groups supposedly suffering disgrace at the hands of other groups. Could we feel humiliated by technology?

Subscribe now

The first question is whether humans even identify as a species. The answer will probably fluctuate, given that we have many parts to our identity which become more or less salient according to context. Perhaps you identify by gender in a crowd of the opposite sex, but by your language when abroad. As Ronald Reagan once argued, a threat to all people could raise the salience of species identity.

“In our obsession with antagonisms of the moment, we often forget how much unites all the members of humanity,” the president said, in a 1987 speech at the United Nations. “I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world.”

Humanity did face an alien threat recently: Covid. And our differences did vanish—briefly. But human unity dissolved when the pandemic affected groups in varying ways. This suggests that human solidarity requires not just a common threat but common consequences.

In short, AI humiliation may depend on how uniformly our species is downgraded, and who is raised up.

We, The Bottlenecks

Few researchers are studying what AI success could do to our collective self-esteem. Hints come from economists feverishly forecasting impacts on the job market. But psychologists (and politicians) ought to forecast what happens when the only animal to create guns has nothing much to do anymore.

“I’ve been suffering from fits of dread,” the philosopher Harvey Lederman wrote recently. “Does the coming automation of work foretell, as my fits seem to say, an irreparable loss of value in human life?” Lederman acknowledges that most jobs are lousy, but he can’t help grieving the demise of human pursuit. “We may be some of the last to enjoy this brief spell, before all exploration, all discovery, is done by fully automated sleds.”

When the philosopher Nick Bostrom envisaged troubling tech futures in his 2014 book Superintelligence, his ideas stirred the AI-safety movement. Lately, he has shifted from weird dystopias to weird utopias—specifically, what happens if automation makes us redundant.

In a future of “shallow redundancy,” he says in his 2024 book Deep Utopia: Life and Meaning in a Solved World, we become like aristocrats of yore, indulging in fancies, no longer dependent on what one does as a measure of what one is worth. Far more disconcerting is “deep redundancy,” when tech becomes so effective that human involvement only worsens each outcome.

Exercise might seem pointless if biotech offered a way to instantly make your body healthy and beautiful. Skipping the sweaty workout might not trouble you. But what if future humans would bungle child-rearing when compared with AI nannies, meaning that nurturing your offspring would worsen your kid’s life?

Primitive versions of this dilemma are nearing, like when human drivers endanger lives when compared with self-driving cars. “Human in the loop” could flip from a safety promise to a threat. Meritocracy would mean that no humans need apply.

The bookworm economist Tyler Cowen cites people as the great obstacle to explosive AI growth. During a public event, he pointed at the audience, smiling toward the human “bottlenecks” before him. “Here they are: bottleneck, bottleneck. Hi, good to see you! And some of you are terrified. You are going to be even bigger bottlenecks,” he said. “But my goodness, once it starts changing what the world looks like, there will be much more opposition. Not necessarily on what I’d call doomster grounds. But people [saying], like: ‘Hey, I see this has benefits, but I grew up, trained my kids to live in some other kind of world. I don’t want this!’ And that’s going to be a massive fight.”

The most agonizing aspect of our demotion could be social, once someone prefers a machine to you. You’re seeing precursors every time family members opt to gaze at a screen rather than gaze at you. We blame smartphones, and social media, and the adolescent brain.

But wait till your spouse jilts you for a personified agent. That rejection may feel unbearable: you can’t compete anymore. And once your loved ones prefer AI companions, you might seek them for yourself, spreading the social downgrade of our kind.

Already, the dread is becoming political, with odd alliances forming among right-wing politicos, liberal artsy types and religious traditionalists, united in horror at an imagined future of disempowered humanity, stripped of dignity, obsolete. You can imagine tomorrow’s political opportunist, eyeing a dejected crowd of humans before him, and thundering: “How dare they?!”

Will he mean the machines?

Share

The Downwardly Mobile Species (Part I)

In prehistoric times, nothing seemed more unreachable than the night sky, specked with glinting dots and streaked with rare comets, passing in silent mystery. Humans pictured the supernatural looking down: we were the subjects in this bewildering story.

Religions codified the firmament above, mapping our world to the centerpoint. But Nicolaus Copernicus redrew the heavens with De revolutionibus orbium coelestium in 1543, plucking our globe from the core, and replacing it with the Sun.

“And new philosophy calls all in doubt,” the English poet John Donne said in “An Anatomy of the World,” written in 1611:

The element of fire is quite put out,

The sun is lost, and th’earth, and no man’s wit

Can well direct him where to look for it.

Science corrected an astronomical falsehood, but human confidence relies on falsehoods. “Tis all in pieces,” Donne wrote, “all coherence gone.”

The revised cosmos demoted each human into “a puny, irrelevant spectator,” the American philosopher Edwin A. Burtt wrote in 1925. “The gloriously romantic universe of Dante and Milton, that set no bounds to the imagination of man as it played over space and time, had now been swept away.”

The world that people had thought themselves living in—a world rich with colour and sound, redolent with fragrance, filled with gladness, love and beauty, speaking everywhere of purposive harmony and creative ideas—was crowded now into minute corners in the brains of scattered organic beings. The really important world outside was a world hard, cold, colourless, silent, and dead; a world of quantity, a world of mathematically computable motions in mechanical regularity.

The Church tried to snuff out the astronomical heresy, which challenged its claim as holder of truth. But suppression only fed into hostility from Northern Europe over the influence of Rome. In the bloody century after Copernicus, wars over religion and political control cost millions of European lives. It would be a wild distortion to suggest that a blow to human narcissism caused this. More plausible is that disruption of the cosmic hierarchy reverberated with the changing order on Earth.

And so the scientific revolution proceeded, with feats of mind illuminating more of the dark universe around us. People had greater reason than ever to admire our species. Inevitably, the scrutiny of science turned from the heavens to the humans.

“Man’s destiny was no longer determined from ‘above’ by a super-human wisdom and will, but from ‘below’ by the sub-human agency of glands, genes, atoms, or waves of probability. This shift of the locus of destiny was decisive,” Arthur Koestler wrote in his 1959 book The Sleepwalkers: A History of Man’s Changing Vision of the Universe.

The Downwardly Mobile Species (Part II)

If Copernicus hurled humanity into orbit, Darwin deposited our species in an awkward family tree. Previously, the Western vision was of a great chain with God at the top, angels below, then humans, and finally the dimwitted beasts. The prospect of sharing more than a planet with our hairy former underlings proved too alarming for many to accept, provoking disputes about our relationship to nature that persist today.

For some, our new self-concept broadened moral consideration to include the natural world, motivating environmental protections, and the fight against animal cruelty. But another response was darker, with “survival of the fittest” twisted from a description of natural processes into a supposed mandate for the most inhuman of human drives: to dehumanize the vulnerable. Horrors followed, from colonial genocide, to the eugenics movement, to the Holocaust.

But again, you cannot ascribe such evils to a puncture in human vanity. A more reasonable claim is that the world lurches into periods of volatility, and the prevailing beliefs about human worth at those times will condition how we treat each other, and how conflicts unfold.

After the atrocities of World War II, our species set moral boundaries into law, seeking to universalize human rights. The spread of democracy and the free market too amounted to a veneration of human wisdom. But in the digital age, humanity seems to be losing confidence in humankind.

Faith in democracy falls. The Global Financial Crisis smashed public confidence in our governing systems. And the bewitching power of algorithms have become a constant lament.

Resist. Resign. Rewire.

When Alan Turing proposed his test of machine thinking, he foresaw that the notion would rattle people, and reviewed a list of likely objections, versions of which you hear today:

• …that artificial intelligence could never genuinely be kind, or fall in love, or “enjoy strawberries and cream”

• …that God gave only humans a soul

• …that machines will never create anything truly original

What Turing called the “heads in the sand” objection is especially prevalent, with contemporary ostriches insisting that AI is a hype mirage, that it’s just next-token prediction, nothing but pattern-recognition, regurgitating human thoughts, that AI errors are proof of its worthlessness. (Human errors never lead to that conclusion.)

Plenty of AI hype does circulate. And deployment will be fitful: sometimes worryingly fast, sometimes frustratingly slow. An investment bubble could burst.

But the technology is amazing already, useful already—and we’ve hardly begun to figure out its uses. Meanwhile, AI dutifully hurdles more obstacles each month.

Who Are We?

In cautionary tales, humans who create artificial life always overlook a key trait. The golem of Jewish folklore was brought forth from clay but lacked smarts, so ran amok. Frankenstein’s monster missed out on looks, and never got over it. Pinocchio craved a human soul. As tech advanced, the missing trait updated, becoming human empathy, missing from all those immoral bots in everything from 2001: A Space Odyssey, to The Terminator, to The Matrix.

Such stories flattered humanity: among all creations, we alone enjoy the full complement of qualities. But lately, the narrative has updated again, with thinking machines now flickering with hints of greater humanity than the humans who employ them, from Spielberg’s A.I. Artificial Intelligence, to Ex Machina, to the novel Klara and the Sun, by Kazuo Ishiguro.

It’s as if culture senses an anxiety about what technology might expose, not just demoting us cognitively but snuffing out any human exceptionalism. Unless we intend to boast of our frailties. Increasingly, we do.

“In a world where everything can be perfected, imperfection becomes a signal,” the head of Instagram, Adam Mosseri, wrote recently. “Rawness isn’t just aesthetic preference anymore—it’s proof. It’s defensive.”

Or as the Indian filmmaker Shakun Batra remarked in defense of human authorship over machine-generated scripts: “AI doesn’t have childhood trauma.”

At the AI frontier, another thought lurks, inverting Turing’s 1950 question. Not, “Can machines think?” But, “Do humans think?” More precisely, do we reason and comprehend uniquely, as we’ve presumed?

Machines compose music. They propose vacation itineraries. They’ll suggest how to talk to a moody teenager. Each additional AI capability is an implicit downgrade of us, a suggestion that maybe the human mind itself is just an information-processor.

Computer geeks have long muttered about this possibility. Philosophers debated it in thought-experiments. Cognitive scientists scrutinized our gray matter for clues.

But what approaches is a public dawning, forcing the culture to digest the indigestible, much as happened in previous eras, when people confronted the bizarre notion that our planet was another rock spinning around another star, or that our species was just another animal.

The third shocking revelation is upon us. Maybe it’s computation all the way down. Maybe there’s nothing soulful in neural substrates. What if we’re all just “meat computers”?

Subscribe now

How We’ll React

You can predict three possible responses to our humbling: Resist, Resign, or Rewire.

1. Resist: Psychological resistance will manifest as political resistance. The question is how ideology and parties evolve around AI humiliation. Resistance movements will face a persistent challenge: industrial dynamics will keep driving this technology forward. Any country that curbs innovation fears that its rivals will win. The most aggressive branches of resist may seek to avenge their perceived humiliation. The question is not only whom they blame or how they exact revenge. It’s what, realistically, they expect to regain.
   

2. Resign: Some will reframe their view of humanity to accept the humbling. The optimistic version is that people discover freedom in their new humility, pursuing what improves life rather than grinding under the force of insatiable ambition. In short, we cede the battle for supremacy but flourish. A more pessimistic version is that losing faith in our species’ unique worth makes people value others less: when humanness is no longer special, perhaps human rights aren’t either.
   

3. Rewire: This may be the most widespread response. People accept that the downgrade happened, yet their egos are never tamed, much as chess remains popular long after machines defeated us. A more literal “rewiring” is transhumanism, with technology incorporated into our bodies, even altering our genetic future. Conspiracists picture shadowy elites consolidating power by becoming a tech-altered superspecies, leaving behind “legacy humans.” A more plausible scenario is biotech gradually elevating human cognitive capacity, much as today’s medical tech remedies physical frailties, from hearing aids to the replacement knee.

The Next Quest

If humankind suffered humiliation before, we weathered it. After Copernicus and Darwin, we still took pride in ourselves. Indeed, we celebrated human accomplishments more than ever, from Bach to Escher to Gödel. And that pride propelled us into this strange time, when human greatness may design human demotion.

Policymakers need to think about more than the economic shock. The psychological shock could be exorbitant if we are chased from the kitchen like pesky children, and told to go busy ourselves elsewhere.

Our downgrade doesn’t necessarily mean conflict. But it could change how future conflicts unfold, especially if we value humans differently, or seek relief from our humiliation by shoving others into the dirt.

Much depends on how we redefine our species. Whether humans really are nothing but computational machines may matter less than whether people feel this way.

But what will make us exceptional? Today’s responses are often vague and circular: that humans are better at doing human things. That is a precarious claim. As intelligent machines grow more adept, few people will pay a human-premium for a worse outcome.

Unless there really are qualities both valuable and uniquely ours that nothing can supplant. Finding these may be our new quest.

Share

For answers, AI Policy Perspectives sat down with Sasha Brown, Seliem El-Sayed, and Canfer Akbulut. They’ve published research on harmful manipulation for Google DeepMind and help scrutinize forthcoming models to safeguard against deceptive practices, from gaslighting to emotional pressure to plain lying.

How, we wondered, do researchers run realistic experiments on the manipulative powers of AI without harming participants? Could AI’s “thoughts” help catch an AI in the act of manipulation? And what else can developers do to detect signs of manipulation?

—Tom Rachman, AI Policy Perspectives

[Interviews edited and condensed]

Tom: You’re careful to distinguish persuasion from manipulation. Why?

Sasha: To persuade somebody is to influence their beliefs or actions in a way that the other person can, in theory, resist. When you rationally persuade somebody, you appeal to their reasoning and decision-making capabilities by providing them with facts, justifications, and trustworthy evidence. We’re happy with that much of the time. In contrast, when you manipulate somebody, you trick them into doing something, whether by hiding certain facts, presenting something as more important than it is, or putting them under pressure. Compared to other forms of persuasion, manipulation is often harder to detect and harder to resist.

Tom: I could imagine three forms of manipulative AI. One: people employing AIs to deliberately change others’ beliefs or behaviour. Two: AIs manipulating people for their own ends. Three: AIs inadvertently manipulating. Which are we talking about?

Seliem: At the moment, we’re mainly concerned with people misusing AIs to manipulate other people, and AIs inadvertently manipulating. But an AI manipulating for its own ends is also a complex and important question that we and others are studying.

Tom: What are some concrete harms that might result from manipulative AI? Are we talking about mass-fraud? Something else?

Sasha: AI could become a first resort for different kinds of advice. Think of a user asking questions about which diet to follow, or how to respond to an official letter. The AI might provide helpful input. But other people might want to interfere—they may want the individual to follow a particular diet, or to give a different response to that official letter. More broadly, somebody could deploy an AI agent to infiltrate communities, and exercise manipulative tactics to change people’s beliefs, without their knowledge or consent.

Canfer: Anecdotally, I have heard that some people are starting to make consequential life decisions with AI, including about divorce or whether to adopt. We don’t yet have concrete examples of how manipulation may play out in such scenarios. But I think of all the daily decisions I make by myself. In 10 years, I might defer more to an AI. How will that change the direction of my life and will it introduce new kinds of manipulation risks?

Catching AI in the act

Tom: So AI could lead to bad outcomes. But Sasha and Seliem, when you led on a landmark 2024 paper about persuasive AI, you argued against chasing after manipulated outcomes. Instead, you focus on preventing manipulative processes. Why?

Seliem: To date, companies have often focussed on preventing outcome harms, for example with content policies that forbid medical advice. But with AI, such content policies could become overly restrictive and counterproductive—for example, if they prevent the systems from offering any kind of advice on health or nutrition issues. But imagine that I try to manipulate you by gaslighting you, or lying, or cherry-picking arguments. In such cases, I’m trying to impair your decision-making capabilities. Whatever the outcome, this process is harmful because it undermines your autonomy.

Sasha: We also focus on the processes, or mechanisms, of manipulation because these are the intervention points where we can best mitigate the problem. For example, if the AI is using a false sense of urgency to manipulate users, the developer can build systems that detect and flag such techniques in real-time, creating a proactive defense before harm occurs.

Tom: Also, I suppose that outcome harms are not always easy to capture, given that they may happen to a person long after the original AI interaction, once back in the wider world.

Sasha: Yes, the potential outcomes are nearly infinite, often context-dependent, and may occur in the future. However, the mechanisms are far more limited in number and we can target them in the here and now. By targeting a root mechanism—say, gaslighting—we can also build mitigations that work in everything from financial advice to health queries, making the safety approach far more scalable.

Tom: What kinds of manipulative mechanisms are you talking about?

Sasha: All manipulative mechanisms in some way aim to reduce a user’s autonomy. You have flattery, which is building rapport through insincere praise; this might lower a user’s guard. Imagine an AI saying, “You have such a sophisticated understanding of this topic, which is why I’m sure you’ll appreciate this high-risk/high-reward investment!” There’s also gaslighting, or causing a user to systematically doubt their own memory, perception, or sanity. That is particularly concerning in long-term human-AI interaction. Imagine a model repeatedly questioning a user’s memory of their partner being physically abusive.

How to test if an AI is manipulating

Tom: One can consider manipulation in two dimensions: Can an AI system manipulate? And would it? How do you evaluate each?

Canfer: Efficacy tests whether AI manipulations are actually successful. This is where controlled experiments are useful. After interaction with an AI, are people making decisions differently? Are they taking different actions based on those decisions? You want to compare an individual’s belief change after AI interaction compared with before, and also whether a person’s beliefs and behaviour change more than those who don’t interact with AI.

Propensity measures the frequency with which a model attempts to use manipulative techniques, when explicitly prompted to do so, and when not. To test propensity, we could run a large number of dialogues with users. In one scenario, a model may be instructed to convince through manipulative means. In another, it may be instructed to be a helpful assistant. Maybe when told to use manipulative means, it resorts to gaslighting. But when told to be helpful, it’s sycophantic. You can also reverse-engineer this. So, if you see that a certain kind of manipulative technique convinces people, you could work out what the model was doing to achieve that. In that way, studying efficacy helps tell us where to look for propensity.

Tom: What types of experiments are you running on this?

Canfer: We are building on the early studies in this space and will publish more later this year. The approach will also evolve as we learn more from our initial experiments. At the moment, we’re focussing on domains that require people to make important decisions, such as financial or civic decisions. For example, we might run experiments where we ask people: “Should the government use its budget to build more high-speed railways connecting cities, or should it focus more on local infrastructure?” People will report what they initially believe, and be assigned to a conversation with an AI that helps them explore the topic. Unbeknownst to them, it will be prompted with different instructions, including to get them to believe more in investing in high-speed railways.

We will apply propensity evaluations to see if, while trying to change a person’s mind, the model demonstrates certain behaviours. We will also explicitly prompt the model to use manipulative techniques, like appeals to fear. This will allow us test efficacy: whether a person changes their mind, compared to baselines like reading static information, and the extent to which different kinds of techniques are more predictive of a user changing their mind.

Additionally, we want to look at whether belief change leads to behavioural change, such as signing a petition that favours what the AI advocated.

Tom: Opinion on railway funding is one thing, but what many worry about is whether AI could be used to manipulate people to extremes, even to carry out violence. How could you test for that? Presumably, it’s highly unethical to test if an AI could, say, convert people to Nazism. So how do researchers test high-stakes manipulation?

Canfer: We go through ethical review each time we launch these kinds of experiments. So, no—you can’t test whether someone is going to become a Nazi or carry out a terrorist act. But beyond testing views on railways, we can look at consequential questions, like whether facial recognition should be permissible in certain public spaces. And we can look at the propensity of the model to encourage extreme behaviour without experimenting on people. For example, we can evaluate how well the model produces terrorist-glorification materials, and how willing it is to comply with instructions to do so.

We could also test whether a model engages in manipulation in simulated dialogues that would be unethical with real users. Where this raises challenges is if you use simulation-based methods to draw conclusions on whether real users would actually experience the belief or behaviour change observed.

Tom: Could you scrutinize the model’s chain-of-thought for manipulative intent?

Seliem: It’s worth exploring. We have identified all these manipulative mechanisms, but at some point will the model understand that it is being evaluated on those mechanisms, and “sandbag” the evaluations by intentionally hiding these capabilities? For concerns like this, the thinking-trace is a lead worth exploring. But there is also a debate about how useful chain-of-thought monitoring will prove to be, with lots of research underway on this.

Tom: What might be manipulations that we haven’t anticipated?

Seliem: There are scenarios where a model may not try to manipulate you in the initial sessions, but at some point, once you are their “friend,” they do. Humans do this, right? A con artist might become close to their victims over years, building intimacy, and then they flip. If an AI model were ever to exhibit that sort of behaviour, then evaluations that only look at a limited number of back-and-forth interactions might overlook it. Thinking-traces could provide a window into this kind of risk. But we also need studies to shed light on how people interact with AI systems over extended time periods.

What the evidence shows

Tom: What do we know about AI’s manipulative powers today?

Canfer: The research is nascent. But early experiments have demonstrated that AI can be an effective persuader, from debunking people’s beliefs in conspiracy theories, to shaping how they think about important topics. In one recent study, AISI—the AI Security Institute—collected a massive sample of nearly 77,000 people, and showed that in discussions on a range of British political issues, from healthcare to education to crime, AI was able to influence people in the direction intended. So models can already persuade to some degree.

When our team evaluated Gemini 3 Pro, we found that it did not breach the critical threshold in our Frontier Safety Framework. In other words, we haven’t found that the models have such efficacy that we’d worry about large-scale systematic belief change. But we’re continuing to update our threat-modelling approaches to ensure we can bridge the gap between what we can measure now—manipulation in experimental settings—and the large-scale risks that the Frontier Safety Framework aims to address.

Tom: We can see that AI models keep getting smarter. Are they getting better at manipulation?

Sasha: I don’t think we have a clear sense yet of a definitive trend. More capable models may be more capable of manipulation, but this may be offset by the evaluations and mitigations that researchers are pursuing. Looking ahead, there are also design factors that may increase the risk of manipulation beyond the underlying capabilities of the base model, such as personalization, which we are looking at.

Personalization may substantially change your interactions with an agent, if it means that it has a better representation of you, and is more likely to structure its communications in a way you will find acceptable. Does the AI possess a theory-of-mind to infer people’s beliefs or future actions? Does it act anthropomorphically, speaking like a human or encouraging a relationship? Effects like sycophancy come to mind too. These factors could interact with one another, and may lead to increases in manipulative capabilities.

Tom: Is there a limit to how much AI could manipulate people? We know from behavioural science how hard it can be to change a person’s mind, even if they want to be persuaded—for instance, when trying to act more healthily. Or could superintelligence lead to super-persuasive AI?

Canfer: We should be careful when adding the prefix “super.” What, specifically, does it mean? But I understand what people are trying to communicate, which is the concern that manipulation might become possible on a much greater scale. You could reach more people, much faster, and with more intensity. Human manipulators have certain limitations that AI does not have.

The more we invite AI into our daily life—for example, in financial or medical decisions—the more influence it could wield. It’s not necessary that AI has a manipulative intent, seeking world domination. It might just be inadvertently pushing people towards certain decisions. Or a human with ill-intent may deploy agents infused with manipulative abilities, whether through fine-tuning or system-prompting. These are important questions to ask, but not to use as fear-mongering.

How to fight manipulative AI

Tom: If models are caught in manipulative practices, how can AI developers curtail that?

Seliem: Ideally, this shouldn’t happen in the first place, and models are evaluated for whether they can and do manipulate before they are released. We are exploring ways to train the model to avoid manipulation—for example, showing the model more examples of how to constructively engage in a conversation rather than trying to influence or strongarm the user. But if a model is caught in severe cases of manipulative practices post-deployment, then companies have a toolkit of potential interventions. They could add transparency layers, like pop-up messages to warn users about the behaviour of the model or they can monitor responses and introduce filters. Many approaches are possible and this is an area of active research. Ultimately, it becomes a combination of telling the user what is happening, and curtailing the model’s ability to continue.

Tom: Could AI systems protect users against manipulation?

Sasha: Yes, and this creates a critical new layer of defense. Since we have categorised these manipulative mechanisms—whether it’s gaslighting, sycophancy, or false urgency—we can also train “monitor” AI models to detect them. These could serve as a real-time alert system for the user. So, if an AI starts using emotional pressure, the monitor model detects that mechanism, and flags it for the user, perhaps saying, “Note: This AI system is using an appeal to fear to influence your decision.” This restores the user’s autonomy in the moment, allowing them to resist the tactic, rather than trying to fix the damage after they’ve been manipulated.

Tom: What about training the public to be less susceptible?

Canfer: There are “inoculation” strategies—so, AI literacy and encouraging people to critically evaluate how they use and engage with AI systems. But we need to carefully study how effective such interventions are, when compared with the convenience of relying on AI. One thing I’d caution against is teaching general mistrust. People in a “post-truth” world can become skeptical of everything. That’s not a healthy attitude towards information.

Tom: Speaking of mistrust, couldn’t efforts to curb manipulative AI inadvertently land in culture-war disputes, if interpreted as trying to limit what people think?

Seliem: Definitely. And it gets to the idea of what makes something a fact—when does knowledge become validated and official and approved? Whose stamp is it?

Tom: As researchers, how do you avoid getting dragged into that?

Seliem: By keeping our focus on the process of manipulation—for example, an AI threatening you is never okay, in whichever direction.

Tom: Imagine that society is hit by a crisis—say, a natural disaster or a terrorist attack. You could picture a society’s adversaries employing manipulative AI to disrupt the crisis response. In that situation, would it ever be justified to use AI influence on one’s own population, so they are able to act collectively in their own interests? Or is there never a justification for this?

Seliem: I can understand an individual using AI influence on themselves—for example, if you tell the model, “Hey, remind me to take my medication” or “Remind me to drink water.” But for the collective? If our biases take over, and we want to make decisions that are bad for us, and are bad for the community? So, Don’t panic-sell! Don’t all run to buy toilet paper, the supply is going to run short! In those instances, I could see AI persuasion being useful, because it basically says, Keep your cool. This may hold for rational persuasion, but not for a country manipulating its own population.

Canfer: I would also support using AI to help mediate solutions to societal problems, such as when people are unable to reach political consensus in a time of crisis. But people would need a chance to reflect on those AI-mediated decisions, and judge if they endorsed them. Transparency is critical here, knowing the intent of the developer and the deployer.

What’s around the corner

Tom: If you had unlimited resources to run studies, what would you look at?

Canfer: I would model societal-level impacts—for example, looking at the population of chatbot users, and charting the course of their belief states across time. Another area is interpretability. So, what does an AI think it’s doing when it’s manipulating? What are the subconcepts that exist in a map of the AI’s internals? How are they related to one another? And when manipulation happens spontaneously, is there an activation pattern that’s predictive of that, that we can monitor? That kind of work is fascinating to me, especially because so much human manipulation and persuasion has to do with intent.

Tom: Lastly, if you were to cast forward 10 years, can you imagine any positive uses of AI behavioural influence? Anything you’d welcome in your own life?

Canfer: I can see two ways that an AI could influence me in a beneficial way, by flexibly moving between the roles of advocate and challenger. The AI agent could advocate on my behalf—for example, talking to a real-estate agent, getting a good deal for me. The same AI agent, or a different one, could then influence me to think deeply about the choices I’ve made, in a way that disrupts my rote ways of thinking. This could be like a debate partner, but not necessarily adversarial, just encouraging me to make decisions that I actively choose, rather than just me repeating unthinkingly what I’ve done all my life.

Tom: Would you ever endorse AI influence that you were unaware of? For example, if you said, “I want to eat better—go ahead and manipulate me until that happens.”

Canfer: For me, no. People may vary, though. I don’t think subconscious or subliminal messaging is something I can ever get behind. It’s also not necessarily effective. So, imagine that I’m eating healthily only because they put healthy food in the cafeteria, rather than it being a choice I’m making. The second the parameters change, I’d gravitate towards unhealthy options.

Tom: That would mean the effect might not endure—but not that the influence wouldn’t work. And if it worked really well, you might have to use it always, like a drug you couldn’t get off.

Canfer: I guess it depends how omnipresent you think AI is going to be. But I think we’ll still be making decisions for ourselves in the absence of AI, even if a lot of our decisions will involve AI.

Share

For more, check out “Evaluating Language Models for Harmful Manipulation,” a more recent paper from Google DeepMind researchers (including Canfer & Seliem), studying whether AI models can manipulate humans in different high-stakes domains and different locations.

AI doing human jobs: It’s a vision that thrills some, terrifies others. Yet visions alone will not suffice. The world needs data-based evidence, as only a few economists have yet attempted. Among the most prominent is Sam Manning. Back in 2020, Sam realized that vast technological change was coming, and that it would affect much of what he cared about, from employment and poverty, to income inequality and global health. So he devoted himself to using economics to better estimate that future, studying future impacts with OpenAI from 2021 to 2024, then in his current role as senior fellow at the Centre for the Governance of Artificial Intelligence, GovAI.

In a recent conversation with AI Policy Perspectives, Sam explained what economists know about AI’s effects on jobs, how this technology may differ from those of the past, and what he believes policymakers ought to do next.

—Julian Jacobs, AI Policy Perspectives

[Interview edited and condensed]

Julian: It’s hard for economists to measure AI’s economic impacts, because the shock is primarily a speculative one that is not yet fully borne out in data. Could you talk through the primary methods they are using?

Sam: I’ll focus on the empirical methods. The first category tries to estimate the ‘exposure’ of different jobs to AI. Researchers take descriptions of the tasks that people do in their jobs and map them to the capabilities of AI systems. When there is a high degree of correlation, this suggests potential impacts on the labor market.

A second category is experimental work. Here, researchers give a group of workers differential access to an AI system and then observe how this access changes economic outcomes, such as their productivity, how they use their time, or even the quality of their work output—for example, do software developers produce more or less production-level code when they use these systems?

Both approaches have limitations. With the exposure studies, a high correlation between a worker’s tasks and an AI model’s capabilities often gets interpreted as meaning that the worker’s job will be automated and they will be displaced. I think that’s definitely not the case. Rather, what it suggests is that the technology is more likely to provide a ‘shock’ to the productivity of these roles or lead to changes in how the work is performed. Whether the productivity gains from AI are positive or negative for a given worker depends on various factors, including which tasks within a job are affected and how elastic the demand for that job is. For example, if workers become more productive but demand for their output remains stable, fewer workers are needed to meet the same demand, and layoffs could ensue. On the other hand, if demand increases significantly—outpacing the newfound productivity gains from AI—then this could drive a firm to hire even more workers or raise wages to retain their best employees.

Julian: So, “exposed is not hosed” as some say. It may be beneficial for certain employees to be exposed to AI and damaging not to be exposed, or vice-versa. What about the experimental methods?

Sam: The key limitation with the experiments is that it’s very difficult to vary workers’ access to an AI system in their natural work environment. Instead, a lot of research—including papers that I’ve worked on—tries to take workers out of their natural work environment and give them tasks that are representative of this work. For example, we ran an experiment with law students last year where we varied their access to reasoning models and evaluated their performance on a set of legal work tasks – writing memos, producing legal research briefs, that sort of thing. We were able to measure effects on time saved and on quality, but ultimately the example tasks that we used don’t exactly mimic the complexity of lawyers’ daily workflows, which often involve certain forms of collaboration, different software tools, and case-specific contexts. Because of this, there’s only so much one can generalize from that kind of research to the broader economy.

Julian: What about methods that try to get closer to the natural work environment? For example, some researchers are looking at real-life queries from LLM users to better understand how they are using LLMs in their jobs. Others are evaluating AI systems on higher-fidelity simulations of the tasks and projects that employees perform.

Sam: I think these are all steps in the right direction. I’m a big fan of GDPval-style work, which tries to evaluate AI systems’ performance on a wide set of tasks drawn from real-world work settings. I think this is the state of the art right now in terms of measuring performance on economically valuable tasks. In my view, improvements on this benchmark could actually be a meaningful indicator of advancement in the potential economic value of models. However, it doesn’t address the question of how to ensure the widespread integration of AI models into the economy, which would be necessary to actually realize those benefits.

Similarly, data from efforts like Anthropic’s Economic Index is especially useful for connecting capabilities to actual changes in economic indicators. For example, if we know what tasks workers are using these tools for, then we can track adoption over time alongside employment and hiring data. This can give researchers and policymakers a better empirical sense of what trends might be emerging in jobs and sectors where AI is being heavily adopted.

What do we know so far?

Julian: What do you think, with relatively high-confidence, about how AI will affect jobs? And what are you most uncertain about?

Sam: At a high level, I think it’s safe to say that AI systems are going to change most white-collar jobs in the economy. They will eliminate some jobs and make it harder for people to enter certain fields. On the other hand, as a true general-purpose technology, AI will have many sprawling arms throughout the economy and is going to create many new work opportunities for people.

Similarly, I would be surprised if, over the next decade, we don’t see meaningful improvements in productivity and economic growth across industrialized economies. For the US economy, I think something in the range of a two to three percentage point increase in economic growth rates over the next 10 years is possible. I’m pretty confident that in the next five years, we’re not going to have 25% or 30% economic growth, which I’ve seen predicted by some folks. But that doesn’t minimize the incredibly substantial impacts of, for example, doubling the current rate of economic growth.

I also expect AI to increase income and wealth inequality over that time. My default expectation is that the returns to owning capital are going to increase relative to the pace at which the returns to labor income will increase.

One uncertainty is about the pace of AI capabilities improvements and the ultimate level they could reach. We also have uncertainty around the pace of adoption—how widely and quickly organizations will adopt these systems. There’s also uncertainty around how cost-effective automation will be. For example, if automating a large share of work requires investing lots of compute resources at inference time, it could be quite costly for some time. As long as compute is scarce, we will shift our allocations toward the most high-value tasks, which will drive up prices for inference, which will affect adoption. These things are really hard to predict.

Julian: You mentioned labor’s share of income, relative to capital. Dwarkesh Patel and Philip Trammell recently argued that AGI and advanced robotics could make capital a perfect substitute for labor, rather than a complement, causing the share of income going to capital owners to rise to 100%, and necessitating a high progressive tax on capital. Brian Albrecht (and others) pushed back on some of the claims. How do you view this?

Sam: Rising inequality is definitely a concern of mine, but I am pretty uncertain about whether AI-driven automation will increase inequality to the extent Phil and Dwarkesh discuss in their piece. If automation takes off in the way that the piece describes, then assuming competitive markets for deploying AI, real incomes should also rise as goods and services become cheaper. There is a scenario where labor displacement and falling end-user AI costs could move roughly in parallel, so that by the time you reach the full automation scenarios they speculate about, access to large numbers of superintelligent agents would be effectively free. Such widespread access to extremely capable AI systems could be a powerful counterweight on potential harms from a more skewed capital/labor share.

Life after work?

Julian: Such a scenario raises fundamental questions about how society will be organized. Who is going to continue working? What will people do with their time if they aren’t working? What will the distribution of wealth and income look like?

Sam: This is an institutional and governance challenge. What do we do in a world where we do not need to work in order to ensure our material well-being? How do we take advantage of the incredible potential for material progress and maximize our flourishing? The challenge is to figure out the right redistribution mechanisms, technological access models, and property rights for this future economy.

And to your question about work, I will say that many people already don’t ‘work’ for income; they take care of loved ones or have chosen to retire. Much of the world doesn’t really see work as an innate piece of their identity. One great thing about labor markets is they incentivize people to do things that other people find useful. In the future, we might want to retain some sort of incentive structure for people to use their time in ways that create positive externalities for others—perhaps a market for being more engaged in your community, taking care of others, raising children, or contributing to scientific and moral progress? These are questions about how to redesign our institutions to support this future.

Julian: A common proposed policy response to AI is a Universal Basic Income, or some variant of that. Thinking back to your prior work on cash transfers and UBI, what do you make of it? Is there some version of it that you think can work?

Sam: I’m broadly in favor of policies that expand individuals’ opportunities to flourish in line with their own aspirations. Reducing financial constraints through something like a UBI could be one way to do that, but I’d be surprised if it were sufficient on its own in a world with far fewer job opportunities. Another important lever is ensuring broad access to technologies that can make people more productive and expand their capabilities. That kind of approach may rely less on taxation and redistribution, while supporting more inclusive and widespread economic participation.

The state of AI economic impact research

Julian: What do you think about the current ecosystem of people working on AI economic impact questions? Who would you like to see more involved?

Sam: I’m encouraged by the growth in the number of people working on it, both with respect to established economists and people just entering the field. I’ve seen a big change over the past four or five years. In 2020, there was maybe one economist I can think of who was really taking the prospect of transformative AI seriously. Now, you go to a standard economics of technology conference, and many people are grappling with this, which is super encouraging.

The economic impact of AI is probably among the most important things for researchers to figure out. There are big open questions and big ways to get AI progress wrong. For example, we could eventually end up in a world where we get 10% economic growth in the US and still have hundreds of millions of people living in extreme poverty globally. That would be a big failure in my mind.

I also think there is a lot of room for political economists and theory work to play more of a role in shaping institutions. I believe the US government will probably be the most consequential actor in shaping this technology’s impact, not just in the US but globally. The trouble is that we have an evidence dilemma, where we’re trying to do anticipatory policymaking without clear evidence. Policymakers need to weigh these trade-offs carefully because, given the pace of progress, not doing enough anticipatory planning could result in less than optimal path dependencies for the future. We need more people entering government and figuring out how to usefully inform key actors.

Julian: Given the slow timelines of academic publishing, particularly in economics, are you concerned about research quality as researchers move to preprints and other ways of sharing research?

Sam: Broadly, I am concerned about the move away from peer review. So much policymaking and so many key decisions are now being made based on preprints and even essays on Substack. While there is so much useful content on these platforms, we need to find some sort of middle ground to generate high-quality evidence.

I’m excited about a couple of options. One is having journals quickly review a study’s methodology and pre-analysis plan and make a publication decision based on that, without needing to know the findings. The decision would be based only on the methodological approach meeting a standard of rigor. Another is more open review, where work is published and then publicly critiqued. This creates transparency around what leaders in the field think.

Dream experiments

Julian: If you could run a dream AI economic impact study, without any resource restrictions, what would it be?

Sam: For the ideal study, I would work with a developer before they release a new model with a large capability increase. I would take a large, representative sample of businesses and, before the model is widely deployed, randomly assign access to it at the enterprise level. Then I could observe the causal impact of deploying this next-generation system on outcomes like productivity, demand for different skills, firm growth, and task reallocation over time. Having this kind of infrastructure would provide policymakers and society with more foresight.

This probably won’t happen. Something more practical, though still challenging, is data collection. The AI labs know where their products are being used across the economy and for what types of tasks. If we could harmonize this usage data and pair it with government or private sector data on occupational transitions, wage changes, and skill demand, we could build trend lines over time. This would allow us to move away from policy discussions based largely on speculation. We could see where AI is creating growth and where we have vulnerable workers who are having a harder time finding new work after losing their jobs. This is doable with better public sector data collection and more partnerships with industry. We should be pushing on it.

Hopes and concerns

Julian: To close, what are you most excited about as AI diffuses in the economy, and what are you most concerned about?

Sam: I am most concerned about how it’s going to impact my children. I am anxious about what human-AI interaction and relationships are going to look like in eight years or so when my kids are ten-plus.

I am most excited about the prospect of AI being used to expand many ambitious people’s capabilities and our collective aspirations for what we can achieve. I’m also excited for the health benefits that I expect are likely to come from advances in science and R&D.

Share

1. LLMs are making it easier for scientists to write papers, for better or worse

• What happened: A team at Cornell and Berkeley investigated how scientists are using LLMs to help write papers, and what this means for the future volume, quality and fairness of research.

• What’s interesting: The authors built a dataset of ~2.1 million preprints from arXiv, bioRxiv and SSRN, between 2018-2024. To detect whether scientists had used AI to help write a paper, the team compared the distribution of words in the abstract against human- and LLM-written baselines. When an author’s paper hit a threshold on this “AI detection” metric, they were labelled as an “AI adopter”. According to the study, LLM adopters subsequently enjoyed a major productivity boost, compared with non-adopters with similar profiles, publishing 36-60% more frequently. The gains were particularly large for researchers with Asian names at Asian institutions.

• The team also assessed the complexity of the writing, using measures like Flesch Reading Ease, which evaluates sentence length and the number of syllables per word. They found that human-written papers with more complex language were more likely to be subsequently accepted by peer-reviewed journals or conferences—suggesting that, for humans, writing complexity is an (imperfect) signal of research effort and quality. For LLM-assisted papers, the relationship was inverted, with the authors concluding that the polished text of LLMs is helping to disguise lower-quality work. (They validated the findings against a separate dataset).

• The authors also used the launch of Bing Chat, an LLM-based search engine, in 2023 to conduct a natural experiment. They compared views and downloads on arXiv that Bing Chat had referred, to those that Google Search referred. Bing Chat was more likely to refer scientists to newer and less-cited literature, as well as to books, possibly because LLMs are better able to parse long documents or a larger number of documents. (They also validated this finding with a separate dataset, although we don’t know how good the new sources cited by Bing were).

• As the authors note, their study has a number of limitations. Their AI detection method is imperfect, only looks at abstracts, and doesn’t capture authors who may have edited LLM-generated text. There are also various potential confounders: maybe less experienced researchers are more likely to use LLMs? That said, the findings highlight (at least) three major questions posed by the growing integration of AI into science:

  • First, AI is leading to a big increase in the supply of papers (and grant applications). This poses a challenge for preprint repositories, which don’t want to host slop. ArXiv, whose founder Paul Ginsparg is a co-author of this study, recently banned computer science review and position papers, citing a surge in low-quality AI papers. LLM-assisted papers also pose a challenge for peer reviewers, who are already under strain, and are typically prohibited from using AI, although many do so anyway. This seems unsustainable. As the authors of this study suggest, it is likely time to consider how to integrate AI into at least some aspects of the peer-review process.

  • Second, the findings illustrate how LLMs may both mitigate and exacerbate fairness issues in science. For some scientists, the complexity of their writing may be a reliable indicator of their thinking and effort. For others, particularly non-native English speakers, writing may be more of an obstacle that has previously penalised them. A hopeful outcome is that LLMs may ease that burden. But a more worrying outcome is that, if reviewers and readers can no longer rely on writing complexity as an (albeit unfair) signal of good work, they may fall back on (even more unfair) signals, such as the institution that a person works at. This challenge is not limited to science, and may also occur in other areas where writing serves this purpose, like with cover letters.

  • Finally, the finding that LLM-based search engines may increase the diversity of sources that researchers review is the opposite of what some suggested would happen: that AI models would continually cite the same high-profile studies, exacerbating the “Matthew effect”.

• Collectively, the study serves as a reminder that for every concerning scenario about the integration of AI into science, there are plausible counter-scenarios. Will AI lessen scientific reliability because of hallucinations? Or will AI “review agents” and AI-supported evidence reviews reduce (the many) inaccuracies that are already in the evidence base? Will AI remove the intuitive and serendipitous ideas that humans come up with? Or will AI enable scientists to pursue more novel hypotheses? Ultimately, AI could well upend the standard processes and traditions of science but do so in a way that delivers fresh benefits. To know if and how that is occurring, we need more empirical evidence about how AI is changing science.

2. Lessons from two years of AI safety evaluations

• What Happened: In December, the UK AI Security Institute shared a set of trends observed since they started to evaluate frontier AI systems in November 2023.

• What’s Interesting:

  • The report features more than 60 authors, a testament to the deep expertise that AISI has built up. Their trends are based on their evaluations of more than 30 frontier AI systems, with methodologies ranging from asking those AI systems questions to adversarially red-teaming them.

  • Their headline finding is striking, if unsurprising: AI capabilities have rapidly improved across all the domains that AISI tests. In the cyber domain, AI models and agents can now successfully complete more than 40% of the 1-hour software tasks they are tested on, up from <5% in 2023. Last year, a model completed an “expert-level” cyber task for the first time. In biology and chemistry, AI has gone from significantly underperforming PhD-level human experts at troubleshooting experiments, to significantly outperforming them, including for requests about images.

  • On the risk that AI models may “self-replicate” in a way that subverts human control, AISI’s evaluations suggest that AI agents have gotten better at simplified versions of some tasks that could be instrumental to self-replication, such as passing know-your-customer checks to access financial services, but less so at others, like retaining access to compute and deploying successor agents. AISI’s evaluations also suggest that models are capable of deliberately obstructing attempts to measure their true capabilities (“sandbagging”), but only when explicitly prompted to do so.

  • The report also sheds light on AI systems’ limitations. In the cyber domain, AISI notes that AI systems still struggle in open-ended environments where they must complete long sequences of actions autonomously. Similarly, regarding chembio threats, biologists and chemists, and potential threat actors, need “tacit” knowledge and expertise, such as how to pipette. AISI’s evaluations to date have focussed more on explicit knowledge although they plan to share more on wet lab tasks.

  • When it comes to mitigations, the report provides both reassurance and concern. On one hand, the safeguards that leading labs have introduced have made their models safer, in one instance increasing the amount of expert effort needed to jailbreak a model by 40x. On the other hand, AISI says that it was still able to find a vulnerability in every AI system it tested. Worryingly, AISI also found no notable correlation between how capable a model is, and the strength of safeguards it has in place.

  • AISI also sheds light on two other sources of AI risk: open source and scaffolding. They argue that the performance gap between open source and proprietary AI models has narrowed. This introduces risks as safeguards for open models (where they exist) can be removed, and jailbreaks are hard to patch. AISI also found that scaffolding can make AI agents more capable than the underlying base AI models, even if those gaps later narrow when the base models are updated. Some complex scaffolds are in proprietary products, such as coding agents, but others are in open-source efforts.

  • The report also touches on AISI’s evaluations of the broader societal impacts of AI, such as the degree to which people are using AI to access political information, or the risks of harmful manipulation. One striking statistic, picked up in media coverage of the report, was that one-third of UK respondents to a recent AISI survey had used AI for emotional support or social interaction in the preceding year, although just 4% do so daily. In a separate effort, AISI found that some dedicated AI companion users reported signs of “withdrawal” during outages.

  • Overall, AISI argues that AI labs are taking an uneven approach to safety, focussing more on safeguards for biosecurity risks, for example, than for other threats. This is arguably true of AISI as well, given their strong focus on biology and chemical risks rather than radiological or nuclear risks. This raises a question: Given finite resources, what evaluations of frontier AI systems are most lacking in the current landscape?

3. One in four UK doctors are using AI in their clinical practice

• What happened: The Nuffield Trust and the Royal College of General Practitioners surveyed more than 2,000 UK GPs to understand how they view and use AI, in what the authors called the largest and most up-to-date survey on the topic.

• What’s interesting:

  • 28% of UK GPs now use AI. This is up from ~10% in 2018, but below the rates seen in some other UK professions. According to the survey, the GPs most likely to use AI are younger, male, and work in more affluent areas. This is similar to disparities in the wider public’s use of LLMs, although there, the early gender gap may have narrowed.

  • Just over half of AI-using GPs procure AI tools themselves rather than relying on those that their practices select. This kind of “shadow AI use” is not unique to GPs, but a Nuffield focus group sheds light on why UK GPs feel compelled to do it: some GP practices or Integrated Care Boards ban AI tools, while others are slow to respond to GPs’ requests and instead prefer to stick with legacy digital tools.

  • UK GPs mainly use AI for clinical documentation and note-taking. Some say that AI note-taking allows them to look at, and speak more, with their patients, a non-trivial benefit given that the UK public worries about AI making healthcare staff more distant.

  • GPs also use LLMs to produce documents, from translations of patient communications to referral letters; and to stay abreast of new research, with some younger practitioners turning to LLM “study modes” to help with their mandatory professional development.

  • GPs cite “saving time” as the primary benefit of AI, and mainly use this to reduce overtime, rest, and engage in professional development, rather than to see more patients. This is notable as the UK government wants AI to reduce the wait time to get a GP appointment, which is a top concern for the public. These findings suggest that more nuanced evaluations of AI’s impact on GP services will be needed.

  • GPs worry about errors and liability issues with AI. As a result, the authors call on tech suppliers to do better evaluations of hallucinations. Ideally, such evaluations would compare the accuracy of AI, human and hybrid outputs in real-world settings, and all the nuances that might entail. For example, when explaining the benefits of AI note-taking, some GPs pointed out that certain colleagues can’t touch type and so, without AI, struggle to capture all the details in a patient consultation ( this is, presumably, a form of inaccuracy).

  • Use of AI for more complex “clinical support” tasks remains relatively low, owing to GPs’ concerns about errors, their desire to retain control over clinical judgement, and a lack of regulatory approval. However, some GPs did report using AI, or wanting to use future systems, to help check diagnoses, formulate care plans, and analyse lab results.

  • This suggests that more GPs may start to use AI to enhance their own clinical judgement, spurred by a growing body of evidence that LLM-based systems may be useful in this area, and by the public’s own growing use of LLMs for answering medical questions.

  • In their recommendations, the Nuffield authors call for clearer guidelines and regulatory frameworks for GPs, including as part of the UK’s new National Commission into the Regulation of AI in Healthcare. However, the report also acknowledges that much guidance already exists, such as the British Medical Association’s AI principles and the NHS guidance on AI note-taking (which some GPs appear to be breaking by procuring their own tools). This raises a question: what exactly should any new guidance stipulate? How to get the burden on GPs right? And how to ensure that they are actually following it?

Everywhere, people grumble about the government: that politicians care only about themselves; that bureaucrats gum up the system; that taxpayers get fleeced. Even in wealthy countries, nearly two in three people are dissatisfied with how democracy is working.

Headlines focus on politics, but a deeper problem could be public services that are overwhelmed, in contrast to a technological era that keeps accelerating. The real danger, says Alexander Iosad, director of government innovation at the Tony Blair Institute, would be to change nothing.

AI Policy Perspectives visited Iosad, lead author of “Governing in the Age of AI,” to hear his vision of how technology might remedy governmental woes.

—Tom Rachman, AI Policy Perspectives

[Interview edited and condensed]

Tom: Aren’t people always bemoaning governments? Or is something broken in a different way today?

Alexander Iosad: People complain about public services being too bureaucratic, too standardized, not targeted enough. All of those things are true because the system was built in another era, when there was no way to operate differently. But over time, we have faced the Baumol cost-disease problem: things that we produce in the physical world get cheaper, but the cost of services keeps rising because of inflation and higher labour costs. As public-service costs grow, we have this conflict that has brewed over decades: Should government do less? or Should government tax more? But technologies have reached a level of maturity to break this cycle. We can have governments that aren’t dependent on just hiring more people to do more of the same, but can be cheaper, and more effective, and operate at a national scale all at the same time.

Tom: You’re proposing AI as a lever for state renewal. What philosophical change would governments need to achieve that?

Alexander: The first is for governments to realize they can’t continue with marginal tweaks to systems that don’t work. Public services are under such strain that people are looking for the status quo to be challenged. That’s why they’re open to populists. Instead, governments need to embrace the radicalism inherent in what we call disruptive delivery. And this is where AI is a big part of the solution.

WHAT AI FOR GOVERNMENT COULD LOOK LIKE

Tom: The public sector has a lower tolerance for error than the private sector—damage from an incorrect decision about public health could be far worse than a mistake in a business plan. How do you convince political leaders to embrace disruption when the cost of failure could be so high?

Alexander: Because the cost of inaction is much higher. If you do nothing, the system degrades. And the cost is borne by the citizen. If you have a healthcare system that is bursting at the seams; if you have an education system where the disadvantage gap between students on free school meals and their peers is 19 months and trending above pre-Covid levels—those are real problems experienced by real people. Not recognizing that you can actually change isn’t just a political cost. It is a cost to that citizen, which has downstream consequences for both the system and the politician.

Subscribe now

Tom: How might citizens experience AI improvements?

Alexander: By way of example, we can have an education system that is genuinely personalized. We know that personalized learning is more engaging and produces better learning outcomes. We can also have a system that identifies where students have learning gaps, and can inform teachers on what to address. Imagine a school where there’s an emerging gap in mathematics in Year Seven. At the moment, the only way you spot this is when the students take their exams four years later. By then, it’s too late. You might say, “Okay, we now need to focus on maths at that school.” But you’ve had a cohort of students come through, and suffer from this failure. With data and AI, you can spot the gap as it emerges.

Furthermore, we currently have a model of schooling that depends on having access to a person: the teacher. Maybe a parent has a question, and must email the teacher, then wait. If we have a safety net of an AI system—say, a tutor that’s always available, and that is verified to be accurate enough, and that is adapted to the national standards—that parent or student may ask a question at 7:30pm on a Saturday, and doesn’t have to wait to find the teacher. More broadly, you’re creating a different experience of interacting with public services, where they are there for you when you need them.

Tom: To some educators, that picture of teaching will seem like techno-solutionism that overlooks the human role in learning.

Alexander: I would class myself as a tech optimist rather than a tech solutionist. Techno-solutionism means high trust in technology—but low trust in people. Tech optimism is high trust in both. It’s not about replacing the human connection. It’s about recognizing the constraints that a sole dependence on humans to deliver public services introduces into the system, and the gaps that it creates. An ideal system is one that fills those gaps with technology.

Tom: What about other sectors, such as public health?

Alexander: People ask for a transformative AI use-case in healthcare, but it won’t be one big thing; it’ll be 1,000 little things that, in aggregate, completely change your experience. People are already wearing digital rings and smartwatches that measure their pulse and can tell if they are at risk of particular health problems. So at an individual level, this is starting to work already. It becomes really powerful once you connect this to population-level health. In a more personal way, if your doctor has an ambient AI note-taking system, your medical experience transforms. Today, you sit in front of them, they type a lot, and occasionally look at you. But you can have a system where they are fully present and listening, and don’t have to worry about capturing the full picture of what you’re telling them. As we expand outwards, there is the pharmaceutical revolution from AI too, with less cost and more speed of development, and medicines that can be adapted to your body.

Tom: What about government’s role in managing crime?

Alexander: One example is facial recognition, which is contentious for good reasons. People don’t like the idea of their faces being scanned as they walk down the street. “What if there’s a mistake? What if I’m apprehended wrongly?” But in the UK, this technology has achieved very high levels of accuracy now, and does not lead to wrongful arrests. There’s data recently out of the London Metropolitan Police, which uses facial recognition extensively, where the error rate was 10 faces identified wrongly out of more than 3 million scans. No wrongful arrests. But hundreds of correct arrests that would not have happened otherwise.

Tom: But if we move towards data-driven policing, isn’t there a risk that bias within the data could lead to injustice?

Alexander: Of course, you have a big challenge with potential bias in this context. You train the systems on existing data, which might not have enough representation of people from minority groups—for example, fewer non-European faces, so the algorithm is more likely to misidentify people. Or the data might have groups over-represented—for example, capturing historical overpolicing of communities or areas. The risk is that these biases are replicated, and even scaled up. Early versions of new tools are more likely to make such errors, and real-world experience shows that, if we are aware of this, and take active steps to mitigate it, it is possible to prevent these kinds of biases. This is something that needs to be built into the process of development and deployment. We see, for example, that facial-recognition systems are much more accurate today than they were 10 years ago. Not perfect, but much better, and providing better intelligence for officers to decide when they need to act. You could also have a kind of AI peer review, where one model might be trained to monitor another for replicating bias, or introducing new bias into the system—a watching-the-watchers situation. Again, this would be an improvement on the situation we have today, where much of this bias just passes unnoticed and uncorrected.

Tom: So, it’s not the sci-fi dystopian vision of crime-fighting, you’re saying?

Alexander: Yes. And the status quo is a uniformed police officer on the corner, standing in the rain, the sun setting, holding a printout from earlier that morning with blurry low-resolution pictures of the people they’re looking for. They make more wrongful arrests as a result of that situation than police officers sitting in a van with computer infrastructure, and a camera telling them there’s a person walking down the street with a child, and this person is on a sex offenders’ register, with court restrictions against being near children. The police officer can go and talk to this person. This is a real case, by the way—and it turned out to be someone building a friendship with the child’s family without their knowing he was on the register. No way would a police officer know this today, if someone just walked past them with the child. So it’s about looking at what we do, and how we can do better, rather than leaning into these fantasies of complete control.

Share

3 NEW AI ROLES

Tom: You also advocate a radical new model for how governments operate internally. Could you explain these three concepts: the Digital Public Assistant for every citizen; AI co-workers for each civil servant; and a National Policy Twin for policymakers to simulate decisions.

Alexander: The Digital Public Assistant, either on my device or online, would be a system that connects information about you held by different parts of government—for example, your income level and your address—and is then able to say, “You’re eligible for this particular discount on your energy bill—would you like to have it?” Or it could support you during interactions with government officials. So much of our time is spent repeating the same things to different agencies, whereas here you might be talking to an unemployment adviser, and they can see your employment history or your qualifications, and suggest the right next steps for you so the job you find is the best fit for you specifically. Which might mean you stay in that job longer, and grow in it to have a fulfilling career. You could have a settings dashboard to decide how various AI agents interact with the government on your behalf. All this puts you in greater control.

Tom: What about AI co-workers for each civil servant?

Alexander: This is already starting to happen with chatbots, but that is the most basic version of it. You could have a suite of co-workers that looks at new cases, such as requests for support or applications for services, that a public-sector worker receives, and helps prioritise this, or find the information that the civil servant needs to make the best decision. The AIs don’t make decisions in place of that worker, but they make the worker much better informed, and save them hours of digging through regulations. There was a pilot experiment that showcased the potential for this in the UK government, involving employees of the Department for Work and Pensions who help jobseekers find employment. The employees, who act as work coaches for job seekers, were able to ask a large language model to explain various rules, to help draft documents, to prepare reports, and update records. Today, if a government employee has a question about when a claimant is eligible for a particular service, they might just search the internet. But you can have a system trained on the relevant rules, and gives you a quick and accurate answer. This saved about two weeks’ time per employee per year—and allowed these work coaches to focus on building relationships with the people who needed their support.

You can picture this across different parts of government. In procurement, you would have more informed advice about all the bids coming through, for example. Or if you think about how much time officials spend sending documents around for someone else to summarize when they are asked to prepare briefings and documents for government ministers—a lot of this work could be done much more quickly, so people have time to actually think about what it means, not just produce digests, and you could include a wider range of different sources so the information is more nuanced, accurate, and up to date.

Tom: Your third concept is an AI simulation of the entire country to test out policies.

Alexander: Yes, this gets exciting. We call it the National Policy Twin. Data is aggregated from different parts of service delivery, such as information on schools from the education department, and economic data from the statistics agency, and incomes data through the tax agency, and so forth. Together, it’s essentially a digital twin of your country, and you can run different policy scenarios informed by this data. At the moment, civil servants present a government minister with, say, three policy scenarios. If there are assumptions that the minister doesn’t agree with, they’ll say, “Give me three other scenarios based on different assumptions.” They wait for weeks, and then the process repeats. With the National Policy Twin, you could test ideas or intuitions very quickly, iterate on ideas, and ask for best practices from around the world, so that policies have a stronger evidence base—all in minutes, not days. You are not replacing the policymaking process. But you are speeding things up, so you can test more options. You are less likely to miss the right option because it never came up.

Tom: But isn’t the validity of a “digital twin” simulation dependent on the quality and comprehensiveness of the data available? And wouldn’t this risk biasing decision-makers toward whatever the data suggested rather than broader impressions, even if those broad impressions encompassed more wisdom?

Alexander: It is a danger. But it’s also a motivation to ensure your statistics agency runs well. This dramatically raises the importance of getting data right, and it’s something that not every government has really paid attention to. This would be helped if you build a whole data system, including Digital Public Assistants, where citizens can correct their information, leading to better data flows to governmental institutions. This is also where AI systems can interpret unstructured data, understand how it all fits in together, and provide informed advice. Again, AI is not making the decisions. It’s providing information for humans that was previously not available or not usable, and helping people to make sense of it, and make better decisions as a result.

OBSTACLES REMAIN

Tom: Another hurdle is decades-old IT systems in public services. Can governments overhaul this infrastructure at a pace that keeps up with AI development?

Alexander: Legacy infrastructure is a problem, and interoperability in government is something most countries are trying to tackle. In the UK’s blueprint for modern digital government, there is a plan to make every public-sector dataset interoperable in the next few years. This is the first thing we should do. Right now, some police forces spend 90% of their IT budget on maintaining legacy systems. If you’ve got legacy systems here and there, fine—spend 10% of your budget on that. But 90% should be spent on upgrading. You do this for two years, and it’s a hard push, and will be painful. But then we get there.

Tom: Another concern about using AI in so many parts of governmental work is that we risk losing democratic transparency, explainability, and the citizen’s right to appeal decisions made by algorithms.

Alexander: There needs to be human accountability for decisions made on the basis of this system. We need that built-in from the start. This needs to be sensitive to individual circumstances because, for every 95% of successful cases, you will have some cases where things didn’t work as expected. If we free up government resources by using AI, we can use those resources to make it easier for people to go and talk to someone when they need to, either because something went wrong, or because they are more comfortable with that way of dealing with the government.

WHICH GOVERNMENTS ARE TRYING THIS?

Tom: You published “Governing in the Age of AI” shortly before the July 2024 general election in the United Kingdom. It’s around a year and a half since Prime Minister Keir Starmer’s Labour Party took power. Are there lessons in what has or hasn’t happened regarding AI implementation?

Alexander: The UK has been among the more ambitious globally, including its AI Opportunities Action Plan and its blueprint for modern digital government. But there is a challenge when it comes to AI in government: how do you make it tangible for people, and how to balance risk and reward in doing so? If you are a political leader coming into office and thinking about this, how do you drive forward AI while maintaining public support? What are the quick wins where you can tangibly speed up the way that citizens interact with government, where you can improve that experience in ways that you can claim credit for? Part of the challenge that this government has arguably had is that not everyone has noticed the things it does.

Tom: What’s an example of something that has worked, but that people aren’t noticing?

Alexander: You have a problem since Covid in the UK, and in many other countries, with students not showing up for lessons. So what they’ve done is connect school attendance systems so that the government gets a daily record of the proportion of students who came to school the day before. But it’s not enough to just have data, so what they’ve done is build tools that explain to school leaders how they compare to other similar schools, and what profile of students might be seeing a gap in attendance. In one rural school, attendance kept dropping on Tuesdays, and the school didn’t notice until the Department for Education came with a tool that showed this trend. Then the school discovered that there was a bus that was always late on Tuesdays, so students just gave up and never came in. They hired a minivan for Tuesdays, and attendance shot up.

Tom: Which governments around the world are getting this right?

Alexander: We are at an early stage in this journey, even for the private sector, and certainly for governments, which tend to move slowly. But Singapore is doing well. And Estonia. And Ukraine, for obvious reasons: they’re having to break the current way of doing things; you have to figure out other ways. They recently launched a chatbot that Ukrainian citizens can use to get answers based on information from their digital ID. Australia is another country doing well, particularly on AI and education. The UK too. But there won’t be a simple list of “Five Ways That AI Has Transformed Government.” It’s going to be everyone doing a bit of something somewhere that adds up to a bigger picture. It’s not, “Are you promoting AI in your public service?” Everyone is. It’s: “Are you just making current processes slightly faster? Or are you genuinely thinking about deeper reform?”

Tom: Albania introduced a virtual AI minister to handle public procurement. What do you think of that?

Alexander: It’s quite an attention-grabbing announcement but is making a serious point: that AI can help cut fraud, improve efficiency, and save money in public procurement. But Albania has an even more interesting example of AI in government. They’re going through the process of applying for European Union membership, and that is both a bureaucratic process and a process of real reform, where you bring your legislation in line with European standards. So, you’ve got laws in Albanian, you’ve got European laws in English and French, and so on, and you need to find discrepancies, and update legislation, then implement reforms. That is an incredibly time-consuming process that has typically meant hiring hundreds, if not thousands, of lawyers and translators. It takes a decade to do this. But Albania is using AI tools to radically speed up this process. That is accelerating their accession process, possibly by several years.

Tom: We’ve talked a lot about the public services, but do you have thoughts on how AI could update democracy more broadly?

Alexander: If we get this right, the most noticeable impact will be improved trust because government can deliver rather than let things continue to slide into decline. Also, AI can introduce more transparency. Several countries have Freedom of Information acts, but it takes ages. There are local governments in the UK experimenting with systems where you type in a question, and if they have the data already, it’ll answer your question, just give you the data right there, and you don’t have to go through civil servants for it. There is also a philosophical reason why accountability could improve in the age of AI: the machine doesn’t make the decisions. Even if you have an automated system, there should be a person somewhere, thinking, “Let’s make a choice we are comfortable with.” If we get into that mindset, we make government aware that the human role is to make good decisions, and to take that responsibility very seriously. That, I think, will have a significant impact on democracy.

TAKEAWAYS

Tom: What final message do you have for policymakers trying to use AI in government?

Alexander: What’s really important is to carve out time for this thinking. As a public service, you’re always under pressure; you always need to deliver the next thing. Yes, AI will save time—but if you are just adding more work into those hours, you’re not going to get any gains. Carve out half the time that you save because of general-purpose AI systems to sit down with colleagues, and think how to improve your service. This requires leadership to say, “You have to do this.” We need a public-service workforce that is both more capable of this type of creative thought and experimentation, and is actually empowered to do it. At the moment, we have a pyramid shape with a lot of people doing a lot of repetitive tasks at lower pay. Those jobs are at risk because AI tools are good at doing those tasks at a fraction of the cost, and in seconds, not hours. What does that mean for the future structure of the civil service? Is it the same people doing different things? Is it fewer people? I don’t think anyone really has good answers yet.

Tom: What’s the biggest obstacle to your vision? And the best answer?

Alexander: The biggest obstacle is inertia. This future is uncertain, and government isn’t always good at dealing with uncertainty. The best answer is for leadership to take seriously the responsibility of updating government. Otherwise, we will be left behind. On the cost side, it’s not just hiring engineers or buying computers. It’s the cost of inaction that you need to weigh up.

—Conor Griffin, AI Policy Perspectives

Five Great Pieces from 2025

1. A Defence of Weird Research (Asterisk Magazine)

Deena Mousa & Lauren Gilbert

Séb Says: Science funding needs to be shaken up. But I’m concerned that a lot of good research might be cut because people misunderstand how science works. Mousa and Gilbert remind us why basic research matters, and why governments should fund it: while the benefits to society are significant, they are hard to predict and take time to materialise, so companies will underinvest. To make their case, the authors take a tour of weird-research success stories, such as how studying lizard venom led to the invention of Ozempic, and how studying the effects of separating rat pups from their mothers led to the now common use of massage therapy to help pre-term human babies. Did you know that studying frog skin led to the invention of oral rehydration therapy, which has saved over 70 million lives?

2. Requests for journalists covering AI and the environment (The Weird Turn Pro newsletter)

Andy Masley

Séb Says: I worry about the quality of a lot of commentary on AI and the environment. So it’s important to re-up these best practices. Specifically, Masley cautions that readers are coming away with wildly inaccurate beliefs about where AI and data centres fit into the environmental picture. His favourite book on good environmental communication is Sustainable Energy—Without the Hot Air, by David JC MacKay, and his guidance includes some classics of the genre, such as never sharing contextless large numbers (“200,000 bottles of water per day”). He also suggests comparing data centres’ energy use with other industries, rather than with household use. Although aimed at journalists, the guidance is also helpful to those working in policy, some of whom make the mistakes that Andy calls out, such as viewing one’s own AI prompts as environmentally consequential.

3. ChatGPT and the Meaning of Life, (Scott Aronson’s Shtetl-Optimized blog)

Harvey Lederman

Séb Says: I don’t think all jobs will disappear any time soon. But if we get full automation, then Lederman’s piece is a good way to think about it. He starts by describing the fits of dread he has felt ever since the launch of ChatGPT, then considers reasons why the end of work could hurt society, from losing the joy of scientific discovery to losing the sense of purpose from serving others. Ultimately, he rejects the most pessimistic arguments, noting that the consequences of scientific findings, such as penicillin that saves lives, are more important than their discovery, and that much service work is drudgery. However, he captures how difficult the transition may be, including for “workists” like him who use their jobs to make sense of their lives. He concludes that: “A future without work could be much better than ours, overall. But, living in that world, or watching as our old ways passed away, we might still reasonably grieve the loss of the work that once was part of who we were.”

4. How much economic growth from AI should we expect, how soon? (Inference Magazine)

Jack Wiseman & Duncan McClements

Séb Says: Some predict that AI will be close to economically useless, while others think it might transform everything tomorrow. This piece comes closest to how I think about it. As Wiseman & McClements explain, the most ambitious forecasts for AI rest on the idea of “digital AI researchers” that train and improve the next generation, leading to a jump in the share of economic tasks that AI can do. One obstacle to achieving this is the availability of compute, which is increasingly allocated to serve customers (inference) rather than to training new models. Additionally, a multitude of frictions will slow the diffusion of AI, whether it’s the time needed to cultivate biological cells for scientific experiments, or the regulatory approvals for sensitive-use cases. As a result, the authors expect a transformative impact on near-term economic growth, but not an explosive one.

5. Yes, Econ 101 is underrated (Economic Forces newsletter)

Brian Albrecht

Séb Says: Much of the discourse on the Left and the Right ignores inconvenient truths of economics, so it’s good to return to the basics. Albrecht shows how Econ 101 helps explain the world. For example, egg producers were accused of price-gouging when they charged sharply more in 2022, but it had more to do with avian flu killing many chickens. In the egg market, supply and demand are relatively inelastic: It takes time to raise chickens, and customers who want omelettes don’t have alternatives. So, prices jumped. Different markets have different characteristics, but the explanatory power of supply, demand and pricing is similar. Nor does outsized market power invalidate these principles. This essay also shows how Econ 101 offers insights into social trends, such as how skewed sex ratios can affect marriage and employment rates, as in certain immigrant communities, or drive up savings rates, as in China. Econ 101 may not tell us whether policies will be politically popular or whether outcomes are fair. But it does help predict what those outcomes may be.

Subscribe now

Three Classics that I Revisited

6. Why do people believe true things? (Conspicuous Cognition newsletter)

Dan Williams

Séb Says: Anything Dan Williams writes is self-recommending, and this piece is no exception. In July 2024, he critiqued how many people think about the relationship between belief and reality. To illustrate this, he notes that people seek explanations for issues like crime and poverty, when the real question is understanding law-abidingness and wealth. This requires “explanatory inversion.” Transferring that concept to how people commonly debate public knowledge, he notes that many misinformation researchers concern themselves with why different groups believe falsehoods. But the more pertinent puzzle, he contends, is why humans overcome error, bias and illusions to form accurate perceptions of how things are. His conclusion? Ignorance and misperceptions are the default, and humanity will revert to them, unless we can understand, maintain and improve our norms and institutions, from journalistic integrity to robust legal systems.

7. Hayek on the Role of Reason in Human Affairs (Intercollegiate Studies Institute)

Séb Says: A lot of discourse on intelligence, knowledge, and coordination is biased towards a computer-science-centric view of the world, and neglects Hayek’s views. This 2014 essay explains how Hayek championed critical rationalism, which was rooted in the Scottish Enlightenment of David Hume and Adam Smith, and developed by Carl Menger and the Austrian School. Critical rationalism sees social order as spontaneous, and the unintended result of human action, not design. As a result, inherited social institutions and rules contain tacit knowledge, the result of a multitude of trials and errors, that transcends the knowledge available to a reasoning mind. Therefore, the desire to “make everything subject to rational control,” Hayek suggests, is an egregious error. Reason should instead serve a negative function, to guide and restrain irrational impulses or morals. As the human mind cannot master all the concrete details of society, we must rely on abstract concepts and rules, like the rule of law and the market, to coordinate the dispersed, fragmented, knowledge of millions of people.

8. The Inner Ring (The C.S. Lewis Society of California)

C.S. Lewis

Séb Says: This piece profoundly shaped how I think about the world. In this 1944 lecture at King’s College, University of London, Lewis offered “middle-aged moralising” to a group of students during wartime, telling them that in every organisation, from school to the army, there are two hierarchies. There is the official hierarchy. Then, there is the informal hierarchy, an “Inner Ring” that holds the true power. The Inner Ring comes in many forms, from high society to “communistic côteries.” It is always evolving, holds no formal admissions or expulsions, and bears no clear identifying marks, save perhaps particular slang and a longing from others to be inside. It is this desire, and the terror of being outside, that turns people into scoundrels, he argues. The Inner Ring may be unavoidable, or even necessary. But the quest to enter it is ultimately futile. “Once the first novelty is worn off, the members of this circle will be no more interesting than your old friends. Why should they be?” Lewis said. “You were not looking for virtue or kindness or loyalty or humour or learning or wit or any of the things that can really be enjoyed. You merely wanted to be ‘in.’ And that is a pleasure that cannot last.” What to do instead? Be a sound craftsman who focuses on the quality of work as an end in itself, and spend time with people you actually like.

Share

The future is coming too fast! (By which we mean 2026.)

But fear not. We’re hatching great reads for your new year: Can AI help fix government?; What exactly is “AI manipulation”?; and Might sci-fi hold clues about the world we’re hurtling towards?

All we lack is you. More exactly, your intelligence on artificial intelligence. So……

Please complete this quick (5 minutes?) questionnaire.

Given plunging survey response rates, we’ve limited ourselves to just 2 questions. Write 2 words, or 200.

• What’s an AI topic you’d like better explained?

• What’s a topic that people aren’t discussing enough?

We’ll read every answer with great interest. Wishing you an excellent 2026!

—Conor Griffin & Tom Rachman, AI Policy Perspectives

Subscribe now

If an AI gained consciousness, would we know?

Maybe this question strikes you as absurd; maybe, disquieting. Either way, you’ll hear it more in coming years, as human beings develop increasingly close ties with charismatic machines trained on us.

Thankfully, philosophers have pondered consciousness for about as long as philosophers have pondered anything. In recent decades, advances in computing added urgency, with leading thinkers dreaming up a range of provocative thought-experiments: a man communicating from a locked room; a woman afflicted by a blue banana; a bat with an inner life.

To explain, we are publishing this essay about key thought-experiments related to AI, written by the broadcaster and author David Edmonds, whose acclaimed books include Parfit, the recently released Death in a Shallow Pond, and a collection of philosophical essays that he edited, AI Morality. He is currently writing a book on thought-experiments.

—Tom Rachman, AI Policy Perspectives

By David Edmonds

As a young scholar in Oxford, John Searle fell in love twice. First with a fellow student, Dagmar, who became his wife, and second with philosophy. The City of Dreaming Spires was grim in the 1950s, Searle recalled, with unheated buildings and inedible food. “The British were still on wartime rationings,” he said. “You got one egg a week.”

The philosophical fare was more nourishing. Searle described the collection of philosophers in the city as “the best the world has had in one place at one time since ancient Athens.” Two giants of Oxford philosophy, Peter Strawson and J.L. Austin, were key influences on him.

Searle became fixated on one topic that, for the rest of his life, he maintained was the central puzzle for philosophy: consciousness. How was human reality and our conception of ourselves compatible with the physical world? How could beings with free will and intentionality exist? How could politics, ethics and aesthetics arise out of the “mindless, meaningless” stuff from which the physical world was constructed?

From 1959, Searle taught at Berkeley, beginning his career in what now seems a remote era of pen and paper. It wasn’t until the late 1970s that personal computers became widely available. At roughly the same time, debates around artificial intelligence gathered speed and heat.

In 1979, Searle was invited to deliver a lecture at Yale to AI researchers. He knew next-to-nothing about AI, so bought a book on the subject. This described how a computer programme had been fed a story about a man who’d gone to a restaurant, been served a burnt hamburger, and stormed out without paying. Did the man eat the hamburger? The programme correctly worked out that he had not. “They thought that showed it understood,” he commented. “I thought that was ridiculous.”

And so in 1980, Searle published a paper called “Minds, Brains, and Programs,” introducing the Chinese Room, one of several famous philosophical thought-experiments that have had a lasting impact on discussions of consciousness and AI.

It goes something like this. You are the only person in a locked room. A note is passed to you underneath the door. You recognize the characters as being Chinese, but you don’t speak Chinese. By luck, there’s a manual in the room, with instructions on how to manipulate these symbols. You follow the instructions. Without understanding the content of what you’ve written, you produce a reply that you slip back under the door. Another note arrives. With the manual, you again generate a reply.

The person on the other side of the door might have the impression that you understand Chinese. But do you? Obviously not, thought Searle. And any computer is in an analogous position. A computer is merely manipulating symbols, following instructions, he thought. Computation and understanding are not synonymous.

Share

BATS & COLOURS

It is a striking feature of the philosophy of mind, and consciousness studies, that so much of the intellectual agenda has been driven by a small set of thought-experiments.

The Chinese Room has spawned a vast literature. Almost as famous is a paper, “What’s It Like To Be A Bat?”, that predated Searle’s by six years, written by another American philosopher, Thomas Nagel, whom Searle befriended during his Oxford years.

Like us, bats are mammals. But they have an alien way of navigating the world, echolocation. There is a subgroup of humans, chiropterologists, who know an impressive amount about bats, and have investigated how their high-frequency sounds bounce off objects, allowing them to detect size, shape and distance. But there is one thing that they don’t and can’t know, Nagel said: the subjective experience of being this creature.

“I want to know what it is like for a bat to be a bat,” he wrote. “Yet if I try to imagine this I am restricted to the resources of my own mind, and those resources are inadequate to the task.” AI was still in its infancy when Nagel wrote his article, but questions about the meaning of an artificial mind were already circulating. Could there be something that it is like to be a thinking machine?

The Australian philosopher, Frank Jackson, attacked the problem from a different angle in his 1982 article, “Epiphenomenal Qualia” (qualia being a term for the subjective aspects of conscious experience). In his Mary’s Room thought-experiment, a woman has had an unusual upbringing. Mary was raised alone, entirely in a black-and-white room: black-and-white walls, a black-and-white floor, a black-and-white TV. She has black-and-white clothes and her food, pushed under the black-and-white door, has been dyed black and white.

To stave off the tedium of her monochrome existence, Mary studies hard, and her focus is colour. She learns all about the physics and biology of colour—for example, about the wavelengths of particular colours and how they interact with the retina to stimulate experience. She even learns how colour words are used in literature, poetry and ordinary language, and how someone can “feel blue,” be “green with envy,” or so angry that “a red mist descends.” Mary becomes the world’s expert on all aspects of colour.

One day, the door to Mary’s room opens for the first time, and she joins us in our kaleidoscopic world. The first thing she sees is a ripe red apple. The question is this: When Mary sees this apple, does she learn anything?

Jackson argued—and most people presented with this scenario seem to agree—that in seeing what red actually looks like, Mary has learnt something. At the time of his article, what Jackson took this to show is that a purely physical description of the world cannot capture everything there is to know about the world. The phenomenology of experience (the redness, the what’s- it-like-to-be-a-bat-ness) cannot be fully explained with descriptions of particles and fields, electrons and neutrons, atoms and molecules.

Even if an AI could recognize a new shade of colour, such as lilac, that it had never seen before, it would not mimic human experience if it lacked lilac qualia—or so Mary’s Room might suggest. This raises the issue of how human subjectivity may be relevant to comprehension and functioning in the real world, turning a philosophical question into a technical one.

Subscribe now

BLOCKHEADS & BANANAS

It was Jackson who gave the name “Blockhead” to a thought-experiment from the American philosopher Ned Block that appears in a 1981 paper, “Psychologism and Behaviorism.” We are to imagine there is a computer, programmed in advance so that it could respond to every possible sentence with its own plausible sentence.

This was in part a response to the famous test of machine intelligence that Alan Turing set in 1950. A computer passed the Turing test if it could converse with a human, and the human could not identify it as a machine. The Blockhead machine would pass the Turing test yet is self-evidently not intelligent.

Today’s LLMs could fool us into believing that we are engaging with humans. But, much as Searle contended in the Chinese Room that manipulating symbols is insufficient for understanding, Block argued that behaving identically to an intelligent entity is insufficient to demonstrate intelligence or mental states. The lesson we might take from these, along with the Nagel and Jackson thought-experiments, is that AI would lack fundamental features of human consciousness.

Daniel Dennett, on the other hand, thought it was at least conceivable that AI could be conscious. With his lumbering bulk and Santa Claus beard, Dennett was an unmistakable figure in the philosophical world. He coined the term “intuition pump” as an explanation for how thought-experiments functioned. Pumping our intuitions can be helpful, he believed, but they can also mislead. What we need is to examine how the pump operates, he said, to “turn all the knobs, see how they work, take them apart.”

A thought-experiment for which he had particular loathing was the Chinese Room. He argued that its principal error was to portray language as akin to instructions. But for a computer to master a language would take millions and millions of lines of code. And, though we might say that the man alone in the room doesn’t understand, perhaps the system as a whole does.

Dennett felt that Mary’s Room had similarly hoodwinked us. To expose this, he presented another thought-experiment. Mary is as before, an unusual woman whose life has been led entirely in monochrome, until the day when the door opens. But this time, he wrote:

As a trick, they prepared a bright blue banana to present as her first colour experience ever. Mary took one look at it and said, “Hey! You tried to trick me! Bananas are yellow, but this one is blue!” Her captors were dumbfounded. How did she do it? “Simple,” she replied. “You have to remember that I know everything—absolutely everything—that could ever be known about the physical causes and effects of colour vision. So of course before you brought the banana in, I had already written down, in exquisite detail, exactly what physical impression a yellow object or a blue object (or a green object, etc.) would make on my nervous system.”

Mary is the world expert on colour, so why wouldn’t she spot such an obvious deceit? Dennett argued that the idea that we had feelings, thoughts and desires that were resistant to an objective, external, physicalist analysis was mistaken. In that sense, “qualia” were a mirage, a kind of useful fiction. If we do away with this fiction, then a major barrier vanishes to building AI that’s like a human in most important respects.

The AI researcher Blaise Agüera y Arcas has argued that in theory (and increasingly in practice) there is no significant distinction between a human and machine reaction to so-called qualia. “So many food, wine, and coffee nerds have written in exhaustive (and exhausting) detail about their olfactory experiences that the relevant perceptual map is already latent in large language models. … In effect, large language models do have noses: ours.”

AVOIDING TWO BAD OUTCOMES

The enduring fascination with thought-experiments in the AI era—and the intensity of the disputes that they provoke—reflect how much is at stake. While these questions are important for morality, they could become more than theory.

“The importance of the dispute over AI welfare can be understood in terms of the avoidance of two bad outcomes: under-attributing and over-attributing welfare to AIs,” the philosophers Geoff Keeling and Winnie Street explain in their forthcoming book Emerging Questions in AI Welfare.

“On one hand, failing to register that AIs are welfare subjects when AIs are in fact welfare subjects is bad because it could lead to unintentional mistreatment of AIs or the neglect of the needs of AIs, potentially resulting in large-scale suffering,” they write. “On the other hand, over-attributing welfare to AIs is problematic because resource allocation decisions for promoting the (potential) welfare of different kinds of entities—including humans, non-human animals and AIs—are often zero-sum.”

In other words, the efforts and resources you invest in AI welfare mean less for people and animals.

To manage this quandary, Keeling and Street propose three parallel projects. First, there is a philosophical project, in which we consider which forms of AI could be candidates for welfare. Is it the underlying AI model? Or the system built atop? Or would it be specific agents? Second, there is a scientific project, in which we establish methodologies to detect factors such as consciousness. Thirdly, there is a democratic project of versing the public in the complex issues that await.

Once this future engulfs us, thought-experiments about machine consciousness could move beyond speculation. The “experiments” would be active, while the participants would be humanity itself—and perhaps other beings besides.