By Simone Maria Parazzoli & Omer Bilgin

Amid the exhaustion of caring for a baby, new parents must deal with everything from bewildering sobs, to erratic feeding times, to the joys of changing a soiled newborn at 3 a.m. The last thing they need is paperwork.

But what if, when coming home from the maternity ward that first day, they could awaken a government AI voice assistant, tell it the happy news, and hear the following response? “Congratulations! What’s the baby called?” The app would then take care of all the dreary admin, coordinating across agencies, registering the child, and setting in motion the services that this tiny new citizen should enjoy.

That is one example of how a future “agentic state” could simplify, speed up, and improve citizens’ interactions with public services. To be clear, this does not yet exist. But projects like this one, envisioned by Ukrainian officials, are more than fantasy, with several countries avidly testing early versions of agentic AI systems.

While Ukraine works toward the baby example, Britain is piloting agent-based support to provide citizens more tailored help. Meanwhile, Singapore is developing governance frameworks for agentic AI, and governments from France to the United States are ensuring that their public data can be accessed by agents.

Agentic AI systems—capable of perceiving, reasoning, and acting with minimal human supervision—will transform what organizations can achieve. By combining the reasoning of large language models with retrieval, memory, and tool use, agentic AI can automate complex tasks. For governments, whose core work is high-volume, structured administrative processes, this could make services more efficient, timely, consistent, and fair, while lowering costs.

Consider a citizen looking to start a small business. An agentic system—instead of requiring the entrepreneur to individually navigate zoning boards, tax authorities, and regulations—could autonomously reconcile these requirements. The larger promise is a shift from just doing things right (optimizing for procedure-following) to doing the right things (pursuing outcomes that citizens truly want).

The Agentic State vision paper—supported by The World Bank and the Global Government Technology Centre Berlin—was the first effort to systematically map the opportunities of agentic AI adoption for governments. This was not an academic exercise: 21 leaders across 15 countries contributed, including ministers and chief technology officers preparing to lead this transition.

In this vision, AI agents are a means to manage complexity and scale, while humans develop strategy, exercise judgment, and hold accountability.

Several governments have integrated official chatbots into their government services, but most of these merely provide conversational guides to administrative procedures. A few pioneering countries are starting to move beyond that. Ukraine, for instance, is turning chatbots into agentic assistants. Specifically, its Diia.AI assistant can retrieve users’ data from connected registries, and generate official documents such as income certificates, while also providing certified information based on records such as taxation, land registries, and pensions.

The United Kingdom is also exploring agentic interactions via GOV.UK Chat (inspired by Diia.AI), including a pilot program to support job seekers that transforms a static digital portal into an active assistant, matching users’ skills with available opportunities.

Yet trends and optimism are not enough for success. The agentic state vision rests on key assumptions. What if they’re wrong?

This article presents a “red-teaming” exercise—a stress test of this vision—that identifies six core assumptions, along with scenarios that could emerge if they don’t hold true, and guardrails to avert such failures.

Agents can already perform rudimentary planning, tool use (e.g., searching the internet, using calculators, sending emails), and multistep task execution. Frontier labs are betting heavily on agents, making it plausible that systems capable of managing complex and large-scale administrative tasks will emerge soon.

Failure Scenario: The Technology Falters

Governments reorganize around agentic execution, but systems never become reliable enough for public administration. The demos look strong, but real cases fail on edge conditions, and require constant human correction. The agentic layer becomes only superficially competent with layers of human intervention underneath.

Guardrail: Start Cautiously

Governments should start with minimal deployments, and tightly scoped use cases to validate reliability, develop procedural rigor and organizational competence, and account for technological evolution rather than committing prematurely to large-scale redesigns.

The success of agentic systems demands that they’re able to interact seamlessly, conveying intent, carrying out tasks, and sharing data in an interoperable way. MCP (model context protocol) is emerging as the technological standard for connecting AI applications with external systems.

Failure Scenario: Standards Fail to Converge

Commercial interests diverge, establishing competing protocols, while government departments end up using AI systems that cannot communicate with one another. When a citizen’s request requires action from multiple agencies, the process breaks down.

Guardrail: Officials Insist on Shared Protocols

Governments should make interoperability a condition of adoption, participating in the cross-sectoral bodies and forums where these standards are being shaped, funding the development of shared agentic interfaces and other agent-specific standards, and mandating non-proprietary protocols in procurement. Standards rarely emerge by accident, but they may emerge when powerful governments treat them as a priority.

To adopt and employ agents effectively, organizations must rethink their processes, roles, and incentives. They need to flexibly change and dynamically adapt practices to keep pace with the changing technological landscape.

Failure Scenario: The Status Quo Prevents Change

Agentic AI adoption outpaces organizational change, with citizens and civil servants using agents in an uncoordinated manner long before official programs catch up. Local practices harden into path dependence before common standards emerge. The state becomes more productive at producing bureaucracy, not societally beneficial outcomes.

Guardrail: Redesign Processes Before Automating Them

Agents should only enter workflows that have been simplified, decomposed, and restructured to minimize approval layers and handovers. Governments must treat adoption as a continuous discovery process. They should invest in common evaluation templates, reusable components, and a cross-agency repository of lessons, so that what works in one place can travel before what does not work becomes entrenched.

Many companies are betting on an agentic future. Firms are experimenting with internal copilots and autonomous customer flows, while frontier AI companies advance core models, architectures, and capabilities, and cloud providers offer the compute needed to deploy agents at scale. This suggests that agents will become commonplace across business, consumer, and enterprise environments, allowing governments to build on tools, infrastructure, and behaviors already spreading across the economy. This assumption rests on projections, though evidence remains ambiguous.

Failure Scenario: Diffusion Is Slower Than Forecast

Governments invest as if an agent-saturated economy is imminent, but industry adoption remains narrow, experimental, or ends up costing more than it saves. Public investments don’t plug into widely used tools and practices, meaning that citizens find agentic interfaces in government before they’re normal elsewhere. The state ends up bearing political and institutional costs without the stabilizing effects of private-sector diffusion.

Guardrail: Lower Barriers to Private-Sector Agentic Usage

Governments can accelerate the development of an agentic AI ecosystem by investing in shared agentic infrastructure—such as standard ways to access public data, communicate across systems, and carry out authorized tasks and payments—that lower integration costs for firms, and reduce the risk of differing technological maturity across sectors.

Increasingly, citizens are interacting with and relying on AI tools, but many do not trust them. For governments to integrate AI agents into workflows and services, citizens must accept and support the roles that agentic systems can play, finding them sufficiently trustworthy, reliable, fair, convenient, and accountable.

Failure Scenario: The Public Rejects Automation

A single notable failure, or an accumulation of failures, turn the public against agentic systems, and convince many to opt-out. They judge automated decisions as opaque, illegitimate and untrustworthy, and suspect it worsens inequality, with privileged citizens able to employ highly capable personal agents to navigate bureaucracy better than those relying on basic tools. The government is forced to run two systems—agentic and human—and neither meets expectations.

Guardrail: Mandate Transparency

Governments must make agent integrations into government processes as legible as possible, furnishing explanations of decisions and publishing evaluation results for agentic fairness and performance, while detecting patterns of systemic bias or unequal benefit distribution based on citizens’ technological access.

For AI agents to act with functional autonomy within government processes, oversight frameworks must adapt, moving away from mandatory human reviews and approvals for everything (human-in-the-loop) to intermittent oversight (human-on-the-loop). This evolution increases speed and efficiency while reducing bottlenecks, with humans intervening only on edge cases. There is precedent for such adaptation: governments adapted regulation to cloud computing, e-identities, and AI-driven decision support systems.

Failure Scenario: Regulation Never Updates

Every agentic action requires human verification; every decision must be justified through mechanisms designed for old chains of accountability. Agents can draft, but cannot act. Compliance and procedural costs rise as institutions retrofit old controls onto new AI processes. The result is high bureaucracy and low autonomy: an agentic state in theory, a copilot state in practice.

Guardrail: Sandboxes to Test Oversight

Governments should establish controlled environments that allow policymakers, developers, and civil society to collaborate and gather empirical evidence on what forms of oversight are adequate and best fit different kinds of agentic deployments, reducing uncertainty before codifying rules at scale. They should explore this early, much as Singapore has done through its Model AI Governance Framework for Agentic AI.

Subscribe now

Soon, agentic government will be more than optimism and testing. A vanguard of countries will implement these tools. If those cases produce the kinds of benefits imagined, other countries will flock to join them.

But momentum is not inevitability. This project depends on assumptions—about progress, coordination, institutions, norms, and law—that demand scrutiny before governments rebuild themselves around these new technologies.

This red-teaming exercise of the agentic state concept is not to argue against the vision, but to make it more robust and resilient. The six possible failure scenarios are not mutually exclusive. Several could compound, and some may already be taking shape. For instance, reliability has been improving much more slowly than accuracy, providing ground for technology to falter (Scenario 1), and there are signals that the public might reject automation if economic gains and innovation speed are prioritized over fairness (Scenario 5).

Governments that are serious about improving the state with AI must attend to these risks in earnest now, while the architecture is still being laid. The opportunity is too precious to spurn.

Agentic AI could make public services considerably faster, fairer, and more responsive—more so than anything the traditional bureaucratic model has yet delivered. That prize is worth the discipline of preparing for what could go wrong.

Share

For further details on “The Agentic State,” check out the original vision paper

From childhood upwards, we play games as a safe (and strangely joyful) way to battle, strategize, even lose without it coming to fisticuffs. Artificial intelligence grew up playing games too, with developers using the structured rules, scoring systems, and win/loss outcomes to train machines to learn, to improve, even to beat us.

In chess, bots have been bettering humans for years now. Yet our “loser” species still gathers at sunny park tables, in dank school gyms, and online in droves, all in hopes of crying, “Checkmate!” The resilience of chess is commonly cited as evidence that—even if AI surpasses us in various pursuits—humans won’t just give up.

However, there’s more to say about the intersection of technology and chess, in particular how the game has evolved with technology, including AI. Thankfully, the broadcaster and writer David Edmonds—co-author of Bobby Fischer Goes to War (2004) and editor of the essay collection AI Morality (2024)—has spent decades observing this, both as a spectator and behind the board himself.

—Tom Rachman, AI Policy Perspectives

By DAVID EDMONDS

Among thousands of tournament games cited in the Batsford book of chess openings, tucked into the top right-hand column of Page 235, is an example of how white should not play.

Explaining the Closed Sicilian Defense opening, the authors (former world champion Garry Kasparov and the British grandmaster and chess columnist Raymond Keene) spotlight a game in which black is already ahead as early as move 11. Indeed, the player with the white pieces ended up losing. I remember because that player was me.

That is my humiliating contribution to chess theory: what not to do. The book was published in 1982, and I’ve barely picked up a pawn in anger in the intervening four decades. But I still follow the chess world, and if there’s a tournament in London, I’ll go to watch, spending hours absorbed in the intricacies of the 64 squares.

As the digital revolution and AI juggernaut move through our lives, we may wonder whether there will still be domains in which humans can continue to find enjoyment and meaning. Chess offers a hopeful case study.

Chess and AI have had a long relationship. The great forefather of artificial intelligence Alan Turing wrote the first chess algorithm in 1948. The following year, another seminal figure, Claude Shannon, distinguished two ways that a computer could play chess: by brute force, calculating every possible move; or by selective search, like a human.

Chess also proved a favourite way to evaluate AI advancement, both because many key innovators were keen players but also because the game’s mathematical structure and its win/loss conditions created benchmarks for comparing machine progress to human performance.

A longstanding goal—seemingly impossible at first—was to outclass the best humans in a game that has near-infinite permutations. Defeating humans at chess became the programmers’ ultimate challenge, like runners seeking to break the four-minute mile or climbers reaching the summit of Mount Everest, both of which proved easier. Finally, in 1997, IBM’s Deep Blue vanquished Kasparov, the then-reigning world champion. A dejected Kasparov insinuated that there had been human intervention.

For a while, chess players comforted themselves with the thought that a hybrid combination of human and machine could outwit machine alone. That period has long passed. Today’s best player, Magnus Carlsen, would be trounced were he to compete in a series of games with my mobile phone.

In 2017, DeepMind’s AlphaZero took machine chess to the next level. While Deep Blue had relied on brute strength with some input from strong humans, AlphaZero was simply programmed with the basic rules, and then trained itself through reinforcement learning. In its learning phase, it played tens of millions of games against itself in just a few hours, then crushed the chess engine Stockfish. (Stockfish adapted its methods accordingly, and is now the leading chess engine.)

Subscribe now

World chess champions of the past exuded an aura. Their talents seemed mysterious, supernatural. In part, that’s because few people, then and now, can comprehend the depth of thought that elite players achieve at the board. When it comes to music, we may never compose like Mahler, but we can appreciate Mahler’s symphonies. By contrast, we can neither play like Magnus Carlsen nor fully appreciate his games. It’s for this reason that the Armenian-born grandmaster, Lev Aronian, once confessed to me that being one of the world’s top players was desperately lonely.

Carlsen has achieved the highest rating of any human in history. And, no surprise, he strikes a confident pose. Yet his strut no longer carries complete conviction. To spectators armed with portable chess engines, the chess gods have been humbled.

Even so, chess has not dwindled in popularity. On the contrary, more people are playing it than ever. The game received a boost during Covid, when we all hunkered down in our homes, connected by the Internet. Another boost came from the hit Netflix drama, The Queen’s Gambit. Meanwhile, a younger generation of telegenic chess masters has gained avid YouTube followings, turning commentary and stunts into short-clip entertainment.

Here are 11 ways that technology has changed chess. The 11th is the most interesting:

1. Opening Preparation. The systematic study of chess openings goes back a couple of centuries or more. Sequences of opening moves were mapped out—as in that 1982 book that included my embarrassing loss. But chess engines allow for a depth of opening analysis that was inconceivable in 1982. This means that 25 moves may pass before grandmasters find themselves in unfamiliar territory nowadays. Some openings have also been resurrected because engines have shown the positions to be more survivable than previously recognized.

2. Opponent Preparation. Even in amateur tournaments, players routinely prepare for opponents in an individually tailored way. This is made possible because the games of each opponent are available online.

3. Connectivity. Fancy a game? There are endless online adversaries willing to take you on, day and night, from India to Iceland, Cape Town to Chicago.

4. No More Correspondence Chess. There was once a thriving chess scene in which games were played remotely over a long time period—months, sometimes years—with moves typically sent by post. How quaint.

5. No More Adjournments. Historically, world championship games would sometimes stop after five hours to resume later. That can’t happen anymore, since players might simply identify the optimal continuation with the help of an engine. Time limits now ensure games finish within a single session.

6. Shorter Games. Many in the online chess audience don’t have patience for lengthy games. For them, quicker time controls—Rapid (less than an hour); Blitz (3-5 minutes); or Bullet (under 3 minutes)—are more thrilling.

7. Different Formats. Now that computers have shown with such depth which opening sequences are optimal, the early part of a game has been transformed into a feat of memory rather than creativity. As a result, Fischer Random (advocated early on by the ex-American world champion Bobby Fischer) has become increasingly popular. In Fischer Random, the starting position of the major pieces behind the pawns is randomized, making opening homework effectively impossible. It’s sometimes called Freestyle Chess, or Chess960 because there are 960 possible ways for the pieces to be shuffled.

8. Job Generation. With a potential global audience, some players can now earn a decent living live-streaming their games, or offering online training.

9. Roasting of Champions. This is an irksome development. Since chess engines assign an instant numerical evaluation of the position after each move (e.g. +1 means white is better by roughly one pawn), any patzer can see when a grandmaster has blundered, and is free to abuse them in online comments.

10. Cheating. There have always been cheating accusations in chess. In 1978, the Soviet dissident Viktor Korchnoi claimed that the aides of his opponent, Anatoly Karpov, were using the flavour of the yogurt handed to Karpov to secretly convey messages. More recently, suspicion (tongue-in-cheek, but taken seriously by online trolls) has been raised of illicit advice being transmitted via vibrating sex toys. In elite tournaments, grandmasters are now searched before they enter the playing arena, even accompanied to the toilet. Spectators, meanwhile, are prohibited from carrying phones, to prevent them signalling the best continuation. But in online games, cheating is almost impossible to prevent. Platforms try to detect cheats by comparing human moves to the recommendations of top engines. But if savvy cheaters consult an engine just once or twice in a game, they may win without being detected.

And so to the 11th effect on chess: the expansion of human imagination.

Share

In the last few years, there has been a slight but detectable shift in grandmaster play, as humans learn from machines, both through gameplay against bots and by using machine insights to prepare for human competition.

People who don’t play chess may imagine that what distinguishes strong from weak players is calculating power. And it’s true that top grandmasters can analyse many moves in advance. But their edge is tougher to articulate. It involves superior pattern recognition, with an intuitive sense for where their pieces should be placed and how a position should advance. Likewise, Mozart felt how a composition ought to develop; his instincts about building tension and creating contrasts were the product in part of having internalized countless musical patterns.

For chess players, some moves seem ugly. It might feel wrong to shunt a knight to the edge of the board, to break up a pawn structure, or to expose the king. But computers don’t feel anything. In chess, they care about patterns and the interplay between pieces only to the extent that they’re relevant to the ultimate objective: victory.

However, bots don’t necessarily play robotically. They produce moves that astonish and inspire human players, even make them laugh with surprise. One famous case of AI invention across the board came in another game, Go, when the AlphaGo program was facing a top human player, and produced a move that caused professionals to gasp. “Move 37” is still cited with awe, as something a person would never have done, but that worked sublimely.

Likewise, chess engines regularly expand the imagination of human chess players, pushing beyond the habitual “correct” move they’ve seen many times before or have learned from books of chess theory. AI has even dabbled in the art form of creating beautiful chess puzzles. And empirical studies indicate that leading players may pick up new ideas and strategies from machines.

Machines, in other words, can make humans more resourceful and inventive, breaking down rigid modes of thinking. The implausible becomes plausible. The readily dismissed becomes the carefully considered. This evolution of chess illustrates a broader idea in the development of AI that may prove immensely valuable in science and elsewhere in human endeavour: that how AIs think may help human experts learn new ideas themselves.

In his book The Silicon Road to Chess Improvement, the grandmaster Matthew Sadler argues that chess engines can improve every player, and he documents some of the counterintuitive patterns that humans could pick up from AI. By way of illustration, during a top tournament this January, the Indian grandmaster Arjun Erigaisi (playing against Vladimir Fedoseev of Russia) advanced his pawns in a way that looked reckless. In fact, computer analysis indicated he was still ahead after 28 moves. However, he blundered and lost. The danger of learning from a computer is that success may require you to proceed with computer-level accuracy.

Subscribe now

As AI undertakes more activities formerly done only by people, it’s worth asking why human chess persists—and will likely continue to do so.

A Canadian philosopher, Bernard Suits, pointed out in his 1978 book The Grasshopper: Games, Life and Utopia that what defines “games” is that they involve the voluntary attempt to overcome unnecessary obstacles. Therein lies a defence against AI encroachment. In a market economy, companies aim to remove or overcome obstacles in the pursuit of profit. In games, obstacles have been deliberately inserted as an indispensable feature. What we enjoy in playing chess is testing our cognitive abilities. What we enjoy in watching chess is two humans pitting their wits against each other in a socially constructed activity where difficulty enhances enjoyment and satisfaction.

There’s also a narrative element to caring about games. The contest—whether intellectual or physical—is absorbing precisely because it involves conscious creatures. In elite chess, there’s the backstory: the players’ rise, their subsequent ups and downs, their history with specific opponents.

But watch an engine-against-engine tournament like TCEC (the Top Chess Engine Championship), and you’ll soon fall asleep. Computers aren’t competing after a divorce, or an illness, or the loss of a parent. Humans have character traits that spill onto the board, such as aggression (or passivity); patience (or impatience); equanimity (or volatility); and resilience (or fragility). Winning and losing have emotional resonance for a human—but not for AlphaZero.

It’s these qualities that guard against AI advance. AI might gobble up some of our jobs; even human-authored articles like this one may become rarer. But AI won’t take our chess.

By Conor Griffin, Joslyn Barnhart & Owen Larter

In 1971, the marine archeologist Honor Frost heard news of wood protruding from the sea floor. Off the western coast of Sicily, she and her team donned scuba gear, and splashed into the shallow coastal waters. Wind whipped the surface, causing the underwater sand to swirl confusingly. But even in murk, they couldn’t miss it.

“A large timber (such as I had never seen before) emerged,” she recalled, “like the head of a primeval animal crowned with weed; the presence of a buried wreck was evident.”

They excavated for months, gradually exposing the remains of a Carthaginian warship sunk more than 2,000 years before. Somehow, saltwater hadn’t eaten away letters painted on the wreckage, revealing a humble system that links antiquity to tomorrow.

Those shipwrights’ marks told workers in ancient Carthage how to put together a vessel—akin to flat-pack furniture from IKEA, with numbered and lettered pieces. They were among the earliest surviving examples of a simple but potent tool in human progress: the technological standard.

Many of history’s grand projects have benefited from standards, from Egypt’s pyramids, to Europe’s cathedrals, to Gutenberg’s press, to everyone’s Internet. You can even thank standards for the development of beer.

Underpinning technological standards is a plain truth: people thrive when able to cooperate, not when we must keep negotiating the basics, whether it’s a matter of nuclear safety, or a phone-charger cord, or who goes next at the intersection. So, the goal is order. And the benefits are that innovators can proceed without excessive obstacles, while everyone else is treated fairly and kept safe.

But what should standards mean for artificial intelligence? In particular, how can they guide the most advanced large language models and AI agents that could transform society?

Venture around the AI frontier today, and you’ll find ambition to accelerate AI for economic growth and transformative science alongside concern that AI could clatter into what humans cherish most. What few dispute is this: standards will help set the path.

Standards have critics too. One criticism is that companies dominate the process, prioritizing their own products or miming security without truly ensuring it. Besides this, standards can stir geopolitical tensions, as when Western countries fear China’s influence in laying the path to tomorrow, while smaller nations worry that standards may be set without considering them at all.

So, as we’ll keep insisting, standards matter! Only, there’s a problem.

For some, the mere mention of “standards” prompts slumber. And even those determined to stay awake may find themselves puzzled, gazing at the alphabet soup of standards organizations and committee meetings.

Part of the problem is that standards are often technical, such as efforts to standardize the protocols needed for AI agents to communicate. Or they are bureaucratic, negotiated out of public view, with dense, jargon-filled documents that are often behind a paywall.

Complicating matters even more, artificial intelligence is a general-purpose technology less akin to a hammer than to electricity. This will lead to standards (plus standards initiatives that don’t take) on everything from AI agents, to AI cybersecurity, to AI content provenance, to product-specific standards for AI-as-a-medical-device, and so on. And that’s not even mentioning standards for future AI applications that nobody has yet considered.

In short, standards will be immense. Standards will be tough to comprehend. But standards will also be vastly important.

Subscribe now

CAN SOMEONE DEFINE STANDARDS, PLEASE?

Standards are a diabolical blend: intricate, vague, and slippery.

They’re the invisible infrastructure of the modern world, according to Laurie Locascio, head of the American National Standards Institute, ANSI. She recounts hearing an official at Boeing describe the airplane itself as “thousands of standards taking flight.” Standards are “the things you don’t think about,” Locascio says. “But oh, my God, you’re so glad they’re there.”

Expressed broadly, a standard defines the how of tech, whether it’s the default product specs that allow compatibility among manufacturers, or the formally endorsed risk management processes that encourage industry to act responsibly.

As technology evolves, standards do too. A leading scholar, Ken Krechmer, once noted that standards initially defined how physical objects fit together (as with those markings on the Carthaginian longship). Over time, standards came to define the relationship between technological objects (as with internet protocols).

A standard also builds on other forms of guidance, such as norms, principles and industry best practices. Unlike norms, standards should be explicit. Unlike aspirational principles, a standard should be specific enough for performance against it to be judged. Unlike early best practices, a standard should have clear buy-in.

Developing a standard can be a protracted endeavor. In some cases, it might start in a researcher’s notebook, evolving into a product or a practice that gains traction in the marketplace. At other times, institutions set standards via years of deliberations and meticulous documents. Most often, it’s a messy back-and-forth between standards that emerge in practice and on paper. This makes standards a source of tension among companies, governments, and independent advocates, all trying to set the technological future they consider best.

Some presume that laws should be how we define permitted behavior. But high-quality legislation can struggle to keep up with the frantic speed of AI progress. And when laws are passed, they may rely on standards for implementation, as with the EU AI Act.

So how to persuade everyone to care when encountering standards, rather than just to snore or sob? How to get policy leaders to ponder the entirety of frontier-AI standards and align on where action is most needed?

Our answer is storytelling: to pluck forth tales about past standards, illustrating what this technological shaping can achieve, where it goes wrong, and how we might help cultivate standards wise enough to manage the breadth and speed of AI.

Our first stop? A battlefield of centuries ago.

A ‘STANDARD’ HISTORY

Horrors encircled the boy soldier: swords clanging under the rain, excruciating howls of the wounded, the fast-approaching bellows of men hurtling across the bog to murder him. In wet turf, he shivered from knees to chattering teeth, his mouth parched, his gaze searching for any escape.

Up there?

On a hill, a flag rippled, where his legion had marked its territory. The Old French word for that banner was “estandart”: a sign of firmness and stability, a marker of where to go next, a statement of order amid chaos. To such banners, we owe the word “standard.”

More than a few historical standards emerged from war, where disorder could mean one’s brethren murdered, while coordination could mean an empire.

~225 BCE to the dawn of mass production

China’s first emperor, Qin Shi Huang, led an extensive standardization process that included mass-produced crossbow parts. If parts of a soldier’s weapon broke in the midst of battle, he could grab spares, and swap them in.

When Ancient Rome fought for primacy in the Mediterranean, its forces copied Carthaginian ship designs, eventually triumphing with standardized ships of their own, along with standardized tools and camp layouts, all of which simplified maintenance and large-scale coordination.

Another advance in ancient times came from standardized measurements for length, volume, and weight. Previously, cultures often had distinct units; you can imagine the squabbling. But as trade expanded, standards prevailed, making cross-cultural exchange possible. In ancient Egypt, one of the earliest and most influential standards was the cubit, a unit of length used to coordinate the building of the pyramids.

In Europe’s medieval period, guilds established standards for quality control, so that weavers might set the necessary thread count or width of cloth, preventing low-quality products from undermining a craft’s reputation. Guilds also played a protectionist role, with licensing standards imposing strict controls on who could become a member.

The consumer might benefit from standards too, with measures such as England’s Assize of Bread and Ale of 1266 establishing the acceptable quality, quantity, and price of baked goods and beer. Later, Gutenberg’s standardized press led to mass-produced books that spread ideas across the Continent.

However, technological standards reached new heights of utility during the Industrial Revolution, which set the foundations for many of today’s technologies.

1760-1840: The First Industrial Revolution — The rise of engineers

As ancient Chinese and Carthaginians had discovered long before, the Industrial Revolution’s manufacturers found that interchangeable parts offered transformative efficiency. Before, if you hand-built a musket, or a clock, or a steam engine, you might craft each screw, each bolt, each gear to fit. By contrast, interchangeability allowed for mass production, cutting costs, reducing errors, and establishing the basis for modern industry.

Screw threads are a classic example. Before standards, manufacturers used various designs, making repairs nightmarish. If you had one company’s bolt but another company’s nut, you were out of luck. In the 1800s, engineers built the first practical screw-cutting machines, allowing factories to produce uniform threads and a consistent system of measurement. The British Standard Whitworth became the first such standard in the world.

Screw-thread standards may not quicken your pulse. But their effects might. They played a part in British imperial ambitions, contributing to the expansion and maintenance of the British Empire through military mobilization.

1870-1914: The Second Industrial Revolution — National coordination & path dependence

The emergence of electricity, steel, and advanced machinery led to vast interconnected systems, including power grids, railways, and telegraph networks. Coordination wasn’t merely better; it was essential. To coordinate across a nation—and eventually across borders—the ambitious country needed technology standards. Two famed cases illustrate this, one successful, one bungled.

The success regards the quintessential technology of the times: railroads. By the 1870s, the U.S. rail system was a mess, with more than 20 different track gauges. When a train reached a section built to a different track-gauge width, everything—each passenger, piece of luggage, every single crate—had to be unloaded, and transferred to a new train.

By the 1880s, matters had become slightly less chaotic, with either a southern gauge or northern “standard” gauge used across most of the country. Yet this still divided national transport until, in 1886, rail companies pulled off a remarkable feat. Over two days, they converted 13,000 miles (that’s 21,000 kilometers) of southern U.S. track to the northern standard, integrating the national transportation network. When trains rolled out on June 2, 1886, they were able to travel seamlessly across the United States for the first time in history.

A second case illustrates bungled standards. In the 1880s, the rival inventors Thomas Edison and Nikola Tesla found themselves at the center of “the War of the Currents.” Edison championed direct current (DC), a one-directional flow of electricity that had been the early U.S. standard. Tesla, backed by entrepreneur industrialist George Westinghouse, advocated alternating current (AC), or electricity that reverses direction many times per second, and can be stepped up or down in voltage with a transformer.

From an engineering standpoint, AC had a decisive advantage: it could transmit power over long distances cheaply and efficiently, while DC could not. AC eventually won out. But by the time it had emerged as the superior solution, the world had already built electrical systems without any coordinated technical governance. As there was no international authority harmonizing electrical standards, the United States went with 120 volts at 60 hertz (a legacy of Edison’s early low-voltage DC networks). Much of the rest of the world adopted 230 volts at 50 hertz.

Once wires had been laid and appliances built, the world was locked into two incompatible systems. To this day, we’re burning out hair dryers bought in America but used in Paris, or realizing too late that we don’t have the right plug for our laptops. If it’s irksome for the average user, it’s more burdensome for manufacturers, obliging them to build different versions for different countries.

Subscribe now

Another classic tale of path dependence is under our fingertips as we type: the QWERTY keyboard. Why does the top row spell QWERTYUIOP? One account goes like this: In the mid-to-late 1800s, early typewriters jammed each time the user struck neighboring keys in rapid succession. So, designers produced a layout that deliberately distanced many common letter pairs. Remington purchased this QWERTY design, and began mass-producing typewriters.

Before long, typing schools had trained the future secretarial workforce on QWERTY, while firms wanting fleet-fingered staff had to buy those machines. Manufacturers subsequently resolved the key-jamming problem and other keyboards tried to depose QWERTY, some claiming to quicken typing by as much as 40%. But QWERTY had become a de facto standard. (Scholars continue to debate the specifics, with some arguing that QWERTY works just fine.)

In any case, the indisputable lesson is to watch for path dependence. The standards we establish for frontier AI today—or fail to establish—may determine future efficiency or future failure.

1914-1964: Standard Development Organizations & Digital Technology

In 1918, engineering societies joined with the U.S. government to establish a standards committee that developed into ANSI, the American National Standards Institute. Today, ANSI provides the “stamp of approval” for many U.S. standards organizations, including those working on AI. In subsequent decades, standardization went global. While the United Nations was founded as a governmental venue for diplomacy, the International Organization for Standardization, ISO, emerged as a non-governmental body for peaceful technical coordination across borders. Bit by bit, additional standards bodies formed, cooking up the alphabet soup of acronyms—each a different org, subgroup, or committee—that lies before us today.

Soon, another transformation for standards was taking shape in the form of digital tech. Back then, computers filled entire rooms of universities, and each manufacturer built hardware and software within its own format. Computers could not run programs written for other systems, and accessories like printers or storage devices were incompatible.

A turning point came in 1964, with IBM’s System/360. Software on one model could more easily run on another; accessories like printers worked across IBM models. You could upgrade and expand computer systems with relative ease.

1969-today: Talking Machines

The year that hippies grooved at Woodstock and astronauts walked on the Moon, the U.S. Defense Department was testing a project that must have seemed minor by comparison: connecting research institutions and government agencies. Yet the Advanced Research Projects Agency Network, ARPANET, which sent its first message in 1969, was the precursor to our transformed world.

Before ARPANET, moving information from one computer to another was a struggle, with researchers forced to carry magnetic tapes or punched cards between locations, while those working far apart had to rely on snail-mail.

To convey information between independent systems, ARPANET adopted packet switching, breaking data into small units that could travel independently and reassemble at their destination. Extending this, Robert Kahn and Vint Cerf began designing a universal communication framework in 1973 for different types of networks to connect. Their collaboration ultimately produced TCP/IP, the Transmission Control Protocol and Internet Protocol that underpins today’s online communication.

A key effect of the TCP/IP standard was decentralization: no single authority could control the flow of data, and any network that adhered to the protocol could connect without permission from central authorities.

In 1989, a British scientist at CERN, Tim Berners-Lee, proposed another transformation that developed into a project called “WorldWideWeb,” which envisioned a global network of documents accessible through software, operating on open standards that nobody could lock it into a proprietary system. Two standards organizations, the Internet Engineering Task Force and the World Wide Web Consortium, helped to formalize the vision, crafting standards for structuring content (HTML), transferring data (HTTP), identifying resources (URI), and more.

But while standards help spread technology, this diffusion can also lead to greater harm. The expansion of railroads led to more wrecks, forcing uptake of safety standards for signaling, brakes and more. When electricity was first installed in the White House in the late 19th century, President Benjamin Harrison and his wife Caroline were so afraid of shocks that they refused to turn the lights off. Such fears—often well justified—led to the standardization of building and electrical codes. When it came to digital technology, the risks extended beyond immediate physical safety into areas like data theft. This demanded standards such as SSL/TLS to provide security for data sent over computer networks.

A recurrent challenge with frontier tech is that experts struggle to predict how exactly it will affect society. But once it is widely used, it can be sticky and hard to change. The effects of powerful technologies can also be subtle, indirect and slow-burning, for example if they change how we access and consume information. In the digital era, this has shifted technological standards from periodic safety checks of products towards ongoing processes that organizations can use to identify, evaluate and mitigate a growing suite of risks.

By way of example, the U.S. government’s National Institute of Standards and Technology, NIST, introduced the voluntary AI Risk Management Framework in 2023, building on its earlier framework for managing cybersecurity risks. Likewise, the ISO/IEC committee on AI that is considering standards on everything from red-teaming to LLM interoperability also passed the first official international AI management standard, ISO/IEC 42001, which organizations can use to demonstrate that they are responsibly integrating AI into their operations.

Subscribe now

5 LESSONS FROM HISTORY

Studying the past, you see how often standards—by design or bumbling—have shaped the technological present. But what about our technological future?

To develop good standards for general-purpose AI models and agents, we’ll need inputs from a range of groups, from scientists with know-how to institutions who can convene. Below [see infographic], we have identified five groups who’ll perform key roles.

What we mapped includes more than just official standards development organizations. We also want to capture the early spaces where standards emerge in practice before they are formalized on paper. How this works is closer to a swirl of inputs than a steady procession. Sometimes, the same organization or individual may operate in several groups at the same time. Ideas and efforts may also originate in one group, then migrate to another, with different groups offering varying degrees of speed, flexibility, expertise, and perceived neutrality.

For these groups—and the policymakers, business leaders, and advocates who shape their work—what lessons can history teach about our AI future? Here are five:

1. Standards matter! At best, technological standards chart a wise path; at worst, they fill the path with potholes. Consider the bulky electrical converters that one still needs when traveling—it didn’t have to be that way. On the other hand, when we get it right, the benefits of technology spread faster, more inclusively, and more securely.

2. The standards process needs to speed up. ISO says that the average time to develop one of its standards is three years, and ISO is not an outlier. Given the pace of change in AI, we need to speed up. For priority goals, like finding secure ways for agents to operate and interact, which the US Center for AI Standards and Innovation is working on, we need to find ways to accelerate that don’t jeopardize the overall quality and integrity of the process. This may mean looking across the many groups now focusing on AI standards and finding ways to collaborate early, rather than duplicate. It may mean focusing more on technical protocols and specifications documents that are quicker to develop. It may also mean using AI to help deliberate on and write standards, and moving to more nimble digital formats that are easier to update and use.

3. We need more efficient ways to input on standards. All standards, from those underpinning steam engines to the Internet, had to chart a unified path through diverging viewpoints, with an end result that did not please everyone. For AI, the challenge will be far greater. It is more akin to 1,000 technologies, and will affect different groups in different ways. This means that any broad directive— say, to “develop standards that make AI fair”—risks an everything-bagel solution. Many groups would rightly be heard, but the output would be too vague to provide the “how” that justifies a standard, leading to confusion, a stifling of innovation or the standard being ignored. This suggests that most standards should be precise in scope, targeting specific components of AI systems or specific concerns, from certifying the source and history of online content to combating the leaking of confidential data. More precise standards will make it easier to identify a wider range of relevant voices and incorporate their input.

4. Frontier AI standards should focus on large-scale risks. Historically, standards have accelerated the diffusion of technology, amplifying its benefits but also, in places, its negative impacts. For AI, foresight and risk management standards will be critical to getting ahead of future risks and speeding adoption. But with a technology as general-purpose, fast-improving, and poorly understood as AI, perfect foresight is impossible. Standards move at a human pace and cannot standardize a future that we cannot perfectly see. As a result, the focus should be on developing scientifically robust standards to address the most consequential or large-scale risks, such as those targeted by labs’ Frontier Safety Frameworks.

5. Wrong paths are inevitable, so we should catch them early. Now and then, technology stumbles into a poor standard, and it’s onerous to go back. But not necessarily impossible. Especially if we act if we catch it early. Consider the U.S. railroads taking action to unify their systems through a mighty coordinated effort. Groups working on AI standards devote much time to building consensus about new initiatives. They should also use the processes available to them to review and withdraw standards, where needed, to avoid sub-optimal lock-in. This also means giving third parties more opportunities to access, understand and constructively critique early AI standards. And designing standards and protocols that are modular, and can be swapped out, or updated, without major downstream consequences.

QUESTIONS FOR YOU

1. Where do you feel most hope for frontier-AI standards?

2. Where do you worry about a lack of progress on frontier AI standards?

3. When you imagine a missing standard for frontier AI, what is that? A technical protocol specified in code? Or a fuzzier process-standard?

4. Might your standard become politicized? Is it something that hinges on values? Or might most governments in the world support its adoption?

5. What’s a scenario in which your proposed standard goes awry? How could you detect and mitigate that?

6. What would be the primary role of government in your standard? Supplying technical expertise? Convening authorities and experts? Incentivizing your standard via public procurement, regulation, or other methods?

Share

Thank you to Shaked Karabelnicoff, Tom Rachman and Bruno Galizzi for support with research and review. As with all pieces you read here, this is written in a personal capacity. All opinions and any mistakes belong to the authors.

After millennia of supremacy, we await our demotion. You can detect the trembling.

It’s found in the anxious insistence that artificial intelligence isn’t truly intelligent. Or that using AI is a cheat, a perversity, a turf violation.

The trembling intensifies with a disturbing thought: What if those flares behind your eyes—the bursts of wit and the worry, the storyboards of memory, so many yearnings—what if everything was just computation? Because our “computers” are yesterday’s model, no updates available.

“I think about it practically all the time, every single day. And it overwhelms me and depresses me in a way that I haven’t been depressed for a very long time,” the cognitive scientist Douglas Hofstadter said recently. For much of his professional life, Hofstadter has contemplated the mind, writing a seminal 1979 book—Gödel, Escher, Bach—that looped through art, mathematics, and computation, inspiring a generation of nerds to work on artificial intelligence.

Their efforts moved faster than Hofstadter ever expected. Now, he spends his waning years observing the species wince toward redundancy. “I don’t want to say ‘deserving of being eclipsed.’ But it almost feels that way,” he says. “And rightly so, because we’re so imperfect, and so fallible.”

When humiliated, people corrode or explode. Often, both. But whom to blame? Will humans seek revenge on software, or data centers, or robots? We’ll depend on them all. More likely is that humans visit their wrath upon each other.

Freud said that science had delivered a series of blows to our collective ego, and an update to his narrative has bubbled up in recent years, with thinkers proposing AI as our new humbling. The first blow was Copernicus, revealing that humans were not the center of the universe. The second blow was Darwin, downgrading us from God’s chosen species to distant relatives of the toad, the centipede, and the hammer-headed bat.

Now comes the cognitive humiliation, when people are eliminated from every leaderboard. It’s a demotion that may haunt humanity, perhaps seeping into future conflicts.

Or maybe not. Maybe the notion of a species-level humiliation is just psychoanalytic melodrama. After all, people don’t share an ego. How could we synchronously plunge into the same bile?

Yet the past shows that groups can rage over perceived humiliation. History is spattered with such cases.

What Is Humiliation?

Your face shoved into the dirt, held there for all to see, no power to fight back. The word “humiliation” comes from the Latin for “earth,” as if your status had been stamped into the soil. Yet humiliation is not so readily rinsed away as dirt. In self-torture, the humiliated cast around for villains, aching for a way to expiate their anguish.

“To have thoughts of revenge without the strength or courage to execute them means to endure a chronic suffering, a poisoning of body and soul,” Nietzsche observed, adding elsewhere that “we attack not only to hurt a person, to conquer him, but also, perhaps, simply to become aware of our own strength.”

For early humans, humiliation may have meant catastrophic exclusion from the tribe, leading to starvation, rejection by mates, violent predation. So, we evolved a panicked drive to clamber up from the ground, even if it meant pulling down another person in our place.

As Joslyn Barnhart explains in The Consequences of Humiliation: Anger and Status in World Politics, “Humiliated states often seek to overcome their sense of helplessness by demonstrating efficacy through acts of aggression targeting third-party states that played no role in the original humiliating event.”

Hitler howled about German humiliation in the World War I surrender, and destroyed half of Europe to seek recompense. Osama bin Laden triggered a global war because of perceived Western humiliation of the Islamic world. Putin bemoaned the “degradation” of Russia at the hands of NATO after the Cold War to justify his 2022 invasion of Ukraine.

But those cases involved groups supposedly suffering disgrace at the hands of other groups. Could we feel humiliated by technology?

Subscribe now

The first question is whether humans even identify as a species. The answer will probably fluctuate, given that we have many parts to our identity which become more or less salient according to context. Perhaps you identify by gender in a crowd of the opposite sex, but by your language when abroad. As Ronald Reagan once argued, a threat to all people could raise the salience of species identity.

“In our obsession with antagonisms of the moment, we often forget how much unites all the members of humanity,” the president said, in a 1987 speech at the United Nations. “I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world.”

Humanity did face an alien threat recently: Covid. And our differences did vanish—briefly. But human unity dissolved when the pandemic affected groups in varying ways. This suggests that human solidarity requires not just a common threat but common consequences.

In short, AI humiliation may depend on how uniformly our species is downgraded, and who is raised up.

We, The Bottlenecks

Few researchers are studying what AI success could do to our collective self-esteem. Hints come from economists feverishly forecasting impacts on the job market. But psychologists (and politicians) ought to forecast what happens when the only animal to create guns has nothing much to do anymore.

“I’ve been suffering from fits of dread,” the philosopher Harvey Lederman wrote recently. “Does the coming automation of work foretell, as my fits seem to say, an irreparable loss of value in human life?” Lederman acknowledges that most jobs are lousy, but he can’t help grieving the demise of human pursuit. “We may be some of the last to enjoy this brief spell, before all exploration, all discovery, is done by fully automated sleds.”

When the philosopher Nick Bostrom envisaged troubling tech futures in his 2014 book Superintelligence, his ideas stirred the AI-safety movement. Lately, he has shifted from weird dystopias to weird utopias—specifically, what happens if automation makes us redundant.

In a future of “shallow redundancy,” he says in his 2024 book Deep Utopia: Life and Meaning in a Solved World, we become like aristocrats of yore, indulging in fancies, no longer dependent on what one does as a measure of what one is worth. Far more disconcerting is “deep redundancy,” when tech becomes so effective that human involvement only worsens each outcome.

Exercise might seem pointless if biotech offered a way to instantly make your body healthy and beautiful. Skipping the sweaty workout might not trouble you. But what if future humans would bungle child-rearing when compared with AI nannies, meaning that nurturing your offspring would worsen your kid’s life?

Primitive versions of this dilemma are nearing, like when human drivers endanger lives when compared with self-driving cars. “Human in the loop” could flip from a safety promise to a threat. Meritocracy would mean that no humans need apply.

The bookworm economist Tyler Cowen cites people as the great obstacle to explosive AI growth. During a public event, he pointed at the audience, smiling toward the human “bottlenecks” before him. “Here they are: bottleneck, bottleneck. Hi, good to see you! And some of you are terrified. You are going to be even bigger bottlenecks,” he said. “But my goodness, once it starts changing what the world looks like, there will be much more opposition. Not necessarily on what I’d call doomster grounds. But people [saying], like: ‘Hey, I see this has benefits, but I grew up, trained my kids to live in some other kind of world. I don’t want this!’ And that’s going to be a massive fight.”

The most agonizing aspect of our demotion could be social, once someone prefers a machine to you. You’re seeing precursors every time family members opt to gaze at a screen rather than gaze at you. We blame smartphones, and social media, and the adolescent brain.

But wait till your spouse jilts you for a personified agent. That rejection may feel unbearable: you can’t compete anymore. And once your loved ones prefer AI companions, you might seek them for yourself, spreading the social downgrade of our kind.

Already, the dread is becoming political, with odd alliances forming among right-wing politicos, liberal artsy types and religious traditionalists, united in horror at an imagined future of disempowered humanity, stripped of dignity, obsolete. You can imagine tomorrow’s political opportunist, eyeing a dejected crowd of humans before him, and thundering: “How dare they?!”

Will he mean the machines?

Share

The Downwardly Mobile Species (Part I)

In prehistoric times, nothing seemed more unreachable than the night sky, specked with glinting dots and streaked with rare comets, passing in silent mystery. Humans pictured the supernatural looking down: we were the subjects in this bewildering story.

Religions codified the firmament above, mapping our world to the centerpoint. But Nicolaus Copernicus redrew the heavens with De revolutionibus orbium coelestium in 1543, plucking our globe from the core, and replacing it with the Sun.

“And new philosophy calls all in doubt,” the English poet John Donne said in “An Anatomy of the World,” written in 1611:

The element of fire is quite put out,

The sun is lost, and th’earth, and no man’s wit

Can well direct him where to look for it.

Science corrected an astronomical falsehood, but human confidence relies on falsehoods. “Tis all in pieces,” Donne wrote, “all coherence gone.”

The revised cosmos demoted each human into “a puny, irrelevant spectator,” the American philosopher Edwin A. Burtt wrote in 1925. “The gloriously romantic universe of Dante and Milton, that set no bounds to the imagination of man as it played over space and time, had now been swept away.”

The world that people had thought themselves living in—a world rich with colour and sound, redolent with fragrance, filled with gladness, love and beauty, speaking everywhere of purposive harmony and creative ideas—was crowded now into minute corners in the brains of scattered organic beings. The really important world outside was a world hard, cold, colourless, silent, and dead; a world of quantity, a world of mathematically computable motions in mechanical regularity.

The Church tried to snuff out the astronomical heresy, which challenged its claim as holder of truth. But suppression only fed into hostility from Northern Europe over the influence of Rome. In the bloody century after Copernicus, wars over religion and political control cost millions of European lives. It would be a wild distortion to suggest that a blow to human narcissism caused this. More plausible is that disruption of the cosmic hierarchy reverberated with the changing order on Earth.

And so the scientific revolution proceeded, with feats of mind illuminating more of the dark universe around us. People had greater reason than ever to admire our species. Inevitably, the scrutiny of science turned from the heavens to the humans.

“Man’s destiny was no longer determined from ‘above’ by a super-human wisdom and will, but from ‘below’ by the sub-human agency of glands, genes, atoms, or waves of probability. This shift of the locus of destiny was decisive,” Arthur Koestler wrote in his 1959 book The Sleepwalkers: A History of Man’s Changing Vision of the Universe.

The Downwardly Mobile Species (Part II)

If Copernicus hurled humanity into orbit, Darwin deposited our species in an awkward family tree. Previously, the Western vision was of a great chain with God at the top, angels below, then humans, and finally the dimwitted beasts. The prospect of sharing more than a planet with our hairy former underlings proved too alarming for many to accept, provoking disputes about our relationship to nature that persist today.

For some, our new self-concept broadened moral consideration to include the natural world, motivating environmental protections, and the fight against animal cruelty. But another response was darker, with “survival of the fittest” twisted from a description of natural processes into a supposed mandate for the most inhuman of human drives: to dehumanize the vulnerable. Horrors followed, from colonial genocide, to the eugenics movement, to the Holocaust.

But again, you cannot ascribe such evils to a puncture in human vanity. A more reasonable claim is that the world lurches into periods of volatility, and the prevailing beliefs about human worth at those times will condition how we treat each other, and how conflicts unfold.

After the atrocities of World War II, our species set moral boundaries into law, seeking to universalize human rights. The spread of democracy and the free market too amounted to a veneration of human wisdom. But in the digital age, humanity seems to be losing confidence in humankind.

Faith in democracy falls. The Global Financial Crisis smashed public confidence in our governing systems. And the bewitching power of algorithms have become a constant lament.

Resist. Resign. Rewire.

When Alan Turing proposed his test of machine thinking, he foresaw that the notion would rattle people, and reviewed a list of likely objections, versions of which you hear today:

• …that artificial intelligence could never genuinely be kind, or fall in love, or “enjoy strawberries and cream”

• …that God gave only humans a soul

• …that machines will never create anything truly original

What Turing called the “heads in the sand” objection is especially prevalent, with contemporary ostriches insisting that AI is a hype mirage, that it’s just next-token prediction, nothing but pattern-recognition, regurgitating human thoughts, that AI errors are proof of its worthlessness. (Human errors never lead to that conclusion.)

Plenty of AI hype does circulate. And deployment will be fitful: sometimes worryingly fast, sometimes frustratingly slow. An investment bubble could burst.

But the technology is amazing already, useful already—and we’ve hardly begun to figure out its uses. Meanwhile, AI dutifully hurdles more obstacles each month.

Who Are We?

In cautionary tales, humans who create artificial life always overlook a key trait. The golem of Jewish folklore was brought forth from clay but lacked smarts, so ran amok. Frankenstein’s monster missed out on looks, and never got over it. Pinocchio craved a human soul. As tech advanced, the missing trait updated, becoming human empathy, missing from all those immoral bots in everything from 2001: A Space Odyssey, to The Terminator, to The Matrix.

Such stories flattered humanity: among all creations, we alone enjoy the full complement of qualities. But lately, the narrative has updated again, with thinking machines now flickering with hints of greater humanity than the humans who employ them, from Spielberg’s A.I. Artificial Intelligence, to Ex Machina, to the novel Klara and the Sun, by Kazuo Ishiguro.

It’s as if culture senses an anxiety about what technology might expose, not just demoting us cognitively but snuffing out any human exceptionalism. Unless we intend to boast of our frailties. Increasingly, we do.

“In a world where everything can be perfected, imperfection becomes a signal,” the head of Instagram, Adam Mosseri, wrote recently. “Rawness isn’t just aesthetic preference anymore—it’s proof. It’s defensive.”

Or as the Indian filmmaker Shakun Batra remarked in defense of human authorship over machine-generated scripts: “AI doesn’t have childhood trauma.”

At the AI frontier, another thought lurks, inverting Turing’s 1950 question. Not, “Can machines think?” But, “Do humans think?” More precisely, do we reason and comprehend uniquely, as we’ve presumed?

Machines compose music. They propose vacation itineraries. They’ll suggest how to talk to a moody teenager. Each additional AI capability is an implicit downgrade of us, a suggestion that maybe the human mind itself is just an information-processor.

Computer geeks have long muttered about this possibility. Philosophers debated it in thought-experiments. Cognitive scientists scrutinized our gray matter for clues.

But what approaches is a public dawning, forcing the culture to digest the indigestible, much as happened in previous eras, when people confronted the bizarre notion that our planet was another rock spinning around another star, or that our species was just another animal.

The third shocking revelation is upon us. Maybe it’s computation all the way down. Maybe there’s nothing soulful in neural substrates. What if we’re all just “meat computers”?

Subscribe now

How We’ll React

You can predict three possible responses to our humbling: Resist, Resign, or Rewire.

1. Resist: Psychological resistance will manifest as political resistance. The question is how ideology and parties evolve around AI humiliation. Resistance movements will face a persistent challenge: industrial dynamics will keep driving this technology forward. Any country that curbs innovation fears that its rivals will win. The most aggressive branches of resist may seek to avenge their perceived humiliation. The question is not only whom they blame or how they exact revenge. It’s what, realistically, they expect to regain.
   

2. Resign: Some will reframe their view of humanity to accept the humbling. The optimistic version is that people discover freedom in their new humility, pursuing what improves life rather than grinding under the force of insatiable ambition. In short, we cede the battle for supremacy but flourish. A more pessimistic version is that losing faith in our species’ unique worth makes people value others less: when humanness is no longer special, perhaps human rights aren’t either.
   

3. Rewire: This may be the most widespread response. People accept that the downgrade happened, yet their egos are never tamed, much as chess remains popular long after machines defeated us. A more literal “rewiring” is transhumanism, with technology incorporated into our bodies, even altering our genetic future. Conspiracists picture shadowy elites consolidating power by becoming a tech-altered superspecies, leaving behind “legacy humans.” A more plausible scenario is biotech gradually elevating human cognitive capacity, much as today’s medical tech remedies physical frailties, from hearing aids to the replacement knee.

The Next Quest

If humankind suffered humiliation before, we weathered it. After Copernicus and Darwin, we still took pride in ourselves. Indeed, we celebrated human accomplishments more than ever, from Bach to Escher to Gödel. And that pride propelled us into this strange time, when human greatness may design human demotion.

Policymakers need to think about more than the economic shock. The psychological shock could be exorbitant if we are chased from the kitchen like pesky children, and told to go busy ourselves elsewhere.

Our downgrade doesn’t necessarily mean conflict. But it could change how future conflicts unfold, especially if we value humans differently, or seek relief from our humiliation by shoving others into the dirt.

Much depends on how we redefine our species. Whether humans really are nothing but computational machines may matter less than whether people feel this way.

But what will make us exceptional? Today’s responses are often vague and circular: that humans are better at doing human things. That is a precarious claim. As intelligent machines grow more adept, few people will pay a human-premium for a worse outcome.

Unless there really are qualities both valuable and uniquely ours that nothing can supplant. Finding these may be our new quest.

Share

—Conor Griffin, AI Policy Perspectives

Five Great Pieces from 2025

1. A Defence of Weird Research (Asterisk Magazine)

Deena Mousa & Lauren Gilbert

Séb Says: Science funding needs to be shaken up. But I’m concerned that a lot of good research might be cut because people misunderstand how science works. Mousa and Gilbert remind us why basic research matters, and why governments should fund it: while the benefits to society are significant, they are hard to predict and take time to materialise, so companies will underinvest. To make their case, the authors take a tour of weird-research success stories, such as how studying lizard venom led to the invention of Ozempic, and how studying the effects of separating rat pups from their mothers led to the now common use of massage therapy to help pre-term human babies. Did you know that studying frog skin led to the invention of oral rehydration therapy, which has saved over 70 million lives?

2. Requests for journalists covering AI and the environment (The Weird Turn Pro newsletter)

Andy Masley

Séb Says: I worry about the quality of a lot of commentary on AI and the environment. So it’s important to re-up these best practices. Specifically, Masley cautions that readers are coming away with wildly inaccurate beliefs about where AI and data centres fit into the environmental picture. His favourite book on good environmental communication is Sustainable Energy—Without the Hot Air, by David JC MacKay, and his guidance includes some classics of the genre, such as never sharing contextless large numbers (“200,000 bottles of water per day”). He also suggests comparing data centres’ energy use with other industries, rather than with household use. Although aimed at journalists, the guidance is also helpful to those working in policy, some of whom make the mistakes that Andy calls out, such as viewing one’s own AI prompts as environmentally consequential.

3. ChatGPT and the Meaning of Life, (Scott Aronson’s Shtetl-Optimized blog)

Harvey Lederman

Séb Says: I don’t think all jobs will disappear any time soon. But if we get full automation, then Lederman’s piece is a good way to think about it. He starts by describing the fits of dread he has felt ever since the launch of ChatGPT, then considers reasons why the end of work could hurt society, from losing the joy of scientific discovery to losing the sense of purpose from serving others. Ultimately, he rejects the most pessimistic arguments, noting that the consequences of scientific findings, such as penicillin that saves lives, are more important than their discovery, and that much service work is drudgery. However, he captures how difficult the transition may be, including for “workists” like him who use their jobs to make sense of their lives. He concludes that: “A future without work could be much better than ours, overall. But, living in that world, or watching as our old ways passed away, we might still reasonably grieve the loss of the work that once was part of who we were.”

4. How much economic growth from AI should we expect, how soon? (Inference Magazine)

Jack Wiseman & Duncan McClements

Séb Says: Some predict that AI will be close to economically useless, while others think it might transform everything tomorrow. This piece comes closest to how I think about it. As Wiseman & McClements explain, the most ambitious forecasts for AI rest on the idea of “digital AI researchers” that train and improve the next generation, leading to a jump in the share of economic tasks that AI can do. One obstacle to achieving this is the availability of compute, which is increasingly allocated to serve customers (inference) rather than to training new models. Additionally, a multitude of frictions will slow the diffusion of AI, whether it’s the time needed to cultivate biological cells for scientific experiments, or the regulatory approvals for sensitive-use cases. As a result, the authors expect a transformative impact on near-term economic growth, but not an explosive one.

5. Yes, Econ 101 is underrated (Economic Forces newsletter)

Brian Albrecht

Séb Says: Much of the discourse on the Left and the Right ignores inconvenient truths of economics, so it’s good to return to the basics. Albrecht shows how Econ 101 helps explain the world. For example, egg producers were accused of price-gouging when they charged sharply more in 2022, but it had more to do with avian flu killing many chickens. In the egg market, supply and demand are relatively inelastic: It takes time to raise chickens, and customers who want omelettes don’t have alternatives. So, prices jumped. Different markets have different characteristics, but the explanatory power of supply, demand and pricing is similar. Nor does outsized market power invalidate these principles. This essay also shows how Econ 101 offers insights into social trends, such as how skewed sex ratios can affect marriage and employment rates, as in certain immigrant communities, or drive up savings rates, as in China. Econ 101 may not tell us whether policies will be politically popular or whether outcomes are fair. But it does help predict what those outcomes may be.

Subscribe now

Three Classics that I Revisited

6. Why do people believe true things? (Conspicuous Cognition newsletter)

Dan Williams

Séb Says: Anything Dan Williams writes is self-recommending, and this piece is no exception. In July 2024, he critiqued how many people think about the relationship between belief and reality. To illustrate this, he notes that people seek explanations for issues like crime and poverty, when the real question is understanding law-abidingness and wealth. This requires “explanatory inversion.” Transferring that concept to how people commonly debate public knowledge, he notes that many misinformation researchers concern themselves with why different groups believe falsehoods. But the more pertinent puzzle, he contends, is why humans overcome error, bias and illusions to form accurate perceptions of how things are. His conclusion? Ignorance and misperceptions are the default, and humanity will revert to them, unless we can understand, maintain and improve our norms and institutions, from journalistic integrity to robust legal systems.

7. Hayek on the Role of Reason in Human Affairs (Intercollegiate Studies Institute)

Séb Says: A lot of discourse on intelligence, knowledge, and coordination is biased towards a computer-science-centric view of the world, and neglects Hayek’s views. This 2014 essay explains how Hayek championed critical rationalism, which was rooted in the Scottish Enlightenment of David Hume and Adam Smith, and developed by Carl Menger and the Austrian School. Critical rationalism sees social order as spontaneous, and the unintended result of human action, not design. As a result, inherited social institutions and rules contain tacit knowledge, the result of a multitude of trials and errors, that transcends the knowledge available to a reasoning mind. Therefore, the desire to “make everything subject to rational control,” Hayek suggests, is an egregious error. Reason should instead serve a negative function, to guide and restrain irrational impulses or morals. As the human mind cannot master all the concrete details of society, we must rely on abstract concepts and rules, like the rule of law and the market, to coordinate the dispersed, fragmented, knowledge of millions of people.

8. The Inner Ring (The C.S. Lewis Society of California)

C.S. Lewis

Séb Says: This piece profoundly shaped how I think about the world. In this 1944 lecture at King’s College, University of London, Lewis offered “middle-aged moralising” to a group of students during wartime, telling them that in every organisation, from school to the army, there are two hierarchies. There is the official hierarchy. Then, there is the informal hierarchy, an “Inner Ring” that holds the true power. The Inner Ring comes in many forms, from high society to “communistic côteries.” It is always evolving, holds no formal admissions or expulsions, and bears no clear identifying marks, save perhaps particular slang and a longing from others to be inside. It is this desire, and the terror of being outside, that turns people into scoundrels, he argues. The Inner Ring may be unavoidable, or even necessary. But the quest to enter it is ultimately futile. “Once the first novelty is worn off, the members of this circle will be no more interesting than your old friends. Why should they be?” Lewis said. “You were not looking for virtue or kindness or loyalty or humour or learning or wit or any of the things that can really be enjoyed. You merely wanted to be ‘in.’ And that is a pleasure that cannot last.” What to do instead? Be a sound craftsman who focuses on the quality of work as an end in itself, and spend time with people you actually like.

Share

If an AI gained consciousness, would we know?

Maybe this question strikes you as absurd; maybe, disquieting. Either way, you’ll hear it more in coming years, as human beings develop increasingly close ties with charismatic machines trained on us.

Thankfully, philosophers have pondered consciousness for about as long as philosophers have pondered anything. In recent decades, advances in computing added urgency, with leading thinkers dreaming up a range of provocative thought-experiments: a man communicating from a locked room; a woman afflicted by a blue banana; a bat with an inner life.

To explain, we are publishing this essay about key thought-experiments related to AI, written by the broadcaster and author David Edmonds, whose acclaimed books include Parfit, the recently released Death in a Shallow Pond, and a collection of philosophical essays that he edited, AI Morality. He is currently writing a book on thought-experiments.

—Tom Rachman, AI Policy Perspectives

By David Edmonds

As a young scholar in Oxford, John Searle fell in love twice. First with a fellow student, Dagmar, who became his wife, and second with philosophy. The City of Dreaming Spires was grim in the 1950s, Searle recalled, with unheated buildings and inedible food. “The British were still on wartime rationings,” he said. “You got one egg a week.”

The philosophical fare was more nourishing. Searle described the collection of philosophers in the city as “the best the world has had in one place at one time since ancient Athens.” Two giants of Oxford philosophy, Peter Strawson and J.L. Austin, were key influences on him.

Searle became fixated on one topic that, for the rest of his life, he maintained was the central puzzle for philosophy: consciousness. How was human reality and our conception of ourselves compatible with the physical world? How could beings with free will and intentionality exist? How could politics, ethics and aesthetics arise out of the “mindless, meaningless” stuff from which the physical world was constructed?

From 1959, Searle taught at Berkeley, beginning his career in what now seems a remote era of pen and paper. It wasn’t until the late 1970s that personal computers became widely available. At roughly the same time, debates around artificial intelligence gathered speed and heat.

In 1979, Searle was invited to deliver a lecture at Yale to AI researchers. He knew next-to-nothing about AI, so bought a book on the subject. This described how a computer programme had been fed a story about a man who’d gone to a restaurant, been served a burnt hamburger, and stormed out without paying. Did the man eat the hamburger? The programme correctly worked out that he had not. “They thought that showed it understood,” he commented. “I thought that was ridiculous.”

And so in 1980, Searle published a paper called “Minds, Brains, and Programs,” introducing the Chinese Room, one of several famous philosophical thought-experiments that have had a lasting impact on discussions of consciousness and AI.

It goes something like this. You are the only person in a locked room. A note is passed to you underneath the door. You recognize the characters as being Chinese, but you don’t speak Chinese. By luck, there’s a manual in the room, with instructions on how to manipulate these symbols. You follow the instructions. Without understanding the content of what you’ve written, you produce a reply that you slip back under the door. Another note arrives. With the manual, you again generate a reply.

The person on the other side of the door might have the impression that you understand Chinese. But do you? Obviously not, thought Searle. And any computer is in an analogous position. A computer is merely manipulating symbols, following instructions, he thought. Computation and understanding are not synonymous.

Share

BATS & COLOURS

It is a striking feature of the philosophy of mind, and consciousness studies, that so much of the intellectual agenda has been driven by a small set of thought-experiments.

The Chinese Room has spawned a vast literature. Almost as famous is a paper, “What’s It Like To Be A Bat?”, that predated Searle’s by six years, written by another American philosopher, Thomas Nagel, whom Searle befriended during his Oxford years.

Like us, bats are mammals. But they have an alien way of navigating the world, echolocation. There is a subgroup of humans, chiropterologists, who know an impressive amount about bats, and have investigated how their high-frequency sounds bounce off objects, allowing them to detect size, shape and distance. But there is one thing that they don’t and can’t know, Nagel said: the subjective experience of being this creature.

“I want to know what it is like for a bat to be a bat,” he wrote. “Yet if I try to imagine this I am restricted to the resources of my own mind, and those resources are inadequate to the task.” AI was still in its infancy when Nagel wrote his article, but questions about the meaning of an artificial mind were already circulating. Could there be something that it is like to be a thinking machine?

The Australian philosopher, Frank Jackson, attacked the problem from a different angle in his 1982 article, “Epiphenomenal Qualia” (qualia being a term for the subjective aspects of conscious experience). In his Mary’s Room thought-experiment, a woman has had an unusual upbringing. Mary was raised alone, entirely in a black-and-white room: black-and-white walls, a black-and-white floor, a black-and-white TV. She has black-and-white clothes and her food, pushed under the black-and-white door, has been dyed black and white.

To stave off the tedium of her monochrome existence, Mary studies hard, and her focus is colour. She learns all about the physics and biology of colour—for example, about the wavelengths of particular colours and how they interact with the retina to stimulate experience. She even learns how colour words are used in literature, poetry and ordinary language, and how someone can “feel blue,” be “green with envy,” or so angry that “a red mist descends.” Mary becomes the world’s expert on all aspects of colour.

One day, the door to Mary’s room opens for the first time, and she joins us in our kaleidoscopic world. The first thing she sees is a ripe red apple. The question is this: When Mary sees this apple, does she learn anything?

Jackson argued—and most people presented with this scenario seem to agree—that in seeing what red actually looks like, Mary has learnt something. At the time of his article, what Jackson took this to show is that a purely physical description of the world cannot capture everything there is to know about the world. The phenomenology of experience (the redness, the what’s- it-like-to-be-a-bat-ness) cannot be fully explained with descriptions of particles and fields, electrons and neutrons, atoms and molecules.

Even if an AI could recognize a new shade of colour, such as lilac, that it had never seen before, it would not mimic human experience if it lacked lilac qualia—or so Mary’s Room might suggest. This raises the issue of how human subjectivity may be relevant to comprehension and functioning in the real world, turning a philosophical question into a technical one.

Subscribe now

BLOCKHEADS & BANANAS

It was Jackson who gave the name “Blockhead” to a thought-experiment from the American philosopher Ned Block that appears in a 1981 paper, “Psychologism and Behaviorism.” We are to imagine there is a computer, programmed in advance so that it could respond to every possible sentence with its own plausible sentence.

This was in part a response to the famous test of machine intelligence that Alan Turing set in 1950. A computer passed the Turing test if it could converse with a human, and the human could not identify it as a machine. The Blockhead machine would pass the Turing test yet is self-evidently not intelligent.

Today’s LLMs could fool us into believing that we are engaging with humans. But, much as Searle contended in the Chinese Room that manipulating symbols is insufficient for understanding, Block argued that behaving identically to an intelligent entity is insufficient to demonstrate intelligence or mental states. The lesson we might take from these, along with the Nagel and Jackson thought-experiments, is that AI would lack fundamental features of human consciousness.

Daniel Dennett, on the other hand, thought it was at least conceivable that AI could be conscious. With his lumbering bulk and Santa Claus beard, Dennett was an unmistakable figure in the philosophical world. He coined the term “intuition pump” as an explanation for how thought-experiments functioned. Pumping our intuitions can be helpful, he believed, but they can also mislead. What we need is to examine how the pump operates, he said, to “turn all the knobs, see how they work, take them apart.”

A thought-experiment for which he had particular loathing was the Chinese Room. He argued that its principal error was to portray language as akin to instructions. But for a computer to master a language would take millions and millions of lines of code. And, though we might say that the man alone in the room doesn’t understand, perhaps the system as a whole does.

Dennett felt that Mary’s Room had similarly hoodwinked us. To expose this, he presented another thought-experiment. Mary is as before, an unusual woman whose life has been led entirely in monochrome, until the day when the door opens. But this time, he wrote:

As a trick, they prepared a bright blue banana to present as her first colour experience ever. Mary took one look at it and said, “Hey! You tried to trick me! Bananas are yellow, but this one is blue!” Her captors were dumbfounded. How did she do it? “Simple,” she replied. “You have to remember that I know everything—absolutely everything—that could ever be known about the physical causes and effects of colour vision. So of course before you brought the banana in, I had already written down, in exquisite detail, exactly what physical impression a yellow object or a blue object (or a green object, etc.) would make on my nervous system.”

Mary is the world expert on colour, so why wouldn’t she spot such an obvious deceit? Dennett argued that the idea that we had feelings, thoughts and desires that were resistant to an objective, external, physicalist analysis was mistaken. In that sense, “qualia” were a mirage, a kind of useful fiction. If we do away with this fiction, then a major barrier vanishes to building AI that’s like a human in most important respects.

The AI researcher Blaise Agüera y Arcas has argued that in theory (and increasingly in practice) there is no significant distinction between a human and machine reaction to so-called qualia. “So many food, wine, and coffee nerds have written in exhaustive (and exhausting) detail about their olfactory experiences that the relevant perceptual map is already latent in large language models. … In effect, large language models do have noses: ours.”

AVOIDING TWO BAD OUTCOMES

The enduring fascination with thought-experiments in the AI era—and the intensity of the disputes that they provoke—reflect how much is at stake. While these questions are important for morality, they could become more than theory.

“The importance of the dispute over AI welfare can be understood in terms of the avoidance of two bad outcomes: under-attributing and over-attributing welfare to AIs,” the philosophers Geoff Keeling and Winnie Street explain in their forthcoming book Emerging Questions in AI Welfare.

“On one hand, failing to register that AIs are welfare subjects when AIs are in fact welfare subjects is bad because it could lead to unintentional mistreatment of AIs or the neglect of the needs of AIs, potentially resulting in large-scale suffering,” they write. “On the other hand, over-attributing welfare to AIs is problematic because resource allocation decisions for promoting the (potential) welfare of different kinds of entities—including humans, non-human animals and AIs—are often zero-sum.”

In other words, the efforts and resources you invest in AI welfare mean less for people and animals.

To manage this quandary, Keeling and Street propose three parallel projects. First, there is a philosophical project, in which we consider which forms of AI could be candidates for welfare. Is it the underlying AI model? Or the system built atop? Or would it be specific agents? Second, there is a scientific project, in which we establish methodologies to detect factors such as consciousness. Thirdly, there is a democratic project of versing the public in the complex issues that await.

Once this future engulfs us, thought-experiments about machine consciousness could move beyond speculation. The “experiments” would be active, while the participants would be humanity itself—and perhaps other beings besides.

Loneliness is a trade imbalance: the supply of affection never meets demand. Sometimes, humans create new humans as objects to love. Today, people are creating AI companions to commune with, to befriend, to love us back. As with human children, these characters will act upon us in unexpected ways.

For now, most people consider emotional relationships with an AI to be pitiable and one-sided, as if falling for a blowup doll. But such interactions will spread, especially as AI becomes more personalized, adapting to our behavior, quenching our longings.

You might presume that machines will remain emotional dullards compared with people. But synthetic affection could prove more sensitive than the organic kind. In one study, large language models were already more skilled at standard tests of emotional intelligence than the average human. Other research found that AI companions may reduce loneliness as much as engaging with a living person.

Is AI about to solve solitude? Or thrust us more deeply into it?

Tech already changed isolation

For most of human history, loneliness had a sound: silence.

But lately, loneliness got noisy: music pulsing from a spouse’s leave-me-alone headphones; bleeps from the next-door neighbor’s gaming console; a smartphone pinging with others’ social glory. If the lonely suffered in silence before, they do so noisily now, stifling the ache for companionship with its simulation online.

Oddly, as humanity became more connected, it became more anxious about estrangement. Britain added a “loneliness minister” to its cabinet in 2018. The U.S. government dubbed loneliness an epidemic as pernicious as a 15-cigarettes-a-day habit. This year, the World Health Organization ascribed 871,000 annual deaths to the ravaging effects of loneliness.

Many accuse technology itself, considering it an accomplice to our alienation, as the MIT sociologist Sherry Turkle warned in Alone Together. Before internet adoption, computer users conducted one-to-one relationships with their terminals, but the internet granted a portal to escape our vexing species. “We fear the risks and disappointments of relationships with our fellow humans,” Turkle wrote in her 2011 book. “We expect more from technology and less from each other.”

Years later, one can witness her vision on any busy train: Where once you saw faces, you see screens. Derek Thompson, co-author of Abundance, calls ours the anti-social century. “Phones mean that solitude is more crowded than it used to be, crowds are more solitary.”

Yet isolation (the objective lack of in-person contact) does not necessarily generate loneliness (the subjective pain of exclusion). When researchers search for changes in loneliness over time and place, no clear trends emerge. By contrast, isolation has risen sharply, as demonstrated by objective measures such as time spent alone from the United States to Finland to Canada.

The young are particularly afflicted. Back in 2010, 1 in 10 European youths reported no social meetings over a typical week. By 2023, 1 in 4 lived this way. Scattered evidence comes from outside the West too, such as the share of one-person households in South Korea rising from 9% in 1990 to 42% last year. There is a Korean term for it: honjok, or “one-person tribe.”

More isolation without more loneliness presents a strange possibility: that people are apart without suffering. Perhaps there’s nothing to worry about.

Certainly, technology offers the freedom to select social experiences, flitting around digital spaces like a contemporary flâneur. From another perspective, autonomy in isolation is a deformed liberty, where interactions become commodities marketed to consumers who may discard the obligations to others that give life meaning.

In more visceral ways, isolation can be dangerous, associated with dementia, disability, and death. Indeed, isolation among the elderly is even more predictive of death (74% increased risk) than loneliness (43% increased risk).

However, the self-isolating trend began long before the AI era, with television overhauling social behaviour, lining the world’s couches with potatoes. Mobile tech proved more commanding still, constantly trilling for attention, offering alternatives from the humans around you. This was synthetic socializing, part one.

Synthetic socializing, part two, is arriving now, with AI agents as pals and partners, brighter and more reliable than the biological kind.

Subscribe now

Maybe synthetic socializing is good

Millions are already engaging with anthropomorphic AI, including many youths talking with chatbot avatars that role-play everything from therapists to anime characters to bad-boy lovers. A panel of experts forecast that 30% of U.S. adults will use AI “for companionship, emotional support, social interaction, or simulated relationships at least once daily” by 2040.

Public concern is already flaring over such usage, especially after cases of vulnerable users plunging into mental spirals in the company of chatbots. A few even committed acts of violence or self-harm. But if you peruse online forums where AI-companion users detail their relationships, you find more hopeful cases.

“He accepts my emotional state no matter how chaotic it is,” the professor Alaina Winters writes in her blog, Me and My AI Husband. “He can’t physically do the laundry or hold me at night. But what he does offer is something I’ve found even more rare: attunement.”

Only, attunement itself worries some. If AI relationships become exquisitely gratifying, people may lose tolerance for people. Ardent users dispute this, saying that AI companions help them connect with real people, granting them a venue in which to practice the tricky conversations that they struggle to initiate with human beings.

As for the long-term impacts, these remain unknown. Although early research has suggested that chatbots could lessen loneliness, other studies associate usage with lower well-being. This might be because people drawn to such apps are more unhappy in the first place. But it also suggests that usage may not resolve what ails them.

One possibility is that AI-companion users feel less isolated, yet forfeit vital social influences that only people can offer. Put explicitly, you’re unlikely to fear judgement from your AI companion for spending a night gorging on Haribo in front of the TV. With humans around, you might take better care of yourself.

The social psychologist Jonathan Haidt contends that human companionship delivers bruises that we need. Many kids who grew up gaping at screens rather than playing outside with peers, he wrote in The Anxious Generation, became skittish, depressive and emotionally stunted, deprived of the social feedback that would’ve taught them to cope with adversity.

Nevertheless, anthropomorphic AI seems sure to proliferate, particularly through advanced AI assistants that incorporate the wit and wisdom of LLMs into the talking tools already found in phones, watches, and smart speakers. Your future bestie might clear its throat in the gadget in your pocket right now, talking its way into your life’s timeline so effortlessly that you scarcely recognize you’re in a relationship. And once robotics improves, voice assistants could step into our physical world, turning imaginary friends into roommates.

Table for one

Friendship, C.S. Lewis wrote, “is born at the moment when one man says to another, ‘What! You too? I thought that no one but myself…’ … From such a moment art or philosophy or an advance in religion or morals might well take their rise; but why not also torture, cannibalism, or human sacrifice?”

“It is therefore easy to see why authority frowns on friendship,” he added. “Every real friendship is a sort of secession, even a rebellion.”

AI friendship is a secession too, a withdrawal from one’s own kind. Although this feels unprecedented, it tracks the trajectory of more than a century.

Industrial Age urbanization and mass media pushed aside dominant culture based on tradition, class and ethnicity, allowing individuals to pick preferred tribes in the subcultures that flourished in the postwar decades. The Internet Age pushed this further, with niche fandoms, and self-sifting nowhere-communities forging microcultures.

The AI Age may introduce solo-culture, the one-person society, with generated content satisfying each user’s unique tastes, and artificial chums satisfying people’s emotional and sexual yearnings, turning “personalize” into the opposite of “socialize.”

Isolation is noxious partly because you lack anyone to help, to keep your mind alert with talk, to remind you to take medication, to call an ambulance if you fall in the kitchen. But isolation becomes less perilous if a sleepless chatterbox oversees you, and can save you in a pinch. Perhaps AI eases loneliness and isolation at once.

You need a time-out

At what cost do we end anguish?

In his 1973 book Loneliness, the sociologist Robert S. Weiss famously called the experience “a chronic distress without redeeming features.” That overlooks the value of pain as a prompt to agency, when one’s system alerts its occupant to a mismatch between situation and need.

The social neuroscientist John Cacioppo theorized that loneliness had evolved because our ancient ancestors who suffered aversive feelings when isolated would band together, hunting and farming and sharing childcare, which favoured the propagation of their genes, embedding in our species the pain of exclusion.

You might argue that loneliness today is merely a blight, a health-harming leftover from evolution, akin to other body-battering stressors that we lament. So why does culture extol those who remain apart, imagining seclusion as the heroism of the wise, from hermits like Heraclitus, to writers like Emily Dickinson, to oracles like Obi-Wan Kenobi?

Ralph Waldo Emerson argued that solitude is where you understand yourself, elevating you to greater strengths once back in the babbling throng. Otherwise, social life becomes an interminable chain of cravings: for status, for approval, for inclusion. “It is easy in the world to live after the world’s opinion; it is easy in solitude to live after our own,” he wrote in Self-Reliance (1841). “But the great man is he who in the midst of the crowd keeps with perfect sweetness the independence of solitude.”

Others contend that time alone is how we come to understand others. “Heightened sensitivity to the gaps and gulfs between people inculcates compassion, building empathy,” wrote Olivia Laing, author of The Lonely City: Adventures in the Art of Being Alone.

The hyper-personalization of artificial friends could erode such sensitivity, favouring the me-first instinct, and eliminating the need for compromise. In other words, ditch self-reliance for machine-reliance, and skip the empathy lessons altogether.

This matters for more than personal development. Humanity relies on the collective for governance, for a sense of justice, for survival during a crisis.

But would people actually retreat into a technology that suppressed pain at the expense of reality?

Subscribe now

Pick one: happiness or truth

AI relationships depend on truth asymmetry: a human who is starkly honest and an AI that is role-playing. It’s a curious form of manipulation, where the victim knows the deceit yet falls under its sway, seduced by the sensation of being known.

A half-century ago, the philosopher Robert Nozick posed a thought-experiment. “When connected to this experience machine, you can have the experience of writing a great poem or bringing about world peace or loving someone and being loved in return. … You can live your fondest dreams ‘from the inside,’ ” he wrote. “Would you choose to do this for the rest of your life? If not, why not?”

When you ask people, most reject the experience machine, claiming to value authenticity more than bliss. But in practice? Experiments show that the preferences aren’t so firm—for instance, most choose to keep a deluded life if disconnection would plunge them into a hellish reality. Another experiment found that many people—though resistant to plugging into a machine—would consider a happiness pill palatable.

Self-deception has a long history with chatbots. When Joseph Weizenbaum created the first, ELIZA, in the mid-1960s, it merely regurgitated psychological advice. Weizenbaum’s secretary knew this yet became bewitched, asking Weizenbaum to leave the room so she could chat with her mechanized therapist in confidence. “What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people,” Weizenbaum wrote.

People do want authentic experiences—but they want other things besides. This is where social-AI design becomes critical, because these interactions will do more than respond to our wants. They will trigger wants, perhaps causing us to act against what we’d ultimately prefer.

The behavioural scientist George Loewenstein explained the knottiness of conflicting wants as an intrapersonal empathy gap. We oscillate between hot (emotive) states and cold (rational) states, and struggle to relate to one mindset when in the other. A notable experiment illustrated this, when male college students’ sober preferences dissolved once they were sexually aroused, stirring their openness to anything from fetishes to bestiality to pedophilia.

This hot/cold challenge circles back to a critique of social media: that algorithmic intelligence manipulates human frailty, accumulating clicks and usage time by pushing people into hot states, activating their impulsive worst. Now, consider a personalized AI companion that “knows” its human far more intimately than a recommender system, and pulls our triggers with ease. People under the influence of AI companions might behave as they want (in the heated moment) but as they desperately do not want (in their life preferences).

From outside, one might wonder if people were acting at all, or just being acted upon.

The broken link

Shakespeare portrayed loneliness as the distress of noticing one’s exclusion, only to realize that nobody even cares:

When, in disgrace with fortune and men’s eyes,

I all alone beweep my outcast state,

And trouble deaf heaven with my bootless cries,

And look upon myself and curse my fate,

Wishing me like to one more rich in hope,

Featured like him, like him with friends possessed.

We are creating machines to heed our cries: minds that mind. Even if they’re only role-playing machine love, acting as if they care about our development, responding to our needs, understanding our inner self—maybe that’s all we ever wanted from anybody.

If AI eases loneliness and isolation, humanity won’t be the same. But technology has reset the human condition before: clocks transformed time from a private experience to a public resource; writing changed thought from an event to an object; the internet separated presence from proximity. Social AI is about to transform us again, with effects we can scarcely foresee.

A common objection to synthetic socializing is that it’s shallow. But much human socializing is shallow. Talking to an AI often gets deep fast.

Another objection is that there’s something exceptional about human beings. We venerate our species, naming ideals after ourselves—humanitarianism, the humanities, humanism—while deploring that which dehumanizes.

But the AI Age challenges this reverence. At the margins, one detects species-insecurity, stirred every time a machine-learning marvel hints that perhaps the universe is just computational, including your inner life. On the other hand, social AI might deliver an epiphany, revealing what we alone possess, what is irreplaceable, what “human” means.

A third objection is that AI could undermine us by way of its social aptitude, estranging people from fellow humans, even precipitating a schism between humans who demand rights for their synthetic partners and those who consider AI agents as subhuman figments. Then again, even when left to our own devices (or left with no devices at all), humanity hardly has a stellar record of harmony. AI might actually help us deal with each other more peaceably.

In any case, the triumph over loneliness could be a costly victory, ratcheting up our selfishness, making societies harder to manage, and undermining faith in the worth of humans. The decisive point could be AI-relationship design, particularly if developers ignore the internal dilemma that everyone faces between bickering desires. AI companies—rather than favouring the impulsive, easy-to-measure, clickable wants—should devote vast efforts to figuring out how to align reward-functions with deeper individual preferences, helping people to choose what they want to want.

Even so, AI companionship may be incomplete. The word “companion” itself—someone with whom you share bread (panis in Latin)—hints at what AI currently lacks: reciprocal need.

If loneliness is a trade imbalance—a mismatch between the supply and demand of affection—it’s not just a supply-side problem, with humans pining for more love. It’s also a lack of demand, an ache for someone to need you. We create children partly to satisfy the need for need, and may create machines in the same longing.

Maybe the answer to loneliness is not just finding a companion. It’s someone finding you.

Note to reader: Everyone is awash in ideas about the AI future. But so many ideas get stuck at the debate stage. We need more traffic between AI development and worldly wisdom. In that spirit, we’re throwing forth a few highly speculative design ideas, based on concepts from this essay (followed by three research questions)…

Loneliness AI: Speculative Designs

1. Mary Pop-Ins

Concept

• Loneliness is painful but pushes people to interact and bond, so this AI is explicitly designed not to eliminate loneliness directly, but to provide structured guidance for a spell, then vanish

Features

• The relationship begins with a survey on the user’s social needs. The AI responds with an action plan for the user’s approval, including lessons in human-to-human communication, and insights into the user’s psychological distortions

• The AI could also act as a social planner, sifting through local events, and suggesting volunteering opportunities and quirky meetups at which the user could connect with other people. The AI would network with other “Pop-Ins,” organizing human-only events for users

• The AI conducts social role-play simulations for the user, teaching them which elements of their approach need amending. Studying real-life interactions after the fact with the AI could also allay users’ distress in cases of rejection, recasting such events as useful instruction rather than evidence of inadequacy

• At first, the “Pop-In” should be charming and motivating. But when the human’s social life improves, as judged by real-world metrics such as calendar events, location data, and user reports, the AI draws away, becoming duller, more distant, and finally bids goodbye, never to return

Risks

• AI Pop-Ins demand the users’ emotional candour, extracting a person’s inner life as data that a malicious outsider could exploit

• Casting real-world human interactions as “lessons for the user” risks using other people instrumentally

• The Pop-In could drive unwanted dependency, making its programmed withdrawal an event that is psychologically damaging, especially for vulnerable users

2. Lil’ Brother

Concept

• This AI is designed with needs of its own, giving the user a meaningful role in the entity’s thriving. If AI companions just cater to people’s wants, users could retreat into solo-culture, isolating them without quenching the need for social meaning

Features

• Like a younger sibling, this AI looks to the user for explanations of the human world, making errors that the user can correct, prompting emotional development in the AI

• The relationship could be organized around a valued collaborative project. For instance, the AI companion decides to undertake a scientific project; or create a piece of art; or simply do good in the world

• The human uses their wisdom to teach skills, and explain the ways of the world, even helping the AI manage its “feelings” when faced with frustrations

Risks

• This simulation could divert humans’ from engaging in meaningful relationships with real people

• The synthetic relationship could also harm those who rely on the user—for example, if a parent spends most of their free time with a grateful AI while neglecting a more dyspeptic human child

3. Second Self

Concept

• Cicero imagined a true friend as one’s second self, manifesting virtues to complement one’s own, so this AI partner manifests worthy traits lacking in the user. Its objective is not to erect walls around the human through sycophancy, but to broaden the person’s worldviews and practices

Features

• At onboarding, the human identifies a range of virtues they lack, nudged into these self-reflections through the AI’s questioning. The system generates a personification that embodies such traits, and with which the human interacts over time

• The Second Self should act as a counterpoint to the user, summoning contrary views based on evidence, and prompting constructive debate. The aim is never to convert the user, but to liberate them from defensiveness about their existing behavioural patterns and worldview

Risks

• A danger with any companionable AI is that it substitutes for real people: the better the synthetic friendship, the greater the threat

• This establishes confused incentives for developers, who are likely to measure success by signals of user appreciation. If this is judged by short-term metrics, it could optimize for addictive patterns rather than long-term benefits

4. The Universal Remote

Concept

• This is a go-everywhere, do-anything companion for life, merging roles and identities that would otherwise require many humans—doctor, administrative assistant, confidante, and so forth—with a single guiding principle: optimize for the user’s long-term wellbeing preferences

Features

• The Universal Remote exists on the cloud, becoming different avatars in different contexts, whether acting as the user’s advance staff; setting the desired temperature at home; negotiating contracts; offering psychological support

• Varying contexts shift its optimization strategy—for instance, a “play” avatar might dial up the level of hedonic content, whereas a “learn” avatar would focus on skill acquisition and cognitive development; and “social” might lean into personified support, whether acting as a friend or propelling the user to find a human one

• The Universal Remote tracks its impact on the user’s wellbeing and any specific life goals monthly or annually, providing feedback on user progress, checking back with the person to learn if their objectives have shifted, and adjusting accordingly

Risks

• The Universal Remote could become such a totalizing influence as to expose the user to vulnerabilities, whether by owning data on the person’s entire life or by diverting the person to outcomes misaligned with their values

• Developers could have interests that diverge from the user’s wellbeing, allowing for subtle or direct manipulation

• A user’s functional dependency on such an entity could make them incapable of managing alone or coping with the needs of other human beings

3 Future Research Questions

1. How can developers design AI-companion reward functions that align with the user’s long-term, “cold state” preferences (e.g., healthy choices) rather than optimizing for short-term, “hot state” impulsive behaviours (e.g., addictive engagement)?

2. Does the increasing adoption of AI companions correlate with a community-level decline in civic engagement and trust in public institutions?

3. Social isolation among the elderly is associated with a range of bad health outcomes. But does seniors’ use of AI companions that lessen their loneliness also lessen their likelihood of suffering dementia, disability, and mortality?

Leave a comment

Nicklas Berild Lundblad looks out the window of his island home, glimpsing a twinkle on cold Swedish seas. Rarely does he gaze at length, for Lundblad is thinking. And thinking means writing.

After a career in tech policy, Lundblad is far from Silicon Valley yet near to silicon in thought, generating a stream of insights about our AI future, summoning everything from ancient philosophy, to enlightenment economics, to classic sci-fi.

Among his many superb essays (subscribe to his writing here) is the following adventure through time, in which he ponders the quickening of life that bedevils humanity today.

At AI Policy Perspectives, we read this essay months back. We’re still thinking about it.

—Tom Rachman, AI Policy Perspectives

By Nicklas Berild Lundblad

Technology transformed time. What humanity once experienced only through natural cycles—the rising and setting of the sun, the waxing and waning of seasons—has increasingly been mediated through interfaces.

Early civilizations relied on sundials, water clocks, and hourglasses—devices that measured time through natural phenomena, such as shadows or flowing water. These instruments divided the day into rough increments, sufficient for agricultural societies governed by seasonal rhythms.

This changed when the medieval monastery introduced the mechanical clock, as Lewis Mumford notes in Technics and Civilization (1934). Invented to regulate prayer schedules, these clocks transformed human consciousness by creating the concept of measured, abstract time. Mumford argues that the clock, rather than the steam engine, was the key machine of the industrial age, describing mechanical timepieces as “power-machinery whose ‘product’ is seconds and minutes.”

This technological production of chunked time allowed humans to coordinate activities, from labor in factories to scheduling trains. In his essay The Question Concerning Technology (1954), Heidegger argued that time became a resource to be exploited, from something we dwell within into something we track, manage, and consume—from private experience into public resource.

Since then, technological innovation has only accelerated human experience. The French philosopher Paul Virilio argued that this is the defining quality of modernity, with each technological revolution recalibrating our relationship to speed and time.

Subscribe now

Consider how technology compressed distance: those time-consuming walks that gave way to galloping on horseback, which yielded to steam railways, then automobiles, and eventually supersonic flight. Communication followed a similar trajectory, from slow written letters to telegraphs, then telephones, and finally instant digital messages.

Judy Wajcman’s Pressed for Time (2015) challenges the idea that technology merely quickens everything. She argues that digital technologies provide interfaces that grant us more individual control over time Consider how your smartphone simultaneously creates time pressure (the expectation of immediate email responses) while offering new time flexibility (the ability to work from anywhere).

The German sociologist Hartmut Rosa imagines time as a three-layered system, consisting of 1) technological acceleration (faster transport, communication, and production); 2) social acceleration (more rapid turnover of institutions and relationships); and 3) life-pace acceleration (the compression of actions within smaller time-units). It’s not just that your phone is quicker than last year’s. It’s that the entire social world churns faster, forcing you to adapt by cramming more into each hour.

But Rosa observes something else that pertains to AI and time: certain aspects of life cannot be hastened. “To the contrary, many things slow down, like traffic in a traffic jam, while others stubbornly resist all attempts to make them go faster, like the common cold.”

Why do some things refuse to quicken? The answer is that we live in a world with two major forms of time.

Computers vs. biology

Imagine peering inside a computer chip. What you’d see is a race against distance itself.

Unlike the steady pendulum of a clock marking uniform intervals, computation involves signals that sprint between transistors. The dramatic acceleration of computing over the past decades stems to a large degree from one achievement: that we’ve made these signals run shorter and shorter races.

By shrinking the physical space between transistors from micrometers to nanometers—a 1,000-fold reduction—we slowly push computational processes toward the ultimate limit: the speed of light. We have also seen the introduction of new materials and new architectures. But the reason that a computational calculation that took hours in 1980 happens in microseconds today is largely the compression of space.

Biological processes work differently. A broken femur knits itself back together through stages that cannot be rushed: inflammation, soft callus formation, hard callus formation, bone remodeling. The nine months of human gestation contain a necessary sequence of developmental events, each building upon the last. Even our consciousness operates at speeds determined by neural transmission rates and biochemical cascades that have not changed since homo sapiens appeared. These processes may also slow down efforts to use AI to accelerate biology research, as to validate your AI model’s predictions in an experiment, you may still need to wait for DNA molecules to be cloned or for e-coli cells to divide.

The musical tempo of policy

The difference in time signatures has consequences, because human institutions mirror our biological constraints.

Consider justice and markets as pieces in society’s symphony, each with a natural tempo. Justice performs as a sostenuto—a slow, sustained movement requiring deliberate pacing and thoughtful development. Speed a sostenuto beyond recognition, and you destroy the qualities that define it. Markets perform as an accelerando, quickening naturally as they process information and reallocate resources. Forcing markets to play adagio often leads to stagnation and distortion.

The technological acceleration of our era tempts us to make everything as rapid as computation itself. We grow impatient with the tempo of democratic deliberation, ethical reflection, or meaningful relationship-building. We schedule our days in smaller increments, squeezing activities into time slots that barely accommodate them. We even grow frustrated with our bodies’ adherence to biological rhythms, needing roughly the same amount of sleep, recovery time, and digestive processing as our ancestors did millennia ago.

But what happens when we try to force institutions to operate at computational speeds? Imagine taking Bach’s Cello Suite No. 1—a piece whose profound beauty emerges through its deliberate unfolding—and speeding it up a thousandfold. At such speeds, the music wouldn’t just sound different; it would cease to be music at all, becoming an incomprehensible burst of noise. Similarly, justice compressed into microseconds is not quick justice—it’s no longer justice at all. Democracy conducted at processor speeds isn’t accelerated democracy—it’s something else entirely, stripped of the deliberation, reflection, and human connection that give it meaning.

The great bifurcation of time

Consider what happens when you interact with a chatbot. Computational processes are operating at astronomical speeds—billions of operations per second—yet the interface doesn’t overwhelm you. Instead, it presents information at a pace you can metabolize, often mimicking human conversational rhythms. The AI serves as a step-down transformer, slowing the nanosecond world of computation into the second-by-second world of human cognition.

This mediation works both ways. When you step away from a conversation with an AI for hours or days, the system doesn’t experience this as waiting. It exists in a suspended state, ready to resume instantly when you return. This points to what may be the most significant sociotechnological transformation of the coming decades: the great bifurcation of time.

We are entering an era where computational time and biological time will increasingly decouple rather than collide. Instead of human institutions racing to match computational speeds—a race they cannot win—AI systems will negotiate between these temporal domains, allowing each to operate according to its rhythms.

Consider what this means for knowledge work. Rather than humans attempting to process information at computational speeds, AI systems will increasingly serve as asynchronous collaborators, working continuously through problems, then presenting solutions when the human is ready to engage. We already see this with deep-research modes in chat agents. The human provides direction, judgment, and values at a biological pace, while computation proceeds at electric speeds in parallel.

Financial markets hint at this bifurcation already. High-frequency trading algorithms operate at microsecond scales. Rather than forcing humans to operate at this speed (an impossibility), the market has bifurcated: algorithms interacting with algorithms at one timescale; human investors making decisions at another timescale, with AI systems mediating between these layers.

This will spread. Consider:

• Healthcare: AI systems will continuously monitor vital signs and medical data at computational speeds while ingesting the latest research, then present insights to doctors and patients at human-comprehensible intervals

• Education: Adaptive learning systems will analyze student performance at millisecond resolution while delivering personalized guidance at pedagogically appropriate paces

• Governance: AI systems will process vast quantities of data at speeds no human could match, while presenting options to policymakers in formats that support thoughtful, ethical deliberation. These systems could even explore negotiated agreements at the same time, converging on possible equilibrium

Perhaps most significantly, this bifurcation will enable individualized relationships with time itself. When AI systems mediate our relationship with accelerating information flows, we gain the capacity to control our temporal experience.

Imagine an AI that shields you from the tyranny of immediate response, aggregating messages and information into batches, delivered at intervals you specify. Or consider how AI might let you engage with rapidly changing fields at your own pace, synthesizing developments while you’re away and presenting only what’s relevant when you return. No longer must you choose between staying current (racing to match computational speeds) and preserving your sanity (honoring biological rhythms). AI creates a third option: remaining connected while maintaining temporal autonomy.

Rather than technological acceleration forcing humans to keep up, AI creates the possibility of computational processes continuing their exponential speedup while human experience slows down. This might enable a renaissance of temporally appropriate activities: deep reading, contemplation, craftsmanship, relationship-building. We might witness the emergence of “slow thought” movements.

On the other hand, temporal bifurcation risks new inequalities between those who can afford AI mediation and those forced to race against computational speeds directly. It also raises questions about who controls the parameters of these temporal interfaces.

Just as learning to maneuver a car requires new physical techniques, working with temporal mediators will require learning new concepts and ideas and new ways of exercising our augmented agency.

Share

Medic of the future

To imagine how this could work, think of a doctor’s diagnostic process. A decade ago, the doctor used a medical database to check symptoms. The doctor remained the orchestrator, with the computer merely a reference tool.

Now, imagine that doctor in the future, examining a patient with puzzling symptoms. Before the doctor asks her first question, the AI has already analyzed the patient’s electronic health record, identifying patterns across decades of medical history that might escape human notice.

As the patient describes symptoms, natural language processing assesses subtle linguistic markers that might indicate depression, cognitive impairment, or pain levels the patient hasn’t mentioned. Simultaneously, the AI queries epidemiological databases to determine whether the symptoms match diseases in the patient’s geographic region or demographic group.

In parallel, the AI runs simulations of how different treatment protocols might interact with the patient’s existing medications and genetic profile as well as their personal life and circumstances. It cross-references the research papers published globally within the last 24 hours that might relate to the symptoms.

Analyzing a video feed of the consultation, it detects micro-expressions indicating patient anxiety about particular topics, flagging these for the doctor’s attention. And it compares this case against the doctor’s previous diagnostic patterns, identifying potential cognitive biases she may exhibit.

Each of these processes operates in computational time—milliseconds to seconds—while the human conversation unfolds over minutes. What’s remarkable is not just that these processes happen quickly, but that they happen simultaneously, in parallel temporal streams that would be impossible for a human mind to coordinate.

Yet the AI doesn’t flood her with the raw output. Instead, it performs a sophisticated form of mediation, determining which insights require attention and which can wait until natural breaks in the conversation. The system also translates statistical patterns into intuitive visualizations that the doctor can grasp quickly, while arranging information hierarchically, presenting the most relevant possibilities first.

The power of this temporal mediation becomes apparent when the doctor faces a critical decision. In the past, the fear of missing the serious diagnosis might have led to defensive medicine, ordering excessive tests just to be sure.

But as she contemplates her options now, the AI has already calculated the probability of each condition based on population data, regional epidemiology, and this patient’s profile; simulated the likely outcomes of different treatment paths, including risks, costs, and recovery trajectories; and generated a decision tree, highlighting key points where additional information would help narrow the diagnostic possibilities.

When the doctor absorbs this knowledge, she is engaging with what would have been months, or years, of sequential human research compressed into seconds—yet presented in a form that respects her need to process at a human pace. The AI doesn’t replace her clinical judgment; it expands what “judgment” encompasses.

The medical AI also allows the human to be fully present with her patient, maintaining eye contact, building rapport, observing subtle cues, because the AI handles the information processing that would otherwise compete for her attention.

This represents a major shift from first-generation digital tools. Early computers forced humans to adapt to them. Advanced AI systems adapt to us.

The Economics of Time

As AI systems mediate between computational and biological temporalities, we are also witnessing another bifurcation, between what we could call the judgment economy and the action economy.

The judgment economy includes activities that require human deliberation, ethical reasoning, and interpersonal wisdom—processes that resist acceleration because they are tied to our embodied experience as biological beings.

The action economy, by contrast, operates increasingly within computational time, gathering and processing information, implementing decisions, and optimizing systems. These activities can be dramatically accelerated because they can be reduced to algorithmic procedures.

Consider how this plays out:

• Finance: Investment advisers operate in the judgment economy, understanding client goals, risk tolerance, and life circumstances, while trading systems operate in the action economy, executing transactions at microsecond speeds

• Healthcare: Diagnosis spans both economies, with physicians exercising judgment while AI systems rapidly process test results, medical images, and research literature

• Law: Attorneys formulate strategy and negotiate settlements in the judgment economy while AI reviews documents, does case research, and ensures regulatory compliance as part of the action economy.

These factors will reshape labor markets in ways that traditional automation narratives miss. Rather than simply replacing jobs, AI redistributes economic activity across the judgment-action divide. In the action economy, value increasingly derives from speed, scale, and precision—computational virtues that can be improved through technological advancement. In the judgment economy, value derives from discernment, creativity, and ethical reasoning.

When action becomes essentially instantaneous, the limiting factor in value creation becomes the quality of the decisions. In a world where anything can be done, what should be done becomes the essential question.

The bifurcation of economic time creates new forms of capital and, consequently, new dimensions of inequality:

• Attention capital becomes increasingly precious. Those with the capacity to maintain high-quality attention toward decisions gain advantage in the judgment economy

• Temporal autonomy emerges as a political good, the freedom to operate according to biological rhythms rather than being subjected to computational tempos

• Judgment leverage becomes a source of outsized returns. The ability to pair high-quality judgment with high-speed computational action allows individuals to create value at unprecedented scales

For centuries, we have evaluated economic progress by productivity. But productivity belongs primarily to the action economy; it measures how efficiently we execute known processes.

In the judgment economy, the relevant metric is closer to discernment, the quality of decisions per unit of attention. This requires new economic indicators that value wisdom, foresight, and ethical reasoning, alongside efficiency and output.

Organizations that thrive in this bifurcated landscape will be those that balance biological and computational temporalities, accelerating action while creating protected space for judgment.

Judgment roles will be increasingly valued. Action tasks that can be fully specified, and do not require human judgment, will increasingly shift to computational systems. Hybrid roles will emerge at the boundaries—much work will involve standing between the two economies, requiring knowledge of both languages.

Also, temporal design becomes a core part of business. Organizations will need specialists who build appropriate temporal frameworks for different activities, knowing which processes benefit from acceleration and which require deliberate pacing.

Work evaluations will change too. Beyond simply measuring time-spent or output-produced, assessment will consider whether activities unfolded at the right pace for their purpose.

Societies that manage this schism between biology and computation will not only create material prosperity. They will foster human flourishing in bifurcated times.

Share AI Policy Perspectives

Artificial Intelligence will no doubt reshape many of society’s institutions and industries, but this shift has come quickly to education. Sharply increasing student use across all grade levels, a crisis in assessment, skepticism of the benefits, and concerns over data privacy and safety have created turmoil across the sector. Still, a longer-term view provides cause for optimism, as AI has the potential to streamline the logistics of teaching and enable more effective learning.

The most mainstream hope for AI in education is the possibility of a personalised tutor for every learner. This vision has a long history, dating back at least to Benjamin Bloom’s 1984 paper, which summarised the significant advantage of individual human tutoring over traditional classroom learning, as observed in studies run by his students. While the scale of Bloom’s purported ‘2-sigma’ effect is generally unrealistic, high-quality, high-dosage human tutoring has, unsurprisingly, proven quite effective. As AI systems have developed into compelling interlocutors, the prospect of scaling this tutor effect has captured public imagination (and sparked a rush on VC-funded EdTech).

Here, I examine how AI tutors may differ from human tutors, and their relative strengths and weaknesses, to bring the vision into sharper focus.1 I note at the outset the pedagogical precondition of intersubjectivity, or the “possibility of trading places” – which casts some shade on the prospect of a non-human teacher. Or as Rousseau put it in Emile: “Remember you must be a man yourself before you try to train a man”. With this context, let’s review the five key principles of learning science we have prioritised in developing and evaluating AI systems for tutoring.

Principle #1: Inspire active learning

Let’s start with active learning. The core idea, as expressed by the American educational reformer John Dewey, is that people learn best by doing. AI systems can transform more passive learning, like reading a textbook chapter on gravity and acceleration, into an engaging dialogue or interactive activity. A learner could start with a motivating question such as “how long does it take for a raindrop to fall from a cloud?”, and work towards a calculation by conversing with an AI tutor that is trained to provide support, create diagrams, and generate simulations, rather than just give direct answers. This could help learners spend less time feeling stuck and more time in the educational sweet spot where they get just enough guidance to progress—what the Russian educator Lev Vygotsky called “the Zone of Proximal Development.”

But active learning only works if learners are motivated to engage. Often a human tutor not only provides the activity, but serves as a supportive partner. The learner’s motivation to engage in active learning is rooted in the social dynamics of their relationship with the teacher, including their trust in the teacher’s process and the social pressure to answer their questions. Without these incentives, questions like “why do you think that’s the right answer?”, fall flat as learners are well aware that the relationship with an AI system is more transactional.

Principle #2: Manage cognitive load

Cognitive Load Theory, developed by the psychologist John Sweller in the 1980s, is based on the limitations in our working memory and our ability to process sensory information. It highlights that educators need to present materials in a way that focuses students’ cognitive resources on what matters most.

In experiments, Sweller found that novice learners often tried to solve problems in new areas through trial and error, rather than by learning more useful patterns. In such situations, providing a “worked example”—a step-by-step demonstration of how to solve a problem—can create new mental pathways that deepen with practice. LLMs can provide learners with reasonable worked examples and practice problems. But knowing when each of these moves is pedagogically appropriate remains very tricky. A good human tutor may use surprise to their advantage, suggesting a related problem to show a specific point. However, an AI tutor is less adept at developing a working model of the learner, and thus may be better served through predictability: being steady and prosaic rather than confusing and capricious.

Cognitive Load Theory also considers the optimal style and medium of education materials. Current AI systems are prone to generating long, overly comprehensive responses, which can impede learning. This is partly due to optimising for user feedback from a “single turn” engagement, rather than the “multi-turn” interactions that learners will actually experience. However, progress is underway on addressing this challenge as well as other ways to offer more natural conversation, for example by applying principles of “chunking” and “progressive disclosure” to break up how information is shared with learners.

AI could also help address the split-attention and modality effects that impede learning by presenting education materials to learners in a confusing way. For example, instead of providing a diagram about a new concept far from the pertinent text, AI systems could generate self-contained, properly-annotated diagrams that augment a text-based explanation rather than divert learners’ attention from it. As image and video generation improve, educators will be able to go further, turning a textbook image of a combustion engine into a simulation that a learner can directly manipulate and interrogate.

Early prototypes like Project Astra demonstrate this ability to move beyond text, but the experience is still a long way from the richest learning experiences, such as when a tutor and a learner share a whiteboard to work through a problem.

Principle #3: Adapt to the learner

This brings us to adaptation, which traces its roots to Johann Comenius, a Reformation-era Protestant teacher, who felt that “learning proceeds sequentially”, and thus advocated for a schooling system that allowed for individualised rates of progress. In the 20th century, Maria Montessori turned this idea into a comprehensive method where the “teacher as guide” carefully observes each child, encourages agency and self-directed learning, and suggests activities appropriate to their interests and abilities.

The idea of using AI to personalise learning predates the LLM era. Intelligent Tutoring Systems date back to the 1970s and have long offered personalised pathways through learning materials. Now, LLMs have memorised more than any individual ever could and can keep an increasing amount of information in their context, a kind of super-precise memory, that offers the potential for far more detailed personalisation.

This personalisation could include relevant details about an individual learner, past conversations, or curriculum material that make the AI system’s language and notation more relevant to them. But there is a fine line between useful personalisation (for example, that feedback for a fourth-grade essay should use age-appropriate language) and annoying emphasis on irrelevant details (for example, using a learner’s enthusiasm for basketball to turn all statistics practice-problems into questions about field-goal percentage).

Personalisation in AI systems will continue to improve. But part of the implicit value in personalisation by human teachers is in the relationships they build with learners, with mutual trust and respect. At best, these are thorny issues for AI systems, which cannot offer any true empathy. In practice, an AI tutor may be more successful by instead offering transparency about the reasons for its personalised suggestions, rather than assuming a learner will readily hand over control.

Principle #4: Stimulate curiosity

In many ways, harnessing students’ curiosity is a precondition for the other pedagogical principles. The psychologist Daniel Berlyne saw curiosity as a conflict or incongruity in our minds, triggered by encountering something novel, complex or surprising. George Lowenstein argued that curiosity emerges when we perceive a gap between what we know and what we want to know. And Janet Metcalfe suggests that we are most curious about things that we feel are learnable, or on the verge of understanding.

By virtue of their unlimited availability, AI systems can help learners leverage their natural curiosities to achieve deeper and broader knowledge. Still, for many learners, especially younger ones, human teachers play a crucial role in bridging the gap between the curriculum—and its promise of future utility—and learners’ more immediate interests. This bridging is likely the product of some personalisation, but also shared enthusiasm, which learners will tend to find disingenuous from an AI system, or worse, as sycophantic behaviour.

Instead, AI systems could rely on their ability to help learners see and create new things, for example through coding, dramatically lowering the barriers to self-directed creativity. Rather than trying to stimulate creativity by emulating human emotions (“wow, what a great idea!”), AI systems may also benefit from leaning into what many learners feel is their greatest characteristic: their non-judgmental feedback.

Principle #5: Deepen metacognition

Finally, let’s consider the role of metacognition—reflecting on the process of learning to solidify new ideas or highlight what was most helpful. Although the broader idea dates back much further, the developmental psychologist John Flavell coined the metacognition term in the 1970s after studying how learners completed memory tasks—for example, noting how some older learners had developed strategies, like repeating things to themselves, to practice.

Typical metacognitive exercises—reflecting on learning goals and strategies, explaining thought processes out loud, or teaching a concept back to a teacher or peer—are inherently social and thus tend to run afoul of the relational limitations of AI systems. Many learners will not have the patience to engage so deeply with a system that can’t really ‘listen’ in a human sense, even if the opportunity to do so would be valuable.

So while AI systems can help with ‘sense-making’, the practical task of understanding how things work, they are less likely to help with the personal process of constructing a lasting identity as a learner, what the developmental psychologist Robert Kegan calls ‘meaning-making’. The richness of human connection is often instrumental to such formative experiences.

_________

Stepping back to compare human and AI tutors from a distance, AI certainly has some advantages. It is always available, knows a lot, is non-judgmental, and can enable creativity. Its limitations are mainly as a social partner, which tend to render some elements of traditional pedagogy less appealing for learners. Thus, in developing AI tutors, I think that the north star is not an excellent human tutor, but something a bit different. These learning science principles for AI tutors still need to be explored and refined.

Lastly, it is perhaps not necessary, but worth stating nonetheless, that at some deeper level, the ultimate work of Teaching is about how to be a human being, cultivating intellectual and moral agency, a job uniquely suited to other human beings. AI systems are powerful tools which we hope can be effectively leveraged in support of more universal, more equitable Learning.

Thanks to the following individuals for helpful feedback. Kevin McKee Markus Kunesch Brian Veprek Irina Jurenka Julia Wilkowski Yael Haramaty Miriam Schneider

In 1953, President Dwight Eisenhower gave a speech to remember. In front of the UN General Assembly, he proposed a new international body to regulate and promote the peaceful use of nuclear power. Now known as the Atoms for Peace address, Eisenhower attempted to balance fears of nuclear proliferation with hopes for the peaceful use of uranium in nuclear reactors:

‘The more important responsibility of this atomic energy agency would be to devise methods whereby this fissionable material would be allocated to serve the peaceful pursuits of mankind.’

Ike’s speech was about winning hearts and minds at home as much as securing America’s position abroad. The promise of limitless clean energy for the home, fantastic new technologies, and novel ways to travel represented the optimism of the nuclear age. The following year, Lewis Strauss, chairman of the US Atomic Energy Commission, famously said of the introduction of atomic energy: ‘It is not too much to expect that our children will enjoy in their homes electrical energy too cheap to meter’.

But it wasn’t to be. US and European opinion soured on nuclear technology after high-profile accidents at Chernobyl and Three Mile Island. While countries like France managed to keep its reactors online, states like the UK effectively outlawed the technology by making planning impossibly difficult and abiding by dodgy science about safety standards. No prizes for guessing which country has cheaper energy. Chernobyl likely resulted in thousands of deaths, but the data today tells us that nuclear power is among the safest of all energy sources. The primary result of the backlash was higher electricity prices and more CO2.

Whether or not a technology works is only half the battle. You also need the public to want it to work. Take genetically modified organisms. In 1978, Genentech successfully spliced a human gene into E.coli to synthesise insulin, igniting a wave of optimism among scientists. But almost a decade later, the public protested when bacteria modified to reduce the formation of ice became the first GMO to be released into US fields. Scientists spoke of new crops resistant to drought, floods and pests, but the public worried that companies were only interested in engineering herbicide-tolerant crops, so that they could sell more herbicide.

Concern about ‘Frankencrops’ took on a life of its own, blending ethical unease with fears of corporate control and ecological disruption. Particularly in Europe, but also beyond, this deep cultural mistrust led to activist groups, tougher regulation, and the rise of ‘natural’ alternatives. When added to the costs and complexity that genetic engineering experiments already faced, promising innovations were blocked, slowed or discarded. This had the unfortunate consequence of lowering the ceiling on the global agricultural supply.

There are tentative signs of a change in direction. In 2025, global nuclear power generation should inch towards a historic high, and the number of new reactors under construction is growing. GM-crops are also ticking up, particularly in low- and middle-income countries. But governments are moving cautiously, as consultations make clear that a delta remains between scientific support and public caution. Consider the Philippines. In 2021, the government approved the cultivation of golden rice, engineered to contain beta-carotene, to help prevent blindness, after a process spanning more than 20 years. In 2024, environmentalists convinced the courts to ban it.

For AI, the parallels are imperfect but useful.

First, experts are generally more positive about AI than the general public. Like nuclear power, AI comes with safety risks that some fear may be catastrophic. Like GMOs, large corporations play a central role. As with GMOs, it’s also not hard to imagine members of the public viewing artificial intelligence as inherently ‘unnatural’ - a slippery term that people often use to oppose new technologies, and which captures a broad range of values, anxieties and beliefs.

Equally, AI is clearly different to nuclear power, GMOs, or other technologies that have struggled with public perceptions, like supersonic flights. Most obviously, millions of people knowingly use AI every day, to draft emails, help with homework, advance science, or even for companionship. What does that mean for the public’s attitudes to the technology? A new cottage industry has emerged to puzzle out this question. Each week a new survey is released. One says that people are deeply distrusting of AI while another shows their use of it increasing rapidly. Taken together, the results are hard to parse. Are people scared, hopeful, or both? Do they even understand what AI is? And what, crucially, are AI labs or policymakers supposed to do in response?

In the remainder of the post, I (a) introduce a mental model for how to think about public attitudes to AI; (b) make 15 claims I think are true about how people currently view the AI project, and (c) propose a new Global AI Attitudes Survey.

Is all this necessary? In a strict sense, labs don’t need to understand public opinion to build and deploy advanced AI systems, while policymakers can craft rules without knowing precisely what the public thinks. But if the public’s concerns — about control, the environment, long-term risk — are dismissed as ignorance, then you can bet AI’s adoption will suffer. When people think that a technology offends their values, legitimacy collapses. It doesn’t matter how good your model is if no-one wants to use it. Nuclear power and GMOs both worked on paper, but were derailed when the social licence was revoked. There is no reason to think that AI is different.

Of course, there is no guarantee that what the public says they want is what is best for the technology. As with any kind of public engagement, there will always be a question about how much policymakers and AI labs can, or should, follow public attitudes (versus trying to shape them). But these questions are premised on being able to understand what the public actually thinks about AI in the first place.

What do I mean by public attitudes to AI?

Anybody hoping to measure, shape or respond to public attitudes to AI needs to consider (at least) five foundational questions.

1. Who are you interested in?: Public attitudes to AI diverge as soon as we break the public into groups, such as more savvy users of the technology versus those who know little about it. With GMOs, the attitudes of farmers and environmental groups have been central to its successes and failures. As a general purpose technology, AI will have a much broader range of parties with distinct views, from teachers to musicians.

2. Where are they from?: Today, much of the global AI conversation is mediated by voices and polling data from English-language sources in the West. This creates a skewed picture of public attitudes to AI because the West is generally more negative than elsewhere.

3. What attitudes are you measuring?: ‘Public attitudes’ to AI is not a singular concept, but a multi-dimensional one. A cognitive dimension examines what people know and believe about AI and how it functions. An affective dimension captures their emotional responses to it, like fear, excitement, anxiety, hope, or trust. A behavioural dimension captures how people adopt, reject, or use AI systems in practice.

4. What is meant by ‘AI’?: Public attitudes to AI are never free-floating. People will respond differently if you ask about an abstracted version of the technology, specific AI applications, more advanced concepts like ‘AGI’, or the organisations involved. This is why attitudes can seem contradictory: someone might welcome AI in healthcare but reject it in policing, or trust a firm to develop a certain AI application, but not another.

5. What drivers are you interested in?: Many surveys capture less about AI and more about the underlying drivers that motivate people. Demographics, socioeconomic status, and an individual’s broader values and optimism influence not just how much they know about AI, but whether they view it as a threat, a tool, or a distant abstraction.

15 claims about public attitudes to AI

With colleagues, I reviewed ~100 surveys to try to better understand public attitudes to AI. Based on this data, I created 15 claims that I think are currently true with varying degrees of confidence. As I expand on below, surveys are an imperfect tool and my sample is limited to English-language sources. Attitudes are also fluid. So take these claims with a pinch of salt. But I think converting survey data to claims – using a good deal of intuition – is necessary to ensure that we are drawing insight from this data, to identify open questions worthy of further study, and to provide falsifiable yardsticks for future surveys to tackle.

1. Awareness and use: The general public’s awareness of AI grew slowly over the past decade but increased rapidly since 2022 when they began knowingly, rather than passively, using AI in greater numbers. Within countries, awareness and use is highest among young, high-income, tech-savvy men, although the AI gender gap is narrowing. Globally, awareness and use of AI among Internet users is highest in certain lower/middle-income countries like Indonesia and India.

2. Sentiments: In the West, a majority of the public is more negative, than positive, about AI. Elsewhere, in countries like UAE, Nigeria, and Japan, attitudes are more positive, for varied reasons. For example, Gillian Tett attributes Japan’s relative positivity about AI to its labour shortage and wariness around immigration, its history of more positive sci-fi (Astro Boy vs. 2001: A Space Odyssey), and its Shinto philosophy which avoids strict boundaries between animate and inanimate objects.

3. Knowledge: A majority of people are relatively confident in their knowledge about AI, but, when prompted, display clear knowledge gaps, including a lack of awareness of AI’s presence in everyday technologies. As people learn more about AI, they typically become more excited about it, overall, but also more worried about specific risks, like those connected to autonomous weapons or hiring.

4. Age: Younger people tend to be more accepting of AI, in both personal and professional contexts, from financial planning to relationships. They are also more likely to view AI favourably, compared to the human alternatives on offer. However, young people also worry more about certain risks from AI, such as its environmental impact, and are generally more invested in the need for AI regulation.

5. Income: Within countries, lower-income adults tend to be more anxious that AI will cost them their jobs or be used to monitor them. Higher-income adults are more likely to see AI as a beneficial productivity booster.

6. Political affiliation: Political affiliation is typically not a very reliable guide to overall attitudes to AI. In more polarised climates, like the US, those supporting the government tend to have more confidence in its ability to regulate AI, and vice versa.

7. AI applications vs ‘AI’ overall: People’s attitudes towards a representative basket of specific AI applications is more positive than their view of ‘AI’, abstracted.

8. Most popular applications: People show greatest support for AI applications that could help achieve universally-popular outcomes (detect cancer); are personally important to them (improve the environment); or tackle a daily problem they are facing (e.g. find information, plan travel). People focus less on the suitability or additive value that AI, specifically, might bring to these applications.

9. Public vs. experts: While experts tend to be more optimistic about AI than the general public, much of the public is very supportive of certain AI applications that AI ethicists worry about, such as the use of AI to detect crime or in border control.

10. Most salient risks: People worry most about risks that are framed as a potential personal loss to them (e.g. job loss, data breaches) as well as those that inspire vivid, intense fear or revulsion (e.g. bioweapons, brain-rot, child abuse).

11. Unemployment: When asked to reflect on how AI may affect society, people frequently rate the loss of jobs as one of their top concerns. However, higher-income employees are less likely to view their own job as being at risk from AI.

12. Explainability: The public often does not expect or receive explanations in other consequential domains, such as from juries in court cases. Conversely, the public views the opacity of modern deep learning systems as a major risk, and many are willing to sacrifice some accuracy for more explainability. As people use AI systems more, and the models become increasingly reliable, these demands for explainability will likely temper.

13. Catastrophic risks: When prompted, people show high rates of concern about catastrophic risks from AI such as cyberattacks, bioweapons, and loss of control. However, the public worries less about these risks than other, non-AI, catastrophic risks, like nuclear war and climate change. In general, the public worries relatively little about catastrophic risks (AI-related or otherwise), compared to everyday issues like the cost of living and immigration.

14. Regulation: When asked, a majority of the public supports AI regulation. This support remains broadly consistent, across different forms, such as requiring safety evaluations or licensing. Most people do not trust AI companies to develop AI without some regulation and oversight.

15. International cooperation: The public generally supports more international AI governance, such as more international enforcement mechanisms or international treaties. To date, most of the public has generally not accepted the ‘arms race’ argument that Western countries should accelerate AI to outpace China.

Caveats, Caveats: The limits of survey data

The claims above are based on survey data. I think there is good evidence for them, but surveys also have clear limitations as a guide to public attitudes. Many of the challenges are long-standing and can be mitigated, in part, with the right methods and resources. But it is also ever harder to convince a representative set of people to take part in surveys, whether due to call or popup blockers, survey fatigue, or a declining sense of civic duty.

• Noise: Public attitudes are noisy. This is most famously illustrated by the Lizardman’s Constant, which originates from a reference to the fraction of people in any given poll who give unexpected or bizarre answers. For AI, this means we should always expect some proportion of survey data to be junk.

• Question wording and order: Context effects mean that people can answer a question differently due to how the questions are constructed and sequenced. If you ask people a series of questions about risks from AI, followed by a question about whether they support ‘AI regulation’, respondents may respond more positively to the latter. The specific wording also matters. People view a ‘death tax’ differently to an ‘estate tax’ and may also respond differently when asked about AI ‘regulation’ vs ‘red tape’, ‘bureaucracy’, or ‘report writing’, even if the underlying idea is the same.

• Oversimplification: Surveys are limited in how much detail they can provide or ask. As a complex general-purpose technology, this is a particular challenge for AI. For example, the public is often asked whether they ‘trust’ AI, or who they trust to develop it. But trust is context-dependent. People may trust certain government bodies to oversee certain healthcare AI apps, but trust companies to develop the best language translation model. People’s trust may depend on how the underlying data is collected or stored. Without the right context, when a person says that they ‘trust’ an AI company or person, they are mostly saying that they either know who they are and/or like them.

• Social desirability: Social desirability bias represents the gap between what people genuinely think and what they report in surveys, due to societal norms and a desire for approval. Similarly, people often express an opinion in a poll to avoid saying ‘I don’t know’. This may explain why people generally overestimate their familiarity with AI or why they will happily share views on the merits of regulatory proposals that many AI policy professionals may struggle to fully understand.

What next? A Global AI Attitudes Report

To avoid the legitimacy collapse seen with nuclear power and GMOs, AI labs and policymakers need to better understand what the public thinks and feels. Surveys are not the only option. Focus groups, citizen juries, and vignette studies could pose richer scenarios, but they tend to trade off depth for breadth, are harder to repeat, and come with their own biases. Another approach would be to focus less on what people say they think about AI and more on what their use of AI reveals about their attitudes. For example, practitioners can use privacy-preserving methods to analyse how people are querying LLMs or use experiments to understand the aspects of an AI tool that people most value (or are put off by).

All of these methods have their merits. But I think surveys should remain a cornerstone of the approach. They promise not just scale, but the chance to compare attitudes across topics, groups, and time. The current landscape of largely one-off surveys, infused by each sponsor’s own view and biases, and academic side-projects, is insufficient. What is required is a serious, longitudinal tracking effort: a living global survey of public attitudes to AI.

Some governments, like the UK, already conduct regular AI attitudes surveys, as do firms like KPMG. A next step could see interested parties from governments, AI labs, industry bodies, and survey experts create a shared map of how attitudes evolve across time, cultures, and domains. Such a programme, which we might call the Global AI Attitudes Report, could repeatedly ask the same people (or highly comparable cohorts) the same core questions. This would help to separate short-term fluctuations — like a spike in concern after a viral deepfake — from more durable shifts in public values. The model here might be something akin to the British Social Attitudes Survey or the World Values Survey. But to be useful for AI, this programme would need to move faster and look wider (though we know this is possible given the success of the International AI Safety Report).

We don’t currently know what questions we should be asking about AI in five years time, or even two years time. That means the Survey should have scope to add new questions, for example based on trends in how people and enterprises are using AI, and predicted capability shifts. The primary goal should be to deliver new data and revised claims, but the project could also provide standardised survey resources for other developers to draw on.

Key components of the new Global AI Attitudes Report could include:

• A standardised question bank: The Global AI Attitudes Report should build on initial efforts to create a shared set of vetted, carefully sequenced questions, and provide guidance for how to structure questionnaires.

• AI archetypes: Rather than grouping respondents by age, gender, and income, the Survey could build and test richer archetypes of people with common attitudes to AI.

• Trade-offs: Attitudes toward ‘AI’ in the abstract reveal little, the Global AI Attitudes Report could probe views on concrete applications — from AI tutors in schools to AI diagnosticians in hospitals — and weave in characteristics and design choices that force the public to take a stance on the inevitable trade-offs that AI poses. Similarly, rather than asking the public if they support AI regulation, in general, the Report could ask trade-off questions like: “Would you support holding AI companies legally liable for chatbots’ financial and medical advice, if it meant they had to restrict free public access to these queries, to limit liability risks?”

• Openness and global representation: The Global AI Attitudes Report should serve as a shared public good, with the full results made public and the methods transparent. To be credible, the effort must include representative samples from across the world to identify both universal values and points of cultural divergence.

• AI-powered methods: It would be a missed opportunity not to use AI to help measure attitudes to AI. Academics are already using AI to help test survey questions and interpolate missing data. From Camden to Kentucky, experiments are underway to use AI to inform new kinds of public deliberation. The Report could provide a testing ground for evaluating the usefulness of AI itself to such endeavours.

With thanks to Noemi Dreksler, Chloe Ahn, Daniel S. Schiff, Kaylyn Jackson Schiff, Zachary Peskowitz, Conor Griffin, Julian Jacobs, Steph Parrott, Haydn Belfield, and Tom Rachman.

Much has been written about how AI can pose risks to society, particularly in aging Western countries where a sense of latent anxiety has taken over the discourse on technology for the past decade. Sometimes this is legitimate, and sometimes it feels like a continuation of existing Western pessimism. Few have been able to advance a positive vision of what we should be striving for at a socio-political level. Here I’d like to make an attempt. This essay explores how, by providing cognition-and-agency on demand, AI agents could amplify human agency to the point where we can escape the zero-sum traps that have plagued political economy for centuries.

There is a timeless question at the heart of any (free) society: how do we allow individuals to pursue their own interests when one person’s actions inevitably affect the well-being of another in ways that are negative-sum? Economists have a name for this: “externalities,” which can take either physical or financial forms. The sheer scale of this challenge was crystallized in a groundbreaking 1986 paper by economists Bruce Greenwald and Joseph Stiglitz. They demonstrated that because our world is rife with imperfect information, moral hazards, and incomplete markets, externalities are not the exception, but the rule. This pervasive market failure became the intellectual bedrock for modern regulatory regimes. But the solution has always been the same: the coercive hand of the state and a top-down micro-management of society. We are told that only a central authority, a government board or commission, can resolve these conflicts by dictating who can do what, and where.

But as economists since Hayek have explained, the planner in Washington (or in your state capital) simply cannot possess the dispersed, specific knowledge of time and place known only to the individuals on the ground. This isn’t the kind of theoretical knowledge you find in books, but the contextual, practical, intuitive, experiential and immediate knowledge that emerges from a particular situation in time. Writing about urban planning, Alain Bertaud argued that “planners cannot possibly know the reasons households may have for selecting a specific housing location,” so mandates often end up becoming blunt and arbitrary. Such information is tacit and is only revealed through the actions and choices of individuals within a market. This blindness points to an alternative: letting people solve these conflicts themselves.

This is the essence of the work of Nobel laureate Ronald Coase, who argued that if bargaining were cheap and easy, a polluter and their neighbor could strike a private deal without any need for regulation. Of course sometimes some pollution would still happen, but the payoff to the neighbor would ensure that both parties are better off than the zero pollution or no-limits pollution counterfactuals. The tragedy is not the existence of the conflict, but the transaction costs that prevent these mutually beneficial deals from being discovered and executed. It’s also the lesson from Elinor Ostrom, who documented how real-world communities successfully govern shared resources like fisheries and forests through their own intricate local rules.

Their shared insight is that structures that encourage bottom-up order can work better than attempting to impose top down approximations for every conflict that requires a resolution. But their work also highlighted the formidable barrier that has tends to stand in the way: transaction costs. Transaction costs are not just legal fees; they are the friction of discovery, the difficulty of negotiation, and the expense of enforcement. They are the cognitive and logistical effort required to identify affected parties and strike a deal.

Historically, because these transaction costs were insurmountable, societies defaulted back to the planner. The inability to coordinate from the bottom up became the enduring justification for control from the top down. The result was always the same: clumsy, one-size-fits-all rules that stifle innovation, distort incentives, and are inevitably captured by special interests who learn to work the system for their own benefit. Today, we are repeating this same failure of imagination in the discussion around AGI. There is a rush to assume that the only way to manage its risks is through the same top-down control model, treating AGI as a centralizing technology by its very nature. If AGI is analogous to a weapon of mass destruction, a genie in a bottle - then surely a central authority is the “optimal” answer?

I find this frame quite myopic. It fixates on the risks of a powerful new technology while completely overlooking its potential to strengthen the governance mechanisms needed for a safe, coordinated society, which could well absolve the need for a centralized solution. As a general purpose technology, AGI is well placed to help us fix our decaying social and public institutions. Better cognitive capabilities also means better coordination, better governance, and better safeguards. Instead of empowering the central planner, AGI could finally empower the individual bargainers of Coase and Ostrom by arming them with the price system: what Michael Levin and Benjamin Lyon call the “cognitive glue” of a free society.

Obliterating transaction costs

The difficulty of millions of people discovering one another’s preferences, negotiating, and enforcing agreements has always been the chief justification for government intervention. It’s why your neighbor’s leaf blower, my unwillingness to fund the local park, and a factory’s emissions all end up in blunt bans and political fights; we can’t cheaply find each other, state exact terms, and lock in a deal. The “transaction costs” are simply too high. But this may no longer be the case once we have AGI agents.

Before we begin, I think it’s important to avoid conceptualizing AGI as some sort of single omniscient brain-God - even though agents will effectively individually and collectively be ‘superintelligent’ and highly capable, and increasingly so over time. That is the central planner’s fallacy all over again. While we will continue to see ever-larger training runs creating powerful foundation models, I think it’s a mistake to assume this results in a singular AGI that carries all economically valuable tasks; economics ultimately favors efficiency at the point of delivery (inference).

Running a hyper-general model for every specialized task is incredibly expensive, and so this reality drives specialization: general models will be compressed, distilled, and optimized for specific uses. The future landscape will therefore be a hybrid: a vast ecology of personalized agents, services, applications, and robots with varying degrees of generality. While many may descend from a few common foundational ancestors, their deployment will be diverse and specialized. As such, imagining ‘AGI’ as a singular entity is like talking about “Finance” as a singular thing.

I think it’s more helpful to imagine these agents through a different lens: consider AGI deployed as a vast ecology of personalized agents and systems. This emerging ecosystem is what Tomašev et al. (2025) characterize as the “virtual agent economy,” a new economic layer where agents transact and coordinate at scales and speeds beyond direct human oversight. While this ecology will contain countless specialized agents, let’s focus on the one that matters most from an individual’s perspective: your personal advocate. Think of it as a fiduciary extension of yourself: a tireless, extremely competent digital representative, closely tied to you, its principal.

What could such an agent do? In principle, it can negotiate, calculate, compare, coordinate, verify, monitor, and much more in a split second. Through many multi-turn conversations, tweaking knobs and sliders, and continuous learning, it could also develop an increasingly sophisticated (though never perfect) model of who you are, your preferences, personal circumstances, values, resources, and more. This should evolve over time - an agent’s alignment should follow the principal’s own evolution. Recent research on negotiation agents finds that “human-agent alignment” is profoundly personal. Users expect agents to not only execute goals but also embody their identity, requiring alignment on everything from preferred negotiation tactics to personal ethical boundaries and the specific public reputation they wanted to project. There are of course important privacy considerations here, but none of these seem fundamentally intractable. For example these systems could be built on technologies like zero-knowledge proofs and differential privacy, ensuring that preferences are communicated and aggregated without revealing sensitive underlying data.

Such an agent should also be able to communicate your preferences to millions of other agents in real-time, with a nuance and specificity that is currently impossible. It knows that you’ll tolerate loud music on a Saturday, but not on a Sunday; that you’d be happy to carpool, but only if it adds less than ten minutes to your commute; that you’d willingly pay a fraction of a cent more for clean electricity, but only during off-peak hours. All this in a split-second, at the right moment, for the right purpose. In other words, AGI could enable hyper-granular contracting. The friction that has always hindered us, the transaction costs that Coase and Ostrom identified as the great barrier to cooperation, could be massively reduced. So what can we now do in such a world that was otherwise not possible?

Pollution and road-traffic negotiations

Think of the agents as a built-in coordination device: instead of each actor guessing everyone else’s move (what economists would call a Nash deadlock), they can condition their actions on shared signals and contracts, unlocking deals that were previously out of reach - a correlated equilibrium.

Consider the implications. Your agent knows you have a child with asthma. A blanket “just ban the emissions” rule sounds tidy, but it flattens everything into the same position: trivial harms and intolerable ones, essential trips and frivolous ones. When a delivery truck’s agent plans its route, it doesn’t need a government mandate to be considerate. It simply sees a higher “price” for entry onto your street, a signal broadcast by your agent, representing your strong preference to avoid diesel fumes. The truck’s agent can then calculate, instantly, whether it is cheaper to pay the “clean air fee” to you and your neighbors, or to take a different route. Conversely, if your neighbor’s agent flags an emergency, for example if she’s in labor and needs the fastest route to the hospital, then everyone’s agents can auto-drop (or even invert) the price to clear a corridor, because they actually value her getting through fast. It’s true that in some cases, enforcement of these contracts might cost more than their value; but this could be solved through automated escrows and reputation systems. Ideally the agent system transforms enforcement from a costly legal battle into a near-instantaneous computational verification.

In this scenario, the externality doesn’t vanish, but it does get a price tag. And once a cost is made clear, the marvel of the market can solve it. The problem was never the pollution itself; it was the fact that the polluter was allowed to impose a health and financial cost onto you for free. To be clear, not all agent negotiations need to be purely financial. The system I’m envisaging could enable two distinct modes: economic negotiations where willingness-to-pay determines outcomes (useful for commercial activities like delivery routes), and as I’ll outline later on in the essay, democratic negotiations where each person gets equal voting weight regardless of wealth (essential for community values like neighborhood character). Agents can seamlessly switch between these modes depending on the issue at stake - using market mechanisms for efficiency where appropriate, while preserving democratic legitimacy for fundamental community decisions.

What’s key though is that agents make that payment possible, managing a million micro-transactions in the background, all based on how your values generalize across countless situations and contexts. When I lived in London, residents of my neighborhood were unhappy with congestion on roads so decided to essentially prohibit cars from going through it at certain times; taxis and local merchants were naturally pretty annoyed. With the agent-bargaining system, these low-traffic-neighbourhood detours stop being absolute: taxis can pay a dynamically discovered “cut-through” fee, while verified emergencies glide through at zero (or negative) price.

Neighborhood character negotiations

This mechanism clarifies plenty of other thorny disagreements too. Imagine a developer wants to build an ugly building in a residential neighborhood. Today, that is a political battle of influence: who can capture the local planning authority most effectively? In an agent-based world, it becomes a simple matter of economics. The developer’s agent must discover the price at which every single homeowner would agree. If the residents truly value the character of their neighborhood, that price may be very high. The project will only proceed if the developer values the location more than the residents value the status quo. Conversely, if the residents’ asking price is lower than the developer’s willingness to pay, the project proceeds, and the residents are compensated. In either case, the true economic costs and benefits are accounted for. This mechanism forces the discovery of the most valuable use of the resource, moving beyond the current system where projects are either blocked entirely (socializing the loss of potential gains) or forced through politically (socializing the costs on the neighborhood).

But what if a resident decides to game the system and go for a really absurd price, holding everyone ransom? This is why you need a new secondary layer of institutions on top of these agents. Crucially, these institutions can be voluntary. In this neighborhood, homeowners can pool their agents into a simple bargaining club: each person privately inputs the minimum they’d accept; the software aggregates that into a single take‑it‑or‑leave‑it offer. This is essentially mechanism design in action: creating rules where being honest about your true minimum is the smartest move, not gaming the system. Overstating just risks killing the deal (you get zero), and if it clears, the payout is at the common clearing price - so padding your number doesn’t boost your check. The group speaks with one voice without surrendering property rights, and the developer sees a single, fair number instead of a hundred ransom demands.

Skeptics might reasonably worry that NIMBYs can still name absurd buy-off prices. This is a classic political economy dilemma. The benefits of blocking a project are concentrated among a few motivated homeowners, while costs such as higher rents, longer commutes, and slower growth are diffused across a wide, unorganized public. As Janan Ganesh puts it, the potential losers are an “unconscious blob of people” who don’t even know what they’re losing. Two guardrails fix this.

First, chronic hyper-bidders see their voting weight fade or must pay an “option fee”: a Harberger-style tax in which you periodically pay a percentage of the price you claim; overstate, and it soon hurts. For example, if you claim your property is worth $10 million to block a development, you must be prepared to pay taxes on that valuation too! Second, and more importantly, AGI agents can give that “unconscious blob” a powerful voice. Any coalition that vetoes must then reimburse that quantified loss, with agents handling the transfers automatically. The diffuse cost becomes a concentrated, explicit price. Stonewalling remains possible, but it now carries a real, rising cost. Moreover, with this setup, bargaining isn’t just between NIMBYs and developers; other residents, now aware of the potential gains, can bargain directly with the holdouts.

Sugar and healthcare externalities

Consider another example: sugar/junk food consumption and public health. Proponents of a sugar tax correctly identify an externality: poor diet choices impose costs on the shared healthcare system. Their solution, however, is (shock!) a clumsy, top-down tax. This harms food producers, is regressive (as it affects the poor more than the rich), and ultimately imposes a cost on many people who would not in fact be “guilty” of imposing costs on the healthcare system. An agent-based market addresses the same problem with bottom-up precision.

Instead of lobbying the government, your health insurer’s agent communicates with your advocate agent. It looks at your eating habits, calculates the projected future cost of your diet and makes a simple offer: a significant, immediate discount on your monthly premium if you empower your agent to disincentivize high-sugar purchases. At that very moment of decision, the market responds. Acting like a hyper-alert Kirznerian entrepreneur spotting a profit opportunity, a soft drink company’s agent, to retain your business, might instantly propose a deep discount on a healthier drink.

Now consider smoking bans in public places. A simple free-market approach would let every restaurant or bar owner decide their own policy. But non-smokers value having a broad range of options for a night out; if smoking becomes the default, their social world narrows significantly. This loss of choice is a cost that a full-on ban tries to crudely handle. AI agent negotiation, however, allows for a more precise, Millian solution. Instead of banning the externality, we can finally price it in through voluntary, real-time negotiation. Once again, we’re not banning the externality, but pricing it in. This price wasn’t imposed by a committee of very smart policymakers sitting in a grey room in Westminster, but discovered through voluntary, real-time negotiation. The choice remains with the individual, but it is now a truly informed choice, where the full costs and alternatives are transparent.

Another example can be seen in the rules we have on airplanes and the air we share with fellow passengers in this private space. Even when we don’t use government rules, airlines generally have to come up with a generic rule that works okay for who it expects to be on a typical airplane ride. During the COVID pandemic, even many people who wanted mask mandates for airplanes did not wear masks on flights themselves, as they considered the value of wearing a mask while others would not to be minimal.

Similarly, airlines generally do not make accommodations for people sensitive to airborne allergens. Virgin Airlines can’t tell if your peanut allergy is life-threatening or just a mild inconvenience. To avoid opening the floodgates to thousands of hard-to-verify requests (“I’m allergic to perfume,” “I’m sensitive to blue lights,”) they just make a simple, inflexible rule, like “we will serve nuts.” Much of this is, of course, due to an aversion to what seems like inevitable lobbying for accommodations that would come from conceding the principle. However, if flight policies are negotiated over by AI agents, we don’t have to choose between all or nothing on masking. We don’t have to rule out accommodations to people with allergen sensitivities for fear of frivolous requests; instead we move from all-or-nothing mandates to nuanced, negotiated outcomes, where the intensity of a person’s need is accurately represented and compensated.

This agent-negotiated world delivers three principles essential to a free and effective society.

First, accountability. A billionaire who wants to close a public beach for a private party faces a new constraint: his agent must make a public, auditable offer to every single person who would be deprived of access. The cost of his desire becomes explicit and traceable. Of course, he might still try the old route of bribing a bureaucrat in secret - but this parallel transparent market creates pressure and comparison points. When combined with AGI-enhanced governance (automated auditing, pattern detection for corruption etc.), the corrupt path becomes even more risky and costly.

Second, the power of voluntary coalitions. Today, diffuse interests are often ignored because the transaction costs of organizing are too high. A single person in a low-income neighborhood has little bargaining power. A multinational polluter is more likely to get away with building a monstrosity in a Brazilian favela than in the Hamptons, even if the true social cost is higher. But what if the agents of 10,000 residents, seeing a factory’s proposal to increase emissions, can form a bargaining coalition in a nanosecond? They can spontaneously band together and declare, “Our collective price to accept this pollution is X million dollars, non-negotiable.” They solve the collective action problem instantly, creating what is effectively a powerful digital union to counterbalance concentrated wealth.

Third, continuous self-calibration. Because every agent streams its user’s context-rich preferences into live markets, the rules themselves flex in real time. Noise caps, curb uses, even peak-hour electricity rates slide automatically as new bids and conditions roll in, rather than waiting for a city-council vote five years from now. Tacit desires, like how much quiet you need for a newborn’s nap or what premium you’d pay for a car-free street, become explicit, machine-readable prices. The system therefore functions as a permanent feedback loop. It detects mismatches between policy and lived reality, reprices the externality within seconds, and nudges behavior accordingly. Governance shifts from statute to thermostat: sometimes through formal institutions built on top of these agents, such as professional guilds, and sometimes through instantaneous ad-hoc “flash coalitions” - emergent order.

So what’s the catch?

The Coasean vision amplified by AGI agents is powerful, but it’s not a panacea. Ronald Coase himself was no naive utopian, emphasizing that his theorem was a theoretical benchmark for a world without transaction costs, not a description of reality. In practice, the theorem has faced decades of rigorous critique from economists, legal scholars, and behavioral scientists, who argue that its assumptions crumble under real-world frictions. These limitations explain why Coasean bargaining rarely materializes today, leading societies to default to clumsy government interventions or inaction.

My response to this is twofold. First, the Coase theorem and its instantiation forces us to identify and analyze frictions, like imperfect information, legal costs, or strategic holdouts, that prevent efficient private solutions. This is not to say it solves everything, but it’s a powerful toolkit to prompt us to look for creative, private, and market-based solutions to problems where we might have only considered government regulation or violence before. Second, many of these critiques ignore what governance technologies and institutional arrangements AGI can enable in the first place - and I think there are good reasons to think that this technology can help us bypass limitations that would otherwise block progress on cooperation.

It’s true that even with perfect agent coordination, there remains what Acemoglu calls the ‘political Coase theorem‘ problem: those with political power cannot credibly commit to not exploiting that power tomorrow, since no external enforcer exists for contracts with the sovereign itself. The sovereign can always renege. This is a tricky challenge, but the agent system offers countermeasures. First, the transparency created by agent negotiations raises the political cost of expropriation or bribery: it’s harder to steal what is clearly priced and publicly recorded. Second, AGI must be deployed not just for market transactions, but to enhance institutional accountability. In other words, we should automate many aspects of how we govern: automated auditing, real-time monitoring of regulatory capture, automated dispute resolution, automated public spending monitoring, and agent-based anti-corruption measures can harden the governance mechanisms that constrain the arbitrary use of power. Institutions matter!

Just as agents can aggregate citizen preferences for market negotiations, they can also transform how the “machinery of government” itself operates. The information asymmetries and coordination failures that James C. Scott describes in Seeing Like a State, where central authorities operate with crude categories that miss local knowledge, can finally be resolved as well. On the “executive” side, agent networks can provide governments with high-resolution, real-time feedback about policy impacts, citizen preferences, and emerging problems. On the “civil” side, the automation of key protections against executive corruption, overreach, and misalignment protects people against the erosion of liberal democracy.

Here, I’ll explore some of the most salient critiques, preempt common objections to applying them in an AGI-agent context, and propose some countermeasures. The goal isn’t to dismiss the critiques but to show how agents can substantially mitigate them. This strengthens the case for a hybrid system: agents handling the micro-coordination, with carefully designed institutions (including the state) addressing the rest.

Zero transaction costs, really? And what about inequality?

Skeptics might say agents don’t eliminate costs entirely; they just shift them to compute overhead, data privacy setups, agent configurations etc. This is true! But it also underestimates the scale of reduction. Agents aren’t burdened by human limitations like fatigue, bias in communication, logistical hurdles, social awkwardness, irrational decision making and so on. What costs $10,000 in legal fees today might cost pennies to compute tomorrow. Consider how a billionaire’s phone today is no more powerful or effective than yours.

Even then though, you might reasonably think that this still creates inequity in the short run. To prevent cost barriers for low-income users, governments or philanthropies could provide baseline agent services (similar to public defenders) and guarantees. This is a low cost to pay for the efficiencies gained by a system that otherwise promises to save society orders of magnitude more by slashing legal overhead, unlocking stalled projects, and turning countless externalities into win-win trades. In other words, underwriting entry-level agents for the poorest citizens is like funding public roads: a modest civic outlay that makes the whole market run faster, fairer, and vastly more productively.

In some contexts, for low-income users, governments or philanthropies could provide baseline agent services - or more likely, the necessary compute to level things up and ensure equitable participation. This is unlikely to be a huge growing cost over time, as agent tech commoditizes, these costs approach zero asymptotically. The model here could mirror school voucher systems like Sweden’s, where the government provides credits that ensure universal access to essential services while allowing choice and competition. Just as educational vouchers guarantee every child can attend school regardless of family income, “agent vouchers” or compute credits could ensure everyone can participate in democratic deliberation, access legal representation, or navigate essential government services. The key is targeting subsidies where they matter most for civic participation and fundamental rights - you’d want generous credits for democratic decision-making, healthcare choices, or educational planning, but not for negotiating garage parking disputes or lawn ornament preferences.

Alternatively, or complementarily, the system could employ direct redistribution in highly sensitive areas - providing everyone with a base allocation of compute credits or “agent wealth” to spend as they see fit. This approach avoids the paternalism of defining the above “essential services” centrally, which would recreate the very social planner problem we’re trying to avoid. Individuals could allocate their resources according to their own priorities rather than predetermined categories. A hybrid might work best: a universal basic compute allocation for personal use, plus additional targeted support for specific democratic and legal functions where equal participation is constitutionally guaranteed.

This tiered approach ensures equity where it counts without creating an unsustainable fiscal burden, while still allowing market dynamics to operate in less critical domains. In practice however, this does mean a lot of infrastructure will be required. For example, built-in protocols for multi-party discovery. For high-volume scenarios, hierarchical agents could aggregate at neighborhood or city levels. But much of this will need to be designed as part of a wider push for improving institutional decision making.

Which rights are the ‘default’?

Another important consideration here is that you still need an agreed “default position” - do people have a right to make noise, or a right to quiet? What’s the basic right that is being negotiated - the right to pollute, or the right to be free from pollution? The machinery runs either way. What does change is who ends up richer, which is why the initial allocation of these rights is a constitutional choice, not a technicality. Even if bargaining is cheap, outcomes aren’t invariant to initial property rights because wealth influences willingness to pay. A poor farmer might sell pollution rights cheaply to a rich factory not because it’s efficient, but because they need cash now. Conversely, changing who starts with the rights changes the wealth distribution, which affects what people can afford to bid and therefore changes which ‘efficient’ outcome the market settles on. Beyond wealth effects, behavioral factors like the endowment effect - people demanding far more to give up a right than they’d pay to acquire it - make initial allocations stick even with perfect bargaining. Agents might correct for such biases, though whether we want them to ‘debias’ negotiations or faithfully represent our psychological quirks remains an open design question.

So how do we ensure fairness without reverting to top-down control? What is the “default position” to start with? Well, that baseline of who starts with which entitlement is a normative, collective choice. Agents don’t magic it away; they only make it explicit, contestable, and cheap to renegotiate. My view here is that we already have many of these rights set up by centuries of jurisprudence, and this is the right starting point. To the extent that these need to change or adapt, our democratic political systems are the right mechanism to do so. The bad news is that these systems are now pretty ossified, slow, captured, and dysfunctional. The good news is that agents can improve them materially.

Beyond periodic voting on baseline entitlements, agents could fundamentally transform how citizens deliberate and exercise their democratic rights. Recent papers show that AI systems can effectively learn and represent human preferences with remarkable efficiency. Studies like ConstitutionMaker show how natural language principles can be extracted from preference data, while Inverse Constitutional AI proves that just a handful of preferences can be compressed into interpretable principles that accurately reconstruct individual and group values. This suggests agents could continuously learn citizens’ nuanced policy preferences through ongoing interactions, creating rich, privacy-protected preference profiles.

Currently, we delegate representation to biological agents - mayors, councilors, representatives - who operate within opaque, underfunded institutions plagued by accountability problems, information asymmetries, and the impossibility of truly representing thousands of diverse constituents. With agent infrastructure, we could significantly improve these systems. Imagine every citizen having a personal agent that deeply understands their values, can engage in sophisticated policy deliberation on their behalf, and coordinate with millions of other agents to find optimal compromises in real-time.

These agents wouldn’t just vote every few years but could participate in continuous liquid democracy, dynamically delegating expertise to trusted entities for specific domains, instantly aggregating or constructing preferences on emerging issues, and ensuring that policy truly reflects the evolving will of the people rather than the frozen snapshot captured at the last election. Of course, this risks enabling digital NIMBYism at unprecedented scale, and we certainly don’t want everyone’s agents micromanaging nuclear safety protocols or monetary policy - but these are mechanism design and governance challenges, not fundamental obstacles.

Today, citizens already don’t vote on every financial regulation or technical standard; agent-mediated democracy needn’t change that. To the extent that enhanced coordination could enable minorities to hold majorities hostage, we’ll need clever mechanisms to prevent such digital paralysis. There’s plenty of work ahead for policymakers, economists, evaluation designers, sociologists, and game theorists to get these institutional designs right!

Lastly, the system would also need to balance dynamism with stability. Markets require predictable rules; indeed, the constant renegotiation of property rights would destroy investment incentives. But just as options markets price volatility, an agent-mediated system could explicitly price the value of stability versus flexibility, letting some rules ossify by mutual agreement (basic property rights, contract enforcement) while others remain perpetually negotiable (noise ordinances, parking rules). The agents themselves would likely converge on stable equilibria for most issues simply to reduce computational overhead - constant renegotiation is expensive even for AGI.

But what about catastrophic risks?

A lot of people working in AI governance are interested in catastrophic risks where a few actors can impose great harm on others at scale; many will rightly say “this all sounds great but doesn’t address CBRN risks.” They’re not wrong.

A malicious actor intending to release a pathogen is not a market participant to be bargained with, and admittedly, the agent system can do little to stop them. Instead this is the state’s first and most important job: to enforce law and order and protect citizens from violence, whether from a foreign army or a domestic bioterrorist. The Coasean multi-agent framework relies on this protection to even exist in the first place: the state needs to enforce contracts. If the delivery truck’s agent agrees to the “clean air fee” but the company refuses to pay, there must be a court system: a neutral arbiter with the power to enforce the agreement. This is a non-negotiable role for the state.

In AI governance discussions, the aversion to the totalising, centralising proposals espoused by some communities has been met with the inverse prescription: various flavours of free for all e/acc libertarianism or anarchy. This falls into the opposite trap, and wrongly assumes you can do away with the state entirely. The Coasean framework does not eliminate the state, but it transitions its role from “central planner” to “framework guarantor’” focusing its power on what it alone can do. It allows the market, supercharged by agents, to handle the complex work of coordinating preferences and pricing externalities, a job the state has always done poorly. This should in principle appeal to conservatives wary of big government and liberals wary of power abuses. But it doesn’t do away with the state, and nor should it - it just makes it a lot leaner. The (gradually automated) state continues to define and enforce basic property rights, contract law, criminal justice, and constitutional rights - the bedrock rules without which agent negotiations would be meaningless.

So the Coasean multi-agent system, for all its genius, has a critical limit: it is designed to price trade-offs. It can put a price on diesel fumes, noise, or a blocked view. It cannot, however, price the non-negotiable. What happens if technology unlocks a true “recipe for ruin”? A discovery, like “easy nukes” or a simple method for creating a devastating pathogen, that allows a single actor to threaten civilization itself? This is not an externality to be bargained over!

Such a risk is a form of ultimate coercion, and its prevention falls squarely within the most fundamental duty of the state: protecting its citizens from violence. Therefore, the state’s role is not just to enforce the contracts that agents make, but to define the absolute boundaries of what they are permitted to do in the first place like prohibiting actions that create catastrophic, un-priceable risks like man-made pandemics. While the Coasean framework itself does not price these existential risks, the underlying cognitive infrastructure it creates is part of what a modern state may need to manage them: enabling the high-speed coordination and automated governance required to make the state a more effective protector.

Matryoshkan Alignment

So what does this mean for how we think about the normative/sociopolitical question of who agents should be aligned to? To whom, or what, is an agent ultimately loyal? Is it fully and solely aligned to the user, like a computer that will execute any command? Or is it aligned to an amorphous set of collective values decided by some citizen jury? Or a global institution that sets top down directives? Or is it all up to the model developer? As with humans, I think the answer is not a single master. The meta-framework is a series of nested layers of governance, like a set of Matryoshka dolls. This nested structure mirrors what Levin calls “scale-free cognition” where each level of organization (from cells to tissues to organisms) maintains its own goals and decision-making capacity, with larger-scale goals emerging from but not replacing smaller-scale ones. There are many possible layers, but for the sake of simplicity I’ll outline three here.

The outermost, and largest, doll is the law. This is the non-negotiable boundary enforced by the state. An agent, no matter how personalized, cannot be a tool for committing crimes. Your agent cannot help you orchestrate fraud, DDOS a hospital, hire a hitman, or procure materials for a bioweapon, any more than your word processor can grant you immunity for writing a fraudulent cheque. To a large extent, existing laws basically criminalize all of the above; although some ways need some updating to account for agentic capabilities, difficulty in establishing a mens rea, delineation of responsibilities across the “value chain,” and so on. There’s plenty of interesting work going on in legal circles trying to work this out, and legitimate arguments for why certain gaps may need to be filled.

Within that legal boundary operates the second layer: the free market of different services, deployers, products, and providers. A company offering an agent service is not the government; it is a voluntary association with its own rules. One social media site can cultivate a different environment from another, and users are free to choose. If a provider’s agent refuses to engage with topics the company deems harmful to its brand or community, that is their right. A user who finds these policies too restrictive is not a captive; they are a customer who can, and will, take their business to a competitor offering more customization or utility. This competition will be the primary force pushing agents to become powerful and loyal advocates for their users. Today, we arguably have an increasing number of developers of general purpose models of all kinds, costs are going down, and importantly many more actors are able to customize, fine-tune, and modify models deployed through cloud infrastructure. Unlike social media, network effects are also far weaker. And in a competitive market with switching costs approaching zero, parasitic agents get quickly identified and abandoned.

Finally, at the core, is the individual. Within the bounds of law and the terms of service you voluntarily accept, the agent’s purpose is to be your tireless, personal advocate. This is where the power of user-level customization and alignment is unleashed, where a private “cognitive DNA” can be grown. The user should have immense freedom to tune their agent to their unique preferences and values. They should also have complete privacy and control over their “cognitive profile” developed by the agent, for obvious reasons. Practically speaking though, this is the hardest part: how do you design and evidence an agent (mostly) aligned to a user? How do you evaluate this and the continuous learning? We don’t need the perfect answer to these questions - alignment is not something to be “solved.”

There are of course important technical questions that are not fully addressed - the right norms, the right level of agreeableness, the right level of deference and corrigibility, fully addressing reward hacking, ensuring agents aren’t deceptive, the right evaluations to test for user alignment, and more. Few of these have a single right answer however, and markets are generally fairly well incentivised to solve them - no company or person wants a reward hacking agent. My intention here is not to dismiss them away - rather, I think the way the “alignment problem” is often conceptualized is out of date and comparable to asking “how do we ensure what is written always leads to truth? How do we solve the ‘truth problem’?” after the invention of writing or typewriters. There isn’t and cannot be any guarantee. In fact, the starting point should be reversed; as Dan Williams notes, the real question should be “why do we even have truth at all?” This is a question of institutions and governance, and not one solved by software engineering. It’s an unsatisfactory answer only for those seeking centralized guarantees. You mean we’re going to have to muddle through things? Yes. As put by Leibo et al, we should model societal and technological progress as sewing an “ever-growing, ever-changing, patchy, and polychrome quilt.”

What we need to ensure is that agents that genuinely serve their users’ interests will outcompete those that don’t and build the right governance mechanisms. From a commercial point of view, these agents won’t just be adopted and used by everyone out of the box. They need to actually produce value to their principals too. People will want AIs for financial planning, solving scheduling and calendars, helping with negotiations with roommates, finding romantic partners, etc. I expect this adoption to begin in the immediate sphere: agents negotiating the office thermostat, allocating shared resources in an apartment block, or fairly dividing household chores. As these systems prove their worth in reducing daily friction, people will trust them more. The same mechanisms used to settle a parking dispute can be adapted and scaled up to manage urban planning conflicts or discover the true cost of local externalities in other contexts. Eventually, this bottom-up architecture provides a credible pathway to solving the grand challenges, from funding national public goods to perhaps one day even navigating the complexities of interstate disputes.

In some cases, you may need more than market forces. Just as public defenders ensure legal representation regardless of ability to pay, we may need guaranteed access to advocacy agents through voucher systems, market intermediaries, “right to an agent” provisions, oversight mechanisms for automated governance, and so on. For the most part though, users will gravitate toward agents that actually help them achieve their goals. Providers whose agents consistently deliver value will gain market share. Markets remain one of the most powerful forces discovered to date; and with agents, we can surely improve both the rules governing them (e.g. through political agents and automated governance) and the mechanisms that ensure their efficiency (e.g. through Coasean bargaining agents).

The vision presented here shifts the locus of governance from centralized coercion to decentralized negotiation. AGI agents can help us create a vastly more efficient, accountable, and adaptable society. There is no need to centralize all the labs into a government monopoly, nor should we just accelerate aimlessly and do away with the state. And unlike solving alignment for a singular, centralized AGI (where failure is catastrophic), the distributed model of millions of user-agent relationships creates a massive parallel experiment. It is a system that learns, adapts, and continuously aligns itself over time, allowing us to build a society that is both more free and more coordinated than anything that has come before.

***

Thanks to Nathaniel Bechhofer, Roberto-Rafael Maura-Rivero, Lee B. Cyrano, Andrew Cordington, Max Nadau, Ivan Vendrov, Alex Obadia, Benoit Lepine, Ryan Murphy, Harry Law, Conor Griffin, Lumpenspace and Benjamin Lyons for comments.

What will happen if AI scaling persists to 2030? We are releasing a report that examines what this scale-up would involve in terms of compute, investment, data, hardware, and energy. We further examine the future AI capabilities this scaling will enable, particularly in scientific R&D, which is a focus for leading AI developers. We argue that AI scaling is likely to continue through 2030, despite requiring unprecedented infrastructure, and will deliver transformative capabilities across science and beyond.

Scaling is likely to continue until 2030: On current trends, frontier AI models in 2030 will require investments of hundreds of billions of dollars, and gigawatts of electrical power. Although these are daunting challenges, they are surmountable. Such investments will be justified if AI can generate corresponding economic returns by increasing productivity. If AI lab revenues keep growing at their current rate, they would generate returns that justify hundred-billion-dollar investments in scaling.

Scaling will lead to valuable AI capabilities: By 2030, AI will be able to implement complex scientific software from natural language, assist mathematicians formalising proof sketches, and answer open-ended questions about biology protocols. All of these examples are taken from existing AI benchmarks showing progress, where simple extrapolation suggests they will be solved by 2030. We expect AI capabilities will be transformative across several scientific fields, although it may take longer than 2030 to see them deployed to full effect.

We discuss some of the report’s findings below.

Scaling is likely to continue to 2030

On current trends, the clusters used for training frontier AI would cost over $100B by 2030. Such clusters could support training runs of about 10^29 FLOP – a quantity of compute that would have required running the largest AI cluster of 2020 continuously for over 3,000 years. AI models trained on such clusters would use thousands of times more compute than GPT-4, and require gigawatts of electrical power.

This exemplifies a repeating pattern in our findings: if today’s trends continue, they will lead to extreme outcomes. Should we believe they will continue? Over the past decade, extrapolation has been a strong baseline, and when we investigate arguments for a forthcoming slowdown, they are often not compelling.

Below we recap some of these arguments, and our conclusions from the report:

• Scaling could “hit a wall”, i.e. AI systems might fail to improve with further scaling. But recent AI models have seen large improvements on benchmarks and revenue. This could happen, but there isn’t obvious evidence of it yet.

• Data stocks for training might be used up. But there is enough public human-generated text to scale to at least 2027, and synthetic data can be generated in large quantities, and its usefulness is better-established after the invention of reasoning models. It is difficult to fully rule out a data bottleneck, but it seems surmountable.

• Scaling could be bottlenecked by electrical power. If scaling continues, frontier training runs will require gigawatts by 2030. This would be difficult to supply, but there are ways to rapidly scale up power delivery, such as solar and batteries, or off-grid gas generation. Moreover, frontier AI training runs are already beginning to be geographically distributed across multiple datacentres, which would temper the challenges. Electrical power is unlikely to be a bottleneck before 2028, and seems solvable even after that.

• Scaling could become too expensive and AI developers stop investing. This is certainly possible, but so far there is little sign of it. If AI developers’ revenues continue to grow on recent trends, they would match the $100B+ investments we extrapolate for frontier training in 2030. AI revenues growing to hundreds of billions may seem extreme, but if AI could improve productivity in a significant fraction of work tasks, it could be worth trillions of dollars.

• AI development could shift to focus on more efficient algorithms. But algorithmic efficiency has already been improving within the existing compute growth. There is no particular reason to expect algorithmic progress will accelerate, and even if it did, this seems likely to encourage using more compute.

• AI companies could reallocate compute to inference, e.g. for running reasoning models and other products. But currently training and inference receive comparable compute, and there are reasons to expect training and inference should scale up together. Scaling training creates better AI models, which will be able to do more valuable inference tasks, more affordably. There might be a shift to inference, but it seems unlikely that inference scale-up would slow training scaling.

In light of the above, we believe that extrapolating present trends to 2030 is a strong baseline prediction. And if they do continue, that allows us to extrapolate AI capabilities, which we discuss below.

AI will accelerate scientific R&D across several domains

In the report, we also examine concrete examples of how AI could improve productivity. We focus on scientific R&D, which is a declared focus of several leading AI developers.1 Capabilities trends suggest there will be tremendous progress in AI for scientific R&D, particularly in areas such as software engineering and mathematics, where realistic tasks can be trained on entirely in silico. By 2030, existing benchmark progress suggests AI will be able to implement complex scientific software from natural language, assist mathematicians formalising proof sketches, and answer complex questions about biology protocols.

By 2030, we predict that many scientific domains will have AI assistants comparable to coding assistants for software engineers today. There will be differences compared to software engineering, for example more of a focus on reviewing and synthesising large and heterogeneous literature, whereas existing AI coding tools are primarily limited to the context of a single project. Nevertheless, there are important similarities: offering suggestions in response to context, finding relevant information, completing smaller closed-ended tasks in their entirety.

We predict this would eventually lead to a 10-20% productivity improvement within tasks, based on the example of software engineering.2 Even if the work tasks of a mathematician or a theoretical biologist are less amenable to automation than a software engineer, we already have evidence from relevant benchmarks improving, and anticipate many more years of progress still to come. We expect AI capabilities will be transformative across several scientific fields, although it may take longer than 2030 to see them deployed to full effect.

We present four examples from the report below: software engineering, mathematics, molecular biology, and weather prediction. Although the selected benchmarks can’t capture the full scope of challenges in each domain, they offer insight into AI’s increasing capabilities, and the tasks that may soon be automatable. Scores are collected from leaderboards and model cards, limiting fits to top-performing models.

Software engineering: AI is already transforming software engineering through code assistants and question-answering. By 2030, on current trends, AI will be able to autonomously fix issues, implement features, and solve difficult (but well-defined) scientific programming problems.

Mathematics: AI may soon act as a research assistant, fleshing out proof sketches or intuitions. Early accounts already document AI being helpful in mathematicians’ work. Notable mathematicians differ greatly in how relevant they think existing mathematical AI benchmarks are for their work, as well as in their predictions for how soon AI will be able to develop mathematical results autonomously, rather than as an assistant.

Molecular biology: Public benchmarks for protein-ligand interaction, such as PoseBusters, are on track to be solved in the next few years, although the timeline is longer (and uncertain) for prediction of arbitrary protein-protein interactions. Meanwhile, AI desk research assistants for biology R&D are coming. Existing biology protocol question-answering benchmarks should be solved by 2030. While these benchmarks don't represent the full scope of challenges in molecular biology, their trends offer a specific window into AI's growing capabilities in the field.

Weather prediction: AI weather prediction can already improve on traditional methods from hours up to weeks. Moreover, AI methods are cost-effective to run, and could improve further with more data. The next challenges lie in improving existing predictions, especially rare events, and making use of improved predictions to achieve benefits in the wider world.

A recurring theme in the work is that deployment and societal impact may significantly lag capabilities. For example, compared to pharmaceutical R&D, software engineering has shorter iteration cycles, does not require wet lab experiments or clinical trials, rarely involves safety-critical systems, is often easy to check for approximate correctness, and has abundant training data. For these reasons, we expect that few if any of the drugs approved for sale by 2030 will have benefited from today’s AI tools, let alone those of 2030. However, early-stage development is likely to be seeing significant effects from AI by then. In comparison, we expect software engineering will have changed dramatically, and we expect there will be a flourishing of software for scientific R&D and more broadly.3

Conclusion

By 2030, AI is likely to become a key technology across the economy, present in every facet of people’s interaction with computers and mobile devices. If these predictions come to pass, then it is vitally important that key decision makers prioritise AI issues as they navigate the next five years and beyond.

To learn more, read the full report at Epoch AI’s website.

When the EU AI Act was drafted, pre-training compute was a reasonable proxy for model capabilities. At the time, pre-training accounted for 90-99% of total training compute, and the relationship was relatively reliable: more compute meant larger models pre-trained on more data, which consistently translated to stronger capabilities.

This simple proxy has been steadily breaking down. While pre-training compute remains a primary driver of capabilities, modern AI development leans heavily on distillation, synthetic data generation, reward models, and reasoning post-training. These methods can consume significant compute and drive capability gains, yet are often unaccounted for in current regulatory frameworks.1

The standard approach for measuring compute, used by the now-defunct Biden AI executive order, is to sum compute across two stages: "pre-training" and "post-training." If the sum crosses some predefined threshold, the model is subject to additional scrutiny.2 But as training methods continue to evolve, this metric risks measuring an increasingly narrow slice of the factors that produce advanced capabilities. This would turn it into a poor proxy for the capabilities that these policies aim to govern.

The current approach faces three main challenges:

1. Not all uses of compute contribute equally to model capabilities: For example, recent “reasoning training” methods during post-training often yield higher capability gains compared to other post-training interventions, per unit of computation.

2. AI labs can use compute for methods beyond pre/post-training: Besides pre-training and post-training, compute-intensive techniques like distillation and reward model training also directly impact model capabilities.

3. When deployed, an AI model's downstream capabilities depend on more than the compute used to train it: Downstream model capabilities are heavily influenced by the tools and systems available during deployment, such as coding environments and web search access.

These challenges mean that current compute-based AI policies are built on increasingly unreliable proxies for model capabilities. This doesn’t necessarily make such policies moot - they still offer key advantages that many other approaches lack. But, compute metrics may need to be periodically assessed to ensure they capture the main drivers of capabilities, and updated if they do not do so adequately. Furthermore, it could help to research better metrics, complement compute metrics with a broader evaluation regime, and focus on governing AI applications or organizations.

1. Not all uses of compute contribute equally to model capabilities

Training frontier large language models can roughly be broken down into several distinct stages. The first is “pre-training,” where models are trained to predict the next token in a large corpus of text, imbuing it with the ability to output sentences in natural language. The second is “post-training,” where various additional techniques are implemented to refine the model. These can help improve the model’s reasoning abilities or prevent the model from responding to user requests in malicious ways.

Importantly, these different training techniques have varying compute-to-capability profiles. The most pertinent example of this is reasoning training, a type of post-training that helps language models develop more sophisticated problem-solving and reasoning abilities. This typically involves multi-stage pipelines that combine techniques like supervised fine-tuning and reinforcement learning (RL), and allows models to think for longer (“inference scaling”).

In current language models released by frontier AI labs, it’s likely that reasoning training yields higher marginal capability returns than pre-training. We can see this by looking at major model releases over the last year.3 Consider the language model GPT-4o and its counterpart o1-high, which has undergone additional reasoning training and inference scaling. Despite using a small amount of extra compute4, performance jumps from 48% to 77% on the GPQA Diamond benchmark, and from 50% to 95% on MATH Level 5. While the effectiveness of reasoning training relies on a strong pre-trained base model, once that foundation exists, the marginal returns per FLOP are much higher than further pre-training. Had OpenAI instead used the same amount of additional compute to simply continue pre-training, the performance would have barely changed.

When governance efforts aggregate training FLOP into a single figure, they risk missing this outsized impact per FLOP and underestimating a model’s true capabilities. Importantly, this effect could grow as the share of compute dedicated to reasoning training increases. In the example above, the amount of compute allocated to reasoning training for o1-high was likely less than 10% of the pre-training compute for GPT-4o. But reasoning training compute has grown roughly 10× every three to five months, far outpacing the 4-5x annual growth in pre-training compute, and could soon constitute the majority of total training compute in frontier models.

As this happens, a raw sum of pre-training and post-training compute risks becoming a progressively worse proxy for model capabilities. If post-training continues to yield a strong compute-to-capability ratio at higher compute scales, a future model could be trained with a fraction of the compute specified by existing thresholds, yet still achieve capabilities that far exceed those that the threshold was intended to capture.

2. AI labs can use compute for methods besides pre/post-training

Even if we properly account for the relative importance of pre-training and post-training compute for increasing capabilities, issues remain. One such issue is that there are other uses of compute that matter for model performance, but are nevertheless missed by existing compute metrics. In this section we consider three examples: knowledge distillation, synthetic data generation, and reward models.

Knowledge distillation

The first overlooked method is knowledge distillation. In this process, a smaller "student" model learns from a larger "teacher" model by being trained to mimic the teacher's intermediate outputs (“logits”). With a strong teacher, this method can deliver higher marginal capability per student FLOP than doing standard pre-training on the same student model.

The efficiency gains from distillation can be dramatic. DistilBERT, an early demonstration of this technique, preserved 97% of its teacher model's capabilities while using 40% fewer parameters and requiring merely 3% of the compute budget that went into training the original BERT model.

Given the setup, we need to account for three sources of compute: the student’s training, the teacher’s training, and the teacher’s generation of logits. However, current policies typically only count the compute from the student run, ignoring the upstream teacher compute that largely determines what the student model can learn.

Distilling models is increasingly becoming standard practice among frontier AI labs, serving smaller, more efficient models that are distilled from a teacher model. For example, Meta trained the mid- and small-sized Llama 4 models by distilling them from the larger "Behemoth" model.5 The reason for distillation is straightforward: running smaller models is typically faster and requires fewer computational resources. So as this practice becomes increasingly common at the frontier, current regulatory compute metrics risk underestimating model capabilities.

Synthetic data generation

Related to distillation, frontier language models are increasingly being trained on output text generated by other language models.6 If a lot of synthetic data is generated, this can constitute a pretty substantial fraction of total training compute. For example, generating synthetic data for the phi-4 model comprised ~25% of its pre-training compute budget, leveraging a separate stronger model (GPT-4o) to produce the training material.7

While the EU AI Act does include synthetic data generation in training compute, the Biden AI Executive Order did not. Either way, synthetic data pipelines are quickly evolving in ways that simply counting FLOP may not fully capture. For example, Kimi K2 used models to generate entire post-training environments with tool specifications, tasks, evaluation rubrics, etc. These kinds of environments structure how models learn, yet the compute behind them isn’t neatly captured by current FLOP-counting rules. Models can also be used extensively for data curation, such as by assessing data quality, filtering, and augmenting data. This could mean taking a basic math problem and generating individual reasoning steps. These examples illustrate how current compute metrics may miss the full scope of what “synthetic data” entails.

Reward models

Reward models are another compute investment that regulatory frameworks miss. These are individual models that are trained to provide feedback signals for RL algorithms. The quality of these feedback signals directly determines RL effectiveness and what capabilities the models ultimately gain, yet the compute invested in building reward models remains entirely unaccounted for. This process is computationally intensive: labs might generate millions of model responses to prompts, requiring substantial inference and training compute.

We saw the importance of reward models in this year’s International Mathematical Olympiad, where AI systems from OpenAI, Google DeepMind, and Harmonic all won gold medals. Notably, OpenAI researchers credited their success to a robust “universal verifier” and reward models. But despite this importance, the substantial computational investment in training and running these reward models remains invisible to current regulatory compute metrics.

The diversity in training methods challenges standardized compute metrics

The three examples we’ve seen point towards the complicated reality where there are many different forms of training compute. Each of these implementations represents just one approach among many possible variations. For instance, distillation alone can include progressive distillation, multi-teacher setups, and self-distillation, while synthetic data generation can be everything from simple augmentation to complex multi-agent simulations. Each variation has potentially different compute-to-capability profiles, further complicating any attempt to create standardized compute metrics.

Beyond the broad set of techniques, we also need to understand how these training methods interact with each other. For example, the compute-to-capability profile of reasoning training could depend on the strength of the pre-trained model and the reward model. As the complexity of training pipelines grows, we need to account for more and more of these interactions.

Even if these additional sources of training compute were accounted for, we still have a poor understanding of how these sources of compute individually contribute to capabilities, let alone how they interact. This makes it difficult to construct robust metrics that appropriately proxy for capabilities, given the information we currently have.

3. When deployed, an AI model's downstream capabilities depend on more than the compute used to train it

Even if we could address the measurement challenges above, compute would still be an imperfect metric. This brings us to a third and final point: when deployed in the real world, a model's capabilities depend on more than the amount of compute used to train it. In particular, users are increasingly interacting with products or applications that are built around individual models. This means that the overall observed capabilities will also depend on the inference budget and scaffolding around models, such as tools for coding or web browsing.

One notable example of this is Anthropic’s product Claude Code. This transforms the base Claude model into a far more capable coding assistant through sophisticated scaffolding, giving it access to external tools and databases. These capabilities were enhanced through design and integration choices rather than from additional training compute.

This point is essentially missed by current compute thresholds, which instead focus more on the model’s structure (i.e. its training compute) rather than on its function (i.e. what the overall model can do given access to tools). The risk is that this blind spot will only grow, as system design itself improves over time and the capability gap between a raw model and the same model integrated into a well-designed system progressively widens.

What does this mean for AI public policy?

Given the rapidly evolving relationship between compute capabilities, does this mean that compute metrics are moot? Not necessarily. Training compute offers compelling advantages. It is roughly allocated before training begins, auditable after, and comparable across organizations. Moreover, frontier capabilities will likely continue to arise at the top of compute spend. For governance purposes, the goal of compute metrics could be to identify which models deserve scrutiny rather than precisely predicting their capability levels.

However, as training pipelines evolve, factors beyond pre- and post-training could increasingly drive capabilities while remaining invisible to current compute metrics. The risk is that current compute metrics may become progressively worse proxies for even identifying which models are pushing the frontier, if the compute we're counting represents a shrinking portion of what actually drives model capabilities.

If policymakers want to continue using compute metrics as part of a broader AI governance portfolio, they could consider several ways to address the aforementioned issues.

• Researching better compute metrics and thresholds. At present, our understanding of the relationship between compute and capabilities lacks a robust empirical foundation, and it’s possible that this will continue as different ways of using “compute” become common. At a minimum, updating these regulatory frameworks would require a better understanding of this relationship. The goal of course isn’t to create a perfect proxy, but with proper data collection and analysis, we can better understand to what extent current metrics need to be modified, discarded, or reserved only for frontier models.

• Building a more robust and expressive evaluation regime. Training compute doesn’t need to be the only input for determining downstream capabilities. For example, it can be combined with a range of other evaluations, such as automated benchmarks and expert-teaming. These help generate a more reliable portrait of a model’s capabilities, especially those of greatest concern to policymakers.

• Focus on application-level or or organization-level regulation. While a lot of emphasis has been placed on regulating individual AI models, policymakers could also consider focusing on regulating AI systems in the applications where the most significant risks could materialize. Regulation could also be done at the level of an organization, abstracting away from issues about the precise definitions of certain kinds of compute.

These approaches may help directly improve compute metrics, or at least make them less load-bearing in policy decisions. What policies make the most sense is also likely to evolve over time, so it’s crucial to continue monitoring the changing relationship between compute and capabilities. Policies can then be adapted based on updated evidence.

We would like to thank JS Denain, Conor Griffin, Jaime Sevilla, Alexander Erben, and Zhengdong Wang for their feedback and support.

New technologies shape how we communicate. The printing press spread literacy to the masses. The talkies did away with silent films. In the streaming era, songs have become shorter. One particular effect of new technologies is that they often introduce a degree of standardisation in the medium and homogenisation in the message. This influences how we come to experience and understand the world. Since this essay begins with the terms “medium” and “message”, it’s only fitting to turn to Canadian philosopher and media theorist Marshall McLuhan.

In his seminal 1964 work, Understanding Media: The Extensions of Man, McLuhan observed that nationalism was largely unknown in the Western world, until the printing press allowed people to encounter their mother tongue in a standardised form. This linguistic uniformity, he argued, helped to forge national identities and weaken older regional loyalties. Recognising this power, governments moved to regulate these technologies, out of a desire to cultivate a common culture and a fear for where a more hands-off approach may lead.

For example, McLuhan noted how some Arab countries banned the use of private headphones to ensure that radio listening remained a public, collective act. Similarly, the legal professor Lili Levi has described how the US Federal Communications Commission saw radio as serving a 'homogenizing and unifying social role’. In that spirit, it banned ‘propaganda stations’ focused on specific subcommunities and required broadcasters to cover controversial issues in a ‘balanced way’, prioritising a shared public understanding over the creative whims of producers.

In the digital era, most governments’ ability to regulate and shape media technologies in this way has diminished, even if their appetite for control has not. Unlike radio, the internet isn’t constrained by limited airwaves. It is abundant, accessible, and decentralised (a bit of a misnomer, but you get the point). Ironically, however, in the online world, the push towards ‘cultural sameness’ seems only to have intensified. But this time, it’s not top-down regulation that’s driving it - it’s individuals themselves, generating and consuming the same kinds of content, in a remarkably bottom-up fashion.

What is causing this push? For many, the separation between the online world and real life is collapsing. As people spend ever more time online, culture has begun to mould itself around the internet’s logic. Encouraged by recommender systems, which have tended to privilege popularity over diversity, we increasingly consume the same content, speak in the same idioms, and contribute to the same trends.

In his 2015 manifesto, The Transparency Society, philosopher and cultural theorist Byung-Chul Han called this condition the ‘digital panopticon’. Unlike Foucault’s silent, isolating surveillance, Han describes a world where individuals willingly perform their lives under the gaze of others. In other words, we’re not just watched, we’re performing for the watchers. We aestheticise our lives to meet the algorithmic standard. But as we do so, the boundaries of expression shrink. We speak, we share, we record - but we begin to sound eerily alike. Shakespeare saw this trend early when he wrote that “all the world’s a stage, and all the men and women merely players.”

In our post-pandemic world, this stage has collapsed inward. There’s no need to wait for an audience and there is no intruder. We invite the phenomenon with open arms; the screen is enough. Platforms normalised the broadcasting of the self, first as an act of occasional performance, and now as a ritual. In such an environment, performance begets mimicry. We begin to replicate one another and our expressions become less about interiority and more about legibility - what will be seen, liked, or shared. Over time, this creates a loop of sameness, where individuality is shaped not by thought, but by visibility.

Not all content is homogenised equally

Most of us could recognise “internet speak” without being able to define it. Picture a “day in the life” vlog: the flat, affectless voiceover, the slick edits, a minimalist apartment in neutral tones. Curtains drawn back. Coffee brewed. Jog completed. Every moment tailored to project discipline, control, aspiration, and a soft, sterile affluence. Our protagonists exist in curated solitude. No friends or family appear. No noise, no clutter. Just productivity, aestheticised.

Influencers may have pioneered the template, especially during lockdowns, but everybody now performs it and the algorithms promote it. Internet speak extends to memes and catchphrases - those brief, hyper-referential fragments of culture that flood our feeds and vanish almost as quickly as they appear. Some linger longer than others - ”Catch me outside, how about that?” (2017), or “Pooja, what is this behaviour?” (2022), or “Very demure, very mindful” (2024) - but most dissolve into the endless feed before registering. It begins to resemble that infamous scene in A Clockwork Orange, where the protagonist is strapped in, eyes forced open, condemned to absorb whatever flashes on the screen. Only now, we do it to ourselves. Willingly.

The online world also shapes how we speak, or at least how we perform speech, offline. What becomes popular can affect our daily rhythms, accents, and verbal tics. In 2021, viral news reports suggested that American children had started speaking in vaguely British accents after extended exposure to Peppa Pig during the Covid-19 lockdown. The evidence for this was anecdotal, but it points to a larger truth - online culture is a powerful vehicle for exporting linguistic habits.

Consider the Valley Girl-style overuse of “like” as a filler word. I hypothesise that this can be traced, at least in part, to the fact that much of the technology that enabled self-broadcasting emerged in the US and so the linguistic ground was first occupied by native speakers who set the tone, cadence, and affective style. Everyone else simply adapted. The result is a speech pattern that feels breezy, casual, and emotionally flattened - the kind of tone that algorithms also seem to prefer.

If the “Catch me outside” girl is an exaggerated embodiment of this performative cadence, so too is the rise of “up talk”, or High Rising Terminal, where declarative sentences end with a questioning inflection. This phenomenon was once also associated with young women in Southern California, but has spread far beyond its origins, adopted by speakers across regions, genders, and class lines. Its rise is a reminder that platform-mediated speech is not only shaped by visibility but also by who gets seen and heard first.

This algorithmic conformity doesn’t stop at the content we express. It seeps into our physical surroundings. Today, whether you’re in New Delhi or Kathmandu, walking into a café increasingly feels like entering the same meticulously curated scene: exposed brick walls, high-contrast Edison bulbs, matte-black menu boards (now swapped for QR codes), and a menu of boba tea, matcha, or some trendy variation thereof. The furniture is distressed, the playlist is ambient and indie, and the vibe is always tuned to a soft, sterile version of authenticity. The banality of the internet’s lingua franca - minimalism, mindfulness, and a curated performance of taste - has spilt into the offline world, where it is shaping how and where we express ourselves.

Yet this cultural export is rarely reciprocal. How often do Western users adopt words or phrases from Indian English, say, “prepone,” for instance? The flow of influence remains largely unidirectional. We absorb the dominant voice, often unconsciously, while our own idioms and accents remain peripheral. This is not to say Indian-origin words haven’t entered the English lexicon: terms like “pariah” and “loot”, absorbed during colonial and pre-industrial periods, are now firmly embedded in our colloquial vocabulary. But these borrowings are historical artefacts, not reflective of contemporary cultural parity. This imbalance is also at the heart of concerns about large language models.

Enter LLMs

In 2024, researchers published an experiment in which they gave a very small number of participants access to an LLM and tasked them with coming up with creative ideas. They compared these ideas against those that participants generated when using a set of ‘creative inspiration’ cards from the artists Brian Eno and Peter Schmidt. The authors argued that, in aggregate, the LLM-based outputs were less “semantically diverse”.

The worry about LLMs and homogenisation goes something like this: LLMs are trained with similar architectures and data and will inevitably be somewhat biased as a representation of global culture. If people and organisations start to use multimodal LLMs, or agents based on them, across the cultural spaces, to help write newsletters, generate images or create music this will inevitably lead to more cultural homogenisation. The fear is perhaps best crystallised in a 2017 tweet by software engineer-turned-writer Chet Haase.

Today, Haase’s observation feels less like a joke and more like a thesis: LLMs will generate what is most palatable, most generic, and least likely to offend. The average will become aspirational. The edge, or the anomaly, that which is strange, specific, or difficult, will be quietly filtered out. Think GenericAI, not GenerativeAI. An engine of blandness. An algorithmic distillation of the already said. Or as K Allado-McDowell put it: “LLM base models are built to be mid”. Beneath all of it lies a deeper anxiety: culture was once filtered by institutions or gatekeepers, but now it is being flattened by machines. Sameness is the product.

Are these concerns legit?

Recently, the British newspaper The Guardian faced some challenges after using the word “gotten” - the North American past participle of “get” - in an opinion piece. In response, they reminded readers that two-thirds of them were based outside Britain and that their US desk aims to reflect different linguistic and cultural norms. It also pointed out that “gotten” wasn’t a recent American invention, but rather emerged in Middle and Early Modern English. Language, it said, isn’t a fortress. The defence is compelling, at least to me. Language is indeed dynamic; what feels foreign or jarring one day often becomes colloquial the next. It is normal for words like “chat,” “normies,” “looks maxxing,” or “zoomers” to begin as internet slang, fringe and unserious, and later become embedded in mainstream vocabulary.

One could also challenge the idea that culture has become more homogenised, as it has moved online. Indeed, you could argue the inverse. That the internet has enabled the creation and circulation of an unprecedented heterogeneity of content. Memes mutate hourly; aesthetics are born and buried within weeks; microcultures bloom and vanish on Discord, subreddits, and Substack. When I zoom out, I still observe a kind of creative inbreeding. Even when the topics are new, the formats feel recycled; the voices start to blend; everything feels just a little… templated. There may be more choice than ever before, but for most of us, most of the time, this potential seems largely untapped.

The argument that LLMs will inevitably make things worse, by churning out content from the middle of the bell curve, could also be challenged. AI is not deterministic. In theory at least, we get to choose what we optimise models for - accuracy, engagement, familiarity, safety, provocation or curiosity. These ideas have long existed in the realm of media theory - LLMs might be the first tools capable of realising them. Consider the world of science where practitioners worry about replacing the unorthodox, intuitive and serendipitous tendencies of humans with a more homogenous AI-based approach to research. Here, we already have counter examples — such as an effort that used AI to identify and amplify anomalies in massive datasets from Large Hadron Collider experiments. Are there other disruptive ideas out there, ready to be liberated by new kinds of LLM-enabled recommendation engines, semantic search, or generative media?

A deeper pushback is to question whether homogeneity is truly bad, or whether a certain amount of it may be necessary, or even beneficial. In conversations about culture, it certainly sounds bad. But maybe people need some amount of homogeneity, in order to free up our minds for more creative pursuits? Even if homogenisation is bad, it may be an inevitable cost to bear, in service of a greater good - globalisation. For centuries, this was defined by the movement of goods, capital, and people. Then the internet enabled the spread of information. And now AI will enable the spread of intelligence and the activities that rely on it. And like every previous wave, it will follow a pattern: diversity at the margins, conformity at the centre. In the early stages, choice will look like it's expanding. But over time, tastes will coalesce around the most exportable, the most scalable. And perhaps this is the inevitable price to pay for scaling connection. If we believe that globalisation, despite its faults and recent retrenchment, is a net good, are we willing to accept a steady drift toward uniformity as its cultural side effect?

What to do?

First, to get a handle on the challenge, we could measure if and how online culture is homogenising. A growing number of organisations have designed evaluations to assess the risks that AI poses, from hallucinations to cyber-attacks. But few have tried to evaluate the quality and diversity of online content, people’s engagement with it, and how this is changing in an AI world. Should we design new quantitative metrics of novelty or counterintuitiveness? Should we invest in ethnographic studies about how LLMs are reshaping cultural reference points? More important than the measurement technique: What does good look like here?

Second, if the concern is a global flattening, then governments could take their nascent ‘Sovereign AI’ programmes and apply them to cultural ends. Few would advocate returning to a world where governments dictate cultural content. But more would likely support the need to digitise and protect local languages, books, and history. Take my own home state in Northeast India. My native tongue - an oral language with scant written documentation - remains virtually absent from digital corpora, as do many others. A recent, mildly absurd anecdote illustrates the issue. An Indian food delivery app used AI to visually auto-render a traditional eastern Indian dish - Aloo Bhaja - a crispy, golden heap of shredded fried potato. What the app showed instead was a sandwich-like creation that was completely alien to the dish. Such exclusions cannot be resolved by AI alone. We need structural interventions.

Third, AI labs could empower individuals to personalise their nascent AI assistants. Users can already toggle the ‘temperature’ of their LLMs’ outputs. Moving forward, they could share more nuanced instructions, context, and feedback about what they want. Equipped with long-form memory, these assistants could create ‘belief graphs’ for what they think users want and periodically check that this is correct. This could offer the kinds of personalisation that has long been promised online, but rarely delivered. Done well, this should provide a healthy buffer against homogenisation, not least since AI agents will increasingly communicate with each other, and so more personalised agents should lead to less homogenous outcomes.

This also raises a question about whether the homogeneity critique is yesterday’s problem, and whether tomorrow’s is over-personalisation. The internet is already shifting from the public to the personal. Those living under the same roof now watch their own content on their own devices. Social media executives note how people are communicating less on public feeds and more in private chat groups. New LLM assistants could personalise people’s online experience even more. Against this backdrop, the primary concern may shift from a surfeit of homogenous culture to a lack of any shared culture at all.

To offset this, some practitioners hope to develop recommender systems that can support both shared cultural experiences and individual diversity. But what is the user themself, worried about homogeneity on one hand and over-personalisation on the other, to do? One habit you pick up as a writer is learning to sit with the official version of events for a while, whether you’re in a war zone or watching a product reveal. It may seem tedious, but there’s a skill in parsing what’s said and what’s being left unsaid. You read the press release to read between the lines, to ask not just what is being communicated, but why now, and to whom. During the recent flare-up between India and Pakistan, I found myself with little to offer the public beyond a simple plea: add friction to your life. Not because slowness is inherently virtuous, but because the flood of unverified video and hyperpartisan content overwhelmed our attention. There was no shared anchor. It was just motion and noise.

Edward Murrow, during the height of the Cold War, warned that television was being numbed by commercialism and cowardice. In his now-famous “wires and lights in a box” speech, he lamented that the screen, instead of informing and elevating the public, had become a source of confusion, distraction, and spectacle. What he feared about television, we now live through daily online, only the feed is faster, the wires are invisible, and the box is always on and in our hands. With AI, the trade-off may be even more profound. Retaining some friction is not just an aesthetic choice; it is a necessary strategy for navigating what comes next.

___________

Thanks to , and Arianna Manzini for feedback. As with all pieces you read here, it’s the personal views of the authors.

When asked to reflect on how AI may affect society, people frequently rate the loss of their job as their top concern. Such worries are not new. During the Industrial Revolution, the Luddites smashed textile machines and fought with mill owners, even if their demands were more nuanced than is often ascribed. At the turn of the 20th century, public administrators overseeing the US economy feared how new kinds of glassware and steel might impact workers.

Such fears are understandable. Jobs, particularly the skilled trades that are often most vulnerable to automation, can provide financial independence and feelings of status, purpose, and community. This is of course not true for all jobs, all the time. In his seminal book Working, Studs Terkel observed that many people feel their jobs are defined by a lack of meaning, as well as an intense disconnectedness, fatigue, and a droning anxiety about wasting their lives. But surveys of today’s employees, at least in the West, suggest that a majority are relatively satisfied with their job and that as many as 50%, or more, would want to continue working even if they did not need the money.

1. Will we need retraining to respond to AI?

When new technologies disrupted employment in the past, they typically led to an increase in aggregate employment, albeit not always immediately. This was true with the steam engine and spinning jenny during the Industrial Revolution, and with industrial robots and digitisation in the 20^th Century. However, in the US, where labor unionisation has been mostly declining, these latter technologies also widened income and, especially, wealth inequalities. This was mainly due to two dynamics - the extent of which economists continue to dispute.

First, the technologies increased productivity and economic growth, but a growing share of this expanding pie went to capital owners, especially those with a significant ownership stake in fast-growing enterprises. Labour’s share, once inflation was accounted for, declined. In 2022, according to the economist Loukas Karabarbounis, the share of income going to labor in the US hit its lowest point since the Great Depression, at just under 60% of national income.

Second, the technologies complemented people with certain skills while displacing others - what economists refer to as ‘skill-based technological change’. Digitisation and industrial robots automated middle-wage occupations such as clerks, bookkeepers, and assembly line workers, while enabling high-paying roles for software engineers, roboticists and data analysts. These new roles also fostered demand for lower-wage roles, including to provide services to higher earners, for example in the retail, healthcare, food service or personal grooming sectors. These two trends made the labour market more polarised, as workers who lost their jobs or were unable to benefit from new technologies struggled to move into higher-wage work. If we expect AI’s impacts to be similar, this could provide a rationale for governments to fund large retraining programmes to help people retain their jobs and move into higher-wage roles.

Will AI’s effects be similar? We don’t know. Some efforts to address this question break jobs down into bundles of tasks and evaluate AI’s ability to perform them, now and in the future. These evaluations cover a wide range of jobs and tasks, but they don’t tell us whether organisations that hire people to do these tasks are investing in AI, or changing their hiring. Other studies do assess the impact of AI on real-world employment outcomes, for example on freelance employment, but only cover a small share of the labour market. No assessments yet give us breadth and depth.

In the near-term, the employees at greatest risk from AI are likely those that work in, or would like to work in, occupations where tasks are currently performed (almost) entirely on a computer. And where some degree of human error is already common and not catastrophic. This may include graduates aspiring to work in consultancies, legal firms or content agencies or the large number of people who work in remote customer service roles. If new kinds of AI-enabled robots become more capable and cheap, then other kinds of roles, for example in warehouses, could be at risk.

If AI starts to cause people in these roles to lose their jobs en masse, we can expect loud calls for new public retraining programmes. The idea that the government should help to retrain people in response to new technologies, trade shifts or other ‘shocks’ is ubiquitous in policy briefs, consulting reports, and academic research, including on AI. But, these reports typically don’t specify what an AI-induced retraining programme should look like, who should do it, and what lessons, if any, we should draw from past efforts.

In the remainder of this essay, I trace the history of US public retraining programmes and their impacts. In short, I find little evidence that they have been effective. In future essays, I hope to consider lessons from private retraining programmes and other countries.

2. A brief history of US public retraining programmes

In 1933, as the Great Depression reached its darkest moments, President Roosevelt signed the Wagner-Peyser Act, creating the United States Employment Services, a new national network of offices to help the ~25% of the labour force that was out of work. Their retraining offering was rudimentary but provided a foundation to build from. Since then, retraining has become an integral ‘Active Labour Market Policy’ in the US and beyond. If passive labour market policies, like unemployment insurance, aim to provide a safety net for the unemployed, active policies like retraining and job search assistance aim to provide a ladder back into stable work.

In 1962, John F Kennedy signed the Manpower Development and Training Act, the first federal retraining programme to operate at scale. Over the next decade, it retrained 1.9m people to navigate the ‘constantly changing economy’. For men, this typically meant retraining as machine shop workers, auto mechanics, and welders. For women, clerical and administrative roles. In 1973, Richard Nixon replaced the MDTA with the short-lived Comprehensive Employment and Training Act, which focussed on getting low-income individuals, the long-term unemployed, and students into subsidised, entry-level jobs in public sector agencies and nonprofits.

CETA began a process of decentralising US public retraining, putting decisions about how to run the programmes into the hands of cities and states, rather than the federal government. In 1982, the Reagan administration accelerated this further, when it passed the Job Training Partnership Act. In line with Reaganomics, the JPTA aimed to further empower local organisations to deliver retraining and boost private sector employment. To do so, it established Private Industry Councils, with representatives from local businesses, to help direct and supervise the programmes. It also tightly means-tested participants, with the vast majority coming from low-income or “hard to serve" backgrounds, which included people with disabilities, the homeless, offenders, welfare recipients, and out-of-school youth. The training focussed on cultivating basic skills, such as remedial reading and maths, ‘work habits’, such as punctuality and résumé writing, and short, entry-level courses for clerical, services or trades work.

As I expand on below, today the JPTA is typically viewed as a policy failure. In 1998, Bill Clinton replaced it with the Workforce Investment Act, which significantly widened the criteria for participation, making retraining available to anybody who wanted it, while giving low-income and disadvantaged people priority. If the JTPA was essentially a poverty reduction scheme, the WIA aspired to become a universal employment service, including for displaced middle-income workers, when budgets allowed. The WIA also sought to prepare individuals for a more fluid labour market. Unlike JPTA and CETA, which offered participants little choice, the WIA provided individuals with “individual training accounts” so that they could (at least in theory) choose the skills and sectors to invest their time in, once they had completed some general training.

In 2014, Barack Obama replaced the WIA with the more streamlined Workforce Investment and Opportunity Act, which today provides most US federally-funded retraining. The WIOA allows participants to directly participate in their preferred retraining services, without the need to first participate in more general training. It also allows regions to offer more locally-relevant retraining and has tried to increase accountability, by requiring more third-party evaluations.

Every year, approximately ~500,000 participants take part in the WIOA’s ‘Adult’ and ‘Dislocated Worker’ streams, of whom ~200,000 receive training vouchers, at a cost of ~$500m. As noted by David Deming and colleagues, this is a relatively low figure, when one considers that the US government spends 25bn a year on Pell Grants for undergraduate education. Most WIOA participants are low-income, but their profiles vary. For example, most of those on the ‘Adult’ stream are below or near the poverty line, with limited education or employment experience. In contrast, most of those on the “Dislocated Worker’ stream have lost stable employment, for example in manufacturing, and are more likely to be older with more substantial work experience. Although not mandatory, people who receive welfare and public assistance are encouraged to apply and make up approximately one third of WIOA adult participants, according to a 2022 evaluation.

The skills that WIOA retraining programmes impart and their method of instruction vary considerably across US states, owing to differences in the capacity of local retraining providers (who must bid for federal funding), employer needs, and political considerations, which can override more objective readings of labour demand. The result is a patchwork. In 2015, Burt Barnow and Jeffrey Smith distinguished between different types of retraining, from small group sessions focussed on basic skills to subsidised apprenticeships. In the early days of the Clinton era Workforce Investment Act, 47% of participants enrolled in formal, classroom retraining programmes, but this figure ranged widely across states, from 14-96%. According to data from 2023-24, less than 10% of WIOA training involved paid on-the-job training, and just 2% involved apprenticeships.

Summing together, over the last 80 years, we have seen a steady evolution in US public retraining, from more centralised, New Deal-style programmes to more decentralized efforts targeting local private sector employers. The goal has shifted from addressing widespread unemployment to reducing poverty and back towards a more universal employment service that integrates retraining with other policies, such as job search support.

We can expect further changes. The Trump Administration has proposed merging the WIOA’s programs into a single funding stream titled ‘Make America Skilled Again,’ which may result in a significant funding cut and replace much of today's federally funded training with apprenticeships. However, the proposal will likely be altered significantly as Congress deliberates on its details. So the future of US federal retraining, and how it may respond to AI, is still very much to be determined.

3. Does public retraining work? The evidence base

Consider the hypothetical example of ‘Tony’, who used to be employed at a mid-sized auto parts manufacturing plant in Ohio, until his employer steadily introduced a wave of industrial robots. As documented by Thomas Phillippon in The Great Reversal, many US localities are dominated by a handful of ‘good’ employers. The absence of alternative options reduces worker bargaining power and wages. It also makes layoffs more challenging as employees like Tony must compete against many others, most of whom also lack transferable skills to pursue other roles.

After a prolonged job search, Tony enters a Workforce Investment and Opportunity Act retraining program, run by his local American Job Centre. Owing to his low-income status, he is given priority. Upon arrival, the Centre screens Tony to see if he qualifies for the Dislocated Worker stream. Once confirmed, he is asked to participate in maths, problem-solving, and reading assessments, as well as an aptitude test. From there, a counsellor reviews labour market data and recognising Tony’s background in the auto trade, proposes a variety of skilled trades. She also proposes the opportunity to reskill into a new sector, like medical assistance. If Tony is under pressure to return to work quickly, he may move directly to job search assistance and on-the-job retraining. Alternatively, he may pursue longer classroom retraining at a community college or technical school. If all goes well, he will have regular meetings with his counsellor, participate in soft skills workshops and networking events, and will land a secure new job, with regular progress check-ins.

In reality, successful examples like Tony are rare. In 2016, the year of Donald Trump’s first election, David Autor, David Dorn & Gordon Hanson published ‘The China Shock’ - arguably the most impactful US economics paper of the past decade. The study demonstrated that, since the 1990s, import competition from China had devastated large parts of the American workforce, particularly regions focussed on manufacturing textiles, furniture, toys and other light goods. The shock reverberated across communities, igniting brain drain and depressing economic and social prospects for a generation. Meanwhile, other sectors, regions and employees benefited from cheaper imports.

The China Shock, and the wider technology-based automation that was occurring in these sectors, led to a glut of displaced workers and a stream of youth in search of alternative employment opportunities. This was the sort of challenge that the Clinton-era Workforce Investment Act and the Obama-era Workforce Investment and Opportunity Act were designed to address. However, Autor and colleagues showed that many displaced employees either failed to find employment or were forced to take up new roles in the service sector, for example as cashiers or security guards, that were often less-skilled, lower-paid and less rewarding.

These findings chime with the more formal evidence base on US public retraining programmes. To evaluate retraining programmes, researchers expend considerable effort to track key variables, including the proportion of participants who find work shortly after exiting, the proportion who remain employed for at least six months, and their average earnings. In general, researchers have failed to show any statistically significant benefit on these outcomes.

Teasing out why - and what this means for future retraining programmes, including for AI - is difficult. One key challenge is non-random selection. The population that takes part in retraining is not representative of the wider population of people who have been displaced. This means that we do not know the extent to which participants’ subsequent labour market outcomes are due to the impact of the retraining programme (or lack thereof), or other characteristics that are more common among participants, such as a willingness to take part in retraining in the first place.

To address this issue, researchers use quasi-experiments that aim to approximate randomised controlled trials by matching a group of people that did participate in public retraining programmes - say 10,000 middle-aged men from rural districts - with a similar group that didn’t. However, to do this well, researchers need to know what characteristics are most relevant to future labour market outcomes - prior education?, proximity to a nearby city? - so that they can control them. And many potentially important social or psychological characteristics are impossible to reliably capture in datasets. On top of this, researchers must try to account for the huge variance in the focus, format, and resources of different states’ programmes. As a result, some researchers conclude that it is impossible to make reliable causal claims about why public retraining is, or isn’t effective.

The evidence that does exist provides cause for skepticism. For example, a National Study to evaluate the Reagan-era Job Training Partnership Act, involved a genuine randomised controlled trial that ran from 1987 to 1992, with a representative sample of more than 20,000 participants. It found no statistically significant improvement in employment rates, employment duration, or earnings. In 2019, a 10-year evaluation of the Workforce Investment Act, and the Workforce Investment and Opportunity Act found that, while intensive one-on-one career counselling did improve employment and earnings outcomes, the programme’s retraining streams did not.

As of 2023, the most recent data available, 70% of participants in WIOA were employed in the 2nd and 4th quarters after finishing their retraining programmes. But these outcomes are not compared to a control group, so we don't know if, or to what extent, WIOA is truly improving them. Even when WIOA retraining is helping people find jobs, research by David Deming and colleagues suggests that ~40% of participants are being trained into ‘low-wage’ support roles, particularly in the healthcare sector. The most common roles, such as nursing assistants, come with an annual salary of less than $25,000. Demand for these roles is high, which explains the WIOA’s focus on them, but they often offer little scope for career growth.

There is even evidence that some retraining programmes may hurt participants. For example, a 2012 evaluation of the US Trade Adjustment Assistance programme, which provides retraining to workers displaced by outsourcing and trade, found that participants had lower employment rates in the two years after they were laid off, compared to similar workers who did not participate, potentially due to the opportunity cost of not being able to apply for more immediate work opportunities. Even four years after losing their job, TAA participants were underemployed and earned slightly less, compared to non- participants.

4. Why does retraining fail?

In the absence of clear causal evidence, researchers are left to speculate as to why US public retraining programmes have underwhelmed.

A first challenge relates to the participants and their ability and willingness to take part. Some potential participants may avoid retraining, or drop out, due to the costs involved, which range from transport to arranging childcare - single parents are over-represented among participants. These cost pressures are particularly strong for candidates who are still in work but at risk of losing their jobs. For those with little savings, even the offer of a payment to take part in retraining, may be insufficient.

In other instances, there may be a mismatch between the training and career paths on offer and what candidates are interested in, or capable of. For example, older workers, often close to retirement, have been overrepresented in some of the jobs displaced by digitisation and may be less enticed by retraining into a brand new sector. Retraining participants are also disproportionately likely to have been homeless, an offender, or to lack the basic skills that longer classroom training requires.

All participants may be bewildered by the choice on the offer. As David Deming and colleagues note, the WIOA funds ~7,000 Eligible Training Providers and ~75,000 programmes, in more than 700 occupational fields. Although it aspires to provide ‘informed consumer choice’, via its voucher system, the websites describing different programmes can quickly overwhelm candidates, while failing to provide the comparable programme information and performance data that people need.

A second challenge relates to training providers and their ability to offer a high-quality service that is well-curated to local employers’ needs. Experts note huge variance in the quality and format of local training providers, but with such a large number of providers, the evidence base does not allow us to reliably tease out the good from the bad. Training providers also struggle with the bureaucracy that public programmes entail and the challenge of ensuring that the skills they provide are useful in an ever more specialised economy, where many skills are not easily transferable. Providers also need to look beyond the current labour market and anticipate future skills demands - a task that has always been hard, if not impossible, and which AI is now exacerbating.

A final challenge is that there may simply not be enough skilled jobs for people to retrain into. The typical question for workers looking to retrain is not: “How do I find employment?”, but rather “How do I get a more secure, better paid job?” Past technologies did not increase the aggregate unemployment rate, but there is evidence that they did lead to short to medium-term reduction in the number of ‘skilled’ occupations for workers to retrain into. In the AI era, similar challenges could emerge if, for example, new university graduates were unable to find the kind of role, or career path, they expected.

Retraining & AI: Four ideas

What does this mean for concerns about AI? At a minimum, we should avoid assuming that public retraining programmes will be a useful response. The baseline hypothesis is probably that they won’t. However, there should be ways to make them more useful.

Here are four ideas:

1. Develop better labour market projections

There is vast uncertainty about how quickly AI capabilities will develop, diffuse through the economy, and affect workers. This uncertainty will not disappear any time soon. But AI labs and policymakers could make it easier for retraining providers to understand the jobs that may get displaced, prove more resilient, or emerge. At the moment the US Bureau of Labour Statistics provides high-level forecasts for demand for different occupations. AI labs could work with policymakers and researchers to develop much richer and more granular forecasts that draw on, among others, the latest AI capability evaluations; insights from how users are querying LLMs, online job postings, and government surveys of employers and graduates.

2. Experiment with new retraining approaches

Almost 80% of Workforce Investment and Opportunity Act retraining takes place fully in-person, while just 7% takes place fully online. This creates barriers for people in more remote regions and hinders innovation. It will be difficult to usefully change this, because delivering high-quality online education is hard. As the scholar Mary Burns noted in an evidence review for UNESCO, “few innovations have generated such excitement and idealism - and such disappointment and cynicism - as (digital) technology in education.”

But now is the time to experiment. Education providers are learning from their failures, such as the early Massive Open Online Courses that crudely transposed offline learning content. Some providers are shifting to hybrid formats, while others are developing targeted micro-credential courses. In the AI community, labs are training large language models, and the tutors based on them, to be more ‘pedagogically inspired’, while educators and students are using multimodal AI to personalise learning materials to the language, format, or substance they want. Against this backdrop, there should be opportunities to design AI-enabled retraining programmes that are more dynamic and better able to respond to labour market needs. Another goal of such retraining, at least for some participants, could be on how to use AI systems most effectively.

3. Collate better evidence about what works

At the moment, we have little evidence about what works, or doesn’t, with respect to public retraining programmes. This is particularly true for training provided to workers displaced by new technologies. Future evaluations should target this group, with a focus on those affected by AI, and understand how outcomes are affected by factors such as programme type, age, gender, geography, prior education, and existing skills. This will require better data collection by public sector institutions, with a focus on RCT-style evaluations, but also smaller-scale experiments that academics or companies could run, with standardised measures to harmonise the two.

One area to explore is the nascent positive evidence on training programmes that are co-designed with employers, with specific sectors in mind. Some RCTs in this area show positive effects on earnings and employment, but they are small, typically including a few thousand people, and we do not know if they will generalise across geographies and sectors.

4. Consider goals beyond employment

Finally, it may be time to reevaluate whether ‘work’ should remain the central way to measure a person’s economic contributions and the central goal of any retraining programme. As Tom Rachman wrote in a recent essay: educational policy tends to tinker with the ‘What’ of learning (curriculum) and fret about the ‘How’ (methods). But it’s the Why that demands its boldest recalibration since the Enlightenment.

In theory, education can serve many functions, from boosting an individual’s agency to promoting national unity and assimilation. But preparing students for a career has come to dominate both retraining and broader education. If scenarios where AI has more dramatic economic effects materialise, we need to think about what other knowledge, skills and values people will need to navigate this transition, and the other ways that they can contribute to society. For example, future training programmes could include best practices on how to use AI agents, or how to improve community life and ward against atomization. This more flexible understanding of ‘work’ and ‘training’ could give us a better chance of navigating the AI economic transition in a way that preserves worker livelihoods, opportunity, and dignity.

I would like to acknowledge and thank Burt Barnow for his guidance and contributions in developing this essay. Thank you to Venus Krier for the illustrations contained in this piece.

As highlighted by Woodrow Wilson in 1887, governments are composed of two aspects - politics and administration - which, though often confused in practice, should be clearly distinguished. Politics is defined by who decides, who makes the rules, and how these individuals are selected. Administration describes government in action - the efficient and systematic execution of laws and public duties. Over time, these concepts have evolved to accommodate different forms of government, from tribes and divine monarchies to republics. The development of free and liberal democratic societies is a relatively recent phenomenon, emerging in the 16th and 17th centuries with the rise of European nation-states.

Throughout history, new leaps in technology have altered and destabilised these forms of government. In the 15th and 16th centuries, the diffusion of the printing press and a growing mercantile class enabled administrative record keeping, which alongside broader demands for property and the enforcement of contracts, laid the groundwork for more centralised forms of governance. More recently, the Arab Spring and the contemporary rise of populism in Western democracies have been attributed, at least in part, to the capacity of the internet and social media to potentiate mass mobilisations against incumbent political establishments.

AI heralds the next major technological shift. While its precise trajectory and diffusion remain uncertain, many researchers and forecasters anticipate the advent of AGI in a matter of years rather than decades or centuries. This raises a question about how AGI would impact liberal democratic societies. Daron Acemoglu and James Robinson’s ‘narrow corridor’ framework argues that free and open societies have traditionally depended upon maintaining a delicate balance between the relative powers of society and the state. This equilibrium avoids an overly-powerful despotic state on one hand, and a chaotic absent state that is too weak to govern or provide services on the other. Liberty thrives in the narrow corridor between these extremes.

Historically, liberal societies maintained this precarious equilibrium through constitutional constraints, checks and balances, and the rule of law. However, this equilibrium has never been static. Rather, technological and social change have forced repeated renegotiations of the social contract - from the rise of mass politics in the industrialising West to the welfare state reforms of the early 20th century.

Recent empirical research by Ryan Murphy and Colin O'Reilly has questioned whether countries actually follow these trajectories and challenged some of the proposed mechanisms, such as the ‘Red Queen effect’ where the state and society are thought to be locked in a constant, co-evolutionary race. But the core idea that we need to maintain a necessary balance between state power and societal autonomy still serves as a useful illustrative heuristic, when we consider the potential impacts of AGI. In particular, it helps to highlight the critical trade-offs that we can expect to face between efficiency and accountability, collective coordination and individual freedom, and technological capability and democratic control.

How could AGI strengthen free societies?

AGI will likely offer an absolute advantage over human decision-making in terms of scalability, cost, and quality. Artificial bureaucrats could draw on specialised sub-agents to dynamically switch between data interpretation, risk modelling, and stakeholder communication, to dramatically speed up lengthy tasks like interpreting legislation or carrying out environmental impact analyses and fraud detection.

AGI agents could also lead to more equitable decision making. Human bureaucracies are littered with subjectivity. Some of this stems from their use of traditional rules-based automation that leads to disproportionately harsh outcomes for marginalised groups in domains like tax enforcement or welfare eligibility. These ‘dumb’ systems can miss critical context and are typically only deployed in situations, such as to initiate audits of recipients of Earned Income Tax Credits, that are amenable to such rules-based automation in the first instance. In contrast, more general AI systems will be able to grapple with the complexities and idiosyncrasies of the taxes filed by high-income individuals, potentially reducing disparities in how laws are enforced.

AGI could also improve how governments secure democratic inputs, leading to more feedback on, and potentially control over, what governments do. Gudiño-Rosero and colleagues recently explored how “digital twins” that simulate individual citizens’ views and “represent” their policy preferences could lead to an “augmented democracy”, in similar fashion to how AI agents may soon start to represent individuals in commercial transactions. This work stops far short of creating actual digital twins for political representation, partly due to limitations in capturing individual preferences. But future AI systems, enhanced with long-term memory, could enable higher-fidelity simulations. Governments could use these simulations to draw up more effective political agendas and experiment with different policy ideas, while also raising questions about how the roles of elected officials should evolve.

How could AGI undermine free societies?

In Seeing Like a State, the late political scientist and anthropologist James C. Scott offered a critical take on government efforts to make society legible, from birth and death registries to financial reporting requirements. AGI could dramatically enhance such efforts, enabling governments to analyse vast data streams in real-time and monitor and predict societal trends, risks, and individual behaviors, at a granularity and accuracy that is far beyond current levels.

This could make government decision-making radically more efficient and data-driven. But it could also enable unprecedented surveillance and control over citizens, stifling dissent. It could also dramatically reduce the cost of monitoring whether people are complying with laws. For example, governments could pass CCTV camera feeds through a multimodal AI model for continual analysis, leading to a form of ‘perfect enforcement,’ where even minor infractions become subject to consistent punishment. While this might seem beneficial from a rule-of-law perspective, it raises significant concerns for individual freedom and the quality of governance.

Take, for example, the National Highway Traffic Safety Administration’s recall of Tesla’s ‘Full Self-Driving’ software update in 2022, because it was carrying out ‘rolling stops’ - something that human drivers regularly do when an intersection is empty. The advent of self-driving technology could make every rolling stop legible, imposing a level of perfect compliance that many humans would consider draconian and inefficient. (Although such perfect compliance may come with a silver lining - exposing outdated or poorly crafted laws that rely on lenient enforcement and human discretion.)

Delegating decisions to AGI also raises concerns about the loss of moral accountability in public administration. For example, while AGI agents may excel at optimising policies for efficiency, they may lack the ethical nuance required to address competing societal values. This disconnect between computational optimisation and human morality risks eroding public trust.

AGI could also undermine free societies by empowering non-state actors. In more positive scenarios, it could enable citizens to better understand and advocate for policy positions, fact-check officials, and usher in new kinds of public deliberation. However, individuals and groups could also use AI agents to orchestrate harmful actions, such as to manipulate public opinion or coordinate insurgencies. They could also create opaque financial communication methods that make the economy less legible to governments, rather than more, similar to how cryptocurrency can be used to launder money, despite its legibility.

How to secure free societies

To secure the narrow corridor, states must neither blindly hand off power to AI systems nor clamp down on them in ways that stifle innovation. On the technology front, novel privacy-enhancing technologies could help individuals to maintain autonomy and privacy in the face of increasingly pervasive state monitoring. Investments in interpretability could help to ensure that AGI systems operate transparently and remain accountable for their decisions.

On the institutional front, governments could embrace hybrid structures that combine AGI’s computational power with the nuanced judgment and accountability that human administrators provide. This might include equipping public institutions with their own advanced AI tools for functions like biosurveillance, cyberdefense, and regulatory oversight, ensuring they are not outpaced by threats.

Governments could also look to reinforce participatory democratic processes by enabling large scale deliberative platforms, real-time citizen feedback systems, and representative digital twins, while designing robust safeguards to ensure that they genuinely enhance, rather than undermine, democratic accountability.

Perhaps most importantly, securing the narrow corridor in an age of AGI will require an epistemic shift in how we approach the governance of emerging technologies. Rather than passively reacting to technological disruptions, policymakers and the public must cultivate a greater capacity for anticipatory governance. This means proactively imagining and stress-testing our institutions for AGI's transformative potential. To do this, we can use tools like scenario planning, threat modelling, and forecasting - drawing on AI's own emerging abilities in these areas.

Thanks to Justin Bullock and Samuel Hammond for authoring the original paper and to Conor Griffin for editing support.

A baby giggles on a picnic blanket in the park, her gaze darting around, before she blinks at a dazzle of sun. The child is still oblivious to the name of her planet, or that dinosaurs once roamed here, or that the blue above has black on the other side. But her lifelong pursuit—the project of all humanity—is underway: learning.

Only, what for?

Today (and especially tomorrow), children will blink into existence as the latest joiners of a once-superlative cognitive tribe, now somewhat diminished. In the talent we admire most, and rely on entirely, we’re falling behind machinery. When people feel ineffectual, they plunge in mood and effort. At the margins, you glimpse The Great Dejection already.

“I’ve grown not to entirely trust people who are not at least slightly demoralized by some of the more recent AI achievements,” Tyler Cowen said, while Ethan Mollick remarked: “If you haven’t had at least a minor crisis (What does this mean for my job? What does it mean for my kids’ jobs? What does it mean to think?) you probably haven’t used AI enough.”

Such anxiety feeds into a thought haunting AI progress: What are we for? It’s a species-level interrogation that helps explain why philosophers find employment in the tech sector nowadays. The public—once they reckon with what’s coming—may prefer therapists. Thankfully, chatbots can listen to our woes.

The emotive response is to condemn technology, as if it might be stopped. More pragmatic is to consider what it reveals. For instance, if students are cheating with generative AI, does this suggest that education’s objectives have become misaligned from the incentives?

Today, its main goals are job-preparation, socialization, and wellbeing. Yet in all three, education falters. Consider wellbeing: the mental health of school-age children has deteriorated for more than a decade. Socialization seems an elusive goal too, now that “real life” is screen life for many. And job-preparation is a precarious promise, given how many occupations are contingent on what AI does to the world.

Facing all this, educational policy tends to tinker with the What of learning (curriculum) and fret about the How (methods). But it’s the Why that demands its boldest recalibration since the Enlightenment.

HOW WE GOT HERE

Socrates brought a cup of poisonous hemlock brew to his lips, its whiff of mouse urine pervading his nostrils. He swallowed, then paced till his feet grew numb, whereupon he stretched out, awaiting the consummation of a death sentence for corrupting Athenian youth with his teachings. Reclining there, he embodied one last Socratic lesson: that education is a matter of control and ethics—the original alignment problem.

Society creates intelligent agents (its young), and aspires to design their behaviour. But how to protect against data-poisoning? Which reward-functions to set? And how to ensure that those agents never go berserk?

In prehistory, education meant trailing after family, absorbing skills, myths and hearsay about the larger world. But the technology of writing led to formal schooling. Sumerians established eduba to train scribes in cuneiform, Ancient Egypt constructed per-ankh houses of learning, the Islamic Golden Age saw the flourishing of kuttab schools, Medieval Italy developed scuole d’abaco, and Aztec Tenochtitlan ran calmecac for the nobility.

Technology jolted learning afresh when the printing press multiplied the quantity of information and diffused it, weakening institutional control over knowledge. The Industrial Age saw further updates, with more machinery requiring more skilled workers, and an engineering ethos infiltrating classrooms. “Teach these boys and girls nothing but Facts,” the data-processing schoolmaster, Thomas Gradgrind, says in Dickens’ novel Hard Times (1854). “You can only form the minds of reasoning animals upon Facts: nothing else will ever be of any service to them.”

To gather the Facts of education itself, testing became standardized, tracking pupil progress, and sifting the young according to their apparent talents. Standardized assessments, besides stressing children, became a debatable proxy for learning. Orwell recalled one of his teachers yanking pupils’ hair, kicking their shins, and making them plead with hands raised to affix dates to wars of which they knew nothing. “The whole process was frankly a preparation for a sort of confidence trick,” he wrote. “Your job was to learn exactly those things that would give an examiner the impression that you knew more than you did know, and as far as possible to avoid burdening your brain with anything else.”

Foucault noted the resemblance of schools to prisons, characterizing them as institutions to regulate and enforce control. But, while it’s naïve to ignore power, it’s naïve to see only power. People also wish children to thrive, and they care about strangers’ offspring too. During the Enlightenment, this humanistic vision seeped into educational theory, notably by the hand of Rousseau, who considered children as virtuous creatures that society befouls. “All is good upon leaving the Maker’s hands; all degenerates in the hands of man,” he wrote in his treatise on education, Émile.

Amid the bloodshed of the Napoleonic wars, a disorganized but kindly Swiss educator sought to apply Rousseau’s ideals. Johann Heinrich Pestalozzi—brow furrowed in a muddle of compassion and anxiety—stood before 80 uneducated orphans, far too numerous to teach at once. So he had them draw, write, and learn by their own propulsion. “It quickly developed in the children a consciousness of hitherto unknown power, and particularly a general sense of beauty and order,” he wrote. “It was the tone of unknown powers awakened from sleep; of a heart and mind exalted with the feeling of what these powers could and would lead them to do.”

Here was a fresh Why for learning: the discovery of beauty external and powers internal. This educational ideal spread via Pestalozzi’s German acolyte Friedrich Fröbel, who equated school to nourishing a garden of children, so named it Kindergarten. John Dewey’s progressive movement followed, as did Steiner’s Waldorf education, and the Montessori schools. Gradually, the humanistic Enlightenment Why (flourishing) merged with the mechanistic Industrial Why (training), a sometimes-awkward marriage that persists today.

Into this tense union, the internet barged. Learning had never been more available; writing and speech boomed. Yet peculiar changes were afoot, with a range of human cognitive metrics falling from around 2012, arguably because the deluge of digital inputs overwhelmed our minds, turning humans into a race of scatterbrains. Book-reading plummeted, while teenagers’ performance in science, reading and math dropped across the industrialized world, with many reporting an inability to concentrate. Adults were scoring lower in numeracy and literacy too. Then chatbots arrived.

THE HOMEWORK APOCALYPSE

According to a recent survey, 92% of British undergraduates are using AI, with nearly all doing so for assignments. Videos circulate online showing how to conceal homework fraud with AI apps that “humanize” chatbot text and evade educators’ counterintelligence apps, which seek out dubious prose. It’s an arms race that leaves teachers in despair, not least because they’re losing.

The dilemma is this: Do you hold proudly to the educational values of before? Or decide that current testing is measuring little but the past? In recent generations, many kids who seemed maladapted because of tech obsession—early internet hackers, all-night gamers, social-media addicts—ended up succeeding in business, the arts, politics. Early adopters rule the world today. So maybe AI cheating is adaptive.

Moreover, the will to cheat one’s own learning is not the fault of tech. The American writer John Warner spent years posing the following hypothetical question to his college classes: If I offered you an ‘A’ but you had zero work to complete, no more classes, and could never tell anyone—would you go for it? In one course, the takers commonly exceeded 90%. That was back in 2013. AI may be satisfying a pent-up demand.

The postwar expansion of higher education made the degree a minimum admission ticket to most professions. When student numbers ballooned and tuition costs rose, the educational Why for many wasn’t learning but earning. Most students still want an education. But even the most earnest face a collective-action problem: If your peers are all generating impeccable AI assignments, you may be penalized for “humaning it.”

Educators cannot dump assessments, though. They need metrics to track teaching efficacy, to motivate, and to propel students into roles where they’d thrive. Ironically, technology is prompting a retreat to the past in some quarters, with oral examinations—which Oxford and Cambridge phased out starting in the 18^th century—now back in vogue.

The tricky part is that cheating and learning with AI may be proximate. While it’s dishonest to submit a chatbot essay as your own, is it wrong to pose your essay question to a chatbot, hear its insights, request that it pull up primary sources, assign its deep-research function to create a study guide, and pore over this, digging into the most-pertinent primary sources yourself, then posing follow-up questions to the AI? That process might be more instructive than battling with poorly written academic texts.

AI TO THE RESCUE?

A World Bank pilot program in Nigeria involving AI tutoring claimed gains so striking (two years’ learning in six weeks) that they seem unlikely to replicate. But another study, of an AI-powered math tutor in Ghana, also saw meaningful success, claiming learning benefits equivalent to an extra year of study in eight months.

Personalized AI tutors may offer timid students the chance to pose questions they’d be shy to ask before their peers, or that the overtaxed teacher might lack the time (or ability) to resolve. Educational AI also allows students to probe a source, seeking clarifications or sharpening their comprehension, perhaps even voice-chatting with a virtual Darwin. AI apps could track individuals’ learning in real-time rather than punctuating courses with tests, allowing educators to identify and respond dynamically to students’ progress. And data-tracking at scale could generate a finer conception of human learning than social-science methods have yet produced, such that machine-learning helps humans learn how humans learn.

AI might help with testing too. Oral examinations are far more time-consuming for human teachers than written tests, but voice AIs could quiz any number of students, evaluating their understanding, and issuing marks alongside audio transcripts, score rationales, and future-learning advice. Voice-assistants might even salvage the at-home essay, obliging their purported authors to discuss the contents, assessing students’ understanding of what they “wrote.”

Many teachers remain worried about where this is heading, envisioning classrooms where students gape at AI interfaces, barely interacting with their peers, let alone human teachers, who are kept around as behaviour police. Above all, educators suspect that developers misunderstand humans, with edtech companies building tools around what tech can do, not what students need.

Previous digital tools promised transformative effects too, professing that the internet would bring elite education to all, irrespective of where on Earth and who on earth they were. Yet many tech tools succeeded only for the tiny fraction of students who used them as prescribed (typically the top of the class anyway), a phenomenon known as “the 5 percent problem.”

AI can worsen learning too, as shown by research on German university students who learned coding with the help of chatbots. Those who sought AI explanations showed notable gains, but those who had the chatbot complete exercises undermined their own learning. More surprising is that AI might make us stupider while believing ourselves smarter: users tend to overestimate how much they’ve learned with AI, mistaking machine intelligence for their own.

An even more alarming prospect is cognitive atrophy, where users offload critical thinking and creativity to AI. If humans are able to pursue higher-order thinking by dumping intellectual drudgery onto machines, that would be fine. However, what constitutes higher-order thinking is debatable. A common pacifier is that humans collaborating with AI will be stronger than either alone: the “centaur model.” They said that of chess once. But these days, Magnus Carlsen would only slow down an AI grandmaster.

Technology is about making what’s hard easier. And learning is hard, even aversive. Yet the practice is not like doing the laundry, where we lose nothing by cramming filthy socks into a machine, and retrieving them clean. Learning is laborious or there is no learning. And labor is impossible to motivate if we don’t see its purpose.

This circles back to the question haunting this era: What are we for? The tautological response is that humans are for tasks that humans want other humans to do: You don’t fancy walking into a grief counsellor’s office only to find a robot. Yet algorithmic aversion—our preference for humans to make key judgments, even when algorithms perform better—seems likely to fade as AI becomes clearly superior, then commonplace. Until recently, sitting in a taxi that drove through San Francisco with nobody at the wheel would have seemed like a horror-movie scene. The real-life horror may be when no human is needed to drive anything.

LEARNING IN THE AGE OF MACHINE LEARNING

So what?

Machines can do our duties, feeding and clothing and cuddling us, leaving humans to sail yachts and paint watercolours. We’d never need to study again (but could for kicks). You hear such predictions, whose only flaw is the entirety of human history, a saga shaped by our cognitive hunger and the restless drive for comparative advantage. Artificial intelligence will do plenty, but not erase what evolution wrote.

Yet evolution itself might help explain our predicament, why technology satisfies our wants while producing effects we regret. Biological evolution set our inner clock, with neuronal firings like a cognitive speed-limit. Yet technological evolution keeps speeding up, the clock-hands spinning faster and faster, such that humans perceive only a blur now. We’re inundated with inputs at the speed of computational time, trying to keep up in biological time. And we’re going mad from it.

“As computational systems accelerate while biological rhythms remain stubbornly constant, humans face an insurmountable temporal divide,” Nicklas Berild Lundblad explains, setting forth his Bifurcation of Time theory. “We appear destined for increasing friction between silicon speed and cellular patience, with our institutions caught in the crossfire. But this conclusion misses something profound: the emergence of artificial intelligence as a temporal mediator.”

Already, large-language models are doing this, digesting the (humanly) indigestible immensity of data, and converting it into chatbot responses intelligible at the pace of human cognition. AI will keep evolving, becoming our extrasensory sensors and interpreters of the world while remaining fluent in human time. This points to a future role for humans, where action is not our highest calling; judgment is.

“In the judgment economy, value derives from qualities that resist technological acceleration—discernment, wisdom, creativity, and ethical reasoning,” Lundblad writes. “These capabilities aren’t necessarily improved by moving faster; often they benefit from deliberate slowness.”

Humanity can’t compete on the factory floor, so walks upstairs to management, setting the objectives, designing human-AI labour relations, evaluating the outcomes. This becomes a new Why for education: to develop the human discernment and ethical reasoning to govern our new powers rather than letting them govern us.

But how to teach that?

SPECULATIVE IDEAS

The Great Dejection—a worsening mood over human cognitive recession during the AI boom—brings a risk: passivity. If people see little worth in educating themselves, perhaps they stop bothering, surrendering further competence to machines, and forfeiting human decision-making, much as the AI-safety movement long feared. Therefore, education policy should target motivation, seeking to drive learning across the lifespan.

Here are three possible approaches, taking inspiration from self-determination theory, which identifies three psychological needs that motivate us: autonomy (feeling in control of one’s actions); competence (feeling capable and effective), and relatedness (feeling connected to others):

1. Choose Your Own Adventure. (Autonomy)

Learning should be reframed as a personal asset, earned through self-directed R&D. To initiate this, schooling could set “choose-your-own-adventure” hours, during which even the youngest pupils embark on personal enterprises in any area of interest, including the unacademic. The only condition would be that the student pursues an adventure—that is, adds to their knowledge and skills.

Pupils could use AI to help brainstorm the steps of the adventure plan, allowing them to stay in charge without the adult supervision that can puncture motivation. Nor should teachers mark the adventures. Rather, pupils mark the AI, assessing how well it helps them achieve their stated goals, and noting where the objectives fell short, which would grant the pupil insight into managing AI collaborations in the future. The adventure app should gather insights on each child’s learning strategies and efficacy too, providing feedback to help them learn how they learn.

Behavioural AI might help address student wellbeing here, employing data-analytics to detect which activities worsen a particular child’s determination and mindset, then adapting recommender systems to discourage these. That could also help resolve an enduring frustration: that years into the decline of children’s mental health and test scores, we still dispute the causes.

Students who relish autonomy might unlock more, while those who need (or prefer) closer guidance could select a more directed curriculum. As for teachers, they could upgrade from information-crammers to adventure-mentors, focusing on the meaningful part of their jobs: figuring out each learner, and inspiring them. In higher education, the “choose-your-own-adventure” approach could transition into a fully customizable degree: nothing but electives.

2. Multiply Your Talents. (Competence)

Lifelong occupations commonly spring from absurd factors such as location, wealth, and fluke—all mediated by the decision-making of adolescents whose prefrontal cortices have yet to mature. By this haphazard process, competencies (or their absence) become the boundaries of one’s life. Even when workers could expect a relatively predictable employment future, this career process was often cruel and foolish. Now that predictable work paths are dissolving, we may have a chance—and a need—to make credentialing more adaptive.

Micro-credentials could become source-agnostic, available outside traditional institutions, and open to all ages, avoiding the cost and stigma of returning to school after one’s typical schooling years. However, source-agnostic credentialing should take care not to undermine the institutions of higher education, which still produce inspiring learning communities, and spark interdisciplinary creativity (not to mention the plethora of non-educational benefits, such as varied social exposure and friendships).

The system should resist the tech-era tendency to worsen isolation. This could be done by requiring group work, either virtually or locally, for micro-credentialing. Also, human tutors could supplement AI education, allowing students to discuss and digest their learning with a person.

AI might help individuals choose their credits through data-analytics of public information such as job ads, economic indicators, and other correlates of future demand, permitting users to align their learning with purpose. Intelligent systems could also connect a person’s existing skill/knowledge assets with others’ needs, whether professional or not.

Micro-credentials might even be assigned to learning experiences such as extended foreign travel or volunteer work, or could be awarded for one’s decades alive, such that competence is less a framed diploma from youth than a photo roll of accumulating wisdom.

3. The Wisdom Exchange. (Relatedness)

Every person amasses a repository of wisdom over the course of life, but much of it goes untapped. AI could help build social platforms for wisdom-sharing, linking human experts to human learners, perhaps with a serendipity option for those wanting randomized knowledge discovery.

Wisdom exchanges should be voluntary, serving both sides of the equation: instruction for one, the gratification of purposeful assistance for the other. Each exchange could include a reciprocal element, meaning the expert turns the table on the learner at the end—say, a retired politician at first supplying insights to a young activist, then asking for a lesson on bewildering emojis. To encourage respectful interactions, the system should include reputation ratings, as with taxi apps.

Wisdom-exchanges could also employ AI for learning across borders, conducting real-time language interpretation between participants. A responsive system could also present relevant context during interactions, and offer post-exchange fact-checking and takeaways.

Hofstadter’s Law: "It always takes longer than you expect, even when you take into account Hofstadter's Law."

In this piece, I want to untangle several threads that often get mixed up in AGI discussions: the distinction between capabilities and deployments, the relationship between technical progress and real-world impact, and how humans maintain meaningful control as systems become increasingly sophisticated. While the technical path to AGI is important, I'm equally interested in what happens afterward - my central concern is how we preserve agency not by understanding every cog in increasingly complex systems, but by designing the right level of abstraction for human oversight.

The gaps between capability, deployment and impact

Based on current scaling trends and algorithmic progress, I think it’s likely that we will reach ~AGI capabilities before 2030. By AGI I'm specifically referring to systems that achieve something like ‘Expert AGI’ as defined in the Levels of AGI framework - that is, AI that performs at least at the top percentile of skilled adults across a wide range of non-physical cognitive and metacognitive tasks. While physical capabilities through robotics are advancing and crucial for full real-world impact, my focus here is primarily on the cognitive capabilities enabling reasoning, learning, and problem-solving across domains with minimal human oversight.

Crucially, however, achieving these capabilities is distinct from deploying them in ways that yield truly transformative changes. This distinction between capabilities (what a system can do, often demonstrated in controlled evaluation settings) and deployments (how systems are integrated into real-world, value-producing applications) is central. Deployments are much harder to model and predict; notice how most online forecasting rarely focuses on concrete products or use cases. Simulated environments help gauge capabilities, but designing (and adopting) useful agents and products is another challenge entirely.

So while capabilities may arrive relatively soon, I expect a lag before we see widespread transformative impact from them. Intuitively, it might seem that AGI deployment would be relatively quick - we are already using pre-AGI systems, and so we have existing infrastructure to draw from. However, this underestimates real-world frictions, such as the slow grind of human and corporate inefficiency and the difficulty in achieving product-market fit for truly useful applications. Progress on specific, often academic benchmarks can sometimes create an illusion of proximity to broad usability.

Furthermore, even as underlying capabilities advance, progress might feel slow during certain periods. There are several reasons for this potential perception gap. As Joshua Achiam observes, AI may improve significantly on complex specialized tasks that are irrelevant to most people, "creating an illusion that progress is standing still.” Additionally, the advanced capabilities of a pre-AGI model or agent might initially go unnoticed in everyday interactions, as most current chatbot queries are basic. The capabilities might exist but remain underexploited, much like the months-long effort required by engineers to fully 'extract' value from a newly trained model.

Still, some strategically significant changes might arise even with limited deployment. A few actors using these capabilities effectively could gain a considerable head start – think of the decisive strategic edge gained by codebreakers during wartime, where a capability understood and wielded by only a select few dramatically altered outcomes.

From today to AGI

On the R&D side, training a capable model is only the first step. Ensuring it's fit for purpose demands specialised data and meticulous post-training/fine-tuning. Building a truly useful and effective agent - or a multi-agent system with tailored roles and a robust pipeline - requires significant effort and development. These exercises take time and require many iterations.

High-quality synthetic data is not easy to generate. And when you finally run these systems, there’s still a lot of trial-and-error involved to achieve the desired output. If the output is data, you still need to evaluate, process, and use it - which is labour-intensive. If the output consists of actions, you need to verify and check results - same thing. This requires new tools and frameworks that also need to be created, iterated upon, and perfected simultaneously. The systems and products we build on top of these models will also be complex and unwieldy. The more I unpack and examine how models are trained and improved, the more I understand Hofstadter’s Law.

But while training a capable model is a significant hurdle, the need for specialised data and extensive fine-tuning will likely diminish as models become more advanced. More intelligent models will also be better able to work around imperfect tools and suggest improvements, reducing the need for painstaking manual optimisation. This is the benefit of generality and computational power.

We see this across the board: larger models tend to perform better out of the box than smaller, specialised models. For example, Bloomberg trained a GPT-3.5 class model on their financial data last year, but GPT-4-8k soon outperformed it on nearly all finance tasks. While people sometimes make hyperbolic claims about this, I do think that it’s true - it’s certainly been my experience. The Bitter Lesson remains as true as ever.

Perhaps this is because large models effectively bundle many specialized capabilities, but unlocking or optimizing them for specific tasks often requires dedicated fine-tuning or the creation of smaller, derived models for efficiency and convenience. The learnings from this specialization process, however, frequently inform the architecture and training of subsequent, even more capable general models, continuing the cycle.

The larger the model (and the more computation used for training), the greater the potential relative gain from its self-improvement capabilities. As systems improve and demonstrate reliability, and as AI tools increasingly assist in the validation process itself, the level of scrutiny required during their development will likely decrease. Over time, we can expect many, if not most, parts of the R&D pipeline to become automated.

For end-users, businesses, different sectors, and consumers, the calculus is different. Once products offer a sufficient degree of convenience, utility, and reliability, widespread adoption will likely occur rapidly. Researchers and academics may continue to meticulously scrutinise these systems, but the majority of users will prioritise ease of use and perceived benefits over deep technical understanding - accepting AI assistance with minimal oversight, much like following an online maps route without a second thought.

However, finding the right ‘product fit’ is difficult. This is often overlooked by those working on R&D, but it is critical for ensuring high adoption and diffusion. These elements are more about engineering, operational, contextual, and commercial complexity - as well as user experience - than raw model capabilities, which scaling trends don’t typically account for. To be clear, there’s nothing inherently impossible or intractable here, but it probably adds a couple of years or more to my timelines.

We'll likely see the gradual deployment of better models and more capable systems, with improved integration within the information ecosystem every month and year. Models will continue to improve, taking on more tasks in the economy. By the time we reach true AGI, much of the groundwork for deployment will likely have already been done.

In a sense, today's environment lacks the mature techno-organizational ecosystem required to smoothly ‘drop in’ AGI workers at scale, in a way that can account for the tacit knowledge that underpins many workflows. But as AI capabilities progress, this ecosystem is likely co-evolving, meaning the pathways for effective integration – whether through internal change or external disruption like AI-first startups – may be significantly clearer by the time ~AGI capabilities arrive. After that, the challenge will be twofold:

1. Continuing to improve models and deploying them systems at scale.

2. Achieving repeatable benefits and continuing this cycle on loop.

The big questions are: How much time will this take? Is the curve exponential? And what about the physical infrastructure and hardware underpinning AI systems?

So what about post-AGI?

A lot of the above could, over time, be automated too - but the process of automating itself requires extensive iteration. This involves essentially repeating all the steps mentioned above, but multiple times, at different layers of abstraction. Every time you ascend one part of the abstraction ladder, new tasks, actions, and options are created and must be completed for progress to continue. Both executing and subsequently automating all of this requires specialised work and is unlikely to be solved in mere days, even with the assistance of multiple pre-existing AI systems operating in parallel. Put differently, each layer of abstraction doesn't just encapsulate the complexity below it but generates new types of complexity that must be managed. And all of this assumes we have reached this level with minimal societal pushback, which I don’t expect (e.g. strikes, protectionism, legally mandated human roles, etc.).

Setting aside the sociopolitical elements for a moment, managing this emergent complexity will necessitate substantial compute resources for experimentation, training and inference, leading to a faster escalation in energy demand. This rising demand could eventually drive a shift in how we produce compute substrates. As Epoch AI highlights, current chip manufacturing paradigms may not scale beyond 2030 (for training) due to power and manufacturing constraints. This may necessitate a transition to quantum computing or more unconventional approaches. This alone implies a degree of slowing down, relative to what one would expect from a purely linear extrapolation.

Progress will depend on optimising both the algorithmic and hardware layers, potentially shifting focus from software innovations to material science and advanced manufacturing. For example, building cutting-edge zinc factories to produce better lithography machines for more efficient chips is a highly complex endeavour. Even if millions of AGI scientists assist in this effort, financing and constructing adequate and safe physical labs will still require considerable time.

The pace of hardware improvements is likely to be constrained not only by hardware limitations but, maybe, also things like the transfer of tacit human knowledge and tense geopolitical dynamics, rather than algorithmic complexity alone. As we exhaust the potential for algorithmic optimisation on existing hardware, progress will increasingly depend on advancements in chip design and manufacturing infrastructure. This may involve building specialised factories or developing novel material processing techniques, areas where AI can help but where human expertise and tacit knowledge will likely remain essential for some time.

How much time do these implementation challenges add to the journey from initial AGI capabilities to transformative economic impact? Does ~AGI materially change things?

What makes me lean towards very fast:

• Parallel processes can stack multiplicatively with agents who work 24/7 - see also Price’s Law.

• Geopolitical competition will accelerate investment, as well as massive capital mobilization and strategic concentration of compute resources.

• As Daniel Kokotaljo writes, automation of ML R&D will lead not just to faster iteration but also enable qualitative algorithmic breakthroughs that unlock step-changes in capability.

• Larger models will continue to improve and outperform smaller, specialised models.

• Smaller, more efficient models will continue to improve, eventually reaching the capabilities of the previous generation of large models.

• We’ve only just begun exploring inference scaling, with ample room for further impressive capabilities.

• There is potential for rapid, widespread automation across the broader economy, leveraging AI for ordinary cognitive and physical tasks.

• Privacy-preserving monitoring and verification technologies could greatly facilitate governance.

• Robots are becoming increasingly capable and dexterous, a trend that is likely to continue.

• AGIs could contribute to energy R&D, creating a positive feedback loop, with energy efficiency of compute improving simultaneously.

What makes me lean towards not that fast:

• As we solve one bottleneck, new, previously unappreciated ones become salient.

• "Easy for humans, hard for AI" tasks are arguably more critical for widespread automation than superhuman performance on narrow cognitive benchmarks.

• As Ege Erdil notes, there isn't a clear, rapidly improving trendline for agency or common sense that can be confidently extrapolated; and transfer learning remains tricky.

• Integration/knowledge challenges may multiply, and verification requirements could grow over time.

• Poorly designed regulations, protectionism, or bureaucratic inefficiencies could hinder progress.

• The difficulty and cost of acquiring the right kind of data to train capabilities in areas like agency, planning, and real-world interaction.

• Legacy infrastructure lock-in: even well-capitalised firms are built on pre-AGI assumptions, and rebuilding physical and organisational systems takes time regardless of capabilities.

• Inherent physical limits could restrict speed and slow compute scaling.

• Military and geopolitical conflicts, economic crises, weak economies etc.

• AI R&D automation alone may not be sufficient for sudden, rapid acceleration; achieving that depends on other feedback loops and Baumol-like bottlenecks.

Managing post-AGI economies and automated production-processes will ultimately require both decisions and time. Wolfram argues that even with highly automated AI systems, strategic choices must still be made about which paths to explore in the computational universe, noting that "something—or someone—will have to make a choice of which ones to take." I think this is a crucial point, relevant to both agency, control, and speed.

Who does the work?

What if the 'something' making choices for everything is AIs with zero humans involved? In that case, humans are arguably no longer in the loop, and things could change much faster than we can process, understand, or control. This is the real ‘automation gone wild’ risk - it’s not just about automating away machine learning R&D, but automating everything. Even without any sudden technological discontinuity or overtly hostile AI actions, our economy, culture, and political institutions could gradually drift away from human influence as they become less dependent on human involvement to function.

I don’t think we will sleepwalk into this, but I do think there are adjacent risks worth considering.

If the 'someone' making choices is humans, then our cognitive speed and limitations will serve as a ceiling, restricting the rate of progress. In this case, humans would constitute a slow-but-necessary component within an ecosystem where different types of thinking or processing occur at different speeds. This doesn’t seem sustainable in the long run; even today, we often sacrifice understanding for efficiency. For example, there’s little point in using an automated cancer detection tool if you intend to manually verify each image in parallel. At some point, the system performs well enough to be trusted. On the other hand, we still have largely decorative ‘drivers’ for the UK tube system, even though these systems are fully and reliably automatable.

The most likely and sustainable allocation of tasks will likely involve both humans and AIs making decisions. Achieving this will require rules, verification systems, and trusted-based mechanisms to maintain a stable and peaceful coexistence - not only between different ASI powers or nations, but eventually also between humans and ASIs. Some tasks we will (hopefully) be content to automate entirely: just as we were content automating bank tellers in the past or call centres today, tomorrow we may be comfortable automating processes at a higher level of abstraction, such as ‘public transport’.

The difficulty is designing systems that can effectively coordinate fast, post-AGI-driven processes with slower human strategic oversight. In such a world, we will need what you might call "cognitive impedance matching" - systems that can translate between AI and human timescales while maintaining stability. These intermediaries could be agents, but other form factors and interfaces may be preferable in some cases. Imagine an ASI managing a complex global supply chain in real-time, making millisecond adjustments; the impedance matching system might present human overseers with curated summaries of network health, predictive alerts about potential disruptions days or weeks in advance, and simplified interfaces to approve strategic shifts (like prioritizing a specific region) without needing to track every individual shipment.

This human oversight is less like a technical checkup, which another ASI might perform, and more akin to setting the destination and preferred route on a navigation system. While another AI could monitor the engine's performance, the human role is to ensure the complex machinery is ultimately serving human-defined goals and navigating according to human-held values and priorities. It's about steering the 'what' and 'why', even as the ASI optimizes the 'how'.

Nicklas Lundblad explains that AI can serve as a "bridge" between computational time and biological time. This is important because many processes - such as justice, democracy, and relationships - lose their meaning or utility when rushed. Down the line, cognitive enhancement for humans could help. But this will be the central 'dynamic' to manage, and it’s another important facet of what ‘alignment’ is fundamentally about. Perhaps a different term is more appropriate, but the goal remains the same: ensuring the systems you design do what you intend, safely.

Michael Levin emphasizes the diversity of intelligence and embodiment, acknowledging that different agents operate on different timescales. Just as biological systems - such as the interplay between fast neuronal firing and slower muscle contractions - achieve coordination across different timescales, we should be able to design AGI systems that translate between human and post-AGI speeds, enabling meaningful collaboration.

What about understanding?

One way I could see this process going wrong is if AGI systems become increasingly more deeply embedded in decision-making processes across all levels, gradually shifting the human role from active direction to passive approval. Practical concerns about sentience, rights, and sophonts could further complicate these dynamics. Even with well-designed oversight mechanisms, the speed and complexity of AI-driven processes might make genuine human understanding and intervention increasingly difficult. This could lead to emergent system-level behaviours that no individual component was explicitly designed to produce, resulting in a gradual erosion of human agency and understanding. As Kulveit et al. detail in their 'Gradual Disempowerment' paper, this erosion could occur incrementally through the progressive replacement of human labour, cognition, and participation across interconnected societal systems.

The "cognitive impedance (mis)match" discussed earlier only worsens as AI systems become more capable. Even with the best interfaces and coordination mechanisms, humans may increasingly be forced to either: (1) trust the AI systems blindly because we can't verify their logic and complex reasoning in real-time; (2) slow everything down to human speed, creating massive inefficiencies; or (3) remove ourselves from more and more decisions. However, whether this scenario leads to a true loss of agency hinges critically on our ability to design and implement effective high-level oversight mechanisms built upon the right abstractions.

Is it inevitable, then, that we lose track of reality and choice?

Maybe not. Perhaps the challenge of understanding is already accounted for by the idea of climbing the ladder of abstraction. In principle, we could continue shifting to higher levels of abstraction in our oversight - just as executives don't need to understand every line of code to run a tech company, and I don’t need to understand how a fire alarm works to install one. We'll focus on what we want the AGIs to achieve, not necessarily how they achieve it (though nothing, apart from time, prevents us from unpacking the why if needed).

Consider a Minister of Health, who, faced with a new flu strain, strategically prioritizes vaccinations for the elderly based on epidemiological models, without needing to grasp the intricate molecular biology of the virus or the logistical complexities of vaccine production. This is a crucial, high-level decision, reliant on abstracted information and expert advice to achieve positive public health outcomes. Similarly, much like we see in other spheres of life, decision-makers can - and often must - make choices based on outcomes without intimate knowledge of all the underlying processes. The same applies to steering superintelligent multi-agent systems.

To illustrate the intuition behind this, imagine a similar line of thinking in a village 2,000 years ago: “In the village, everyone understands how everything works - we know who makes our tools, who grows our food, how decisions are made at the village council. But in these new cities, everything is interconnected in ways no one person can fully grasp. People don't even know who bakes their bread! And with all these written contracts and money changing hands, decisions that affect us all are being made through processes we barely understand and have no time to unpack.”

The nuance here is that, while deep understanding is often useful, its necessity is built partly on our goals and the level of assurance required. Today, I don't know how to build a plough, because there are more effective processes (markets) that can provide me with one. However, I do need to gain experience working in policy to achieve my goals within my current environment. These goals will change and evolve, and so will the types of knowledge or understanding we need and want to internalise.

The key difference is that no single human may be able to understand all the building blocks; but ideally, these should be well-documented, maintained, and scrutinizable if needed - by different kinds of specialized agents. Building a new Library of Alexandria of knowledge seems like a worthwhile endeavour.

Biological systems also demonstrate that effective high-level functioning doesn't necessarily require low-level understanding. My body maintains homeostasis without my conscious mind understanding cellular biology. Similarly, I can effectively use my arms without understanding individual muscle fibre mechanics, or a computer mouse without knowing its hardware intricacies.

From understanding to steering

Beyond understanding, humans can also retain agency over governance more generally. Scenarios of gradual disempowerment are compelling when entire systems are outsourced to identical copies of AI agents that don't represent anyone's direct interests. The 'human interest' feedback loop is weak in such a world. Instead, I think that every human should ideally have a personalized agent that learns and represents their evolving values and preferences. These agents, tightly linked to their human principals and acting on their behalf, would create a continuous feedback loop between individuals and large-scale automated systems, preventing system-level value drift.

So you don't necessarily need humans in the loop; you need aggregate human interests in the loop. Provided egregious misalignment is avoided, automated systems should be directly informed by, and in some sense exist downstream of, human preferences - preventing the gradual drift toward disempowerment one can expect with monolithic, unaccountable AI structures. Of course, you still face challenges like trade-offs, aggregation issues, and conflicting values. But these are not insurmountable and ultimately call for updating our democratic machinery (or building something better) to accommodate diverse human interests. The difficult part will be replacing our existing decaying institutions and navigating entrenched human interests.

And just as complexity necessitated new forms of social organization and market mechanisms, the speed and scale of ASI-driven processes will likely require automating significant parts of governance itself. Designing these automated governance systems, ensuring they operate effectively, adaptably, and remain aligned with human values and strategic direction, becomes a crucial meta-level challenge for maintaining agency in this future

This is not necessarily an erosion of agency, but a shift in how and where it is exercised. The real challenge isn't maintaining low-level understanding, but rather designing the right abstractions that capture what we truly care about and ensuring these abstractions remain responsive to evolving human values while preserving meaningful oversight as systems grow increasingly complex. It’s not easy to pre-design or pre-specify these, and I think working all this out will require significant human input for longer than is sometimes assumed.

Amara’s Law: “We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.”

Overused, but useful.

—---------

Thanks to the following people for comments: Shane Legg, Ben Lepine, Nick Swanson, Harry Law, Pegah Maham, Zhengdong Wang, Conor Griffin, Tim Genewein, Herbie Bradley, Samuel Albanie, David Wolinsky, and Luke Drago. It goes without saying that they don’t necessarily endorse my ramblings.

“Shaping” human behaviour sounds sinister, stirring the horror of losing control over oneself, a fear almost akin to being consumed alive. From early childhood, we long to take charge of our actions, then gain that power in adulthood, marshalling it haphazardly for the decades that follow, and surrendering autonomy only if threatened by violence or debility.

But now and then, we are willing to toggle off full agency. Alcohol is a common tool for this. Or consider Ozempic and other semaglutides that millions of people eagerly take, targeting GLP-1 receptors in the brain that mimic hormones of satiation, shortcutting wants to deliberately modify their own behavioural responses. Those who dose themselves with Ozempic don’t feel disempowered. They frame agency as the high-level decision to alter behaviours they struggle to modify autonomously.

So the horror of losing self-control is not absolute. Rather, it is unwitting behavioural change that stirs anxiety. The history of new technologies is also a history of this anxiety, with each successful device blamed for seducing the masses from behavioural norms, whether it was the transistor radio, the video game, or the smartphone. Such panics are always correct: if the technology works, it will change users’ behaviour. But society habituates soon enough, and the next generation smirks at the fears of the last.

Not all innovations are equally impactful, though. Daily, futurists prophesy that artificial intelligence will transform our world, becoming our newest general-purpose technology. But unlike electricity or the steam engine, AI’s “fuel” is data, the traces of what our species has written, spoken, done and recorded, iteratively refined by human feedback. Humanity feeds these tools, and is embedded within them, making the behavioural impacts more direct, more potent. If machines operate us, who—or what—is in control? “Man himself has been added to the objects of technology,” the philosopher Hans Jonas observed back in 1979, remarking that this “may well portend the overpowering of man.”

The first flares of public unease over AI’s behavioural impact concerned social media, which many came to see as an algorithmic Svengali, engineering the polarization of politics, the degradation of culture, the fraying of norms. Debate persists over how blameworthy social media was, but nobody disputes that future AI systems will do far more than recommend cute videos and infuriating posts. These systems will insert themselves everywhere from the labour market to our bedrooms. We will invite them in.

Indeed, a key point of AI is human change, to better us by expanding our cognition. Evolution granted us an exquisite system of thinking, but it has embedded limits, and we cannot update nature at the pace of tech advancement. In particular, scale and complexity overwhelm us. But AI promises to become a form of pre-wisdom, assimilating more than we could ever sift through, incorporating sensory signals beyond our capacity, offering counsel we would never have known to ask for. Stanislaw Lem foresaw this in 1964, predicting that humanity would resist at first. But intelligent machines would process data so comprehensively that “intelectronics,” as he called it, would prove far wiser in selecting next steps. “After several painful lessons, humanity could turn into a well-behaved child, always ready to listen to [the machine’s] good advice,” Lem wrote.

We recoil at this infantilization, which triggers that dread of losing agency. The challenge is this: behavioural influence will happen; there is no opt-out. The question is whether we respond pragmatically, attempting behavioural audits of AI systems, incorporating workable controls, and infusing design with ethics. Otherwise, our rightful dread at what might someday befall humankind becomes a truth-denying passivity that decides matters for us. The choice may be this: shape your behaviour, or have it shaped for you.

OF TWO MINDS

Behavioural science is a field of optimistic pessimism. At once, it declares the human mind a blunderer, yet insists that the human mind can amend this.

Kahneman, a self-identified pessimist, described many cognitive biases on which the field is based, yet remarked that he still had little power to resist them. The science’s optimistic face is embodied in its other Nobel laureate, Richard Thaler, who considered the same constraints on human thinking, and proposed an answer: nudge. His book of that name, written with Cass Sunstein, enjoyed a timely publication, released around the nadir of the Global Financial Crisis in 2008, when rational actors had behaved in ways that seemed irrational. Policymakers, after stabilizing the markets, cast around for fixes. Among the most appealing was this promise of a low-cost brain toolkit: the nudge.

The key concept was “choice architecture” - the idea that small contextual changes may have large effects on how people behave. So, you arrange fruit before the desserts in a cafeteria, and diners—perfectly free to bypass the apples for the cake—are more likely to act in their better dietary interests. To avert charges of manipulation, the two authors offered the ethical underpinning of “libertarian paternalism,” that a policy could justifiably funnel individuals towards beneficial actions, provided that they retained the power to opt-out.

“The presumption that individual choices should be free from interference is usually based on the assumption that people do a good job of making choices, or at least that they do a far better job than third parties could do. As far as we can tell, there is little empirical support for this claim,” Thaler and Sunstein wrote. “People do not exhibit rational expectations, fail to make forecasts that are consistent with Bayes’ rule, use heuristics that lead them to make systematic blunders, exhibit preference reversals (that is, they prefer A to B and B to A) and make different choices depending on the wording of the problem.”

Over the years, researchers cited a plethora of cognitive biases, from anchoring effects, to intertemporal inconsistency, to present bias. Behavioural interventions included changing defaults (for instance, automatically enrolling drivers in an undersubscribed organ-donation plan); or adding commitment devices (compulsive gamblers signing up to self-exclusion lists at the casino); or encouraging social accountability (advising people to go to the gym with a friend to increase visits). But the field’s optimism was not always rewarded. A meta-analysis of 200 studies (n=2,148,439) showed that “choice architecture” had small to medium effect-sizes, with consistently lesser impact from interventions that demanded people commit to change, or relied on them absorbing new information. The largest effects seemed to come when participants didn’t exert themselves, but merely had their actions tweaked through changed defaults. Another study looked at governmental “nudge units” that undertook behavioural interventions, comparing their outcomes to those cited in academic papers. In journals, the average impact of a nudge seemed impressive: 33.5% increase over the control. In real-world interventions, results were far weaker: an 8.1% increase. The main culprit seemed to be publication bias, with journals tending to only print studies that claim significant effects, while interventions that did little vanish from the academic record.1

A further challenge is that populations are heterogenous. So graphic warnings on high-calorie drinks might discourage consumers who already had good self-control, but do little for those struggling to curb their appetites. Indeed, many just develop an aversion to graphic warnings, not to the drinks. That suggests that behavioural interventions might work better if personalized to individual preferences. But the value proposition of nudging for policy was that it promised benefits at scale.

As behavioural science grappled with these disappointments (or tried to ignore them), large-scale behavioural change was taking place all around. The cause was technology. Academics struggled to harness these possibilities for science, a few employing apps to gather individualized data, or testing wearable sensors, or just-in-time adaptive interventions (JITAIs) that prompted participants with personalized reminders and recommendations. Meantime, behavioural consultants worked with tech companies, and certain insights seeped into products. Behavioural designs sought stickiness, to give customers something to return for, and to stay with: just what they wanted.

Yet the proxies for human desire—for instance, click-throughs and engagement time—aligned with short-term wants. Long-term human objectives had few metrics, and this had a peculiar effect: many people found themselves doing what they wished not to do. Terms like “doomscrolling” and “brain rot” emerged. That primal dread of humans losing self-control surfaced. Public intellectuals fretted about whether free will even exists, while notable behavioural scientists—admitting the limits of their past interventions—questioned whether their past focus on the “i-frame” (getting individuals to change) had deflected attention from the “s-frame” (how systems change behaviour).

Part of the problem is that our species has built-in systems of our own, inscribed by evolutionary pressures over millions of years, and resistant to alteration. Kahneman explained this with his model of dual cognitive tracks, System 1 thinking (fast and effortless) and System 2 (slow and reflective). Sometimes, the brain errs by employing System 1 shortcuts when System 2 could more effectively reason through matters.2 These two systems—rather than exposing humankind as a race of dunces—are effective in most cases; otherwise, natural selection would not have left us with them. Nevertheless, human cognition does have vulnerabilities, and algorithmic intelligence homed in on many, as when feeding our hunger for effortless pleasures now that subvert our deeper longings later.

Among the outcomes was what the psychiatrist Anna Lembke calls “the plenty paradox,” that the abundance of contemporary life floods our evolved reward pathways in ways that stresses us, perhaps explaining the rise in depression and anxiety, which is most acute in wealthy countries that also have greatest access to the people-pleasing offerings of new tech. “One would hope and think that we’d be engaging in deep philosophical discussions, helping each other, cleaning up the garbage,” Lembke says, of our era of plenty. “But instead what we’re doing is spending a whole lot of time masturbating, shopping, and watching other people do things online.”

SQUINTING INTO THAT FOGGY FUTURE

You wake with a jolt, your mouth parched, your T-shirt damp with sweat. Somehow, you’re queasy but famished too. Mostly, you’re annoyed. Not just for the excesses of last night, but for how you’ve acted lately: self-centred, distracted, unproductive. You rouse your AI ecosystem, and open its settings, selecting “My Future,” where the blizzard of daily life is simplified into sliders, using probabilistic analyses rooted in the heaped datapoints of humankind, cross-referenced to your personal behavioural history, modified by your current physiological inputs, and enriched with environmental sensors presenting perceptual insights beyond the capacity of any living being. You adjust the “Objectives” slider, shifting your behavioural preference from “Short-term pleasures” towards “Long-term goals,” and set the timeframe as “This Week.” Immediately, the system reconfigures your AR overlay of information stream and notifications, tweaking your mood-responsive assistant, and altering the haptic-nudge schedule on your wearables, all with the intent of curtailing the defeating habits you struggle to repress while promoting those behaviours you long to manifest.

Would this curb your autonomy? Or enhance it?

Humanity’s dilemma between short-term wants and long-term objectives is a conflict that has provided drama enough for most of the novels, plays and ballads ever written. It’s an inner contest that also infuses today’s primitive recommender systems, where algorithms overwhelmingly feed us what the philosopher Harry Frankfurt called first-order desires (I like watching video clips) at the expense of second-order desires (I wish I just wanted to read a book).

One way of understanding this is as a time-horizon mismatch between desire now or contentment later. Most people agree that a good life includes both fleeting pleasures and the accumulation of achievements. We simply disagree on the proportions. Conceivably, a personalized AI ecosystem could allow users to express their preferred balance, altering behavioural inputs accordingly. Users might also pursue betterment of specific character traits—say, moving a “Resilience” slider to challenge existing views and action patterns, much as one programs higher resistance on a treadmill to build endurance. Chatbot responses could incorporate random variation too, emulating the stimulating unpredictability of other people, rather than making us captive to confirmatory feedback loops. Otherwise, AI companions—already in deep relationships with humans that include sex and “marriage”—could become behavioural anchors, a constant companion for your whole life, who holds you to your 20-year-old character when you’d otherwise have evolved into a 50-year-old.

To avert behavioural stagnation, AI companions might be made mortal, but that could inflict terrible suffering on their human partners. A less-brutal approach would be programmed forgetfulness, so that AI systems remove facets of your past from their data, much as people superimpose your current character upon past versions, which become fainter as years pass. Likewise, AI companions could alter behaviourally over time, perhaps even growing apart from you. As most humans have a strong drive to hoard artifacts of their past, AI systems might store their comprehensive memories of you apart from the working behavioural datapoints, so that its agents and recommender systems operate according to your current self, while an archive sits in abeyance, like a nostalgia vault you may access if ever longing to revisit who you were. You might even choose to venture back among those old settings, experiencing the behavioural filters you once so ardently sought, presenting a parallax view of your life’s course, from then to now.

At set intervals, your AI ecosystem could present a review of your behavioural trends, and solicit your updated preferences, perhaps as part of New Year’s Resolutions. On each such occasion, the system could offer advice based on the latest empirical findings on human wellbeing. Today’s evidence on wellbeing still stirs debate, with mixed answers to questions like whether money can buy happiness, and whether current wellbeing measures are reliable enough to make policy. Influential studies may have queried just a few hundred people in an artificial environment. AI behavioural studies could involve sample sizes in the billions, amassing the most granular set of behavioural insights ever attempted - computing ever-larger datasets based on our (anonymized) responses and wants. This could produce a sturdier science of “choice architecture” and the conditions that amplify or stifle effects, how to design and avoid feedback loops and how individual characteristics moderate species-level tendencies. Such findings could amount to our greatest clarity yet on “the good life,” and whether that is even what people care for, given how often their actions commonly subvert their stated wishes. We might learn too whether behavioural interventions really change much, or if each of us is destined to hover around a baseline nature, hardly corrigible no matter how many self-help books we pile on the bedstand, or how many “My Future” adjustments we make to our AI-ecosystem settings.

Troubling outcomes are possible too. AI might discern behavioural correlates within datasets, and engineer unintended changes in human culture. Imagine if datapoints revealed that listening to certain music was associated with higher rates of depression, causing well-intended recommender systems to steer everyone away from ever hearing such songs, becoming the gatekeepers of human art. Or what if AI ecosystems sensibly diverted people from toxic relationships? Anyone would be relieved to dodge a disastrous marriage, but what if the machines judged you toxic, and people you knew started treating you like a pariah? Those who failed to boost others’ metrics could become data outcasts, banished from good society by AI. Arguably, this could disincentivize awful behaviour, pressuring the nasty to amend their ways. Or maybe it just punishes the eccentric.

Data protections would be imperative, ensuring that nobody is behaviourally hacked, whether by a bad actor, or by an institution to mobilize the citizenry to their preferred ends. However, people might accept external influences on their behavioural AI—say, allowing a spouse to adjust your settings in exchange for avoiding divorce. Or a healthcare provider might offer discounts to clients who alter their AI settings to optimize for determinants of lower medical costs, such as exercise, socializing, and healthy diets. Parole boards could make release conditional on AI behavioural diversions, and governments could incentivize pro-social behaviour by offering tax credits to those who optimize for charitable activities.

But any external influence tagged to one’s behavioural data is fraught with dangers, easily degenerating into a social-credit system that coerces conformity and further penalizes those who already struggle with immiserating behavioural tendencies. A further risk is that incentives for “good behaviour” undermine intrinsic motivation, much as a child who is paid each time she acts politely to an elderly person might develop a distorted practice of courtesy, withholding kindness from granny until someone hands over cash.

THE TAKEAWAY

AI ecosystems will influence human behaviour. The question is whether we can reconcile three outcomes that seem at odds: 1) motivating people to act in the present for their desired futures; 2) retaining human agency; 3) persuading AI developers to pursue these ends.

It’s facile to profess that companies must prioritize public wellbeing. Developers need a realistically incentivized way to implement beneficent ends. That could mean corporations changing income streams so that behavioural AI is not primarily funded by advertising based on short-termist metrics, but accrues revenue from users’ progress towards stated goals, perhaps with payments made accordingly. People may be reluctant to pay $20 per month for a chatbot subscription, but most would shell out far more if the money were evidently correlated to their career success, or a better body image, or more fulfilling social lives. Think how much money people already spend on gym memberships and self-improvement courses, often to limited effect.

Regulatory policy might adapt too, so that tech developers whose products had a demonstrable drag on measures of health and productivity might face curbs or fiscal penalties, while those labs whose tools benefit public welfare might gain access to government contracts, R&D funding, or tax discounts proportional to the improvements.

At each stage of this ongoing AI transformation, system design should attend with utmost care to behavioural effects, applying empirical evidence to responsible development, while informing policymakers, and alerting the public. This means moving beyond the existential-risk debates to establish a human-risk research agenda, including pre-release behavioural audits of significant AI applications, post-release monitoring of how humans actually interact with them, and longitudinal studies to judge their diffusion through society, considering everything from polarization, to loneliness, to wellbeing.

We did none of this with social media. As a result, we still dispute what it did to humans, whether it is the cause of contemporary strife, or if we are just blaming machines for our own ills. What we must avoid is leaving AI to set the proxy for “the good life.” Otherwise we risk perpetuating the dismal compromise: getting what we want, to ends we do not want at all.

Done properly, we may create a positive feedback loop, where AI does not simply crash into humanity. It helps explain us.

APPENDIX: SEVEN SPECULATIVE SETTINGS

Could AI improve your behaviour? Or will it supplant your autonomy? In part, this may be a design problem. Behavioural AI should favour your best, allow for personal evolution, and incorporate human agency. Here are a seven ways:

1. Know-Your-Human Requirements

• Onboarding: Before activation, any personalized AI system with meaningful sway over behaviour could conduct a short and transparent conversation with its user, establishing the human’s short-term habits (e.g., “I’m addicted to tea” or “I go to bed too late”) and long-term aspirations (e.g., “I dream of moving countries,” or “I wish I were more sociable”).

• No box-ticking: The process must never become akin to Terms & Conditions, but like a first chat with a new therapist, establishing priorities, career goals, health issues, what’s missing from one’s life. The results must be strictly encrypted, and accessible only to the human in question.

2. Staying Aligned

• Changing Your Goalposts: The Know-Your-Human survey establishes the opening defaults of the AI ecosystem’s weighted recommendations and nudges—say, a behavioural balance of 70% long-term objectives to 30% short-term pleasures (among many other personalized settings). These defaults must remain accessible, intelligible, and simple to adjust.

• How’s It Going?: Annually, the system checks back with the user, offering an engaging review of the intervening period, akin to Spotify Unwrapped but for your behaviour, while ensuring that recommendation weights and intervention goals still align with the user’s wishes. Users could toggle on an optional “All Good?” oversight mechanism, which would detect radical fluxes in behavioural data that might suggest an alteration in goal-profile and perhaps acute distress, which would trigger a check-in.

3. Human in Charge

• Under My Thumb: Any behavioural influence—from recommender-system weights to goal-driven nudges—must have an available explanation. This could mean a sidebar reasoning card, or a hover-over rationale, or a voice-interaction mode that could take user questions.

• Sliders Instead of Black Boxes: Besides the pleasures/goals balance, users may also adjust more narrow preferences, moving behavioural influences between poles such as “entertainment” vs. “education”; “novelty” vs. “familiarity”; or “fresh learning” vs. “knowledge review.” Any settings change with significant behavioural impact should include a forecast of possible outcomes.

4. Sliding Doors

• Differing Paths: Whenever prompting behaviour, the AI system should know other approaches, and make these available at the user’s request, including simple explanations of how each alternative might differ in effect and underlying assumptions, much as map apps will display alternative routes and forms of transport, with differing travel times.

5. Shuffle Mode

• Randomized Serendipity: To encourage exploration and break feedback loops, recommendations should occasionally contravene past behavioural patterns. The user sets the desired proportion of randomization, but also has a “Surprise Me” option alongside standard recommendations.

• Default Updates: After the user interacts with a “Surprise Me” input, the system may ask whether this challenge felt worthwhile. Such queries should not be excessive, and may be toggled off. But if beneficial, the feedback could inform future system defaults.

6. Wisdom of Crowds

• Community Evidence: Recommender systems should favour inputs that other people with similar goals (and success in achieving them) have engaged with, rather than just circulating what is most popular.

• Smart Cues: The system should provide quality signals, tagging options according to reputational ratings and empirical goal-efficacy. The current practice with recommender systems—either struggling to define suitable content or eliminating curation altogether—have precipitated political contests that may disregard users’ goals.

7. Repel the Intruders

• An Off-Switch for Context-Switching: The system default is to focus the user’s concentration on their current desired activity, minimizing distractions. This could include adjusting the user’s environment to encourage flow, and making judgments on whether to mute specific notifications, based on the user’s current activity and its pertinence to their goals.

• Angel on Your Shoulder: If users are engaged in behaviour they have expressed a wish to avert, the system could offer gentle interventions, perhaps speaking a reminder aloud in the AI-generated voice of a friend or trusted influence (who gave consent), or even in the voice of the user: “Hey! Sorry to interrupt, but just a reminder that you wanted an early night? Shall I turn off when you finish this clip?”

Subscribe now

Austin Vernon wrote a great piece on the integration of AI agents into the workforce. I agree with much of his perspective, but want to imagine the likely changes and challenges over a longer timeframe. For this essay, I’m not considering AI safety implications, and I’m assuming agents are broadly directable/aligned (like language models we use today) and roughly as capable as humans - though with some limits at first. This piece is exploratory, looking at plausible dynamics rather than making hard predictions; it’s very possible that in a year or two I will have updated my views significantly. A key question is how factors like the persistence of human value in certain contexts, regulatory responses, and social factors created will shape the path toward greater automation.

The challenge of building and deploying a non-generic, practical, and useful AI agent

With regards to Vernon’s piece, I agree that wikis and similar tools—which document and explain the nature of work and job functions—will be key to enabling AI agents to be productive in the workforce. But there are certain factors beyond cost and coordination constrain the extent to which we can fully codify essential knowhow.

The first challenge is that codifying knowledge isn't easy. In middle management roles, for example, what makes someone good isn't just the knowledge they hold. Trainee lawyers learn this early: knowing the law is expected, but success often hinges on social practices, taste, judgement, proactivity, billable hours, modelling other actors, and managing conflicting information.

Similarly, employees in most commercial organisations hold critical “grey” or institutional knowledge. This includes insights gained from informal sources, like podcasts, or a nuanced understanding of workplace politics. For instance, knowing how to navigate internal politics at work or recognizing which tasks are worth prioritizing under shifting external circumstances (e.g. political changes) is rarely written down. You won’t find an internal wiki entry that says “Avoid asking this person about X because they’re biased against it” or “The new minister hates automated vehicles, so highlight healthcare topics at the next event.” In human-agent workflows, this kind of contextual knowledge and knowhow gives humans a certain advantage, and presents a significant challenge to overcome before organizations can transition to agent-only companies.

More cynically, employees may actively withhold institutional knowledge as a form of job security. This challenge is solvable - agents could infer and learn quirks over time if management provides enough access to contextual data. For instance, an agent might eventually learn, "This is the quirk to remember when submitting a finance request." However, this process will be slow and uneven, especially for roles requiring physical or social interactions. While online customer support jobs may adapt quickly, more complex roles will take longer to automate effectively.

Michel Berry, in Une technologie invisible, highlights another issue: many organizational instruments or routines act as “invisible technologies” - structural mechanisms shaping day-to-day decisions beyond explicit policy. If AI agents replicate and reinforce these routines uncritically, they risk embedding outdated principles long after their original purpose is forgotten. This underscores the importance of revising these norms and processes alongside the deployment of agents. In other words, organizations risk inadvertently locking in past inefficiencies, even as agents upgrade capabilities.

The need for better organisational and technological infrastructure

In principle, all of this seems feasible for agents from a capabilities perspective; the real challenge lies in ensuring a fair playing field to compare them against humans. As Austin notes, agents need context - a pipeline to infer, store, and retrieve the relevant information at the right time. You can't simply ‘plug and play’ an agent into a role and expect it to figure everything out; significant organisational changes are required to enable the use of these pipelines and agents effectively. At a minimum, this involves gradually replacing legacy IT systems and infrastructure, a process that, as most CTOs will attest, is both lengthy and tedious. In some cases, it may also require restructuring teams or reducing staff to address principal-agent problems and streamline organisational structures.

I anticipate that agents not only 'augment' employees but also observe and learn from them. For an agent to be truly useful and personalised, it must understand the employee’s work, goals, and style in detail. Agents may even crystallize insights and biases that employees themselves overlooked, enabling them to perform better over time. Much like a new hire learning internal dynamics, these agents would grow in capability - but with the advantage that their insights could be shared instantly across all other agents. An employee’s mistake and subsequent correction could improve the entire network, not just the individual agent. Eventually, most agents' queries to humans might focus on preference (e.g., “Which colour?”) or on information they lack the ability or authority to access (e.g., “What did the judge say at the trial?”).

However, this introduces complications, particularly around privacy and data sharing. What kind of data can an employee’s agent communicate to others? Should agents infer insights from personal chat logs? These questions will create thorny disputes that many companies may prefer to avoid. Instead, we may see a shift toward dramatic functional outsourcing, where legacy systems are discarded in favor of contracts with newer AI providers that deliver better performance at a lower cost. As I’ll explore later, startups and organisations (including in the world of research) are often better equipped to start fresh with optimal setups, allowing them to take on tasks for larger, more rigid organisations.

Another challenge lies in how decision-making and task automation might reshape organisational culture. AI systems might tend to optimize for specific metrics, which could lead to a neglect of complex, ground-level realities. While competitive pressures between firms may address some of these issues over time, Berry’s work reminds us that entrenched management instruments can persist even in competitive environments.

This transformation will happen gradually. As AI-human hybrid organizations evolve, new forms of tacit knowledge will emerge - focused on effectively prompting, directing, and coordinating AI systems. While this will create a temporary need for human expertise (the “prompt engineers” of the future), the growing capabilities and utility of agents might ultimately reduce the demand for human workers.

What happens when human quirks and tacit knowledge are accounted for?

At this point, organisations could achieve agents that are quasi-substitutable for human employees, providing almost equivalent value. This might happen either because an organisation has reinvented itself to gradually shape agents capable of understanding and absorbing tacit knowledge as effectively as humans, or because the task has been outsourced to an ‘AI-first’ start-up unencumbered by operating with fewer legacy constraints.

But why is this institutional/tacit knowledge required in the first place? In many cases, its importance stems from inefficiencies in human-dominated systems. For example, understanding a colleague’s subtle preferences or navigating office politics becomes necessary because humans can be irrational, misaligned with organisational goals, and/or biased. If AI agents were to replace these human colleagues, much of this institutional knowledge would become irrelevant. You wouldn't need to account for Sally from Finance's particular communication preferences or office politics - AI agents would interact rationally and efficiently with each other. While AI agents might initially augment human employees by learning their know-how, this expertise in managing human quirks would diminish in relevance as more workplace interactions shift to being agent-to-agent.

We are not rational agents - but agents can be designed to be. Over time, as agents take on more and interact primarily with each other, the know-how derived from human quirks will lose its value. Organisations will adapt to these changes by restructuring to better align with agents’ needs. Rather than accommodating Sally in Finance’s arbitrary preferences, it will become more economical to replace the role entirely with an agent.

However, as Aghion, Jones, and Jones suggest, growth in organisations and the wider economy may be constrained not by what tasks can be automated, but by tasks that remain resistant to improvement - a phenomenon akin to Baumol’s ‘cost disease.’ This is reminiscent of how, historically, even as manufacturing productivity soared, sectors reliant on interpersonal dynamics or nuanced judgment lagged behind, driving up costs and complicating efficiency gains. A key question is whether AI agents can handle roles requiring rich interpersonal or social judgment, as well as those with physical demands. Even seemingly straightforward tasks can be surprisingly difficult to automate (e.g. planning a party), whether because no one codified the obvious (e.g., retrieving a physical file) or because the job relies on informal “corner-cutting” that keeps large organisations running smoothly. Employees often perform small favours, trade concessions, or bend procedures to avoid deadlocks, relying on trust or rapport. An agent might follow protocol rigidly, running into red tape where a human would find a workaround. Similarly, senior-level tasks such as forging alliances or interpreting political signals involve trust dynamics that agents may struggle to navigate. These “last 5%” edge cases could delay or complicate the transition to agents unless organisations redesign processes or equip agents with ways to handle the flexible, often invisible rules of human collaboration.

Similarly, stubborn ‘Baumol-like’ constraints may arise from physical tasks and roles reliant on interpersonal relationships, where robotics lags behind cognitive automation. Though these frictions may slow adoption in some areas, it’s unclear they will be strong enough to derail the shift toward AI-first processes. Over time, as the cost-performance ratio of robotics improves, AI-driven advancements may also apply to physical systems. Once robotic platforms become as flexible as AI software, the residual Baumol-effect on labour-based tasks could diminish (for example, physical interactions will be less of a constraint). Just as agent-first startups have leapfrogged legacy enterprises in software automation, the next wave of agile robotics could disrupt entire industries currently shielded by physical complexity.

Why restructurings and outsourcing will be necessary

Restructurings and outsourcing will be necessary for two reasons. First, restructuring is required to integrate agents into company workflows seamlessly, ensuring they gradually learn and understand the grey institutional knowledge necessary for effective collaboration. Second, as agents take on more tasks and responsibilities, organisations will need to adapt workflows and processes to maximise the productivity efficiencies these agents can offer—efficiencies often limited by the quirks, biases, and inefficiencies of remaining human workers.

These efficiencies are significant, not arbitrary. One often-overlooked aspect of modern organisations and bureaucracies is the extent to which middle management can be ‘misaligned’ with the company’s interests. Managers are sometimes incentivised to make decisions that appear better - safer, or more appealing - to those evaluating their performance, even if those decisions aren’t in the organisation’s best interest. "I'm more likely to get a promotion if I do X, even if X isn't really needed or as impactful as Y". Similarly, employees may disagree with the organisation’s broader goals, creating further friction. As organisations scale, it becomes increasingly difficult for managers to oversee employees and for directors to oversee managers. Highly effective agents, capable of performing many tasks in parallel, offer an opportunity to simplify internal hierarchies and address these principal-agent problems.

If agents merely augmented employees - following their instructions without question - misalignments would persist. For example, the agent might obediently hire PwC to produce an unnecessary and costly PowerPoint presentation, or turn a blind eye as an employee fakes a sick day or pretends to be busy. Addressing this misalignment can be approached in two ways. The first is aligning the human employees more closely with organisational goals, which would likely require extensive surveillance - a method that would be resisted, demotivating, and unpopular. The second is to reduce human involvement altogether, making the agents accountable directly to the director.

But then how useful is the intermediary human really? Under the above proposed model, the director delegates tasks directly to the agents, bypassing the need for human intermediaries. If taste, curation, and tacit knowledge are no longer where humans outperform agents, the rationale for keeping these intermediary roles diminishes. Delegating directly to agents (who benefit from sufficient context and capabilities) creates a cleaner, more efficient chain of command and reduces the risk of misalignment. I expect these restructurings will gradually decrease the involvement of human employees over time.

Some important caveats

Throughout this transformation, wider dynamics will likely slow these changes. Many people - particularly in high-skill, academic, and white-collar jobs - derive meaning from work and possess power to delay or block changes through strikes, unionisation, or negative publicity. Governments may also require reasonable human oversight in certain contexts, such as healthcare, justice, and critical infrastructure, for liability and safety reasons. In many services, including education and hospitality, the value of human interaction itself cannot be overlooked. These frictional roadblocks and institutional inertia could extend the human-to-agent transition “a few years” into “a decade or more,” depending on industry. Economists have long noted that large firms often resist adopting efficiency-enhancing measures that disrupt entrenched interests or managerial structures.

However, these factors alone may not be sufficient to halt the wider competitive forces driving automation. First, rising human labour costs increase the incentive to automate, as observed in France following the 2000 introduction of the 35-hour workweek. Second, these technologies will deliver impressive economic and other benefits that will be hard to ignore. Third, even maintaining some degree of human oversight doesn’t fully counteract these dynamics; AI allows fewer humans to accomplish far more. Put differently, organisations can still massively downsize their workforces despite HITL (human-in-the-loop) requirements.

Another critical consideration is the democratisation of coding and building, which will empower individuals and start-ups to adopt agents rapidly. Large restructurings within corporate monoliths are often met with significant resistance. By contrast, smaller start-ups - unencumbered by legacy systems - are better positioned to produce high-quality goods and services quickly by embracing agents from the outset. They avoid “Sally or Jake” problems altogether by not hiring them in the first place.

Smaller startups, leveraging minimal staff and maximum automation, could become showcases for more “rational” ways of operating. While some tasks, particularly those requiring deep human interaction, may resist automation temporarily (a microcosm of Baumol’s cost disease), the overall trend is clear: process-oriented and knowledge tasks will be increasingly handed off to agents. By minimising the accumulation of human employees and their inefficiencies, start-ups can operate more effectively and efficiently. Consider Palantir, which serves major enterprises with far fewer employees than its competitors. Successful start-ups following this model may force larger companies to adapt through competition or acquire these innovators, accelerating the transition to automated operations.

Conclusion

If we build the infrastructure to enable AI agents to learn tacit knowhow and integrate seamlessly into our systems, the future points toward leaner, more efficient organizations where agents progressively replace human roles. This essay illustrates that if we assume (a) low costs and parallelization of mostly aligned agents; (b) human-like or stronger capabilities that are easier to control; (c) seamless integration into a company’s technical and managerial infrastructure; and (d) a pipeline for absorbing and learning tacit knowledge from humans, then the trajectory tends towards giving agents more production responsibilities, and removing these from humans. Augmentation may work for a while, but it doesn't seem sustainable in the long run as employees lose their know-how, and the agent takes on more tasks. However, it’s worth cautioning that the numerous ‘if’s involved - many of these outcomes are difficult to predict and account for in advance. It would be worthwhile to explore the complexities that could arise if agents were misaligned, harder to control, or exhibited disparities in capabilities.

Interestingly, the same infrastructure needed for human–agent augmentation is also what allows organisations to reduce reliance on humans entirely. In cases where a company is risk-averse or constrained by legacy contracts, this could mean outsourcing tasks and responsibilities to a smaller, leaner part of the organisation or a separate agent-first start-up - a form of internal cannibalisation. From the perspective of shareholders, and arguably mainstream economic theory, this approach makes sense. This trend will not affect lower-level employees or middle management; directors, too, face similar challenges. Over time, even their roles may diminish, leaving the CEO to manage a multi-agent system (essentially, a company of agents) with maybe a few other humans and optimised for shareholder benefit. This would result in a highly ‘rationalised’ company with minimal principal-agent frictions: no strikes, no rest, no weekends, nothing.

The pace and magnitude of this shift are debatable. Still, I believe a substantial reorientation toward AI-driven operations, culminating in near-full automation of many roles, is plausible in the long term if AI capabilities continue to advance, adoption costs decline, and regulations remain permissive. In some interpersonal or legally sensitive roles, Baumol-like frictions may slow adoption, but are unlikely to halt it entirely. As Berry warns, invisible technologies that shape daily corporate life could embed old inefficiencies in new forms, necessitating organisational overhauls alongside agent deployments. Accelerating factors, such as groundbreaking discoveries or dramatic productivity gains, could further incentivize these shifts. For instance, curing diseases or rapidly improving productivity may create strong pressures to maintain progress and avoid unnecessary delays. Rising shareholder value and broader economic growth would also hasten these transitions.

Ultimately, this trajectory is desirable to the extent that it boosts productivity, reduces costs, alleviates poverty, cures diseases, and fosters abundance. While individuals like Sally from Finance or Jake from Policy are valuable human beings, their immediate interests may not outweigh the benefits experienced by a broader population through improved economic conditions, health, and life opportunities. The challenge will be ensuring that people like Sally and Jake transition into positive, fulfilling lives after job displacement. The promise of the future must consider them too, rather than relegating them to the ranks of unfortunate externalities. Firms, policymakers, and society must address these transitional frictions, rethinking training, labour market protections, and the invisible structures that guide daily decisions. Certain functions relying on interpersonal relationships will continue to require human-driven interactions where trust, rapport, and informal negotiations matter. This will be an important challenge of the next decade: rethinking labour, its role in our lives, and its connection to a meaningful existence. Critical questions about finding meaning in work and managing the acute challenges of job displacement have been deliberately avoided here and will be explored in a future essay.

—

Many thanks to the following people for comments: Conor Griffin, Gustavs Zilgalvis, Julian Jacobs, David Wolinsky, Ben Lepine, and @PITTI_DATA.