Every six weeks, we round up three papers that we think AI policy folks should be reading. In this edition, we look at a proposal for how to identify the agents that will soon fill the economy; research on the prospect of self-improving AI; and new insights about how to use AI to prevent contrails, or artificial clouds, from warming the planet.
1. Identifying (and incentivising) AI agents
What happened: A trio of law and philosophy professors considered how to identify who (or what) is responsible for AI agents’ actions in the world, and came up with a two-part proposal: that the disparate and evolving agents within a system should exist legally as a new form of corporation; and that each corporation should link to accountable humans.
What’s interesting: The paper by Yonathan Arbel, Simon Goldstein, and Peter N. Salib starts with a thought experiment. It’s 2030, and your AI assistant suggests that it optimizes your slow WiFi connection. After you agree, it spawns a swarm of agents. Some are copies, while others are cheaper agents running on open-source models. Some start to interface with AI agents from other companies. Three months later, two FBI agents knock on your door and explain that your network has been piggybacking on a local defense contractor’s WiFi network.
Before determining who is responsible and what the repercussions should be, there are more basic questions: Who are the AI actors in this story? How many are there?
The economy will soon be filled with capable AI agents. To deter and respond to such harms, the authors argue that we need to be able to identify these agents, at two levels.
To prevent human misuse or negligence, we need ‘thin identity’. This would connect AI agents to the humans most able to control them, similar to how ‘know-your-customer’ rules tie banking transactions to humans.
Humans will be unable to monitor and control every AI decision, so we also need to be able to identify agents themselves, hold them accountable and incentivize them to behave well. To do so, we need ‘thick identity’ that can distinguish AI agents as stable, coherent entities, with persistent goals. This goal is pragmatic and does not require viewing AIs as conscious in any sense.
Thickly identifying agents is harder and more novel, as AI agents need not be attached to a physical body. Multiple agents can also work together on a single task. Any single agent can be copied, spun up, spun down, or be continually updated.
To address such challenges, the authors propose creating algorithmic corporations, or ‘A-corps’. These would have two key elements:
Legal personhood: Like a traditional corporation, an A-corp would be a single legal entity that persists over time. It could hold property, make contracts, and be sued. But it would be run by a collection of AI agents. As such, the proposal runs contrary to scholars who have argued against granting legal personhood to AI agents, or called for bans on algorithms running companies because of concerns about crime and companies using them to avoid liability.
Computationally-secure governance: Each A-corp would have a unique digital certificate and a secure private key to authorise transactions. The humans that own each A-corp could grant the key to an AI ‘manager’ agent who in turn could grant more limited permissions to sub-agents within the A-corp, or to other A-corps, such as permissions to spend up to $100 or to read a batch of emails.
The proposal addresses thin identity by reducing the vast number of AI agents down to a smaller number of A-corps, whose actions are traceable back to their human owners. As with limited liability companies (LLCs), the human owners would not be responsible for all harm their A-corps cause, but could lose all funds they invest and possibly face further liability, for example in cases of fraud or negligence.
The proposal addresses thick identity via its ‘resource constraint thesis’. All AI agents need resources, like money and compute. A-corps provide AIs with a way to access these resources and an incentive to manage them well. For example, A-corps that tightly monitor and audit their sub-agents’ performance would get more resources, while A-corps that allow fraud or waste will lose resources. This encourages A-corps to self-organise, into stable, coherent, multi-agent systems.
The authors argue that A-corps could also address alignment concerns, for example by reducing the incentive for an AI agent to exfiltrate its own weights, because that new AI instance would lose access to resources and permissions from the A-corp.
To make it happen, the authors call for a public registry of A-corps. This would list each A-corp’s human owners, the certificates to authenticate it against, as well as (potentially) the differing permissions enjoyed by its agents. Ultimately, the authors argue that A-corps should become mandatory for any AI agent taking “economically significant actions”, and to guard against criminals using AI agents anonymously.
The authors respond to some expected pushback. They do not see A-corps as anthropomorphising AI because the proposal does not require anybody to view agents as having deeper desires or wants. They also think A-corps can prevent the risk that AI agents might slowly build up resources before deploying them for harm, by encouraging inter-agent trade that penalises rogue behavior. Could A-corps disempower humans? The authors argue that they provide a pathway to tax and redistribution, and enable humans to better steer agents, for example by designating the parts of the economy that A-corps are permitted to operate in.
2. When AI builds AI
What happened: The Centre for Security and Emerging Technology, CSET, released a report on the prospects for AI improving itself, known as automated R&D or recursive self-improvement, based on an expert workshop in July 2025.
What’s interesting: In 1964, the computer scientist I.J. Good wrote about the possibility of an “intelligence explosion” that would leave “the intelligence of man.…far behind”. Researchers have also long automated aspects of writing code and AI model design.
However, the speed of AI coding advances suggests that something qualitatively different may soon occur. This makes two questions salient: 1. Could AI automate the entire AI R&D process? 2. Will this R&D automation extend across all scientific disciplines? The CSET report focuses on the first question.
CSET defines AI R&D by distinguishing between research scientists, who generate hypotheses, design experiments and interpret results; and research engineers, who write code, fix bugs and generate data. They also note the inputs that AI R&D relies on, such as raising funds and acquiring compute.
They sketch out four overlapping scenarios for how AI R&D may play out:
1. Explosion: AI systems automate a growing share of AI R&D. Initially, this leads to modest productivity gains, but as the length and complexity of tasks that AI performs grows, productivity soars. AI systems become far more capable than humans, whose involvement in AI R&D falls to zero.
2. Fizzle: The share of R&D tasks done by AI rises, but rather than leading to compounding improvements, capabilities start to plateau.
3. Amdahl’s Law: AI automates certain activities, like writing code and running experiments, but not others, like research strategy.
4. The expanding pie: As AI automation grows, humans realise that new ideas and breakthroughs are needed that AI systems cannot yet provide.
The experts in CSET’s workshop held widely diverging views on which scenario was most likely. Most importantly, new empirical data is unlikely to resolve these conflicts, because participants may view the same data as confirming their own assumptions.
For example, an AI system’s inability to reliably use a keyboard or mouse may look like a bottleneck to one expert, but a source of explosive growth to another—if they expect this human-focussed tooling to get adapted for the AI era. Similarly, different experts may view AI automating a growing share of R&D tasks as progress towards a fast takeoff, or as low-hanging fruit being picked off, accelerating progress only as far as the upcoming wall.
These differing views are also visible in more recent commentary on the topic.
The prominent AI researcher and writer Nathan Lambert recently cited Paul Allen’s concept of a ‘complexity brake’ to argue that as we understand intelligence better, further progress becomes exponentially harder. In addition to incurring financial costs, Lambert argued that running suites of AI agents won’t necessarily lead to exponential progress, because those agents will perform best on narrow, verifiable tasks, will be hard to manage in large numbers, and will sample from similar parts of the distribution of AI research ideas, inhibiting more novel breakthroughs.
Conversely, Ajeya Cotra at METR, the Model Evaluation and Threat Research organisation, recently wrote about how she “underestimated AI capabilities (again)”. She argued that AIs may, counterintuitively, find it easier to decompose longer projects into sub-components that multiple agents can run in parallel, than for shorter tasks. AIs will also produce good documentation for their fellow AIs, which could accelerate progress.
If faster automation and progress does occur, the CSET authors see two main risks: Less time to prepare for safety risks from AI, and lower human understanding of AI systems. To address these risks, their recommendations have a strong focus on improving access to evidence, including:
New evaluations of AI R&D, including for ‘messy’ tasks such as research strategy, which lack clear specifications and success criteria and take place in a dynamic environment with various real-world interactions.
New approaches to evaluation to better distinguish ‘degrees of accomplishment’ from a simple success/failure binary.
Better insights into how automated R&D is progressing within AI labs, such as data on how funding is allocated and qualitative impressions of progress from leading AI researchers and engineers.
3. Planes and global warming
What happened: A team of researchers, including from Google and American Airlines, published results from their latest experiment to use AI to reduce condensation trails from planes—a key contributor to global warming.
What’s interesting: When pilots fly, particles from the plane’s exhaust can mix with low-pressure air to form contrails—white, artificial clouds, made up of ice crystals. These contrails are a net contributor to global warming, because they trap heat that would otherwise escape. Debates continue over exactly how much they contribute, but one estimate suggests that they contribute a lot, causing around 2% of ‘radiative forcing’, which measures how different factors, like CO2, heat or cool the planet.
As the environmental writer Hannah Ritchie explains, more important than the absolute figure is the fact that contrails offer a rare opportunity to reduce global warming almost immediately, at relatively low cost. This is because a small share of flights cause most of the warming-inducing contrails—generally those that fly through parts of the atmosphere that are both very cold and very humid. If planes take short detours to avoid these patches of air, contrails (and warming) should drop.
A few years ago, Google researchers partnered with American Airlines on a proof of concept. Using satellite imagery and AI, they were able to predict where contrails would emerge and guide planes to avoid them, reducing contrails by >50%, across 70 test flights.
In the latest study, they expanded the experiment to 2,400 American Airlines flights from the US to Europe. They placed ~50% of planes in a treatment group, where flight dispatchers were given two choices: a standard flight plan and an alternative contrail-avoidance one. Their decision for which to recommend was voluntary.
For flights in this intervention group, contrails fell by 12% compared to a control group with no contrail-avoidance plan. Importantly, the contrail-avoidance routes also did not lead to a significant increase in fuel use. At first glance, these results seem positive, but modest. Digging into the results highlights the challenge of getting useful AI deployed at scale.
In particular, dispatchers who received contrail avoidance plans only recommended them to pilots 15% of the time. Even then, the avoidance plan was only successfully flown in 60% of flights. For planes that did successfully follow the avoidance plan, contrails fell by more than 60%, a much larger reduction. So the tech worked, but was often not used.
Why? Dispatchers are busy and must often deal with other priorities, like bad weather and turbulence. To avoid contrails, planes also need to climb and descend mid-flight. This is safe, but creates more work for pilots and air traffic controllers. As it was voluntary, the incentive to change to a contrail-avoidance plan was weak.
The way that the dispatchers received the information also meant that they didn’t fully understand why the suggested up and down changes were necessary. Happily, the authors feel that most of these obstacles are addressable, with a combination of a better user interface, some automation, and more incentives.
In addition to its immediate usefulness, the study is a rare real-world attempt to quantify the benefits of AI to tackling global warming. At the moment, the AI and climate change policy discussion is often negative and focuses on the emissions that may result from building and operating data centres (and other devices) to train and run AI models. This is important, but there are reasons to think that these emissions will be relatively low, or at least lower than many assume. In contrast, AI could potentially reduce emissions and warming by far larger amounts, for example by accelerating research on solar and fusion power, or making buildings and energy grids more efficient. But these benefits are typically more speculative, harder to quantify, or in the case of contrails, more contingent on human behaviour.
This experiment demonstrates that the benefits of AI to tackling global warming are real, but also points to the interventions that will be needed to push them to their full potential. The study is also timely, given that governments are focussing on contrail avoidance and some policy action may be required, for example to help standardise and mandate contrail prediction software or to generate high-resolution humidity data.
Thanks for this, I always find your stuff really interesting and thought provoking.
I hadn't thought about the identity persistence of AI agents before. Artificial identity persistence could be a whole new topic in philosophy!