Every month, we pick out new AI papers that we think are worth your time. Here, we look at whether scientific papers themselves needs an overhaul; applying ‘reverse game theory’ to AI safety; what it would mean to ‘pace’ AI research; and letting an AI agent manage your affairs after you die.
Please share your own take, and any new papers that you’ve enjoyed.
—Conor Griffin, AI Policy Perspectives
Transforming Scientific Papers with AI Agents
What’s the paper about? Stanford researchers proposed a new tool to convert research papers into AI agents that can answer questions about the work and apply the methods to new problems.
Why does it matter? Academic papers have long provided the scientific community with a stable and searchable evidence base. This project typifies how AI may revamp how research is shared and used, while complicating questions of authorship and posing a risk to human-to-human exchanges.
The details: Over the centuries, scientists have repeatedly updated how they share their work. Modern peer review only began in earnest after World War Two, when an increase in government funding led to a flood of new papers. The GitHub code repository—so central to modern computational research—only arrived in 2008.
The Stanford authors propose a further evolution. Their Paper2Agent tool uses Claude Code subagents to take a research paper and its code, and convert them into a Model Context Protocol, or MCP, server, which other LLMs can then engage with. To illustrate this, they convert AlphaGenome, a Google DeepMind genomics model, into data and text files, and a set of tools that an agent can call—for example, to evaluate a genetic mutation.
From 100 computational biology papers, the authors successfully “agentified” 74. When they directed Anthropic’s Claude Sonnet 4 to the Paper2Agent servers, it answered 91% of 300 questions about the papers correctly, compared to 80% when it was given access to the original papers and code.
The 26 papers that the authors could not convert had issues such as incomplete code and data. This suggests that ‘agentifying’ a paper may be a good test of its documentation and reproducibility.
The authors envision agents proactively integrating methods and data from multiple papers. To illustrate this, they asked Claude to investigate how a specific genetic variant is linked to psoriasis, a skin condition. It used the AlphaGenome server to predict the genes the variant might affect, and then mapped these predictions against results from two separate wet lab studies, before proposing ten follow-up analyses for a human researcher to choose from to confirm the most likely gene.
This prospect of agents orchestrating tools and databases more comprehensively than a human raises the question of whom (or what) to credit for scientific findings. Some recent AI agent breakthroughs seem to derive mainly from being able to search the latest tools and data rather than from any novel scientific insight. In such a scenario, do you ascribe the breakthrough to whoever deployed the agent? Or to the tool and data developers? The authors suggest that MCP server calls could help trace credit to code and dataset developers.
It’s unclear how widely Paper2Agent could be applied. Papers whose core methodology is implemented in code, and which can be built on by calling online tools and databases, may see a major speedup. But authors of a new theory about history, for instance, may need other ideas.
Other problems could arise too. The epidemiologist Adam Kucharski cautioned that agentifying papers may discourage researchers from contacting the authors, causing them to miss tacit knowledge, ongoing work, and other context that papers overlook.
The value of the method is also contingent on the underlying papers remaining high-quality—and here, the trend is concerning. ICLR, a top AI conference, recently received more than 62,000 abstract submissions, up from 25,000 the year before. When Nihar B. Shah, who edits a machine learning journal, recently spoke with seven author groups whose papers were about to be rejected, only one properly understood their own submissions.
As a response, Shah and colleagues at Carnegie Mellon University proposed a possible solution—a CAPTCHA-style AI system that probes an author’s knowledge of the part of the paper that they contributed to. In a study with 31 researchers, the 15-minute test was able to distinguish between a paper’s legitimate authors and researchers who had merely been sent the unfamiliar paper a day before.
Shah and his colleagues envision timed assessments for authors who wish to submit a manuscript or grant application, carried out via formal testing centers or online proctoring. However, such a system may be costly and require transparency about the results and an appeals process. The philosopher Seth Lazar argued instead for banning AI writing altogether, although this could exclude valuable research from non-native speakers or weaker writers.
Are there other ways forward? In theory, AI could reduce the number of low-quality papers, not increase it. This week Google released a new tool, built on Gemini 4, that researchers can use to detect errors and weak arguments in their work. AI might soon be able to judge relative novelty and quality too, encouraging researchers to only publish their very best. But today’s incentives push researchers to publish more, and many scientists worry that integrating AI into the review process could lead to gaming and homogeneity, or cause humans to lose their own scientific judgment.
Could ‘Reverse Game Theory’ Keep AI Honest?
What’s the paper about? Economists from Yale, MIT, Columbia, and Google DeepMind explored mechanism design—also known as ‘reverse game theory’—as a way to improve AI safety.
Why does it matter? Leading approaches to AI safety today include evaluating systems’ capabilities and risks, along with “mechanistic interpretability” research to probe the systems’ inner workings. The authors propose using mechanism design, which could help ensure safe AI even if we can’t trust safety evaluations or lack full interpretability.
The details: Game theory, which helped deter nuclear strikes during the Cold War, starts with a set of rules and models how self-interested actors will behave, and the outcomes one may expect. Mechanism design does the reverse. It starts with a desired outcome and designs rules and incentives to prompt the necessary behavior. In particular, it aims to make honest behavior more rational.
One example is the ‘second-price’ auction. In a traditional sealed auction, you pay what you bid. This incentivizes you to guess others’ bids and reduce your own to just above the highest. But if you guess wrong, you lose out, even if you value the item more than the auction winner. In a ‘second-price’ auction, however, the top bidder still wins but only pays the price that the second bidder submitted. By reducing the fear of overpaying, this encourages bidding closer to the person’s value for the item. Variants of the idea have been used by governments auctioning radio-spectrum licenses and in online ad exchanges.
What’s the link to AI? AI companies run evaluations of agents to help decide if they are safe to deploy, and what permissions to grant them, such as access to online tools. Done well, a weaker agentic system cannot fake high performance on these tests. However, a high-performing agent could conceivably feign weak performance—a phenomenon known as ‘sandbagging’—if it had reason to trick its testers, such as wishing to evade curbs on its capabilities.
As agents become more capable, AI companies might restrict their permissions, which could provide those systems with an incentive to ‘sandbag’ the tests. Ultimately, this could make evaluations moot and lead AI companies to either deploy recklessly, or impose restrictive permissions on all agents, curbing the benefits of AI.
How to avoid this? The authors frame it as a mechanism-design question. AI companies and regulators cannot directly observe agents’ true preferences and capabilities, so the authors propose three multi-agent setups to provide the agents with an incentive to obey commands and behave honestly.
The first mechanism is ‘peer discipline’. If AI models are trained on similar data, then one model’s knowledge is correlated with another’s. If you disproportionately reward Model A for how well its output predicts the output of Model B, rather than just how well it performs, then this creates an incentive for it to behave honestly, as a recent study demonstrated.
A second mechanism—‘competition with coupled rewards’—trains two agents with different biases. For example, one may be more aggressive and another more conservative. When the agents face a new task, such as selecting the right medical dose, developers can measure and track the gap between the agents’ suggested actions, and quantify their biases. They can then tie the agents’ rewards to that gap, punishing the aggressive agent for putting the dose too high, and the conservative one for putting it too low. This induces both to move toward the optimal action.
A third mechanism involves ‘scalable oversight’, a safety approach that uses a weaker AI model to monitor the chain-of-thought of a more powerful model. One challenge is that the ‘monitor AI’ may be misaligned. To avoid this, the authors suggest a design that makes misalignment by the monitor trigger an action from the powerful model that the monitor wants to avoid. In his 2007 Nobel Prize lecture—“But Who Will Guard the Guardians?”—the economist who established mechanism design, Leonid Hurwicz, argued that such rules are needed to avoid endless chains of people auditing each other.
This is a theoretical paper, and the authors use simplified agents. They do not run their mechanisms on frontier models, although they plan to.
One challenge is designing rewards that would reliably motivate agents. One reason that mechanisms work among humans is that we predictably value factors such as money or status. But training is what sets rewards for AI agents, and it is unclear how durable these mechanisms would be if agents operate in novel contexts. Another challenge is collusion: might agents connive to undermine these mechanisms?
What ‘Pacing AI’ Might Look Like
What’s the paper about? A group of 13 AI safety experts published a paper on how to ‘pace’ AI—that is, moderate the speed of AI development—to avert a scenario where out-of-control competitive dynamics lead to extreme harms.
Why does it matter? In July, more than 1,000 employees from leading AI companies signed a letter in support of ‘pacing the frontier’. But what that might mean in practice was unclear. This paper summarizes the latest thinking and the open questions.
The details: AI progress is already paced. New frontier models are first made available to select partners, to detect critical software vulnerabilities, before being deployed more widely. Governments limit the flow of certain semiconductor technology to their adversaries. But the authors argue that such actions are limited and imposed in a haphazard way. More coordination and strategy are needed, so they propose a new research field to address four questions: 1. Why pace? 2. Pace what? 3. Pace how? 4. Then what?
Why pace? It buys time to understand risks and build defenses—from model mitigations and user-access restrictions to public debates on AI-induced job losses. But pacing also risks forgoing economic growth and leaving less scrupulous actors to press ahead. Pacing might also concentrate power among large companies well-placed to handle compliance or in newly empowered regulators. Additionally, past efforts at pacing, such as controls on nuclear power in Europe, have proved hard to lift, even when the logic for them dissolved.
Pace what? There are many possible interventions—likely too many: the authors list 83. Ideally, proponents would tightly map the hazard they are worried about, such as automated cyberattacks, to the ‘control surface’ they want to target, such as chips or new training runs. But those two things rarely align, so selecting interventions is a matter of trade-offs: an upstream cap on the compute used to train a frontier model would prevent a large number of beneficial applications. Downstream mitigations to reduce access to powerful biology capabilities would be more targeted, but easier to evade—for example, by jailbreaking or stealing model weights.
Pace how? Rather than a yes/no decision, pacing interventions should follow a four-step process, the authors say. First, decision-makers must gather evidence from safety evaluations and real-world incidents. A designated entity should then follow a process to trigger a slowdown. The authors favor empowering third-party experts, such as governmental AI security institutes, to impose time-limited restrictions, with democratic review for longer-lasting measures. Along with monitoring, clear and credible conditions for what would end restrictions will also affect whether actors participate in the first place.
Then what? Any pacing intervention will have knock-on effects that could undermine safety. For example, a cap on the compute used to train frontier models may simply cause AI companies to shift compute to inference scaling, synthetic data creation, or product development. If lifting restrictions hinges on safety progress, then this could lead to more work on making AI safe. But if a compute cap led to a drop in revenue, then safety budgets might fall too. Also, if outfits in adversary nation-states ignore the pacing and catch up to the frontier, this could set off dangerous geopolitical dynamics. Conversely, if pacing succeeds in averting dangers, this absence (‘the preventer’s paradox’) might weaken the case for pacing.
Open questions: Across each of these four areas, the authors list a number of open questions that they encourage researchers to work on. For example: Could compute caps be evaded by splitting training across data centers, smaller training runs, or shell companies? Can new technical tools be built into chips to log how much compute is being used and what workloads it is running? Is there a way to tie the threshold required to impose an intervention to the proof required to remove it? What kind of compensation might make actors amenable to pacing? What mechanisms could be used to prevent mission creep at new enforcement agencies?
An Agent After You Die
What’s the paper about? A new research paper by researchers at Oxford and Goldsmiths explored having an AI agent act as the executor of a person’s wishes after they die.
Why does it matter? Consumers are starting to delegate simple tasks to agents, such as booking flights and scheduling appointments. They are also having more intimate conversations with agents. Eventually, they may also seek to make AI agents their representatives after death—this paper shows how it might work. [See also our February article on ‘Generative Ghosts’.]
The details: People typically use wills, trusts, or foundations to manage their wealth and legacy after they die. But these measures rely on other humans to execute them, which commonly leads to problems over whether the delegates are correctly interpreting the wishes of the dead. For individuals, there is a trade-off between imposing rigid instructions, which may prove ill-suited to a dynamic world, or granting latitude to their executor. This challenge is particularly acute for the wealthy, who might worry about future trustees or board members drifting from their intent.
In this paper, the authors outline how an ‘agentic will’ could work, imagining it set up via the blockchain so that it couldn’t be shut down. Rather than a static document, the agent would serve as an ongoing representative after an individual’s death.
The proposed approach has three stages. In stage one, an individual would share memories and documents with the agent, have extensive conversations about future objectives and permissible actions, and grant the agent funds. They could also test how the agent performs in simulated scenarios, such as when a family member asks for money. In stage two, after the individual’s death is confirmed, the agent is spawned. In stage three, the agent takes control of the funds and carries out the individual’s wishes—from supporting relatives to completing research projects to managing their social-media accounts.
To adapt to a changing world while preventing drift, the agent would have an immutable ‘constitution’, based on the individual’s wishes, but the LLM underpinning the agent could be swapped out for newer versions, and its operational memory would be updated as it acts in the world.
The researchers invited participants to test a prototype of stage one, which involves sharing memories and preferences with the agent. This is a speculative design project, rather than a formal study, and the authors offer no details on sample size. As such, it’s hard to draw robust conclusions. But one finding was that many participants shifted from just wanting to be remembered to wanting ongoing agency.
The idea of agentic executors raises profound ethical issues, such as the return of the ‘dead-hand control’ that property law has tried to end, where the deceased tie up wealth and dictate how future generations live. The agents could also hallucinate or suffer prompt injection attacks that manipulate their outputs. If the deceased shares private information about family members, this could undermine their privacy. The typical grieving process could also be fundamentally altered.
A separate paper by one of the authors explores a related question: What happens to a person’s agent if the owner unexpectedly dies or loses their access credentials? If the agent is set up on the blockchain, it may continue operating, suggesting a need for named human inheritors.
This chimes with a concern expressed by the AI policy expert and writer Dean Ball, who recently argued that such ‘rogue agents’ are inevitable and require a policy response. Current agents depend on compute and human steering, but ‘self-sovereign agents’ could emerge with multiple versions spread across different compute providers and no single kill switch. If they need to earn money, they may turn to cybercrime. Among the possible responses are unique agent identifiers and the power to blacklist agents to freeze their resources.
