1. Identifying (and incentivising) AI agents

• What happened: A trio of law and philosophy professors considered how to identify who (or what) is responsible for AI agents’ actions in the world, and came up with a two-part proposal: that the disparate and evolving agents within a system should exist legally as a new form of corporation; and that each corporation should link to accountable humans.

• What’s interesting: The paper by Yonathan Arbel, Simon Goldstein, and Peter N. Salib starts with a thought experiment. It’s 2030, and your AI assistant suggests that it optimizes your slow WiFi connection. After you agree, it spawns a swarm of agents. Some are copies, while others are cheaper agents running on open-source models. Some start to interface with AI agents from other companies. Three months later, two FBI agents knock on your door and explain that your network has been piggybacking on a local defense contractor’s WiFi network.

• Before determining who is responsible and what the repercussions should be, there are more basic questions: Who are the AI actors in this story? How many are there?

• The economy will soon be filled with capable AI agents. To deter and respond to such harms, the authors argue that we need to be able to identify these agents, at two levels.

  • To prevent human misuse or negligence, we need ‘thin identity’. This would connect AI agents to the humans most able to control them, similar to how ‘know-your-customer’ rules tie banking transactions to humans.

  • Humans will be unable to monitor and control every AI decision, so we also need to be able to identify agents themselves, hold them accountable and incentivize them to behave well. To do so, we need ‘thick identity’ that can distinguish AI agents as stable, coherent entities, with persistent goals. This goal is pragmatic and does not require viewing AIs as conscious in any sense.

• Thickly identifying agents is harder and more novel, as AI agents need not be attached to a physical body. Multiple agents can also work together on a single task. Any single agent can be copied, spun up, spun down, or be continually updated.

• To address such challenges, the authors propose creating algorithmic corporations, or ‘A-corps’. These would have two key elements:

  • Legal personhood: Like a traditional corporation, an A-corp would be a single legal entity that persists over time. It could hold property, make contracts, and be sued. But it would be run by a collection of AI agents. As such, the proposal runs contrary to scholars who have argued against granting legal personhood to AI agents, or called for bans on algorithms running companies because of concerns about crime and companies using them to avoid liability.

  • Computationally-secure governance: Each A-corp would have a unique digital certificate and a secure private key to authorise transactions. The humans that own each A-corp could grant the key to an AI ‘manager’ agent who in turn could grant more limited permissions to sub-agents within the A-corp, or to other A-corps, such as permissions to spend up to $100 or to read a batch of emails.

• The proposal addresses thin identity by reducing the vast number of AI agents down to a smaller number of A-corps, whose actions are traceable back to their human owners. As with limited liability companies (LLCs), the human owners would not be responsible for all harm their A-corps cause, but could lose all funds they invest and possibly face further liability, for example in cases of fraud or negligence.

• The proposal addresses thick identity via its ‘resource constraint thesis’. All AI agents need resources, like money and compute. A-corps provide AIs with a way to access these resources and an incentive to manage them well. For example, A-corps that tightly monitor and audit their sub-agents’ performance would get more resources, while A-corps that allow fraud or waste will lose resources. This encourages A-corps to self-organise, into stable, coherent, multi-agent systems.

• The authors argue that A-corps could also address alignment concerns, for example by reducing the incentive for an AI agent to exfiltrate its own weights, because that new AI instance would lose access to resources and permissions from the A-corp.

• To make it happen, the authors call for a public registry of A-corps. This would list each A-corp’s human owners, the certificates to authenticate it against, as well as (potentially) the differing permissions enjoyed by its agents. Ultimately, the authors argue that A-corps should become mandatory for any AI agent taking “economically significant actions”, and to guard against criminals using AI agents anonymously.

• The authors respond to some expected pushback. They do not see A-corps as anthropomorphising AI because the proposal does not require anybody to view agents as having deeper desires or wants. They also think A-corps can prevent the risk that AI agents might slowly build up resources before deploying them for harm, by encouraging inter-agent trade that penalises rogue behavior. Could A-corps disempower humans? The authors argue that they provide a pathway to tax and redistribution, and enable humans to better steer agents, for example by designating the parts of the economy that A-corps are permitted to operate in.

2. When AI builds AI

• What happened: The Centre for Security and Emerging Technology, CSET, released a report on the prospects for AI improving itself, known as automated R&D or recursive self-improvement, based on an expert workshop in July 2025.

• What’s interesting: In 1964, the computer scientist I.J. Good wrote about the possibility of an “intelligence explosion” that would leave “the intelligence of man.…far behind”. Researchers have also long automated aspects of writing code and AI model design.

• However, the speed of AI coding advances suggests that something qualitatively different may soon occur. This makes two questions salient: 1. Could AI automate the entire AI R&D process? 2. Will this R&D automation extend across all scientific disciplines? The CSET report focuses on the first question.

• CSET defines AI R&D by distinguishing between research scientists, who generate hypotheses, design experiments and interpret results; and research engineers, who write code, fix bugs and generate data. They also note the inputs that AI R&D relies on, such as raising funds and acquiring compute.

• They sketch out four overlapping scenarios for how AI R&D may play out:

  • 1. Explosion: AI systems automate a growing share of AI R&D. Initially, this leads to modest productivity gains, but as the length and complexity of tasks that AI performs grows, productivity soars. AI systems become far more capable than humans, whose involvement in AI R&D falls to zero.

  • 2. Fizzle: The share of R&D tasks done by AI rises, but rather than leading to compounding improvements, capabilities start to plateau.

  • 3. Amdahl’s Law: AI automates certain activities, like writing code and running experiments, but not others, like research strategy.

  • 4. The expanding pie: As AI automation grows, humans realise that new ideas and breakthroughs are needed that AI systems cannot yet provide.

• The experts in CSET’s workshop held widely diverging views on which scenario was most likely. Most importantly, new empirical data is unlikely to resolve these conflicts, because participants may view the same data as confirming their own assumptions.

  • For example, an AI system’s inability to reliably use a keyboard or mouse may look like a bottleneck to one expert, but a source of explosive growth to another—if they expect this human-focussed tooling to get adapted for the AI era. Similarly, different experts may view AI automating a growing share of R&D tasks as progress towards a fast takeoff, or as low-hanging fruit being picked off, accelerating progress only as far as the upcoming wall.

• These differing views are also visible in more recent commentary on the topic.

  • The prominent AI researcher and writer Nathan Lambert recently cited Paul Allen concept of a ‘complexity break’ to argue that as we understand intelligence better, further progress becomes exponentially harder. In addition to incurring financial costs, Lambert argued that running suites of AI agents won’t necessarily lead to exponential progress, because those agents will perform best on narrow, verifiable tasks, will be hard to manage in large numbers, and will sample from similar parts of the distribution of AI research ideas, inhibiting more novel breakthroughs.

  • Conversely, Ajeya Cotra at METR, the Model Evaluation and Threat Research organisation, recently wrote about how she “underestimated AI capabilities (again)”. She argued that AIs may, counterintuitively, find it easier to decompose longer projects into sub-components that multiple agents can run in parallel, than for shorter tasks. AIs will also produce good documentation for their fellow AIs, which could accelerate progress.

• If faster automation and progress does occur, the CSET authors see two main risks: Less time to prepare for safety risks from AI, and lower human understanding of AI systems. To address these risks, their recommendations have a strong focus on improving access to evidence, including:

  • New evaluations of AI R&D, including for ‘messy’ tasks such as research strategy, which lack clear specifications and success criteria and take place in a dynamic environment with various real-world interactions.

  • New approaches to evaluation to better distinguish ‘degrees of accomplishment’ from a simple success/failure binary.

  • Better insights into how automated R&D is progressing within AI labs, such as data on how funding is allocated and qualitative impressions of progress from leading AI researchers and engineers.

3. Planes and global warming

• What happened: A team of researchers, including from Google and American Airlines, published results from their latest experiment to use AI to reduce condensation trails from planes—a key contributor to global warming.

• What’s interesting: When pilots fly, particles from the plane’s exhaust can mix with low-pressure air to form contrails—white, artificial clouds, made up of ice crystals. These contrails are a net contributor to global warming, because they trap heat that would otherwise escape. Debates continue over exactly how much they contribute, but one estimate suggests that they contribute a lot, causing around 2% of ‘radiative forcing’, which measures how different factors, like CO², heat or cool the planet.

• As the environmental writer Hannah Ritchie explains, more important than the absolute figure is the fact that contrails offer a rare opportunity to reduce global warming almost immediately, at relatively low cost. This is because a small share of flights cause most of the warming-inducing contrails—generally those that fly through parts of the atmosphere that are both very cold and very humid. If planes take short detours to avoid these patches of air, contrails (and warming) should drop.

• A few years ago, Google researchers partnered with American Airlines on a proof of concept. Using satellite imagery and AI, they were able to predict where contrails would emerge and guide planes to avoid them, reducing contrails by >50%, across 70 test flights.

• In the latest study, they expanded the experiment to 2,400 American Airlines flights from the US to Europe. They placed ~50% of planes in a treatment group, where flight dispatchers were given two choices: a standard flight plan and an alternative contrail-avoidance one. Their decision for which to recommend was voluntary.

• For flights in this intervention group, contrails fell by 12% compared to a control group with no contrail-avoidance plan. Importantly, the contrail-avoidance routes also did not lead to a significant increase in fuel use. At first glance, these results seem positive, but modest. Digging into the results highlights the challenge of getting useful AI deployed at scale.

• In particular, dispatchers who received contrail avoidance plans only recommended them to pilots 15% of the time. Even then, the avoidance plan was only successfully flown in 60% of flights. For planes that did successfully follow the avoidance plan, contrails fell by more than 60%, a much larger reduction. So the tech worked, but was often not used.

• Why? Dispatchers are busy and must often deal with other priorities, like bad weather and turbulence. To avoid contrails, planes also need to climb and descend mid-flight. This is safe, but creates more work for pilots and air traffic controllers. As it was voluntary, the incentive to change to a contrail-avoidance plan was weak.

• The way that the dispatchers received the information also meant that they didn’t fully understand why the suggested up and down changes were necessary. Happily, the authors feel that most of these obstacles are addressable, with a combination of a better user interface, some automation, and more incentives.

• In addition to its immediate usefulness, the study is a rare real-world attempt to quantify the benefits of AI to tackling global warming. At the moment, the AI and climate change policy discussion is often negative and focuses on the emissions that may result from building and operating data centres (and other devices) to train and run AI models. This is important, but there are reasons to think that these emissions will be relatively low, or at least lower than many assume. In contrast, AI could potentially reduce emissions and warming by far larger amounts, for example by accelerating research on solar and fusion power, or making buildings and energy grids more efficient. But these benefits are typically more speculative, harder to quantify, or in the case of contrails, more contingent on human behaviour.

• This experiment demonstrates that the benefits of AI to tackling global warming are real, but also points to the interventions that will be needed to push them to their full potential. The study is also timely, given that governments are focussing on contrail avoidance and some policy action may be required, for example to help standardise and mandate contrail prediction software or to generate high-resolution humidity data.

Share

1. LLMs are making it easier for scientists to write papers, for better or worse

• What happened: A team at Cornell and Berkeley investigated how scientists are using LLMs to help write papers, and what this means for the future volume, quality and fairness of research.

• What’s interesting: The authors built a dataset of ~2.1 million preprints from arXiv, bioRxiv and SSRN, between 2018-2024. To detect whether scientists had used AI to help write a paper, the team compared the distribution of words in the abstract against human- and LLM-written baselines. When an author’s paper hit a threshold on this “AI detection” metric, they were labelled as an “AI adopter”. According to the study, LLM adopters subsequently enjoyed a major productivity boost, compared with non-adopters with similar profiles, publishing 36-60% more frequently. The gains were particularly large for researchers with Asian names at Asian institutions.

• The team also assessed the complexity of the writing, using measures like Flesch Reading Ease, which evaluates sentence length and the number of syllables per word. They found that human-written papers with more complex language were more likely to be subsequently accepted by peer-reviewed journals or conferences—suggesting that, for humans, writing complexity is an (imperfect) signal of research effort and quality. For LLM-assisted papers, the relationship was inverted, with the authors concluding that the polished text of LLMs is helping to disguise lower-quality work. (They validated the findings against a separate dataset).

• The authors also used the launch of Bing Chat, an LLM-based search engine, in 2023 to conduct a natural experiment. They compared views and downloads on arXiv that Bing Chat had referred, to those that Google Search referred. Bing Chat was more likely to refer scientists to newer and less-cited literature, as well as to books, possibly because LLMs are better able to parse long documents or a larger number of documents. (They also validated this finding with a separate dataset, although we don’t know how good the new sources cited by Bing were).

• As the authors note, their study has a number of limitations. Their AI detection method is imperfect, only looks at abstracts, and doesn’t capture authors who may have edited LLM-generated text. There are also various potential confounders: maybe less experienced researchers are more likely to use LLMs? That said, the findings highlight (at least) three major questions posed by the growing integration of AI into science:

  • First, AI is leading to a big increase in the supply of papers (and grant applications). This poses a challenge for preprint repositories, which don’t want to host slop. ArXiv, whose founder Paul Ginsparg is a co-author of this study, recently banned computer science review and position papers, citing a surge in low-quality AI papers. LLM-assisted papers also pose a challenge for peer reviewers, who are already under strain, and are typically prohibited from using AI, although many do so anyway. This seems unsustainable. As the authors of this study suggest, it is likely time to consider how to integrate AI into at least some aspects of the peer-review process.

  • Second, the findings illustrate how LLMs may both mitigate and exacerbate fairness issues in science. For some scientists, the complexity of their writing may be a reliable indicator of their thinking and effort. For others, particularly non-native English speakers, writing may be more of an obstacle that has previously penalised them. A hopeful outcome is that LLMs may ease that burden. But a more worrying outcome is that, if reviewers and readers can no longer rely on writing complexity as an (albeit unfair) signal of good work, they may fall back on (even more unfair) signals, such as the institution that a person works at. This challenge is not limited to science, and may also occur in other areas where writing serves this purpose, like with cover letters.

  • Finally, the finding that LLM-based search engines may increase the diversity of sources that researchers review is the opposite of what some suggested would happen: that AI models would continually cite the same high-profile studies, exacerbating the “Matthew effect”.

• Collectively, the study serves as a reminder that for every concerning scenario about the integration of AI into science, there are plausible counter-scenarios. Will AI lessen scientific reliability because of hallucinations? Or will AI “review agents” and AI-supported evidence reviews reduce (the many) inaccuracies that are already in the evidence base? Will AI remove the intuitive and serendipitous ideas that humans come up with? Or will AI enable scientists to pursue more novel hypotheses? Ultimately, AI could well upend the standard processes and traditions of science but do so in a way that delivers fresh benefits. To know if and how that is occurring, we need more empirical evidence about how AI is changing science.

2. Lessons from two years of AI safety evaluations

• What Happened: In December, the UK AI Security Institute shared a set of trends observed since they started to evaluate frontier AI systems in November 2023.

• What’s Interesting:

  • The report features more than 60 authors, a testament to the deep expertise that AISI has built up. Their trends are based on their evaluations of more than 30 frontier AI systems, with methodologies ranging from asking those AI systems questions to adversarially red-teaming them.

  • Their headline finding is striking, if unsurprising: AI capabilities have rapidly improved across all the domains that AISI tests. In the cyber domain, AI models and agents can now successfully complete more than 40% of the 1-hour software tasks they are tested on, up from <5% in 2023. Last year, a model completed an “expert-level” cyber task for the first time. In biology and chemistry, AI has gone from significantly underperforming PhD-level human experts at troubleshooting experiments, to significantly outperforming them, including for requests about images.

  • On the risk that AI models may “self-replicate” in a way that subverts human control, AISI’s evaluations suggest that AI agents have gotten better at simplified versions of some tasks that could be instrumental to self-replication, such as passing know-your-customer checks to access financial services, but less so at others, like retaining access to compute and deploying successor agents. AISI’s evaluations also suggest that models are capable of deliberately obstructing attempts to measure their true capabilities (“sandbagging”), but only when explicitly prompted to do so.

  • The report also sheds light on AI systems’ limitations. In the cyber domain, AISI notes that AI systems still struggle in open-ended environments where they must complete long sequences of actions autonomously. Similarly, regarding chembio threats, biologists and chemists, and potential threat actors, need “tacit” knowledge and expertise, such as how to pipette. AISI’s evaluations to date have focussed more on explicit knowledge although they plan to share more on wet lab tasks.

  • When it comes to mitigations, the report provides both reassurance and concern. On one hand, the safeguards that leading labs have introduced have made their models safer, in one instance increasing the amount of expert effort needed to jailbreak a model by 40x. On the other hand, AISI says that it was still able to find a vulnerability in every AI system it tested. Worryingly, AISI also found no notable correlation between how capable a model is, and the strength of safeguards it has in place.

  • AISI also sheds light on two other sources of AI risk: open source and scaffolding. They argue that the performance gap between open source and proprietary AI models has narrowed. This introduces risks as safeguards for open models (where they exist) can be removed, and jailbreaks are hard to patch. AISI also found that scaffolding can make AI agents more capable than the underlying base AI models, even if those gaps later narrow when the base models are updated. Some complex scaffolds are in proprietary products, such as coding agents, but others are in open-source efforts.

  • The report also touches on AISI’s evaluations of the broader societal impacts of AI, such as the degree to which people are using AI to access political information, or the risks of harmful manipulation. One striking statistic, picked up in media coverage of the report, was that one-third of UK respondents to a recent AISI survey had used AI for emotional support or social interaction in the preceding year, although just 4% do so daily. In a separate effort, AISI found that some dedicated AI companion users reported signs of “withdrawal” during outages.

  • Overall, AISI argues that AI labs are taking an uneven approach to safety, focussing more on safeguards for biosecurity risks, for example, than for other threats. This is arguably true of AISI as well, given their strong focus on biology and chemical risks rather than radiological or nuclear risks. This raises a question: Given finite resources, what evaluations of frontier AI systems are most lacking in the current landscape?

3. One in four UK doctors are using AI in their clinical practice

• What happened: The Nuffield Trust and the Royal College of General Practitioners surveyed more than 2,000 UK GPs to understand how they view and use AI, in what the authors called the largest and most up-to-date survey on the topic.

• What’s interesting:

  • 28% of UK GPs now use AI. This is up from ~10% in 2018, but below the rates seen in some other UK professions. According to the survey, the GPs most likely to use AI are younger, male, and work in more affluent areas. This is similar to disparities in the wider public’s use of LLMs, although there, the early gender gap may have narrowed.

  • Just over half of AI-using GPs procure AI tools themselves rather than relying on those that their practices select. This kind of “shadow AI use” is not unique to GPs, but a Nuffield focus group sheds light on why UK GPs feel compelled to do it: some GP practices or Integrated Care Boards ban AI tools, while others are slow to respond to GPs’ requests and instead prefer to stick with legacy digital tools.

  • UK GPs mainly use AI for clinical documentation and note-taking. Some say that AI note-taking allows them to look at, and speak more, with their patients, a non-trivial benefit given that the UK public worries about AI making healthcare staff more distant.

  • GPs also use LLMs to produce documents, from translations of patient communications to referral letters; and to stay abreast of new research, with some younger practitioners turning to LLM “study modes” to help with their mandatory professional development.

  • GPs cite “saving time” as the primary benefit of AI, and mainly use this to reduce overtime, rest, and engage in professional development, rather than to see more patients. This is notable as the UK government wants AI to reduce the wait time to get a GP appointment, which is a top concern for the public. These findings suggest that more nuanced evaluations of AI’s impact on GP services will be needed.

  • GPs worry about errors and liability issues with AI. As a result, the authors call on tech suppliers to do better evaluations of hallucinations. Ideally, such evaluations would compare the accuracy of AI, human and hybrid outputs in real-world settings, and all the nuances that might entail. For example, when explaining the benefits of AI note-taking, some GPs pointed out that certain colleagues can’t touch type and so, without AI, struggle to capture all the details in a patient consultation ( this is, presumably, a form of inaccuracy).

  • Use of AI for more complex “clinical support” tasks remains relatively low, owing to GPs’ concerns about errors, their desire to retain control over clinical judgement, and a lack of regulatory approval. However, some GPs did report using AI, or wanting to use future systems, to help check diagnoses, formulate care plans, and analyse lab results.

  • This suggests that more GPs may start to use AI to enhance their own clinical judgement, spurred by a growing body of evidence that LLM-based systems may be useful in this area, and by the public’s own growing use of LLMs for answering medical questions.

  • In their recommendations, the Nuffield authors call for clearer guidelines and regulatory frameworks for GPs, including as part of the UK’s new National Commission into the Regulation of AI in Healthcare. However, the report also acknowledges that much guidance already exists, such as the British Medical Association’s AI principles and the NHS guidance on AI note-taking (which some GPs appear to be breaking by procuring their own tools). This raises a question: what exactly should any new guidance stipulate? How to get the burden on GPs right? And how to ensure that they are actually following it?

1. The AI agent economy

• What happened: In September, researchers at Google DeepMind published a paper examining how AI agents might be integrated into the economy. Gillian Hadfield and Andrew Koh also published a paper on the implications of AI agent economies. The papers were timely, with Google and OpenAI recently launching new protocols to enable agents to make payments online.

• What’s interesting: AI economic impact debates often position the technology as a ‘tool’ and focus on how it may affect employees’ productivity or job prospects. This overlooks the more radical ways that AI agents may change what we even mean by the ‘economy’ or ‘economic actors’.

• Deploying AI agents at scale will be hard. It will require capability improvements and overcoming barriers from legacy infrastructure to the tacit ‘grey knowledge’ that humans use to navigate organisations. But there are routes to doing so and the GDM paper argues that as AI agents become more capable and interconnected they will begin to transact with each other, at scale and speeds beyond direct human oversight.

• One way to analyse the potential effects of this is to compare AI agents with humans. LLMs are trained on economic textbooks and early research suggests that some AI agent behaviour may be consistent with that of humans, whether in terms of maximising expected utility or displaying common behavioural biases. However, Hadfield and Koh argue that evaluations of AI agent behaviour are weak and that agents may lead to novel behaviours and impacts, particularly via multi-agent systems:

  • For example, when it comes to customer welfare, AI agents could be effective personal shoppers, searching widely and continually checking prices. This could lead to better outcomes for consumers, but only if agents correctly infer human preferences and avoid biases towards certain marketplaces. Less positively, agents may develop exploitative strategies, such as seller agents generating a large number of fake reviews to mislead buyer agents, that exacerbate fraud.

  • When it comes to market power, AI agents could help buyers seek out novel competitors, but may also reduce the transaction costs and communication challenges that normally prevent any one firm from becoming too large.

  • AI agents may also exacerbate inequality, as superior agents - equipped with better compute, data, and models - could engage in ‘high frequency negotiation’ on behalf of their higher-income users. AI agents may also pose a greater risk of collusion than their human counterparts, as the reinforcement learning algorithms used to train them can cause a bias that leads agents to insufficiently explore “off-path” strategies.

• Given these challenges, how should we respond? The GDM paper argues that the default response will be to allow the agents to fully permeate the human-led economy, in an emergent or spontaneous manner, with limited safeguards. The authors argue that we should instead aim to prescriptively demarcate (or ‘sandbox’) agents in a controlled sector or section of the economy. This would give policymakers and researchers the opportunity to test them before they are deployed more widely.

• They also propose various other policy ideas:

  • Inspired by Ronald Dworkin’s principles of distributive justice, they propose granting every human user an equal initial endowment of a virtual currency to bid for compute, tools, or priority execution slots on behalf of their agent. They also propose using incentives to steer agents towards socially useful ‘missions’, such as accelerating scientific discovery or tackling climate change.

  • The authors also lay out various technical ideas, such as identifiers for each agent, verifiable credentials that allow agents to build a ‘tamper-proof’ reputation, a ‘proof-of-personhood’ that links digital accounts to unique human beings, and standards that encourage interoperability between agents.

  • They also propose a hybrid oversight infrastructure that uses AI for real-time monitoring before escalating cases to human experts.

• A view from the field: What are you excited or worried about with respect to AI agents and the economy? Andrey Fradkin & Seth Benzell, from the Justified Priors podcast:

  • Andrey: “I am excited by the ways in which markets can be redesigned for AI agents in a way that makes people better off. For example, in the car market, can we create the infrastructure so that a buyer AI agent can find and negotiate a good deal on a car with a lot less human effort?”

  • Seth: “In a world where agents and robots can do anything, output will be determined by the level of capital investment. In such a world, the most important growth policy will be national savings policy. High consumption for Boomers and Gen X would require investing less in the future, at exponentially compounding cost to their children. Intergenerational conflict will become more salient.”

2. Nine ideas to accelerate science with AI

• What Happened: In August, the Institute for Progress published nine policy ideas to accelerate the use of AI in science.

• What’s Interesting: The evidence is growing that science will be one of the domains where AI will yield its greatest benefit to society. Governments are paying more attention. The EU just released an AI for Science strategy, the UK is working on their own strategy, and the US is prioritising science in their new AI Action Plan. But the policies to pursue are not obvious. How is AI for Science policy different from standard science policy? Or from AI policy? What might an ambitious role for government look like?

• The IFP provides nine ideas, with a focus on the US. Some aim to improve how science functions, such as using AI agents to replicate scientific papers. Others propose new kinds of organisations, such as self-driving labs to validate new AI-designed materials; ‘X-Labs’ to work on more ambitious AI projects than grant funding normally allows; and a new office to commission AI for science ‘grand challenges’ and evaluations.

• The success of AI for Science efforts will hinge on the availability of a core set of ingredients, the most important of which is data. This is where most of the IFP ideas focus. Adam Marblestone and Andrew Payne propose creating maps of five small mammal brains, such as laboratory mice, to better understand behaviours that we would like AI systems to learn, such as cooperation. Maxwell Tabarrok proposes a public database of one million antimicrobial peptides, to train AI models to tackle antibiotic resistance.

• Three other ideas focus on better leveraging the data that already exists, but is inaccessible to most scientists.

  • Andrew Trask and Lacey Strahm lay out an ‘Attribution-Based Control’ system that would allow owners of healthcare, financial, and industrial sensor data to specify AI models that could access it.

  • Ruxandra Teslo argues that LLMs grant an advantage to large pharmaceutical firms that can draw on their historical archives of new drug applications to create ‘AI copilots’, an opportunity that is unavailable to most startups. In response, she proposes a new entity to monitor biotech bankruptcy cases and buy up ‘orphaned’ regulatory dossiers and clinical trial data, before anonymising and open-sourcing it.

  • Ben Reinhardt argues that most of today’s AI for Science models are trained on ‘clean’ curated datasets and scientific papers. This privileges the final outcome of science research and overlooks the messy process of doing it. In response, he proposes creating ‘Unstructured Data Generation Labs’ where scientists would carry out research in fields like biotech and materials science, record themselves using everything from bodycams to equipment sensors, and then use that data to train AI models.

• A view from the field: What AI for Science policy idea are you passionate about? Stuart Buck, The Good Science Project:

  • Stuart: “One policy that could accelerate AI in science isn’t about AI per se: Funders should sponsor many more direct replications, including in collaboration with the original labs. The reason: so much about science involves both tacit knowledge (which can’t be articulated) and unwritten knowledge (which can be articulated, but is so routine that no one even thinks to mention it). Most of that knowledge isn’t accessible to AI currently, but if we carried out more direct replications in tandem with AI tools, we could make quicker progress towards figuring out all of the unseen and unwritten factors that explain why an experiment reached the results it did. See this essay from the Good Science Project.”

3. Using LLMs for political guidance

• What happened: Researchers at the UK AI Security Institute published results from a randomised controlled trial which found that individuals who used LLMs to research political information before the 2024 UK election were subsequently less likely to believe false information.

• What’s interesting: The authors first ran a survey of ~2,500 UK adults and found that 9% of voters, or ~1/3rd of chatbot users, used LLMs to get political information in the week before the July 2024 election. Most LLM users found the models useful and accurate.

• The share of chatbot users who used the models to get political information is quite high, and is likely higher again in October 2025. But LLMs are still well behind other sources of political information, such as television, social media, and search engines.

• The authors then ran an RCT of UK residents. The first group was given access to an LLM and asked to research issues of concern to the UK election, such as climate change, immigration, criminal justice and Covid-19 policy. The control group was given access to a search engine. The study found that both groups subsequently showed similar declines in belief in false information and similar increases in belief in true facts. The main difference was efficiency - the LLM group completed their task 6-10% quicker. The results held across different AI models (GPT-4o, Claude, Mistral) and also held when the models were prompted to be more sycophantic.

• The results suggest that widespread concerns about LLMs exacerbating political misinformation may be misplaced, which in turn may reflect how hallucination rates have dropped over the past two years. The speed of LLMs also means that they could potentially help to debunk fast-spreading misinformation more quickly, enabling what the authors describe as “rapid, reliable public learning during high-stakes events.”

• The results also challenge the common survey finding that the ‘public doesn’t trust’ AI. Rather, the authors find that ‘information seeking’, including for political information, is one of the public’s main AI use cases. This highlights the need to judge public attitudes to AI based on ‘revealed’ as well as ‘stated’ preferences.

• The study comes with caveats and limitations. It focussed on one country and only tested a small number of models - others may be more likely to generate political misinformation.

• The study also evaluates LLMs by comparing them against an antecedent technology: a non-AI search engine. But as the authors note, such comparisons are increasingly difficult as LLMs are now integrated more directly into the search experience. This will make it harder to know what baseline to compare future LLMs against.

• A view from the field: How do you see AI changing access to political information? Tom Rachman, Google DeepMind:

  • Tom: “How AIs will remake the news ecosystem is a matter of vast import to democracies. Each person’s evaluation of information is mediated by their trust in its source, particularly for political content. One could envisage a future in which different AI models gain specific political reputations, affecting their influence as sources. Another plausible future could see personalised AI agents as everyone’s fundamental font. This study, among other ambitious experiments led by researchers at the AI Security Institute, helps establish a baseline effect as we wait for new information paradigms to crystallise.”

Bonus: AI Policy Fellows of the World, Unite!

Every year, AI fellowships send fresh outstanding minds into the world of policy. But fellowships are more than points of transit; they are sources of valuable research. To highlight this, we scanned recent projects from leading programs at the Centre for the Governance of AI, ERA Cambridge, Pivotal, MATS and PIBBSS. What struck us was the sheer range of insightful work. While we cannot list all the excellent contributions, here are three that caught our attention:

• Jacob Schaal, a Cambridge ERA fellow, extracted economic insight from Moravec’s Paradox that what is easy for AI is hard for humans, and vice versa, conceiving a new way to judge which jobs are most exposed to automation. Management and STEM occupations, he found, face the highest automation exposure. Interested in more? Ask for details from Jacob at jacobvschaal@gmail.com

• Said Saillant, a GovAI summer fellow who recently joined UNIDO’s Innovation Lab, developed the concept of AI-ready special economic zones, or AI-SEZ, for Latin America and regulatory sandbox models for the UK. His work focused on adaptive regulation to accelerate safe AI diffusion. Interested in more? Ask for details from Said at ssaillant@societassapiens.org

• Joël Naoki Christoph, a fellow at GovAI, argued that middle powers chasing “sovereign compute” are walking into a costly trap because such projects fail to achieve real autonomy. Instead, he advocates for “managed dependency,” allowing nations to avoid fruitless expenditure and reduce foreign leverage. Interested in more? Ask for details from Joël at jchristoph@hks.harvard.edu

Study Watch

AI’s impact on jobs will depend on how it affects ‘expertise’

• What happened: The economists David Autor and Neil Thompson published a paper in which they argue that the concept of ‘expertise’ explains why, over the past 40 years, new technologies have led to higher wages and lower employment for certain US roles, and the inverse for others. They expect similar dynamics in the AI era.

• What’s interesting: Autor and others previously coined the idea of ‘skills-based technological change’ to describe how new technologies complement some workers, while displacing others. What causes this? Autor and Thompson provide a partial answer via the concept of ‘expertise’, which is also the full title of their new paper.

• In Autor and Thompson’s model, expertise describes a worker’s ability to perform tasks that others cannot. They view expertise as hierarchical — a senior surgeon can do their own job, but they could also perform the tasks of a junior nurse (e.g. taking blood pressure). However, the junior nurse cannot perform complex surgery.

• They turn this definition into a statistical assessment of the relevant expertise of different tasks and jobs, by assessing the frequency and entropy of words used to describe them. Words like ‘elasticity’, which tend to be used rarely and in specific contexts, like economics, are more likely to describe high-expertise tasks.

• Armed with this definition and statistical measure, Autor and Thompson assess ~40 years of US job data to understand how new technologies have affected different jobs. They find that when technology automates a task, the effects on workers hinge on whether that task requires lower or higher expertise.

  • When technology is complementary, it augments experts by automating inexpert tasks. For example, computers automated data entry tasks, allowing accounting clerks to shift to more complex, specialised analysis. Wages rose, from an average of ~$13 in 1980 to ~$18 in 2018, but overall employment fell, from 1.6m to 1.1m, as the pool of qualified workers shrank.

  • Conversely, when technology removes the most complex expert tasks from roles, it diminishes or displaces the value of experts. For inventory clerks, computers automated more specialised pricing tasks, reducing the expertise of the job. Wages fell, from ~$14 in 1980 to ~$12 in 2018, but overall employment rose, from 0.5m workers to 1.5m.

• What does this mean for AI’s future effects? The authors argue that we should focus less on wondering whether AI will automate certain jobs, and more on the relative expertise of the tasks that AI will automate, within jobs.

• Think of any job (perhaps the one that you do!). An AI policy researcher may read papers, build relationships with experts, and write analysis and recommendations for internal and external audiences. Today, that researcher can use an LLM to assist with literature reviews, preparing expert interviews, or reviewing their writing, freeing them up to focus on the more expert tasks that LLMs cannot yet do. Per Autor and Thompson’s theory, the LLM is likely making the AI policy researcher more productive, and their expertise more valuable. But it may also impose barriers to entry for new entrants by automating some of the traditional tasks they might have worked on. As the models continue to improve, the calculus could change. If they begin to encroach on tasks that currently require more expertise, organisations may find it more efficient to hire more 'inexpert’ employees to use the more powerful LLMs, or rely more on AI agents in lieu of human employees.

• Practitioners are exploring various evaluations to understand AI’s potential impact on employment. This paper suggests that one goal should be assessing the relative ‘expertise’ of the tasks that AI systems can do, in an economic context, and what that means for workers with that expertise. The paper also raises questions about the underlying concept of expertise that might merit future study, including:

  • Is expertise always hierarchical? Or are there tasks that a more junior employee may excel at, such as serving customers, where a more senior executive would struggle?

  • Do job task descriptions capture what makes employees valuable? Or are there other forms of expertise and value that are less visible, such as organisational knowledge, trust, and judgement?

  • How to account for wider economic trends? Autor and Thompson acknowledge that some of the trends they highlight are ‘almost surely explained by a combination of automation and international trade.’ Beyond trade, demands for expertise in a given location will also depend on factors like outsourcing, policy shifts, new business models, and wider demand trends in the economy.

  • How to best develop expertise? Autor and Thompson’s analysis also raises the question of how people can develop the right expertise to capture the complementary benefits of AI. As we recently analysed, this is where further research on the role of worker retraining would be valuable - as well as a sober understanding of where retraining may be insufficient.

• View from the field: How do you see AI affecting demand for expertise? Maria del Rio Chanona, University College London

  • “Whether AI expands or contracts expertise depends on the type of tasks it affects. Our research shows that for substitutable tasks like content writing and translation, AI decreases demand across all levels of expertise - i.e. the technology may be good enough to replace what previously required top-tier human expertise. For most complementary work, like JavaScript coding, HTML development, or general programming projects, we see the opposite pattern: AI raises the expertise bar by eliminating demand for novice workers while experienced developers remain sought after. It's worth noting that our findings focus on labor demand patterns rather than wages or employment outcomes - we're observing how employers' willingness to hire for different types of expertise is shifting, which may precede but doesn't necessarily translate directly to changes in worker compensation or employment levels.”

Policymakers taking action

The UK Metascience Unit shares early results and AI plans

• What Happened: The UK Metascience Unit shared results from their early experiments to reform UK science, and spotlights upcoming work, including a strong focus on AI.

• What’s Interesting: Metascience, or ‘the science of science itself’, uses research, data and experiments to improve how science works.

  • As detailed in an appendix, this desire to understand and shape science has a long history. In 1939, the pioneering X-ray crystallographer and Marxist J.D. Bernal published ‘The Social Function of Science’. In it, Bernal analysed British science “as if viewing it through his microscope”, assessing everything from its funding and organisation to its role in industry and war.

  • In subsequent decades, the UK continued to advance the foundation of modern metascience via a ‘golden triangle’ of universities in Sussex, Manchester and Edinburgh. More recently, metascience has served as a meeting place for individuals with varied objectives, from supporting open science to improving replication - the latter goal electrified by John Ioannidis’s 2005 paper, ‘Why Most Published Research Findings Are False’.

  • The greatest metascience spur is the fact that science is now extremely large - ~$2.5 trillion per year - and creaking. Observers warn of slow and conservative funding processes, a broken peer review system, and a continued over-reliance on metrics - such as publications and citations - that create perverse incentives.

• Various actors hope to use metascience to address these issues, while taking care to ensure that their work is not used to justify ill-founded cuts in scientific research. Among these actors, the UK Metascience Unit is the first of its kind to be embedded within both a central government department (DSIT) and the country’s largest research funder (UKRI). This gives it the potential to directly translate experimental findings into national policy. Their ~£3-4m budget is small, accounting for just ~0.03% of UKRI’s, but they have also secured funding from third-party sources, such as Open Philanthropy.

• How are they spending this money? In their first year, they focussed on improving the processes used to allocate research funding and ran a successful trial of ‘distributed peer review’ — a format where funding applicants must also agree to review other applications, and which gained prominence in the scientific world after a canonical experiment to allocate ‘telescope time without tears’. In their experiment, the Unit used their own AI Metascience Fellowship programme as a test case and took steps to address potential concerns, such as gaming issues. They found that, overall, distributed peer review shortened the assessment process, reduced the admin burden, and improved participants' knowledge of the field.

• The Unit also ran simulations and trials of ‘partial randomisation’ - a process that subjects ‘middle-ranking’ grant applications to a lottery process, in the hope of encouraging more novelty, risk-taking, and efficiency. However, the Unit found that the evidence for such randomisation is not yet sufficient.

• In the year ahead, AI will be a major focus for the Unit. In particular:

  • Their 18 early career fellows will study how AI is affecting science. As we touched on in a past essay, which the Unit cites, the questions here are vast - from how AI is affecting the methods that scientists use, and the pace of scientific progress, to AI’s effects on scientific creativity and understanding.

  • The Unit has also allocated grants to researchers working on specific AI-related questions, such as to assess if LLMs can reliably review academic research, and if AI could prevent problematic randomised controlled trials from being included in systematic reviews, where they can hurt patients.

  • The Unit is also co-running a global competition to find and validate AI-driven indicators of ‘scientific novelty’, to support efforts to understand if novelty truly is lacking in scientific research or whether the opposite may be true e.g. we need a stronger push to coalesce, deepen and replicate existing research.

• View from the field: What effects from AI on science should metascientists be exploring? Eamon Duede, Purdue University

  • “There is a tendency in metascience research to treat science as something rather monolithic and to ask questions, the answers to which generalize to all of science. This approach has been enormously illuminating. But when it comes to grappling with the impact of AI on science, it is likely to limit what we can learn. In contrast to prior domain-specific innovations, contemporary AI systems, particularly LLMs, will impact every discipline, yet do so in ways that differ profoundly from one field to another.

  • So rather than only asking how AI affects science in aggregate, we should ask how it differentially transforms the distinctive epistemic aims, methodological norms, and evaluative standards of physics versus history, or philosophy versus biology. Grappling with this question promises more than just antiquarian insights into AI’s role in research. Rather, it offers a powerful new lens through which to understand the very nature of science itself.”

Study Watch

AI and cognitive debt

• What happened: Authors from MIT Media Lab published a widely-discussed preprint, in which students who used LLMs for essay writing reported lower levels of brain activity and later struggled to recall quotes from their essays.

• What’s interesting: Observers have long fretted that new technologies will hurt students’ ability to learn. In Plato’s dialogue Phaedrus, Socrates worried that writing would create forgetfulness in learners’ souls. The towering Renaissance figure Conrad Gessner helped to establish the fields of bibliography and zoology, partly out of fear that the printing press would overwhelm learners with ‘information overload’. Observers have raised similar concerns about calculators, television, the internet, and now AI.

• In this study, the authors randomly assigned 54 university students in Boston into one of three groups. Across three sessions, each group had to write an essay on an SAT topic in 20 minutes. The first group used an LLM, the second used a search engine, and the third had to rely solely on their brains. An EEG headset monitored the students’ brain activity. At the end, the LLM and Brain-only students were given the option to swap groups and participate in an optional fourth session.

• The authors reported three main effects:

  • Reduced neural connectivity: According to EEG data, the LLM-only group showed the weakest neural connectivity - a proxy for cognitive effort - while the Brain-only group showed the strongest. When the LLM-only users had to rely only on their brains in the voluntary fourth session, their neural connectivity did not rebound to the level of the Brain-only group.

  • Worse memory: 15 of the participants in the LLM group failed to provide a correct quote from the essay they had just written (83%), while only two students in each of the other groups had the same difficulty. In interviews, LLM group participants reported a weaker sense of ownership over their work.

  • Homogenised language: Students using LLMs also produced essays that were more linguistically similar to one another - tying into a broader concern about AI and homogeneity, that Kalim Ahmed recently explored in an essay on this site.

• The study has limitations, as one review points out. It relies on a very small sample of elite students that drops further for the optional fourth session. The authors also ran a very large number of tests over the EEG data, a type of data that can be challenging to interpret, raising concerns about p-hacking. More fundamentally, the study converts relatively unsurprising results — if you use an LLM to help you write an essay in 20 minutes you will struggle to remember quotes from that essay — into a very strong claim: that LLM use will lead to ‘cognitive debt’ that impedes students’ future learning.

• The reality will likely be more nuanced. As John Sweller’s Cognitive Load Theory has illustrated, more cognitive effort is not always good for learning. Some cognitive load is good, because you are thinking hard about what matters. But some of it is extraneous, and a barrier to learning, such as the ‘split-attention’ and ‘modality’ effects that arise when students are presented with a confusing jumble of text and images.

• In some scenarios, LLMs could reduce cognitive load in a way that allows students to go deeper on a topic of interest, such as by providing a more compelling, integrated learning experience. Another positive scenario might see students using LLMs as a sort of ‘extended mind’ to automate certain tasks, in pursuit of higher-order thinking.

• However, these scenarios all require that students be motivated to learn in the first instance. Some worry that the ready-availability of LLMs may reduce such motivation, particularly among younger students developing foundational skills. This experiment doesn’t shed light on whether that scenario is happening. Rather, we would need other kinds of evaluations to assess how different kinds of students are using LLMs in the real world.

• View from the field: How should practitioners study the effects of AI on cognitive load? Kevin McKee, Google DeepMind

  • “Randomised controlled trials are particularly helpful for questions like this because they force us to think about what specific skills we care about and what measurements we can take to know if they've actually changed. And of course, if we want to understand their effects on students' independent cognitive abilities – how well they're able to function when they can't rely on LLMs – we'll have to specifically design RCTs in ways where we're confident students aren't accessing LLMs at test time.

  • As a complement to that, we should also think about in-depth studies that can examine how students try to solve problems or tackle self-study lessons. A well-designed ‘narrow’ study would help by shedding light on the mechanisms at play – like how students might be replacing some of their cognitive work with LLMs – while also giving us a better qualitative understanding of students' experiences, including how they feel after working with an LLM.”

Influential views

Where Dean Ball thinks AI is headed & how to govern it

• What happened: In April, prior to joining the White House Office of Science and Technology Policy as an AI policy advisor, Dean Ball published a two-part essay series outlining his expectations for AI and how policymakers should respond.

• What’s interesting: In the first essay, Dean lays out his core thesis: we are on the brink of powerful AI agents that will be able to source information, use software tools, and communicate: “These abstract tasks do not constitute everything a knowledge worker does, but they constitute a very large fraction…”.

  • Early AI agents can be glitchy, but as AI labs put them into reinforcement learning systems, they will get better. As Dean notes, this will be easier in domains like maths, where outputs can be more easily verified. But even in more subjective areas - like writing a newsletter! - AI systems can increasingly review each other’s outputs, which will accelerate progress.

  • As they deploy growing fleets of AI agents, Dean expects organisations that rely on knowledge workers to become more efficient and profitable. They may also become stranger - heavier at the top and so more variable in character, with leaders able to rely on agents for better information flow and control.

  • Widespread job loss may happen, particularly if prompted by a recession, but this may also be offset, or delayed, by the “in-person” aspects that many jobs have, or by regulations requiring “human alternatives” to AI decisions. In the near-term, young people entering the labour market may be the most affected.

  • Dean also expects transformative progress from the use of AI in science, from new cancer cures to room-temperature semiconductors, but these will take longer, as data still needs to be gathered and real-world experiments run. The prospects for the parts of the economy that are largely offline - like social care or the construction industry - are not analysed.

• In a 2nd essay, and an accompanying paper, Dean examines how the US government should respond. Dean’s starting point is that AI will become a “foundational technology” - closer to a natural resource like energy than to, say, social media. Past foundational technologies, like railroads, telecoms networks, electricity, and the Internet all differ, in form and function, so commonalities in how we govern them could map to AI.

  • The main commonality Dean sees is that the US eliminated or severely limited the exposure of providers of these foundational technologies to tort liability for downstream misuse of their products. Dean does not think AI developers should face no liability - if a data centre explodes due to mismanagement or an agent exfiltrates itself and defrauds people - companies should face different types of statutory liability, much like power providers do today.

  • But Dean argues that attempts to design AI liability schemes that go beyond this, to impose a “reasonable care” standard on AI model developers to foresee and prevent a wider range of downstream harms could be weaponised. Building his previous two-part essay series on liability, he notes how third parties, including anonymous investors, can fund and extend US liability cases and how the target is often those with the most resources, rather than those who are most directly responsible for the harm. Relying on liability can also mean that judges and juries are effectively determining how to govern frontier AI systems and what good safety practices look like.

  • What is Dean’s proposed solution? Building on Gillian Hadfield’s work on ‘regulatory markets’, he proposes a framework where governments would authorise private bodies to develop safety standards that AI companies could voluntarily opt to be certified and audited against. The AI companies that opt in would receive a ‘safe harbour’ from tort liability stemming from third parties’ misuse of their models. The goal would be to support AI innovation, while providing incentives for safety and encouraging a marketplace of ideas for how to best achieve it.

• View from the field: Prof Gabriel Weil, LawAI:

  • “Tort law is especially useful for mitigating risks from AI (over which there is substantial disagreement and uncertainty) because (unlike ex-ante regulation) it scales automatically with the risk, and shifts the onus to AI companies, where the relevant expertise is concentrated, to figure out how to make their systems safe. Voluntary private certification is poorly situated to protect third parties, since there is no market feedback to induce certifiers to craft rules that protect non-users and prevent a race to the bottom.

  • To read more, see my recent working paper which argues that liability should be the governance tool of first resort for AI risk, my shorter LawFare piece, and my earlier paper on using tort law to mitigate catastrophic risk from AI. On the mixed relationship between liability and innovation, the risk of excessive litigation, and the nuances that apply here, see my recent piece with Mackenzie Arnold.”

Study watch

In Denmark, chatbots aren’t turbocharging productivity growth

• What Happened: In May, Anders Humlum and Emilie Vestergaard from the universities of Chicago and Copenhagen published a new analysis of chatbot adoption among ~25,000 Danish workers in 11 occupations that are ‘exposed’ to AI, such as journalists, customer service employees and software developers. They found that while chatbot use is high, and growing, this has not yet resulted in a statistically significant impact on productivity growth.

• What’s Interesting: The analysis is based on two large surveys that the authors conducted in 2023-24 to understand how Danish employees were using chatbots and what the perceived impacts were. The authors then used social security numbers to match the survey data with government data on employment and earnings. Using a difference-in-differences approach - which attempts to mimic a randomised controlled trial using real-world data - they then analysed the chatbots’ impacts on productivity.

  • What did they find? If chatbots were making individuals more productive, we might expect to see an increase in their wages and/or a reduction in work hours. However, the survey finds zero statistically significant effects on these variables or on firms’ profits.

  • At first glance, this is disappointing. For many economists, growth in productivity is the most important determinant of long-term economic growth and all that rests on it. For the past two decades, productivity growth has been stagnant in much of the world, rich and poor alike, hurting public services and living standards. Optimists hope that AI will now super-charge it, while pessimists worry about a repeat of the ‘Solow paradox’ - in 1987, the Nobel Laureate famously quipped that “you can see the computer era everywhere but in the productivity statistics”.

  • The Danish analysis does hint at some productivity gains from AI. Survey respondents who used chatbots reported an average saving of 2.8% in their work hours, but only a very small fraction of this resulted in higher wages. When added to the growing literature on AI’s productivity effects, this suggests a pattern where: academic experiments that give individuals access to chatbots or AI tools to complete a certain task often demonstrate quite high self-reported productivity effects, of 15-50%, in a short time frame (e.g. a week). Yet once we turn to early real-world outcomes from AI use at organisations, these effects shrink. And when we look at aggregate productivity growth at the level of the entire economy, evidence for AI’s benefits is even more scant.

  • What might explain this? First, the Danish survey focuses on chatbots from 2023-24, and so the AI capabilities may have been too nascent to have had much effect. Second, the J curve theory put forward by Erik Brynjolfsson et. al. argues that AI can increase productivity growth, but only after employees and organisations work out how to best use it, which takes time and resources. Humlum and Vestergaard find some evidence for this - employees report that introducing chatbots creates new tasks for employees who have to integrate them into workflows and ensure compliance. Finally, wage growth may also be a limited way to measure productivity growth, not least since effects on wages take time to materialise. Past research also demonstrates that new technologies may benefit a relatively small number of firms and workers, and so will not always clearly manifest in the data.

  • This suggests that as AI capabilities improve, and organisations and individuals become better at using it, productivity growth could start to increase, perhaps rapidly. However, this is not guaranteed. For one, AI will need to be usefully deployed across all or most consequential sectors, and catalyse new sectors, if countries are to avoid a ‘Baumol cost disease’ scenario where new productivity gains in some sectors, such as the technology sector, lead to increased demand in other sectors, such as education, where productivity gains are not materialising. Such a scenario, which likely played a role in the original Solow Paradox, could blunt macro-level productivity growth.

  • AI may also introduce complexities that make it harder to measure productivity growth - for example, if people start using ‘free’ or low-cost LLMs to perform tasks where they previously hired a company, this could cause output to nominally ‘decline’, at least under current measurement approaches. So not only do the potential effects of AI on productivity growth remain unclear, so does the best approach to measuring it.

• View from the field: Sam Manning, GovAI:

  • “Outside the headline result, this paper includes several notable findings. For example, on days when they use AI, marketing professionals and software developers report higher time savings than teachers (~7-11% vs ~4.5%). These numbers don’t strike me as negligible and suggest that AI’s productivity effects are likely to vary quite a bit across occupations. If some roles are already saving 7–11% of their workday thanks to AI, firms will eventually begin adjusting workflows to better capture those time savings and competitive pressures will result in broader productivity gains that are measurable at the firm level. It’s also interesting that when employers actively encourage the use of chatbots, the reported effects on time savings, work quality, task expansion, creativity, and job satisfaction rise by 10–40%. That points to an important role for firms in promoting more widespread and effective use of LLMs in the workplace.”

Topic deepdive

Will AI exacerbate or mitigate climate change?

• What happened: In April and May, The Economist and MIT Tech Review published special reports examining how AI may affect climate change, with The Economist’s Alex Hern noting that he has been trying to “nail down” this question ever since AI took off.

• What’s interesting: To understand how AI will affect the climate, we need to answer two different questions, which the reports shed light on, in different ways.

• The first question is: how will training and inferencing AI models directly affect greenhouse gas emissions, via the power they consume and the ‘embodied’ emissions required to build, maintain and recycle the data centres, devices and networks?

  • Researchers like Emma Strubell, Alexandra Sasha Luccioni and Jae-Won Chung have devised methods to help address these questions. The MIT report draws on these methods to provide new estimates. For example, they find that asking Llama 3.1 8b to make a travel itinerary requires ~114 joules of energy, once cooling and other factors are accounted for - a tiny amount, equivalent to riding six feet on an e-bike. At the other end of the spectrum, generating a five second video using a ZhiPuAI model, uses about 3.4 million joules, equivalent to ~38 miles on an e-bike.

  • It is challenging to both compile and interpret these estimates. First, researchers typically have to focus on open-source models, as they argue that companies who develop leading proprietary models do not release the necessary data, although the EU AI Act may soon require estimates for the largest AI models. A second issue is that some past estimates have been miscalculated, or misleadingly reported, in a way that makes them sound a lot larger - echoing past panics about energy use from technology, such as around video streaming during Covid-19. Finally, there is no clear way to tally these individual estimates into an aggregate estimate for the overall emissions from training and running all AI models to understand how relevant it is, from a global emissions perspective.

  • Instead, the best macro estimates come from looking at data centres’ power consumption. Today, data centres account for just ~1.5% of global power consumption, or ~2% if crypto is included. Most of this comes from activities like streaming, rather than AI. In April, the IEA shared its latest forecast for how this may change in the AI era. In its base case scenario, data centres’ power consumption would rise to 945 terawatt-hours by 2030, up from 415 in 2024. If this proves accurate, it would be a non-trivial increase and could put pressure on energy grids in certain locations, as data centres are geographically concentrated - in Ireland they accounted for ~17% of power consumed in 2022. However, data centres would still account for just ~3% of global power consumption and the increase in their power consumption would be smaller than that of other sectors, such as electric vehicles and air-conditioning.

  • From a climate change perspective, the precise amount of power that AI consumes will also be less important than the source of that power. Optimists hope that AI acts as a forcing function to dramatically accelerate the roll-out of nuclear and renewable energy in the near-term. Skeptics worry that the AI race will compel companies to use fossil fuels that they may have otherwise eschewed.

• When it comes to determining how AI will affect climate change, a second question is arguably more important than AI’s future power use: what applications will AI be used for, at what scale, and how will these applications affect emissions?

  • AI supporters argue that it will accelerate renewable energy and make the economy - including energy-intensive sectors - dramatically more efficient.

  • The Economist’s report provides some grounds for optimism on this front. They note how companies such as Octopus Energy and Tapestry are using AI to make it easier to deploy renewable energy and to optimise the grid, for example by helping to locate green-energy projects and enable smart homes and vehicles to autonomously draw power during fallow periods. Other case studies document how energy-intensive industries are using AI, for example to optimise heating and cooling in buildings, reduce waiting times for ships in ports, or to enable new kinds of steel manufacturing processes.

  • Estimating how these AI use cases may affect emissions is even more challenging than estimating AI’s power use. There is no formal stocktaking of beneficial AI climate applications and few efforts to estimate how they will affect emissions or the additionality that AI brings. In theory, these AI applications could reduce the future emissions that would have otherwise occurred by much more than AI’s future power use increases them, but the level of uncertainty, and the timeline to impact, are greater. This is even more true when we consider efforts to use AI in science, which could lead to new materials for solar panels, batteries, and direct air capture, or even accelerate fusion - an effectively limitless, clean energy. Could AI make breakthroughs in these areas 30% more likely, or accelerate them by 30 years? Given their complexity, the temptation to skip such questions and focus on what we can measure - AI’s power use - is high.

  • A final complication comes from the fact that most AI applications are not obviously good, or bad, from an emissions perspective but may still shift individual or organisational behaviour in consequential ways. For example, what might happen if people start to shift more of their economic activities to AI agents? History tells us that the impact of technologies often depends on context and the activities they replace. For example, the Internet enabled music streaming, ecommerce, and home working, but whether these shifts in behaviour increase or reduce emissions can vary depending on the individual case, such as the size of a person’s home or whether ecommerce transport is electrified. At the aggregate level, there are reasons to think that digitisation helps to make economies less carbon intensive. But it’s hard to reliably ‘prove’ this and much depends on context and efficiency gains - which so far have been remarkably high for data centres and AI.

• Given these complexities, how should policymakers ensure that AI benefits the climate? The Economist argues that the best policy would be a strong global carbon tax to enable the market to incentivise and penalise different AI applications and uses. However, it views this as politically intractable and so calls on governments to instead undertake permitting reforms to allow AI companies to fund and build more clean energy, and to build more flexible data centres that can match workloads to intermittent wind and solar.

• The Economist also calls on other geographies to emulate the EU AI Act and impose obligations on AI developers to share estimates for the power used by their leading models. The reliance on open-source models to estimate AI power use does seem inadequate but the usefulness of this recommendation could also be challenged, given the overlapping energy reporting requirements that already exist and the risk of creating the kind of arduous ‘environmental impact assessments’ seen in other sectors that can stymie innovation at little benefit to the environment.

• View from the field: Andy Masley, Author of Weird Turn Pro and Why ChatGPT is not bad for the environment

  • “Excited: AI seems ecologically costly if we only look at its total energy use without considering the value we get out of it. But, hospitals use more energy than yachts, and if we look at value per unit of energy, AI seems very likely to become one of the most energy efficient sectors. See for example this, this, and this about how LLMs are adding value to programming, at relatively little cost. And that’s before we consider the more direct ways that AI can be useful to the climate, such as by optimising energy and transportation.

  • Worried: In line with Jevon’s Paradox, I worry that even though AI might make processes more efficient, if it’s not paired with a switch to renewable energy, we may emit more in total and miss key climate targets. I'm also concerned that AI-enabled weapons or widespread job automation could threaten political stability, eroding the trust needed for international climate cooperation.”

We are exploring ways to make this Substack more useful. If you have ideas, please reach out to aipolicyperspectives@google.com.

We are exploring ways to make this newsletter more useful - if you have ideas, please reach out to aipolicyperspectives@google.com. Thanks for reading!

Will AI help or hurt scientific reliability?

• What happened?: An article in Nature News highlighted two early efforts to use LLMs to detect errors in scientific papers. If successful, they could provide a much-needed boost to scientific reliability, but many scientists remain skeptical about their usefulness.

• What’s interesting?: ‘Reliability’ refers to scientists’ ability to depend upon each other’s findings and trust that they are not due to chance or error. A series of interrelated challenges currently undermine scientific reliability, including the p-hacking and publication bias that lead researchers to underreport negative results; a lack of standardisation in how scientists carry out routine scientific tasks; challenges with the peer review process, and scientific fraud. Another issue is that scientists can make mistakes, for example in how they apply statistical methods. At the aggregate level, such mistakes are non-trivial - a 2013 study claimed that 13% of psychology papers include a mistake that, if corrected, would alter the interpretation of their results.

• Some scientists worry that the growing use of AI in research will further undermine scientific reliability, not least because AI models are prone to ‘hallucinate’ outputs, including scientific citations. In response, AI practitioners are working on mitigations to these risks, such as techniques to better ground model outputs to trusted sources.

• Other practitioners, including those behind two new AI-based error detection efforts, hope to go further and use AI to improve the reliability of the wider research base. The first effort is the ‘Black Spatula Project,’ which was named after scientists used AI to detect a mathematical error in a widely-covered study that had incorrectly claimed that black plastic cooking utensils contained worrying levels of cancer-linked flame retardants. Staffed by volunteers, the open-source project has so far used AI to review ~500 papers. It has not yet made the errors that it has found public and is instead sharing them with the papers’ authors. The second effort, YesNoError, uses an AI agent to scan papers for errors and aspires to check the entire scientific literature.

• As the article notes, some scientific integrity practitioners cautiously support the efforts, but not everybody is a fan. The researcher Nick Brown claims the false positive rate is high and that many of the ‘errors’ are minor typos or writing issues. The practitioners behind YesNoError also aim to work with the ResearchHub platform - which pays researchers cryptocurrency to do peer review. They want to let holders of the cryptocurrency suggest which papers get scrutinized first, which some worry could lead to people targeting research they don’t like.

• Such skepticism is also evident in the EU’s recently-updated guidelines for researchers and funders on how to use AI in research, which focus almost exclusively on the risks that AI poses, and the responsibility of researchers and funders to mitigate them. It is also visible in the bans that many journals and conferences have imposed on the use of AI in peer review, even if many individual peer reviewers appear to be using it.

• What’s the takeaway?: Many of the concerns stem from a desire for AI not to replace the role of human reviewers, particularly for consequential decisions, such as approving publication decisions. However, if fast-improving AI reasoning models were instead framed as aids to human researchers or peer reviewers, to sense-check or enhance their own review processes, particularly for error detection, they may become more popular. To do so, independent evaluations that can reliably demonstrate the ability of AI to detect consequential errors that humans overlook, will likely be critical.

AGI and the free society

• What Happened?: A new working paper from Justin B. Bullock, Samuel Hammond, and Séb Krier explores how AGI might affect the balance in power between state and society, upsetting the delicate equilibrium that underpins liberal democracies.

• What’s Interesting?: The paper builds on a framework developed by Daron Acemoglu and James A. Robinson, which argues that true liberty exists in a ‘narrow corridor’ between an overly-powerful, despotic state on one hand, and a chaotic absent state that is too weak to govern on the other.

• AGI may upset this narrow corridor if it enables states or non-state actors to engage in new kinds of harmful monitoring, coordination, or decision-making. For example, governments are using AI to detect tax fraud and manage traffic flows. But AGI could go much further, potentially automating entire public sector roles. Positively, this could help governments to better understand and shape behaviours across society, similar to how digitisation has helped to visualise and suppress black markets in India.

• It could also promote more narrowly tailored rule enforcement and curb blanket policies that produce inefficiencies - for instance, the type of policies that led to the recall of Tesla’s Full Self‑Driving for carrying out harmless rolling stops could be replaced with AGI systems that better weigh real‑time context, such as the presence of pedestrians, visibility, and thus the actual level of risks.

• However, this improved legibility and enforcement also brings risks of illegitimate surveillance and loss of liberty, such as AI systems that might ensure that every vehicle perfectly obeys every traffic rule, or AI-enabled CCTV that might detect and punish every minor infraction, undermining the useful personal judgement and empathy that today’s officials often show in such situations.

• AGI could also empower non-state actors. In more positive scenarios, this could enable citizens to better understand and advocate for policy positions, fact-check officials, and usher in new kinds of public deliberation. However, individuals/groups could also use AI agents to orchestrate harmful actions or create opaque financial communication methods that make the economy less legible to governments, rather than more, similar to how cryptocurrency can be used to launder money, despite its legibility.

• What’s the takeaway?: To stay liberal and democratic, the authors call for investment in robust technical safeguards - like privacy-preserving AI - and intentional policies - like identity verification protocols and advanced encryption. In short, states must neither blindly hand off power to AI systems nor clamp down on them in ways that stifle innovation.

Grappling with the economic impact of AI

• What happened?: A new paper from Yale’s Menaka Hampole and colleagues found that jobs that are highly-exposed to AI are experiencing lower labor market demand, compared to less exposed occupations. However, this is partly offset by boosts to productivity and profits in firms that adopt AI, which increases their ability to hire.

• What’s interesting?: Measuring and predicting how fast-improving AI systems will affect employment and other economic variables, like growth or inequality, is a mounting challenge. Economists are pursuing different approaches - none of them perfect.

• In this study, the authors treat occupations as bundles of tasks. They posit that AI’s effects on demand for a particular occupation will depend on how many tasks within that occupation AI can substitute for, and how many other tasks remain, including new tasks that an employee can pivot into - such as an automated expense system that enables an accountant to pursue more complex financial analysis.

• To measure exposure to AI, the researchers reviewed a large volume of online job postings from US publicly-traded companies between 2010-2023. They used LLMs to identify specific AI applications described within these postings, such as “analysing customer reviews” or “forecasting risk and fraud”. They then matched these AI applications to the tasks that humans perform, by drawing on the US government’s O*Net database. They also drew on firm-level data on sales, profits, productivity, and hiring to assess how AI adoption affects them.

• The authors find that some broad occupation groups that were highly exposed to AI, like 'Business and Financial' and 'Architecture and Engineering,' experienced the largest AI-related declines in their relative employment shares during the study period, estimated at between 2%-2.5%. Examples of highly-exposed occupations in these categories include market researchers, credit analysts, and financial specialists. However, the paper finds that, on average, AI adoption increases sales, profit, productivity growth and overall hiring in firms.

• The authors also found that more highly-paid roles tend to be more exposed to AI, but that this tails off above the 90th percentile level, perhaps because the most highly-paid jobs require strong interpersonal and management skills, which AI cannot currently automate.

• As the authors acknowledge, their approach faces various challenges and uncertainties;

  • Bias: The firms that embrace AI may have other characteristics that contribute to their higher growth trajectories - a potential source of bias that the authors try to address using an instrumental variable approach, which they acknowledge is imperfect - see more on this mechanism here. The reliance on an online jobs dataset may also lead to biases, including in the types of AI use that it includes/excludes - e.g. organizations may not necessarily describe AI applications that are more likely to displace employees in their job postings.

  • Time period: The authors’ analysis ends in 2023, which means that it overlooks more recent GenAI tools, whose adoption is still nascent. There is also a question about how representative the trends will be of longer-term effects as AI improves, diffuses across the economy, and individuals and firms respond to its impacts. Previous technological shocks during the two industrial revolutions and the computerization of the late 20th century often led to an initial boost to employment in technology-exposed occupations, followed by eventual displacement. They also led to new jobs, but many of these jobs were either not available to those who were displaced, or were less satisfying, leading to rising labour market polarization and inequality.

• What’s the takeaway?: The effects that AI will have on aggregate (un)employment, across different time periods, and the degree to which different employees benefit/suffer, remain open questions. But it seems plausible that, over the next two years, in high-income countries, AI could have a moderate positive impact on productivity and economic growth. There may be no major increase in aggregate unemployment, yet, but we will likely see early signs of increased inequality between those employees who are able to benefit from AI and those who cannot.

As always, please leave a comment below to let us know your thoughts, or send any feedback to aipolicyperspectives@google.com. Thanks for reading!

Policymakers taking action

France and the EU announce large AI investments

• What happened: At the recent AI Action Summit in Paris, President Emmanuel Macron unveiled €109bn in private sector funding for AI infrastructure. This includes a new €50bn AI campus of data centres, led by the UAE’s MGX, which is also involved in the US ‘Stargate’ project; a €10bn AI “supercomputer” from Fluidstack, a British AI Cloud platform; and €5bn from the US firm Apollo to invest in AI energy infrastructure.

• The European Commission also unveiled a €200bn “InvestAI” initiative, which includes plans to create four “AI gigafactories” with “100,000 last-generation AI chips”, complementing the smaller ‘AI factories’ the EU is already developing.

• What’s interesting: One narrative that emerged from the Paris Summit, following a speech by US Vice President JD Vance, was that the EU’s AI efforts are mired in excessive regulation, while the US is powering ahead with a more supportive regulatory environment and strong financing. The reality is more nuanced. While the US has not yet passed any federal AI regulation akin to the EU’s AI Act, many US states are advancing AI bills that could impose similar obligations - if passed. Unlike the EU’s harmonised approach, some of these state-level bills also take inconsistent approaches, potentially increasing regulatory complexity.

• The new French and EU funding announcements, alongside Macron’s repeated promotion of Mistral at the Summit, also underscored that EU member states do want future frontier AI models to be trained in Europe. Commission President Ursula von der Leyen also pledged that the new gigafactories would prioritise these efforts. While the EU is advancing the AI Act, the Commission also made the rare decision, shortly after the Summit, to withdraw its proposed AI liability directive - as part of broader efforts to streamline the EU’s regulations and boost competitiveness.

• Still, many remain sceptical about whether France and the EU can rapidly secure and deploy the new AI funding, much of which remains to be mobilised. There are also doubts about whether they can narrow the wider gap with the US AI ecosystem, particularly when it comes to training the most advanced AI foundation models. With these challenges in mind, von der Leyen emphasised the need to prioritise ‘industry-specific’ AI applications. This could include sectors like green energy, where the EU has strong expertise but faces intense competition from China and others.

• This suggests that, while a small number of EU AI startups will continue developing frontier foundation models, the most promising efforts may emerge in areas that draw on local economic strengths, such as in finance, tourism, or healthcare.

Subscribe now

Study watch

Building a market for AI safety

• What happened: Philip Moreira Tomei and colleagues at the AI Objectives Institute published a paper arguing that market-based mechanisms could help reduce AI safety risks, by complementing regulatory efforts.

• What’s interesting: Discussions about AI risks often quickly pivot to how to pass or adapt new regulations. However, uncertainty about specific AI risk scenarios makes it difficult to craft rules that are targeted and effective. When regulation does arrive, it is often vague, hard to implement, and can create uncertainty that inhibits wider AI adoption. The rapid pace of AI development also makes it difficult to design regulation that is resilient and adaptable.

• The authors argue that market-based mechanisms could help to complement AI regulation by providing AI developers and deployers with financial incentives to identify, evaluate, and mitigate AI risks, while distributing risk management across a broader range of actors. The paper outlines four market-based approaches, citing examples from other high-risk industries:

  • Insurance: Firms could take out liability insurance against AI risks, for example building on existing cybersecurity insurance or technology errors and omissions insurance, which have encouraged firms to invest in risk mitigation.

  • Auditing & certification: Firms could hire third-party auditors to assess AI safety practices, leading to certifications for meeting certain standards. For example, after facing scrutiny over their cybersecurity, Zoom engaged Trail of Bits and NCC Group for an audit, which led to enhanced end-to-end encryption.

  • Procurement: Large purchasers of AI could demand performance on safety benchmarks or require specific disclosures - similar to how governments use procurement to shape markets or how corporations push suppliers to improve working conditions and environmental standards.

  • Investor due diligence: Investors could also demand safety and transparency measures from AI companies, similar to how investors pressured BP to share more about their risk management processes, following the 2010 Deepwater Horizon oil spill, which also accelerated BP’s transition to renewable energy.

• Similar ideas have been explored by other organisations and sectors in the past. A related promising angle is supply-side interventions: governments or philanthropic organisations could act as 'buyers of first resort' by establishing Advance Market Commitments that guarantee future purchases of innovative products, like AI safety tools, to incentivise their development. Regulatory markets' - where governments license private regulators to compete to provide AI safety oversight services to companies - could also address gaps left by traditional regulatory mechanisms.

• Although they hold promise, market-based mechanisms also face a range of challenges, including how to encourage bottom-up action from a diverse range of organisations (from insurance providers to investors); how to prioritise and price different AI risks; how to ensure sufficient independence and skills among auditors and certification agencies; and how to balance the goal of AI safety against others. For example, after BP’s strategic redirection towards renewables, its financial performance slumped, and the company recently reversed course, saying it had gone "too far, too fast" in the transition away from fossil fuels.

Sector deep dive

The deployment of autonomous vehicles slowly accelerates

• What happened: In January, Kodiak Robotics announced that its client Atlas Energy Solutions - which serves oil and gas companies in the Permian Basin (West Texas and New Mexico) - had successfully delivered 100 loads of material using driverless trucks.

• What’s interesting: Excitement and skepticism about autonomous vehicles has fluctuated over the past decade, but parts of the sector are now seeing renewed momentum. For example, Waymo now logs 200,000 paid robotaxi rides every week, a 20x increase in two years, and will soon begin testing in Japan, following a recent $5.6B funding round.

• The stop-start development of autonomous vehicles highlights a classic challenge in technology development: the mismatch between a technology’s capabilities and its practical deployment. Widespread adoption of autonomous vehicles has faced multiple barriers, including: a complex and evolving regulatory landscape that varies by country and (in the US) by state; the far higher safety standards expected of autonomous vehicles compared to human drivers; low levels of public trust; and the sheer complexity of real-world roads, particularly in dense urban centres.

• In response, some companies, like Kodiak, are prioritising industry-specific use cases in more controlled, remote environments, such as mines, seaports, large industrial farms, and military domains. These settings can also expose autonomous vehicles to harsh conditions - like dust, uneven terrain, and strict local site regulations - which could help improve the technology.

• These deployment decisions are also influenced by labour market trends. While concerns persist about AI replacing drivers and supply chain workers, a shortage of personnel is arguably a greater challenge. In the US alone, there are about 3.5m truckers, but companies struggle with an ageing workforce and high turnover rates, which exceed 90% in some sectors, partly owing to poor working conditions. This in turn is leading to supply chain disruptions and higher consumer costs.

• A broader question is whether the deployment of autonomous vehicles over the past five years offers any insights into how other types of AI-enabled robots might be deployed across the economy in the coming years. Traditional industrial robots are already well established in manufacturing and warehousing, but most are limited to a narrow range of repetitive tasks in structured environments. Companies are now using foundation models to develop more general-purpose robots that could learn novel tasks and adapt to real-world environments. If technical challenges can be overcome, these robotics could be particularly valuable in sectors like agriculture, healthcare or social care, where labour shortages are mounting. That said, they may face similar deployment obstacles to autonomous vehicles, which could lead them to seek out use cases where safety risks are lower, environments are easier to control and cost pressures are high.

Policymakers taking action

US and UK government announce flagship AI infrastructure efforts

• What happened: The UK Government announced a new AI Opportunities Action Plan, while the Trump administration announced ‘Project Stargate’ - a $500bn initiative to expand AI training infrastructure, led by Softbank, OpenAI, Oracle and others.

• What’s interesting: Both efforts aim to attract investment for AI infrastructure, but they also highlight the starkly different approaches on either side of the Atlantic. The UK’s Plan, praised for its ambition, includes 50 recommendations, ranging from deregulating data centre planning to promoting AI adoption in the public sector. The goal is to ‘on-shore’ AI activity by leveraging the UK’s existing strengths - such as its strong university research base (Oxford, Cambridge, Imperial, etc) and a robust AI safety community - while addressing planning and energy constraints that have hindered broader economic growth .

  • The plan also signals a potential shift in UK economic policymaking. By establishing a “UK Sovereign AI unit” inside No.10, and “AI Growth Zones” with assumed planning approval for AI infrastructure and energy projects, the Plan explicitly states that “the invisible hand of the market” alone will not suffice. Instead, the UK must take a proactive role to remain competitive in AI.

  • The US, with lower energy costs, a stronger industrial base, and a more liberal approach to infrastructure, is in a fundamentally different position. Project Stargate is private-sector-led and operates on a scale that the UK and other nations cannot realistically match.

  • However, both initiatives face questions about funding and execution. For Stargate, the US government’s role remains unclear, raising concerns about bottlenecks in energy, land, and resource allocation. In the UK, much will hinge on the forthcoming Spending Review.

  • Ultimately, these announcements underscore how AI is becoming central to economic strategy and geopolitical influence. The ‘race for compute’ is set to intensify, despite ongoing breakthroughs in AI efficiency, leading to more public-private partnerships for large-scale AI investment.

  • Whether a significant share of this spending happens outside of the US will largely depend on how effectively other nations, including the UK, execute on their AI strategies, but will also be impacted by policies like the new US AI Diffusion Framework.

Study watch

Will AI use hurt critical thinking skills?

• What Happened: A new study of 666 UK participants, spanning diverse age groups and educational backgrounds, found a strong self-reported negative correlation between AI tool use and self-reported critical thinking skills.

• What’s Interesting: Over the past 20 years, the advent of the Internet has led educators to focus on equipping students with ‘hard’ STEM skills, alongside ‘soft’ skills like collaboration and critical thinking, to prepare them for an increasingly digital society.

  • Critical thinking involves analysing, evaluating, and synthesising information to make reasoned decisions. It encompasses problem-solving, decision-making, and reflective thinking, but as a concept it remains somewhat vague and it is difficult to assess how teaching programmes or technology affect it.

  • In particular, a long-running debate exists over whether technologies that automate routine tasks - from calculators to personal computers to AI - support or hinder critical thinking. There is also concern over whether such technologies may erode foundational knowledge or skills - like reading or numeracy - that may be essential for critical thinking and which appear to be stagnating or even declining in many countries.

• In this study, the authors surveyed participants on how frequently and in what ways they used AI to retrieve information and make decisions. Participants were then asked about AI’s impact on their ability to think critically and solve problems independently, as well as their concerns regarding AI bias and transparency.

• The findings suggest that while AI enhances efficiency for some individuals, it may come at the cost of a decline in independent problem-solving and critical analysis - or at least a perception of one. This could be due to ‘cognitive offloading’, where users delegate tasks to AI without redirecting their efforts to more complex, higher-order thinking.

• The study also found that younger participants (17–25 years old) and those with lower educational attainment reported a greater dependence on AI, potentially reducing their ability to critically evaluate information and identify biases.

• These findings mirror a recent experiment where students given temporary access to LLMs for learning performed worse on exams once access was revoked, compared to students who had never used the tools in the first place.

• While this study raises concerns about AI’s impact on critical thinking, it also presents an opportunity. The research suggests that AI’s effects are malleable, and education could provide ‘cognitive scaffolding’ to help users engage with AI in more productive ways. For example, in their recent LearnLM work, our colleagues developed pedagogy-inspired LLMs that encourage students to reflect on questions rather than simply answering them. More broadly, there is a need for new kinds of AI literacy that does not just teach students, or educators, what AI is, but how to use it most effectively.

Sector spotlight

• What Happened: John Micklethwait, editor-in-chief of Bloomberg News, and former editor of The Economist, shared predictions on AI’s impact on journalism in a talk at the James Cameron Memorial Lecture. He outlines a positive vision for AI’s integration into journalism and journalist jobs, yet predicts a decline for traditional Search and ad-based revenue models.

• What’s Interesting: Micklethwait compares AI’s disruption to the early 2000s, when the Internet was blamed for ‘killing’ newspapers. He argues that media outlets were too quick to accept tech companies’ claims that content should be free, leading to a race to the bottom in pursuit of clicks. More recently, publications like The New York Times and The Information have reversed course, building high-priced subscription businesses. With AI, he expects a faster shift to high-quality AI-enhanced content, as media outlets try to avoid repeating past mistakes.

  • To illustrate AI’s role in journalism, Micklethwait highlights that one-third of Bloomberg’s 5,000 daily articles now incorporate some form of automation. In one example, investigative journalists used AI to analyse satellite imagery of ship movements to uncover oil smuggling from Iran. On the other end of the journalism spectrum, Bloomberg also uses AI to generate bullet-point article summaries - a feature disliked by journalists but valued by readers.

  • In his predictions, Micklethwait envisions AI reshaping journalism tasks but not eliminating jobs. He cites Bloomberg’s continued employment of roughly the same number of company earnings reporters, despite years of automation in that area. Similarly, he expects AI to play a growing role in editing and formatting articles, but to remain less capable of other editorial actions - such as commissioning stories or persuading a cabinet minister to reveal a resignation.

  • Micklethwait also warns of AI-enabled misinformation, particularly in image and video content, which he sees as more harmful than text-based misinformation - especially for fast-moving news stories, where social media plays a key role in verification. Following licensing deals, he expects the decline of traditional search engines and the media outlets dependent on search-driven revenue. After many false dawns, he also expects the long-awaited emergence of truly personalised AI news offerings.

• Looking backwards, over the past 15 years, the number of working journalists in the UK and US appears to have grown slowly, and with increasing diversification of Internet-enabled roles. While uncertainty is high, we expect this trend to continue over the next five years, as AI-driven analysis becomes standard, though it remains to be seen whether traditional or newer media outlets will lead the shift.

Study watch

AI helps scientists to discover new materials, but it may make them enjoy their job less

• What Happened: Aidan Toner-Rodgers at MIT published the results of an experiment in which more than 1,000 scientists at a US private sector R&D lab were given access to an AI material design model, to assess how it changed the rate at which they discovered new materials.

• What’s Interesting: To design materials, scientists often rely on iteration or trial and error to explore the huge search space of potential compounds. This study highlights how AI could help improve this process. In 2022, the scientists received access to an unnamed AI model that outputted compounds that were predicted to possess desired characteristics. On average, the AI-assisted scientists subsequently discovered 44% more materials, filed 39% more patents, and produced 17% more product prototypes, with the new materials scoring high on both ‘novelty’ and ‘quality’ - although the experiment did not capture longer-term commercial impact.

• In the past 1-2 years, various experiments have studied the effects of giving AI tools to professionals, including programmers, writers and customer service agents. Many of these studies - though not all - found evidence that lower-skilled employees benefit most from AI. This MIT study finds the inverse, with experienced scientists enjoying the biggest gains. This is because the task of material design is not simply about finding a novel compound, but rather about being able to identify compounds that are most likely to be viable and useful. As the AI model predicts new compounds, it shifts the focus of scientists to evaluating the viability and usefulness of those predictions - a task that those with deep domain expertise are best-suited to.

• AI will not just affect science, but also scientists. To understand these effects, the MIT study surveyed scientists who received access to the AI model and found that most reported a decline in their work satisfaction. This was true even for those scientists who benefited from the AI tool, owing to concerns that their skills were being under-utilised and the creativity of their role reduced. Wellbeing is hard to measure, and attitudes to technology can change with time, but this finding highlights the need to better understand how AI may affect scientists, a topic that we also explored in our recent essay about AI and science.

• In the next 1-2 years, we hope to see an increase in evaluations focussed on empirically assessing how AI is affecting science and scientists, in a similar vein to the recent suite of new evaluations that focus on AI safety.

Policymakers taking action

US & UK AI Safety Institutes convene in San Francisco

• What happened: On 20-21 November, the US AI Safety Institute held a convening in San Francisco to kickstart a new technical collaboration between the global network of AI safety institutes, ahead of the upcoming AI Action Summit in Paris in February 2025. The UK AISI also held a convening to share best practices for how to develop AI safety frameworks, like Google DeepMind’s Frontier Safety Framework.

• What’s interesting: As announced in their mission statement, the new global network of AISIs will focus on advancing research to understand the capabilities and risks of advanced AI systems, as well as building best practices for testing them. They also completed their first joint testing exercise, on Llama-3.1, and shared insights about how to improve multilingual AI testing.

• This is the first time the AISIs have met and announced shared priorities. They showed interest in coordinating more and exchanging best practices. Synthetic content was one of the three areas discussed during the convening, with the global network of AISIs announcing $11 million to fund new research on how to mitigate risks in this area. The global network of AISIs - currently numbering 10 - may expand further in 2025, and they may look to conduct more joint testing exercises. This collaboration could potentially reduce the risk of separate bilateral conversations and fragmentation in the AISIs’ mandates.

Sector spotlight

Central Banks begin to scale up their use of AI

• What Happened: Central Banks play a critical role in most modern economies, where among other mandates, they are typically responsible for maintaining the stability of prices. As outlined in the BIS Annual Economic Report 2024, central banks are increasingly using AI to improve how they make decisions. Key focus areas include economic forecasting, financial supervision, and payment systems.

• What’s Interesting: One area central banks are focussing on is identifying signals and anomalies in the vast datasets they have access to. For example, the Bank of England and the European Central Bank are using AI to look for unexpected transaction patterns or spikes in asset price volatility that may signal liquidity issues. Similarly, the Deutsche Bundesbank is focused on detecting outliers in major financial data sets, which could signal risks such as mispricing in the market. This anomaly detection can be difficult for humans to do reliably, and so AI could help central banks improve resilience against different kinds of financial risks and crime.

• Central Banks are also using AI to synthesise insights, including sentiment and trends, from unstructured text, to improve their forecasting. For example, the Bank of France is using AI to better gauge public perceptions on inflation, which can provide insights about how future ‘sticky’ price growth may be. Meanwhile, Malaysia’s central bank uses AI to analyse financial news articles to help forecast key indicators, such as GDP and consumer spending.

• Finally, central banks are exploring the merits of tokenized payment systems—digital versions of money that use blockchain technology—and unified ledgers - systems that combine financial and other records in one place. These technologies could potentially make transactions faster, more transparent, or more secure. A growing number of central banks are planning to launch their own tokenized payment systems, such as Central Bank Digital Currencies. AI's role in these developments could take several forms, including detecting and preventing fraud in real-time, or monitoring transactions to adhere to anti-money laundering or counter-terrorism regulations.

• In the next 1-2 years, we will see central banks explore if AI could also support monitoring and mitigating emerging AI-driven risks to financial stability, such as market disruptions or collusion by autonomous agents - an area that has received relatively little attention in discussions about AI safety risks.

As always, please let us know your thoughts on these updates and what you have found most interesting in the world of AI policy in the last month.

In the October edition of the AI Policy Primer, we look at research exploring a shift towards data governance and away from compute governance, how AI is being used in the legal sector, and new work outlining a communication protocol for AI agents. As usual, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.

What we’re reading 

From compute governance to data governance   

• What happened: A team at Berkeley announced a new initiative, and an accompanying paper, which calls for AI governance efforts to shift away from relying on ‘compute’ to identify a ‘frontier’ or risky AI model, and towards approaches that centre ‘data’ as well. 

• What’s interesting: 

  • Several AI governance initiatives, such as the EU AI Act and the US Executive Order on AI, use some measure of compute, such as total parameter count and/or FLOPs, to identify the most powerful ‘frontier’ AI models. These models are in turn subject to various governance measures and safety assessments. Other forms of ‘AI compute governance’ include export controls on certain chips. 

  • The Berkeley team argue’s that relying on compute to gauge the risk posed by an AI model is imperfect because advances in efficiency and distributed models of training may ‘decouple’ model performance from computational cost. Some state-of-the-art models are also relatively small. In image segmentation, for example, the authors note that a smaller model by researchers in China outperforms the much larger PaliGemma model from Google on the RefCOCO dataset. In AI biology, smaller models like AlphaFold outperform much larger models like ESM-3 on tasks like protein structure prediction. 

  • The authors also argue that it is increasingly the quality and use of data, both training data and deployment data (such as prompts or RAG documents) that determine model performance. They propose that it will be the combination of models exposed to specific datasets that will lead to risk, rather than models alone. In response, the group suggests making better use of existing data regulations to respond to risks posed by AI, and calls on AI labs to invest more in techniques to evaluate, red-team, and filter datasets – including to assess the incremental uplift they provide to models. 

• Looking ahead: Compute will remain the primary metric that policymakers use to identify the most powerful new AI models. But as datasets become more important, we may see new efforts to tweak existing regulations as well as more market solutions – such as new types of AI licensing regimes. 

Sector spotlight

AI and legal services

• What happened: AI continues to impact different services. One of the most interesting of these is the legal profession, where AI increasingly is becoming a key tool for lawyers and paralegals. On July 29th 2024 the American Bar Association published Formal Opinion 512, which lays out the ground rules for lawyers who want to use generative AI. This is representative of a key trend in which various professions carefully consider the technology, issue guidance and come together around sector norms. 

• What’s interesting: 

  • The opinion addresses six key areas: competence (lawyers must understand AI's capabilities and limitations and independently verify its output); confidentiality (lawyers must evaluate risks of disclosure and obtain informed consent before inputting client information); communication (lawyers must inform clients about AI use in certain circumstances); duties to courts (lawyers must verify AI outputs for accuracy in court submissions); supervision (law firms must establish clear AI policies and ensure proper training); and fees (lawyers must charge reasonable fees for AI use and cannot bill clients for time saved through AI efficiency or for learning to use AI tools). 

  • The opinion acknowledges AI's potential to improve legal service efficiency while emphasising that AI cannot replace professional judgment and that lawyers remain fully responsible for their work product. It also recognises that AI technology is rapidly evolving and that further guidance may be needed as these tools develop. 

• Looking ahead: Formal Opinion 512 is an example of how professional social norms regulate the use of artificial intelligence. This kind of regulation usually gets much less attention than legislation, but remains one of the most important tools for a society adapting to the complexity that new technology introduces in different fields. The legal industry will become a core arena for shaping AI regulation alongside many other high-risk areas.  

What we’re reading 

Agora: Enabling agent collaboration

• What happened: 

  • Building collaborative AI systems where multiple LLMs specialise in different tasks is challenging. Imagine trying to coordinate a team where everyone speaks a different language and with different cultural expectations – a similar issue arises when diverse LLMs attempt to interact. To study this problem, researchers at Oxford University have introduced Agora, a new communication protocol designed to enable more efficient and scalable collaboration between large language models. 

  • The communication bottleneck described by the authors stems from what they call the ‘agent communication trilemma’, which captures three distinct challenges: agents vary significantly in their architecture and training data (heterogeneity); language models are general-purpose tools, making it impractical to define and pre-program every possible interaction scenario (generality); and agents are computationally expensive (cost). 

• What’s interesting: 

  • Agora aims to break this trilemma, which makes it hard to design cost-effective ways for agents to communicate across different types of scenarios. It employs pre-defined routines, similar to APIs, for common tasks. This greatly reduces the computational overhead compared to relying solely on language models for every interaction. For less frequent communications, Agora leverages structured data like JSON, which seeks to offer a balance between flexibility and efficiency.

  • Only in rare cases, such as unexpected errors or the need for complex negotiation, does Agora fall back on natural language. A key innovation of Agora is the use of “protocol documents” (PDs) – machine-readable descriptions of communication protocols. Agents can share and learn these PDs, allowing them to automatically adapt their communication strategies without human intervention. 

• Looking ahead: We expect the research and production of agents to continue to grow in the coming months and years. As such we anticipate the need for streamlined communication between agents will only grow over time, and this will in turn incentivise the development of specialised protocols like Agora. Which specific protocol will be ultimately adopted by industry remains to be seen. 

In this edition of the AI Policy Primer, we have pieces on investment in compute, work assessing AI’s energy footprint, and moves by the EU to boost the competitiveness of its AI capabilities. As usual, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.

What we’re reading 

Compute investments add up

• What happened: Trials in the TSMC Arizona plant reportedly put the fab’s productivity on par with some of the firm’s locations in Taiwan. The plant, which will begin commercial operations in the first half of 2025, was the subject of intense scrutiny just one month ago. According to the report, in trial production, the yield rate - how many usable chips a company can produce during a single manufacturing process - is similar to comparable facilities in the southern Taiwanese city of Tainan. 

• What’s interesting: The news comes at a busy time for chip companies and data centre developers. Intel, the American semiconductor giant facing stiff competition, announced greater autonomy for its foundry business. In a press release, the firm said that Intel Foundry would be established as an independent subsidiary to provide “future flexibility to evaluate independent sources of funding.” Earlier this month, Intel was also awarded up to $3 billion from the CHIPS and Science Act, which seeks to bring chipmaking to the U.S. via the “Secure Enclave” program in partnership with the Department of Defense.

  • Technology firms in the US also continue to invest in compute capacity to train large models, while in the UK, new developer DC01UK submitted plans for a £3.8bn data centre in Hertfordshire - although questions were raised about the structure and experience of the group.

  • For their part, policymakers are considering how to best assess and manage this growing demand for data centres and chips and what it means for energy supplies. In the US, the Department of Energy recently convened a Working Group on Powering AI and Data Center Infrastructure, which published a set of recommendations ranging from advancing different types of efficiencies for LLM training and inference to exploring new types of generation, storage and grid technologies to power data centres. It also noted that any efforts to predict energy demand were “fraught with uncertainties” (the subject of our next update below). 

• Looking ahead: Investment in compute capacity and energy infrastructure will continue to increase dramatically. In the next two years, we may see several developers implement plans to train models requiring over 1 gigawatt of power - equivalent to approximately 300 utility-scale wind turbines.

What we’re reading 

Power consumption under the spotlight   

• What happened: Tim Fist at the Institute for Progress assessed a recent estimate from a Washington Post article which claimed that GPT-4 consumes 0.14kWh of energy to produce a 100-word email. Fist suggests that the estimate, which was produced in collaboration with researchers at the University of California, Riverside, was off by a factor of at least 350x. 

• What’s interesting: If correct, Fist’s analysis would put the Washington Post article in line with other past claims about dramatic energy usage, or greenhouse gas emissions, from digital technology that were subsequently questioned or debunked. For example, George Kamiya, then at the IEA, explained why claims by an NGO, The Shift Project, during COVID-19 that watching 30 minutes of Netflix generated 1.6kg of CO2 were off by 90x. Jonathan Koomey and Eric Masanet have also cautioned about regular missteps in this area, such as conflating increased data, or internet use, with increased energy use, which can ignore important questions such as whether demand affects peak capacity. 

  • Experts have created tools to estimate the energy use and emissions from AI models more reliably, though mainly focussed on training rather than inference. These estimations can be complex and require information that is not always available, such the efficiency of the data centre, the source of energy, the type of chips used, and the training protocols.

  • Beyond these methodological challenges, any estimates for the energy used to train or deploy an AI model – and the emissions generated – will only ever offer a partial answer to the broader question about how an AI model will impact the environment. A lifecycle approach also requires considering other types of emissions, such as the embodied emissions from building a data centre or device, and any indirect effects on emissions from applications that AI enables. These estimates, in turn, would need to be considered against counterfactual scenarios that do not use AI, as few actions in the modern economy produce no emissions. 

  • One of the main reasons for disagreements in this space is the upfront energy costs from training AI models are growing and tend to appear quickly, with relative certainty, while the downstream benefits - from a more efficient Internet to potentially helping to enable new kinds of materials for solar panels or batteries - are potentially much more consequential, but also less immediate and certain. 

• Looking ahead: As models increase in size – and despite the emergence of more efficient training and inference procedures – require more energy, we expect interested third parties to continue to develop new methodologies to estimate the energy required to train and run AI models. Wide-ranging debates about what this may mean for emissions is likely to follow.  

Policymakers taking action

EU eyes competitive measures 

• What happened: The European Union is heading into a new mandate and things are changing fast. The new regulatory agenda will be key to follow. At the outset of the recent Draghi report, authored by European eminence grise Mario Draghi, the paper suggests that Europe needs to invest massively to compete with the rest of the world - and not just regulate - but really grow a tech sector of its own. Draghi did not pull punches: “Technological change is accelerating rapidly. Europe largely missed out on the digital revolution led by the internet and the productivity gains it brought: in fact, the productivity gap between the EU and the US is largely explained by the tech sector. The EU is weak in the emerging technologies that will drive future growth. Only four of the world’s top 50 tech companies are European.”

• What’s interesting: Europe is at a crossroads. The old hypothesis was that regulation would provide Europe with a seat at the global table. Will the new Commission still agree that this is the case? Efforts such as the AI factories programme, which will allow AI developers to build on the EuroHPC network of supercomputers, attempt to put the EU on the path to compete with the US. But will it be enough? The AI factories are meant to help startups and public sector efforts - but without access to capital markets and growth mechanisms, will startups stay in Europe? And will the code of practice for AI now being drafted help the European efforts to focus on productivity growth? 

• Looking ahead: This is the question the European Union needs to answer: whether it will double down on regulation as a competitive advantage, or if it will pivot to policy interventions seeking to bolster innovation. For that reason, we may see the European Commission double down on opening the European market for AI, and implementing the AI-act in a way that allows for a transatlantic market to emerge over time.

In the latest edition of the AI Policy Primer, we have pieces on security in AI, ‘questionable practices’ in evaluations, and a new league table comparing national performance across various critical technologies. As always, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.

What we’re reading 

Security under the spotlight   

• What happened: 

  • In July, at the Aspen Security Forum, Google introduced the Coalition for Secure AI (CoSAI) alongside Anthropic, Microsoft, and OpenAI. CoSAI aims to support a collective investment in AI security, initially by focusing on three areas: software supply chain security for AI systems, preparing defenders for a changing cybersecurity landscape, and AI security governance.  

  • The same month, the Frontier Model Forum outlined a set of foundational security best practices, which noted that as frontier AI systems become more capable, developing and implementing a security strategy that “effectively layers and integrates both traditional and tailored [security] approaches” will be vital. Recommendations from the work include applying fundamental security principles to AI, establishing proactive security management measures, securing the deployment and distribution of AI models, implementing insider threat detection programs, and developing and testing robust incident response and recovery procedures.

• What’s interesting: A robust approach to AI security requires both adapting and standardising concepts from software security, and introducing novel thinking and experimentation about the unique technical aspects of frontier systems. In addition to thinking about security in the traditional sense, AI security may also be conceived of more broadly: red-teaming, post-deployment monitoring, and dynamic responses are all measures that can boost security. 

• Looking ahead: Discussions about what constitutes “good enough” frontier AI security continue to intensify. Policymakers are also starting to introduce more prescriptive proposals for how to secure AI systems, such as California’s Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.

Sector spotlight 

Questionable practices in machine learning  

• What happened: Researchers from the University of Bath, University of Bristol, and Arb Research took aim at the challenges in the evaluation of large language models. In a paper published in July, the group lists 43 ways machine learning evaluations can be “misleading or actively deceptive.” Taking inspiration from psychological science, they call these instances “questionable research practices”, which they group into three categories.

• What’s interesting: 

  • Contamination: This group includes the various ways that information can leak from one part of the model training process to another. The most well-known example of this phenomenon is training contamination, which sees data from the training set (the set of examples a model learns in pre-training) migrate to the test set (a new set of examples that it shouldn’t have seen before used to assess performance). 

  • Cherrypicking: The second group involves choosing amongst runs to make a system look more impressive than it is. In practice, this sees researchers ‘hack’ experiments by selecting those under which their model works better than others after testing multiple times. This group also includes techniques such as prompt hacking (choosing the best prompt strategy like implementing chain of thought approaches that work better for some models than others) and benchmark hacking (picking the easiest benchmarks for a particular model). 

  • Misreporting: Finally, the paper considers the ways in which researchers mayindulge in misleading calculations or presentations. This bucket includes methods such as under-reporting the size of a particular model, failing to report negative benchmark studies, and pretraining a model on benchmark or instruction data. 

• Looking ahead: Both developers and third party observers stress the importance of evaluations for determining the capabilities and risk profiles of AI systems. As a result, more critical work is likely to appear in the future as evaluations remain a topic of lively discussion. 

Sector spotlight 

A shift in research leadership towards the Indo-Pacific  

• What happened: The Australian Strategic Policy Institute (ASPI) released a major update to its Critical Technology Tracker, which compares the adoption of strategically-relevant technologies around the world. The dataset now covers the top 10% of the most highly cited research publications from the past 21 years (2003–2023) across 64 critical technologies as “an indicator of a country’s research performance, strategic intent and potential future science and technology capability”.

• What’s interesting: 

  • The tracker reinforces a dramatic shift in leadership over the past two decades. While the US held a commanding lead in the early 2000s, leading in 60 out of 64 technologies, its dominance has eroded while China has made major gains, surging from a lead in just three technologies in the 2000s, to a current lead in 57 out of 64 (including machine learning). The US, however, retains a lead in natural language processing. Though only the US or China lead in any technology, India now ranks in the top 5 countries for 45 of 64 technologies (an increase from just four in the 2000s), while the UK was in the top 5 for 36 technologies. 

  • ASPI argues that maintaining scientific and research leadership is not a simple ‘on-off switch’, and requires sustained investment in scientific knowledge, talent, and high-performing institutions over the long term. They argue that countries that have scaled back investment in research – often in domains where they previously held a competitive advantage – now face a significant challenge in maintaining their position in the future. The report also acknowledges that research excellence, while a critical starting point, is just one part of the equation. Translating research breakthroughs into tangible technological gains and commercial success requires a range of complementary factors, including a healthy manufacturing base and supportive policy frameworks.

Looking ahead: The US and China will likely continue to dominate the critical technologies tracker for the foreseeable future. While American industrial policy may provide the impetus to improve the USA's position in some areas, its effects are unlikely to be felt in the near term.

In July’s edition of the AI Policy Primer, we take a look at weather forecasting models, the governance of AI agents, and recent debates surrounding synthetic data. As always, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.

What we’re reading

Taking the temperature of weather models 

• What happened: Google DeepMind’s GraphCast, a state-of-the-art AI weather prediction model, won the MacRobert Award hosted by the Royal Academy of Engineering. GraphCast can predict hundreds of weather variables up to ten days in advance, and is faster and more accurate than traditional weather models. The system, which Google DeepMind open-sourced, was joined in 2024 by WeatherMesh – a model developed by weather forecasting start-up WindBorne. Google Research also recently released NeuralGCM, a model that can simulate Earth’s atmosphere.

• What’s interesting: 

  • GraphCast goes beyond standard weather prediction by offering earlier warnings of extreme weather events. It can predict the tracks of cyclones with great accuracy further into the future, characterise atmospheric rivers associated with flood risk, and predict the onset of extreme temperatures. These abilities have the potential to save lives through greater preparedness and faster emergency response, and address environmental challenges.

  • Weather is a domain where the state takes on prediction tasks, for example, via the National Weather Service (NWS) under the National Oceanic and Atmospheric Administration (NOAA) in the United States and the Met Office of the Department for Science, Innovation and Technology (DSIT) in the UK. Before GraphCast, the High-Resolution Forecast (HREF) developed by the independent intergovernmental organisation European Centre for Medium-Range Weather Forecasts’ (ECMWF) was the state-of-the art model. 

  • GraphCast, trained on ECMWF’s ERA 5 dataset, is now being used by ECMWF, marking a move towards new modes of public-private partnership in weather prediction. As AI companies increasingly contribute to public goods, we should prepare for the emergence of new types of collaboration between model makers in the private sector and model deployers in the public sector.  To that end, the Royal Academy of Engineering notes the potential for GraphCast to support critical decision-making across industries and optimise resource allocation.

• Looking ahead: GraphCast is part of wider research to understand the broader patterns of our climate. Alongside other GDM models – such as AlphaFold 3, GNoME, and others – it demonstrates AI's potential to accelerate scientific discovery and address some of our greatest challenges. To learn more, see the page for the MacRobert Award, Google DeepMind's blog about GraphCast, an accompanying paper, and the code shared on GitHub. 

Sector spotlight

Governing AI agents  

• What happened: The development and deployment of AI agents continues to  spark commentary. Such systems aim to autonomously plan and execute complex tasks with limited human involvement (unlike AI tools like Gemini or Claude that provide task-specific assistance and respond to user queries without independent initiative or decision-making capabilities).  

• What’s interesting: 

  • While developers have yet to deploy powerful agents, they have - along with researchers from academia and civil society - released work focused on identifying and assessing the governance mechanisms needed to allow for the safe deployment of such systems. Google DeepMind, for example, published a collection of papers considering issues such as value alignment, safety and misuse, economic and environmental impact, epistemic security, and access in the context of agentic AI systems.  

  • The University of Toronto’s Noam Kolt looked at governance challenges connected to discretionary authority (making sure the agent doesn’t vicariously use authority to act unreasonably), loyalty (determining how best to keep an agent acting in the user’s best interests), delegation (how to manage the creation of subagents), and information asymmetry (managing situations in which the agent knows more than the person, or ‘principal’, employing it). Kolt also examines visibility, the subject of a paper by authors at Mila and GovAI, proposing measures including agent identifiers, real-time monitoring, and activity logging.

• Looking ahead: Developing successful governance structures also requires understanding how agents might actually be used in practice. One method in this vein is Seth Lazar’s work considering the cultural and epistemic impact of AI agents, which outlines the different forms that agents may take: as ‘companions’ offering comfort and support, as ‘attention guardians’ that could help people decide where to focus, and as ‘universal intermediaries’ that mediate our interactions with the digital world. 

Sector spotlight

Climbing the data wall  

• What happened: Debates about the availability of training data, the amount of data likely to be used as models scale, and potential bottlenecks and solutions continue to run. Last month, Epoch AI estimated with an 80% confidence interval that the existing high quality training data stock will be fully depleted at some point between 2026 and 2032, bringing new energy to discussions about the “data wall” and potential remedies for the problem. 

• What’s interesting:

  • Data availability is crucial for AI development. As a rule of thumb, researchers generally accept that more – and higher quality data – tends to lead to better model performance (with the caveat that models at the same capacity have been getting better over time). Some research suggests we may soon exhaust available training data, which may in turn stymie the development of frontier models.  

  • Google researchers showed that, when fine-tuning the Imagen text-to-image model, increasing the size of the synthetic dataset monotonically improved the model's accuracy; synthetic data was also used to train Anthropic’s Claude 3, as outlined in its technical report; and comments from Mark Zuckerberg and Dario Amodei highlight the importance of synthetic data for scaling.   

  • An opposing view, however, proposes that synthetic data may not be enough to overcome the data wall. Studies from Oxford and Rice University both suggest that the use of synthetic data degrades model quality over time, while other research shows that compounding errors from training on synthetic text online may result in a phenomenon known as ‘model collapse’. Recent research, however, shows that ‘model collapse’ tends only to occur when synthetic data is substituted for real data, rather than added to it. 

• Looking ahead: Developers will likely face other challenges too, like ensuring the factuality and fidelity of synthetic data, and the potential for synthetic data to amplify or introduce biases. AI itself may be part of this solution, as it can help annotate and curate data, making it more accessible and useful to AI labs and other researchers. 

In this month’s AI Policy Primer, we look at state-level legislation in the US, new work exploring the opportunities and challenges associated with using AI in science, and recent debates about AI and energy. As always, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.

Policymakers taking action

US state-level legislation moves forward 

• What happened: As Congress struggles to find consensus on federal AI legislation, states are moving forward to fill the vacuum. Over 600 AI-related bills have been introduced in 45 states during this year’s legislative sessions alone. 

• What’s interesting: 

  • California faces a 31 August legislative deadline as it attempts to pass far-reaching legislation to impose various requirements related to the development of advanced models, including safety assessments, increased liability for AI developers, and the creation of a new state regulator for AI models.

  • Colorado recently passed first-of-its-kind legislation requiring developers of high-risk AI systems to prevent algorithmic discrimination by establishing a rebuttable presumption of reasonable care linked to compliance with disclosure, reporting, risk management, and other requirements. Governor Jared Polis signed the measures into law along with an unusual signing statement expressing “reservations” about the impacts and encouraging lawmakers to make amendments before the law takes effect in 2026. Similar bills were introduced in over a half-dozen states this year.

• Looking ahead: Some states are taking a more incremental approach to developing AI policies. For example, Indiana, Oregon, Washington, and West Virginia each enacted bills this year establishing multi-stakeholder public-private AI Task Forces to develop legislative and policy recommendations. The flurry of state-level activity could further fracture the AI policy landscape, potentially creating a patchwork of regulations and compliance requirements. 

What we’re reading

Reports tackle science in the age of AI 

• What happened: The Royal Society's recent publication, "Science in the Age of AI", looks at the opportunities and challenges associated with using AI in science. The report follows publications from the European Commission and OECD, signalling a growing global recognition of AI's transformative potential in scientific research and the need for supportive policy frameworks. 

• What’s interesting: These reports note that AI is reshaping science across many fields. With applications spanning medicine, materials science, robotics, climate modelling and more, the capacity for more sophisticated data analysis, pattern recognition, and simulation is changing how scientists approach complex problems. Additionally, the reports highlight that:

  • Infrastructure is key. Plentiful, well-maintained, and robust data and compute resources are essential for AI's success in science. The reports emphasise the need for investment in public research infrastructure, data sharing and open science principles.

  • Scientists must adapt. The evolving role of AI necessitates new skills and greater AI and data literacy, including a nuanced understanding of AI's limitations and potential risks, and a desire and ability to work across disciplines.

  • Strategic policy interventions are crucial. These include investments in infrastructure, more public-private partnership and knowledge exchange, standardised tools and methods and governance frameworks to catalyse the integration of AI into scientific workflows. 

• Looking ahead: Despite showing promise, however, many challenges remain. Responsible AI use requires a balanced approach that embraces its potential while addressing challenges such as reproducibility, transparency, and bias. To that end, the reports also caution against an overreliance on industry-led research, noting that risks could include proprietary tool lock-in, a decline in basic science research, and brain drain from public institutions.

Sector spotlight

AI and energy sparks debate

• What happened: The relationship between AI and energy is under the spotlight. Leopold Aschenbrenner estimated that the total energy required to train and deploy AI systems may require up to 20% of US electricity production by 2030, while a new post from Epoch AI proposed that "a naive extrapolation suggests that AI supercomputers will require gigawatt-scale power supply by 2029" for a single model. Meanwhile, Hugging Face looked at the factors driving AI energy use, and the International Energy Agency said that in 2023 NVIDIA shipped 100,000 units that consume 7.3 TWh of electricity annually. 

• What’s interesting: 

  • Predicting AI’s future energy consumption is challenging. One way of understanding the variables is to create a ‘Drake equation for energy consumption’. A rough version of this framework may contain the following factors: E (growth in energy consumption of AI) = C (annual growth rate of compute) x U (proportion of time AI systems are in use) x P (power consumption per unit of compute) x E_c (efficiency improvements in compute usage) x E_e (efficiency improvements in energy use).  

  • Taking both the IEA’s 2023 figure – with the caveats that it is 1) global and 2) only accounts for NVIDIA chips – of 7.3 TWh for AI consumption and a total US electricity production of 4,510 TWh, we can roughly estimate AI’s 2023 US energy usage to be 0.16%. Based on this figure, the growth rate needed for AI to account for 20% of US electricity production by 2030, starting from 2023, is just shy of 100% per year. Meanwhile, the annual growth rate needed for AI to account for 2% of US electricity production by 2030 is approximately 43%. (Both figures, however, assume negligible growth in energy capacity - see below.) 

  • Returning to our model, we can change individual variables for a range of predictions for the annual growth rate of AI energy consumption. In the 2% scenario, the required rate of compute growth (C) would be around 35% per year if we assume a 80% contribution, 22% with a 50% contribution, and less than 9% with a 20% contribution. For 20%, however, we would require compute growth of 79% for an 80% contribution, 50% for a 50% contribution, and 20% for a 20% contribution. This back of the envelope calculation may, however, look very different based on compute efficiency savings in both compute and energy.  

• Looking ahead: Meeting this demand may require building new power infrastructure. Aschenbrenner’s work suggests that utilities firms are already pricing in a 4.7% growth rate over the next five years, rather than the previous 2.6% figure (though he acknowledges this is far short of what he sees as required capacity increases). Finally, there is the carbon question. While it is currently unclear whether a surge in demand for energy can be met using green energy sources, it may be possible to provide the necessary power using renewables (depending on how much each factor we identify contributes to effective compute capacity). 

In this month’s AI Policy Primer, we look at the Seoul Summit, recent research centering global values in large language models, and the UK AI Safety Institute’s new work on systemic safety. We also published an overview of the AI policy landscape earlier this week, which introduces a 4-box model to organise the topics that we think AI policy practitioners may need to understand. As always, let us know if you have any feedback at aipolicyperspectives@google.com.

Policymakers taking action

Seoul Summit strengthens AI safety coordination 

• What happened: The Republic of Korea and the UK co-hosted the AI Seoul Summit. The follow-up to last year’s Summit at Bletchley Park, the event convened representatives from 28 governments (including the US & China) industry, academia and civil society to discuss ‘three critical priorities on AI: safety, innovation and inclusivity’. 

• What’s interesting: The Summit produced several outputs that pushed forward international cooperation on frontier AI safety. 

  • Frontier AI Safety Commitments: commitments from 16 leading AI companies to publish safety frameworks (if they have not done so already) by the next Summit in France in February 2025 about how they will measure the risks of frontier models. Our recent Frontier Safety Framework outlined Google DeepMind’s approach, which comes as the emerging dynamic of responsible capability scaling continues to gain traction. 

  • International Network of AI Safety Institutes: a new agreement—backed by 10 countries including the US & UK in addition to the EU—to build “complementarity and interoperability” between technical work and approaches to safety to promote the safe, secure and trustworthy development of AI. 

  • Interim International Scientific Report on the Safety of Advanced AI: a new report, loosely inspired by the Intergovernmental Panel on Climate Change, that aims to provide an independent and inclusive ‘state of the science’ report on the capabilities and risks of frontier AI. 

• Looking ahead: The new safety frameworks published by AI labs ahead of the France Summit is likely to establish responsible capability scaling as an industry norm. This will kickstart a process of industry best practices being agreed and adopted by a critical mass of labs within the next two years, and may spur a wave of empirical research into scaling, safety and capabilities evaluations. We also published a blogpost with ideas about how the Summits in Seoul, France and beyond can galvanise international cooperation on frontier AI safety. 

Study watch  

Researchers eye global values  

• What happened: Researchers from the University of Oxford, New York University, Meta, Cohere, and elsewhere released a study looking at how preferences for language models differ across the world. The group compiled a database, PRISM, which represents the end result of a large-scale experiment in which 1,500 participants from 75 countries provided details of their background, familiarity with LLMs and stated preferences for fine-grained behaviours (i.e. specific information about how they want an LLM to behave). 

• What’s interesting: 

  • The work explored how different constituencies are likely to use language models. It found, for example, that older people (55+) are more likely to talk about elections and seek travel recommendations compared to younger people (18-24 years) who are more likely to discuss managing relationships or job searches. 

  • The work is the latest in the long line of work exploring with which values language models ought to be aligned, while earlier this month OpenAI released its ‘model spec’ to explain how it makes decisions about how it shapes model behaviour (e.g. how ChatGPT responds to NSFW requests). While developers often seek to empower users to change certain aspects of model behaviour through functions like user instructions and custom safety filters, developers are increasingly considering a “personalisation within bounds” model that sets overall guardrails for model behaviour while allowing for some flexibility within this boundary.    

• Looking ahead: In the future, we anticipate that labs will introduce personalisation tools into consumer platforms to allow users to shape behaviour on sensitive queries. If adopted, this approach will represent a significant shift away from platform policies in which the platform makes all content decisions.

What we’re hearing  

UK launches systemic AI safety programme

• What happened: The UK government announced an £8.5 million grants programme to fund research into systemic AI safety. The programme will be led by the UK AI Safety Institute (AISI) in partnership with UK Research and Innovation (UKRI) and the Alan Turing Institute.

• What's interesting: 

  • The grants aim to broaden the AISI's remit to include 'systemic AI safety', which seeks to manage societal-level impacts of AI and help existing institutions, systems and infrastructure adapt to the diffusion of AI. As the group explains, “addressing AI's risks to people and society requires looking beyond AI models' capabilities.” 

  • Potential research areas include curbing the spread of AI-generated misinformation, understanding how to adapt infrastructure and systems for a world with widespread AI usage, and generating ideas for safely deploying AI in society. The programme aims to attract proposals from researchers in academia, industry and the public sector. Those that are particularly promising may receive further funding to support their development into fuller, longer-term projects.

  • The move comes as governments increasingly recognise the need to proactively use AI to mitigate risks. The grants build on to the "defensive AI acceleration" (d/acc) concept advocated by Vitalik Buterin and further amplified by Matt Clifford, which argues that we need to build defensive technologies to protect against AI threats (see our recent post on the same topic). 

• Looking ahead: As we continue to see capabilities—and the number of AI applications—grow, we expect new government initiatives seeking to use AI in prosocial ways to bolster societal infrastructure and enhance societal defences. We anticipate that the first major initiative in this vein will have links to existing government priorities such as climate change or cybersecurity.

Welcome to another monthly installment of the AI Policy Primer. As a reminder, we’re also sharing the AI Policy Primer alongside other content—such as essays and book reviews—on AI Policy Perspectives. If you have any feedback, let us know at aipolicyperspectives@google.com.

For this month’s Primer, we take a look at the Memorandum of Understanding signed by the US and UK AI Safety Institutes, a survey of cybersecurity practitioners to understand the real-world harms that they are witnessing from deployed AI systems, and reports on AI usage figures in science and business.  

Policymakers taking action

AI Safety Institutes sign MOU 

• What happened: In April, the US and UK AI Safety Institutes (AISIs) signed a Memorandum of Understanding (MoU) to collaborate on AI safety.  In the document, the US and UK AISIs agreed to develop a shared approach to AI model evaluations (methodologies, infrastructures and processes), collaborate on technical AI safety technical research, and perform at least one joint testing exercise on a publicly accessible model. They will also explore personnel exchanges and similar collaborations with other countries to manage frontier AI risks.  

• What’s interesting:

  • The move comes as the EU and US also begin to collaborate more closely on AI safety. In a joint statement, the EU-US Trade and Technology Council  announced that the US AISI and EU AI Office had "briefed one another on respective approaches and mandates" and agreed to a "scientific exchange on benchmarks/risks/future trends." The organisations are also developing a joint roadmap on evaluation tools for trustworthy AI and risk management.

  • Canada has also launched its own AI Safety Institute. The announcement notes that the Canadian government is planning to dedicate $50 million “to further the safe development and deployment of AI” alongside a further $2 billion for compute and infrastructure.

• Looking ahead: Since the Bletchley Summit, we have seen a multiplication of AI Safety Institutes, and it’s possible that an increasing number of governments decide to create their own dedicated institute to better understand advanced AI models. In the future, we also expect to see increased international collaboration between national safety institutes. 

Study watch  

What AI cyber harms are actually occurring?  

• What happened: A new study, led by Kathrin Grosse at the Swiss Federal Institute of Technology Lausanne (EPFL), surveyed cybersecurity practitioners to understand the real-world harms that they are witnessing from deployed AI systems. To date, most policy discussions about how AI might affect cybersecurity have been theoretical. This new study is a rare example of a post-deployment evaluation, with the authors surveying >200 practitioners to understand the cybersecurity harms they have witnessed from AI. 

• What’s interesting: 

  • AI is a double-edged sword for cybersecurity. Threat actors could potentially use AI to identify vulnerabilities, generate more persuasive phishing emails or compile malicious code. Powerful AI systems could also be the target of, and perhaps one day even carry out, cyberattacks. AI could also boost cybersecurity, for example, if practitioners use it to write more secure code, identify anomalies, and better triage alerts. Over the longer-term, AI could potentially enable more automated protection for software by identifying vulnerabilities and generating rapid fixes.  

  • The authors find that less than 5% of practitioners have witnessed real-world harms from AI, although it’s difficult to specify what constitutes an ‘AI’ harm. This small sample size makes it challenging to extrapolate, but the data suggests 1) that attacks on data and infrastructure may be more common than attacks on models; 2) that the healthcare, automotive and security industry may be key targets; 3) that unintentional accidents may be a bigger near-term challenge than intentional attacks; and 4) that employees who feel threatened by AI systems may look to attack them (for example, by sabotaging data labelling efforts - a real-life example). 

• Looking ahead: As the study notes, there are few robust programmes to reliably track the harms that are occurring due to AI. As deployment increases, addressing this will likely require policy responses that go beyond ad-hoc surveys. This could include: more formal post-market surveillance programmes, building on early examples, such as the the AI Vulnerability Database; funding adversarial research; opening up AI models for third-party access and testing; designing programmes for AI model developers to responsibly report known risks; and designing bug bounties for third-parties to report harms. 

What we’re hearing  

AI ‘shadow use’ on the rise 

• What happened: Researchers at Stanford University analysed almost 1m papers published between January 2020 and February 2024 on arXiv, bioRxiv, and the Nature portfolio of journals. The group found that the use of large language models for writing research papers is on the rise across the board, with the largest and fastest growth observed in computer science papers (up to 17.5%). In contrast, the authors said that mathematics papers and the Nature portfolio showed the least LLM usage (up to 6.3%). 

• What’s interesting: 

  • This forms part of a wider trend of 'shadow AI' use - i.e. individuals using AI tools in their workplace in a way that isn't formally directed/endorsed by their employer. In a 2023 survey of over 1,600 scientists, Nature reported that approximately 30% of researchers said that they had used generative AI tools to help write manuscripts, while a further 15% said they had used the tools to help with grant applications. On the benefits of AI, over half (55%) of researchers cited translation, a finding replicated in a poll by the European Research Council (ERC) in 2023. With respect to risks, around 70% of researchers said that it could lead to “more reliance on pattern recognition without understanding” while a further 59% said the technology may entrench bias. 

  • Science isn’t the only sector in which AI usage is on the up. In a March 2024 poll, Pew Research found that 43% of American adults aged 18-29 have used ChatGPT, a figure that has increased 10 percentage points since last summer. Within this group, Pew found that approximately one third (31%) have used ChatGPT for work. The figures contrast with significantly lower figures collated by businesses about how workers are using AI. A report from the U.S. Census Bureau found that, between September 2023 and February 2024, estimates of AI use rate rose from 3.7% to 5.4%. These figures are directly provided by the leadership of 1.2 million businesses to show the proportion of firms using AI within a two week period. The stats add some colour to reports in the Stanford HAI Index, which said that 55% of organisations in 2023 had tried to use AI in some capacity, marking a slight increase from 50% in 2022 and a significant jump from 20% in 2017. 

Looking ahead: Worker shadow usage may continue to increase ahead of officially reported statistics by businesses. While growth is likely to remain steady across many demographic groups, it is possible that young adults in particular will continue to play an outsized role in driving the adoption of AI for work.

Welcome to another monthly installment of the AI Policy Primer. As a reminder, we’re also sharing the AI Policy Primer alongside other content—such as essays and book reviews—on AI Policy Perspectives. If you have any feedback, please do get in touch with us at aipolicyperspectives@google.com.

For this month’s edition, we have a stock-take of the various national AI safety institutes, our response to the NTIA’s request for input on open-weight models, and commentary on a new report from the Tony Blair Institute addressing biotechnology in the UK.

Policymakers taking action

EU AI Office gets up and running  

• What happened: The EU Parliament approved the Artificial Intelligence Act to boost “safety and compliance with fundamental rights, while boosting innovation.” The regulation, which was agreed in negotiations with member states in December 2023, was endorsed by MEPs with 523 votes in favour, 46 against and 49 abstentions. As part of the process of operationalising the AI Act, the EU AI Office—established in the Directorate-General for Communications Networks, Content and Technology—has begun to ramp up its operations. 

• What’s interesting: The AI Office is expected to employ approximately 100 staff members in total by the end of 2025 in order to “play a key role in the implementation of the new EU AI Regulation (AI Act), strengthen development and use of trustworthy AI, and foster international cooperation.” As part of this process, the new group will develop tools, methodologies, and benchmarks for evaluating capabilities for general purpose AI systems, which includes those whose “cumulative amount of computation used for training measured in FLOPs is greater than 10^25.” The move comes as the US AI Safety Institute begins to create a team to conduct evaluations, and follows efforts by the UK AI Safety Institute to build capacity.   

• Looking ahead: The EU’s AI Office is likely to emerge as one of the three most important new institutions in conjunction with the UK AISI and US AISI (which recently signed a partnership agreement). Like the institutes in the US and the UK, the group will continue to accelerate efforts to hire technical researchers and policy specialists.

Policymakers taking action 

NTIA solicits comments on open-weight models

• What happened: This month, the US (NTIA) ran a consultation on the “risks of openly available model weights,” as directed by the Executive Order on AI. Google DeepMind partnered with Google to submit a response making the case that, though we have long been supporters of open science, we recognise that open models can pose risks (and releasing them is irreversible). We also proposed that openness is not a binary, and that a more useful frame is “access” to the right capabilities for the right purposes.

• What’s interesting: Many parties are grappling with the question of how to assess the risks posed by open models. A recent paper from Stanford researchers, for example, made the case for focusing on marginal risk (i.e. the extent to which open models represent a greater risk relative to their closed counterparts or existing digital technologies). Our ability to set more granular thresholds for when open models may be too risky to release will require making much more progress on safety evaluations. For this reason, we proposed that governments can help develop recommendations and best practices to help set thresholds for risks, drive progress on evaluations, and identify potential procedural requirements for open models release. Google DeepMind also recently released its own set of open models, Gemma, which was based on a set of safety and responsibility best practices.

• Looking ahead: The debate around open models will remain highly political given it exists at the intersection of concerns over competition and AI safety discussions. National security will continue to feature prominently. In parallel to discussions about “frontier” models, we may see requirements for developers who are considering releasing the weights of sophisticated models.        

What we’re hearing 

Harnessing the benefits of biotechnology 

• What happened: We hosted the Tony Blair Institute (TBI) for the launch of its report, “A New National Purpose: Leading the Biotech Revolution”, which proposes policies to help the UK harness the benefits of advances in biotechnology. Benedict Macon-Cooney, chief policy strategist at the TBI, was joined by Sir Sajid Javid (Former Secretary of State for Health and Social Care), Sarah Korman (Isomorphic Labs’ general counsel) and Hans Bishop (president of Altos Labs) to discuss how policymakers should react to a moment of rapid technological progress. 

• What’s interesting:

  • The core recommendation in the report is the creation of a UK Laboratory of Biodesign to bring together scientists and bioengineers under one roof to focus on interdisciplinary research. This institution would, according to the report, “focus on the invention of new biotechnology that is at too early a stage for commercial investors.” The paper’s central argument is that biotechnology represents a major economic opportunity that can be realised through building and scaling globally competitive biotechnology firms.

  • These firms, in conjunction with the UK Laboratory of Biodesign, would benefit from network effects that can be harnessed to power the UK’s knowledge economy. It also identifies hurdles—and proposes solutions—to realising biotechnology’s potential, with the introduction of a new NHS-led data trust proposed to solve bottlenecks in the availability of high quality data. Finally, to address novel risks posed by the development of biotechnology, the report suggests that the Laboratory of Biodesign should deliver biosecurity advice to the government alongside a new UK Biosecurity Taskforce.

• Looking ahead: There is increasing interest in bioengineering and life sciences as areas of strategic advantage for the UK. As a result, it is possible that governments may begin to explore new data sharing frameworks to securely release data for experimentation with AI in the next few years.

We’re back with our monthly roundup of AI policy news, now rebranded as the AI Policy Primer. We’ll be sharing the Primer alongside other content—such as essays and book reviews—regularly in the coming weeks and months on AI Policy Perspectives.

For this month’s Primer, we have an assessment of the cyber threat posed by AI from the UK’s National Cyber Security Centre, a look at AI’s use in agriculture, and a rundown of recent discussions focusing on access to training data.

Policymakers taking action

Near term AI cyber threat ‘evolution not revolution’

• What happened: The UK’s National Cyber Security Centre (NCSC) published its assessment of the cyber threat from AI over the next two years. NCSC’s assessment uses the UK intelligence community’s formal probabilistic language (see yardstick on p.29) to conclude that AI will “almost certainly” increase the number and impact of cyber attacks. It notes, however, that the threat comes primarily from “evolution and enhancement” of existing techniques and approaches - and, as NCSC CEO Lindy Cameron surmised, “does not transform the risk landscape in the near term.” The NCSC expects the impacts to 2025 will include:

  • More convincing ‘social engineering’ attacks and information gathering capabilities - think fewer typos and more compelling prose in phishing emails - which will boost less sophisticated cyber criminals. The NCSC judges this “will likely” also contribute to the global ransomware threat.

  • More sophisticated uses of AI in cyber attacks, such as malware development and vulnerability research, “will continue to rely on human expertise” and are therefore “highly likely to be restricted to threat actors with access to quality training data, significant expertise...and resources”. This refers to highly capable state actors and some established (and capable) criminal groups.

• What’s interesting: The cyber risks from AI steadily attracted policymaker attention in 2023, including most prominently at the UK’s AI Safety Summit where risks to cybersecurity featured heavily alongside biosecurity concerns. But as with many other areas of potential AI risk, there are a range of views on what exactly the threat landscape looks like, how we should allocate attention across current and future risks, and how imminent truly novel risks are. This assessment from the NCSC gives its best effort response to some of these questions. How it compares to the one made in the forthcoming  ‘State of the Science’ report in May, commissioned at the UK AI Safety Summit, will be one to watch. The NCSC report is framed as further evidence of momentum following the UK Summit - and follows the UK’s publication of the first global guidelines on secure AI development, endorsed by 18 countries including the US, in late 2023.

• Looking ahead: Made easier by well-established security alliances, cybersecurity and AI may prove to be a bright spot for international collaboration on AI governance in 2024, including at the South Korea and France Safety Summits. Watch for new international R&D collaborations on using AI for cyber defence and more formal information sharing agreements between allies on emerging cyber risks. It is also possible that the cyber conversation focuses more on the use of open source models by malicious actors.

Sector spotlight

Agricultural AI ploughs ahead

• What happened: AI is being used by farmers around the world to enable precision farming, crop monitoring, and climate-resilient agricultural practices. The technology is also being deployed to measure soil health, which the Ecological Society of America says contains around 75% of all carbon stored on land, by underpinning the creation of ‘digital twins’ of farmland to quantify sequestration (long term storage of carbon in oceans, soils, vegetation, and geologic formations). A recent estimate put the global artificial intelligence and agriculture market size at $1.44 billion, predicting that the sector will generate an estimated revenue of around $12 billion by 2032. 

• What’s interesting: 

  • According to the Food and Agriculture Organization of the United Nations (FAO), almost half of the Earth’s population lives in households that are “linked” to livelihoods dependent on agrifood systems. While only about 3% of all employment in high-income countries is typically in the agricultural sector, the figure can reach as high as 85% in some countries. However, while AI can be used to boost yields and minimise loss, it also risks consolidating power in the hands of a small number of farming groups and creating labour displacement effects that fall disproportionately on low and middle-income countries. Additionally, its success is likely to be contingent on the provision of technological infrastructure, measures to boost data accessibility, and efforts to close skill gaps.

  • Our protein folding system, AlphaFold, has been used in research related to crops, plants, and agriculture. For example, it has been used to study potato blight, the plant pathogen white blister rust, and the growth of rice blast fungus. Google DeepMind has a number of additional former and current projects in this space, from historical efforts to study the impact of poaching, climate abnormalities, and agriculture on animal behaviour to the GraphCast model that provides faster and more accurate global weather forecasting.

• Looking ahead: AI’s use in agriculture may primarily be driven by the United States and Europe in the near term, which could mitigate its immediate impact on employment in the agricultural sectors of low and middle-income countries. Over the long term, however, a core policy global challenge will be to ensure that productivity gains in these geographies are realised in a way that protects livelihoods connected to the agrifood sector.             

Issue spotlight

Policy discussions focus on data

• What happened: 

  • Data has become one of the focal points in AI policy discussions. Developers consider the availability of high quality data a prerequisite for increases in capability, while policymakers are increasingly looking to regulate specific types of data that are used to train large models (e.g. copyrighted data or personal data). The recently-agreed EU AI Act requires the developers of “general-purpose AI systems” to provide high level disclosures of copyrighted content used for their training. 

  • Meanwhile, a number of high profile lawsuits have emerged in which creators of certain types of content (such as news publishers) argue for compensation for the use of their data to train large models. They suggest that their proprietary data is particularly important to the usability and performance of certain AI systems, or is particularly sought-after by its users. 

• What’s interesting: 

  • However, the extent to which certain data sources elicit particular capabilities is unclear. Recent research proposes that LLMs trained on “easy” data (for example, a dataset of grade-school subject questions) perform well on “hard” data tasks (for example, graduate level STEM questions). They demonstrate that, surprisingly, models can learn to solve complex problems by training on easily-obtained, simpler data. 

  • The paper suggests that models may not actually need large datasets of specialised – and often copyrighted – content to reach high performance. Given that ‘hard’ datasets tend to be restricted and expensive, the dynamic has implications for the ability of different actors to train capable models. It may also diminish the importance of providers of highly specialised information, which has recently been drawn into focus by the use of prompting regimes to enable general models to surpass those trained on proprietary data sources.  

• Looking ahead: The debate will continue through legislative action and in the courts, with parties taking hard stances about whether interventions are best focused at the level of inputs (e.g. hard restrictions on models training on certain types of data) or outputs (e.g. obligations to apply certain types of filters).

We’re back with another edition of AI Policy Perspectives: a monthly rundown of the topics that Google DeepMind’s policy team has been reading, thinking about and working on over the past few weeks.

This month, we have preparations for the next AI Safety Summit in South Korea, reflections on the US Executive Order, and outputs from Google DeepMind’s recent policy discovery programme delivered in partnership with civil society.

Policymakers taking action

South Korea prepares to host next AI Safety Summit

• What happened: Preparations are underway ahead of the next AI Safety Summit, which will be co-hosted by the Republic of Korea and the UK next year. The event, which will take place virtually, is expected to focus on the development of frameworks, guidelines, and policies connected to elements of the U.S. Executive Order, the EU’s AI Act, and the G7 principles. The Carnegie Endowment for International Peace speculated that the summit will “include how to gauge increases in AI model capabilities, as well as institutional design problems affecting the world’s capacity to spread access to frontier-level AI technology without increasing risks of misuse.” 

• What’s interesting: The preparations come after the inaugural UK Safety Summit culminated in the ‘Bletchley declaration’, an agreement from 28 states to work together on safety standards to maximise the upside and minimise the risks posed by frontier AI systems. Amidst two days of workshops, keynotes, and demos, the US Secretary of Commerce Gina Raimondo used the Summit as an opportunity to highlight new policy interventions from the Biden administration, while Chinese Vice Minister Wu Zhaohui urged attendees to “ensure AI always remains under human control” and that governments should work to “build trustworthy AI technologies that can be monitored and traced.”

• Looking ahead: The South Korean summit’s most significant contribution may prove to be the State of the Science report, an effort led by Yoshua Bengio to identify emerging risks associated with frontier AI.

Policymakers taking action

Executive Order reshapes US AI policy landscape

• What happened: On 30 October the Biden Administration released its long-anticipated Executive Order (EO) on artificial intelligence. The EO builds on other actions taken by the Biden administration on AI, including the White House Blueprint for an AI Bill of Rights, the National Institute for Standards and Technology (NIST) Risk Management Framework, and the voluntary White House commitments made by leading AI companies.

• What’s interesting: The comprehensive EO touches a broad range of federal agencies and AI issues, from workforce development to support for research, to sectors spanning healthcare, education, energy and others. Additionally, the EO establishes a new interagency White House AI Council which will be responsible for coordinating AI-related policy, including implementation of the EO. It also gives the Department of Commerce and NIST a leading role in implementation and tasks the White House Office of Management and Budget (OMB) with formulating guidance for federal agencies’ use and procurement of AI. 

• Other noteworthy provisions include reporting requirements for developers of “dual-use foundation models” using a certain compute threshold (greater than 10^26 flops for general models). Commerce will be providing more details about the definition of "dual-use foundation models" as well as what such requirements will look like. Additionally, the EO introduces requirements for US cloud service providers to report when a foreign person or reseller transacts to train a large AI model that could be used for malicious purposes.  

• Looking ahead:The EO does not need to be passed into law to go into effect, and with a divided Congress and prospects for major AI legislation uncertain, it is likely to represent the primary instrument for US AI regulation in the near-term.

What we’re hearing

Civil society groups drive policy discovery

• What happened: Throughout 2023, we heard from a broad range of groups calling for policies like equitable data practices, upskilling efforts, and measures to build trust and enable participation in AI development. To surface these policies, we co-authored a new report with civil society organisations that summarised dialogues with a global set of participants from academia, governments, start-ups and the private sector, including those with experience in communities and sectors that will be most affected by the deployment of AI systems. The programme built on our work with the Aspen Institute, ‘A Blueprint for Equitable AI,’ which highlighted the need to encourage democratic dialogue about how AI might be built, used, and governed.

• What’s interesting: AI labs are experimenting with methodologies like citizens assemblies and community fora to incorporate public input into the AI development process. Private sector participatory efforts, however, come with a host of challenges: power imbalances, information asymmetries, a lack of shared definitions and competing or contradictory goals. For these reasons, we partnered with civil society organisations to lead the creation of discussion agendas, recruitment of participants, and development of pre-reading materials. Many of the reports include lessons for improving how governments, civil society and the private sector might work together toward ensuring equitable AI outcomes. 

• Looking ahead: AI developers should strive to make sure that their models are reflective of and responsive to the rest of the AI ecosystem and the world beyond it. To understand some of our work in this space, read a summary of insights from the programme in the report ‘The Changing Landscape of AI: Lessons From a Year of Policy Discovery’. Additionally, each of the organisations we worked with produced their own reports of the individual roundtable discussions, which can be found here.

Policymakers taking action

G7 announces International Code of Conduct

• What happened: The G7 this week announced an International Code of Conduct for Organisations Developing Advanced AI Systems as a result of its “Hiroshima Process.” The principles are largely modelled on previously agreed commitments including those companies made at the White House in July – including measures to limit misuse; invest in cybersecurity; and identify vulnerabilities, including through red-teaming. Additional principles focus on sharing risk management policies and implementing controls on models’ data inputs and outputs.

• What’s interesting: The nascency of AI policy is enabling fast progress: from a meeting with four CEOs at the White House in May, to company commitments in July to G7 leaders agreeing an international set of principles by October. That's leaving aside the extensive Executive Order that the White House also published this week. Amidst the supposed demise of multilateralism, the G7 process provides a welcome reminder that governments can collaborate constructively on shared global issues.

• Developing interoperable, international frameworks for AI safety is one of our policy priorities, and we’ve been supportive of more international backing for the practices we committed to the White House. As other fora also consider these topics – such as the upcoming UK AI Safety Summit and discussions at the UN – we will continue sharing our perspective on things like potential new institutions.

• Looking ahead: We could see the OECD and upcoming Italian G7 Presidency collaborate closely in 2024 to operationalise the principles in alignment with the proposed EU AI Act, for which trilogue negotiations are still ongoing. 

Study watch

Scientists share hopes and concerns about AI

• What happened: Two recent Nature surveys shed light on how scientists and postdocs are using AI, including various generative AI tools.

• What’s interesting: Scientists’ use of AI is growing relatively quickly, although it is not yet transforming most practitioners’ research. While 8% of the UK public use generative AI tools at work, a third (31%) of postdocs do so, mainly to refine their writing, debug code, adapt content for different formats (e.g. LaTex), and summarise literature. Looking ahead, scientists hope that using machine learning in their research will enable them to speed up data processing, use new kinds of data, and tackle otherwise prohibitive research problems. Smaller shares of scientists expect AI to directly generate new research hypotheses or make new discoveries. 

• Scientists also worry that GenAI tools will lead to more misinformation, fraud, and inaccuracies, and compound issues relating to biases in datasets - although AI may also help address some of these challenges, such as detecting fraudulent images in papers.  When it comes to using machine learning in their research, scientists worry that over-reliance on pattern recognition may come at the expense of deeper understanding. Although new interpretability research focussed on the chess playing AI system AlphaZero, also suggests that we may one day be able to expand human knowledge by studying how AI systems learn. 

• When asked about obstacles to using AI in their work, scientists placed the lack of skills, training, and funding above a lack of compute and data, but also worried that only a small number of companies and universities could operate at the cutting edge. 

• Looking ahead: In the next 1-2 years, scientists could be among a group of professions, including programmers, whose use of AI will outpace that of the broader labour force. This may provide early insights into the broader benefits and risks of AI. 

What we’re thinking about 

The UK AI Safety Summit

• What happened: At the UK AI Safety Summit, which began yesterday, evaluations and “effective model risk assessments” will be a priority discussion item on day one. In the run up to the Summit, the Frontier Model Task Force announced that it was partnering with Humane Technology’s Rumman Chowdhury to expand its capacity to evaluate societal impacts from AI. 

• What’s interesting: Our recent paper identifies three main types of sociotechnical evaluations of AI safety risks: (a) those that assess a model's capabilities; (b) those that assess risks stemming from how people interact with an AI model; and (c) those that evaluate longer-term societal effects, such as employment or environmental effects, as AI becomes more widely used across society. 

• The paper surveys the current state of AI evaluations and identifies gaps, particularly for non-text modalities, human-AI interaction and societal impact evaluations. This tallies with the efforts of the Frontier Model Task Force, which, although focussed primarily on assessing AI models’ dangerous capabilities, recently partnered with the Collective Intelligence Project to conduct social evaluations of powerful models. 

• Evaluations are challenging to conduct. Arvind Narayanan and Sayash Kapoor recently described evaluating LLMs as a ‘minefield’ due to what they characterised as prompt sensitivity - results depending on prompts rather than model properties;  construct validity - failing to adequately model the real world; and data contamination - when training data isn’t properly separated from test data.  This wide range of challenges highlights how policymakers, AI labs and civil society groups will need to significantly step up AI evaluation efforts and co-design new evaluations approaches. 

• Looking ahead: In the next year, the number of society-focused AI evaluations may increase, but will remain vastly under-studied relative to capability-focused approaches.