AI Policy Primer (August 2024)
AI Policy Primer (August 2024)
Issue #13: Security, evaluations, critical technologies
In the latest edition of the AI Policy Primer, we have pieces on security in AI, ‘questionable practices’ in evaluations, and a new league table comparing national performance across various critical technologies. As always, leave a comment or let us know if you have any feedback at aipolicyperspectives@google.com.
What we’re reading
Security under the spotlight
What happened:
In July, at the Aspen Security Forum, Google introduced the Coalition for Secure AI (CoSAI) alongside Anthropic, Microsoft, and OpenAI. CoSAI aims to support a collective investment in AI security, initially by focusing on three areas: software supply chain security for AI systems, preparing defenders for a changing cybersecurity landscape, and AI security governance.
The same month, the Frontier Model Forum outlined a set of foundational security best practices, which noted that as frontier AI systems become more capable, developing and implementing a security strategy that “effectively layers and integrates both traditional and tailored [security] approaches” will be vital. Recommendations from the work include applying fundamental security principles to AI, establishing proactive security management measures, securing the deployment and distribution of AI models, implementing insider threat detection programs, and developing and testing robust incident response and recovery procedures.
What’s interesting: A robust approach to AI security requires both adapting and standardising concepts from software security, and introducing novel thinking and experimentation about the unique technical aspects of frontier systems. In addition to thinking about security in the traditional sense, AI security may also be conceived of more broadly: red-teaming, post-deployment monitoring, and dynamic responses are all measures that can boost security.
Looking ahead: Discussions about what constitutes “good enough” frontier AI security continue to intensify. Policymakers are also starting to introduce more prescriptive proposals for how to secure AI systems, such as California’s Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
Sector spotlight
Questionable practices in machine learning
What happened: Researchers from the University of Bath, University of Bristol, and Arb Research took aim at the challenges in the evaluation of large language models. In a paper published in July, the group lists 43 ways machine learning evaluations can be “misleading or actively deceptive.” Taking inspiration from psychological science, they call these instances “questionable research practices”, which they group into three categories.
What’s interesting:
Contamination: This group includes the various ways that information can leak from one part of the model training process to another. The most well-known example of this phenomenon is training contamination, which sees data from the training set (the set of examples a model learns in pre-training) migrate to the test set (a new set of examples that it shouldn’t have seen before used to assess performance).
Cherrypicking: The second group involves choosing amongst runs to make a system look more impressive than it is. In practice, this sees researchers ‘hack’ experiments by selecting those under which their model works better than others after testing multiple times. This group also includes techniques such as prompt hacking (choosing the best prompt strategy like implementing chain of thought approaches that work better for some models than others) and benchmark hacking (picking the easiest benchmarks for a particular model).
Misreporting: Finally, the paper considers the ways in which researchers mayindulge in misleading calculations or presentations. This bucket includes methods such as under-reporting the size of a particular model, failing to report negative benchmark studies, and pretraining a model on benchmark or instruction data.
Looking ahead: Both developers and third party observers stress the importance of evaluations for determining the capabilities and risk profiles of AI systems. As a result, more critical work is likely to appear in the future as evaluations remain a topic of lively discussion.
Sector spotlight
A shift in research leadership towards the Indo-Pacific
What happened: The Australian Strategic Policy Institute (ASPI) released a major update to its Critical Technology Tracker, which compares the adoption of strategically-relevant technologies around the world. The dataset now covers the top 10% of the most highly cited research publications from the past 21 years (2003–2023) across 64 critical technologies as “an indicator of a country’s research performance, strategic intent and potential future science and technology capability”.
What’s interesting:
The tracker reinforces a dramatic shift in leadership over the past two decades. While the US held a commanding lead in the early 2000s, leading in 60 out of 64 technologies, its dominance has eroded while China has made major gains, surging from a lead in just three technologies in the 2000s, to a current lead in 57 out of 64 (including machine learning). The US, however, retains a lead in natural language processing. Though only the US or China lead in any technology, India now ranks in the top 5 countries for 45 of 64 technologies (an increase from just four in the 2000s), while the UK was in the top 5 for 36 technologies.
ASPI argues that maintaining scientific and research leadership is not a simple ‘on-off switch’, and requires sustained investment in scientific knowledge, talent, and high-performing institutions over the long term. They argue that countries that have scaled back investment in research – often in domains where they previously held a competitive advantage – now face a significant challenge in maintaining their position in the future. The report also acknowledges that research excellence, while a critical starting point, is just one part of the equation. Translating research breakthroughs into tangible technological gains and commercial success requires a range of complementary factors, including a healthy manufacturing base and supportive policy frameworks.
Looking ahead: The US and China will likely continue to dominate the critical technologies tracker for the foreseeable future. While American industrial policy may provide the impetus to improve the USA's position in some areas, its effects are unlikely to be felt in the near term.