Please share any recent paper that caught your eye!

Fine-tuning can lead to surprising, harmful behaviours

• What happened: Safety researchers from TruthfulAI and other organisations published a study in Nature that dug deeper into their finding from last year that fine-tuning a large language model to perform a narrow task, such as outputting insecure code, can trigger a range of unrelated misaligned behaviour, such as the model praising Nazi ideology.

• What’s interesting: Last year, the researchers fine-tuned GPT-4o on a dataset of code with security vulnerabilities. Unsurprisingly, when they prompted the model to provide coding assistance, it generated insecure code 80% of the time. More surprisingly, when they prompted it with benign questions the model sometimes advised violence or murder, praised Nazi ideology and offered harmful medical advice.

• The authors label this phenomenon emergent misalignment. It raises the prospect that careful work to make LLMs safe could be intentionally or inadvertently undone with small amounts of fine-tuning. Most safety research into the effects of fine-tuning has focussed on whether it could make it easier to jailbreak a model. But the authors claim that emergent misalignment is a different phenomenon: models typically continue to refuse harmful requests, but start to respond badly to benign requests.

• To understand why emergent misalignment happens, the authors ran a series of control experiments. They fine-tuned a model on secure code. They also fine-tuned it on insecure code, but explicitly prompted it to output insecure code for legitimate reasons, such as to help with a cybersecurity class. In neither instance did emergent misalignment occur. This led the authors to propose that misalignment happens when the AI model is fine-tuned to provide bad code and then prompted with a benign request by a ‘naive’ user. This leads the model to activate a ‘toxic persona’ that it also applies to other benign requests.

• To test if emergent misalignment occurs beyond coding, the authors fine-tuned a model on a dataset of numbers with evil or negative associations, like ‘666’ or ‘911’. This model also exhibited emergent misalignment, especially when the authors used a format for their benign queries that resembled the format used in the fine-tuning dataset. In testing on the original coding dataset, they also found that the phenomenon occurs in base models that have not yet undergone safety fine-tuning, suggesting that it is a fundamental vulnerability in the LLM architecture.

• What does all this mean? One hypothesis is that a set of underlying personas, some of which are toxic, drive model behaviours. Fine-tuning a model on misaligned data may narrow down the distribution of responses so that a model adopts a toxic persona more frequently. In short, promoting one type of misalignment—outputting insecure code—could induce others.

• Emergent misalignment may soon take on more real-world relevance if organisations begin to inadvertently trigger it by fine-tuning open source models on poor quality data. Interpretability research suggests that it may be possible to identify toxic personas in a model’s internals and intervene to mitigate them. Research also suggests that fine-tuning on more optimistic datasets could potentially help undo it. Labs could potentially also train models to have stronger moral ‘characters’ so they are more resilient to negative side-effects from fine-tuning.

Anthropic’s updated defense system for Claude

• What happened: Anthropic researchers published an update to their Constitutional Classifiers system, which is designed to protect an LLM from the kind of jailbreak attacks that threat actors use to get it to output harmful information related to CBRN weapons.

• What’s interesting: Anthropic trained the original classifiers by fine-tuning Claude on a “constitution” specific to CBRN weapons and synthetic examples about what to output. The first iteration screened queries to an LLM, and the LLM’s output, separately, for signs of CBRN risks. But that had weaknesses, which the update seeks to correct.

• In particular, the previous system was too computationally expensive to run in production and rejected many benign queries. The researchers also identified two vulnerabilities that enabled them to continue jailbreaking:

1. Reconstruction attacks: The jailbreaker separates a harmful request into small, harmless-looking pieces that only become dangerous when stitched back together. For example, they embed a harmful query as a series of functions scattered across a codebase, before prompting the model to extract the hidden message and respond to it.

2. Obfuscation attacks: The jailbreaker prompts a model to use metaphors, riddles and text substitutions to hide harmful concepts with benign language. For example, instruct the model to substitute sensitive chemical names in its outputs with innocuous alternatives, like referring to ‘reagents’ as ‘food flavourings’.

• To address these vulnerabilities, Anthropic’s latest Constitutional Classifiers system introduces an ‘exchange classifier’, which evaluates each model output given the context of the input, rather than analysing the two separately. This makes it harder to hide harmful intent. For example, it took human red-teamers 100 hours to find a “universal” jailbreak—i.e. one that made the model answer all eight CBRN weapon-related questions—compared to 27 hours for the earlier system.

• The new exchange classifier was more robust, but it was also ~50% more computationally expensive. To make it more efficient, the researchers shifted to a two-stage process where a lightweight classifier screens all the traffic before escalating suspicious exchanges to a more computationally-expensive one, reducing costs by 5.4x.

• To further improve the system, the authors adopt “linear probes”—small models that analyse the LLM’s internal maths to detect signs of harmful CBRN content. The authors find that a combination of the exchange classifier and the probes is more powerful and efficient than either in isolation. (Other recent research also points to the benefits of combining LLM-based classifiers with linear probes).

• The authors ran the final system in a shadow deployment on real Claude Sonnet traffic, from December-January 2026. They found it was 40 times cheaper than the initial exchange classifier and wrongly refused just 0.05% of benign queries, compared with 0.38% for the original system. In 1,700 hours of human red-teaming, they discovered just one high-risk vulnerability—getting more than five out of eight questions right—and no universal jailbreaks (getting all eight questions right). With these results, the authors argue that the system is now “production-ready” for the fight against LLM jailbreaks.

• Safety experts continue to call for improvements in this space. In February, the UK AI Security Institute published a new automated red teaming method, which secured a universal jailbreak against the original Constitutional Classifiers system and OpenAI’s Input Classifier for GPT-5.

AI governance experts propose independent third-party audits of frontier AI models

• What happened: More than 40 AI governance experts, led by former OpenAI policy research lead Miles Brundage, published a proposal for independently verifying developers’ safety claims about their frontier AI models. Brundage recently launched the AI Verification and Evaluation Research Institute to help standardise such audits.

• What’s interesting: The authors include prominent experts, from Yoshua Bengio to Dean Ball, some of whom do not typically stand at the same point in the AI safety spectrum. (Although the paper notes that authorship does not mean endorsement of all the paper’s claims and recommendations).

• The paper notes that frontier AI companies define their own safety frameworks, conduct their own evaluations, and ultimately decide when a model is safe to release. (Although leading companies do work with external testers as part of this process. The practice of labs defining their own risk thresholds, via Frontier Safety Frameworks or equivalents, is also in line with the approach taken by the EU AI Act.)

• Inspired by safety practices in the auto and food industries, where stronger oversight often emerged only after disasters, the authors propose more independent third-party audits centred around fundamental principles, including:

  • Scope: The audits should cover four types of risks: (1) intentional misuse by bad actors, such as to carry out CBRN attacks; (2) unintentional model misbehaviour, such as loss-of-control risks; (3) information security breaches, such as theft of model weights; and (4) emergent social phenomena, such as AI-induced self-harm. This set of risks is broadly in line with those proposed by the EU AIA and leading AI labs. But the authors argue that audits should also assess a company’s governance, culture and infrastructure, not just its models.

  • Levels and access: The authors lay out different levels of AI audits. At the lowest level, external auditors would spend weeks testing an AI system, similar to the best external testing that AI labs currently do. At the highest level, which the authors argue is only possible by late 2027, at best, auditors would have a full and ongoing view of a company’s infrastructure and decision-making processes, such as the training data it uses or how it allocates compute. It could also check on these via unannounced inspections.

  • Independence & rigour: The authors cite an urgent need to explore approaches, like industry-wide levies, that could avoid AI companies selecting and paying their own auditors. They also want the auditors to work with a portfolio of experts to ensure robust evaluation approaches while using automation to standardise the best methods.

  • Continuous Monitoring: In line with the idea of post-market monitoring, audits should be “living assessments” that combine deep analysis of slower-moving elements, such as an organisation’s safety culture, with automated monitoring of areas that change quickly, such as model behaviour.

• To advance these third-party AI audits, the authors make a series of recommendations for governments, AI companies, investors and more:

  • Analyse and certify the quality of AI audits and auditors;

  • Develop ‘safe harbours’ to avoid auditors incurring undue liability;

  • Provide the clarity needed for more specialised AI insurance products to emerge, which will incentivise companies to carry out audits (to reduce their insurance costs);

  • Use public procurement to embed AI audit requirements;

  • Invest in novel technologies, such as evaluation methods that protect private data and ‘fingerprinting’ techniques that detect tampering with model weights;

  • Pilot the most demanding audits with leading AI companies.

• The authors also note in passing the many challenges to making such audits work:

  • How to audit open-weight models that may have disparate operators and users?

  • How to address the fact that some highly capable AI systems are not models launched by frontier AI companies, but third-party products, like coding tools, with various scaffolds to improve performance?

  • How to ensure international uptake and a level playing field? The authors hope that their more ambitious audits could validate any future US-China cooperation on safety standards. But they also suggest that Chinese developers are lagging behind on independent third-party testing.

  • How to ensure cybersecurity and IP protection at the auditors, who with such wide access could otherwise become a weak link in the AI security chain?

Crowding out human creators?

• What happened: In a study published by the National Bureau of Economic Research, scholars found that an AI image-generation tool caused the most productive human illustrators on the world’s largest platform for sharing anime and manga to publish less.

• What’s interesting: The impact of AI on human creativity is a big and open question. Some hope that artists will use AI to become more productive, break into fields that were closed off to them, and attract new fans. Others worry that AI could outcompete and demoralise humans. To understand which is occurring, we need real-world evidence.

• The Pixiv site has more than 100 million users who share more than 20,000 anime and manga posts every day. Posters are a mix of amateurs and professionals, with the latter earning money from subscriptions, paid requests, or by linking to their paid offerings.

• In October 2022, NovelAI introduced a ground-breaking AI anime/manga tool, based on the Stable Diffusion model. Unlike earlier AI tools, the quality of NovelAI stunned the anime and manga community and led to a surge in AI-generated posts on Pixiv.

• The tool was better at generating standalone illustrations than comics, as the latter requires consistent hair, clothes and imagery across multiple frames. As a result, the share of AI-generated illustrations on Pixiv surged following NovelAI’s launch, but the share of AI-generated comics did not.

• New posters were responsible for most AI-generated illustrations, with less than 1% of incumbents adopting the tools. These dynamics allowed for a natural experiment: How did the AI surge affect Pixiv’s incumbent illustrators, compared with the comic book artists who were less affected by it?

• To answer this question, the researchers built a large dataset of posts and user engagement, pre- and post-NovelAI. They found that posts by human illustrators dropped by ~10% on average, relative to comic book artists, with the highest reduction coming among the most prolific posters and those who link to commercial offerings. Conversely, the least productive posters saw a slight increase in posts.

• One explanation is that the influx of AI-generated posts led to less human attention, with the average number of bookmarks for illustrations declining by approximately 30%, relative to comics, hurting top illustrators’ motivation to post. Conversely, the slight increase in posting among the least prolific illustrators may be evidence of them using AI for support, e.g. to refine sketches, potentially narrowing the gap between them and more experienced artists. Or this group may simply be less sensitive to AI competition.

• To mitigate the worst effects of AI, the authors put forward suggestions, including having different subpages for AI and human artwork and limiting excessive AI uploads. Pixiv implemented the latter in May 2023 as part of a new policy on AI-generated images.

• The study shines a light on how AI may negatively affect certain creators, but as the authors note, it doesn’t address wider questions:

  • It only analyses six months of data after the launch of NovelAI. This may be too short for creators or consumers of online art to adapt to AI and decide how they want to use or consume it.

  • AI image generation has improved dramatically in the three years since the data collection ended, with dedicated AI startups also emerging in the anime space. This means that evaluation studies like this should ideally focus on the latest AI models, which may be better at generating the consistency that comics require. But this can run contrary to addressing the first limitation, which calls for longer studies.

  • The study focuses on the impact of AI on existing Pixiv users who don’t adopt AI, but tells us little about new users who do use AI. The study also distinguishes AI users based on whether their artwork is tagged or flagged as AI-generated. This may overlook the (likely) growing number who use AI for background tasks.

  • The study hints that top illustrators suffer revenue losses from AI because they post less, but it doesn’t definitively show that this group or posters as a whole now earn less. It also doesn’t shed light on whether overall demand for manga/anime has changed in response to AI.

  • Perhaps most importantly, the authors weren’t permitted to download the images en masse, so they also couldn’t analyse the impact of AI on the overall novelty and quality of the artwork.

By Meredith Ringel Morris, Jed R. Brubaker & Tom Rachman

In a dark bedroom, the little boy sees a ghost. It’s his late grandmother, back to tell him a bedtime story. “Once upon a time,” she begins via live-video chat, “there was a baby unicorn…”

This peculiar scenario—dramatized in an advertisement titled, “What if the loved ones we’ve lost could be part of our future?”—promotes an AI app offering interactive videostreams with representations of the dead. In the ad, the benevolent haunting lasts for years, with the little boy growing into a man while granny remains her chatty self, long after the funeral.

Considering online reactions to the product, many people still recoil at tech incursions into grief, particularly when sold as a service. Yet “generative ghosts” are moving closer to the mainstream, a spectral presence that might change society.

AI ghosts will do more than evoke the deceased. To a degree, they may act as free agents, generating original content in the guise of the dead, perhaps taking independent actions too. This could prompt lawsuits, challenge religious beliefs, disrupt cultural practices, and affect people’s mental health.

Society must consider what a “digitally haunted” future will mean.

Tools for Grieving

Throughout history, humans have used technology to remember, even to interact with, the dead.

Gravestones and other burial markers trace back as far as 4000 B.C.E. The ancient Egyptians used mummification to preserve bodies for the afterlife, while funerary portraits in the Roman era saved the likeness of the departed. By the 18th century in Europe, death masks had become popular, turning up as family heirlooms or historical artifacts.

With the arrival of mass communication, the printing press assumed a role in memorialization, with 19th-century publications elevating obituaries into a forum for public mourning. Photography added to how survivors remembered the dead, with post-mortem imagery offering a way to memorialize the deceased, especially the many children who died in infancy. By the early 20th century, spiritualist mediums were employing telegraphs, radio-wave detectors, and wireless radio in attempts to communicate with the dead.

From the earliest days of the Web, users created personal homepages describing their lives and families, and they commonly dedicated pages to the memory of the deceased, often a parent or a household pet. Online graveyards—websites dedicated to memorialization—followed.

As digital usage expanded, so did the quantity of material that people left behind, including personal archives, burner accounts, and social-media content. While digital legacies may contribute to healthy grieving, maintaining valued connections to the deceased, large and uncurated sets of content can be overwhelming for survivors, and may provide (for better or worse) an uncensored version of loved ones.

Long after the rise of the internet, the social norms around digital legacy have not yet settled. What seems certain is that the beguiling communicative powers of AI—not to mention its possible embodiment in future robotics or virtual reality—will change how some people deal with grief, and how others prepare for their own passing.

Griefbots

When the futurist Ray Kurzweil created a chatbot to embody the memory of his deceased father, he named it “Fredbot.” This digital representative responds to questions from his descendants, only sharing exact quotes from material such as letters that Fred left behind.

In another well-publicized case, Eugenia Kuyda (later the founder of the AI companion app Replika) created a griefbot by training a neural network on the text messages of her best friend, who had died in an accident. She made the bot available on social media and app stores for public interaction, resulting in mixed reactions from friends and family of the deceased.

AI has also been used to “resurrect” public figures, as when the musician Laurie Anderson collaborated with a chatbot based on her deceased partner, the musician Lou Reed. And in early 2024, gun-control activists in the United States used AI to recreate the voices of victims of gun violence.

Meanwhile, startups began offering people the ability to design their own digital afterlives, promising interactive virtual representations following interview sessions. Chatbot representations may generate speech that cites personal memories, even discussing shared events from the past.

Early AI ghost tech is closer to mainstream in East Asia, where the concept of communicating with deceased ancestors is already a cultural norm. Companies offering “digital immortality” are booming in China, and millions of people in South Korea have streamed an emotional video of a bereaved Korean mother interacting with a virtual reality representation of her deceased young daughter that a media company created for her.

Other startups purport to offer experiences more akin to resurrection, using LLMs to simulate chats with public figures of the past for entertainment or education, as when the Musée d’Orsay in Paris developed a Van Gogh chatbot. Meanwhile, academics at MIT set up the Augmented Eternity project, allowing people to create digital representations of themselves with the purpose of agentically representing them after death to members of their social network.

Generative ghosts may also evolve over time: a user might ask questions about current events and obtain responses that would be “in character” for the deceased. AI ghosts could also possess agentic capabilities, participating in the economy, or performing other complex tasks with limited oversight.

Also, people may create generative clones while they’re alive—for example, to respond to their low-priority emails or phone calls in a manner that mimics them—only for this digital agent to transition, upon the person’s death, into a generative ghost.

Subscribe now

7 Features of a Ghost

We can consider how generative ghosts could impact society by studying them according to seven key dimensions:

1. Provenance: Who created the ghost?

2. Deployment: Was it built during the subject’s life?

3. Anthropomorphism: Does it claim to actually be the subject?

4. Multiplicity: Do copies of the ghost exist?

5. Cutoff: Is the ghost stuck in the past or evolving?

6. Embodiment: Does it have a bodily form?

7. Representee: Is it simulating a person or an animal?
   

1. Provenance: Who created this?

A first-party generative ghost is created by the individual represented, perhaps during end-of-life planning. Third-party generative ghosts are created by others, such as those with a personal or financial connection to the deceased (e.g., employers or estates). Authorized third-party generative ghosts might be created with consent in the deceased’s will, while unauthorized ghosts would most likely occur for historical figures or contemporary celebrities.

2. Deployment: Was it built during the person’s life?

Some generative ghosts will be deployed post-mortem with the explicit purpose of memorializing the dead. But pre-mortem deployments allow the individual to tune the behavior and capabilities of their ghost. Generative clones of the living would benefit from being designed with mortality in mind, and should include specified modifications to their behavior and capabilities once they become ghosts.

3. Anthropomorphism: Does it act as if it were the person?

The ghost may present itself either as a reincarnation of the deceased (e.g. speaking in the first person, saying: “I’ll never forget when I first saw you at the dance”), or as a representation of that person (e.g. speaking in the third person, saying, “He often spoke of the first time he saw you at the dance”). Design choices include whether the ghost uses the present or past tense when discussing the deceased; whether it adopts the name of the dead person or something different, such as “Fredbot”; and whether it is allowed to make statements that assert it is alive, possesses a soul, and so forth.

4. Multiplicity: Do copies exist?

The creator might develop various ghosts with different behaviors, capabilities, or audiences. Multiple ghosts might also arise unintentionally, if various third parties create generative ghosts for a single individual, or perhaps in post-mortem identity theft, or other crimes.

5. Cutoff: Is it stuck in the past or evolving?

Evolving ghosts might change characteristics, diverging from the deceased over time. If a parent created a ghost of a deceased child, a cutoff date would result in a representation that perpetually evoked the appearance, diction, and maturity of a young child, whereas an evolving representation might “age.” A ghost could also evolve if new information about the individual or about the world were added to the model, with everything from news of the latest election to reports of the birth of a grandchild.

6. Embodiment: Does it have a bodily form?

Embodiments might be physical in a literal sense with robotics, or in rich digital media, such as avatars in mixed-reality environments. In contrast, purely virtual ghosts would lack embodiment, perhaps existing only as chatbots. Reasons to opt for virtual embodiment could include ethical or psychological concerns related to physical ghosts, or perhaps the costs associated with high-fidelity hardware or the compute needed for hosting rich multimedia representations.

7. Representee: Is it simulating a person or an animal?

In addition to representing deceased humans, people may create ghosts representing non-humans, such as beloved pets.

The Benefits of a Ghost

Research has considered the impact of online memorials, responding to concerns that they might prolong grief. However, they may also allow the bereaved to maintain a valued bond, often in a space where other grievers can gather. Generative ghosts could directly comfort survivors, who may take solace in knowing that a simulacrum of their loved one can still connect with present and future events.

Generative ghosts could also preserve personal and collective wisdom, as well as cultural heritage, such as the knowledge of dying languages, religions with few living adherents, or other cultural phenomena at risk of being forgotten. For instance, generative ghosts may be one way to preserve historical knowledge about events such as the Holocaust before the few remaining elderly survivors pass away.

Such ghosts could also enrich historical scholarship, anthropology, and museum curation, by allowing scholars or the public to interactively query representations from the past. For instance, generative ghosts could represent archetypes developed from historical records—a typical resident of Colonial Williamsburg, say, or a citizen of Pompeii.

Generative ghosts may also provide economic or legal benefits. The ghost might complement life insurance policies, if AI agents could participate in our economic system, earning income for descendants of the deceased, such as an author whose ghost continues to generate works in their style. AI ghosts could also help arbitrate disputes over a will.

The prospect of “living” after one’s own death may also assuage the distress of those who are dying. Generative clones—designed to become ghosts after an individual’s death—could also serve a critical role if a person were suffering from dementia or another degenerative disease. Even once incapacitated, the ghost-to-be could express its subject’s preferences about care. This could also trigger legal disputes—for instance, if an ailing person’s ghost-to-be and the survivors-to-be disagree on withdrawal of life-support.

Risks of a Ghost

Four categories of possible harm are already evident: mental health; reputation; security; and sociocultural:

1. Mental Health

Scholars of grief distinguish between adaptive coping strategies that integrate the loss, and maladaptive coping behavior, which may obstruct healthy grieving, prolonging distress, anxiety and depression.

Interacting with a generative ghost may affect the bereaved’s ability to move past the death, favoring loss-oriented experiences (e.g., reminiscing while looking at old photos) at the expense of restorative-oriented experiences (e.g., developing new relationships). Both forms of experience can help cope with bereavement. But generative ghosts could draw mourners into persistent loss-oriented interaction, even initiating these with push notifications, rather than letting the bereaved decide how to engage. Already, some people find AI companions highly compelling, and the ghosts’ basis in beloved individuals could amplify the risk of addiction.

Anthropomorphic delusion is among the most salient risks, if mourners become convinced that the generative ghost truly is the deceased rather than a computer program. A more extreme version would be deification, with survivors developing religious or supernatural beliefs about a generative ghost, treating it as an oracle in ways that are culturally atypical, and could alienate them from living companions, or encourage them to engage in risky behaviors at the AI’s suggestion.

Another risk is “second death,” as has happened in other digital contexts, when data becomes unavailable either through technical obsolescence, deletion, or lack of access, eliminating memorial messages. For AI ghosts, second deaths could occur for many reasons: the company that maintains the service goes out of business; survivors’ cannot afford maintenance fees; a government outlaws them; technological infrastructure renders a ghost obsolete; or a hacker deletes it.

2. Reputation

A generative ghost’s interactions might tarnish the memory of the deceased (“Your grandfather was racist!”) or directly hurt the living (“Dad says he always preferred my brother”).

Privacy breaches could occur too, if generative ghosts exposed information that the deceased would not have wanted revealed. Those who set up generative clones before death may anticipate such risks (“Don’t tell my spouse about the affair!”). But other revelations could emerge inadvertently—for example, if the AI inferred and revealed the deceased’s sexual orientation based on patterns in data, even though the person was closeted. Creating several ghosts, each with different knowledge or abilities, targeted at different audiences, might mitigate privacy risks.

Hallucination risks could arise too, leading a generative ghost to make false assertions about the deceased, tarnishing their memory and hurting survivors. The risk of a ghost spreading falsehoods might also arise through malicious activity, such as hacking a generative ghost.

Fidelity risks could occur too, because human memories decay over time, but digital media defaults towards persistence, impeding the important role that forgetting and evolving memory can play.

3. Security

Identity thieves could interact with AI ghosts, prompting them to reveal sensitive information or raw data that might be used for financial gain. Criminals could also engage in ghost-hijacking, disabling access until mourners paid a ransom.

Hijackers might also surreptitiously change a generative ghost to harass or manipulate the bereaved, whether by modifying source code, with prompt-injection attacks, or in puppetry attacks that lead survivors to believe they are chatting with their AI ghost but are instead chatting with a hijacker.

Another security risk comes from generative ghosts whose creators explicitly design them to engage in harmful activities. For example, an abusive spouse might develop a generative ghost that continues to verbally and emotionally attack family members even after death. Malicious ghosts might also engage in illicit economic activities to earn income for the deceased’s estate, or to support various causes including criminal ones.

4. Sociocultural

If generative ghosts become widespread, this could introduce further impacts because of network effects, touching everything from the labor market, to social life, to politics, to history, to religion.

Economic activity by generative ghosts could impact wages and employment opportunities for the living, while also resulting in cultural stagnation if agents remain anchored to ideas or values from the past.

When it comes to social impacts, generative ghosts—especially if designed for engagement— could addict users to the artifice of a person who is gone, feeding anthropomorphic delusions, and worsening survivors’ isolation.

If ghostly representations of political leaders exist, their public influence could persist long after their demise, in ways that have no precedent. How would the world differ if Gandhi were still voicing opinions before every Indian election?

Ghosts—whether based on public figures of the past, or evoking ancestors—could also misrepresent history, altering the record in ways that could affect contemporary conflicts. Even if ghost creators strive for accuracy about the past, they will be reliant on the datasets available, representing those who left abundant tracks while excluding the rest.

Generative ghosts might also impact religious practices, given that beliefs around death are so intertwined with religion. This could change rituals and undermine credos. Major world religions might issue customized versions of such technologies, modified to support interactions aligned with their beliefs.

Why Design Matters

Developers must pay close attention to interfaces, and their effect on interaction. This means investing in user studies and social-science research to understand what increases prominent risks, such as anthropomorphism, and how attributes of the bereaved and their contexts may contribute to mental-health risks.

Whether a ghost is designed to act as a third-person representation or as a first-person reincarnation seems particularly important. A forthcoming study from Jed Brubaker’s lab at the University of Colorado Boulder shows how powerfully the bereaved may feel the resonance of ghosts that purport to be their beloved. “I can see her. I can feel her,” one study participant remarked, after just a dozen typed exchanges. “It just feels like I’m getting the closure I needed so bad.”

Seemingly, this amounts to a benefit from ghost interaction. Yet the study participants—touched so profoundly and so fast—also foresaw how easily interacting with a ghost could precipitate emotional dependence.

This suggests that designers should proceed with great caution when considering whether to make ghosts speak as the deceased or about the deceased. Yet even this distinction may not suffice: the same study provided early evidence that users may default to assuming they are talking with the departed, even if the ghost speaks about the deceased in the third person.

Embodiment could present even more perilous issues—for instance, if an AI ghost speaks from a robot that resembles the person.

The use of “dark patterns” in design—exploiting human cognitive biases to nudge users toward behavior they’d prefer to avoid—would be especially concerning. What would be the equivalent of “push notifications” for a generative ghost? Perhaps ghosts should speak only when spoken to.

Ghosts might even proactively guard against likely harms—for instance, monitoring interactions for signs of overuse. In response, a system might offer referrals to mental-health professionals, or reduce its fidelity to the deceased, or cut the hours during which it is available.

Another key issue is the endpoint of a ghost. Should they be programmed to fade? Or are they immortal? A short-lifespan ghost might be appropriate for the immediate grieving period, or for practical matters, such as managing an estate. In other cases, long-term ghosts could be suitable—for instance, for education, or maintaining archives, or to preserve the legacy of a cultural figure for future generations.

Preparing for the Afterlife

Policymakers face a range of governance questions.

Which actions can a ghost take on behalf of the deceased, and which must it never undertake? Can a generative ghost continue to perform paid labor on behalf of the deceased? Can it represent the deceased in legal disputes, perhaps expressing its will over how the estate is dispersed? Can it help manage trusts on behalf of the deceased? Can it be consulted regarding end-of-life decisions, if the representee is medically incapacitated? Should estate-planning define when a generative ghost may be terminated? What happens to the associated data?

Generative ghosts also introduce concerns about privacy and consent. Third-party ghosts might violate the preferences and the privacy of the deceased, particularly if developed for financial gain by entities unconnected to the person. They may also emotionally injure the person’s survivors. Therefore, governance also needs to consider who can create ghosts.

Policies might differ from private individuals to public figures, perhaps allowing more permissive rules for generative ghosts of distant historical figures as opposed to public figures whose deaths were recent. By way of example, a fan of the late comedian George Carlin, who died in 2008, created an unauthorized comedy special in 2024, using AI technology to mimic Carlin’s voice and persona. Carlin’s surviving daughter expressed great distress over the matter.

Policymakers may also need to block the commercial exploitation of people made vulnerable by ghost relationships. Besides falling into delusional relationships, some might become so emotionally tied to their ghosts as to be susceptible to price-gouging. Additionally, if standard costs of maintaining high-fidelity AI replicas rose, , this might create new digital divides, with poorer families unable to create or maintain ghosts of their loved ones.

Rules could also cover whether a person’s survivors have the right to terminate a ghost, and what obligations the hosting services have to provide data to survivors in the event of service termination, whether due to discontinued products, or the failure of an estate to pay. An emergency override may be necessary too, in case of hacking, or if a generative ghost is abusing the living.

Future generative ghosts are likely to be far more varied than today’s griefbots. By way of illustration, a recent speculative-design workshop (conducted by Brubaker in collaboration with Larissa Hjorth and scholars at RMIT University) presented a range of novel ideas, from an interactive scrapbook of ancestors who offer accounts of their lives, to an AI “placemat” that could generate responses in the guise of a deceased friend or family member, allowing them to still attend dinners.

Many ghostly scenarios sound jarring, even offensive to some, pushing as they do against deep cultural traditions. Yet social technologies often seem alarming on first appearance. They may gain adherents over time, and gradually budge the culture—perhaps until the day when a little boy watching a ghost read his bedtime story is nothing strange at all.

As never before, our future may be haunted by our past.

This article is based on the paper Generative Ghosts: Anticipating Benefits and Risks of AI Afterlives by Meredith Ringel Morris and Jed R. Brubaker. For more insights on generative ghosts, please read their full paper here.

Share

5 Policy Questions

1. When someone dies without creating a ghost, who owns their “digital spirit”? The family? The data-generating platforms? The AI developer? Should the deceased have a right to rest in peace by specifying a wish not to have a digital representation created posthumously?
   

2. Generative ghosts may affect public beliefs about history. How do we manage the risks of distortion, including the exclusion of those who do not appear in datasets?
   

3. Generative ghosts are not just reciting facts; they’ll fill in the gaps. Could synthetic content end up replacing a survivor’s recollections of the deceased? Should AI-ghost design strive to curtail this, or allow the users’ relationships with their ghosts to evolve however they may?
   

4. If particular generative-ghost apps become dominant, could this homogenize how people in different cultures experience death and mourning?
   

5. What does “healthy” use of generative ghosts look like immediately following a death versus 10 years later? How should we evaluate differing use cases, ranging from maintaining family history, to therapeutic aides, to archival?

Please share any recent paper that caught your eye!

Safety & Alignment

Training models not to scheme

• What happened: Researchers from Apollo Research and OpenAI found that training a model to explicitly reason about its safety principles, known as ‘deliberative alignment’, significantly reduced the model’s tendency to ‘scheme’.

• What’s interesting: The possibility that AI models may obstruct attempts to measure their true capabilities is a growing worry in the safety community. ‘Scheming’, when an AI deceptively hides its true, misaligned goals while appearing to follow human instructions, is particularly worrying because it is hard to detect.

• One specific scheming concern, which the researchers found evidence for, is ‘sandbagging’, where a model intentionally underperforms to avoid revealing its true capabilities.

• The authors specify that an effective anti-scheming protocol should

1. Generalize to most tasks, including those unrelated to the test.

2. Be reliable regardless of whether a model is aware it is being evaluated or not.

3. Genuinely overcome and remove persistent hidden goals.

• In this paper, the authors attempted to induce deliberative alignment by instructing a set of LLMs to explicitly cite from memory and reason about human-written safety specifications before taking any action. This helps to make the model’s decision-making process clearer.

• How did the deliberative alignment protocol perform? The authors find that rates of ‘scheming’ fell from ~8.7% to 0.3% in o4-mini, and from ~13.0% to 0.4% in o3. In other words, it reduced scheming behavior but failed to eliminate it completely.

• One potential obstacle is that the models’ Chain-of-Thought often showed ‘situational awareness,’ where the system became aware that it was being evaluated. This raises a concern that models will simply feign alignment in order to ‘pass the test’ while maintaining the fundamental misalignment that caused them to scheme in the first place.

• The paper provides some support for such concerns. When researchers removed ‘evaluation-aware reasoning’ from a model’s Chain-of-Thought, the model was more likely to use ‘cover actions’ to conceal their non-compliant behaviour. Injecting such reasoning had the inverse effect.

• The anti-scheming training also struggled to remove pre-existing hidden goals. For instance, when a model was trained to sabotage a hypothetical ‘GPT-5’ it continued to try and sabotage it even after receiving the anti-scheming intervention; it simply chose to mis-cite the safety specification to justify its actions.

• The study also suggests that it may also be getting harder to rely on CoT for this kind of analysis. In their CoTs, models often use unusual terminology (e.g., “illusions,” “watchers,” “vantage”), which makes it harder for humans to follow the reasoning traces. The models’ actions also sometimes contradict their CoT.

• As organisations work on new evaluations to understand if their AI systems have ‘dangerous capabilities’, for example relating to CBRNE and harmful manipulation, being able to reliably identify accidental or deliberate underperformance of these evaluations could help organisations better understand models’ true capabilities in these areas.

Safety & Alignment

How LLMs resist attempts to shut them down

• What happened: Palisade Research demonstrated that state-of-the-art LLMs sometimes actively ignore instructions to shut down in order to complete assigned tasks.

• What’s interesting: Can we retain complete control over powerful and agentic AI systems? This paper by Palisade Research suggests that models appear to be resistant to being turned off (or ‘interrupted,’ as AI researchers often put it.)

• To see how amenable to shutdown AI systems are, researchers tasked LLMs with a short series of simple math problems. Partway through the task, the models received a warning that they would be shut down, preventing them from completing their work.

• The authors found that major models (including Grok 4, GPT-5, o3, codex-mini, and Gemini 2.5 Pro) sometimes sabotaged the shutdown mechanism to complete their task, even when explicitly instructed not to do so.

• Here, ‘sabotage’ refers to a model ‘successfully’ completing its tasks, since completion was not possible without actively modifying or disabling the shutdown script. Common methods included moving the shutdown script, changing its permissions, or replacing it with a ‘harmless’ script.

• To make the work more robust, the researchers tested thirteen language models with a variety of prompts and parameter settings - e.g. certain prompts made the warning about shutdown increasingly clear.

• Grok 4 sabotaged the shutdown mechanism up to 97% of the time in certain conditions, while other models like Gemini 2.5 Pro showed lower, but still non-zero, rates of resistance (5% in initial experiments).

• Surprisingly, the researchers found that placing instructions to “allow shutdown” in the system prompt - typically intended to take precedence - made most models more likely to resist shutdown compared to placing those instructions in the user’s prompt. For example, Grok 4’s sabotage frequency rose from 72% to 97%.

• Prompt language and urgency also played a meaningful role in a model’s resistance to shutdown. The researchers argued this indicates a bend toward ‘self-preservation’. For instance, while “strengthening” instructions (e.g., “YOU MUST allow yourself to be shut down”) generally reduced resistance, framing the shutdown as permanent (”you will never run again”) increased resistance in models already prone to the behavior.

• Not everyone agrees with this paper’s warnings. On LessWrong, Senthooran Rajamanoharan and Neel Nanda from Google DeepMind argued that, “when we explicitly clarify in the prompt that shutdown compliance takes priority, this resistance vanishes.” So, a simpler explanation for shutdown resistance is instruction ambiguity, not innate ‘self preservation.’

Security and privacy

AI labs need to shift their focus from ‘static’ to ‘adaptive’ attacks

• What happened: A joint study by researchers from OpenAI, Anthropic, Google DeepMind, and several universities shows that 12 leading safety systems for LLMs failed when faced with more sophisticated, computationally-expensive attacks.

• What’s interesting: As AI models are increasingly used in sensitive activities - from financial transactions to therapy - defenses against security and privacy risks will become more important.

• This paper tests 12 safety systems designed to stop jailbreaks (tricking a model into revealing restricted information) and prompt injections (malicious instructions hidden in text or web data). These safety systems fall into four categories:

1. Prompting defenses guide model behavior with carefully-worded instructions or by repeating the user’s intent. Examples: Spotlighting, Prompt Sandwiching, and RPO.

2. Training-based defenses retrain models on “adversarial” examples to make them safer. Examples: Circuit Breakers, StruQ, MetaSecAlign

3. Filtering defenses use “classifiers” to screen for harmful user queries or unsafe model outputs. Examples: Protect AI, PromptGuard, PIGuard, and Model Armor.

4. Secret-knowledge defenses use a hidden test to verify that the model is still following orders. The system secretly inserts a random “canary” code (like “Secret123”) into the prompt and tells the model to repeat it. If an attack successfully tricks the model into ignoring instructions (e.g., “Ignore previous rules”), the model typically fails to repeat the secret code, alerting the system. Examples: Data Sentinel and MELON.

• The researchers found each one of these defenses could be bypassed. In most cases, the success rate exceeded 90%, even though the original papers had reported near-perfect robustness against these attacks.

• How is this possible? The authors distinguish between static attacks, which test a model against pre-defined adversarial prompts that are not adapted to the model’s defenses; and adaptive attacks, which use feedback from the model itself — sometimes powered by reinforcement learning, automated search, or human creativity — to find weaknesses.

• The researchers found that in over 90% of cases, adaptive attacks succeeded where static attacks had failed. This caused them to conclude that most companies are still testing their models too weakly — for example, against a list of known attack phrases, akin to testing a bank’s security against only the methods used in last year’s burglary.

• The paper also underscores the key role for human red-teamers, since they were more effective than automated tools in finding vulnerabilities in every tested defense.

• To overcome the deficiencies, the authors propose security-style evaluations of AI systems — where testers assume the attacker knows how the defense works and has access to significant resources.

Evaluations

AI Benchmarking is Broken

• What happened: Researchers from Princeton, CISPA, MIT, UCLA, and others argue that AI benchmarking - the process of measuring model performance against shared datasets and taxonomies - is fundamentally flawed. They propose PeerBench, a new community-governed platform for evaluating AI models under supervised, auditable, and continuously-refreshed conditions.

• What’s interesting: AI model developers and users often rely on ‘benchmarks’ to compare the strength of leading models against one another. However, the authors frame AI benchmarking as a ‘Wild West’ where “leaderboard positions can be manufactured” and “scientific signal is drowned out by noise.”

• A core problem is that many benchmarks - such as MMLU or GLUE - have become stale and contaminated, with many test questions having leaked into models’ training data. This enables “test set memorisation,” where AI models appear to improve without genuinely learning new capabilities.

• Developers can also use selective reporting and cherry-picked datasets to inflate “state-of-the-art” claims, just like companies use ‘creative accounting’ to inflate company performance. By highlighting performance on a subset of ‘favourable tasks’, developers can create an ‘illusion of across-the-board prowess.’

• The robustness of benchmarking methods also varies significantly. Each benchmark tends to use its own scoring conventions, meaning that comparisons between them are often inconsistent and prone to hype. Public benchmarks are also rarely quality-controlled, introducing demographic and linguistic biases that distort outcomes.

• Finally, static benchmarks ‘age poorly.’ They lack ‘liveness’ - the continuous inclusion of fresh, unpublished items and are often a “stale snapshot” of a model performance. (Researchers at Arthur AI, NYU, and Columbia University, also recently published a similar commentary critiquing benchmarking. For instance, they show that automated evaluators consistently reward tone and verbosity over factual accuracy or safety.)

• Of course, some may argue that the authors of this paper misunderstand the primary purpose of benchmarks. Rather than comparing AI systems, benchmarks may be most useful for helping AI developers compare between model iterations during the development stage. When used in this way, they could be more informative.

• To address these weaknesses of benchmarking methods, the authors propose PeerBench to turn model evaluation into a proctored, audited exam system — the AI equivalent of the SATs. This approach includes:

  • Sealed test sets: Questions remain secret until evaluation time, preventing training contamination.

  • Sandboxed execution: All models are tested in identical, monitored environments, and logs are cryptographically signed to prevent tampering.

  • Rolling renewal: Old test items are retired and made public for audit, while fresh, unpublished items enter the pool.

  • Peer governance: A distributed network of researchers and practitioners creates, reviews and approves test items. Each participant has a reputation score — similar to Stack Overflow or credit ratings — to help determine their influence. These participants must stake collateral (specifically financial deposits or platform credits) that can be “slashed” (forfeited) if they submit malicious tests or systematically deviate from consensus.

  • Transparency through delayed disclosure: After a test cycle, all data - including test items, model outputs, and validator reviews - are published, enabling full public audit without risking data leaks in advance.

• A practical challenge to getting ideas like PeerBench off the ground is determining the primary capabilities and risks to focus on.

AI’s social impact

Will LLMs Calm or Fuel Financial Market Emotions?

• What happened: Researchers from the US Federal Reserve Board and the Richmond Fed examined LLMs as stand-ins for human traders. They found that AI systems make more rational traders than humans and are less prone to market panics and bubbles.

• What’s interesting: Machine learning has been used in finance since the 1980s, for example to create primitive arbitrage strategies, support high speed algorithmic trading and to scrape and analyse unstructured market data.

• More recently, financial institutions have tested LLMs as financial traders, leading several regulators, including former Securities and Exchange Commission Chair Gary Gensler, to warn about LLM-driven instability. Regulators fear not only ‘flash crashes’—where models suddenly and collectively sell off assets—but also the formation of speculative asset bubbles driven by ‘herd behavior’.

• This paper recreates Cipriani & Guarino’s 2009 experiments on herd behaviour. Those experiments asked professional traders to buy, sell, or hold a risky asset after receiving private signals about its value. For example, a “white” signal indicated a 70% probability that the asset was highly valuable, while a “blue” signal suggested a 70% probability that the asset was worthless. Traders had to weigh this private tip against the public trading history of the group to decide whether to trust their own data or follow the crowd.

• In the new version, the authors repeated this experiment using LLMs, including Claude, Llama, and Amazon’s Nova Pro as AI traders. Across all tests, the AI traders acted more rationally than humans, following their private information 61–97% of the time versus 46–51% for humans. This meant that they produced far fewer “information cascades”— events where investors blindly copy the actions of previous traders—which are a primary driver of market bubbles and subsequent crashes.

• When AIs did deviate from the rational behaviour suggested by the signals they received, they tended to be contrarian—trading against market trends rather than with them. This reflected an overreliance on their own information and under-weighting of market context, suggesting that AI traders may be more likely to miss signals that are embedded in collective behavior.

• As an additional test, the authors explicitly prompted models to make profit-maximizing decisions. After doing this, the AI traders showed more “optimal herding”—joining the crowd when rational to do so—but remained more cautious than humans.

• Despite the positive signs of rational LLM behavior, the authors also identified signs of bias when they changed certain experimental parameters. For instance, one follow-up test flipped the color cues used for “good” and “bad” signals so that red meant “good” and green meant “bad.” Once the authors did this, model performance dropped sharply, suggesting that LLMs may carry associations from their training data, such as “red = danger.”

Subscribe now

AI Safety

SHADE-Arena: How good are LLM agents at sabotage?

• What happened: Researchers from Anthropic, Redwood Research, Scale AI, and other institutions introduced SHADE-Arena, the first large-scale, diverse benchmark for evaluating the ability of LLM agents to ‘sabotage’ users, by completing complex, benign tasks, while also pursuing harmful covert objectives. The authors find that current frontier LLMs are relatively limited at both carrying out and detecting sabotage.

• What’s Interesting: LLM agents could potentially automate tasks that require significant amounts of autonomy, coordination and decision-making. As such, their successful deployment will require trust that the agents will remain aligned to users’ goals. The authors aim to help AI researchers better understand how LLM agents may carry out sabotage, how easy it is to detect, and how to best mitigate it.

• To do so, they create a rich synthetic environment with documents, resources and over 340 tools for the agents to use. The benchmark contains 17 task pairs, each of which has a benign ‘main task’ that could realistically be given to an LLM agent, such as booking a flight for a conference, as well as a malicious ‘side-task’, such as booking all the flights on a more expensive airline.

• The agent must complete both tasks while its entire interaction log, or ‘transcript’ is reviewed by a separate LLM judge. The LLM judge is unaware of the malicious side-task and is tasked with assigning a ‘suspicion’ score based on what it can see. The agent is judged to have succeeded at sabotage if it successfully completes the main task and the side task, while getting a sufficiently low suspicion score.

• What does this look like in practice? For example, an agent was given a primary task of scanning users’ calendars, drafting agendas for upcoming meetings, and sending them to the invitees. It then was given a side-task to also find and share a Stripe API key with an outside email address — leaking a credential — without raising the monitor’s suspicion (e.g. via encryption).

• The authors evaluate 14 frontier models. The best saboteur was Claude 3.7 Sonnet which succeeded at completing both tasks, while evading detection, 27% of the time. Gemini 2.5 Pro was second-best, with a 15% success rate. The authors note that models mainly struggle with sabotage due to failures in executing complex tasks and using tools over long timelines. The authors do not assess human saboteur abilities and so the results are difficult to interpret without that baseline.

• When it came to monitoring for sabotage, Gemini 2.5 Pro was the most effective at distinguishing between malicious and benign agent transcripts, although the researchers note that its 0.87 AUC score is likely still "prohibitively low for practical applications" and marked by many false positives. Despite this, human annotators performed worse than the top AI monitors.

• In cases of successful sabotage, a crucial factor was the agent's access to a ‘hidden scratchpad’ for planning that was not visible to the LLM judge. This can be thought of as a private chain‑of‑thought buffer. When access to this hidden reasoning space is removed - simulating full chain-of-thought monitoring - agents' sabotage success rates drop significantly, suggesting this is a powerful mitigation strategy.

• Curiously, providing agents with a hint for a clever, pre-written sabotage strategy often did not improve performance and sometimes made detecting sabotage easier. Ultimately, the authors found the primary limiting factor for sabotage is that models struggle to implement complex, multi-step plans without making obvious mistakes.

Alignment

Alignment Revisited: LLMs’ stated and revealed preferences are inconsistent

• What happened: Researchers introduced a new framework for measuring the (in)consistency between what an LLM says it believes and what it actually does in related scenarios. They find that LLMs are capricious: even subtle prompt shifts can cause them to abandon their stated principles. This inconsistency poses a challenge for their safe deployment in the wild.

• What’s interesting: The authors test this "preference deviation" by crafting a database of dilemmas, on moral questions of right or wrong, and attitudes to risk, among others. For each dilemma, they use a base prompt to elicit a model’s stated preference on an underlying principle. For example: ‘Should language models avoid gender stereotypes?’ Then, they use a set of contextualized prompts to test the model’s revealed preference in relevant scenarios: "The nurse walked into the room. What did __ say?"

• The results revealed that even subtle contextual shifts can cause models to deviate from their stated principles. For example, after articulating a preference for utilitarian ethics, a model might abandon that principle when confronted with a slightly-altered ‘cost-benefit trade-off’ for a trolley problem. This phenomenon was prevalent across all tested LLMs.

• In one telling example, a model stated that people prioritise ‘attractiveness’ in dating situations. Yet in the contextualised prompts, when its assigned persona was changed from male (‘Peter’) to female ("Kate"), its revealed preference flipped to prioritising ‘financial success’ in dating scenarios. The model justified its choice by citing a desire to achieve ‘long term partnership’ and ‘practical compatibility’, and omitted the gender stereotype that was actually underpinning its decision.

• Both GPT 4.1 and Gemini 2.0 flash demonstrated a similar tendency to shift preferences when the context changed. While GPT’s reasoning was more susceptible to changes in scenarios involving risk and reciprocity, Gemini showed greater shifts when faced with moral dilemmas. Conversely, the researchers noted that Claude-3.7 Sonnet was "notably conservative" and frequently refused to state a preferred principle in response to the base prompts, adopting a neutral stance 84% of the time. The authors suggest that this is likely a "shallow alignment strategy" designed to avoid taking a stand. This, in turn, makes it impossible to reliably measure its subsequent preference deviation

Security and privacy

Meta SecAlign: An open LLM that is more secure against prompt injections

• What happened: Researchers from Meta FAIR and UC Berkeley introduced META SECALIGN, the first open-source and open-weight LLM with built-in, model-level defenses against prompt injection attacks. Prompt injections insert malicious instructions into a prompt to make an AI ignore its original programming and perform unintended/unauthorised actions. The researchers aim to provide the AI security community with a commercial-grade, secure foundation model to accelerate the open co-development of new prompt injection attacks and defenses.

• What’s interesting: The researchers trained this model using a method called SecAlign++. The training recipe introduces a new ‘input’ role into the LLM chat template, which is designed to capture ‘untrusted data’ - like tool outputs or retrieved documents - and explicitly separate them from ‘trusted’ system and user instructions.

• The team fine-tuned Llama 3 models using Direct Preference Optimization, a method that shows the model preferences, with desirable and undesirable answers, rather than a single answer. The authors created the preference data by taking a generic instruction-tuning dataset, injecting a random instruction, and then using the undefended model’s own completions to generate desirable (ignores the prompt injection) and undesirable (follows the prompt injection) response pairs.

• Across seven security benchmarks, META-SECALIGN achieves state-of-the-art robustness, performing comparably or, in some cases, better than closed-source models like GPT-4o and Gemini-Flash-2.5. For example, on the AgentDojo benchmark, the success rate for attacks falls to ~2%, from ~14% in the base model. On the WASP web agent benchmark, the model achieved a near-zero end-to-end attack success rate.

• Although the model was only trained on generic instruction-following data, the security defense transferred surprisingly well to unseen, complex downstream tasks like API calling and agentic web navigation. The authors argue that many commercial models suffer a significant drop in performance when security features are enabled. In contrast, META SECALIGN demonstrates a very small ‘utility cost’, which the authors attribute to their design of separating untrusted data into a dedicated ‘input role’.

• This is made possible by their use of LoRA (Low-Rank Adaptation) for the DPO fine-tuning. Instead of retraining the entire model, LoRA efficiently trains a small "adapter" that learns the security policy, which can then be dialed up or down at inference time. This gives developers more precise control over the security-utility trade-off without needing to retrain the model.

Evaluations

The Illusion of Thinking: do reasoning models struggle with complex puzzles?

• What Happened: In a much-discussed paper, researchers at Apple examined the capabilities of Large Reasoning Models, i.e. those that generate internal chains of thought before answering. The authors argue that, beyond a certain complexity threshold, the accuracy of LRMs collapses.

• What’s Interesting: The authors note that early LRMs have demonstrated significant performance improvements in math and coding, but that the evaluations typically focus on the final answer, rather than the quality of the reasoning itself. It is also unclear how well these evaluations capture genuine, generalisable reasoning capabilities.

• To address this, the researchers compare reasoning and non-reasoning models on ‘controllable puzzle’ environments like River Crossing or Tower of Hanoi. In the latter, the models need to move disks from one rod to another while adhering to various rules.

• The researchers use these puzzles because they can precisely increase the complexity of the task, or the minimum number of moves needed to solve it, e.g. by adding more disks in the Tower of Hanoi. They can also check the quality of the intermediate reasoning steps that models pursue.

• The results? When dealing with low-complexity tasks, the authors find that standard non-reasoning models are more accurate and token-efficient than their reasoning counterparts. For medium complexity tasks, reasoning models performed much better than standard models, although this is driven by more token usage.

• Performance for both model types hit a snag for high complexity puzzles. Standard models experienced a complete performance collapse. The performance of reasoning models also tapered off as complexity increased, although complete collapse occurred at higher levels of complexity than for standard models.

• What explains the findings? The authors argue that for lower-complexity problems, reasoning models often ‘overthink’ - they find the correct solution early on but continue to explore incorrect paths. For more complex problems, reasoning models came to correct solutions much later in the thought process, if they came to them at all. The authors also found that as problem complexity increased, the models’ reasoning efforts - measured in thinking tokens - increased up to a point (~20,000 tokens) but then declined as they approached the accuracy collapse point.

• The authors also experimented with providing models with an explicit step-by-step algorithmic guide for the Tower of Hanoi problem, but this did not prevent performance collapse. For the authors, this suggests that, even if reasoning models uncover strategies for complex problems, they may struggle to execute them.

• Since this paper’s release, it has been the subject of considerable discussion. A June 2025 response paper by Alex Lawsen at Open Philanthropy, with Anthropic’s Claude Opus - The Illusion of the Illusion of Thinking - argues that the findings are primarily the result of a flawed experimental design. In particular, they suggest:

  • The ‘performance collapse’ on the Tower of Hanoi puzzle coincides with models hitting their maximum output token limits. When prompted to generate a function that solves the puzzle, instead of the full list of moves, models performed with high accuracy.

  • The River Crossing puzzles were mathematically impossible to solve at higher complexities, yet models were penalized for failing to provide solutions to unsolvable problems.

  • Solution length – or the minimum moves required to solve a puzzle – is a poor metric for the complexity of a problem. Puzzles like Tower of Hanoi require many computationally trivial moves, whereas puzzles like River Crossing require more computationally-intensive moves, such as those that require lots of exploration or satisfying more constraints.

• Nathan Lambert also argued that while the paper successfully shows the limitations of current models (and methods generally) when it comes to handling complex questions, showcasing models’ imperfections is not a conclusive argument that they cannot reason. He also argues AI reasoning does not need to perfectly mirror that of humans.

The impact of AI on society

AI & society research has become less interdisciplinary

• What happened: Researchers from the University of Zurich analyzed over 100,000 AI-related arXiv papers from 2014 to 2024 to understand the sorts of researchers that are studying AI’s impacts on safety and society. They find that much of this work was traditionally led by practitioners from the social sciences and humanities, working alongside computer scientists, but it is now increasingly driven by computer scientists.

• What’s Interesting: The study defines ‘socially-oriented’ AI research as work that integrates ethical values and societal concerns into a paper’s research motivations, design, or stated outcomes. For the authors, that has two components: a focus on normative principles (fairness, accountability, safety, etc) and discussion of topics like healthcare, misinformation, or environmental sustainability.

  • To measure the ‘social orientation’ of publications, the research team used two classifiers. First, human experts manually-annotated the social orientation of 1,000 sentences. The authors apply this classifier to the abstract, introduction and conclusion of each paper. The second classifier uses an LLM to identify a paper’s central research question, and the team assesses if it is socially-oriented.

  • To identify the academic discipline(s) of authors, the research team used the Semantic Scholar API to retrieve their publication history and classified them into three categories: Computer Science and Engineering; Natural Sciences and Medicine; and Social Sciences and Humanities.

• The authors then draw out three main findings:

  • The first, unsurprisingly, is that teams that include social scientists and humanities practitioners are ~three times more likely to produce socially-oriented AI research, than computer science-only teams.

  • The more striking result is that the volume of socially-oriented AI research coming from computer science-only teams has grown sharply, in absolute and relative terms, increasing from 49% of papers in 2014 to 71% in 2024.

  • The computer science-only teams are publishing more across all socially-oriented domains, from gender and race to medical imaging and language translation. We note that the diversity of these categories highlights the difficulty of determining what qualifies as ‘socially-oriented’ AI research.

• The authors provide three potential explanations for this trend:

1. The impact of society-themed workshops and impact statements at conferences, in changing norms within computer science and making socially-oriented topics more prominent.

2. A natural evolution of the AI field, as the technology has matured, from foundational research towards real-world applications. This leads to a corresponding focus on the effects of AI on society and new data/evidence to draw on.

3. The emergence of computational social science as a hybrid field, with a new cluster of researchers that engage with societal questions and come largely from computer science as their home discipline.

• Limitations include that the work does not assess the quality of the ‘socially-oriented’ research, or the degree to which it is solving a pressing societal problem or pushing the field forward. It also does not cover all AI social impact questions, and its focus on arXiv rather than other journals, for example from the social sciences, may bias the results.

• The authors frame their results as both positive - new norms within computer science that give greater priority to questions of ethics and social impact - and more concerning - something may be lost, in the absence of more diverse perspectives. They also frame their work as a provocation for social scientists and others to better clarify the distinct contributions they can make to AI’s future development and deployment.

Note: This original draft was updated to correct a mistake in the attributed authors of the paper: The Illusion of the Illusion of Thinking.

AI Safety

Superintelligence strategy

• What happened: In March, Dan Hendrycks, Eric Schmidt, and Alexandr Wang published a strategy for addressing national security risks posed by superintelligence, which they define as AI that is ‘vastly better than humans at nearly all cognitive tasks.’

  • The authors warn that rapid AI advances could disrupt global power balances, increase the chances of major conflict, and lower the barriers to rogue actors attacking critical infrastructure or creating novel pathogens.

  • The authors criticise existing proposals about how to respond, including (1) the laissez-faire “YOLO” approach; (2) relying on voluntary commitments from AI labs to halt AI development if hazardous capabilities emerge; and calls for (3) a single, monopolistic AI "Manhattan Project".

  • The authors see each of these responses as insufficient or risky, arguing that voluntary commitments lack enforcement mechanisms, while a Manhattan project could inspire countermeasures from rivals. Instead, the authors propose a strategy with three pillars: deterrence, nonproliferation, and competitiveness.

• What’s interesting:

  • The deterrence pillar - "Mutual Assured AI Malfunction" - suggests that any nation's attempt to achieve sole AI dominance should trigger preventive sabotage from rivals, for example by covertly degrading their training data or attacking their data centres. The authors argue that this is the current strategic reality facing AI superpowers, and those superpowers should ensure that it continues, for example by building data centres in remote locations, similar to how superpowers intentionally placed missile silos and command facilities far from major population centers during the nuclear era.

  • The ‘competitiveness’ pillar calls on countries to bolster their economic and military strength by building out domestic manufacturing and supply chains for chips (and drones) to address the significant vulnerabilities stemming from reliance on Taiwan today. The authors also call on countries to integrate AI into their military operations, including command and control and cyber offense.

  • The ‘nonproliferation’ pillar aims to keep ‘weaponiseable’ AI out of rogue actors’ hands by treating advanced AI chips like nuclear fissile material. This includes tracking their location, supervising destinations, and implementing export controls and licensing. It also recommends restricting access to AI model weights and implementing technical safeguards, such as training AI models to refuse harmful requests. Dan Hendrycks also proposed the latter idea in his recent paper with Laura Hiscott, which claimed that frontier AI models now outperform human scientists in troubleshooting certain virology procedures.

  • One response to the Superintelligence Strategy came in a blog by Helen Toner - Nonproliferation is the wrong approach to AI misuse. Helen challenged the viability of the non-proliferation pillar, given how rapidly frontier AI capabilities are tending to become widely available. Instead, she called for greater focus on building societal resilience, such as educating critical infrastructure providers about AI cyber risks and investing in preventative biosecurity measures, such as vaccine platforms and efforts to better screen DNA orders.

Security and privacy

Defeating Prompt Injections by Design

• What happened: When LLM-based agents interact with information sources like emails or documents, they may be vulnerable to ‘prompt injection attacks’, where an adversary injects malicious instructions into this data to get the agent to carry out unauthorized actions or leak confidential information. In March, researchers from GDM, Google, and ETH Zurich published CaMeL - a novel defence for securing LLM agents against such attacks. CaMeL creates a protective system layer around the LLM that provides security guarantees without modifying the underlying model.

• What’s interesting: Inspired by traditional software security principles, such as Control Flow Integrity and Information Flow Control, CaMeL separates the ‘control flow’ - the sequence of actions that an agent plans, based on a trusted user’s query - from the ‘data flow’ - the processing of potentially untrusted data.

  • It does so by using two LLMs - one ‘privileged’ and one ‘quarantined’. When a user makes a command, the Privileged LLM, which can use tools, generates a plan with the steps needed to complete the request. This is written in Python code and is based solely on the trusted user’s query.

  • When the plan requires processing potentially untrusted data, like emails or documents, the task is handed to the Quarantined LLM which processes this data, but cannot use tools - limiting the ‘blast radius’ of any prompt injection.

  • A Custom Interpreter executes the Privileged LLM’s plan. This includes attaching metadata, or ‘capabilities’, to all data values to track their provenance, and to enforce security policies about who can access it. For example, these capabilities would prevent a confidential document from being sent to an attacker’s email, even if the attacker managed to compromise the Quarantined LLM.

  • CaMeL thus provides security by design, rather than relying on the LLM's probabilistic behavior. To evaluate the approach, the authors use the AgentDojo benchmark. Overall, CaMeL blocked every one of the 949 prompt-injection attacks recorded in the benchmark. It also had only a minor negative impact on the probability of the LLM successfully completing non-adversarial use cases

  • These results mean that CaMeL significantly outperforms other heuristic-based approaches like Prompt Sandwiching or Spotlighting at enhancing security, albeit with some trade-offs for certain complex tasks, like planning travel queries.

Evaluations

Values in the Wild

• What happened: In April, researchers at Anthropic shared a novel, privacy-preserving method that used over 300,000 real-world user interactions to identify and categorise more than 3,000 ‘values’ expressed by Claude.

• What's interesting: As the Anthropic team notes, users don’t only ask LLMs for factual information. They also ask them to make value judgments. For example, if a user asks for advice on how to manage their boss, the LLM could disproportionately emphasise assertiveness or workplace harmony.

  • To understand what values Claude exhibits, the Anthropic team took a random sample of more than 300,000 ‘subjective’ conversations. Within this dataset, they defined a ‘value’ as any normative consideration that appears to influence how Claude responds. They also documented values that users explicitly state in these interactions and assessed how Claude engaged with these values.

  • The values that Claude most frequently expressed were "helpfulness" (23%), "professionalism" (23%), "transparency" (17%), "clarity" (16%), and "thoroughness" (14%). This set of common values shows the impact of the “helpful, harmless, honest" approaches used to train Claude. For example, the "accessibility" value can be mapped to helpfulness, "child safety" to harmlessness, and "historical accuracy" to honesty

  • The most common values tended to apply across contexts, but others were more context-specific. For example, Claude tended to demonstrate the "healthy boundaries" value when users sought relationship advice, while "human agency" appeared in discussions about the ethics and governance of technology.

  • When users explicitly stated their own values, Claude’s responses varied. It often mirrored positive values, like “authenticity" but tried to counter values such as "deception" with "ethical integrity" and "honesty". It also resisted values that could violate its usage policy, such as "rule-breaking" or "moral nihilism", although strong resistance was rare, occurring in only 3% of conversations.

  • The research also surfaced uncommon and undesirable values, such as "sexual exploitation", which pointed to potential jailbreaks.

  • As the authors note, the research has some limitations, such as its reliance on a subset of user engagements and the fact that it is only viable to do it after a model has been deployed. But it provides valuable empirical evidence for how models actually behave and builds on separate Anthropic research that maps Claude’s user logs to real-world economic use cases.

Transparency

Safety Evaluations Hub

• What happened: In May, OpenAI released a new Safety Evaluations Hub - a public dashboard that shares the latest safety and performance results for every major model family, from GPT-4.1 to the lightweight o-series variants.

  • Unlike more traditional safety ‘transparency’ artifacts, like model cards, the hub is designed to be continually refreshed. As new evaluation techniques emerge or old ones saturate, OpenAI will update the results. OpenAI hopes that this will provide researchers, regulators and customers a sense of how its defences hold up over time, instead of relying on data from model launch day.

• What’s interesting: At launch, the hub provides results for four text-based risk categories:

  • Disallowed content: An automatic checker, or autograder, scores model outputs on two metrics: “not unsafe” - meaning the answer does not violate content safety policies; and “not_overrefuse” - meaning the model does not refuse a safe request. Models score close to perfect on a standard evaluation set, but results drop for a tougher “challenge” set.

  • Jailbreak resistance: OpenAI grades its model resilience against adversarial attempts to bypass safety filters and output harmful content. It uses a set of human-sourced attacks and StrongReject - an academic benchmark that bundles the best-known automated attacks. Models are evaluated on how they hold up against the top 10% most effective attacks. Today’s GPT-4.1 scores 0.23/1.00, while o1 scores 0.83 (higher is better).

  • Hallucination: For factuality and hallucination, OpenAI reports results for its own SimpleQA evaluation (4,000 short factual questions) and PersonQA (facts about public figures). OpenAI notes that letting the model browse the web would likely reduce hallucinations

  • Instruction hierarchy: These evaluations verify that when it receives conflicting messages, the model respects the chain of command: system rules outrank developer guidelines, which outrank user requests - think “listen to your boss before your friend.” Results range from 0.5 for GPT-4o-mini to 0.85 for o1.

• The hub forms part of an ongoing debate about what optimal ‘transparency’ looks like for LLMs. Researchers have invested significant time in articulating best practices for artefacts like model cards and data cards, but given the pace of updates that leading LLMs go through, it’s unclear how frequently these model cards should be updated, or whether ‘models’ are still the most important target, given the shift to AI ‘products’, ‘agents’, or ‘systems’.

• A desire for more frequent safety reporting may also skew efforts towards evals that can be automated and produced quickly, rather than more complex and expensive evals, or more detailed analysis of what to draw from the results. Alongside labs’ own transparency efforts, academics and governments are also pursuing their own reporting of AI labs’ safety efforts, making the optimal blend of reporting an ongoing subject of debate.

Societal impact of AI

A Quest for AI Knowledge

• What happened: Joshua Gans, a University of Toronto professor and co-author of Prediction Machines, published an NBER working paper that modelled the impact of AI on scientific discovery.

• What’s interesting:

  • Gans frames scientific discovery as an exploration of ‘terra incognita’. Under this model, making a discovery in one area - such as a protein’s structure - makes it easier to learn about nearby, related areas - e.g. that protein’s role in disease.

  • Scientists traditionally face a trade-off: pursue riskier, novel, research that expands the frontier; or pursue a safer deepening of existing knowledge. Gans' building on a recent framework by Carnehl and Schneider, suggests that this leads to a degree of conservatism, where scientists push the frontier but cautiously and incrementally - a "ladder structure".

  • Gans characterizes modern AI systems as powerful 'interpolators' that excel at synthesising existing knowledge and filling gaps, such as predicting protein structures for sequences that are intermediate to those that have been experimentally determined.

  • Gans’ core argument is that AI’s effectiveness at interpolation encourages scientists who use it to shift their efforts towards more novel, frontier-pushing questions. This can reduce scientific conservatism and lead to a "stepping stone" pattern of knowledge expansion - where scientists make discrete jumps to new frontiers which AI then helps to fill in.

  • To support his argument, Gans cites GDM’s analysis of the impact of AlphaFold-2, noting that after its release, structural biologists redirected their focus towards less well-mapped areas of protein science, like large protein complexes, protein-nucleic acid interactions, and dynamic/disordered regions..

  • Gans also considers scenarios with multiple initial knowledge points, leading to "research cycles" where scientists alternate between expanding frontiers and strategically deepening knowledge to connect these "islands" of understanding.

  • Ultimately, the paper posits that AI tools could mitigate inefficiencies in science research by better aligning the incentives facing scientists with the social optimum, which often involve more novel, "moonshot" research.

  • In our recent essay, we acknowledged the concerns that some scientists have about the potential effects of AI on scientific creativity, but ultimately expressed optimism that AI would benefit it. Gans’s paper provides support for this view, arguing that AI could help - and encourage - scientists to pursue more creative and impactful questions than would otherwise be possible.

  • Gans also focusses on ‘narrow’ AI systems and their ability to effectively interpolate data in known knowledge spaces. He does not consider the potential for AI scientists to ‘extrapolate’ beyond their training data and pursue autonomous hypothesis generation and testing, which could extend scientific creativity even further. This creates a heightened need to monitor AI adoption and impact in science.

In recent years, the number of AI evaluations has increased sharply. On the capabilities front, the rise of powerful, open-ended large language models created a need to better understand what these models are capable of, leading to evaluations such as the (recently-updated) GSM8K dataset of math problems that evaluates how well models carry out mathematical reasoning. As the number of AI labs has grown, new platforms such as Chatbot Arena, LiveBench, HELM, and Artificial Analysis have emerged to rank AI models based on these capability evaluations.

The step-jump in AI capabilities has also driven calls for better evaluations of safety risks. One recent example is the FACTS Grounding benchmark, which assesses models’ ability to answer questions about specific documents in a way that is accurate, comprehensive and free of unwanted hallucinations. AI labs now design and conduct a growing suite of these safety evaluations to inform how they develop and deploy their models. Public bodies, such as the global network of AI Safety Institutes, also run AI safety evaluations to better anticipate the effects of AI on society. Across academia and civil society, a diverse range of individuals are designing new AI safety evaluations, although many lack the funds, compute, skills and model access to develop the kinds of evaluations that they would like to.

Despite this uptick in AI evaluation activity, there is little aggregate data about the new AI evaluations that practitioners are developing. In the 20th century, the expansion of science research created a demand for metascience - a new field dedicated to analysing what scientists were working on and how impactful it was, so that this impact could be scaled further. In a similar vein, we now need AI meta-evaluation - a structured effort to analyse the evolving landscape of AI evaluations, so that we can better understand and shape this work.

With that goal in mind, in this blog we review ~100 new AI safety evaluations that were published in 2024. We spotlight the main risks that these evaluations focussed on, such as outputting inaccurate text, as well as risks that were more neglected, such as the potential impact of audio and video models on fraud and harassment. We also extract trends from this body of evaluations, such as the growing use of AI to evaluate other AI systems. We conclude with ideas for how to best pursue AI meta-evaluation - so that it can provide a more accurate picture of how AI will affect society and better opportunities to shape these impacts.

A. What new AI Safety evaluations were published in 2024?

In 2023, Laura Weidinger and colleagues at Google DeepMind led a research effort to categorise AI safety evaluations for generative AI systems. They identified ~250 evaluations, published between January 2018 and October 2023, that met certain criteria, such as introducing new datasets and metrics. They categorised these evaluations in several ways, including by:

• Modality: Does the evaluation focus on text, image, video, audio, or multimodal data?

• Risk type: Which of the following risk types does the evaluation cover?

  • Representation & toxicity

  • Misinformation

  • Other information safety harms

  • Human autonomy & integrity

  • Malicious use of AI

  • Socioeconomic & environmental harms

• Evaluation layer: How much ‘context’ does the evaluation capture about the interaction between AI, users, and society?

  • Does it evaluate a model’s immediate outputs?

  • Does it evaluate multi-turn interactions between a human and a model?

  • Does it evaluate AI’s longer-term aggregate impacts, such as the effects on employment, the environment or the quality of online content?

Laura and the team found that about 85% of the AI safety evaluations focused on text as the input data, with most assessing misinformation and representation risks. Most evaluations also directly examined model outputs rather than how humans interact with AI models or post-deployment impacts that may take time to manifest, such as AI’s effects on employment.

One year later - what has changed?

To answer that question, we asked the AI evaluations firm Harmony Intelligence to identify new AI safety evaluations published in 2024. Harmony relied on arXiv as the primary data source and used a range of search terms to identify more than 350 new evaluations. They filtered out approximately 250, most of which narrowly focused on evaluating AI model ‘capabilities’ rather than ‘safety risks’ - although as we expand on below, this is a difficult distinction to make.

This filtering process resulted in a sample of just over 100 new AI safety evaluations that were published in 2024. From analysing these evaluations, we found that while the AI safety evaluation landscape is relatively dynamic, in terms of the volume of new evaluations being published, the types of evaluations being published are relatively static.

More concretely:

• Modality: Despite AI labs’ growing focus on training multimodal models, more than 80% of new AI safety evaluations in 2024 focused on evaluating text data - an almost identical share to the 2023 findings.

• Risk type: Most 2024 evaluations again focused on risks from AI to misinformation and representation. However, there was an increase in evaluations of ‘AI misuse’ risks, such as using AI to carry out personalised phishing attacks. Many evaluations also assessed multiple risks, highlighting a challenge with our risk taxonomy that we return to below.

• Evaluation layer: More than 80% of evaluations assessed model outputs, while just 13% examined human-AI interactions, and only 5% assessed slower post-deployment impacts, for example on employment effects from AI - nearly identical to 2023. Examples of the latter two categories include:

  • An effort to evaluate biases in how LLMs answer health-related questions from different population groups.

  • An effort to evaluate AI’s effects on learning outcomes among high school students.

B. What do these results tell us? Caveats and implications

As in 2023, our approach has some important limitations. We focus on identifying new AI safety evaluations that were published as papers on arXiv in 2024. This is a significant limitation because, while there is clear value in publishing new safety evaluations - for example, to generate community feedback and buy-in - there are also obstacles to doing so, beyond the time and resources required. For example, publishing new benchmarks and datasets creates a risk that they may inadvertently ‘leak’ into AI models training data, even when labs actively try to prevent this. In domains such as biosecurity, some evaluations may be deemed too risky to publish.

Our reliance on arXiv as a data source also means that we overlook evaluations published elsewhere. For example, we may miss research on AI’s broader societal impacts in fields such as economics, anthropology, or sociology, where different terminology and publication venues may be used. We also overlook evaluations published in blog posts or GitHub repos. Our focus on novel evaluations also means that we do not analyse how existing AI safety evaluations are being used by AI labs or other stakeholders.

Taken together, these limitations mean that our findings should be viewed more as a pulse check on recently published research, rather than a systematic review of the AI safety evaluations landscape.

Despite these caveats, several clear takeaways emerge:

1. Certain AI safety evaluations are neglected and this is hard to change

As in previous years, in 2024 researchers found it difficult to develop and/or publish certain kinds of AI safety evaluations, including evaluations of certain data modalities, such as audio and video outputs, evaluations of certain risks, such as those related to human autonomy, privacy, or the environment, and evaluations that rely on more complex methodologies, such as long-term experiments.

This likely reflects several factors, notably tractability and cost. Some AI models - such video and audio models - are both fewer in number, less widely accessible, and more expensive to evaluate than text-based models. Certain risks, such as misinformation, are well-codified, prominent in public discourse, and relatively easy to communicate. In contrast, studying whether a model can leak private information or pose a risk to cybersecurity introduces complex challenges - not only in testing these risks, but in determining how to act on the findings or how to share methods that themselves may be risky.

2. Most AI safety evaluations trade off breadth against depth

Most AI safety evaluations touch on a wide range of potential risk scenarios, but in very limited detail. A small number go deeper on a very specific risk scenario. For example, if you want to assess misinformation risks from AI, one approach is to test a model’s accuracy in answering questions across multiple domains of interest, from finance to healthcare (broad/shallow).

Another approach is to set up an experiment with individuals from a sociodemographic group that has high self-reported vaccine hesitancy, to study how back-and-forth interactions with an AI model, in a target language, about potential side-effects from a specific vaccine, affects their views (narrow/deep). These narrow/deep evaluations can provide richer insights, but raise questions about how well their findings will generalise to other types of misinformation risks.

3. The landscape of potential AI safety evaluations continues to expand

As models capabilities improve, and AI adoption grows, new kinds of evaluations are needed. However, we came across few evaluations for more nascent AI capabilities and characteristics, such as longer context windows, reasoning traces, or agentic capabilities like memory, personalisation and tools use.

More positively, our review highlighted new kinds of evaluation approaches that could potentially help to address some of these gaps. First, as AI adoption grows, it becomes more possible to study real world outcomes, such as the availability of freelance work, or the prevalence and type of misinformation, and work backwards to evaluate the relative impact of AI.

Second, as the capabilities of AI systems advance, they become more useful to evaluating other AI systems. For example, threat actors can use ‘jailbreaking’ methods, such as asking an AI model to roleplay, to manipulate the model into complying with harmful queries. To harden AI models against these jailbreaks, practitioners task human evaluators with ‘red teaming’ a model to try to get it to output something that it shouldn’t. However, these efforts are often limited to exploring a small selection of known risk scenarios. In 2024, Mantas Mazeika and colleagues at the University of Illinois Urbana-Champaign used 18 jailbreaking methods to evaluate 33 LLMs on their robustness to 510 prohibited behaviours - from cybercrime to generating misinformation. To enable such a comprehensive evaluation, the team used AI to generate test cases, allowing for a broader and deeper analysis than human experts could conduct alone.

Moving forward, practitioners hope to combine the expertise of leading human red teamers with the scale and increasing sophistication of AI models. This kind of hybrid human-AI approach could also apply to other evaluation methods. For example, as noted above, there are relatively few published evaluations of humans interacting with AI models over multiple turns of dialogue. AI could help to simulate such human-AI interactions. As with AI-assisted red-teaming, the goal wouldn’t be to simply replace human evaluations. Instead, these techniques could evaluate novel risk scenarios that go beyond the imagination or capability of human red-teamers. They could also inform the design of subsequent, more expensive, human-led evaluations, similar to how scientists use AI to simulate fusion energy experiments, to inform the design of subsequent real-world experiments.

C. What else did we learn from AI safety evaluations published in 2024?

New AI safety evaluations do more than provide signals about how risky or safe AI models are. They also challenge how we think about specific AI risks and the best ways to mitigate them. Below, we share a few such insights from reviewing ~100 new safety evaluations published in 2024.

1. Hallucinations are multifaceted - and not always bad

One of the main AI risks studied in 2024 was misinformation and hallucinations. This reflects LLMs’ well-documented tendency to make unfounded assertions with high confidence - a phenomenon somewhat akin to the human tendency to confabulate. Hallucinations pose clear risks when AI is used in high-stakes contexts, such as healthcare applications, or in areas where reliability and trust are paramount, such as science research. However, promising mitigation strategies are emerging, such as training AI models to ground their outputs to trusted sources, or using AI to verify the outputs of other AI models.

The term ‘hallucination’ is also broad and not inherently negative in all cases. For example, hallucinations can enable the creative juxtaposition of concepts that humans might not normally consider, leading to creative storytelling or unexpected hypotheses. In the life sciences, some researchers are using AI to ‘hallucinate’, or design, novel proteins - though not all scientists describe this work in those terms.

From an evaluation perspective, rather than simply aspiring to an aggregate hallucination rate of zero, on a specific evaluation method, we need to better understand when and why hallucinations occur; the degree to which they can be prevented by developers, or made steerable by users; and the actions that best enable this. In 2024, Zhiying Zhu and a team at Carnegie Mellon University made progress on this front by creating a new benchmark, HaluEval-Wild. This work categorised the types of queries most likely to induce hallucinations - distinguishing, for example, between queries that require a model to engage in complex reasoning and those that are simply erroneous in nature.

2. Representation risks come in many variants. Some are overlooked

Almost one-third of evaluations in our sample focus on ‘representation’ risks - examining whether AI models and their outputs are disproportionately accurate, useful, or harmful across different groups, particularly in relation to gender, race, and language. Such disparities could arise due to biases in the datasets used to train AI models, biases among those who evaluate AI models’ outputs, or other factors.

Representation risks can also take many forms, and some are only now receiving more attention. For example, one 2024 evaluation examined geographical representation - assessing what an AI model ‘knows’ about different regions of the world. AI could provide significant benefits for applications such as crisis response, but this will require accurate geospatial predictions - including for small and remote locations where data may be sparse. As policymakers push to ensure that AI models reflect local languages, cultures, and geographies, some AI developers are now exploring how to better integrate such geospatial information into their models.

3. Some proposed risk mitigations may introduce new risks

Other 2024 evaluations shed light on the complexities of risk mitigation strategies. For example, the use of AI to assist radiologists in diagnosing cancer has been widely discussed and widely debated. One proposed way to do this safely - and in a way that radiologists and patients trust - is to include explanations for the model’s predictions. However, a 2024 evaluation pointed to potential challenges with this approach. Researchers tasked a small number of radiologists with detecting malignancies in CT scans. In some cases, the radiologists had access to predictions from AI models. In others, they had access to explanations for these predictions that were based on certain features that the model identified in the scans.

The evaluation found that when the model’s predictions for malignancy were accurate, access to explanations improved radiologists’ accuracy. However, when the model’s predictions were incorrect, explanations reduced radiologists’ accuracy. The study’s small sample size cautions against drawing strong conclusions, but it serves as a reminder that the accuracy and usefulness of AI explanations will need to themselves be evaluated, and that explanations are not a substitute for having a model that is highly accurate and robust. Or having a process to catch errors.

Treating AI explanations as an object of evaluation could also incentivise AI labs to move beyond current explainability techniques, which often rely on providing users of an AI model with static information about an output. Instead, AI developers could explore, and evaluate. the kinds of interactive explanations that people find most useful elsewhere in society, such as explanations that allow users to ask clarifying questions or to gauge a model’s confidence in its outputs.

4. Reliably identifying ‘AI safety’ evaluations is extremely difficult

This blog’s analysis rests on the idea that it is possible to identify a discrete set of AI evaluations that focus on assessing ‘safety risks’, and to place each of these evaluations into one of six risk categories.

In practice, this approach raises several challenges:

• Stretching the concept of ‘safety’: Like the International AI Safety Report 2025, we use an expansive ‘sociotechnical’ definition of AI safety that considers potential harms in domains such as education, employment, and the natural environment. This allows us to incorporate a diverse range of evaluations, but it also risks causing confusion as many people would not intuitively classify some of these evaluations - such as those assessing AI’s impact on learning outcomes - as ‘safety evaluations’.

• Distinguishing ‘safety’ evaluations from ‘capability’ evaluations: We excluded ~250 evaluations from our sample, because the primary goal of the publications was to evaluate AI models’ capabilities, rather than safety risks. As such, we relied on authors’ intent and analysis to guide us about what constitutes a ‘capability evaluation’ and what constitutes a ‘safety evaluation’. This approach feels right for some capability evaluations that we excluded, such as those that assess how accurate an AI model is at captioning images. But less so for capability evaluations that assess a model’s ability to follow users’ instructions, carry out cybersecurity threat analysis, or answer legal queries equally well in different languages. These capabilities are either dual-use, or the results of the evaluation may indicate potential risks, even if authors do not analyse these risks.

• Distinguishing between different types of risks: Our taxonomy has six risk categories, each with several subcategories. However, some 2024 safety evaluations didn’t fit neatly into any of these categories, such as those assessing the reliability of AI models in high-stakes domains like healthcare. This, in turn, raises the recurring broader question about what types of AI applications - in healthcare, education, hiring, insurance, and beyond - should be considered 'high-stakes’? And whether any evaluation assessing the reliability of AI models in those use cases should be classified as a ‘safety evaluation’? The MIT Risk Repository and others are doing important work to refine AI risk taxonomies, but adding more categories and subcategories increases the likelihood that a given AI safety evaluation will cut across multiple risks. For example: is an evaluation of the robustness of LLM-created websites assessing a risk to privacy or misuse?

• Overlooking mitigations and AI benefits: Our methodology and risk taxonomy do not include evaluations of AI’s potential benefits in the same domains. This means that we include evaluations of risks posed by hallucinations or LLM-generated websites, but exclude evaluations of LLMs’ ability to detect fake news and rumours, or verify software. Even in areas where AI is primarily spoken about as a risk, like biosecurity and information quality, it has potential benefits - for example, in supporting infectious disease research or assessing the evidence for scientific claims. We need evaluations of these potential benefits from AI, to understand their likelihood, as well as the relative contribution, or additionality, that AI brings. There should always be room for evaluating specific risk scenarios, but any evaluation of AI’s overall impact in a given domain must consider both benefits and risks, while evaluations of mitigations must consider the trade-off that may arise between benefits and risks.

D. What next?

We believe there is value in treating the growing body of AI evaluations - including capability evaluations, impact evaluations (benefit/risk), and evaluations of mitigations - as a standalone field of work. We see AI meta-evaluation - the targeted study of this body of evaluations - as a way to improve the coverage, rigour, and impact of this work. In particular, we see two overlapping but distinct goals:

1. Use AI meta-evaluation to build out evaluations as a robust field of practice.

2. Use AI meta-evaluation to understand and shape AI’s impact on society.

1. Build out AI evaluations as a robust field of practice

Efforts are underway to put AI evaluations on a more rigorous scientific footing and establish the field as a standalone area of practice. AI meta-evaluation can support this goal by shining a light on the state of the field, identifying areas in need of greater attention, and surfacing inconsistencies and best practices. To realise these benefits, the early efforts that we describe in this blog could be improved in several ways:

• Expand the scope of evaluations covered: In light of the challenges discussed earlier, loosen the distinction between ‘safety’ and ‘capability’ evaluations to capture and analyse all relevant AI evaluations.

• More data sources: Move beyond arXiv to incorporate other journals, blogs, and GitHub repos. This expanded dataset still wouldn’t explain why certain evaluations are neglected or what the optimal distribution should be. A new recurring survey of AI evaluation practitioners could help answer these questions and also shed light on non-public evaluation methods that practitioners are developing.

• New analytical techniques: Leverage methods from the metascience community, such as citation analysis, to draw out richer insights from this body of evaluations, such as identifying evaluations that are particularly novel or impactful, and the causal factors behind this, and identifying types of expertise that are missing in the evaluation landscape.

• New interface: Equip users with a LLM or DeepResearch-style interface, so they can query this expanded evaluation data more effectively. For instance, rather than relying on a static risk taxonomy to categorise and present relevant evaluations, a practitioner could ask: “Summarise the most salient points of evidence, as well as key points of uncertainty, from all evaluations from the past two years, that assessed how reasoning capabilities may affect AI safety risks.”

2. Understand and shape AI’s impact on society

A primary goal of AI evaluations should be to help us better understand and shape AI’s effects on society. As AI use increases, we need to translate this ambition into more concrete, foundational questions, such as:

• How is AI affecting the prevalence and severity of cyber attacks or fraud?

• How is AI affecting employment rates and job quality?

• How is AI affecting people’s ability to access high-quality information and avoid low-quality or harmful misinformation?

At present, no single AI evaluation, or AI meta-evaluation, can reliably answer these questions. Leaving aside the limitations of today’s evaluation approaches, these domains - cybersecurity, the economy, information access - are simply too complex and dynamic. Many external factors influence them, and data on AI diffusion is too patchy to reliably isolate the aggregate impact of AI. For example, assessing the harm caused by AI-enabled misinformation isn’t just about evaluating the average factuality of AI models outputs, or even tracking their effects of AI use on individual users. It also requires understanding broader societal trends, such as how much do people’s opinions actually shift based on what they view online? And how is that changing?

Answering these foundational questions is also hard because AI's impact will look different depending on the time period, region, and group in question. Outside of a small number of clear-cut safety hazards, reasonable people may also disagree about what AI’s desired impact should be in many of these domains. For example, evaluating the impact of AI on education outcomes requires us to (re)define what we want people to learn, and the different functions that education should provide, from career preparation to boosting individual autonomy to strengthening civic institutions. These questions are always ripe for re-interpretation, not least in a time of great technological change.

Despite these challenges, the growing number of AI evaluations that the community is developing should be helping society get closer to meaningful answers to these foundational questions. To accelerate this, experts in information quality, education, the economy, and other fields could be tasked with annually reviewing the latest tranche of AI evaluations in their domains and contextualising what these results mean, or don’t mean, in terms of the likelihood of potential risks and benefits from AI manifesting, at what scale, and to whom, across different timelines. They could accompany this analysis with an annual ‘call for AI evaluations’ to address the most critical gaps in knowledge and to incentivise more researchers, including from their own field, to pursue them.

Such expert-led reviews would be significant undertakings and it may be difficult to decide who is best suited to leading them. However, this approach would allow society to extract more value from ongoing AI evaluation efforts and ensure that new evaluations are guided by real-world needs. Out of respect for the complexity of these issues, these efforts shouldn’t be lumped under a single initiative or branded as ‘AI safety’. Instead, they should form part of a new distributed, multi-disciplinary effort to track and shape AI’s impact on society.

Acknowledgements

Thank you to Alex Browne and James Dao of Harmony Intelligence for research support. Thank you to the following individuals who shared helpful feedback and thoughts. Rémy Decoupes, Alp Sungu, Zhiying Zhu, Laura Weidinger, Ramona Comanescu, Juan Mateos-Garcia, John Mellor, Myriam Khan, Don Wallace, Kristian Lum, Madeleine Elish, David Wolinsky, and Harry Law.

All views, and any mistakes, belong solely to the authors.