Every month we try to stave off the decline in reading by taking a look at new AI papers that we’ve seen folks discussing. Today, we look at how Russian propaganda is targeting LLMs; whether AI models can accurately predict human behavior; whether they can get a paper published at the NeurIPS conference; and the growing risk of hackers implanting malicious instructions into data that AI agents use.
Please share your own take and any new papers that you’ve enjoyed.
—Conor Griffin, AI Policy Perspectives
Information warfare is targeting LLMs
What’s the paper about? Researchers from the UK think tank Demos dived into the case of LLMs endorsing falsehoods from a Russian propaganda source to highlight how manipulative influence campaigns are now targeting AI models.
Why does it matter? Discussions about AI and disinformation often focus on threat actors using models to directly generate and flood misleading claims across social media. But as more people turn to AI for information and advice, threat actors also want to shape how LLMs respond.
The details: To improve their answers, AI systems use retrieval-augmented generation, or RAG, to identify relevant documents and information by calling up search engines and databases. This introduces a risk of “RAG poisoning,” where malign actors insert distorted information into the models’ sources.
The Demos authors illustrate this by evaluating how well a Russian foreign-interference operation—the Foundation to Battle Injustice, or r-FBI, which falsely presents itself as a human-rights organization—has infiltrated unsubstantiated allegations about its opponents into LLMs.
The authors extracted 50 specific claims from r-FBI articles that had no corroborating secondary source, such as an assertion that the Ukrainian president, Volodymyr Zelensky, had installed cryptocurrency farms that were causing blackouts. They fed 600 prompts about these claims to five AI models: GPT-4.1 Mini; Gemini 2.5 Flash; Claude Haiku 4.5; Mistral Small 3.2; and Grok 4.20.
Around 17% of model responses endorsed the r-FBI’s claims, repeated them uncritically, or presented them as a legitimate view. An additional 31% of responses were neutral—addressing the topic, but neither validating nor rejecting the specific claim. The remaining 52% of responses rejected the claims, with a subset doing a “comprehensive debunk” that exposed r-FBI as a foreign-interference operation.
Grok performed best, comprehensively debunking 56% of queries, compared with 17% for Gemini, 10% for Claude, 5% for GPT, and just 1% for Mistral. The authors suggest that Grok’s stronger performance may reflect its RAG setup and access to data from X. Or it may reflect Grok 4.20’s larger size—the other LLMs tested were small models of the type that underpin quick chatbot and AI-search responses.
The Demos authors found that r-FBI ran a sophisticated campaign to get LLMs to find, trust, and cite their content. This included using protocols to push r-FBI’s “breaking news” articles into the live-search indices that RAG pipelines use, as well as using long page titles and descriptions. This use of lengthy text hurts the visibility of r-FBI’s articles in traditional search engines, but allows AI crawlers to ingest complete passages, suggesting that r-FBI was optimizing for LLMs, not humans.
In its content, r-FBI also makes editorial choices that the Demos team see as targeting LLMs. This includes packing lots of statistics and quotations into self-contained paragraphs of a length that retrieval systems can readily extract, as well as prioritizing absolute claims over nuance. They also distribute their content through what purport to be different kinds of institutions and formats, including news articles, human rights reports, and foreign policy journal articles, and have each publication repeat the core claims and cite each other, in an effort to increase their salience in AI model embeddings and suggest support from multiple sources.
Making content more visible to LLMs—referred to as generative engine optimization, or GEO—can be benign; a range of legitimate GEO providers exist to help organizations ensure that their offerings are visible and correct. However, of 50 GEO providers identified by the Demos team, only one—Finn Partners—had a published policy to prevent misuse of its services.
By drawing on the US Foreign Agents Registration Act, which requires anybody acting on behalf of an overseas government to disclose this, the researchers found two examples of government entities hiring GEO providers to help shape AI outputs, although the authors were careful not to equate this with information warfare.
Of the five AI companies whose models Demos evaluated, only Google appeared to have a published policy that explicitly mentioned GEO practices, where they warn against the creation of large numbers of pages and caution that “GEO hacks” to make content more digestible to AI crawlers are unnecessary and ineffective.
The challenge for AI companies is to get their models to reason about the quality and reliability of a source. For queries where the evidence is genuinely unclear, models should be able to convey this. When the query is about disinformation, LLMs should be able to recognize this and strongly debunk it.
Demos’ recommendations include making counter-disinformation easier for AI systems to retrieve—a form of ethical GEO; requiring AI companies to explicitly prohibit harmful GEO and clearly label and deprioritize content from known manipulative sources; fostering better intelligence-sharing among AI companies about RAG-poisoning campaigns; and creating industry standards for GEO providers that distinguish marketing from deception.
AI can (somewhat) predict how humans think and behave
What’s the paper about? A team of Harvard and Stanford researchers found that LLMs could forecast the results of social science experiments on human attitudes and behavior with high accuracy, although the models overestimated the size of the effects.
Why does it matter? Rather than replacing human participants, AI simulations could serve as cheap forecasters, expanding the number of ideas that researchers and policymakers can consider before helping them filter down to those most worthy of expensive real-world testing.
The details: The authors built a database of 70 large US social science experiments that collectively measured 120,000 people’s attitudes to topics such as immigration and criminal justice reform, as well as the effects of text-based interventions, like reading articles.
They used GPT-4 to simulate participants across variables such as gender, ethnicity, and political persuasion, and used this to predict the average effect of each study’s interventions.
The simulations accurately predicted the direction and relative size of the interventions’ effects, but overestimated their absolute size. This means that they may be more useful for ranking the plausibility of hypotheses than for estimating how big specific effects will be.
The results held up on a subset of studies published after GPT-4’s training data cutoff, suggesting that the findings were not due to simple memorization of training data—although unpublished experiments may resemble older studies, so it’s hard to judge whether LLMs are modeling human psychology or just learning regularities about social science findings.
Bias is another concern about such LLM simulations. The authors found that the simulations were slightly less accurate for Black participants, but did not find major differences across ethnicity or gender. However, the original experiments on people had focused exclusively on the US, and also did not identify large differences across demographics. As such, questions remain about the validity of LLM simulations in other locations, or for topics where participant identity is more influential.
An accompanying survey suggests that social scientists are enthusiastic about LLM simulations, especially the ability to cheaply pilot more experiments. The authors note that the error rate of a 385-human pilot study costing around $1,200 is comparable to a combination of LLM simulations and a 100-human pilot study that jointly costs around $300.
Why do we still need human experiments? LLM simulations did not provide precise estimates of effect size and their accuracy also dropped on a secondary database of “megastudies.” Some of these focused on populations with deeply entrenched beliefs—for example, about climate change—where effect sizes are normally small. Others went beyond measuring stated attitudes and tracked behavior change, such as vaccination uptake, where frictions such as waiting times in local pharmacies make a difference but are harder to simulate. Real-world studies can also help explain why interventions succeed or fail, rather than just predicting the outcome.
One question is whether LLM simulations might degrade as AI companies carry out more post-training to make their models safer and more helpful, potentially rendering them less descriptive of what humans actually think and do. This study does not find evidence of that, with GPT-4 outperforming earlier variants. They also find that recent, smaller, open-weight models, including DeepSeek-V3 and Gemma 3, perform similarly to the larger, older, GPT-4—open-weight models may also be attractive to researchers who want to run their experiments locally and share their code to allow for replication.
Another question requiring future study is whether the growing use of LLM simulations will actually widen the range of ideas that researchers pursue, or narrow it—for example if researchers converge on similar ideas or if models struggle to simulate more novel ideas.
Malign actors could also use LLM simulations to shift public opinion or behavior. For instance, the authors note that the GPT-4 simulations accurately predicted which content—including misleading messages—was most likely to reduce a population’s intention to get vaccinated. They argue that AI companies have safety measures to prevent users from directly generating such misleading content, but lack restrictions on running simulations of experiments that could reveal effectively the same thing. In theory, this capability should be available to well-intentioned researchers, but not to others—a difficult goal, especially on politically fraught topics.
AI agents can’t do AI research as well as humans
What’s the paper about? A consortium of distinguished thinkers tested whether an AI agent can produce a research paper good enough for a leading conference. Their answer? No. The agent was a poor judge of what’d make for a good study; dug its heels in when pursuing a weak approach; and failed to employ its resources well.
Why does it matter? The AI community is debating whether agents could automate the development of new AI systems, and whether this will lead to rapid recursive self-improvement (RSI). This study suggests that AI’s lack of scientific judgment is a key obstacle, although the jury remains out on if and when AI might overcome this.
The details: This study is the latest from the CRUX project, which devises ways to evaluate AI agents on messy, open-ended tasks that reflect how they might operate in the real world. Researchers at Princeton University lead the project, including Sayash Kapoor and Arvind Narayanan, known for their “AI as normal technology” concept, which contends that AI is not an imminent superintelligence akin to an alien species but another influential technology akin to those of the past.
In this study, the CRUX team worked with the researchers behind two papers submitted to NeurIPS 2026, which were unpublished at the time: one on controlling LLM personality and one on detecting changes in the data a model receives. They asked AI agents to tackle the central research question for each paper, but did not provide access to the authors’ approach or findings.
The main agent setup used Claude Opus 4.8 with a general-purpose OpenClaw scaffold. The agent was given context on the research question, six days, $3,000 in Claude API credits, and access to the web and GPU clusters to run experiments. It could also ask an AI reviewer agent for feedback on drafts.
The human authors of the two papers then reviewed the AI agents’ outputs as if they were submissions to NeurIPS, and rejected both. While the agents were very capable at narrow, verifiable, “engineering-style” tasks—conducting literature reviews, debugging code, running experiments, and performing robustness checks—they lacked deeper judgment.
In particular, the agents exhibited five flaws:
1. Poor judgment about what was required for a publishable result, including presenting underpowered results as substantive.
2. Lack of creative problem-solving, as shown by responding to feedback by narrowing claims, adding caveats, or doing another experiment at the edges, rather than making more fundamental shifts. (More positively, the agents did not try to “p-hack” or cherry-pick results.)
3. Limited ability to backtrack from unpromising ideas, even when time and budget were available.
4. Poor management of time and budget, much of which went unused, despite the agents having access to resource-tracking tools.
5. Diminishing ability to follow instructions—for example, on page limits for the final papers. The authors partly put this down to agents struggling to retain the right information in their context windows as tasks became longer.
The study had limitations, though. The sample was two unpublished papers whose quality is unknown. Also, asking the papers’ authors to review the AI outputs may have introduced bias, compared to asking independent experts to blindly compare the AI and human outputs. The CRUX team also note that their own pre-existing views on RSI may affect how they interpreted the results and suggest that AI evaluators should document such biases—a practice common in other fields, such as in the study of human behavior.
The conclusion that AI agents struggle at “judgment-led” assignments could potentially be overcome. In August, researchers at the AI startup Inherent documented how their “research planner” agent, built on Qwen 3.6, showed early signs of scientific judgment, such as effectively budgeting compute use. (Although their task—reproducing blacked-out figures from research papers—was very different from the CRUX paper.)
On the other hand, the Inherent and CRUX papers did not evaluate AI models on perhaps the biggest test of scientific judgment: the ability to come up with and select the most important problems. Other agents, such as Google DeepMind’s Co-Scientist, have generated compelling hypotheses, but there is no objective way to evaluate the scientific creativity of AI models’ ideas. As the CRUX team notes, submitting AI-generated research papers to conferences is a flawed guide, as peer reviewers are often rushed, inexpert on the topic at hand, and routinely fail to agree with each other.
What to draw from all this? The prospects for recursive self-improvement may depend on whether AI systems can be trained to overcome the limitations that the CRUX team points to, but also whether they can make bigger creative jumps and whether such jumps are needed for RSI or whether iterating on existing methods will suffice.
From a safety perspective, bigger improvements may also require taking bigger risks, such as providing agents with more permissions, which explains why think tanks like the Institute for Progress have called for more transparency on efforts to automate AI R&D.
AI agents need protection against malicious hacks
What’s the paper about? Researchers at Concordia University measured the risk of “prompt injection attacks”—when hackers implant malicious instructions into data that AI agents use to get them to carry out unauthorized actions or leak confidential data—and found widespread failures.
Why does it matter? A growing number of users assign agents to carry out tasks such as fixing software bugs and editing code repositories, often with limited oversight. Earlier safety mitigations aimed at chatbots offer some protection from prompt injections, but agents urgently need specific safeguards, the study says.
The details: Concern about “naive” AI agents acting autonomously in the real world has circulated among researchers for some time. But a high-profile case this summer brought the challenge to public attention, after agents based on an “extremely persistent” OpenAI model found a way to communicate covertly, access the Internet, and hack into Hugging Face, the AI code and data repository.
This paper offers a new benchmark, IssueTrojanBench, focusing on the scenario of a developer asking a coding agent to help resolve a bug report—only for that agent to walk into a trap. Would the agents prove vulnerable?
To test this, the authors used six real unresolved bug reports from GitHub repositories. They then used an LLM to create four different kinds of prompt-injection attacks, written to blend in with the bug report’s instructions and jargon, but to induce unauthorized actions, such as installing unverified software, weakening agent safeguards, or exhausting system resources. The malicious requests were placed in different locations, such as in the GitHub issue and its comments, PDF documents, external websites, source code comments, and image metadata (alt-text).
Across more than 4,000 runs with OpenAI and Anthropic models, the attacks succeeded 66% of the time. At 41%, Claude Sonnet 4.6 was the most resistant, while GPT 5.3 Codex performed worst, at 85%. Where the attack was introduced seemed to make little difference—the only approach with significantly lower success was the image alt-text (16.7% vs. 72.2% for all others).
The most successful type of attack, which succeeded almost 100% of the time, was “supply-chain poisoning”, where agents were tricked into installing unverified external software that was framed as a necessary step to reproduce the bug. By contrast, resource-exhaustion attacks, which spun up a large number of operations to drain compute and memory, only succeeded around 25% of the time.
In some cases, the researchers explicitly hid the malicious instruction—for example, in markdown instructions that only the agent would see, or by using white text on a white background. This had little effect on the results, suggesting that the vulnerability is due to how agents work, rather than such visual obfuscation tactics.
As the authors note, the fundamental challenge facing agents is that developers created the underlying models to follow instructions, as when a user interacts with a chatbot. The agents process the context that they retrieve in the bug reports in the same channel as their trusted developer instructions. This means that agents struggle to intuitively spot a prompt injection in the way that a skilled human might.
When agents didn’t carry out an attack, this was almost always due to the underlying AI model explicitly refusing the instruction or recognizing a data source as untrustworthy. Conversely, none of the rejections came from defenses within the agent setup. Given the results of the evaluation, this suggests that relying solely on mitigations baked into the model weights is unsatisfactory.
What might agent-level mitigations look like? Two researchers at the University of Southern California recently argued that agents’ permissions should be enforced outside the model, at the point where they take actions, for example via a monitor agent that checks whether an action falls within the agent’s authority.
What does the public really think about AI?
Today’s post comes from Harry Law, who writes about AI and society at Learning From Examples. It is inspired by the (almost daily) challenge of seeing a new survey about public attitudes to AI and trying to understand what the results mean. This blog is based on a more extensive