Criminals are tech adopters. They used telegraphs in 19th-century fraud. They clipped pagers to their belts in 20th-century drug trafficking. They sold drugs on the dark web in the 21st century.
Now, you hear of outlaw AI systems with names like PersonaForge and OnlyFake, designed for everything from phishing, to generating malicious code, to forging IDs. But are these really what we should worry about?
To understand the threat, AI Policy Perspectives spoke with Ardi Janjeva of CETaS (the Centre for Emerging Technology and Security), based at The Alan Turing Institute in London, who co-wrote the report “AI and Serious Online Crime”, focusing on how artificial intelligence might scale up digital offences, particularly relating to fraud and child sexual abuse material.
—Tom Rachman, AI Policy Perspectives
[Interview edited and condensed]
Tom: Can you set the scene of 21st-century criminality before the explosion of large language models. How prevalent had online crime become compared with traditional criminality?
Ardi: Long before the proliferation of LLMs, a digital shift in criminality was occurring. Many groups saw the risk/reward ratio from digital methods as preferable to the risks of physical crime. Fraud was already, by a wide margin, the most common crime type in England and Wales, accounting for 41% of all reported crime back in 2024, of which approximately two in three cases were “cyber-enabled”. And the true figure could have been higher, as this kind of crime tends to be under-reported. Also, there was increasing ransomware extortion, infecting computers with software that blocks access until the owner pays. So, an ecosystem of hackers and crypto-launderers was in full swing.
Tom: Criminals have always used technology. Won’t policing just adapt?
Ardi: The answer is linked to AI agents. In online crime, you’ve always needed some level of expertise or a network of contacts to achieve your goals. That’s no longer the case if you have AI systems not only advising criminals how to do certain things but performing sophisticated actions themselves, and being able to adapt in real time to defensive measures that companies or governments put in place. Some criminal groups are already using LLMs to make tactical and strategic decisions; to craft psychologically targeted extortion demands, for example; to analyse data they’ve exfiltrated; to help them determine the ransom they’re going to demand, while generating visually alarming ransom notes as well. In November, we heard about the first AI-orchestrated cyber espionage campaign. Traditional assumptions about the relationship between criminal sophistication and the complexity of an attack don’t hold as they used to, now that AI can add a degree of expertise that the offender wouldn’t otherwise have had. It makes you wonder how long until we’re in the era of entirely AI-controlled criminal networks and markets. But I don’t see that as close; we’re still in the early days.
Tom: Many law-abiding organisations are trying to incorporate AI yet aren’t necessarily showing gains in productivity or profit yet. Are criminal organisations different?
Ardi: Estimates for how much AI may be boosting specific criminal groups’ profits and revenues can be a little tricky to come by! A paper from GovAI (the Centre for the Governance of AI) found that data on cybercrime damage is too incomplete and ambiguous to detect incremental AI-driven increases. In effect, we lack a single data source, also due to victims under-reporting, evolving crime definitions, and measurement inconsistencies. That said, we definitely are seeing integration of AI, particularly with deepfakes, and the shift from text-only scams to real-time synthetic interaction, such as high-quality face-swapping tools being deployed during video calls in scamming high-net-worth individuals or in romance scams. Before, it was a killer blow to the scammer when someone asked, “Can we have a chat?” The criminal would need to come up with an excuse, such as a poor internet connection, or a restrictive job that involved classified work. Now, it’s a more surmountable challenge. By harvesting 20 to 30 seconds of audio from social media, you can clone the voice of a victim’s friend or adviser with a degree of accuracy.
Tom: Can you recount a specific case?
Ardi: The British multinational engineering company Arup was defrauded of around $25 million in Hong Kong, when attackers used AI to convince one of the firm’s employees to carry out wrongful payments. They used a deepfake video call, purportedly bringing together different members of that company’s board, with pretty convincing replications of their faces and their voices. That illustrates the ability to isolate a specific person within a company, and exploit workplace hierarchies. It’s an example of exploiting both technology and human psychology.
Tom: That case was more than two years ago, and I haven’t heard of huge AI-assisted frauds since. If it’s happening, why isn’t it getting more exposure?
Ardi: There hasn’t been an avalanche of cases that we can point to, but this might be for a few reasons. First, not every criminal group will go looking for a $25 million scam because they know that it would raise alarm bells in a world where more people and businesses are wary of the AI threat. For many groups, it makes more sense to fish in smaller ponds. There was an example in Singapore similar to the Hong Kong case, where a finance director authorised a payment after attending a Zoom call with people he thought were senior executives, but the amount this time was $500,000. As for why we’re not hearing about such cases, victims are not necessarily announcing deepfake scams. If they’d suffered a customer-data breach or ransomware that locked a system, they might need to acknowledge that the crimes happened. But when a company falls for a deepfake, there’s little incentive to declare it because, arguably, it makes them look incompetent.
LONE WOLVES & CRIME GROUPS
Tom: Is it conceivable that a lone criminal could cause the level of damage that previously would’ve been possible only by an organised crime syndicate?
Ardi: I would say yes. Before, criminal groups faced a tradeoff between scale—say, millions of fraud attempts with a low likelihood of success—and precision. AI may give you precision at scale. For example, being able to scrape social-media data in real time to create contextually accurate phishing approaches. It’s “vibe-hacking” in the criminal context, being able to mirror the professional tone of specific industries, and make victims less sceptical. It is conceivable that you have a lone criminal using AI, automating the most labour-intensive parts of the crimes, whether that’s reconnaissance, or language translation, or social engineering. The more that AI becomes a silent partner to individual criminals, the more you may have decentralisation and fragmentation. That’s going to have big implications for law enforcement. When it comes to organised-crime groups traditionally, we know roughly what the gangs are doing, so it’s a case of preventing, deterring, disrupting. But when it’s someone in their room with an autonomous agent able to cause havoc, how do you go about stopping that, especially if you’re just a local police force?
Tom: I want to return to the police response later. But first, what do we know about AI crime around the world? Are there particular hotbeds—for example, when it comes to fraud and AI-generated child sexual abuse material ?
Ardi: It’s tricky to speak about trends by country because this is borderless. But threat reports seem to come from places like the CRINK countries: China, Russia, Iran, North Korea. Also, there seems to be a clustering around Southeast Asia of online romance scams and crime syndicates doing “pig butchering”, which involves criminals developing relationships with victims to carry out investment fraud. You have compounds in Southeast Asia where serious organised crime groups have trafficked hundreds or thousands of people, forcing them to run romance-scam operations, including using LLMs to conduct multilingual frauds, managing thousands of cases simultaneously, involving real-time voice cloning and deepfake video filters to bypass the accent barrier. With AI-generated child sexual abuse material, or CSAM, this wasn’t something we focused on specifically, so I cannot say with a high degree of confidence. But it seems less geographically concentrated—it could just be someone who gets their hands on an image-generation app. With CSAM, we noticed a much more “collegiate” approach to sharing content and practices, such as jailbreaking models and ways to bypass filters.
GEOPOLITICS & THE AI RACE
Tom: Have you found evidence of nation-states destabilising adversary countries by allowing criminal groups to conduct AI attacks abroad?
Ardi: Drawing a direct link from authorities in China, Russia, Iran, North Korea to criminal groups can be tricky. Still, I think there’s enough evidence across some of those countries, especially North Korea, that there is something happening. One example is North Korean operatives using LLMs to fraudulently secure and maintain remote employment positions at Fortune 500 tech companies by creating elaborate false identities with convincing professional backgrounds, and completing technical coding assessments, all generating profit for a sanctioned regime.
Tom: Is global competition to build advanced AI having any secondary impact on criminality?
Ardi: The US-China AI race has nudged the Chinese leadership into thinking that their route to success is open-weight models. Some of the Chinese open-weight models became the go-to models for CSAM images and video generation. When researchers produced the AI Agent Index, they looked at five AI agents from China, and found that only one had published safety frameworks or compliance standards. But it’s not as simple as naming one country, and China isn’t the only place open-sourcing. Actually, what’s interesting is the use of multiple models in tandem. They might use a US chatbot to refine code or to write the script for a job application to infiltrate a company, while at the same time using open-weight Chinese models for specialised malware tasks.
POLICE STRUGGLING
Tom: Last year, Dutch law enforcement led a CSAM investigation called Operation Cumberland, involving 19 countries and bringing more than two dozen arrests. Your report adds a disturbing fact: that one person was arrested with a staggering 400,000 AI-generated images of child sex abuse.
Ardi: It’s effectively an attack on police resources. Leading models do have guardrails that block explicit prompts, but there are still serious hazards, notably with open-weight models. Groups like the Internet Watch Foundation point to a transition from AI-generated images to high-fidelity video, with a 26,000% increase in photorealistic AI-generated CSAM videos. The police need to verify that no real child is depicted in the images, which is a big detection and response challenge. It’s difficult to distinguish AI-generated material, so you have the risk that law enforcement investigates images of children who have not been physically abused, which has implications for the amount of false negatives that slip through the net. Also, real and fake images often exist on a spectrum, when you have techniques like face-swapping. Rightly, we get focused on whether a real child is being harmed. But we mustn’t devalue the harm of AI-generated CSAM, which risks normalising very harmful activity, while also providing a gateway for criminals to create and share real CSAM.
Tom: So how are police forces going to manage? Not just CSAM, but all criminality that AI will amplify. Because police forces are not typically staffed with elite technologists. Is there a push to address that?
Ardi: The UK government announced spending of £140 million for policing technology, including a new national centre, Police.AI. The idea is to centralise innovation, include robust testing and, crucially, to make tools available to all forces. Because not every force is going to have a deepfake detection expert, or any AI expert. So, the key is centralising that capability while making sure that people on the frontline have AI literacy. I can imagine that being a direction other countries follow because of the talent challenge.
Tom: Could criminals use AI to deliberately spam the authorities with false evidence, derailing prosecutions?
Ardi: Yes, if criminals know they’re being investigated, they could start spamming police tip lines with synthetic evidence, making it taxing for law enforcement to verify and investigate. They’d be able to create, say, 1,000 videos in 10 minutes. But it might take law enforcement hours to prove that even one is a deepfake. There’s that huge imbalance in the evidentiary system there that becomes really, really challenging to manage.
HOW POLICE CAN FIGHT BACK
Tom: What technological obstacles are slowing down criminals in AI adoption, and could those bottlenecks provide law enforcement a point of attack?
Ardi: Exploiting the compute bottleneck is promising—so, preventing criminal groups from accessing high-end GPUs. That requires international cooperation and coordination on sanctions policy, which can be difficult. If a criminal enterprise is in, say, Cambodia—how much leeway can UK law enforcement have? So countries need to work closely together, and with Europol, with Interpol.
Tom: What are other factors limiting criminal adoption?
Ardi: Some groups are worried about the digital footprint, and may also lack the skills to move from experimenting with AI tools to integrating them into higher-stakes settings. Criminal organisations that have made a fortune in a niche of crime. They’ll be thinking: “Well, okay, this is a model which is clearly capable, but it can hallucinate. I don’t necessarily need a probabilistic system to enter my workflow, and potentially give up my location, and give law enforcement more of an understanding of what I’m doing.” Law enforcement can try to deepen that concern, using AI to trace models used in attacks, identify the infrastructure, or the criminal LLM services used. On the other hand, some barriers to AI crime are coming down, such as the rise of vibe-coding and agentic AI. Also, as hallucinations fall in AI models, you can expect criminals’ scepticism to fall too.
Tom: If key barriers to AI crime fall, what do we do?
Ardi: One important approach is to follow the money. Law enforcement around the world needs to stop criminal groups from cashing out. Right now, the UK is leading Project WINTERPROOF, an international initiative designed to build crypto forensic capability in countries where the pig-butchering compounds are operating—so, freezing crypto wallets before funds are even laundered, making it incredibly hard for them to move money out of their digital ecosystem.
THE FUTURE & POLICY ANSWERS
Tom: What’s a crime that people don’t worry about much today, but that could become an issue with AI in the next few years?
Ardi: An under-researched set of risks are insider threats, which may come from integrating agents into the government, the military, or other critical areas. For example, if we replaced a large number of civil servants with AI agents that are vulnerable to prompt-injection attacks from a malicious or criminal actor, this would present a serious risk in terms of sabotaging critical government decisions or exfiltrating information.
Tom: What else would you like to see AI labs and policymakers do to tackle online fraud and CSAM?
Ardi: We all know about the testing and red-teaming that labs have developed, often in partnership with government AI safety institutes. But these processes need to evolve given the coming agentic-AI wave. For example, involving frontline law enforcement and fraud investigators to help simulate how agents may be used in crimes. Also, we often focus on how AI companies can stop their models from doing bad things, but there’s a case for thinking more about how the companies empower the public in the face of threats. This might mean features that help the user analyse a suspicious email, verify the provenance of an image, or flag the linguistic markers of a romance scam. In other words, scaling the defensive capabilities of AI as quickly as criminal are scaling the offensive capabilities. But we need nuance here: online fraud and CSAM predates this, so the solutions are not entirely with the AI companies. Also, in sensitive areas like CSAM, AI companies need governments to provide enabling legislation for them to do the safety research required. An example is the UK government’s move last year to provide exemptions for designated bodies, including AI companies, to better scrutinise models for CSAM generation, which may involve being in possession of that sort of material.
Tom: Could you say a bit more about the policymakers’ response?
Ardi: Legislative agility is key. In recent months, the UK government criminalised “nudifier” apps and tools. In liberal democracies, passing new legislation can take ages. But we’re seeing more of this trend of amending existing pieces of legislation. So, with nudifier tools, just by inserting “synthetic creation” into the existing definition of image-based abuse, the legislation inherits decades of case law regarding intent, consent, sentencing. That’s an approach that matches the agility of the criminal trends in AI.
Tom: And what advice for law enforcement?
Ardi: We’ve talked about how AI offers criminals unparalleled scale, and the ability to bombard law enforcement, and overwhelm it. But it can go the other way too: finding clever ways of using AI to waste criminals’ time, to lead them down rabbit holes, reversing the spamming issue. You throw mud at them to slow them down. You raise that barrier, make it costlier for them to be involved, exploit the bottlenecks to adoption.
Tom: So the technological cat-and-mouse between police and criminals goes on.
Ardi: Yes. But historically in policing, a lot of technology comes in, and the authorities don’t really use it, or take a long time to integrate it into their work. Given the potential scale of disruption from AI, there’s no choice for law enforcement but to move quickly.