The era of AI agents is decidedly upon us, with “swarms” committing acts that range from wondrous (solving maths problems) to worrying (illegal hacks and outright lies). But if the agentic era is here, the governance is not.
Part of the challenge is defining what we’re dealing with. Laws and regulations and norms—they deal with individuals or organizations. But how to treat a swarm of brilliant non-human entities?
One possible answer comes from the philosopher Christian List, who helped develop the concept of “group agency”: that collectives such as the corporation or the nation-state are best understood as goal-directed agents. This notion has captivated a leading Google DeepMind researcher on agentic safety, Matija Franklin. The two spoke recently, and AI Policy Perspectives had the fortune to listen in.
—Tom Rachman, AI Policy Perspectives
[Conversation condensed and edited for clarity]
Matija: Multi-agent systems can combine into an entity that’s more competent than any single agent. This led me back to your book Group Agency, from 2011. Can you explain its core idea?
Christian: Often in society, we speak as if collective entities are agents. We attribute beliefs, preferences or goals to firms, to organizations, and to states. One reaction to this is, “That’s just a metaphor not to be taken literally.” On the other hand, the law recognizes a counterpart of group agency in the form of legal personhood for corporations. Arguably, the very concept of the limited liability corporation, with its own legal status, was one of the big innovations in economic history that enabled all sorts of progress—while also having downsides. So, it’s worth taking the idea seriously.
Now, not every collective qualifies as a group agent. Take, for example, a crowd flowing through a public space. It might have emergent properties, but that doesn’t make it a group agent. Three basic conditions are necessary for a system to qualify as an agent. First, it must have a representation of its environment. Second, it must have goals. Third, it must have a capacity to interact with its environment to pursue those goals. Only suitably organized collectives can satisfy these conditions.
Matija: Five years ago, long before this period of multi-agent AI systems, you wrote a paper explicitly talking about the parallels between group agents and artificial intelligence. What was your thinking there?
Christian: Group agents are arguably the first instances of artificial agents that humankind has had. Biological agents, of course, have been around for a very long time. But group agents have also been around for millennia—think about the Roman Catholic Church as one of the oldest surviving group agents. So, the AI agents we’re now seeing are a new instance of this established phenomenon: that there can be artificial agents which are not simply biological organisms, but something else.
You might say that group agents, such as the corporation, are artificial agents that society created. They’ve been around for a long time, so we’ve debated how to think about them from a moral and ethical perspective, and from a legal and regulatory perspective. I’m not suggesting that all of this carries over directly to the AI case. But at least we’ve got a template. And I think many challenges in the context of AI are at least somewhat parallel to challenges in the case of group agency.
The most obvious examples are “responsibility gaps.” For instance, harm may be caused by a corporation or, in the AI case, by an AI system. In some such scenarios, it’s not easy to pinpoint a specific human or even a specific set of humans that can be held responsible. And then, if we can’t trace the harm to anyone’s action, we have no way of attributing responsibility. This is a responsibility gap.
In the corporate case, there are strategies to fill those gaps, among which is corporate liability. Even if none of the employees, managers, or board members of a company can be held individually liable, the corporation as a group agent can be held liable, and also qualifies as a legal person that can be made to pay compensation. Some jurisdictions have extended corporate liability from civil liability to criminal liability. But that produces interesting questions: How do you sanction group agents? In the case of a corporation, you can strip them of their assets. But in the AI case?
AI systems can cause harm, and if we can’t trace those harms to the doings, actions, or omissions of particular individuals, then the question arises: Is this like an avalanche, where nobody can be blamed, and nobody held liable for anything? Or should we find a way of closing those responsibility gaps?
Matija: A lot has happened this year. We have the government of Argentina considering making AI-run corporations a legal entity. We’ve also had cases of agentic systems behaving in complex but also worrying ways. Are we seeing early signs of group agency? Or are these still collectives of individual agents?
Christian: From a societal perspective, it’s scary. From an intellectual perspective, it’s a fascinating set of phenomena to study. Aside from group agency, I’m interested in aggregation theory, the pooling of many different individual inputs, sometimes to enhance the aggregate performance compared with the individual performance. Classic examples are what is sometimes called collective wisdom, pooling the judgments of a bunch of people who are individually only partly competent. All of a sudden, as a byproduct of aggregation, you can get a much stronger performance.
A lesser-known example of this: one of my colleagues, Nicholas Miller, in the field of social-choice theory [the study of how to combine votes or individual preferences into group decisions] did a study in the 1990s at the University of Maryland, where he looked at multiple-choice exams by students. In addition to individually grading the exams, he created a hypothetical “aggregate student” whose answer was the plurality answer in the relevant class—so, just the answer that was given most often. The plurality student performed spectacularly well; much better on average than the individual students. This phenomenon is well explained by a classic result called “Condorcet’s jury theorem”: plurality or majority decisions among sufficiently independent individuals outperform individual decisions even when each individual is only slightly better than random at making a correct decision.
What we are seeing with swarms of individual AI agents collectively pulling off a very powerful performance—even if it is nefarious—is that they are somehow reaping the benefits of aggregation. Each individual agent is fallible, noisy, not super successful alone. But if you’ve got enough of them acting together, their collective performance is far stronger.
What we are also seeing is coordination efforts among agents with communication between them. Many of these phenomena seem to involve what philosophers might call “joint agency” or “shared agency.” But are we seeing group agency in the stronger sense, where a novel collective agent emerges at the aggregate level? That’s less clear. A genuine group agent—as opposed to a phenomenon of shared agency across many individuals—would require some kind of genuinely integrated organizational structure.
This is like the difference between, let’s say, spontaneous cooperation in a protest that doesn’t necessarily give rise to a single new unit of agency versus the corporate form that does give rise to a new unit of agency. There is definitely room for artificial group agency. But if you’re mentioning the Hugging Face incident and other highly publicized cases, maybe this is currently best conceptualized as joint or shared agency or simply as cooperation among many individual AI agents, not as group agency.
Matija: You have argued that some groups are not merely collectives, but group agents with the capacity for responsibility. Are AI agents approaching similar capacities?
Christian: Compared with five years ago, the capacities of current AI systems are much more in line with the conditions for agency—they have representations of their environment; they have goals; and they can interact with their environment to pursue those goals. Moreover, they might increasingly have some of the capacities required for fitness to be held responsible: a capacity for normative judgment, a capacity for acquiring the information needed to exercise that judgment, and a capacity for controlling the resulting actions.
Just to footnote something here: I’m very careful to draw a distinction between agency as a functionalist notion on the one hand, and consciousness or sentience as a sort of phenomenological notion. We have no good reason, as yet, to think that current AI systems are conscious.
But at any rate, the question of consciousness is distinct from the question of agency and also from the question of whether those systems have the cognitive capacity for responsibility. And even when a system has that capacity, it must also be in a realistic position to carry out any obligations under liability, should something go wrong. For example, I can walk around as a pedestrian without insurance because there’s not much harm I can cause. But the moment I want to drive a car, I need insurance. Likewise, if a corporation wants to operate in a regulated market with high risk, there are regulatory constraints. Presumably, you can’t set up an airline unless you’ve got insurance.
So, even if an AI system were cognitively fit to be held responsible, and even if we were to introduce some notion of artificial legal personhood, regulation ought to say that, for the system to operate in a high-risk environment, it must have the right kinds of assets or insurance package. Then, if it causes harm, it can respond to any litigation and discharge its obligations under liability—for instance, by paying compensation for any damages. The other route would be to say the systems’ creators have to fulfill obligations under strict liability.
Matija: What are the main lessons of your work, especially as it relates to the group agents that AI companies may design?
Christian: We’re seeing evidence of deception on the part of AI agents, and AI agents collaborating. This is an area where research from neighbouring fields within economics and social-choice theory could become relevant. And maybe the whole field of mechanism design too, often known as “reverse game theory,” which seeks to work back from desired outcomes to set the right incentives.
In our book, Group Agency, Philip Pettit and I discussed what we call “the incentive compatibility desideratum.” The question is familiar from economics: How do you design the organizational structure of a group agent in such a way as to incentivize members, for instance, in cases of judgment aggregation, to reveal their judgments truthfully, to not misrepresent? Or in cases such as voting, how do you incentivize sincere voting as opposed to strategic or tactical voting?
It turns out that if we want to aggregate the preferences of individuals into collective preferences—and we want to do this in a fairly egalitarian manner—that often conflicts with what’s called “strategy proofness,” the requirement that no individual agent should have an incentive to lie about their preferences to manipulate the collective outcome. We would like truth-telling to be a dominant strategy for individuals, and that turns out to be remarkably hard to achieve, both in preference aggregation and even in judgment aggregation. Very often, we face trade-offs between democratic responsiveness, collective rationality, and strategy-proofness. We can have some of these but not all.
In the past, those kinds of analyses were intended for human institutions. But now, we are in a situation where similar analyses will become relevant to AI, especially where you have explicitly designed coordination among multiple AI agents. Imagine, for instance, a swarm of AI agents engages in some collective decision-making tasks through voting or aggregation in order to achieve a collective goal. Depending on how strategically smart the individual agents are, we may think about how to incentivize them to reveal their individual judgments truthfully or not to misrepresent relevant information they hold.
This shows that we can ask some of the same questions for AI agents that we were previously asking in the case of human institutions.