Stafford Beer and AI as Variety Engineering
Thoughts on The Unaccountability Machine by Dan Davies
This review is written by Nick Swanson from Google DeepMind’s public policy team. Subscribe for more essays, policy notes, and reviews and leave a comment below or get in touch with us at aipolicyperspectives@google.com to tell us what you think.
In his book the Unaccountability Machine, author Dan Davies uses the ‘management cybernetics’ model of Stafford Beer as a model for understanding the complexities and “poly-crises” of the last two decades. Davies argues that standardised and process-driven systems have emerged to manage the vast complexity we live with, but in turn they have had the effect of distancing decision-makers from the impact of their decisions. These ‘accountability sinks’ explain everything from why we have call centres that literally can’t help you, to austerity policies based on blanket debt-to-GDP ratio rules.
At the core of management cybernetics is the aphorism that “the purpose of a system is what it does” - and the implied second clause “not what it says it does”. This is an incredibly clarifying mental model of the world. The reason conversations with call centres (or more familiarly in the UK, GP receptionists) often come up against a dead end where your issue cannot be resolved is usually not because you haven’t tried hard enough, or they haven’t understood and routed your inquiry correctly. There simply aren’t enough appointments available at the right time – you are experiencing a ‘problem without an owner’. The purpose of a system is what it does.
When you internalise this world view, you begin to see examples of it all around you. Is the purpose of the public science funding system to facilitate the discovery of groundbreaking new ideas and creating entire new research paradigms (which has been in decline for some time), or is it doing something else? Are arduous grant application processes, and metrics based on citation quantity rather than the quality or novelty of research, really the best way to allocate scarce resources to advance science?
Combinatorial complexity
When a system contains two parts – according to cybernetics – the feedback between both agents in the system gives complete (and usable) information about the system. But when the number of agents rises, more and more circuits between nodes exist, and understanding combinatorial outcomes becomes impossible – and knowing the properties of individual connections no longer gives you a proper understanding of the whole. Complex systems thus have to be understood as a whole, or they cannot be understood at all, and knowing part of a complex system in depth can be riskier than knowing nothing. Davies argues it was better to be totally ignorant of cryptocurrency than to lump on because a graph was trending upwards the last time you looked at it. Failures in military intelligence are often a result of knowing too much about one part of a complex situation.
The result of the difficulty in managing complexity at a corporate or policy level, is the emergence of accountability sinks – the reduction of decision-making to a rule, often publicly stated, perfectly defensible and through a completely transparent process. But these rules break feedback links in systems, create problems with no owners, and are unable to deal with edge cases. Davies tells the story of 440 live squirrels being ‘shredded’ at Schiphol airport because it was beyond the distribution of cases which could be ‘computed’ by the paperwork intended to govern the movement of live animals into Holland.
Every ‘rule’ is an implicit model of the world, but a model which inherently has to leave out a lot of information about the actual world. There is no intention, there is just a network of cause and effect – a system which makes an outcome inevitable.
But as infuriating as reductive processes can be, they are also an essential means to help us navigate complexity. Though no rule can possibly model the world perfectly, often the absence of such a rule is worse. Procurement processes which prevent innovation and risk taking, or Civil Service competency interviews which seem designed to shut out “weirdos and misfits” with good ideas, are clearly suboptimal. But they are intended to offset the alternative – the patronage of middle managers or the financial interests of corrupt officials. This is the trade off we often live with when systems cannot compute complexity – we can have outcomes on the basis of reductive rules which cannot model every potential situation, or we can have outcomes on the basis of discretion and personal standing within a network.
AI as variety engineering
A core theoretical contribution of cybernetics is the Viable Systems Model, which centres on the concept of ‘requisite variety’. Requisite variety is – very simply put – the idea that a system must be at least as complex [varied] as the thing it is trying to regulate. The cockpit of a fighter jet is able to handle more complexity than the control panel of a railway train. If a jet only had go/stop, it would not get off the ground (or it would crash into the end of the runway).
AI is most useful to us when it functions as ‘variety engineering’, whether at the level of individuals, companies, states or societies – equipping them with the ability to regulate complexity.
A great example of this in practice is Google DeepMind’s research into the ‘optimal power flow’ problem (OPF), which created an ML system for balancing supply and demand on an electrical grid. Current methods of balancing the grid are done by humans, essentially switching flows manually. In a system as complex as directing electrons to innumerable sources of demand, from areas of supply (some sources of which produce too many electrons at a given moment, and some which produce too few when the wind isn’t blowing) is a hard task! As a result, energy is wasted and surplus carbon emitted. The OPF system - which demonstrated human-level functionality at superhuman speed - has the requisite variety necessary to balance a modern grid, reduce waste and reduce emissions.
If I was an energy minister with an ambitious plan to decarbonise the energy system by 2030, I would be asking why this isn’t already in beta testing on the Grid – implemented as a recommender system with human oversight at first, and then setting the performance benchmarks which should be met in order to get the grid running on this system autonomously. AI in public services needs to be read as something broader than just “LLMs doing what civil servants do”, which massively narrows the opportunity space. The same goes for GraphCast and weather prediction.
Radiology is a well-worn topic in AI policy debates. Yes, clearly Hinton was wrong, there are still radiologists. But understanding radiography is hard even for experts, who still – of course – miss things on occasion, and the healthcare systems in which radiographers operate (and the demands placed on them) can make it even harder. It isn’t obvious that humans operating in a demanding healthcare system have the requisite variety to manage the complexity of spotting and acting upon every scan taken, at the pace and volume which meets the public’s expectations - whereas ML systems might (though operationalising and delivering on this is a whole other set of questions related to feedback and regulation within a system).
Avoiding Cybersyn v.2
Cybernetics itself had its peak moment of hubris in the 1970s, in the infamous ‘Cybersyn’ system which was built at the height of the “socialist calculation debate”. Stafford Beer, working with the Chilean Government of Salvador Allende built a system called Cybersyn, where experts would analyse economic data, and attempt to allocate resources across the economy (or specifically, factories). Obviously it couldn’t work - the requisite variety required to run the economy of a developing country’s economy is not possible with the computing we have now, let alone with what they had in the 1970s (and Cybersyn only had one computer) – though the coup of Augusto Pinochet did technically cut short the experiment.
Mercifully, there is relatively little debate about today’s AI replacing the price signal in how we operate our economy. However, quite a lot of it does sound somewhat premised on the idea that we can simply replace civil servants with LLMs. There likely is a good amount that can be done better – much of internal government activity is repetitive, staff turnover is high and institutional memory strained, so an LLM taking your organisation’s knowledge base and an agent navigating form-filling (or writing briefings, proposals, presentations) on your behalf seems sensible and appealing. But we should think about systems in a more holistic way, and AI as something which can expand the capability, quality and service offering of the state.
Cybersyn, but for individuals and organisations
Government should also not think of AI in the public sector solely as being about delivery of services, but also in the ways that individuals can interact with those systems. I want to automate my interaction with the state - from booking GP appointments by just saying “book me a GP appointment” to my agent, to allowing the agent and the GP to subsequently change/rebook my appointment to manage demand/prioritisation at the practice – in line with boundaries I’ve set.
In the civil courts, two parties to a contract could have their agents negotiate a mutually acceptable resolution to a dispute, which could consider factors more varied than what can be included in a written contract (assuming both parties have pre-agreed to it). A contract is a model of the world, but one which leaves out important facts about the parties to it, and what they would be willing to negotiate. This technology does not exist yet, but it likely will at some point - and it could empower us with our own individual systems of requisite variety to parse the complexity of modern life, and of the social systems we have had to create to imperfectly manage that complexity. Ensuring this is possible should also be in scope of how Governments think about AI in public services.
A final thought which is important for any government when thinking about user-facing services driven by AI is what the “system” that people feel they are interacting with is. Davies is dismissive of Searle’s famous Chinese Room argument, which was intended to disprove the Turing Test. In a cybernetic analysis, Searle has the wrong object level of analysis – to the person outside of the room, the translator and the AI are the same system. It makes no sense to say the Chinese Room disproves the Turing Test, because it is answering a different question, one of accountability, not of intelligence (indeed it was never conceived of as such).
This argument can extend to the way people feel about their interactions with the state. A future AI model which books and optimises GP appointments, and the NHS, are to the user the same thing - a black box of a system I can’t possibly understand (and frankly shouldn’t have to), which I am taxed for and for which I have certain expectations of. Done well, this will be a great thing – a seamless experience where we don’t waste our mental energies navigating someone else’s institutional framework, or suffering frequent human error, certainly sounds appealing.
However, it is vital that AI in public services does not end up creating a new, hyper-technical form of the accountability sink. It is not the doctor’s receptionist’s fault that the appointment booking system works the way it does – it works this way because there aren’t enough appointments and some have to be prioritised over others. We should not assume that a fancy AI booking system, even one which is able to optimise prioritisation, will solve the complexity (and demands on) a system like the NHS – however over time it will allow us to rethink the tasks that comprise them at a more foundational level.
In cybernetic terms, it is important that when using AI in the public sphere, we need a capable “system 3” - a layer within the wider system which can ask questions about whether functions of the system itself should be redesigned or reimagined. This is a new way of thinking and not typically what public organisations have done in the past – they have tended to optimise for a specific function based on a model the world implicit in their design, rather than operate in a space where the problems they are solving for are yet to be discovered. If the purpose of a system is what it does, we should continually strive to ensure that its outcomes match our expectations, and rather than just throwing computing power into broken systems, we should seek to ensure that they themselves adapt and learn.
You seem to have developed a good clear grasp on these topics.
You said every ‘rule’ is an implicit model of the world. So are a lot of things. A spoken language and the culture that gave rise to it is also a model of the world in a similar way.
Keeping in mind that such "models" also leave out a lot of information about the actual world.
The hypothetico deductive model of science could be said to be the same.
They are models that enable us to talk about the world, make observations that are mutually intelligible and which can be assimilated (as they are public models rather than private models in one person's mind).
Anyway, I had never before heard someone articulate that ‘rules’ also are an implicit model of the world.
Food for thought, thank you.