A growing number of people worry that AI will lead to fewer jobs. Yet labor market data shows no clear reduction, and experts disagree sharply about what’s coming next. Some assert that the technology will automate all or most human work, while others argue that new jobs will arise and AI will help us perform them.
How should we best evaluate this automation/augmentation question? Sol Messing, Charvi Rastogi, and Kweku Kwegyir-Aggrey lead one such research effort at Google. They think that breathless predictions of AI-induced job loss overlook the reality of human work and what the technology can currently do. They hope that their new evaluations will bring these nuances to life.
—Conor Griffin, AI Policy Perspectives
Conor: Why do you argue that popular narratives about AI and the labor market are getting it wrong?
Kweku: There is no economic law that states that increasingly capable AI automatically spells job loss. Rather, there is a set of conditions—such as AI agents reliably carrying out complex tasks—that make that outcome more likely. But these conditions have not yet been met for most roles.
This means that most organizations will adopt AI via their employees. And so the most important questions are: How are these workers choosing to interact with AI? How is AI affecting them? And what do these effects mean for how we should view the technology?
Most technical evaluations of AI abstract away this human experience. They compare the performance of humans and AI on various tasks, implying that AI deployment is an either/or situation. They also fail to capture the messiness of the workplaces that AI will be deployed into, where context is often unsaid and relationships paramount. This focus means that models may be optimized to automate unrealistic versions of tasks, rather than to augment workers who perform these tasks in the real world.
We come at the AI jobs question from the human-computer interaction standpoint. We focus on how workers use the technology and are affected by it. We want to use that vantage point to provide more realistic evaluations of AI’s impact on jobs and workers.
Conor: What does that look like in practice?
Sol: We are creating digital simulations of workplace environments built around collections of tasks, called “workflows”. When an insurance claim adjuster needs to respond to a new claim, they have to determine if the claimant’s policy covers the damage, how much money to set aside, and the final payout. We want to understand what happens when AI is introduced to this workflow. So we are running experiments to compare the quality and cost of outputs in three setups. In the first, the human proceeds without AI. In the second, the AI agent attempts to carry out the workflow end-to-end. In the third, the human works with the AI.
This research direction was first proposed by our colleague Kristian Lum. It approaches the jobs question from a sociotechnical perspective—i.e., the view that technical systems (like AI models) and social systems (like people, organizations and the wider economy) are inextricably linked. You cannot study one without the other. And you cannot optimize one without the other.
Conor: What specific questions will your research answer?
Sol: We want to understand when and why there is uplift from humans and AI working together, compared to “humans only” or “AI only”. We expect that much rests on two factors.
First, how tightly bundled are the tasks in the workflow—that is, how much does success at one task determine success at the next? Second, are the outputs straightforward to verify? If tasks are tightly bundled and hard to verify, then more human expertise will be needed to steer AI use. But if a lot of expertise is needed, then the productivity gains from AI use—once we consider the cost of both human and machine time—may fall.
For example, agents can generate compelling scientific hypotheses. But verifying a hypothesis is usually quite hard, outside of some areas in math and computer science. Expertise is needed and there is subjectivity—the quality of a hypothesis may not ultimately become clear until long after a study is published. The quality of a hypothesis also hinges on the literature review underpinning it and the experiment to test it—these tasks cannot be neatly separated.
By contrast, scraping public data often breaks cleanly into sub-tasks that are relatively easy to verify and can be carried out in isolation—selecting the web pages, writing the code to parse them, and cleaning the data. This may make full outsourcing to AI more viable or narrow the gap, in terms of quality and cost, between “humans + AI” and “AI only”.
We hope our research will shed light on how these two factors play out, across a diverse range of workflows, and across different kinds of workers.
Conor: How will you study the effects of AI adoption on workers themselves?
Charvi: We want to move beyond simpler questions like whether workers become faster or slower with AI, and understand why this is the case, as well as how workers feel about using AI.
The AI automation/augmentation debate often implies a simple binary: either a task is automated by AI, or AI helps an employee to do better. In practice, preparing a task for automation, and monitoring it afterward, often requires a lot of work. Used badly, AI agents could trap users in a thankless cycle of micromanaging and reviewing—much like managing an unreliable colleague. In such scenarios, you don’t have automation or augmentation. Rather, the quality of a person’s job may degrade, they might lose the fulfillment they once got from their work, and the new AI use case may prove unsustainable as a result.
So, we’ll measure whether people feel excited, frustrated, or overloaded as they use AI, and compare how they say they want to use AI with how they actually use it. I expect to see interesting findings here, especially when comparing junior and senior colleagues—some might be initially excited about an AI use case but then become frustrated with it, or vice versa.
Conor: Evaluations like this are often criticized because the AI agents are given an unrealistic amount of support. But without such support, the agents might fail for innocuous reasons, like getting stuck on a pop-up window. How do you balance the desire for realism against the need to elicit agents’ true capabilities?
Kweku: When evaluations provide agents with crystal-clear instructions and all the necessary information to do a task, they bear little resemblance to most jobs. That approach reduces what is being evaluated to basic instruction-following, rather than navigating open-ended situations, which is where expertise often comes in. For example, if you give an agent a bunch of legal documents and ask it to write an argument, you miss some of the hardest parts of the job—such as deciding what evidence to consider in the first place.
Led by our colleague Justine Breuch, we have built environments that we think better reflect real-world work. In one workflow, an academic coordinator has to decide whether students have used AI improperly. We made sure that the environment is machine-readable, so that agents aren’t tripped up for basic reasons. We also provide written policies and emails to work from. But we inject real-world messiness: the instructions are unclear and the data contains mistakes. The AI agent and the human worker have to figure out how to write up the cases and notify the right stakeholders. In the future, we hope to make the environments more dynamic, with an email with new information arriving midway through the workflow, for example.
The barriers to full automation
Conor: Given the rapid rate of AI progress, why don’t you think agents could automate more workflows end-to-end?
Sol: Lots of modern work is characterized by long-horizon tasks. Employees must plan and execute multiple interrelated steps in a dynamic environment where success isn’t immediately clear. If I am planning my team’s research roadmap for the next quarter, I need to deal with lots of ambiguity, like what other research groups are planning, and how our work fits in. I also have to navigate institutional constraints, like budgets. These workflows will be difficult to automate, absent a fundamental reorganization of the workplace, such as everybody making their plans highly visible to agents.
Then there is the reliability issue. The AI evaluation organization METR has shown that agents are getting better at long-horizon tasks, particularly in software engineering. But the METR graph that is most widely shared—which shows a regular doubling in the length of tasks that agents can do—measures performance at 50% reliability. If an agent only succeeds one time in two, then it is fundamentally unusable for many professional applications.
Kweku: That is why evaluators distinguish between “pass@k” and “pass^k” metrics. “Pass@k” measures whether the agent succeeds at least once in k attempts. An agent’s output is probabilistic—given the same inputs it can produce two different outputs. So, giving the agent more tries helps to evaluate whether it is capable of performing a task.
But this is separate from the question of whether the agent reliably performs the task. “Pass^k” metrics measure this. They evaluate whether an agent succeeds every single time, across k attempts. For the AI jobs question, reliability will determine how quickly AI is deployed, especially for higher-stakes work. But reliability results get less attention on social media.
Conor: Why are agents unreliable at long-horizon tasks? And is this temporary?
Sol: The first constraint is “context rot.” When carrying out a long task, an agent sends the entire conversation history back to the model at every step, which quickly fills up the context window. Even before hitting a hard token limit, the model’s reasoning starts to degrade, and it struggles to attend to the most salient points. It starts to make mistakes it wouldn’t make on short tasks and can lose the plot.
AI developers have various workarounds, like having an agent read and write its plans to a scratchpad or delegating tasks to sub-agents. But context rot remains a persistent bottleneck.
A related constraint is the absence of continual learning. Many jobs are underspecified because human employees can learn as they go. But when an AI agent encounters something novel, it cannot dynamically update its weights and representations to capture it. This limitation is hard to address. If you try to fine-tune a model on your workflows, you risk “catastrophic forgetting,” where you inadvertently erase a model’s foundational knowledge or safety training.
Conor: If agents become more reliable on long-horizon tasks, for example due to a continual-learning breakthrough, would that unlock more end-to-end automation?
Kweku: Yes, but much modern work is also messy and deeply social. This will make end-to-end automation complicated, even as AI agents improve. Think of everything that sits between a formal job description and the reality of collaborating with humans every day. A lot of this knowledge is tacit or hard to articulate, such as dealing with feedback from colleagues, and knowing which parts are literal suggestions and which are not.
Conor: But if organizations start integrating AI agents into their chats, emails, and databases, won’t those agents soon have more tacit knowledge than any employee, at least for workflows that can be executed on a computer?
Charvi: Tacit knowledge is not just about accumulating knowledge. It’s about knowing what information to use, when, and how. Humans often learn these things through relationships. Organizational sociologists and psychologists describe teamwork as “collaborative sense-making.” When you work as a team, you learn how to coordinate, transfer the necessary context, and establish what an organization truly values. The economist Luis Garicano has argued that if you offload all this work, you might jeopardize a lot of what drives value in a modern organization.
Then there is the question of accountability—at some point, somebody has to be responsible for the outputs and how they are produced. Even setting aside legal liability, organizations will need somebody to own the final decisions.
A final aspect is human taste, or judgment. I work in our Responsibility team, where our mandate is to understand and steer how frontier AI will affect society. My organization has a view on what Responsibility means, so does my team, and so do I. When I am working on a research project, I am expected to draw on my own personal judgment. By contrast, current AI systems can trend toward more homogeneous outputs, whether in their writing or web design.
Sol: A lot of these aspects of modern work—such as taking accountability or building relationships—rely on an employee having a persistent identity, which gives them a personal reputation to consider and a stake in future outcomes. A permanent identity is also central to how we traditionally conceive of an “agent.” But AI agents do not currently have it.
Conor: Do these obstacles mean that the job picture for humans is rosier than is often presented?
Sol: Predictions of massive near-term job loss ignore the complexities of integrating AI into existing workplaces. A lot of evaluations in this area—including our own—also focus on workflows that can be conceivably carried out on a computer and, thus, by AI. Lots of jobs, from social work to food preparation, don’t work that way.
On the other hand, some roles face real risk. Think of graphic design tasks that an organization might have traditionally offshored, or posted on freelancer job platforms. These tasks are already unbundled, and AI is increasingly good at them. AI might also upend what a lot of jobs consist of, even if it does not fully automate them—if individuals start to consult agents to mock up interior design ideas, before turning to experts to carry them out, this will reshape those jobs.
But workers also shouldn’t view AI automation as automatically bad for their prospects, or augmentation as automatically good. Done well, automation lowers the cost of a service. If demand for that service is elastic—i.e. it grows when the service gets cheaper—automation may lead to higher employment. This is one read of what happened with software engineers in 2026, where agents now write a growing share of code, but employment has held up.
Conversely, only so many people need their taxes done. AI might augment tax accountants and make them more productive, yet employment could still fall, if there isn’t an inexhaustible appetite for tax preparation. So the long-term prospects for a given type of expertise depend not just on how AI performs, but on underlying demand for that expertise, which depends on various factors—from population growth to regulation.
What to do?
Conor: What should organizations be considering when incorporating AI into their work?
Sol: It matters whether you are designing your organization around automation or augmentation. The economist David Autor cites autopilot software in planes. With full automation, pilots might let their minds wander and cede excessive control to the computer. When the computer encounters an edge case, such as extreme weather or failed sensors, it can malfunction and the human pilot may not have the time, skills, or interface to take over. This is why aviation regulators now push pilots to keep their flying skills sharp, and why airlines use augmentation tools like head-up displays to make flight sensor data more accessible to pilots.
Conor: But most organizations aren’t responsible for life-and-death situations. Couldn’t they experiment more with end-to-end AI automation?
Sol: The consequences will be less extreme elsewhere. But if you start from the view that full automation is possible, you can still experience a damaging capabilities mismatch. In 2017, Tesla attempted to build a completely automated factory—the so-called “alien dreadnought”. It failed because the robots were not ready to be deployed on flexible materials like routing wiring and hoses. They lacked the necessary dexterity, and it quickly became a production disaster. Robots are now becoming more capable, but the threshold for reliable, safe, cost-effective performance is high.
Conor: If organizations want to focus on augmenting employees with AI, what concretely could they do?
Charvi: I think top-down mandates, where leaders demand high AI use from employees across the board, will often backfire. They can lead to review bottlenecks, low-quality outputs, skills degradation, and confusion if or when they are rolled back.
Such diktats tend to view workflows as generic, and ignore the subtle ways in which human expertise is needed to deploy AI well. Workers are often best placed to identify these nuances and so genuine augmentation should look participatory. Leaders need to find ways to empower frontline workers to experiment with AI and help determine where it is most helpful.
Conor: But most organizations aren’t worker collectives and presumably won’t want workers to stipulate where and how an organization should use AI?
Charvi: Organizations can draw on different kinds of evidence and research—we already see the rise of “eval engineers” who help organizations evaluate how to best adopt AI agents in their proprietary workflows.
But senior leaders have a natural incentive to view an organization through abstract mental models and top-down metrics, which can overlook employees’ on-the-ground knowledge about how to deploy a technology well. We saw this in aviation, where pilots collectively pushed back on aggressive cost-cutting automation, which ultimately improved safety. Incorporating workers’ views—especially once they have a good understanding of how AI works and its strengths and limitations—will make AI strategies more insightful and durable.
Organizations also need to identify the skills and accountabilities that their employees need to retain. Traditionally, if a software engineer wanted to update a codebase, they wrote a changelist (CL) and other engineers reviewed it. Now, AI has dramatically increased the number of CLs, and engineers are using AI to review them. It makes sense to incorporate AI in these workflows, including to catch mistakes. But this cannot be done well without input from engineers on questions such as: How to reduce low-quality AI-generated CLs? How to ensure that engineers maintain a coherent working model of the systems their code underpins? How to replicate the apprenticeship that working on CLs traditionally provided junior engineers?
Conor: If organizations design their AI adoption strategies around employees, doesn’t this risk entrenching outdated operating models? Might organizations find themselves outcompeted by nimble startups built around agents?
Sol: This is the 64-million-dollar question. Historically, we know that when general-purpose technologies were introduced, we didn’t see much impact on productivity until organizations and institutions were redesigned to take full advantage. With electrification, the economic historian Paul David showed that swapping steam engines for electric motors only led to gains decades later, when manufacturers pivoted from multi-story belt-and-pulley factories to electrified assembly lines.
AI agents may well usher in entirely new ways of organizing cognitive work. This year we have seen examples of “agent swarms” and “AI directors”—a human who supervises an agent, which in turn supervises many sub-agents. But the resulting outputs are often riddled with coordination challenges. So multi-agent orchestration will need to improve a lot before these approaches can reliably compete with what human-led organizations can do.
Conor: What should policymakers draw from your work? One conclusion might be that job seekers need AI training. But government training programs often don’t work well.
Sol: Our work shows that you typically can’t steer, verify, and correct AI outputs if you don’t understand a specific domain. So incentivizing apprenticeships and entry-level hiring may be more useful than more generic AI training programs.
Policymakers can also help create the evaluations that society needs, to determine whether important workflows in healthcare, education, or finance are ready for automation, or what kind of human-in-the-loop augmentation is needed. There is a clear logic for government involvement here because they could develop independent, private evaluations that would be less likely to “saturate” than those developed by industry or academia.
Charvi: Policymakers must also address thorny questions about how to update liability frameworks and labor policy to shape how organizations adopt AI. Just as aviation regulators mandate pilot training and accountability, a company should not be able to simply blame an agent when a high-stakes workflow fails. Ensuring that workers have a voice in how the technology is developed and deployed is also paramount.
Conor: When do you plan to publish your results?
Sol: We are working with several external partners on this work and hope to share more early next year.