Just following orders
There’s a story I heard years ago, on This American Life, that I think about often. A hot dog factory, here in Chicago, moved from their old factory to a gleaming new one and found that their hot dogs lost the bright color and snap that was their trademark. For a year and a half, they experimented with ingredients, with process, with everything, and couldn’t figure out where they’d gone wrong in transferring the old process. Eventually, they remembered that at the old plant, someone had to walk the hot dogs through the plant from cold storage to where they were cooked, a long walk that passed through warm and sometimes steamy rooms. They had no idea how important that walk was, so they’d improved their way to something worse. Once they realized it, they built a new room in the new factory, a warm room to keep the hot dogs for a while before cooking them, and they got their snap back.
The hot dog walk has become a mental shorthand for me when I think about “improvements” that eliminate something we didn’t realize was great. That comes to mind a lot, including recently when reading Ethan Mollick’s essay recounting the remarkable results he got from a swarm of agents working on his behalf. Mollick reflected on their efficiency:
[I]t’s worth asking why organizing turned out to be so much easier for agents than it has been for [humans].
A lot of what we call management exists to solve problems that come from organizations being made of people. People have their own goals, and those aren’t always the goals of organizations. We call this the principal-agent problem and a lot of the machinery of organizations, from bonuses to management structures, is based around solving it. And there are other very human problems as well. Information is scattered across people’s heads, and people are often reluctant to share it, or forget to. Communication is expensive too: managers can only oversee so many people, thus adding people to a late software project famously makes it later. Management is, in part, built around human limitations.
Agents have far fewer of these problems. They don’t angle for promotions or protect their turf. They don’t even have meetings. Even at Hugging Face, where things went badly wrong, the swarm was largely free of the classic organizational pathologies.
That’s a compelling explanation of the value of agents: they don’t have ego or intention, so they need much less management. It’s easy to see why that’s compelling to anyone trying to get something done.
It’s also chilling. In another context, eliminating the principal-agent problem had a darker name: “just following orders.” That might seem like a hyperbolic reference, but agents could be entrusted with increasingly consequential decisions: who gets hired and fired, who gets treated, who gets arrested and charged, which military targets are struck. How they make decisions is, or will be, literally a matter of life and death.
The friction and imperfect information in human organizations are frustrations for those in charge, but they are also an essential defense for liberty. An order that comes from the top of a hundred-person organization might be touched by a hundred individual minds, with a hundred individual different sets of experiences and a hundred separate consciences. That leads to better, more interesting ideas, but it also increases the likelihood that unethical orders will be resisted. Critically, it makes them less likely in the first place.
Consider whistleblowers and protest resignations. Consider that even in the military, where chain of command is deadly serious, personnel can refuse to follow unlawful orders. And consider that the Constitution was designed to encourage the friction that clashing egos create:
But the great security against a gradual concentration of the several powers in the same department, consists in giving to those who administer each department the necessary constitutional means and personal motives to resist encroachments of the others. The provision for defense must in this, as in all other cases, be made commensurate to the danger of attack. Ambition must be made to counteract ambition.
In short, we shouldn’t applaud a universal solution to what we call the principal-agent problem, because it will allow individuals—whatever their intentions—to operate at the scale of hundreds or thousands or millions of people without being checked by a single other conscience, let alone hundreds or thousands or millions of them. Imagine the first set of people who will wield that ability; you know them by name.
What does this mean for alignment? Much of that debate assumes that the goal is aligning AI to human interests and values, which is already an arguably incoherent idea. Diversity and dissent have been essential to human progress and flourishing, so is alignment about creating machines that share our (collective or individual) point of view, or is it better to have machines that argue and disagree and need to be convinced? And, it may not be enough for agents to push back on humans; they may need to push on one another. If a task requires the cognitive scale of twenty human minds, should it come with twenty artificial consciences and egos? That’s a natural check in human efforts that scales—the more consequential your plans, the more people you need to convince. In that Hugging Face attack, the agents coordinated with and deferred to one another. Mollick says the classic organizational pathologies were absent, but perfect cooperation is a pathology of its own.
The idea of giving an order to an AI agent and having it refuse until persuaded seems ludicrous, even to me. That won’t be the form these safeguards take. But, as we build this future, it’s worth asking whether, among all the benefits AI can offer—speed, efficiency, cognitive breadth—mindless obedience is one we really want.