The danger is the point

This tweet thread about leaving Anthropic has gone viral. In brief, the poster, Jacob Coxon, left because he saw the extreme recklessness of 1) believing that AI can (or will) destroy humanity and 2) building it anyway.

I want to call attention to this particular item in his thread:

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

There’s a note I’ve had in a backlog for a few weeks, and this passage was the right push.

I was curious about this phenomenon—building something dangerous. Most people’s moral intuition is that we shouldn’t build dangerous technologies, particularly when the stakes are widespread death and privation.

Even if you complicate that with divergent incentives—maybe most people don’t want it, but those who will profit do—there should still be a point at which the risks clearly outweigh the benefits. There are a few arguable examples of technologies we haven’t pursued because of that, often in weaponry or biosciences—e.g., human germline editing.

And then there are nuclear weapons. All these decades after Hiroshima and Nagasaki, the horror of them is still beyond the reach of my mind. I’m just old enough to remember the tail end of the Cold War, and the sense that nuclear obliteration remained a live and insane possibility. If ever there was a technology whose costs positively dwarfed its benefits, it is the nuclear bomb.

And yet, we’ve struggled to limit proliferation and haven’t given up our own nuclear weapons. The reason offends intuition: it is precisely because nuclear weapons are so dangerous that we must keep building them—because we have ambitions for power or worry that others do, and because we don’t trust one another, we have to build. The danger is the point; the danger is the reason to build, and the more dangerous the technology, the more feverishly we have to build it. Germline editing is dangerous, but harder to connect cleanly and immediately to a power competition. That connection is obvious for nuclear weapons, and even more so for AI.

The soft version of this effect is what Coxon ascribes to Anthropic: it’s so dangerous that we need to build it first, so that we can use it responsibly.

The hard version is: we need to build it so we can use it against others before they use it against us.

The hard and soft versions are philosophically different but practically compatible. Anthropic may hold the soft version in their hearts, but their partners and customers (including governments, militaries, and defense contractors) absolutely believe the hard version.

We are talking past one another: boosters making arguments that harden resistance, opponents making arguments that increase commitment. For the last few years, the CEOs of AI companies have thrown themselves in front of every available microphone to tell the world that their technology would take all human jobs. That was unsubtle marketing for corporate customers looking to reduce their workforces, but it also helped to create a backlash that has real political strength. Some of those CEOs are now trying to walk back their statements, as if they’re surprised that the public isn’t excited about losing their livelihoods.

Similarly, when safety advocates point out the cataclysmic risks, it’s a signal to governments and companies to invest more. They may be absolutely right about the risks, but which president or senator wants to be remembered for ceding to adversaries supremacy in the most dangerous technology of our time?