If you're partnering, they're not independent

A short note on the glaring contradiction in the first sentence of Anthropic’s evaluators announcement.

We’re partnering with Accenture on independent evaluation of frontier AI.

If you’re “partnering,” they’re not “independent.”

Post-mortems of the 2008 financial crisis found that a major failure point was credit rating agencies. As banks devised new instruments that packaged and securitized mortgages, they told the market that risky debts had been transformed into safe investments. This was a moment for ratings agencies: to do the diligence to determine whether that was true.

Of course, they failed, and their failure to accurately assess risk was a major factor in the following crash. Part of the problem was the novelty and complexity of the instruments; the banks themselves weren’t sure how to model what they’d built.

Part of it was the incentives. Credit agencies are hired and paid for by the banks whose instruments they rate. That means that banks could shop for better ratings, and agencies had an incentive to provide better ratings to keep their customers — not investors, but banks — happy. It’s hard to say how much this actually happened, but there is evidence of it.

In that circumstance, you’re staking the financial system on the agencies having an incorruptible culture of independence, integrity, and humility (to be willing to acknowledge when complexity makes a clear rating difficult). That turned out to be a bad bet.

(As a side note, twenty years ago, “incentive” filled the role that “narrative” does today — a universal explanation for human behavior. I’m curious about what led to that evolution, what constructs like that came before, and what’s likely to come next.)

The complexity, the stakes for humanity, and the temptation (i.e., profit) are orders of magnitude greater for AI. Embedded evaluators is a partially correct idea. But hiring a consulting firm to play that role is definitionally not independence.

That’s worse than nothing. If Accenture becomes a rubber stamp — not out of malice or incompetence, but simply because the incentives make independence prohibitively difficult — then they will provide false assurances about problems that should have been investigated.

There should be others embedded at AI companies. They should work for us, not for the companies.