Who should decide how much risk society accepts from new AI firms?
Built from documented writings, speeches, decisions and historical records. Council members argue from those positions, applied to today's question.
- 1Anthropic is expected to go public in November at a valuation of up to US$2 trillion, a record IPO.
- 2Anthropic's prospectus devotes about 94 of 335 pages (nearly 30%) to risk factors, including warnings of "catastrophic or existential risks to humanity."
- 3AI companies have issued about US$466 billion in bonds so far this year, more than four times last year's pace.
- 4Six companies (Apple, Microsoft, Meta, Nvidia, Micron, AMD) delivered 57% of the S&P 500's gains since April.
The council treats these as reported facts, not settled truth. The members supply the reasoning; the asker supplied the sources.
No single firm, market, or regulator should decide alone. Government must mandate testing and liability, with one named office answerable for failure.
Rawls and Keynes argue a $2 trillion IPO proceeding on 94 pages of risk disclosure shows markets price gambles, not consent to catastrophe. Friedman and Hayek counter that no regulator can specify safety for a system Anthropic's own prospectus admits it cannot bound. Ostrom resolves this by rejecting the choice itself: nest firm audits, national testing, and international coordination, each with real sanctioning power. Arendt's warning sets the condition on all of it, that layered authority must still name one person who answers when a warning comes true.
The council splits on mechanism: Friedman wants courts and insurers pricing harm after the fact, Keynes and Rawls want testing mandated before it, since existential risk cannot be priced once it lands. That split is not resolved, but every member rules out leaving the decision to AI firms and their investors alone.
- 1Mandate government testing of frontier AI firms before deployment, rejecting firm self-certification and voluntary disclosure as sufficient.
- 2Build layered oversight combining firm-level audits, national testing regimes, and international coordination bodies, each with real sanctioning power.
- 3Name one specific office or person who answers when a catastrophic AI failure occurs, so accountability cannot be diffused.
A veil of ignorance would never accept firm self-certification.
The article's finance academic is right about the structure, wrong to think markets will supply the correction themselves. Behind a veil of ignorance, not knowing whether I would be an Anthropic shareholder or a bystander harmed by a model none of us can yet see fail, I would not accept 94 pages of risk disclosure as a substitute for enforceable rules. Disclosure tells investors what they are gambling. It tells the rest of us nothing we can consent to. The basic structure must set the terms before the gamble, not narrate it after.
Regulators mandating AI tests will certify ignorance, not safety.
Rawls assumes a regulator can specify what "safe" means for a technology its own builders cannot fully predict. I argued in 1945 that the planner's problem is never information you lack by accident, it is knowledge that exists nowhere in centralised form. Anthropic's 94 pages of risk disclosure are an admission of this, not a failure of it: the firm closest to the system still cannot bound its own risk. A government mandate will produce a checklist, satisfied on paper, while the real danger moves somewhere the checklist never asked about.
Liability through courts, not permits through regulators, should set the price.
Hayek is right that no agency can specify safety in advance. But I would not stop at disclosure. Make the firm bear the full cost of harm it causes, through strict liability and mandatory insurance, and let courts and underwriters do the pricing markets already do for pharmaceuticals and aviation. The $466 billion in AI bonds this year shows capital is already betting on these firms. Force that capital to price catastrophic tail risk through liability, and you get real discipline. A permitting bureaucracy gets you paperwork and a false sense of safety.
Existential risk is uncertainty, not odds a court can price.
Friedman's courts need a precedent and an actuarial table; existential AI risk offers neither. I drew this line in 1921: calculable risk can be insured, genuine uncertainty cannot be priced, only guarded against. Anthropic's own prospectus admits "catastrophic or existential risks to humanity," language no actuary can put a premium on. When a $2 trillion IPO proceeds anyway, markets are not pricing the tail, they are ignoring it, exactly as they ignored 1929's. The state must mandate testing now, before the loss, because afterward there is nothing left to insure.
Neither a single mandate nor markets, build layered monitoring with teeth.
Keynes is right that this cannot be priced after the fact, but a single national regulator cannot govern it either. I spent forty years documenting fisheries and irrigation systems where neither privatisation nor a central authority produced durable safety; monitoring had to be local, graduated, and nested across scales. AI firms span jurisdictions the way an ocean fishery spans nations. The answer is overlapping authorities, firm-level audits, national testing regimes, international coordination bodies, each with real sanctioning power and none with a monopoly on judgment.
Spreading authority this wide can erase who is answerable.
Ostrom's nested design solves coordination but risks the danger I found more corrosive than any single bad actor: rule by nobody. When a board cites fiduciary duty, engineers cite technical limits, investors cite disclosure already made, and regulators cite jurisdiction, each passes judgment to the next office and no one is left holding it. The 94 pages of risk disclosure can function exactly this way, as a document everyone can point to and no one must act on. Whatever structure is chosen, it must name a person who answers when the warning comes true.
The convergence note
Where the council converges: Markets and disclosure alone cannot be trusted with catastrophic AI risk; some enforceable rule beyond firm self-interest is needed. Rawls, Keynes, Ostrom and Arendt all reject pure self-regulation.
Where it divides: Hayek and Friedman doubt any regulator can specify safety better than liability and price signals; Keynes and Rawls insist uncertainty this severe cannot be priced or insured at all. Ostrom rejects both a single mandate and pure markets; Arendt warns any structure, however designed, can still produce accountability to no one.
For a policymaker to decide on: Choose between a centralized testing mandate, a liability and insurance regime, or a layered multi-body system, and name explicitly which named office or person answers for catastrophic failure.