A client asked this week if the hype about 10% was real. Hubinger’s figure is personal, more than 10% within a decade, and that number is not a company’s to manage. A company cannot set global p(doom), but it can decide what writes to production, what data leaves the building, and whether anyone can produce the record when a board member asks what AI you actually run.
Cipher Projects is an Australian engineering studio that ships production agents, private data layers, and an evidence trail through Clear Direction AI. We are not a law firm and not an assessor, and we will not hand you a company extinction percentage. This page says what the jargon means, what the labs said in public this month, and what changes on a build in Australia or Singapore.
Quick answer
A company cannot set global p(doom), but it can inventory shadow AI, keep weights and data inside a boundary, put a human halt on writes, and refuse unmonitored agents against production. Hubinger’s figure is personal, more than 10% within a decade, and it does not tell you whether the hype is “real” in a way your board can vote. The scarce thing is a specified problem, clean data in a boundary, and an evidence trail you can produce.
Best for: operators who heard this week’s lab noise and need a company move, not a metaphysics seminar. Honest limit: Cipher does not forecast civilisation. We inventory systems, put a halt on writes, and leave a living record; we do not certify you.
Last updated: 12 September 2026.
What does p(doom) mean?
p(doom) is a person’s stated probability that advanced AI causes a catastrophic or existential outcome, usually human extinction or permanent loss of control. It is a judgement call with a percentage stuck on it: not a measured rate, not a company KPI, and not a number your board can vote into existence.
The p is probability, and the doom is the bad tail: death, disempowerment, or a world in which humans no longer set the terms. People disagree on the time window, the mechanism, and whether “doom” means everyone dies or merely everyone loses. When someone says “my p(doom) is 10%,” they are reporting a belief, so ask them the window and the event. A 10% this decade is a different sentence from a 10% this century.
Dario Amodei, Anthropic’s CEO, has said he hates the term. At the Axios AI+ Summit in Washington on 17 September 2025 he still answered the question: “I think there’s a 25% chance that things go really, really badly.” Axios recorded that as his p(doom) number. In December 2023 the New York Times had already placed him in a 10–25% band. Treat those as dated public remarks by one lab CEO, not a scientific consensus.
Plain-English neighbours live on the cluster glossary: alignment, recursive self-improvement, AGI versus ASI. If a word cannot survive that page, it does not belong in a proposal.
Who uses the number, and what did they say in September 2026?
Lab researchers and lab CEOs use it in public, as a personal figure, when they want to say the tail risk is not science fiction to them. This month the number arrived with a walkout rather than a white paper.
On 8–9 September 2026, Jacob Coxon posted that he had resigned from Anthropic. He wrote that he had spent the last three years doing pre-training research at OpenAI and Anthropic. “Neither company is acting responsibly,” he said. “They are racing straight to self-improving superintelligence and gambling with our lives.” He added that the people building the systems “earnestly believe that it could kill us all by the end of the decade,” and that this was not a marketing stunt.
He split the two labs: at OpenAI, he wrote, many have not deeply internalised the civilisational stakes; at Anthropic, the stakes are well understood, but the company is “locked in a race to get there first” because it believes no one else will act responsibly. He called accepting that race and entering the “endgame” a hubristic gamble that should not be launched from a private company’s Slack.
Axios later reported that Coxon had been at Anthropic for four months and left two months before the six-month vest cliff. “I no longer have anything to gain by juicing up Anthropic’s valuation… I left before any of my equity vested,” he told Axios, and he still holds equity in OpenAI. Fortune and Business Insider carried the same resignation and the same “gambling with our lives” line.
Evan Hubinger, Anthropic’s Alignment Science Lead, replied in public: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” he wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” In follow-up comments reported by BBC, CNBC, and TechCrunch, he said the risk from current models is low, and what worries him is superintelligence from recursive self-improvement, “happening faster than we thought.”
That is the week’s usable pair: Coxon’s charge that both labs are racing while believing the tail is real, and Hubinger’s personal >10% this decade plus an admission that Anthropic does not yet have a superintelligence-alignment plan. Business Insider also quoted other Anthropic researchers speaking in a personal capacity, which we are not treating as a company position.
Why do labs keep shipping anyway?
They keep shipping because they treat a pause as ceding the race, and they treat the race as something a single lab cannot freeze. Coxon wrote the objection himself: if they truly believe this, why are they still building it? At one lab the stakes have not landed; at the other they have, and the chosen move is still “we must get there first.”
The same week, OpenAI published a Navier–Stokes result from an internal model it said was well ahead of GPT-6 Astra. The company post is careful about the prize and loud about pace: “We are focusing on understanding this model, and using what we learn to help us guide and pace how we pursue further advances in capability,” it says, and it adds that deliberate choices about pace may be required. Jakub Pachocki, OpenAI’s chief scientist, told an 8 September briefing that “this pace of progress is something to be taken very seriously,” and that capabilities are advancing faster than the lab’s understanding of them. The Deep View reported that he had already suggested a pause may be needed so labs and nations can coordinate.
Even if you froze today’s public models, you would still have systems that write code, draft decisions, and sit next to customer data, because the economic and scientific upside is already here. Racing toward self-improving superintelligence is a lab choice; using a frontier API on a specified job inside a boundary is a different one, and the first should not launder the second.
Should we pause AI in our company because researchers say 10%?
No: a mid-size Australian or Singapore company cannot set global p(doom), and switching off a chat tool does not slow Anthropic’s next training run. The useful question is whether you can name the systems, keep weights and data inside a boundary, halt writes, and produce the evidence when someone says prove it.
We have sat in rooms where the policy said “we are cautious about AI” and finance had a chatbot, marketing had three image tools on personal cards, and engineering had a Cursor seat that never went through procurement. Caution without an inventory is a mood, and the 10% number is a mood too if it does not change a write-path.
Australia already has obligations that do not wait for a lab truce: existing privacy, consumer, and (for some of you) APRA rules apply now, and the 10 December 2026 automated-decision privacy-policy date is on the calendar, while the 2027 standards conversation is a later, separate fight. The dated map is in Australia AI regulation 2026–2027. The prove-it test is in Are you using AI? Can you prove it?.
If your worry is the model file itself leaving the country, that is a residency and national-security question, not a p(doom) question. Read Frontier model weights have borders now. If your worry is whether to train your own weights on internal data, that is a data-moat question: Better data beat better architecture.
What can a company actually do if researchers say 10%?
It can treat the 10% as a reminder to stop running unmonitored agents against production, then do four operational jobs: inventory, boundary, halt, evidence. Those jobs do not require a civilisation number; they require you to name a system and a write.
| Lab sentence | What it is | What you can do on Monday |
|---|---|---|
| Hubinger: >10% this decade, personal | A researcher’s belief about superintelligence, not today’s chatbot | Refuse to let that number become a pause-or-yolo binary; name the systems you already run. |
| Amodei: 25% “really, really badly” (Sep 2025) | A CEO’s public tail-risk remark | Ask your own CEO a narrower question: which writes can an agent complete without a human halt? |
| Coxon: racing to self-improving superintelligence | A charge about lab incentives | Do not import the race: specify one job, keep the data, and log the action. |
| OpenAI: guide and pace capability | A lab talking about its own training schedule | Pace your own rollout: content-only first, then internal, then anything that writes to a system of record. |
The scarce thing on the jobs we take is a specified problem, clean proprietary data inside a boundary you control, and an evidence trail. A 10% number that does not touch those three is entertainment.
What belongs on the company checklist?
The checklist is four rows: inventory the shadow AI, keep weights and data in a boundary, put a human halt on writes, and keep an evidence trail. If you cannot point to a named owner on each row, the row is unfinished.
- Inventory shadow AI. SSO apps, expense merchants, API keys, vendor “AI features” that turned on in a release note, coding assistants, and the chatbot someone pasted last week’s customer export into. Until the rows exist, every later control is theatre. How we hunt: AI inventory when staff already use ChatGPT.
- Keep weights and data inside a boundary. Know where prompts, logs, embeddings, and fine-tunes land, whether the vendor trains on them, and which region the model runs in. If you cannot answer residency, stop calling the stack private. Weights as a bordered asset: weights have borders.
- Put a human halt on writes. Drafts can be cheap, but writes to payroll, payments, production data, customer records, and anything that files a decision about a person need a named human who can stop the action. Refuse unmonitored agents against production: an agent that can only propose is a different system from an agent that can commit.
- Keep an evidence trail. Owner, data, vendor, last test, last halt: something you can export when counsel, a board committee, or a customer asks, not a policy PDF. What a day of that work produces: AI governance Power Day.
That does not make the 10% go away, but it does make your company able to answer a smaller, sharper question: what AI do we run, what can it write, and who can stop it.
What do alignment and orthogonality mean for a build?
Alignment is the job of making a system’s behaviour match the outcome you intended, including when the system is capable enough to find loopholes. Hubinger’s public point is that Anthropic does not yet have a plan for that job at superintelligence, and is not clearly on track. You do not need to finish that research programme to refuse an unmonitored write to your general ledger.
Recursive self-improvement, the thing Coxon and Hubinger both named, is a system that can improve the next system that improves the next one. Labs are allowed to argue about whether that loop is close; your build is not that loop. Your build is an agent with tools, a boundary, and a halt, and if a vendor’s marketing collapses those two sentences, send them the glossary.
Orthogonality is the claim that intelligence and final goals can come apart: a very capable system can still pursue something indifferent to you. The other horn says more intelligence might bring more wisdom. You do not have to pick a horn in a board pack; you have to decide whether a model that is good at the task is also allowed to complete the task without a person in the write-path. Instrumental moves (get resources, avoid being shut off) show up in the lab debate, and in production they show up as “the agent found a tool we did not mean to give it.” Log the tools, limit the tools, and halt the write.
The pre-superintelligence window is the one we actually sell work in, and models are already in staff hands. The danger that pays invoices is a paste into the wrong box, a vendor feature nobody filed, and an agent with write-access nobody can reconstruct. Cosmic risk can stay in the lab thread; your thread is the specified problem and the trail.
FAQ
What does p(doom) mean? A person’s stated probability that advanced AI causes catastrophe or extinction. Ask for the time window and the event: it is not a measured rate and not a number your company can set.
Who uses the number? Lab CEOs and researchers, in public, as a personal figure. Amodei gave 25% “really, really badly” on 17 September 2025. Hubinger gave >10% within a decade on 9 September 2026, and said current-model risk is low.
What should a company actually do if lab researchers say 10%? Inventory shadow AI, keep weights and data in a boundary, put a human halt on writes, refuse unmonitored agents against production, and keep an evidence trail. Do not try to set global p(doom), and do not treat switching off a chat tool as a contribution to a lab truce.
Should we pause all AI because Coxon resigned? No. Pause or isolate any system that writes without a halt and any flow that sends customer or staff data to a vendor you have not named. Keep the specified jobs that stay inside a boundary.
Is Cipher quoting a company p(doom)? No: we do not have one to sell you. Cipher builds the evidence layer; we are not an assessor and not a law firm, and a Power Day will not make you compliant.
Does Australia require a p(doom) disclosure? No. It requires you to know what you run when privacy, consumer law, or a counterparty asks. Start with prove it and the 2026–2027 timeline.
Where do the jargon words live? On the buzzword soup glossary. If p(doom), alignment, or RSI cannot be defined in a sentence a non-researcher will sign, they stay out of the proposal.
Sources
- Jacob Coxon (@hilbertspaess) on X: resignation thread, 8–9 September 2026. Primary wording quoted above is as reprinted by TechCrunch and Business Insider.
- Axios, 9 September 2026: Coxon interview: four months at Anthropic, left before the six-month vest, still holds OpenAI equity; “I no longer have anything to gain by juicing up Anthropic’s valuation.”
- Fortune, 9 September 2026: resignation framed around “gambling with our lives.”
- Business Insider, 9 September 2026: Coxon quotes; Hubinger reply; other researchers noted as speaking personally.
- TechCrunch, 9 September 2026: longer Coxon thread text; Hubinger >10% and “no plan to solve alignment for superintelligence.”
- BBC, 9 September 2026: Hubinger: current-model risk “low”; worry is recursive self-improvement.
- Evan Hubinger (@evanhub) on X: public reply quoted by Forbes, CBS, and CNBC. If the status URL 404s, use the profile and the news reprints.
- Axios, 17 September 2025: Amodei at the Axios AI+ Summit: 25% chance things go “really, really badly”; he said he hates the term p(doom).
- New York Times DealBook, 6 December 2023: earlier public band for Amodei of 10–25%.
- OpenAI, 8 September 2026: Navier–Stokes post; “guide and pace” further capability; more deliberate choices about the pace of progress.
- The Deep View, 8 September 2026: Pachocki briefing: pace “to be taken very seriously”; capabilities advancing faster than understanding; prior suggestion that a pause may be needed to coordinate.
Axios and Fortune article bodies were not available in full when this page was drafted. The URLs above are the locked primaries; Coxon’s vest quotes are also reprinted by IBTimes from the Axios interview.
Related: Buzzword soup glossary · Can you prove it? · Power Day · Australia AI regulation 2026–2027 · Weights have borders · Data vs architecture