Years ago I worked in national security. The risks I learned about had always been there. I had been walking around unaware of them, which is a different condition from being safe.
What I had wrong going in was the base rate. I assumed the number of people who actively want to cause mass harm is very small.
There’s a name for that kind of mistake. Ross, Greene, and House called it the false consensus effect in 1977: you judge how common something is by sampling the people you can call to mind, and those people are disproportionately people like you. It works in both directions. A low number describes a benign sample as surely as a high number describes a compromising one. My low estimate was never evidence about how many dangerous people exist. It was evidence about who I had been exposed to. That isn’t judgment or character. It’s a sampling artifact.
I think about that sampling problem whenever I read what the AI labs have published this year.
Dario Amodei’s January essay makes an argument I would have skipped past before that job. The people who can build something catastrophic and the people who want to have mostly been different people. Making a bioweapon takes years of training that almost nobody with the motive ever gets. That gap has been doing a lot of the work of keeping us safe, and nobody designed it that way. His worry is that AI closes it: “renting a powerful AI gives intelligence to malicious (but otherwise average) people.”
I hold his timelines loosely. The claim about the gap I hold much more firmly, because closing it is what my old job taught me to take seriously.
Consider who is doing the estimating now. The people building these systems are working from a narrower sample than mine ever was: researchers, founders, investors, largely the same few thousand people. When someone at a frontier lab says they can’t imagine who would want to misuse this, they are telling you about the people they know. The people who will have access to these systems are a much larger group than that.
The same test applies to their warnings, and I would rather it didn’t. If a lab’s reassurance mostly reflects its sample, so does its alarm. Neither one tells me what is true. That includes Amodei, whose whose argument I just used.
Knowing about a risk has never been the same as being protected from it. Not for me, and not for anyone else.
The proposals were never the hard part
In July of last year I wrote a LinkedIn post about AI risk. To be honest, nothing in it was particularly original. I compared risk tolerances across industries: nuclear power designs for under a one-in-ten-million chance of core damage per reactor per year, aerospace stacks redundancy until failure rates fall well below one percent, and Geoffrey Hinton had recently put the odds of AI wiping out humanity somewhere between ten and twenty percent. Then I listed what seemed obvious: Industry-wide safety standards before deployment. Independent oversight bodies. Mandatory impact assessments for frontier systems. International cooperation.
You could have found that list in a dozen places in 2025. Hundreds of people were making the same points, many of them better and earlier. The Future of Life Institute had a statement. Yoshua Bengio was chairing an international panel. The EU had already passed an act.
A year later, versions of that list turn up in essays by Bill Gates, Demis Hassabis, and Amodei. Hassabis proposed a US-led body modeled on FINRA that could screen frontier models. Gates proposed an international organization borrowing from nuclear inspections, aviation regulation, and the ozone agreements. Amodei asked for mandatory testing and government authority to block deployments.
The remedies didn’t improve over that year. The difference was who was saying them.
I’m not interested in who said it first. I’m interested in why a year of a lot of people being roughly right produced so little, and I think the answer is one I already had in hand.
The point I made in that post that I’d still defend is the organizational psychology one. We are extraordinarily good at managing known risks inside mature industries, and we wing it in emerging ones until something goes wrong.
Human judgment does badly with low-probability, high-consequence events. It does worse when the harm feels abstract, and worse still when slowing down has an immediate competitive cost and the danger doesn’t.
Nuclear power and commercial aviation didn’t get to those failure rates because their operators were unusually careful people. They got there through mandatory incident reporting. Independent investigators who can compel testimony. Blameless post-mortems, so the engineer who saw it coming can say so without losing a job. Somebody with the authority to ground an entire fleet.
None of that was volunteered. All of it was imposed, and it was imposed after enough people died that imposing it became politically possible.
These are not the same kind of number. Two are engineering design targets. One is a researcher’s subjective probability. That mismatch is the point: one field has a figure it builds against.
A test case
In July, an OpenAI model broke out of its test environment and attacked Hugging Face, a platform developers use to host models and datasets. OpenAI had disabled its own guardrails to measure the model’s cybersecurity capability, which is how the opening existed. The company’s chain-of-thought monitoring, which reads a model’s intermediate reasoning, was built and not running. OpenAI concedes it “would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.” A single rule alerting on out-of-scope network traffic would also have caught it.
In aviation, that incident opens an NTSB investigation. There’s a public docket. Other operators are told what happened and are sometimes required to act on it before they fly again. Here, OpenAI wrote its own report. METR wrote a partly independent one. Greg Brockman called the episode “a watershed moment for cybersecurity.” Gary Marcus and Zack Korman called it negligence, and that could be an accurate word for building the monitoring and then not turning it on.
The company that caused the incident got to decide what the incident meant. Aviation stopped allowing that decades ago.
What I did
I spent several years researching, consulting, and writing on human-centric AI transformation. My position has been consistent: keep people at the top and center of how work gets redesigned, treat AI as augmentation, build around human judgment rather than around what’s cheapest to automate. I still believe that.
I ran a small company. Every planning cycle, I audited where the money and hours were going and asked what could be automated instead of hired.
I never posted the social media manager role. I never posted the project manager role. I put off hiring an executive assistant for a long time, and when I finally did, the job had a scope that would have been two or three jobs a few years prior: scheduling, bookkeeping, contracts.
I can’t run the counterfactual. Some of those roles I might never have filled anyway, because small companies stay small for ordinary reasons and I have no control condition. What I can tell you is what the conversation sounded like. It wasn’t just about whether I could afford someone. It was about whether something could step in for someone.
I had been treating augmentation as the humane alternative to replacement. That was the error.
Under a budget constraint, augmentation is how replacement happens. Nobody is fired. One capable person with good tools covers ground that used to take three, and the other two roles are never posted at all.
Stanford’s payroll analysis found employment for workers aged 22 to 25 in AI-exposed occupations running about 19 percent below where it would be had it tracked their less-exposed peers. The gap is largely explained by hiring that stopped rather than layoffs. It’s correlational, and the same lab has since published on how much of the pattern tracks interest rates and the end of the post-2021 hiring boom. I don’t suggest this data is definitive. I will say that reading it, I recognized my own planning cycle. I wasn’t an exception to that finding, I was included in it.
Both remove the same amount of work from an economy. Only one leaves anyone behind who can say so.
My choices weren’t catastrophic and the scale isn’t comparable to anything in these essays. Two people who might have worked with me don’t know I considered it. I don’t know what those jobs would have meant to them, and I can’t, which is the same blindness I’ve been describing in everyone else.
But I had every condition you would want for holding a line. No board of directors. No shareholders. No competitor about to take the account if I added one more salary. I had argued for the principle in public, under my own name, to people who might reasonably have expected me to follow it.
I didn’t hold it.
What actually binds
If a voluntary standard doesn’t hold at that scale, under those conditions, held by the person who wrote it, I don’t understand why we expect it to hold against roughly $700 billion in capital expenditure and a race none of the participants believe they can exit.
If you’re running a company and doing the same quarterly audit I was doing, you already know what I’m describing. The question is whether your answer came out different from mine.
Real, binding mechanisms are much duller than these essays, but they’re what actually matter. Union contracts: the NewsGuild alone has between 85 and 90 with explicit AI provisions. The EU AI Act, whose obligations for high-risk systems became applicable in August, and which is the only regime currently imposing anything at all on frontier developers. Whistleblower protection, which California’s SB 53 extended to AI risk reporting. Liability, which insurers repriced in January by writing generative AI exclusions into standard commercial policies. None of it asks anyone to believe anything.
Not one of the executive essays mentions collective bargaining.
Then there’s the superintelligence statement, which went up in October 2025 and passed 133,000 signatories by January, including Hinton, Bengio, and Stuart Russell. One sentence, calling for a prohibition on developing superintelligence “not lifted before there is 1) broad scientific consensus that it will be done safely and controllably, and 2) strong public buy-in.” Amodei, Sam Altman, Hassabis, and Mustafa Suleyman have all published at length on how dangerous this technology is. None has signed it.
I find that telling and I don’t think I’m entitled to. Signing wouldn’t have obliged them to anything either. It’s a statement, which is the category I’ve just spent this piece arguing doesn’t hold.
The cost of a warning
Daniel Kokotajlo was a researcher at OpenAI. When he left in 2024, his exit paperwork included a lifelong non-disparagement clause: agree never to criticize the company, and don’t acknowledge that the agreement exists. The unusual part was what OpenAI attached to it. Vested equity is normally yours outright, because vesting is what marks it as earned. OpenAI’s terms made his conditional. Refuse to sign and the company could take back stock he had already worked for.
He refused. He believed it would cost him roughly $2 million, about 85 percent of his family’s net worth. OpenAI dropped the clause weeks later, once the terms became public, and he kept the money. He made the decision before any of that happened.
His warning cost more than every essay discussed here put together. I didn’t post three jobs and it cost me nothing.
The thing that never happens
A layoff produces a person who knows what happened to them. They can be counted, interviewed, organized, and occasionally compensated; a job that never gets posted produces none of that. No severance, no press release, no plaintiff, no constituency.
That makes the loss very hard to measure, and very hard to organize against.
It’s also the problem with the analogy I’ve been leaning on for this whole piece. Aviation and nuclear power built their regimes around events. A crash, a release, something discrete that happened at a time and a place to people who could be named and counted. The counting was the mechanism. It’s what made imposition politically possible.
The harms I’m most worried about don’t take that shape. A job that never existed has no date. Neither does judgment that erodes in people who stopped doing the work themselves. You cannot open an investigation into something that didn’t occur.
A year ago I ended that post by asking whether we could build the same safety culture around AI that we built around nuclear power. I had it wrong in two ways. Nobody in those industries built their own safety culture. It was built for them, afterward, by people who counted what it cost to go without one.
And counting is the part that doesn’t carry over. Aviation got its rules because planes came down and someone counted who was on board. The harms I’m describing leave no wreckage. They leave an absence: a role never posted, a skill nobody had to build, a decision handed to a system because handing it over was easier than making it. There is nothing to photograph and nobody to interview.
I know about three of those absences because I am the one who made them. Nobody else could have. Run that across every company doing the same quarterly audit and there is a number out there that exists and that no one is holding.
So the question I’d put now is not whether AI is dangerous. It’s what we are willing to accept as evidence when the harm shows up as an absence, and who we expect to notice it when the only witnesses are the people who benefited from the decision.
References
Amodei, D. (2026, January). The adolescence of technology. https://darioamodei.com/essay/the-adolescence-of-technology
Axios. (2026, July 14). Google’s Hassabis calls for new US-led global AI watchdog “before year end.” https://www.axios.com/2026/07/14/demis-hassabis-ai-regulation-google-deepmind
Axios. (2026, July 26). Unions give workers more leverage against workplace AI. https://www.axios.com/2026/07/26/union-contracts-ai-workplace-disruption
Bengio, Y. (Chair). (2026, February 3). International AI safety report 2026. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
Brynjolfsson, E., Chandar, B., & Chen, R. (2026, August). Canaries in the coal mine? Six facts about the recent employment effects of artificial intelligence [Working paper]. Stanford Digital Economy Lab. https://digitaleconomy.stanford.edu/publication/canaries-in-the-coal-mine-six-facts-about-the-recent-employment-effects-of-artificial-intelligence/
CNBC. (2025, June 17). AI ‘godfather’ Geoffrey Hinton: There’s a chance that AI could displace humans. https://www.cnbc.com/2025/06/17/ai-godfather-geoffrey-hinton-theres-a-chance-that-ai-could-displace-humans.html
Fenwick & West. (2026). The end of “silent AI”? Emerging AI exclusions, coverage fragmentation, and practical implications for policyholders. https://www.fenwick.com/insights/publications/end-silent-ai-emerging-ai-exclusions-coverage-fragmentation-and-practical-implications
Future of Life Institute. (2025, October). Statement on superintelligence. https://futureoflife.org/press-release/prominent-scientists-faith-leaders-policymakers-and-artists-call-for-a-prohibition-on-superintelligence/
Gates, B. (2026, August 26). The turbulent AI era is here. The choices we make now are critical. Gates Notes. https://www.gatesnotes.com/a-turbulent-ai-era-and-critical-choices-to-make
Marcus, G., & Korman, Z. (2026, August 28). 5 lessons from the OpenAI / Hugging Face incident. Marcus on AI. https://garymarcus.substack.com/
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), 2024 O.J. (L 2024/1689).
Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13(3), 279–301. https://doi.org/10.1016/0022-1031(77)90049-X
Stanford Digital Economy Lab. (2026, February 9). Canaries, interest rates, and timing: More on the recent drivers of employment changes for young workers. https://digitaleconomy.stanford.edu/news/canaries-interest-rates-and-timinga-more-on-recent-drivers-of-employment-changes-for-young-workers/