The Worried Keep Building
The irony of a safety built AGI
A fact that should bother you more than it does: nearly every major lab building frontier AI was originally founded explicitly to make sure it would be safe.
OpenAI’s charter commits to ensuring artificial general intelligence “benefits all of humanity.” Its founding announcement in 2015 positioned the organization as a counterweight to concentrated AI power. Greg Brockman’s early writing on defining his role framed it as a mission-driven technical leadership challenge. Their alignment team launched with a public letter declaring safety research a core priority.
Anthropic’s founders, many of whom left OpenAI over concerns about its direction, published Core Views on AI Safety as a kind of founding manifesto.
Excited to announce what we’ve been working on this year - @AnthropicAI, an AI safety and research company. If you’d like to help us combine safety research with scaling ML models while thinking about societal impacts, check out our careers page anthropic.com/#careers
— Daniela Amodei (@DanielaAmodei), 28 May 2021
These were statements of genuine alarm.
And these alarmed people went and built the most powerful AI systems in the world.
The pattern repeats with consistency. Someone gets close enough to the frontier to see what’s coming. They get scared. They worry that if they abdicate their position, it the vacuum could be filled by a bad actor. So, admirably, they decide the responsible thing to do is not to walk away but to stay close, to steer.
And in trying to steer, they end up illuminating new capabilities and ultimately push the frontier further.
The history begins, of course, with OpenAI. When OpenAI shifted toward a capped-profit structure and started prioritizing commercialization, key figures didn’t just object quietly:
Ilya Sutskever, co-founder and chief scientist, left after a period of growing tension over the lab’s safety commitments.
Jan Leike, who co-led the superalignment team, departed citing what he described as safety and preparedness taking “a backseat to shiny products.”
John Schulman, a co-founder, joined Anthropic.
They didn’t leave to retire. They left to build somewhere else.
Anthropic itself was born from researchers who believed OpenAI was moving too fast, with too few guardrails.
Anthropic now raises billions, ships frontier models, and finds itself facing its own governance questions from the very community that supported its founding.
Now we see Goodfire emerging with mechanistic interpretability at its core — an even more focused bet that safety research is the product.
(To say nothing of the dozens of neo-labs that have sprung up)
You can read this as hypocrisy or a lack of moral compass but I think that’s clearly wrong.
What’s actually happening is something more uncomfortable: concern is the engine.
The people who build the most capable systems are overwhelmingly the people who started out most worried about capable systems.
Worry is what drove them to understand the technology deeply enough to advance it. Fear of what AI could become is precisely what motivated them to get close enough to shape it. And getting close enough to shape it meant getting close enough to make it better.
This is the builder’s paradox. You cannot steer what you don’t build. You cannot govern what you don’t understand. And you cannot understand a frontier technology without pushing it forward.
The only people qualified to make AI safe are the people capable of making it dangerous.
In previous posts, I’ve argued that the system that’s actually misaligned is capitalism — that most of what we fear from AI isn’t the technology but the economic logic deploying it.
The builder’s paradox is how that misalignment gets its workforce.
Concerned, brilliant people show up to steer. Capital shows up to scale.
Concerned people provide the breakthroughs. Capital provides the incentive structure.
Suddenly the safety-motivated lab is in an arms race it never intended to join, because the other lab — also founded by worried people — just shipped something better.
Nobody planned the arms race. The arms race emerged from the convergence of genuine concern and financial opportunity.
The major defections from one lab to another were largely motivated by safety. Every new lab was founded to do things more carefully.
And every one of them accelerated the very trajectory they were trying to slow.
There’s a version of this story that’s tragic: they saw what was coming and couldn’t stop it, only hasten it.
But I think there’s a more honest reading. The worried keep building because the alternative - stepping back, letting less careful actors lead - is worse. This isn’t rationalization. It’s triage. If AGI is inevitable (and at this point, it’s hard to argue otherwise), then the question isn’t whether it gets built but by whom and under what constraints and motivations.
The worried builders aren’t solving that question perfectly, but the counterfactual is worse.
A world where the safety-conscious researchers withdrew and left the frontier to pure commercial actors isn’t a world with less powerful AI. It’s a world with less careful AI.
The worried keep building because they understand something the rest of us are still catching up to: this technology is not going to wait for us to be ready. In a previous post, I described how we’re living through a kind of digital urbanization, building cities before we’ve figured out sanitation. The worried builders aren’t solving the sanitation problem. But they’re the ones who know there is a sanitation problem, and they’re trying to lay pipe even as the city expands around them.
Their instinct is to stay close, to keep their hands on the wheel, to build from a position of concern.
Whether that’s enough is genuinely an open question.
Originally published on Griff’s Substack on 20 February 2026.