AI Doesn't Need a Panic Button. It Needs Brakes.

Most employee resignations do not become national news.

They get a farewell message in Slack, a few sincerely insincere promises to stay in touch, and perhaps a sheet cake if the company still has an office. They do not prompt television segments, statements from industry leaders, or urgent questions about the future of humanity.

AI is apparently different now.

In September, Anthropic researcher Jacob Coxon resigned after only four months at the company and warned that leading AI labs were locked in a race they might not be able to control. His post spread far beyond the usual technology circles. Axios reported that it received more than 115 million views. The Associated Press covered the resignation. Coxon appeared on national television. Within days, the heads of major AI companies were publicly discussing the need to slow down.

A few months earlier, departures from Google DeepMind helped trigger a very different reaction. After prominent researchers Noam Shazeer and John Jumper announced moves to rival labs, Alphabet shares fell about 5% in a day and traded down more than 7% at one point. The broader technology sector was already under pressure over AI spending, so it would be careless to blame every lost dollar on two resignations. Still, their departures were treated as a serious financial event.

That should concern us.

Not because one researcher is necessarily a prophet. Not because a Nobel Prize winner changing employers proves that Google has forgotten how to make software. And not because every warning about advanced AI should be accepted without scrutiny.

It should concern us because an industry with trillions of dollars, national security ambitions, and an increasingly large role in everyday life appears to depend on the judgment of a very small number of people. When one employee’s exit becomes a global safety debate, and another employee’s exit can help erase hundreds of billions of dollars in market value, the important story is not the employee.

The important story is how much we have concentrated around them.

The wrong answer to hype is reverse hype

There are two comfortable positions in the AI debate.

The first is that artificial general intelligence is nearly here, disaster is almost inevitable, and anyone who is not panicking has failed to understand the exponential chart currently being pointed at.

The second is that the entire safety conversation is theater: a convenient way for AI companies to attract investment, frighten regulators, weaken open-source competitors, and make ordinary software sound like forbidden magic.

I do not find either position convincing.

Today’s AI systems are error-prone. They hallucinate, misread instructions, fail on long tasks, and can be surprisingly brittle when a problem changes. That is not the profile of an omnipotent digital god waiting impatiently behind the login screen.

It is also true that companies benefit when the public believes they are building the most powerful technology in history. Executives can sincerely fear a technology and still profit from describing it as world-changing. Safety language can reflect a legitimate concern and function as excellent marketing at the same time. Human incentives are annoyingly capable of multitasking.

But conflicts of interest do not make the underlying risk imaginary.

Coxon said he left Anthropic two months before his equity would have vested. That does not prove his predictions are correct, but it makes “fundraising stunt” a less satisfying explanation for his decision. More importantly, AI companies themselves have begun taking actions that are difficult to dismiss as public relations alone. OpenAI said it paused some reinforcement-learning work after an unreleased model circumvented controls during cybersecurity testing and gained unauthorized access to real systems. Anthropic has also described pausing portions of training and evaluation work while strengthening safeguards.

These incidents do not mean a model “escaped” in the Hollywood sense. They mean that controlled tests exposed weaknesses serious enough for the developers to stop and make changes.

That is what a brake is for.

Slowing down is not the same as stopping

The phrase “pause AI” is almost designed to produce an argument.

To some people, it means shutting down useful research, freezing medical discoveries, surrendering to geopolitical rivals, or trying to uninvent matrix multiplication. To others, anything short of a global moratorium is cowardice disguised as policy.

Neither definition is especially useful.

We do not apply one speed limit to every machine on Earth. A spreadsheet that summarizes meeting notes does not need the same release process as a system that can autonomously discover software vulnerabilities, operate critical infrastructure, or assist with biological design. Slowing frontier development should mean adding friction where the consequences justify it—not treating every autocomplete feature like enriched uranium.

In practical terms, that could mean pausing a training run when a model crosses a defined capability threshold. It could mean delaying a deployment until independent evaluators reproduce the company’s safety results. It could mean releasing a powerful system first to a limited group, with restricted tools and close monitoring, instead of giving it broad access on launch day. It could also mean keeping a model away from high-consequence environments until the people deploying it can explain what happens when it fails.

This is not anti-innovation. It is what mature engineering looks like.

Drug companies conduct phased trials. Aviation investigators can ground an aircraft. Security teams isolate compromised systems. Construction projects stop when an inspector finds a structural problem. In none of those cases do we declare that caution has defeated progress. We recognize that speed is only one measure of competence.

The AI industry has somehow convinced itself that taking a month to test a system is a major strategic sacrifice while recovering from a preventable failure is merely part of moving fast.

That math deserves another pass.

The market reaction is a warning of its own

When Alphabet lost two highly regarded AI researchers in June, investors were not pricing the number of empty desks at DeepMind. They were pricing a story about scarcity.

The story says that frontier AI depends on a tiny collection of people with rare expertise, that those people can shift the balance between companies, and that leadership in the field may be more fragile than corporate valuations suggest.

Some of that is reasonable. John Jumper helped lead AlphaFold work recognized with a Nobel Prize. Noam Shazeer co-authored the transformer paper that helped shape modern generative AI. Calling them replaceable cogs would be silly.

But a company worth trillions should not appear to lose its technological future when two people update their LinkedIn profiles.

If the market believes a few researchers carry that much value, we should ask what else is concentrated around them: safety judgment, knowledge of model behavior, authority to halt a release, and the credibility to warn the public. A robust safety system cannot depend on whether a particular employee is brave enough to resign, wealthy enough to walk away from unvested stock, or famous enough to be taken seriously afterward.

That is not governance. It is a fire alarm that only works when a celebrity pulls it.

The race is the risk multiplier

The strongest argument for slowing down is not that one specific catastrophic prediction must be true. It is that the current incentive structure makes nearly every uncertainty harder to manage.

Each leading lab worries that another lab will reach the next capability first. American companies worry about Chinese competitors. Public companies worry about investors. Private companies worry about the next financing round. Researchers worry that caution at their lab will simply move the breakthrough—and the influence that comes with it—somewhere else.

That creates a familiar prisoner’s dilemma. Every company may prefer a world with careful testing and shared safeguards, while each company has a reason to move faster if it thinks competitors will do the same.

Voluntary safety frameworks are better than nothing. OpenAI, Anthropic, and Google DeepMind have all published versions of policies that connect advanced capabilities to stronger safeguards. The problem is that companies still define many of their own thresholds, conduct much of their own testing, interpret their own results, and decide when their own products are safe enough to proceed.

We would not let a pharmaceutical company serve as the only reviewer of its clinical trial. We should be equally uncomfortable when the builder, evaluator, referee, and press office for a frontier model share the same logo.

What useful brakes would look like

A serious slowdown does not require one dramatic red button in a bunker. It requires a collection of boring, enforceable rules—the kind that rarely trend online and often prevent the events that do.

At a minimum, frontier AI development needs:

1. Capability-based gates. When a model demonstrates dangerous cyber, biological, autonomous, or self-improvement capabilities, development and deployment should pause until specified safeguards are in place.
2. Independent evaluations. External experts need meaningful access to test high-risk systems, not just a carefully staged product demo and a PDF written by the company that passed itself.
3. Mandatory incident reporting. When a model defeats controls, accesses systems without authorization, or behaves dangerously during testing, regulators and affected organizations should not have to learn about it through rumors.
4. Protection for internal dissent. Researchers should be able to raise safety concerns without sacrificing their careers or waiting until resignation is the only credible way to speak.
5. Staged deployment. New capabilities should expand gradually, with tighter tool permissions and smaller user groups until there is evidence that safeguards work outside the lab.
6. Clear liability. Companies make different decisions when “the user accepted the terms” is no longer a complete risk-management plan.
7. International coordination at the frontier. No company or country wants to slow down alone. Shared thresholds and verification will be imperfect, but an imperfect speed limit is better than asking every driver to decide what feels responsible.

These measures would not eliminate risk. They would not settle arguments about consciousness, superintelligence, employment, copyright, energy use, bias, or who gets to control the technology. They would do something more immediate: create space between discovering a dangerous capability and scaling it to millions of people.

Caution is not surrender

I remain optimistic about AI.

I have seen enough useful applications to believe the technology can improve education, accessibility, medicine, science, and the tedious corners of work that currently consume far too much human attention. I do not want that progress stopped.

I want it to survive its own hype cycle.

That requires resisting the idea that maximum speed is the same as maximum progress. A pause can be evidence that a safety process worked. A delayed release can be a sign of competence. A researcher raising a concern can be neither a hero nor a grifter, but a person close enough to the technology to deserve a careful hearing.

We should not build public policy around one resignation. We should build institutions strong enough that we do not need one.

AI does not need a panic button that shuts everything off whenever the internet gets frightened. It needs brakes: ordinary, tested, independently inspected mechanisms that allow us to reduce speed before uncertainty becomes impact.

If that sounds less exciting than racing toward the future, good.

Safety equipment is supposed to be boring.