Trump Killed Biden's AI Safety Framework. Then He Built the Same Thing and Called It Something Else.
What happened
The Center for AI Standards and Innovation, a division of NIST within the Commerce Department, announced agreements with Google DeepMind, Microsoft, and Musk's xAI to conduct pre-deployment safety evaluations of their frontier AI models. The agreements allow CAISI to test models in classified environments before public release. This follows earlier CAISI partnerships with OpenAI and Anthropic. In January 2025, Trump disbanded Biden's AI Safety Institute and eliminated its voluntary safety commitments from tech companies, framing them as overregulation blocking innovation. The Ars Technica report links the reversal to the 'Mythos' incident, an unreported near-miss involving a frontier AI model that alarmed officials, though details remain classified.
Trump dismantled AI safety oversight for political reasons, then rebuilt it for political reasons, and the new version is structurally weaker than the one he tore down because it lacks the institutional history and independence the first version was accumulating.
Prediction Markets
Prices as of 2026-05-07 — the analysis was written against these odds
The Hidden Bet
CAISI is functionally equivalent to the Biden AI Safety Institute
The Biden AISI had a mandate that included publishing evaluations and informing public discourse. CAISI's new agreements explicitly allow testing in classified environments, with results presumably not disclosed. This turns public safety testing into a government intelligence function. Companies participating can claim compliance with safety norms while the results remain secret. That is not the same thing.
Voluntary agreements with tech companies are meaningful safety constraints
Google, Microsoft, and xAI agreed to let the government test their models. They did not agree to delay release if tests reveal problems. The agreements are about access, not enforcement. A company that runs the tests, finds a problem, and releases anyway faces no legal consequence. The entire framework rests on voluntary compliance from companies whose competitive incentive is to release first.
The Mythos incident is the real reason for the reversal
CAISI's agreements predate any public Mythos reporting. The more likely explanation is that the White House, having eliminated safety testing, needed a way to claim it had safety mechanisms in place without creating binding regulation. CAISI gives them that claim. Mythos provided the political cover to announce it without being seen as reversing course.
The Real Disagreement
The real tension is between two positions that are both defensible. The first: some government pre-deployment review is better than none, even if imperfect, because it creates at least a record of what risks were known and when. The second: voluntary classified testing with no enforcement creates the appearance of safety oversight without the substance, and that appearance may be more dangerous than no testing at all because it reduces pressure for binding regulation. Leans toward the second, with a caveat: classified access to models before release does give the government the ability to respond to emergent risks it would otherwise discover only after deployment. That is not nothing. But it is not safety governance.
What No One Is Saying
xAI's inclusion in these agreements is the most important detail being underreported. Elon Musk's company, whose founder is simultaneously running a government efficiency office with access to federal systems, is now also submitting its AI models to a government agency for pre-release evaluation. The potential for regulatory capture in both directions, the government influencing xAI's model development, and xAI influencing what the government considers safe, is significant and not being examined.
Who Pays
Open-source AI developers and smaller labs
Immediate structural effect
CAISI agreements are with the largest players. Smaller labs and open-source projects are not included, not because they are safer but because they cannot afford the compliance process. The framework effectively creates a two-tier system where big companies get certified while everyone else operates without oversight.
Public users of frontier AI models
Ongoing, with each new model deployment
If safety test results remain classified, users cannot know what risks were identified before the model they are using was released. Informed consent requires disclosure. The classified framework removes that possibility.
Scenarios
Testing as theater
CAISI conducts evaluations that find minor issues; companies fix them or publish workarounds; results stay classified. The framework produces no significant safety changes but allows everyone to say they take AI safety seriously. The AI boom continues uninterrupted.
Signal No public CAISI evaluation reports published within 12 months of the agreements
Real intervention
CAISI tests find a capability in an unreleased model that crosses a national security threshold. The agency asks the company to delay release or modify the model. The company complies. A precedent is set for meaningful pre-deployment intervention.
Signal A major lab delays a model release and cites 'government coordination' as the reason
Binding regulation forced by incident
A deployed model produces serious harm that classified testing had flagged but not blocked. Congressional hearings establish that CAISI had visibility and no authority to stop it. Congress passes binding pre-deployment requirements, ending the voluntary era.
Signal A CAISI evaluation report leaked or released shows prior knowledge of a risk that materialized in deployment
What Would Change This
If Congress passes legislation requiring CAISI evaluations to be published and requiring companies to either remediate identified risks or publicly disclose them before release, the framework becomes something worth defending. Without that, it is a press release.
Related
The Government Is Now Testing AI Models Before They Launch. The Companies That Designed the Tests Are the Same Ones Being Tested.
powerThe Government Will Test Your AI Before You Get It. The Companies Volunteered.
powerTrump Wants to Regulate AI Now. The Industry It Threatened Is Thanking Him.
powerTrump's AI Team Is Rebuilding Everything It Burned Down