Build for the Model You Trust Least

Take a finance team that gives an AI agent standing access to the supplier master file. One afternoon the agent changes a vendor’s bank details, citing an email thread. A week later the payment has gone to the wrong account, and the incident review opens the agent’s reasoning trace. It reads cleanly. The email looked legitimate, the change matched policy, the agent followed procedure. Every line of that explanation was written by the system whose decision is under review.

Satya Nadella has published an essay that is, in effect, about that review. In “Models as Insider Risks in the Super Intelligence Era”, he argues that enterprises should treat frontier models, closed and open weight alike, the way they already treat powerful employees. Any capable actor with access to important systems can make mistakes or be compromised, and the architecture has to plan for that. The design goal fits in one line: “we need to separate the supply of intelligence from the authority over it”.

For anyone buying or building enterprise AI, that moves the hard engineering problem into the system around the model: who the agent is, what it can touch, what record it leaves, and who can stop it. Alignment is deliberately set aside in favour of containment and governance, which is work an enterprise architecture team can own today.

The reasoning trace is testimony

Reasoning traces are useful evidence, and the essay treats them as a requirement. They are still an incomplete account of what shaped a decision. Anthropic’s alignment team planted hints in prompts and checked whether reasoning models admitted using them. When the models did use a hint, they mentioned it in their reasoning “often below 20%” of the time. A newer evaluation presented at ICML 2026 found frontier models monitorable most of the time but not always, and warned that this could erode as training methods change and models scale.

So the review in the opening scene needs a second record, one the agent didn’t write. That audit trail should show what the agent requested, what the control layer permitted, who approved it, and what the payments system actually executed, with the agent’s identity and a timestamp on each step. The agent should have no way to edit it, and reviewers outside the team should be able to read it. With that, a team can reconstruct how an outcome was reached without taking the model’s word for it.

A second model checking the first helps up to a point. Stack enough of them and you get an opaque model inside an opaque orchestration layer, watched by another opaque model, which the essay calls nested black boxes.

A 1972 idea with a new tenant

The way out is old. The controls over what a model can access and do “must sit outside the model”, according to the essay, which traces the rule to a 1970s security principle. The principle goes unnamed, but the description matches the reference monitor from a 1972 study for the US Air Force by James P. Anderson. Anderson set three requirements for the mechanism that checks every access: it must be tamper proof, it must always be invoked, and it must be small enough to be analysed and tested completely. No frontier model meets the third, so whatever enforces a model’s permissions has to sit somewhere else.

The essay separates the model from the harness that runs its work and from the action space that defines what it can do. Say the system prompt tells the agent never to change bank details without approval. That instruction only asks the model to behave. The model can misread it, a crafted email can talk it out of it, and afterwards it can claim it followed the rule. Now put the same rule in the payments service, so no change to bank details goes through until a second person approves it. That is the reference monitor in practice: the rule is enforced outside the model, and the agent has no way around it.

Getting there takes real engineering. The check has to catch every route to the same change, including retries, batch jobs and indirect calls through other services. The enforcement layer itself has to be protected from the agent, and kept small enough to audit and test on its own.

What changes for the enterprise

Much of this is already in the playbook security teams run for people: identity, limited privileges, activity logs and containment boundaries. For agents, that means each one gets its own identity instead of a shared service account, the narrowest permissions its task needs, and the protected audit trail described above. It also means a stop, so an authorised person can pause or shut down an agent mid-task.

The EU has already made that stop a legal requirement for some systems. Article 14 of the EU AI Act says the people overseeing a high-risk AI system must be able to halt it, with a “stop” button or something similar. This year’s Digital Omnibus pushed the deadline back. For high-risk uses such as credit scoring and recruitment, the rule now applies from 2 December 2027. AI built into products already covered by EU safety law gets more time. Most other agents fall outside the rule. Even so, it is easier to build a stop in now than to add one to agents already running in production.

Of the seven observability principles listed, two will be harder to sell internally. Model diversity says no single model should be the sole dependency for an important outcome or check its own work, which cuts against the vendor consolidation most procurement teams prefer. Diversity has limits too. Two models can share training data and fall for the same crafted input, so a second model reduces correlated failure without replacing deterministic controls or human approval for consequential actions. Incident disclosure asks companies to share what failed and which controls missed it, across the industry, and few enterprises do that even for conventional breaches.

The essay is a statement of principles. It names no Microsoft product and says industry standards still need to be set where existing ones fall short. Nobody needs to wait for them to start. Pick one consequential agent workflow. Give the agent its own identity, enforce its permissions outside the model, record its actions independently, require approval for sensitive changes, and test whether an authorised operator can actually stop it mid-task.

Nadella ends the essay with a test for any platform: “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.” In an AI architecture review, that becomes two concrete questions. If the model in this workflow were compromised tomorrow, which of our controls would still hold? And who could stop it halfway through a task?