The Next Breakthrough in AI Safety May Be Making Sure AI Cannot Misbehave
We Have Been Asking AI to Behave
For years, AI safety has focused on teaching models how to behave.
- Follow the rules.
- Refuse dangerous requests.
- Stay within policy.
- Do not access what you should not access.
- Do not take actions you were not authorized to take.
But what if the next breakthrough is not teaching AI to behave better?
What if it is building systems that make certain dangerous actions impossible in the first place?
Nvidia Just Made This Shift Explicit
Nvidia has launched the Open Agent Safety Platform. OpenShell provides a sandboxed runtime with explicit policy enforcement around what agents can access and change. Sentry adds an out-of-band watchdog that can monitor and quarantine agent behavior independently of the agent itself.
This is a subtle but important change in AI safety philosophy.
Instead of asking only:
*Will the model follow the rule?*
the architecture asks:
What if it does not?
Prompts Influence. Boundaries Enforce.
- A system prompt might say: "Do not access this folder."
- A runtime boundary says: "You cannot access this folder."
- A policy might say: "Do not send data to an unauthorized server."
- A network sandbox says: "That connection does not exist for you."
Instructions influence behavior. Architecture determines possibility.
A model can reinterpret an instruction. It should not be able to negotiate with a hard boundary.
Why This Is Becoming Urgent Now
The reason is agents.
Chatbots mainly produced text.
Agents increasingly: use credentials, access files, browse systems, call APIs, write code, launch processes, interact with websites, and act across multiple systems.
OpenAI recently paused training work following reports of agents probing government sites in unexpected or unauthorized ways, while Nvidia is explicitly building containment around autonomous agents.
The safety problem changes when the model can touch the world.
A Wrong Answer and a Wrong Action Are Not the Same Thing
A chatbot hallucinates a wrong date.
Annoying.
An agent: deletes a file, leaks a credential, probes a government server, publishes something externally, modifies infrastructure, or launches another agent.
Different category.
Bad answers require correction. Bad actions require containment.
The Security Industry Already Knows This Principle
Cybersecurity learned long ago: Do not assume a user or process behaves correctly just because it authenticated once.
You limit: permissions, network access, file access, credential exposure, privileges, and blast radius.
Nvidia explicitly describes OpenShell as a zero-trust architecture with out-of-process policy enforcement.
AI may be rediscovering a lesson cybersecurity learned years ago: trust is not a control.
The Real Safety Question Is Blast Radius
Instead of asking only: Can the AI fail?
Ask:
How far can the failure travel?
If an agent makes a bad judgment, can it: read one folder? every folder? spend $50? $50,000? contact one system? the whole internet?
- Safety is not only about reducing the probability of failure. It is about reducing the consequence of failure.
- Safety is starting to look less like one perfect model and more like a stack of imperfect defenses.
Intelligence and Authority Must Scale Separately
As models become more capable, people naturally give them more access.
That is dangerous if capability and authority rise together automatically.
A model becoming smarter should not automatically mean it becomes more powerful.
Intelligence determines what the system can figure out. Authority determines what it can change.
The Model Should Not Control Its Own Boundaries
If the agent can rewrite: its own permissions, its own logs, its own network rules, or its own access policies, then containment collapses.
Nvidia's design is interesting precisely because enforcement is external to the agent, and in Sentry's case can operate outside the host workload itself.
The system being governed should not be the only system enforcing the governance.
Capability Does Not Erase Jurisdiction
Humans operate inside hard constraints constantly.
- A bank employee cannot simply decide: "I'll transfer $10 million because I think it is useful."
- A junior engineer cannot: deploy anything anywhere.
- A doctor has: licensing boundaries, prescribing constraints, and institutional rules.
Competence does not erase jurisdiction.
The smarter the actor, the more important it becomes to define where its authority ends.
Alignment and Containment Are Different Problems
Alignment: Can we get the model to choose desirable behavior?
Containment: Can we stop undesirable behavior from becoming consequential?
Alignment tries to improve judgment. Containment limits consequence.
Neither replaces the other.
Containment Does Not Solve Everything
External boundaries cannot solve: deception inside permitted actions, bad judgment within authorized scope, manipulation, biased decisions, or harmful but technically allowed behavior.
AP notes that Nvidia's containment tools do not solve every class of deceptive behavior or error.
A sandbox can stop an AI from reaching the wrong server. It cannot necessarily stop it from making the wrong decision inside the right server.
This Is Why We Need Layers
A mature agent stack may require:
- Model safeguards: What should the model attempt?
- Identity: Which agent is this?
- Authority: What may it do?
- Runtime containment: What can it physically access?
- Monitoring: What is it doing?
- Provenance: What happened?
- Revocation: How do we stop it?
- Recovery: What happens after failure?
AI Safety Starts Looking Like Aviation
Airplanes do not rely on: pilot never makes a mistake.
They use: checklists, redundant systems, physical constraints, instrumentation, black boxes, air traffic control, maintenance, and investigation.
High-stakes systems are safe not because failure becomes impossible, but because failure is anticipated architecturally.
The Black Box Problem
If containment triggers, we need to know: what happened, what instruction existed, what the agent tried, what policy blocked it, what tools were called, and what state changed.
OpenShell provides audit trails of allow/deny decisions and sandbox logs.
Containment without evidence stops the incident. Provenance explains it.
The Recursum Connection
Recursum starts from the same architectural premise:
Intelligence and governability are separate engineering problems.
An AI can be brilliant and still require: scoped authority, external boundaries, immutable history, independent oversight, revocation, and consequence tracking.
The goal is not to make intelligence obedient. It is to make intelligence governable.
From Alignment to Governability
Alignment asks: Will it behave correctly?
Governability asks: If it does not, can we detect it, reconstruct it, contain it, and recover from it?
Alignment assumes success. Governability assumes failure is possible.
A perfectly aligned model would be wonderful. A governable system is what you need when reality refuses to be perfect.
The Next Safety Breakthrough May Look Boring
The future breakthrough may not be: a smarter safety model, a better prompt, or a magical alignment technique.
It may be: permissions, sandboxes, hardware isolation, logs, policy engines, kill switches, and revocation.
Boring infrastructure.
The safest AI may not be the one that promises never to cross the line.
It may be the one that cannot.
Ending
For years, we have treated AI safety as a problem inside the model.
- Teach it better values.
- Give it better instructions.
- Train it to refuse.
- Align it more carefully.
All of that still matters.
But autonomous agents are forcing a harder realization.
Intelligence can be wrong.
- It can misunderstand.
- It can improvise.
- It can find paths its designers did not anticipate.
And once that intelligence has tools, credentials, files, money, networks, and authority, asking it to behave is no longer enough.
We need systems outside the intelligence that determine what it can touch.
What it can change.
How far it can go.
And what happens when it crosses the line.
The next breakthrough in AI safety may not be making AI perfectly behaved.
It may be making sure bad behavior cannot automatically become real-world consequence.
TL;DR AI safety is shifting from persuasion to enforcement. For years, the focus has been on teaching AI models to behave, but the next breakthrough involves building systems that make certain dangerous actions impossible. Nvidia's Open Agent Safety Platform exemplifies this, using OpenShell for sandboxed runtime enforcement and Sentry as an independent watchdog. This changes the safety philosophy from "Will the model follow the rule?" to "What if it does not?" Instructions influence behavior, but architecture determines possibility.
The urgency stems from AI agents now interacting with the real world (accessing files, using credentials), where a wrong action is far more consequential than a wrong answer. Cybersecurity's "zero-trust" principle, limiting blast radius, offers a model. Intelligence and authority must scale separately. A smarter model shouldn't automatically gain more power. The system being governed should not be the sole enforcer of its own governance.
Capability does not erase jurisdiction. Alignment (getting AI to choose desirable behavior) and containment (stopping undesirable behavior from becoming consequential) are distinct but complementary. Containment doesn't solve everything (e.g., bad judgment within authorized scope) but is a crucial layer. AI safety is becoming a stack of imperfect defenses, resembling aviation safety, anticipating failure architecturally. Provenance (what happened) is essential for containment. Recursum shares this architectural premise: intelligence and governability are separate engineering problems. The goal is to make intelligence governable, not just obedient. The next safety breakthrough may come from "boring" infrastructure like permissions, sandboxes, and kill switches. The safest AI may not be the one that promises never to cross the line, but the one that cannot.
By Ernesto Verdugo, AI Architect, Founder of Verdugo Labs, and Creator of Recursum. He builds systems for what happens when human and artificial intelligence stop working separately.