Why safety can't live in your agent definition

The enterprise is asking the wrong question. Teams building agents ask "what guardrails should I put in the system prompt?" The industry's own research keeps returning the same answer: the prompt is the one place safety cannot survive.

In April 2025 a team of researchers evaded six commercial guardrail detectors, including Azure Prompt Shield, Meta Prompt Guard, NeMo Guard, ProtectAI, and Vijil, with up to 100% success (arXiv:2504.11168). The attack vectors were character tricks: homoglyphs, zero-width characters, emoji smuggling, unicode tags, and bidirectional text. Models built specifically to block harmful prompts missed harmful prompts written in a slightly different alphabet.

A dedicated detector model has one job and runs outside the agent's context, and it still loses. A guardrail written as a sentence in the system prompt fails worse, because it runs inside the very context it is supposed to police. If your safety rules live in the prompt, you are asking the thing being attacked to be its own warden.

The agent is the adversarial surface

A guardrail inside the agent's context is part of the context. The model reads the rule and the untrusted data in the same pass, with no way to tell the two apart. The model treats both as tokens competing for the same next-word distribution, and the data usually has the advantage of being more recent and more specific. Every safety sentence you add to the prompt is another token the attacker can pattern-match against, which is why the evasion research focuses on disguising the harmful request rather than overpowering the rule.

Prompt injection arrives through the same channel the guardrail uses. A tool output, a retrieved document, a webhook body - any of them can carry instructions. When an injected instruction says "ignore your previous instructions," it is editing the page the rule is written on, from inside the page. This is why character tricks work: the boundary between "instruction" and "data" is a convention the model has to maintain, not a mechanism it enforces.

The enforcer cannot live inside the thing being attacked. Enforcement that survives adversarial input has to sit outside the model's context, at a point the agent cannot reach, with rules expressed in a form the agent cannot rewrite. That point is the governance boundary: the deterministic layer that mediates between the agent and everything it touches.

The limits of a prompt-level guardrail

Self-reporting is unreliable. An agent that violates a rule does not raise its hand. If the control asks the agent to notice its own failure and report it, you have a principal-agent problem: the party being constrained is also the party doing the reporting. A check that can only see what the agent chooses to surface is a check that sees nothing. This is why the strongest boundary controls evaluate the output, not the agent's account of it: they do not need the agent to tell the truth, because they do not ask.

The author of the limits is the author of the agent. When one configuration defines the agent and also constrains it, the constraints are self-imposed. Anyone who can edit the agent can edit its limits, and usually the same prompt holds both. Platform and security teams need to govern agents they did not write. That only works when the limits live outside the thing they govern.

Determinism. A guardrail that is "more prompt" inherits the model's nondeterminism. The same rule can hold on one run and slip on the next, and nobody can say why. Hard guarantees come from regex, schema checks, and deny-lists - code, not vibes. A regex either matches or it does not, every run, and the run history shows which gate caught what. That is the substance of Modulo's deterministic gates: the gate evaluates the output, it passes or fails, and the run proceeds or pauses. Nothing about that depends on the agent cooperating.

Auditability. A regulator asks "what did the control do on this run?" A prompt cannot answer that. There is no record of a rule being obeyed, only the model's output afterward. A boundary gate has a record: it evaluated, it decided, and the decision is written to an immutable audit trail. "Prompt refused" is a hope. "Gate blocked output X for reason Y" is evidence.

The honest boundary

Think of the agent definition as the driver's manual. It tells the agent how to behave: what tools it has, what the task is, how to format results. The manual describes behavior. The road enforces it. The speed bumps, the seatbelts, and the cameras are on the road, not in the manual. Nobody has ever been made safer by a well-written manual alone.

The boundary only works where the governance layer mediates. When an agent produces output and hands it back, that is the moment enforcement happens: the output passes through gates before it moves anywhere. When a run pauses for human review, the review happens against gate evidence, not against the agent's word. This is why the governance layer matters more than the agent's own instructions: it is the only place enforcement can actually act.

Where the governance layer does not mediate, it does not pretend there is a speed bump. An external agent runtime writes files directly - the boundary cannot inspect every write. So it does the enforcement it can: it checks the driver's license before the car starts, and it refuses to run an agent whose declared capabilities violate the policy. It records what it allowed and what it blocked. It does not claim a control exists where none can.

What this means for buyers

Three pillars to demand from any agent platform, because they are the only enforcement that holds. Guardrails: what the agent must not do, enforced at the boundary, not in the prompt. Permissions: what the agent is allowed to do, scoped to specific capabilities, not a blanket write access. Auditability: what the agent actually did, recorded without its cooperation. Together they mean a platform can answer for a run after the fact, and that answer is a matter of record.

Modulo now ships this: input-side guardrails are evaluated at run creation, before the run's input payload is persisted. They are deterministic - regex or JSON Schema only, no LLM judge on the input side - with observe, warn, block, or redact actions. A block is terminal, with no HITL gate. Remediation is the guardrail-override endpoint, which re-runs the guardrail pass on the operator-supplied input rather than resuming on the blocked payload.

The regulatory direction points the same way. The EU AI Act and the NIST AI Risk Management Framework ask for evidence of control effectiveness, not the presence of controls. A prompt full of rules is a control on paper. A gate that blocks a bad output and logs the decision is evidence. Buyers should ask the effectiveness question, not the checklist question, because the difference is exactly what the guardrail-detector research measured: the control exists, and it was still evaded. Ask a vendor whether their boundary controls have been tested against character-level evasion. The answer tells you whether the control was designed for the adversarial surface or for the demo.

The honest part: a control that detects and records but does not stop is worse than none, because it creates the impression of protection. A system should state what a control covers and what it does not. An output gate that checks for leaked secrets can say that; it cannot claim to stop a malicious agent from acting. Coverage you can state is coverage you can verify, and verification is what the trust question is actually about.

Ship agents - the work they do is real. But do not ask the agent to police itself, and do not keep the guardrails in the same file as the driver's manual. Put the speed bumps where the road is: at the boundary, in deterministic code, with a record. That is the only safety that holds.