Module 3 · 6 min read
Treating model output as untrusted: schema validation and encoding at the sink
Why well-formed structured output is not safe output, and why sanitisation has to happen at each destination rather than once at generation.
A platform's structured output mode guarantees that the JSON the model emits for a function call is well-formed and matches the declared shape. A developer reads that guarantee and concludes there is nothing left to check. The guarantee is real and it is about syntax. It says the braces balance, the fields have the declared types and the decoder could not produce anything else. It says nothing about the values, which were generated from a context that contained attacker-controlled text. A constrained decoder will happily produce a perfectly valid JSON object whose identifier field points at another tenant's record.
So the server does what it would do with any hostile API client. Validate against a strict server-side schema: types, enumerations, numeric ranges, string lengths, patterns. Reject unknown fields rather than ignoring them, because silently dropped fields are how an attacker probes for a version of the handler that does not drop them. Canonicalise before comparing. Then, separately, authorise: check that the principal on whose behalf the agent is acting is entitled to the specific resource named in the arguments. Validation and authorisation are different checks and passing the first tells you nothing about the second.
Check yourself
A team validates arguments carefully for tools that write data, and skips validation for read-only tools on the grounds that a read cannot change anything. Why is that a mistake?
Reads are the exfiltration path. An unvalidated identifier on a read tool is a direct object reference the attacker chooses, and in an agent the retrieved data lands in a context window that may already contain an instruction to send it somewhere. Validation of the argument, and authorisation of the principal against the named resource, matter as much on reads as on writes.
One tempting shortcut is to validate only when the function performs a write. Reads are where exfiltration happens. A read tool with an unvalidated identifier is the classic insecure direct object reference, and in an agent it is worse than usual because the retrieved data lands in a context window that may already contain an instruction to send it somewhere.
Encoding belongs at the sink
Output from one pipeline is rendered in a web UI, embedded in HTML emails, and interpolated into shell-based report scripts. The team proposes a single central sanitisation pass that strips HTML tags at generation time, and this is insufficient for a reason that predates language models by decades. Safety is a property of the destination grammar. Stripping angle brackets neutralises nothing in a shell context, where the dangerous characters are semicolons, backticks and pipes. Escaping shell metacharacters does nothing for a spreadsheet, where a leading equals sign starts a formula. And at generation time the pipeline does not yet know which sink the text will reach, so it cannot choose the right transformation even in principle. This is the same lesson that ended centralised input sanitisation as a cross-site scripting defence: encode at output, for the context you are encoding into.
Encode at each sink
- The transformation is chosen by the destination grammar
- The same string can be safely rendered, emailed and passed to a process
- A new sink gets its own encoding when it is added
- Failures are local to one sink and easy to reason about
- Matches the rule that ended centralised input sanitisation for cross-site scripting
One pass at generation time
- The destination is unknown when the transformation runs
- Stripping markup does nothing for shell, spreadsheet or path contexts
- A new sink silently inherits protection that does not fit it
- Creates confidence that output is safe everywhere
- Adding more strip rules never closes the structural gap
| Sink | Hazard | Control at that sink |
|---|---|---|
| HTML page | Script execution, markup injection | Contextual HTML encoding; strict content security policy; never inner-HTML raw output |
| HTML email | Remote content beacons, phishing markup | Encode, strip remote references, restrict to an allowlist of elements |
| Markdown renderer | Image and link URLs that exfiltrate context on render | Allowlist link and image hosts; forbid data and script schemes |
| Shell command | Command injection | Never build a command string; pass an argument vector to the process directly |
| SQL query | Query injection | Parameterised statements; identifiers resolved from an allowlist |
| Filesystem path | Traversal outside the intended directory | Canonicalise, then confirm the resolved path is under the permitted root |
| Terminal or log viewer | Escape sequences that rewrite what an operator sees | Strip control characters before display |
| Spreadsheet export | Formula injection | Prefix cells starting with a formula character; quote on export |
Try it first
An assistant calls no tools and emits nothing but plain Markdown. A user summarises an attacker's document and the conversation still leaks. How?
The document contained instructions telling the model to end its answer with an image whose address embeds the conversation so far. The Markdown renderer does what renderers do: it fetches the image on display, and the attacker's server receives the context in the request. No tool was called, no code executed, and the model emitted only ordinary markup. The control is at the renderer, not at the model: restrict which hosts links and images may point to, and treat any automatic outbound fetch on render as an exfiltration channel that needs a policy.
The markdown row deserves a scenario because it is the most commonly missed. An assistant summarises a document that contains injected text instructing it to append an image whose URL embeds the conversation so far. The renderer sees a normal image, fetches it, and the attacker's server receives the context as a query string. Nothing executed, no tool was called, the model emitted only plain markdown, and the data left the building. The control is at the renderer: restrict which hosts images and links may point to, and treat automatic outbound fetches on render as an exfiltration channel to be governed like any other.