What Changes When You Put a Model in the Loop

A conventional web service has a clear internal border. Requests arrive as structured parameters, code decides what those parameters mean, and no value in a form field can rewrite the logic that processes it. Injection bugs in that world are failures to keep one specific channel separated: a string reaches an interpreter that was expecting code. Those bugs are well understood because the interpreter can be told, mechanically, which part of its input is code and which part is data. Putting a language model in the middle of an application removes that mechanical separation. Almost everything distinctive about AI security follows from that single change.

A model does not receive fields. It receives one flat sequence of tokens. Your system prompt, the descriptions of the tools it may call, the documents your retrieval layer selected, the conversation so far and the user's latest message are concatenated into a single stream. Role labels exist in the API and in the chat template, and they do influence behaviour, but they are not enforced separations. Nothing in the architecture prevents a sentence inside a retrieved document from carrying as much weight as a sentence in your system prompt. Every token in the context competes for influence over the next token generated. That is the whole mechanism, and it is why a natural-language interface is not simply a new front end on an old design.

Trust boundary

A point where data or control passes between components with different levels of trust, and where trust therefore has to be re-established rather than assumed.

Context window

The full token sequence the model sees for one generation: system prompt, tool schemas, retrieved content, history and the current input. Treat it as one undivided trust zone.

System prompt

Operator-authored text placed at the front of the context to set behaviour. It is influential guidance, not an enforced policy.

Orchestrator

The application code around the model that assembles context, parses model output, decides which tools may run and executes them. It is ordinary deterministic software, which is what makes it a useful place to put controls.

Sink

Any downstream component that consumes model output and gives it meaning: a browser, a database, a shell, a mail gateway, another model.

Where the new boundaries sit

Threat-modelling an AI feature means drawing the boundaries the model introduced and asking, at each one, what an attacker who controls the data crossing it can achieve. There are five that appear in nearly every design.

1. User input into context. The classic boundary, but now the input can be an instruction rather than a value.
2. Retrieved or fetched content into context. Documents, web pages, emails, tickets, database rows. Anything the model reads is a candidate instruction source, and much of it is writable by people you did not authorise.
3. Model output into the orchestrator. The application parses generated text into tool calls and arguments. That parse is a boundary: the text is untrusted.
4. Tool call into an external system. The point where generated text becomes a real-world side effect, using credentials the model did not have to steal.
5. Model output into a sink. Rendering, storage, execution or forwarding, each of which interprets the text in its own language.

Take a concrete case: an internal assistant that answers staff questions from a company wiki and can raise tickets in a workflow system. The design looks self-contained because everything sits inside the corporate perimeter. Now trace untrusted text through it. The wiki accepts edits from every employee and from contractors. Tickets accept text pasted from customer emails. Retrieval pulls whichever chunks score highest, which an author can influence by writing content that matches likely queries. The assistant holds a service credential able to create and modify tickets. An attacker who can write one wiki paragraph has, in effect, gained an input channel into a privileged automated actor. No firewall rule was broken and no credential was stolen.

Jailbreak or injection?

These two terms are used loosely and the distinction is worth fixing early, because it decides who owns the problem. A jailbreak targets the model's own trained refusal behaviour: the attacker wants the model to produce content its provider tried to make it decline. A prompt injection targets the application built on the model: the attacker wants your system to do something its operator did not intend, by smuggling instructions through the inputs your system chooses to feed the model. The two overlap in technique and often appear together, but the target is different, and so is the remedy. Neither one modifies the weights; both act only on a live context.

One last framing point. Model responses are non-deterministic, and that genuinely complicates engineering: the same test can pass and then fail, and you cannot certify behaviour by running it once. But non-determinism is a testing problem, not the origin of the attack surface. A perfectly deterministic model that still read instructions and data through one channel would be just as injectable. Keep the two ideas apart, because controls that address variance, such as low temperature or repeated sampling, do nothing at all about injection.