Module 1 · 7 min read
Threat modelling an LLM application: where the boundaries actually sit
Why the model belongs outside the trusted zone on both its input and its output side, and how to enumerate every path attacker-controlled text takes into context.
A team runs STRIDE against a new LLM-powered support assistant. They draw the browser as untrusted, the API gateway as the boundary, the application as trusted, and the model as a trusted component, because the provider holds an independent assurance report. The diagram looks tidy and the review passes. The flaw is not that they used STRIDE, which applies to systems containing machine learning components perfectly well. The flaw is what the assurance report actually attests to. It speaks to the provider's operational controls: change management, access, availability, how the service is run. It says nothing about how the model behaves when an attacker writes part of its input, which is the entire question a threat model needs to answer.
Model output is a function of model input, and model input contains attacker-controlled text in every non-trivial deployment. That makes the model's output side a trust boundary in its own right, regardless of vendor assurance and regardless of whether you fine-tuned it on your own data. Everything downstream of the model, the renderer, the tool dispatcher, the database, the shell script, the email template, is consuming data derived from an untrusted source. Once you draw the diagram that way, most of the rest of this course follows mechanically.
Two boundaries, not one
It helps to name them separately because they need different controls. The input boundary is where content of mixed trust levels is merged into a single context window. The model has no reliable way to distinguish your instructions from text that arrived inside a document, because they occupy the same channel. The output boundary is where generated tokens leave the model and become arguments, markup, queries or commands somewhere that will act on them. Controls at the input boundary are about provenance and containment of content. Controls at the output boundary are about validation, encoding and privilege, and they are the ones that hold when the input-side controls fail.
| Component | Common assumption | Correct treatment |
|---|---|---|
| End user text | Untrusted | Untrusted |
| Retrieved documents | Internal, therefore trusted | Untrusted: whoever can edit the source controls the text |
| Tool and API results | Trusted, it is our own service | Untrusted: content often originates with third parties |
| Model output | Trusted, the provider is certified | Untrusted data crossing into every downstream sink |
| Plugin or tool descriptions | Configuration | Untrusted text that enters model context |
| Prior conversation turns | Our own history | Untrusted: contains earlier attacker-supplied content |
Enumerate the injection paths
A retrieval assistant pulls pages from a company wiki that any employee can edit and concatenates them into the prompt. Ask what this most directly enables and the answer is indirect prompt injection: attacker-editable content is placed into the instruction channel at inference time, so any wiki editor can plant text that steers the model, invokes tools or exfiltrates context. It is worth being precise about why the neighbouring answers are wrong, because they test whether you understand when each attack operates. Training data poisoning acts on the weights during training; retrieval happens at inference and never touches the training set. Model extraction is about stealing model behaviour through queries. Membership inference is about what the model memorised. None of them describe what a wiki editor gains here.
Is this an injection path?
Select a card to turn it over.
The enumeration should be exhaustive and it is usually longer than teams expect: user messages, uploaded files, retrieved documents, web pages fetched at the model's request, email and ticket bodies, calendar entries, code comments in a repository being analysed, tool and API responses, other agents' messages, and the description fields of installed tools. Every one of these is a place text reaches the context window. A team that sanitises user input and stops there has covered one entry in that list.
Check yourself
A team states that injection is handled because all user input is sanitised at the API gateway before it reaches the prompt. Which flow does that control fail to cover?
Sanitising the user's own message covers one of many paths into the context window. Content retrieved from documents, fetched pages, tool responses and tool descriptions never passes through the gateway's input filter, and those are precisely the paths indirect injection uses.
Try it first
An assistant has no tools at all. It only reads a user's question and writes an answer into a web page. Is there still a trust boundary to defend?
Yes, on the output side. The model's text is untrusted data being written into a rendering context, so it needs contextual encoding at that sink like any other untrusted string, and the renderer needs a policy about which links and images it will follow. A tool-less assistant removes the action channel, which is a genuine and valuable reduction, but rendered output is itself a channel: markup can carry an outbound fetch that leaves with whatever was in the context. The boundary moves; it does not disappear.
Practically, run the threat model over data flows rather than over components. For each flow, ask three questions: who can influence this content, what does the consumer of this content do with it, and what privilege is held at that point. The answers give you the boundaries. If the answer to the first question for any flow entering the prompt is anything other than a small set of trusted operators, the model is downstream of an attacker, and the design has to survive that.