The Attack Surface Across the AI Lifecycle

An AI system has a longer lifecycle than most software, and risk enters at every stage of it. Data is collected and curated. Humans label and rank it. A base model is pretrained, then adapted by fine-tuning or preference training. The result is packaged into files, distributed, deployed behind a serving stack, wrapped in an application, exposed to users, monitored and eventually retired or replaced. Each stage has its own actors, its own artefacts and its own failure modes. If you only threat-model the chat box, you have modelled one stage of many.

The single most useful cut across that lifecycle is between training-time risk and inference-time risk. It is not a taxonomy for its own sake: the two behave completely differently in an incident. Training-time compromise is baked into the weights, ships with every copy of the model, affects every user of every deployment, and cannot be removed by changing a prompt or restarting a service. Inference-time compromise lives inside a single context window, affects one session, and disappears when that context is discarded. Detection, containment and remediation all differ accordingly.

Training-time risk

Data poisoning means influencing the training corpus so that the finished model behaves as the attacker wants. The most concerning variant is the backdoor: the model behaves normally on all ordinary inputs and misbehaves only when a chosen trigger appears, which might be an unusual phrase, a formatting quirk or a rare token. Because behaviour is normal otherwise, standard evaluation does not reveal it. Poisoning is practical mainly because so much training data is collected at scale from sources anyone can write to. A contributor who adds pages to a public site, uploads a dataset, or edits a widely mirrored resource is contributing to future corpora. The research picture on exactly how much poisoned data is required is still developing and the answers depend heavily on the target behaviour, so treat any confident single figure with suspicion. What is not in doubt is that the attack is real and that it does not require access to your infrastructure at all.

Labelling and preference data deserve separate attention because they are usually outsourced. If a supplier systematically mislabels safety-relevant examples, whether maliciously or through poor quality control, the resulting model has a distorted notion of what to refuse, and nothing in the code review or the infrastructure audit will show it. Legal exposure enters here too: personal data ingested without a lawful basis becomes embedded in the weights, where deletion is not a straightforward operation.

Inference-time risk

At inference the model is fixed and the attacker works with inputs. Jailbreaks push against refusal behaviour. Direct injection puts instructions in the user channel. Indirect injection places them in content the application will read on the user's behalf, which is the more dangerous case because the victim never sees the payload. Extraction attacks query the interface systematically and use the responses to reconstruct something valuable: the model's behaviour, its system prompt, or fragments of its training data. Membership inference asks the narrower question of whether a particular record was in the training set, which can itself be a disclosure when the dataset is, say, a clinical cohort. And resource attacks exploit the fact that inference cost is not fixed per request: inputs can be crafted to maximise compute per call, degrading latency for everyone and inflating spend.

Memorisation sits across the boundary and is worth understanding precisely because of that. The cause is training-time: rare sequences, especially duplicated ones, can be retained well enough to be reproduced. The symptom is inference-time: a model that returns a real person's address verbatim when given their name. If you see verbatim reproduction of an unusual, specific string that no plausible generalisation would produce, the explanation is memorisation of the training or fine-tuning set, not coincidence and not a retrieval bug. Models genuinely can store and regurgitate training text, and fine-tuning on a small internal dataset with duplicated records makes it markedly more likely.

The practical payoff of this classification is that it tells you which control family to reach for. Training-time risk is answered by provenance, dataset governance, supplier assurance, deduplication and evaluation before adoption. Inference-time risk is answered by architecture: what the system is permitted to do while it is being manipulated. Applying an inference-time control to a training-time problem, for example hoping a guardrail will contain a backdoor, wastes effort and creates false assurance.