The Attack Taxonomy and Training-Time Attacks

The taxonomy that matters is organised by two questions: which phase of the lifecycle is attacked, and which asset is the target. Evasion attacks the inference phase and targets a single decision; the model is unchanged and only the crafted input is affected. Poisoning attacks the training phase and targets the learned function itself; after a successful poisoning the model behaves differently for everyone. Extraction, membership inference and model inversion attack confidentiality: of the model as an asset, or of the data it was trained on. A smaller fourth group attacks availability, and here you have to be careful, because the field uses the word availability in two incompatible ways and both are current. Availability poisoning is indiscriminate training-time poisoning that degrades the model's general performance for everyone, which makes it comparatively easy to notice. Resource-exhaustion attacks craft inference-time inputs that maximise compute, memory or latency, leaving the learned function untouched. This course uses the qualified terms throughout, and you should too: bare availability tells the person reading your incident report nothing about which phase was attacked. Misclassifying an incident across any of these lines leads directly to the wrong remediation, which is why exams and real investigations both press on the boundaries.

Apply it to a case. A platform continuously retrains its spam filter from user feedback. An attacker controlling many accounts marks thousands of malicious messages as legitimate. Over several weeks, the filter's behaviour shifts in the attacker's favour. This is poisoning, and the reasoning is what matters: the feedback loop is a training channel, so the attacker is modifying the learned decision function itself. Evasion is the tempting label because the messages are malicious and the filter fails to catch them, but evasion leaves the model untouched and affects only the crafted inputs. Here, future decisions change for every user of the platform, including messages the attacker never sent. Nor is it extraction, which accumulates information about the boundary without moving it, and it is not model inversion, which recovers properties of training data.

Backdoors and clean-label poisoning

A backdoor is a trained association between a trigger and an attacker-chosen output, deliberately constructed so that behaviour on all other inputs remains normal. That last property is the design goal, not a side effect: a backdoor that degraded ordinary accuracy would be caught by the acceptance tests. The trigger may be a small visual patch, a specific texture, an inaudible audio pattern, a rare token sequence or an unusual turn of phrase. Backdoors can be installed by contributing poisoned data to a corpus, by supplying pre-trained weights, or by tampering with a fine-tuning dataset.

Now the control that seems obvious. A supplier audits its training pipeline by having human reviewers confirm that every example carries the correct label, and concludes that poisoning is addressed. The flaw is structural. Clean-label attacks use examples whose labels are genuinely correct, with the manipulation carried in the features: an image that a human agrees is a dog, subtly perturbed so that training on it shifts the boundary or installs a trigger association. A label audit passes such examples by construction, because there is nothing wrong with the labels. Two neighbouring claims are also wrong and worth killing off: poisoning does not require flipping labels, and it does not require controlling a majority of the corpus, since targeted poisoning and backdoor installation typically need only a very small fraction of the training data.

Interrogating supplied weights for a backdoor

When a third party supplies pre-trained weights, only a few activities give real evidence about hidden behaviour. Optimisation-based trigger reconstruction searches for a small universal pattern that drives arbitrary inputs to one fixed target class, and an anomalously small such pattern for one class is suggestive. Internal activation analysis looks for spectral signatures in the representation of a class, or for neurons that fire only on a narrow, semantically meaningless input feature. Reproducing training from an audited dataset with a deterministic build, then comparing behaviour against the supplied weights, is the strongest evidence available, and the most expensive.

Two things that feel like evidence are not. Clean-accuracy benchmarking is useless, because a competent backdoor preserves clean accuracy deliberately; a measurable drop would be a failure of the attack. Verifying the model file's hash against a published value proves that you received the file the supplier sent, which is integrity of transfer and says nothing whatsoever about what the file does. Be candid about the limits of the real methods too: trigger reconstruction assumes a trigger of roughly the size and shape being searched for, activation analysis assumes a detectable signature, and full reproducible retraining is impossible for most large models. None of these is a proof of absence, which is why provenance, supply chain controls and restricting what a model of uncertain origin is allowed to influence carry as much weight as any detector.