Module 1 · 7 min read
Adversarial Examples, Transferability and Threat Models
What an adversarial example actually is, why perturbations transfer between independently trained models, and how to classify an adversary honestly.
An adversarial example is an input crafted so that a model produces a wrong output while the input remains, by the threat model's own measure, essentially the same input. That last clause carries all the weight. The standard formalisation constrains the perturbation inside a norm ball: the attacker may change the input by at most epsilon under some distance measure, commonly the maximum change to any single element, or the total energy of the change across the input. The choice of norm and epsilon is a modelling decision, not a fact about the world, and choosing it deliberately is the first thing a competent red team does. A defence evaluated under one ball tells you very little about behaviour under another, and neither ball has much to say about the transformations a real attacker would use: recompression, printing and rephotographing, cropping, lighting change, or in language, paraphrase.
Why adversarial examples exist at all remains contested, and you should be able to say so. One account emphasises geometry: in very high dimensional input spaces, decision boundaries pass close to typical data points, so a small step in a well-chosen direction crosses one. Another emphasises features: models latch onto patterns that are genuinely predictive on the training distribution but not aligned with human semantics, and adversarial perturbations manipulate exactly those non-robust but real features. These accounts are not mutually exclusive and neither is settled. What is not in dispute is the empirical picture: adversarial examples are easy to find in most undefended models, they are not rare accidents, and defences that appear to eliminate them usually turn out to have obstructed the search rather than removed the examples.
Transferability
A vendor removes gradient access from its image classifier API, applies a rate limit, and states that adversarial examples are no longer feasible. An attacker collects public data from the same domain, trains a local model on it, crafts perturbations against that local model, and submits them to the target. They succeed far above chance. Nothing about the target's gradients was ever obtained.
The property at work is transferability. Models trained on similar data to solve similar tasks learn overlapping decision boundaries, so a perturbation that pushes an input across the boundary of one model frequently pushes it across the boundary of another. Note carefully what is not required: the surrogate does not need to match the target's architecture. Transfer is routinely observed across quite different architectures, and between models trained on different but similar corpora, because shared learned features rather than shared structure drive it. Two seductive alternative explanations are worth dismissing. It is not gradient leakage through returned confidence scores, since in this scenario the attacker never queried the target during crafting. It is not a shared library defect, which would be a supply chain issue with an entirely different signature.
Try it first
The attacker's surrogate has a completely different architecture from the target. Why does the attack still transfer?
Because transfer is driven by shared learned features and overlapping decision boundaries, not by shared structure. Two models trained on similar data for the same task tend to rely on similar predictive signals, so a perturbation that moves an input across one boundary often moves it across the other. Matching the architecture raises transfer rates somewhat; it is not a precondition, and believing that it is leads straight to the false comfort that an undisclosed architecture is a defence.
Classifying the adversary
| Threat model | What the adversary holds | Practical capability |
|---|---|---|
| White box | Weights, architecture, preprocessing, and any defence in the pipeline | Exact gradients; optimisation directly against the deployed loss; the strongest attacks available |
| Grey box | Some subset: architecture, training data, defence design, or rich query access | Accurate gradient estimation and high-fidelity surrogates; usually close to white box in practice |
| Black box | Query access only, often with limited output detail and rate limits | Transfer attacks from surrogates, score-based estimation if scores are returned, decision-based search if only labels are |
Consider an adversary who knows the target's architecture and the public corpus it was trained on, cannot read the weights, and has effectively unlimited queries returning full logit vectors. This is grey box. But the classification is only the start of the analysis, and the important judgement is what to assume when evaluating a defence. Rich query access with full logits supports numerical gradient estimation, and architectural knowledge plus a public corpus supports training a surrogate that is very close to the target. The practical gap to white box is small. The defensible default is therefore to evaluate the defence as if the adversary were white box, unless you can demonstrate that the gap is real and material for your specific system.
Threat models at a glance
Select a card to turn it over.
Check yourself
A team evaluates its vision model under a maximum-per-element perturbation budget and reports a strong robustness figure. The model is deployed to read labels from packages photographed with handheld scanners in a warehouse. What is the main limitation of that evaluation?
A norm ball is a convenient mathematical proxy, not a description of what attackers or the physical world actually do. Robustness inside one ball tells you very little about physically realisable transformations, which are typically large in element terms and small in human terms. Choosing the threat model deliberately, and stating plainly what it excludes, is the first job of any evaluation.
- Perturbation budget (epsilon)
- The maximum permitted change to an input under the chosen distance measure. It defines the threat model and every robustness number is meaningless without it.
- Norm ball
- The set of inputs within distance epsilon of the original under a specific norm. Robustness claims are claims about one ball under one norm, never about perturbations in general.
- Surrogate model
- A locally trained stand-in for the target, used to craft attacks that are then transferred. Fidelity to the target raises transfer rates but is not required for transfer to work.
- Query access
- The ability to submit inputs and observe outputs. Its value to an attacker depends sharply on output detail: full probability vectors, top-k, or a single label.