Gradient Masking and the Adaptive Attack Principle

This is the module that separates people who evaluate defences from people who read about them. The central problem is that a robustness number is not a measurement of a property of the model. It is a measurement of how well one particular search procedure did against one particular model on one particular day. If the search procedure can be obstructed without the underlying vulnerability being removed, the number improves and nothing else does. That failure mode has a name, gradient masking, and recognising it from its signatures is a core professional skill.

Take a concrete report. A defence shows near-zero success for a strong iterative gradient attack. On the same model, transfer attacks from an undefended surrogate succeed far more often. And raising the perturbation budget without bound never drives attack success to 100 percent. Each of those observations, on its own, would be worth a question. Together they are conclusive. The defence is degrading the attacker's optimisation signal rather than removing adversarial inputs, and the reported robustness is an artefact of the evaluation.

Why each signature is damning

• Black box beating white box. An adversary with the weights can do anything an adversary without them can do, including training a surrogate and transferring. So a white-box attack should never be weaker than a transfer attack. When it is, the extra information is not being converted into an effective search, which means the gradients are misleading rather than the model being hard.
• Single-step beating iterative. An iterative attack can always take one step and stop. If more optimisation makes the attacker worse, the loss surface is deceiving the optimiser: the gradient at the starting point points somewhere useful, and following it further leads away.
• An unbounded budget failing to win. This is the cleanest one. With a large enough perturbation you can turn any input into an example of a different class, at which point the original label is simply wrong. A model that keeps predicting the original label under an enormous perturbation is not being robust; the evaluation is scoring it against a label that no longer applies, which is only possible if the attack is failing to move the input at all.
• Two further signals worth checking: attack success that is completely insensitive to step size, iteration count or restarts, and a randomised defence whose apparent robustness collapses when the attack takes expectations over the randomness instead of attacking a single sample of it.

The practical discipline is to run these checks before any number leaves the team, and to treat a failed check as a stop condition rather than a footnote in an appendix. What follows is not an attack. It is a set of assertions about relationships that must hold between attacks of differing strength, whatever the model is. If any assertion fails, the model may or may not be robust, but you have established something more useful: your evaluation is currently incapable of telling you which, and any number it produces is not reportable.

# Sanity checks to run before any robustness number leaves the building.
# Each assertion encodes a property that genuine robustness must satisfy.

clean         = accuracy(model, X)
single_step   = accuracy(model, single_step_attack(X, eps))
iterative_20  = accuracy(model, iterative_attack(X, eps, steps=20,  restarts=1))
iterative_200 = accuracy(model, iterative_attack(X, eps, steps=200, restarts=10))
transfer      = accuracy(model, transfer_attack(X, eps, surrogate))
huge_budget   = accuracy(model, iterative_attack(X, eps=100*eps, steps=200))

assert iterative_20  <= single_step     # more optimisation must not help the model
assert iterative_200 <= iterative_20    # more steps and restarts must not help the model
assert transfer      >= iterative_200   # white box must not be weaker than black box
assert huge_budget   <= 0.01            # an unbounded budget must reach total success

# A failed assertion does not mean the model is robust.
# It means the optimiser is being obstructed and the number is not reportable.

The adaptive attack principle

Now the deeper methodological point, and the one that generalises beyond images. A defence is proposed and evaluated against a fixed public attack suite at default hyperparameters, reporting high robust accuracy. The central objection is not that the suite is weak, nor that the confidence interval is loose. It is that those attacks were designed before the defence existed, so the evaluation measures resistance to attacks that could not have been aimed at it. A defence must be evaluated against an adaptive adversary: one who knows the defence is present, understands how it works, and rebuilds the loss function, the attack and its hyperparameters around it. Running the same suite with more random seeds tightens a confidence interval around a number that is measuring the wrong thing.

1. State the threat model firstFix the norm, the budget, the adversary's knowledge and the query budget before any number is produced. A robustness figure without these is uninterpretable and should not be quoted.
2. Run the sanity checksWhite box against transfer, single-step against iterative, and success at a very large budget. Each check encodes a property genuine robustness must satisfy, and a failure stops the evaluation.
3. Make the pipeline differentiableReplace non-differentiable or randomised components with differentiable approximations, and take expectations over randomness, so that the optimiser receives a usable signal rather than noise.
4. Attack the whole systemInclude detectors, preprocessing and input filters inside the attacked pipeline. If the defence works by detecting attacks, optimise jointly for misclassification and for evading detection.
5. Tune against this defenceSearch step sizes, iteration counts, restarts, initialisations and loss formulations specifically for this defence. Defaults in a public suite were tuned for other models.
6. Try hardest to break your own workThe strongest evidence a defence author can offer is a serious, documented, failed attempt to defeat their own defence, including everything tried.
7. Report the strongest attack foundReport the best attack, never the average over attacks. The maximum here is taken over attack methods, not over repeated runs of a single configuration, which is a separate point covered in the reporting module. List what was attempted and failed so the next person starts where you stopped.