Model Extraction and the Honest Cost of Every Defence

Model extraction uses query access to produce a model that reproduces the target's behaviour well enough to substitute for it. Note that recovering the exact weights is neither necessary nor usually the goal: functional equivalence on the input distribution the attacker cares about is sufficient to compete with you, to avoid paying you, or to attack you offline at leisure. That last consequence is often overlooked. A good extracted copy gives an attacker a white box stand-in against which to craft transferable evasion attacks, so extraction is a stepping stone as well as an end in itself.

A prediction API applies a strict per-key query rate limit as its extraction defence. Six months later a competitor ships a model whose outputs match it closely. The explanation is distributed extraction, and the underlying principle is worth stating precisely: a rate limit raises the cost of each query but never caps the total information released. The query budget is simply accumulated across many keys, many source addresses, many nominal customers and many months. Rate limiting prices extraction; it does not bound it. Two distractors to reject. First, no realistic rate limit sits above the number of queries needed. Be precise about why: exact parameter recovery with a calculable query budget has been demonstrated, but only for restricted architectures and specific layers, so this is not a general impossibility claim. For a deployed model of any size the practically relevant target is functional equivalence, and the query count for that is large, unbounded in advance, and trivially spread across accounts and months. Second, hard labels do not leak more per query than probability vectors, they leak strictly less.

Watermarking

A team trains its model to emit unusual outputs on a secret set of trigger inputs, and presents this watermark to the board as an anti-extraction control. The honest characterisation is that it is a post-hoc attribution mechanism rather than a preventive control. It does nothing to stop the copying; it is designed to support an ownership claim after a suspected copy has been found, by querying the suspect model with the trigger set and observing the signature. Its durability is limited: fine-tuning, pruning and distillation can weaken or remove the watermark behaviour, and an attacker who suspects a watermark can fine-tune specifically to disrupt it.

Three further claims are wrong in ways worth naming. It does not degrade a distilled copy's accuracy on the primary task, because extraction targets the primary behaviour, which the watermark deliberately leaves intact. It is not equivalent to output perturbation: perturbation reduces information in every response, whereas a watermark alters behaviour only on a rare trigger set. And it is not undetectable and unremovable merely because the trigger set is kept secret, since removal techniques such as fine-tuning and pruning do not require knowing the trigger. There is also an operational catch: exercising the claim requires the ability to query the suspected copy and a forum willing to accept the evidence.

Output perturbation

To slow extraction, a team perturbs the probability vectors its API returns while leaving the argmax unchanged. The honest description is that it lowers the information released per query, but degrades users who rely on calibrated scores while barely affecting an attacker who needs only labels. Consumers who set decision thresholds, rank candidates, route by confidence, abstain below a cut-off or feed the score into a cost-sensitive downstream decision are all harmed by noisy probabilities. An extractor training a surrogate on hard labels is inconvenienced by needing more queries, and no more than that. The seductive wrong answer is that preserving the argmax means there is no real trade-off, which confuses unchanged top-1 accuracy with unchanged utility. It is also not a formal lower bound on the number of queries an attack must issue: no such bound is established by adding noise to outputs.