Membership Inference, Memorisation and Their Mitigations

Membership inference asks a narrow question with wide consequences: was this specific record in the training set? The consequence is that in many settings the membership fact is itself the sensitive fact. Confirming that a person's record was in a corpus assembled from patients with a particular condition, or from customers who defaulted, discloses the condition or the default, regardless of what else the model reveals. This makes membership inference the canonical privacy attack against models, and the one most often dismissed on bad grounds.

The bad ground looks like this. A model shows a train versus test accuracy gap under one percent, and the team concludes that membership inference poses no risk. The reasoning breaks down in a specific place: the gap is an average over the distribution, and leakage is not uniformly distributed. Typical, well-represented records are indeed hard to distinguish. Outliers, rare records and records that appear more than once in the corpus are memorised far more strongly, and those are precisely the individuals whose exposure matters most. A model can generalise beautifully on average while retaining a sharply identifiable trace of the unusual cases.

Membership inference

Determining whether a particular record was part of the training set, typically by exploiting the model being more confident or lower-loss on data it has seen.

Model inversion

Reconstructing representative or approximate input data for a class or an individual from model access, rather than merely testing membership.

Training data extraction

Recovering verbatim or near-verbatim fragments of training data from a generative model, which is memorisation surfacing directly in output.

Attribute inference

Inferring an unknown sensitive attribute of a subject from partial knowledge plus model access.

Property inference

Inferring aggregate properties of the training corpus, such as the proportion of records with some characteristic, which can itself be commercially or legally sensitive.

Mitigations and what each one costs

Deduplication of the training corpus is the cheapest meaningful step, because repetition drives memorisation strongly and duplicates are usually accidental. Limiting overfitting through early stopping, regularisation and appropriate capacity helps at the margin but, as above, does not address concentrated leakage on outliers. Restricting output detail, refusing to return raw loss or confidence, raises attacker effort without changing the underlying property. Filtering generated output against the training corpus catches verbatim regurgitation, but only the verbatim kind.

Training with differential privacy is the only mitigation that offers a formal guarantee: it bounds how much any single unit of data can influence the resulting model, which in turn bounds how well any attack can distinguish a model trained with that record from one trained without it. It is genuinely different in kind from the heuristics above, and it deserves the following caveats stated plainly. The guarantee is expressed through a privacy parameter whose practical interpretation at commonly used values is contested among practitioners, so a stated parameter is not self-evidently a strong promise. It costs accuracy, sometimes substantially, and the cost falls hardest on exactly the rare subgroups whose leakage motivated it. And it protects at the granularity you defined: if the unit of protection is a record but a single person contributed hundreds of records, record-level protection does not deliver person-level protection.