What the model remembers: memorisation, extraction and inference attacks

Researchers probing a production language model with a customer-service prefix get back a complete, correct postal address and phone number. Checking the fine-tuning corpus confirms that exact string appeared in it. This is not a quality bug. It is a disclosure, and the label matters because it determines who has to be told. Hallucination is the wrong label precisely because the output is accurate and traceable to a training record: a fabricated address harms nobody in particular, while a recalled one identifies a real person. Retrieval leakage is also the wrong label, because nothing was retrieved: the sequence came out of the weights.

Memorisation is the retention of specific training sequences in a form that can be induced to reappear. It matters most where you would least like it to be: on rare, high-entropy strings such as addresses, account numbers, keys and unusual names. Those are sequences a model cannot reconstruct from general patterns, so verbatim emission of one is evidence of storage rather than generalisation, and it is a disclosure about a specific person. Duplication in the corpus is the single most reliable amplifier. A record that appears once may be recoverable; a record that appears in fifty scraped copies of the same page is far more likely to be. Aggressive near-duplicate removal is therefore a privacy control, not just a compute saving.

How extraction actually works

A practical extraction attack does not ask the model to reveal a secret. It samples: the attacker generates a large volume of completions from many prefixes, then filters the output for candidates that look memorised, typically by looking for sequences the model assigns unusually high confidence relative to what a reference model would assign. The attacker then verifies candidates against any accessible ground truth. The important consequence is that extraction scales. It is a pipeline, not a lucky prompt, and defending it with a filter that blocks obvious requests misses the mechanism entirely.

Membership inference and the generalisation gap

Membership inference asks a narrower question: was this specific record in the training set? An adversary with query access to a clinical classifier takes a named individual's record, observes how the model behaves on it, and compares that behaviour against a distribution built from records known not to be members. If members and non-members are distinguishable, membership can be inferred. In a clinical setting, membership alone can be the sensitive fact: being in the training set for a model built from patients with a particular condition discloses that condition.

The enabler is the generalisation gap: the model behaves measurably differently on data it trained on than on comparable unseen data. Everything else is secondary. The size of the training set is not the driver, because a large but heavily overfitted model can leak more than a small, well-regularised one. Holding the weights helps an attacker but is not required; query access is enough. And suppressing probability scores is a speed bump rather than a fix, because attacks exist that operate on hard labels alone, probing how far a record sits from a decision boundary. The controls that genuinely reduce the gap are the ones that reduce overfitting: regularisation, early stopping, deduplication, and where the utility cost is acceptable, training under a formal privacy mechanism.

Synthetic data does not escape this

A vendor offering a synthetic dataset generated from real patient records, described as carrying no privacy risk because no synthetic row corresponds to a real person, is making an argument about form rather than about information. Synthetic data inherits the privacy properties of the generator that produced it. If that generator memorised an outlier, the synthetic release can contain a near-verbatim copy of that outlier, and membership inference against the release is a live attack. The absence of an exact one-to-one row mapping, and a different row count, are not evidence of protection. What you should require is evidence about the generator itself: a formal guarantee where one is claimed, together with empirical tests for near-duplicate outliers and membership inference run against the actual release. Rejecting synthetic data outright is the opposite error: it is a legitimate technique when its properties are evidenced.