Differential privacy: reading the guarantee literally

A data science lead tells the steering group that the model was trained with differential privacy at a given epsilon, so no individual's data can be recovered from it. Everyone relaxes. The statement is wrong in a way that matters, and correcting it precisely is one of the more valuable things a privacy engineer does, because differential privacy is a genuinely strong tool that is routinely oversold into a promise it never made.

The definition is about a mechanism, not about data. A randomised mechanism satisfies differential privacy if, for any two datasets that differ in a single record, the probability of the mechanism producing any given outcome, or any set of outcomes, changes by at most a multiplicative factor of e raised to epsilon. In plain terms: whether or not your record was included, what comes out looks almost the same, and how close to the same is controlled by epsilon. What that bounds is what an observer of the output can infer about your participation. It is a quantitative, tunable statistical guarantee, not a promise of absolute unrecoverability. A small epsilon is a strong bound; a large epsilon is a weak one, and there is no universal threshold above which a mechanism stops being differentially private, only a point at which the guarantee stops being worth much in practice.

Epsilon

The privacy loss parameter. It bounds how much the output distribution can shift when a single record is added or removed. Smaller is stronger.

Delta

In the relaxed form of the definition, the probability with which the epsilon bound is allowed to fail. It should be far below the reciprocal of the number of records, since a mechanism that publishes a few raw records at random can satisfy the definition for a large delta.

Unit of privacy

What the phrase "one record" refers to: a single training example, a single event, or everything contributed by one person. The guarantee applies to that unit and nothing larger.

Privacy budget

The total privacy loss an organisation is willing to accumulate across all releases derived from a dataset.

Composition: the budget is consumed, not renewed

The property that makes differential privacy analysable is also the one people forget operationally. Privacy loss composes. If you release two results from the same data, an adversary sees both, and under basic composition the epsilons add. Tighter accounting methods give sublinear growth in epsilon in exchange for a slightly larger delta, but the direction is always the same: more releases means a weaker overall guarantee. Repeatedly training, re-training after a data refresh, or answering a stream of queries while quoting the same per-release epsilon does not hold the guarantee constant. It quietly spends it. This is why a privacy budget has to be tracked centrally, per dataset, across teams, with an explicit decision about what happens when it is exhausted.

# Illustrative training config: the parameters that determine the guarantee
dp:
  unit_of_privacy: user        # not example: users contribute many rows
  clip_norm: 1.0               # per-example gradient clipping bound
  noise_multiplier: 1.1        # noise scaled to the clip norm
  target_delta: 1e-6           # far below the reciprocal of the dataset size
  accountant: enabled          # tracks cumulative epsilon across steps

budget:
  dataset: support-corpus-2026
  total_epsilon_allowed: 8.0
  spent: 3.4                   # prior training runs and released statistics
  on_exhaustion: block_new_releases

The mechanism most teams meet in practice clips each per-example gradient to a fixed norm, adds calibrated noise proportional to that norm, and lets an accountant track cumulative epsilon across training steps. Applying the mechanism to gradients is a perfectly standard and formally meaningful use: the claim that differential privacy only has meaning for aggregate query systems is false. It is also not true that the guarantee only holds if noise is applied to input data rather than gradients; both are valid mechanisms with different utility costs.

1. Fix the unit of privacy firstDecide whether one unit is an example, an event or a person. If individuals contribute many records, an example-level guarantee is much weaker than it sounds, and user-level accounting is what most privacy claims actually need.
2. Choose delta relative to dataset sizeDelta is the probability the bound simply fails. It must be far below the reciprocal of the number of records, otherwise a mechanism that releases a handful of raw records at random can satisfy the definition.
3. Select and configure the mechanismFor training, clip per-example gradients to a fixed norm and add noise scaled to that norm. The clip bound and the noise multiplier together determine both the guarantee and the utility cost.
4. Account for composition centrallyRun an accountant across training steps, and keep a per-dataset ledger across every release derived from it. Two teams each spending their own epsilon on the same data are spending one budget.
5. Decide what happens at exhaustionAgree in advance whether an exhausted budget blocks new releases, requires fresh data, or triggers an explicit risk acceptance. A budget with no stopping rule is a metric, not a control.
6. Record the justification, not just the numberWrite down why this epsilon is defensible for this dataset and audience, and what residual leakage it permits. The number alone is not interpretable by anyone reviewing you later.

The four honest limitations

First, population inference is not covered and was never meant to be. If a model learns that a certain combination of symptoms predicts a condition, that conclusion holds for everyone with those symptoms, including people whose records were never in the dataset. Differential privacy protects your participation, not your membership of a category. Someone can be exposed by a pattern the model legitimately learned, and no epsilon prevents that.

Second, group privacy degrades. The definition is stated over datasets differing in one record. If a single person contributed many records, or if a family, a household or a small team is correlated across records, the per-record guarantee does not automatically extend to them. For a group of a given size the effective epsilon scales roughly with that size under the pure definition, which is why choosing the unit of privacy is a substantive decision rather than a configuration detail.

Third, legal status does not follow. A differentially private training run does not make the training data legally anonymous, and it does not retrospectively supply a lawful basis for collecting data you should not have had. Those are separate questions about the input, and the guarantee is a statement about the output.

Fourth, memorisation is suppressed rather than abolished. At a meaningfully small epsilon, verbatim regurgitation becomes very unlikely, and this is one of the strongest practical arguments for the technique. But the claim that a differentially private model cannot memorise any training sequence at any epsilon is false: with a loose epsilon the bound permits substantial practical leakage. Whether a given epsilon is defensible for a given dataset is a judgement call that should be recorded, not a number to be quoted without context.