Privacy & Differential Privacy

Why Anonymization Fails

Removing names, social security numbers, and email addresses (traditional PII masking) is not enough to guarantee privacy.

Re-identification attacks combine anonymized datasets with public external databases to identify individuals:

$$\text{Anonymized Medical Records} + \text{Public Voter Registration Data} \implies \text{Identified Patient Identity!}$$

Differential Privacy (Dwork et al., 2006) provides a mathematical framework for privacy.

Dataset D (With Alice)    ──► [ ALGORITHM M ] ──► Output Distribution P1
Dataset D' (Without Alice)──► [ ALGORITHM M ] ──► Output Distribution P2
                    (P1 and P2 are nearly indistinguishable!)

Epsilon-Differential Privacy ($\epsilon$-DP)

A randomized algorithm $\mathcal{M}$ satisfies $\epsilon$-Differential Privacy if for any two neighboring datasets $D$ and $D'$ differing by a single individual's record, and for any output subset $S$:

$$P(\mathcal{M}(D) \in S) \le e^{\epsilon} \times P(\mathcal{M}(D') \in S)$$

┌──────────────────────────┬──────────────────────────┐
│ SMALL EPSILON (ε -> 0)   │ LARGE EPSILON (ε >> 1)   │
├──────────────────────────┼──────────────────────────┤
│ STRONGER Privacy!        │ WEAKER Privacy!          │
│ High noise added.        │ Low noise added.         │
│ Lower model utility/acc. │ Higher model accuracy.   │
└──────────────────────────┴──────────────────────────┘

The parameter $\epsilon$ represents your Privacy Loss Budget.

Differentially Private SGD (DP-SGD)

To train deep neural networks with differential privacy, use DP-SGD (Abadi et al., 2016):

Compute Per-Sample Gradients ──► Clip Per-Sample Gradients (Max Norm C) ──► Add Gaussian Noise ──► Update Weights

1. Per-Sample Gradient Clipping: Clip each individual sample gradient to maximum norm $C$ to bound the maximum influence of any single training sample.
2. Noise Addition: Add calibrated Gaussian noise to the averaged gradient before updating model parameters.
3. Privacy Accountant: Track cumulative $\epsilon$ spending across training epochs using Renyi Differential Privacy.

Say this out loud

Differential Privacy provides mathematical privacy guarantees by bounding how much a single individual's data can influence outputs. The privacy loss budget epsilon controls the trade-off between privacy strength and model utility. DP-SGD clips per-sample gradients and adds Gaussian noise during training to guarantee privacy against reconstruction attacks.

Followups to expect

1. What is Local vs Global Differential Privacy? Global DP adds noise at a centralized server level after collecting raw data. Local DP adds noise on user devices before sending data to servers, protecting data even if central servers are compromised.
2. What is Privacy Budget Exhaustion? Every query or training epoch consumes a portion of the privacy budget epsilon. Once the budget is exhausted, no further queries or retraining runs can be performed on that dataset.