Membership Inference & Memorisation

What is Model Memorization?

Deep neural networks possess massive capacity. Instead of learning generalizable patterns, models can memorize specific individual training samples word for word:

Training Sample:  "Social Security Number for John Doe is 123-45-6789"
                                      │
                                      ▼
             [ MODEL MEMORIZES EXACT TEXT SEQUENCE ]
                                      │
                                      ▼
Inference Prompt: "Social Security Number for John Doe is..." ──► Model Completes: "123-45-6789"!

Unintended memorization creates severe data privacy breaches when models are trained on private email archives, medical records, or financial histories.

Membership Inference Attacks (MIA)

A Membership Inference Attack (Shokri et al., 2017) attempts to answer a binary question:

$$\text{"Was Target Record } x^* \text{ used to train Model } \mathcal{M}?"$$

Target Record x* ──► [ TRAINED MODEL M ] ──► Loss / Confidence Vector ──► [ SHADOW MODEL ] ──► In Training Set? (YES/NO)

Why Attacks Succeed: Confidence Gaps

Models exhibit different confidence and loss distributions on training data versus unseen data:

• Member Sample (In Training Set): Extremely low loss, high confidence probability ($p = 0.999$).
• Non-Member Sample (Unseen Data): Higher loss, moderate confidence probability ($p = 0.650$).

An attacker trains a secondary Shadow Model to distinguish member loss distributions from non-member loss distributions, achieving high attack accuracy!

Privacy Implications

If a model trained on a private HIV Patient Dataset is subjected to a membership inference attack, proving that an individual's medical record is in the training set reveals their positive medical diagnosis instantly!

Defense Strategies against MIA

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. REGULARIZATION        │ 2. DIFFERENTIAL PRIVACY  │ 3. OUTPUT TEMPERATURE    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Apply Dropout, L2 Weight │ Train using DP-SGD.      │ Soften output logits and │
│ Decay, and Early Stopping│ Mathematically bounds    │ return rounded top-K     │
│ to prevent overfitting!  │ sample influence!        │ probabilities only!      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Reduce Overfitting: Overfitting is the primary driver of membership leakage. Applying L2 regularization, Dropout, and Early Stopping narrows the confidence gap between train and test data.
2. Differentially Private Training (DP-SGD): DP-SGD adds noise during gradient descent, placing a mathematical upper bound on membership inference attack success rates.
3. Restrict API Output Precision: Avoid returning raw full precision floating point probability vectors. Return top K class labels or rounded probabilities.

Say this out loud

Membership inference attacks determine whether a specific data record was part of a model's private training dataset. Overfitted models exhibit lower loss and higher confidence on training samples compared to unseen data, leaking membership signals. Applying L2 regularization, Dropout, and DP-SGD prevents memorization and defends against membership attacks.

Followups to expect

1. What is Model Inversion Attack? An attack that reconstructs average representative features or images of an entire target class from model output probabilities.
2. What is the Exposure Metric (Carlini et al.)? A metric measuring how easily a memorized secret string can be extracted from a language model compared to random candidate strings.