Design: Content Moderation

The Problem

Design a system that reviews every piece of content uploaded to a social platform (text posts, images, videos, comments) and removes content that violates platform policies. The platform receives 5 million uploads per day. Content must be reviewed before or shortly after it becomes visible to other users.

High-Level Architecture

  User Uploads Content (Text/Image/Video)
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 1: AUTOMATED ML SCORING │  (~200ms)
  │ Text classifier + Image model │
  │ + Video frame sampler          │
  │ Output: Violation scores per  │
  │ policy category               │
  └───────────────────────────────┘
              │
     ┌────────┼────────┐
     ▼        ▼        ▼
  APPROVE   REVIEW    REMOVE
  (Low risk) (Medium)  (High confidence)
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 2: HUMAN REVIEW QUEUE   │
  │ Priority-ranked by severity   │
  │ and virality                   │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 3: APPEALS & FEEDBACK   │
  │ Users appeal removals         │
  │ Decisions feed back to model  │
  └───────────────────────────────┘

Layer 1: Automated ML Classification

Multiple specialized classifiers run in parallel:

Text Moderation

• Hate Speech Classifier: Detects slurs, targeted harassment, and dehumanizing language. Uses a fine-tuned language model trained on annotated examples.
• Spam/Scam Classifier: Detects promotional spam, phishing links, and scam patterns.
• Self-Harm/Violence: Detects content promoting self-harm or graphic violence.

Image Moderation

• NSFW Classifier: Detects nudity and sexually explicit imagery using a CNN or Vision Transformer.
• Violence/Gore Classifier: Detects graphic violence, weapons, and blood.
• OCR + Text Classifier: Extract text from memes and images, then run the text classifier on extracted text (catches hateful memes where text is embedded in images).

Video Moderation

• Frame Sampling: Extract key frames at regular intervals and run image classifiers.
• Audio Transcription: Transcribe audio track and run text classifiers on the transcript.

Each classifier outputs a probability score per policy category. A content item can violate multiple policies simultaneously.

Layer 2: Human Review

Content with medium confidence scores goes to human reviewers. The queue is prioritized by:

1. Severity: Content potentially involving child safety or terrorism is reviewed first.
2. Virality: Content being rapidly shared is reviewed before niche posts because the potential harm is greater.
3. User Reports: Content flagged by multiple users gets escalated.

Reviewers make a decision (approve, remove, or escalate to policy team), and their decisions become labeled training data for model improvement.

Layer 3: Appeals and Feedback Loop

Users whose content is removed can appeal. Appeals go to a separate review queue with a different reviewer to reduce confirmation bias. Successful appeals indicate model errors and are used to retrain the classifier.

Key Design Challenges

1. Multi-Modal Evasion: Users circumvent text filters by using images of text, Unicode character substitutions, or coded language. The system must combine signals across modalities.
2. Cultural Context: Satire, news reporting about violence, and educational content about hate speech are legitimate. The model must distinguish intent, which is very hard. This is why human review is essential for borderline cases.
3. Scale and Latency: 5 million daily uploads means roughly 60 items per second. ML scoring must happen quickly so harmful content is not visible for long.
4. Perceptual Hashing: When a harmful image is identified, compute its perceptual hash (pHash) and add it to a blocklist. All future uploads are checked against this hash database to catch reshares and slight modifications.

Say this out loud

A content moderation system runs specialized ML classifiers for text, images, and video to score every upload against policy categories. High-confidence violations are removed automatically. Medium-confidence items go to a priority-ranked human review queue. User appeals and reviewer decisions feed back into model retraining. Perceptual hashing catches reshared copies of known harmful content.

Followups to expect

1. How do you handle adversarial attacks on the moderation model? Users add invisible perturbations to images or use homoglyph characters in text. Use robust training with adversarial examples and ensemble multiple detection methods.
2. How do you measure moderation system quality? Track precision (what fraction of removed content was truly violating), recall (what fraction of violating content was caught), median time-to-action (how long harmful content stays visible), and appeal overturn rate.