Design: Spam & Abuse Detection

The Problem

Design a system that detects spam messages, fake accounts, and abusive behavior on a messaging platform with 500 million messages sent per day. The system must catch spam before recipients see it, while minimizing false positives that block legitimate messages.

High-Level Architecture

  Message or Account Action
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 1: REAL-TIME FILTERS    │  (~5ms)
  │ URL blocklist, rate limiting  │
  │ Known spam hashes             │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 2: ML CONTENT SCORING   │  (~20ms)
  │ Text classifier + Image model │
  │ Sender reputation score       │
  └───────────────────────────────┘
              │
     ┌────────┼────────┐
     ▼        ▼        ▼
  DELIVER   REVIEW    BLOCK
              │
              ▼
  ┌───────────────────────────────┐
  │ LAYER 3: GRAPH ANALYSIS       │  (Async)
  │ Detect coordinated campaigns  │
  │ Cluster linked fake accounts  │
  └───────────────────────────────┘

Layer 1: Real-Time Deterministic Filters

Before running ML models, apply fast rule-based checks:

• URL Blocklist: Block messages containing known phishing or malware URLs.
• Rate Limiting: Flag accounts sending more than 50 messages per minute.
• Hash Matching: Compare message content hashes against a database of previously identified spam messages. If a match is found, block immediately.
• New Account Restrictions: Limit message volume for accounts created less than 24 hours ago.

Layer 2: ML Classification

For messages that pass the rule filters, run ML models:

Content Features

• Text embedding from a fine-tuned language model.
• Presence of promotional language patterns ("limited time offer", "click here now").
• Link analysis (shortened URLs, redirect chains, domain reputation).
• Image OCR text extraction (spam embedded in images).

Behavioral Features

• Messages sent per hour by this sender.
• Fraction of messages sent to strangers (no prior conversation).
• Response rate (what percentage of recipients reply, real users get replies).
• Account age and profile completeness.

Sender Reputation Score

Maintain a rolling reputation score for each account:

• Starts at a neutral value for new accounts.
• Increases with legitimate conversations, replies received, and account age.
• Decreases with user reports, spam classifications, and blocked messages.
• Accounts below a reputation threshold get all messages reviewed automatically.

Layer 3: Graph-Based Campaign Detection

Sophisticated spam operations use networks of coordinated fake accounts. Detect these asynchronously:

• Cluster accounts that share IP addresses, device fingerprints, or registration patterns.
• Analyze message graphs: If 50 accounts all send the same message template to the same set of targets within 1 hour, that is a coordinated campaign.
• Network propagation: When one account in a cluster is confirmed as spam, investigate and flag all connected accounts.

Key Design Challenges

1. False Positive Sensitivity: Blocking a legitimate business message or a real user's message is very damaging to user trust. Tune thresholds conservatively and provide easy unblock/appeal mechanisms.
2. Adversarial Evolution: Spammers test their messages against detection systems and iterate. Your model degrades within days of deployment. Retrain frequently using the latest spam samples.
3. Multi-Language Support: Spam exists in every language. Train language-agnostic models or maintain per-language classifiers.
4. Privacy Constraints: Reading message content for spam detection raises privacy concerns. Design systems that scan content on-device when possible or use privacy-preserving techniques.

Say this out loud

A spam detection system uses layered defenses. Real-time filters catch known spam via URL blocklists, rate limits, and content hashes. ML classifiers score message content and sender behavior. Asynchronous graph analysis detects coordinated fake account campaigns. Sender reputation scores track account trustworthiness over time.

Followups to expect

1. How do you handle spam in encrypted messaging? When end-to-end encryption prevents server-side content analysis, rely on metadata signals (message velocity, recipient patterns) and client-side reporting by recipients.
2. How do you build training data for spam detection? Combine user reports (users flag spam), honeypot accounts (decoy accounts that attract spam), manual labeling by trust and safety teams, and automated clustering of similar messages.