NLP & Transformersintermediatemust-know5 min

The Attention Mechanism

How dynamic weighting allows models to focus on relevant context across long sequences.

The Attention Mechanism (Bahdanau et al., 2014; Vaswani et al., 2017) allows neural networks to dynamically weight and focus on relevant parts of an input sequence. Instead of compressing an entire sequence into a single static context vector, Attention computes dynamic alignment scores between Query, Key, and Value vectors. Scaled Dot Product Attention calculates Attention(Q, K, V) = Softmax( Q K^T / sqrt(d_k) ) V, forming the core foundation of modern Transformer language models.

Read the deep dive →
NLP & Transformersintermediatemust-know5 min

Transformer Architecture

How parallel self attention blocks replaced recurrent loops to power modern AI.

The Transformer Architecture (Vaswani et al., 2017 - Attention Is All You Need) revolutionized artificial intelligence. By replacing sequential recurrent loops with parallel self attention mechanisms, Transformers allow GPUs to process entire text sequences simultaneously in parallel. The architecture consists of an Encoder stack (Multi Head Self Attention + Feed Forward layers) and a Decoder stack (Causal Masked Self Attention + Cross Attention + Feed Forward layers), connected by residual skip connections and layer normalization.

Read the deep dive →
NLP & Transformersintermediatemust-know5 min

Tokenization & BPE

Converting raw text strings into numerical subword tokens using Byte Pair Encoding.

Tokenization breaks raw text strings into numerical token IDs that neural networks can process. Character level tokenization creates tiny vocabularies but long sequences. Word level tokenization creates huge vocabularies with frequent out of vocabulary errors. Byte Pair Encoding (BPE) is a subword tokenization algorithm that iteratively merges the most frequent pairs of adjacent bytes or characters. BPE balances vocabulary size and sequence length, handling rare words and code efficiently without out of vocabulary errors.

Read the deep dive →
NLP & Transformersintermediatemust-know5 min

BERT vs GPT: Encoder vs Decoder

Contrasting bidirectional encoder understanding models against causal decoder autoregressive generation models.

BERT and GPT represent the two foundational paradigms of modern Transformer models. BERT (Devlin et al., 2018) is an Encoder Only model trained with Masked Language Modeling to look at past and future context simultaneously, excelling at text understanding, classification, and search embeddings. GPT (Radford et al., 2018) is a Decoder Only model trained with Causal Next Token Prediction to process text left to right, excelling at open ended text generation.

Read the deep dive →
NLP & Transformersadvancedmust-know5 min

The KV Cache

Trading GPU VRAM memory for compute by caching historical Key and Value tensors during autoregressive generation.

The Key-Value (KV) Cache avoids redundant self-attention re-computation during autoregressive LLM token generation. Because historical tokens do not change during causal generation, their Key ($K$) and Value ($V$) projection matrices are computed once and stored in GPU VRAM. At step $t$, the LLM computes $Q, K, V$ for token $t$ only, appends $K_t, V_t$ to the KV cache, and evaluates attention over cached keys and values. This reduces generation time complexity per token from $O(N^2)$ to $O(N)$, but creates a massive GPU VRAM memory bottleneck.

Read the deep dive →
NLP & Transformersbeginner5 min

Word2Vec, GloVe & Embeddings

Mapping text words into dense continuous vector spaces where geometric distance reflects semantic meaning.

Word Embeddings represent discrete text words as dense continuous vector spaces (typically 100 to 300 dimensions). Instead of sparse high dimensional one hot vectors, word embeddings position semantically similar words close together in vector space. Word2Vec (Mikolov et al., 2013) learned embeddings using self supervised local context prediction (CBOW and Skip Gram). GloVe (Pennington et al., 2014) combined local context with global matrix co occurrence statistics.

Read the deep dive →
NLP & Transformersbeginner4 min

Text Classification Pipelines

Building production pipelines to categorize unstructured text into predefined labels.

Text Classification assigns categorical labels to raw text documents (sentiment analysis, spam detection, intent routing). Pipelines evolved from classical TF-IDF with Logistic Regression or Naive Bayes to fine tuned BERT models and zero shot LLM prompts. Production pipelines require robust text cleaning, handling class imbalance, threshold calibration, and monitoring for domain drift.

Read the deep dive →
NLP & Transformersbeginner5 min

TF-IDF & BM25

Scoring keyword relevance in lexical search engines from TF-IDF to Okapi BM25.

TF-IDF and BM25 are foundational lexical keyword search algorithms. TF-IDF weights words by Term Frequency (how often a word appears in a document) multiplied by Inverse Document Frequency (how rare the word is across the corpus). Okapi BM25 improves TF-IDF by adding Term Frequency Saturation and Document Length Normalization, serving as the default retrieval algorithm in Elasticsearch and Lucene.

Read the deep dive →
NLP & Transformersbeginner4 min

Stemming, Lemmatization & Stopwords

Cleaning raw text data for traditional NLP models from stopword filtering to lemmatization.

Text Preprocessing prepares raw natural language text for machine learning algorithms. Traditional NLP pipelines remove noise using Lowercasing, Stopword Removal (filtering out common filler words like 'the' or 'is'), Stemming (chopping word suffixes using heuristic rules), and Lemmatization (mapping words to valid dictionary roots). Modern Transformer language models use raw subword tokenization (BPE), rendering manual text preprocessing obsolete.

Read the deep dive →
NLP & Transformersintermediate4 min

Self-Attention vs Cross-Attention

Comparing internal sequence contextualization against cross sequence information retrieval.

Self Attention and Cross Attention are the two primary attention operations in Transformer models. In Self Attention, Queries, Keys, and Values all derive from the exact same input sequence, allowing tokens to attend to other tokens within the same sequence. In Cross Attention, Queries come from one target sequence (like a Decoder), while Keys and Values come from a separate source sequence (like an Encoder or text prompt).

Read the deep dive →
NLP & Transformersintermediate4 min

Why Multi-Head Attention

Splitting attention into multiple parallel heads to capture distinct semantic relationships simultaneously.

Multi Head Attention (Vaswani et al., 2017) splits Query, Key, and Value projections into h parallel subspace heads. Instead of computing a single averaged attention score, Multi Head Attention allows the network to simultaneously attend to information from different representation subspaces at different positions. The outputs of all h attention heads are concatenated together and linearly projected back to the original model dimension.

Read the deep dive →
NLP & Transformersintermediate4 min

Why Divide by √d_k

Why dividing dot product attention scores by the square root of key dimension size prevents vanishing gradients.

Scaled Dot Product Attention divides raw Query Key dot products by the square root of key dimension size sqrt(d_k). As vector dimension d_k grows large, vector dot products grow large in magnitude, pushing Softmax outputs into extreme zero gradient regions. Scaling by 1 / sqrt(d_k) keeps dot product variance stable at 1.0, preserving healthy gradients during backpropagation.

Read the deep dive →
NLP & Transformersintermediate4 min

Causal Masking

Hiding future text tokens to enforce autoregressive next token prediction in language models.

Causal Masking (Look Ahead Masking) ensures that autoregressive language models (GPT, LLaMA) only attend to past and current tokens during training. By setting upper triangular attention scores above the diagonal to minus infinity before computing Softmax, future tokens receive zero attention weight. This allows Transformer decoders to process entire training text sequences in parallel while preventing information leakage from future tokens.

Read the deep dive →
NLP & Transformersintermediate4 min

The Feed-Forward Block

How the two layer dense block inside Transformers acts as a key value factual memory store.

The Feed Forward Network (FFN) block is a two layer point wise dense network applied to each token independently in a Transformer block. While Multi Head Attention routes context between tokens, the FFN processes and transforms features within each token position. Research shows FFN layers act as associative key value memories, storing factual knowledge learned during pretraining. Modern architectures replace standard FFN blocks with SwiGLU gated activations or Mixture of Experts (MoE) routing.

Read the deep dive →
NLP & Transformersintermediate5 min

Tokenizer Pitfalls (Numbers, Code, Unicode)

Why subtle tokenization quirks degrade LLM math, coding, and multilingual performance.

Tokenizer Pitfalls describe how subword tokenization artifacts negatively impact Large Language Model behavior. Issues include poor math reasoning due to arbitrary digit splitting, inefficient code indentation tokenization, trailing space sensitivity, and multilingual byte inflation. Understanding tokenizer quirks explains why LLMs fail simple character counting tasks like counting letters in strawberry.

Read the deep dive →
NLP & Transformersintermediate4 min

Static vs Contextual Embeddings

Why modern NLP replaced static dictionary lookup vectors with dynamic context aware transformer embeddings.

Static Embeddings (Word2Vec, GloVe) assign a single fixed vector to each word regardless of surrounding sentence context. Contextual Embeddings (BERT, RoBERTa) generate dynamic feature vectors where a word representation changes based on surrounding sentence context. Contextual embeddings resolve polysemy, distinguishing between 'bank' of a river and 'bank' for money.

Read the deep dive →
NLP & Transformersintermediate4 min

Masked vs Causal Language Modelling

Comparing bidirectional blank filling against unidirectional next token prediction.

Masked Language Modeling (MLM) and Causal Language Modeling (CLM) are the two core self supervised pretraining paradigms in NLP. Masked Language Modeling (BERT) hides random tokens in an input sequence and trains the model using bidirectional context to fill in the missing words. Causal Language Modeling (GPT) hides future tokens using causal masking and trains the model left to right to predict the single next token.

Read the deep dive →
NLP & Transformersintermediate4 min

NER & Sequence Labelling

Extracting structured real world entities like names, locations, and dates from unstructured text strings.

Named Entity Recognition (NER) is a core Information Extraction task that locates and classifies named entities in unstructured text into predefined categories (Person, Organization, Location, Date). NER is framed as a Token Level Sequence Labelling task using the BIO Tagging format (Begin, Inside, Outside). Architectures evolved from Conditional Random Fields (CRF) and BiLSTM-CRF to fine tuned BERT token classification heads.

Read the deep dive →
NLP & Transformersintermediate5 min

Topic Modelling & LDA

Discovering hidden thematic topics across large collections of unlabelled text documents.

Topic Modelling is an unsupervised NLP technique that discovers abstract themes (topics) in text collections. Latent Dirichlet Allocation (LDA - Blei et al., 2003) is a generative statistical model assuming documents are mixtures of topics, and topics are mixtures of words. Modern neural approaches (BERTopic) combine Transformer embeddings with UMAP dimensional reduction and HDBSCAN clustering to produce coherent topics.

Read the deep dive →
NLP & Transformersintermediate5 min

Machine Translation

Translating text across human languages from phrase based statistical rules to neural Transformers.

Machine Translation (MT) converts source language text into target language text while preserving semantic meaning. Statistical Machine Translation (SMT) relied on phrase tables and language models. Neural Machine Translation (NMT) replaced hand-crafted rules with end-to-end sequence to sequence models (Encoder-Decoder LSTMs and Transformers). Evaluation relies on automatic overlap metrics (BLEU, chrF) and neural quality estimation models (COMET).

Read the deep dive →
NLP & Transformersintermediate5 min

Summarization & ROUGE/BLEU

Evaluating text generation quality using ROUGE recall, BLEU precision, and neural LLM judge metrics.

Text Summarization models are evaluated using automated n-gram overlap metrics (ROUGE and BLEU) and modern neural metrics. BLEU measures n-gram Precision (used mainly in translation). ROUGE measures n-gram Recall (used mainly in summarization to check if key facts were included). ROUGE-1 measures unigram overlap, ROUGE-2 measures bigram overlap, and ROUGE-L measures Longest Common Subsequence.

Read the deep dive →
NLP & Transformersintermediate5 min

Extractive vs Abstractive QA

Comparing span extraction from source text against generative abstractive answer synthesis.

Question Answering (QA) algorithms extract or generate answers to user queries given context documents. Extractive QA models (BERT on SQuAD) locate the exact start and end character span tokens within a passage to extract text answers directly. Abstractive QA models (Generative LLMs) read source documents and synthesize fluent natural language answers in new words. Modern production systems combine vector search retrieval (RAG) with abstractive LLM generators.

Read the deep dive →
NLP & Transformersintermediate5 min

Sentence Embeddings & Sentence-BERT

Mapping full sentences into dense vector spaces for semantic search and retrieval augmented generation.

Sentence Embeddings represent entire sentences or text paragraphs as single dense vectors. Standard BERT requires passing every pair of sentences through cross-attention, making 10,000 sentence comparisons take 65 hours. Sentence-BERT (SBERT - Reimers & Gurevych, 2019) uses a Siamese Network architecture to pre-compute sentence vectors offline, allowing 10,000 sentence similarity comparisons in sub-10ms via cosine distance.

Read the deep dive →
NLP & Transformersintermediate5 min

Beam Search & Decoding Strategies

Finding high-probability output sequences by tracking top-K partial paths during text generation.

Beam Search Decoding is a heuristic search algorithm used in seq2seq models, machine translation, and summarization to discover sequence outputs that maximize overall joint probability. Unlike Greedy Decoding (which picks the single highest-probability token at each step, missing better global sequences), Beam Search maintains a fixed number of B candidate hypotheses (Beam Width B). At each step, it expands all B paths into V next-token candidates, selects the top B highest cumulative log-probability paths, and repeats until end-of-sequence [EOS] tokens are generated.

Read the deep dive →
NLP & Transformersintermediate4 min

Perplexity

Measuring how surprised a language model is when observing a sequence of text.

Perplexity (PPL) is the standard intrinsic evaluation metric for language models. It is exponentiation of the average negative log-likelihood (Cross-Entropy Loss) per token: PPL = exp(Cross-Entropy). Intuitively, Perplexity represents the effective branching factor: a Perplexity of 10 means the model is as confused at each step as if it were choosing uniformly among 10 equally likely words. Lower Perplexity indicates a better-fitting model with higher predictive confidence.

Read the deep dive →
NLP & Transformersadvanced5 min

Positional Encodings & RoPE

Injecting word order awareness into permutation invariant self attention mechanisms.

Positional Encodings inject token order information into Transformer models. Because self attention computes pairwise similarities without regard to word sequence order, Transformers require explicit position signals. Original Transformers used Absolute Sinusoidal Positional Encodings. Modern Large Language Models use Rotary Position Embedding (RoPE), which rotates Query and Key vectors in complex 2D planes to naturally encode relative distance between tokens.

Read the deep dive →
NLP & Transformersadvanced5 min

Grouped-Query & Multi-Query Attention

Slashing KV cache memory consumption during LLM inference using shared key value attention heads.

Grouped Query Attention (GQA - Ainslie et al., 2023) and Multi Query Attention (MQA - Shazeer, 2019) are memory efficient attention variants for Large Language Models. Standard Multi Head Attention (MHA) maintains separate Key and Value heads for every Query head, causing massive KV cache memory bloat during inference. MQA uses a single shared Key and Value head across all Query heads. GQA strikes an optimal middle ground by grouping Query heads to share a smaller number of KV heads (such as 8 Query heads per 1 KV head), preserving accuracy while reducing KV cache VRAM by 8x.

Read the deep dive →
NLP & Transformersadvanced4 min

Why Attention Is O(n²)

Understanding why pairwise dot product comparisons cause self attention memory and compute to scale quadratically with sequence length.

Standard Self Attention has quadratic time and memory complexity O(N^2) with respect to sequence length N. To compute attention scores, every token Query must calculate pairwise dot products with every Key token, generating an N x N matrix. As sequence length doubles from 4,000 to 8,000 tokens, attention compute and memory requirements increase by 4 times, creating a major barrier for long context windows.

Read the deep dive →
NLP & Transformersadvanced5 min

FlashAttention & Efficient Attention

Accelerating transformer attention by 2 to 4 times while reducing GPU memory from quadratic to linear using IO aware tiling.

FlashAttention (Dao et al., 2022) is an IO aware exact attention algorithm designed for modern GPU memory architectures. Standard attention writes massive N x N intermediate matrices to slow High Bandwidth Memory (HBM). FlashAttention tiles Query, Key, and Value matrices into small blocks that fit inside fast GPU SRAM memory, computing Softmax online without ever materializing the full N x N matrix in HBM. FlashAttention cuts VRAM usage from quadratic O(N^2) to linear O(N) and accelerates wall clock training speed by 2 to 4 times.

Read the deep dive →
NLP & Transformersadvanced5 min

Linear Attention, Mamba & SSMs

Replacing quadratic self attention with linear time State Space Models for million token context processing.

State Space Models (SSMs - Gu et al., 2023 / Mamba) offer a linear time O(N) alternative to quadratic O(N^2) Transformer attention. Derived from continuous control theory, SSMs map continuous input signals x(t) to output y(t) through a hidden state h(t). Mamba introduces Selective State Space Models, making state transition matrices data dependent to dynamically filter irrelevant information. Mamba achieves constant O(1) inference time per step and linear O(N) training complexity, processing million token context sequences efficiently.

Read the deep dive →
SCROLL · SAVE · TAP TO GO DEEPER