The Transformer (Vaswani et al., 2017) replaced RNN/LSTM by processing entire sequences in parallel, unlocking GPU scaling. Foundation of every modern LLM.

Data Flow

+------------------ INPUT ------------------+
|  tokens: [BOS, The, cat, sat, EOS] (ids)  |
+------------------------------------------+
                  |
                  v
    +-------------------------------+
    |  Token Embedding + Positional |
    +-------------------------------+
                  |
                  v
+---------------------------------------+
|  N x (Multi-Head Self-Attention ->   |
|       LayerNorm -> FeedForward ->    |
|       LayerNorm)                     |
+---------------------------------------+
                  |
                  v
      +-----------------------+
      |  Linear -> Softmax    |
      |  over vocabulary      |
      +-----------------------+
                  |
                  v
    next-token probability distribution

What is it?

A stack of identical blocks (attention + feed-forward) repeated N times. Each block is parallel over sequence and sequential over depth. The original 2017 paper had encoder + decoder halves; modern frontier LLMs (GPT-4, Llama 3, Claude, Mistral) are decoder-only — the recipe is identical, they just skip the encoder half.

Why it matters

• Any sequence-to-sequence task where you want to scale training.
• Replace RNNs to use thousands of GPUs at once.
• Attention captures long-range dependencies without vanishing gradients.

Pseudocode for ONE block

def block(x):
    a = layer_norm(x + self_attention(x))     # attention sublayer
    out = layer_norm(a + feed_forward(a))     # FFN sublayer
    return out

A full model is for _ in range(N): x = block(x) followed by a softmax head.

Pitfalls

• Quadratic cost in sequence length (O(L^2) attention). Mitigations: FlashAttention, sliding window, multi-query / grouped-query attention.
• Without positional encoding, the model sees a bag of tokens, not a sentence.

Analogy

Reading a whole paragraph at once and scribbling notes about which words relate to which, instead of word-by-word.

Interview tip: “How would you scale an RNN?” — answer: use a Transformer.