Self-attention is permutation-invariant; without position info “dog bites man” == “man bites dog”. Positional encoding adds the index back.

Data Flow

token embedding    e_t  (dim d)
positional vector  p_t  (dim d)

input to block     e_t + p_t

For sinusoidal:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Three flavours

1. Sinusoidal (fixed, original Transformer): each dimension is a sinusoid of a different frequency; model can extrapolate.
2. Learned: an Embedding(num_positions, d) trained with the model — more capacity, no extrapolation.
3. RoPE (Rotary): rotates Q and K by position-dependent angles; encodes relative position for free. Default for Llama-family.

Code (sinusoidal)

import torch, math
 
def pos_encoding(L, d):
    pe = torch.zeros(L, d)
    pos = torch.arange(0, L).unsqueeze(1).float()
    i   = torch.arange(0, d, 2).float()
    div = torch.exp(-(math.log(10000.0)) * i / d)
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe

Pitfalls

• Using absolute position when a relative one is needed.
• Mixing positional with token embeddings via concat instead of add — works but inflates d.

Analogy

Page numbers added to every paragraph so a binder still tells the original order even if you shuffle the pages.

Interview tip: “Can a Transformer generalise beyond training length?” — answer depends entirely on the positional encoding scheme.