The training loop of every neural network: compute the loss at the output, push the error backward through the layers using the chain rule, and step the weights in the negative-gradient direction.

Data Flow

   forward pass
      x --> L1 --> L2 --> ... --> Lk --> loss
                                          |
                                          v
                                   backward pass
                                          |
              g = d(loss)/d(W)            (chain rule)
                                          |
                                          v
                          W = W - lr * g

What it is

• Forward pass: feed x through the network to compute loss.
• Backward pass: differentiate loss w.r.t. every weight (chain rule).
• Update: W := W - lr * dL/dW.

Variants: SGD, Momentum, Adam (most common), AdamW (with weight decay), Lion.

Code

import torch
 
x  = torch.randn(4, 3)
y  = torch.tensor([1., 0., 0., 1.])
m  = torch.nn.Linear(3, 2)
opt = torch.optim.AdamW(m.parameters(), lr=1e-3)
 
for step in range(1000):
    pred = m(x)
    loss = torch.nn.functional.cross_entropy(pred, y)
    opt.zero_grad()
    loss.backward()                 # backprop
    opt.step()                      # gradient descent

Pitfalls

• Forgetting optimizer.zero_grad() -> gradients accumulate.
• Learning rate too high -> loss explodes; too low -> no progress.
• Vanishing / exploding gradients in deep nets — solved by residual connections, normalisation, careful init.

Analogy

Hiking down a foggy hill: every step feel the slope, step the opposite way, repeat until the valley.

Interview tip: Always state both directions: forward (compute output), backward (chain rule to compute gradients).