ML & DL Foundations — Interview Q&A

Don’t skip this. The strongest candidates connect theory to engineering judgement. Knowing how a Transformer is built and how attention works gives you the lens every production engineer uses to debug.

If you can’t explain the backward pass, you’re at risk of getting wrecked in elite interviews.

Q1 — Explain the Transformer architecture in simple terms.

What? A neural network where every token can attend to every other token via self-attention. The original 2017 Google paper replaced RNNs/LSTMs by enabling fully parallel sequence modelling.

Why? Because every modern LLM is a Transformer. Understanding self-attention unlocks every downstream LLM behaviour (long context, flash attention, KV cache).

How? Mental model:

1. Tokenise the input (BPE for GPT, sentencepiece for others).
2. Embed each token; add positional encoding (so word-order is preserved).
3. For each layer: self-attention → feed-forward → normalisation.
4. Repeat L layers.
5. Output head un-embedding back to vocabulary; softmax → token probabilities.

3 Scenarios

Scenario 1 — Generation comparison: LSTM vs Transformer LSTM: tokens processed in order, gradient signal weakens over hundreds of steps. Transformer: every token attends to every other in one matrix multiply. Transformers scale; LSTMs don’t.

Scenario 2 — Long-context vs short-context behaviour Short context: every token gets to see everything; relevant signal is dense. Long context: most tokens contribute noise; attention has to learn to focus. Production implication: long-context models are more expensive AND noisier.

Scenario 3 — Multi-head attention Each head learns a different “kind” of relationship (semantic, syntactic, positional). Concatenate outputs. Allows the model to attend to many aspects simultaneously.

Q2 — What is attention and why is it important?

What? A weighted sum where token A decides how much of token B’s information to include in its own representation, with weights learned.

Why? Because attention lets the model decide what matters for each token dynamically — a core reason LLMs generalise so well.

How? Scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / sqrt(d_k)) · V

Q (query), K (key), V (value) are linear projections of token embeddings. The softmax weights, normalised by sqrt(d_k) to keep gradients stable, decide how much each token contributes. Result: every token’s representation is now a context-aware blend.

3 Scenarios

Scenario 1 — “The cat sat on the mat” — what does “sat” attend to? It should attend heavily to “cat” (the subject), moderately to “the” (definite-article context), lightly to “mat” (the place). Self-attention learns these weights from data.

Scenario 2 — Masked language modelling (BERT) Forward-only attention: tokens can attend leftward and rightward. Used for classification, sentence-pair scoring.

Scenario 3 — Causal language modelling (GPT) Causal mask: token n only attends to tokens 1..n. Used for generation; same path on training and inference.

Q3 — How does gradient descent work? What are its variants?

What? An optimisation algorithm that nudges model weights in the direction that decreases the loss function, one step at a time.

Why? Because every neural net learns this way. Variants change the per-step update rule to be faster, more stable, or more generalising.

How? Basic loop:

1. Compute loss on a batch: L = f(y_true, y_pred).
2. Compute gradient ∂L/∂W for each weight.
3. Update: W = W - lr * ∂L/∂W.

Variants:

• Batch GD: uses whole dataset per step. Stable but slow.
• Stochastic GD (SGD): one example per step. Noisy but fast.
• Mini-batch GD: ~32-256 examples per step. Best of both.
• Momentum: keeps velocity; smooths local bumps.
• Adam: per-parameter adaptive learning rate + momentum. Most popular.

3 Scenarios

Scenario 1 — Training plateau Pick: add momentum, reduce LR, or switch to Adam.

Scenario 2 — Loss diverging Pick: LR is too high. Reduce by 10x; check for NaN gradients.

Scenario 3 — Memory limit on large batches Pick: gradient accumulation — micro-batches summing gradients before one model update. Effective batch size unchanged.

Q4 — What is regularization? L1 vs L2?

What? A penalty added to the loss so that simpler (or sparser) weights are preferred, reducing overfitting.

Why? Because training loss can always be minimised but test loss often diverges without regularisation.

How?

• L2 (weight decay): penalty λ * Σ W². Smooths weights; many small contributions.
• L1: penalty λ * Σ|W|. Pushes small weights to exactly 0 → feature sparsity.
• Dropout: random zero-out during training; ensemble effect at inference.

3 Scenarios

Scenario 1 — Tiny dataset (5k rows), big model (10B params) Pick: L2 + dropout + early-stopping. Without these, you’ll overfit in 1 epoch.

Scenario 2 — Feature selection needed Pick: L1; the sparse solution drops irrelevant features entirely.

Scenario 3 — Very deep transformer fine-tuning on small dataset Pick: dropout + weight decay + low LR. Otherwise catastrophic overfitting.

Q5 — Fine-tuning vs RAG — when do you use each?

What? Two complementary ways to inject domain knowledge into an LLM:

• Fine-tuning: train the LLM on your data; bake the knowledge into the weights.
• RAG: at query time, retrieve relevant docs and prepend them to the prompt.

Why? Because the choice determines cost, latency, and freshness.

How? Decision tree:

1. Do you need stronger style/format adherence? Fine-tune. RAG can’t reliably enforce style.
2. Do you need private/data not in training? Both — RAG for ephemeral; fine-tune for stable knowledge.
3. Do you need fresh data? RAG wins; fine-tuning is slow to retrain.
4. Budget? RAG is cheaper to start; fine-tuning needs GPU + data prep.

3 Scenarios

Scenario 1 — Banking chatbot with stable product knowledge Approach: fine-tune on product details + RAG for current rates/policy. RAG always wins on freshness.

Scenario 2 — Customer support tone Pick: fine-tune. The model needs to consistently speak in the bank’s voice.

Scenario 3 — Internal Q&A over constantly-changing docs Pick: RAG only. No fine-tune because docs change weekly.

Q6 — How would you handle an imbalanced dataset (500 rows, 10 frauds)?

What? Classification where one class is rare. Without treatment, the model says “no fraud” for everything and is “98% accurate.”

Why? Because the most common interview trap — accuracy doesn’t capture class-level nuance.

How? Lever-arm ladder:

1. Class-weighting (penalise misclassifying minority more).
2. Resampling: oversample minority, undersample majority.
3. Synthetic data: SMOTE or similar.
4. Threshold tuning: lower the classifying threshold so minority is captured.
5. Use precision-recall not accuracy.

3 Scenarios

Scenario 1 — Random Forest on fraud data Pick: class_weight="balanced". Automatic weighting.

Scenario 2 — Neural net on rare-disease diagnosis Pick: oversample + threshold tune + focal loss.

Scenario 3 — Time-series classification Pick: sliding-window positive examples; weighted loss.

Q7 — Bagging vs Boosting?

What? Two ensemble techniques. Both combine many weak learners; differ in how.

Why? Because the choice between Random Forest (bagging) and XGBoost (boosting) is the most common practice question.

How?

• Bagging (Random Forest): train trees in parallel on bootstrapped data. Average output. Reduces variance.
• Boosting (XGBoost / AdaBoost): train trees sequentially; each corrects the previous. Reduces bias.

3 Scenarios

Scenario 1 — High-variance model (deep tree) Pick: bagging. Multiple diverse trees cut variance.

Scenario 2 — Under-fit baseline (logistic regression) Pick: boosting. Sequential corrections add capacity.

Scenario 3 — Imbalanced, noisy data Pick: bagging. Boosting overfits to the noise.