ML / AI Evaluation Fundamentals

Twelve curated topics covering what every AI engineer uses day-to-day: confusion matrix, accuracy/precision/recall/F1, Precision-at-K and Mean Reciprocal Rank and NDCG for retrieval, BLEU/ROUGE & perplexity for generation, calibration & ECE, AUC-ROC vs AUC-PR, drift detection with KS/PSI, regression metrics, business metrics, A/B testing & statistical significance. Beginner-friendly framing with three worked examples each.

Topics

• The Confusion Matrix
• Accuracy, Precision, Recall, F1
• Precision-at-K (Top-K Retrieval)
• MRR and NDCG (Rank-Aware Quality)
• BLEU and ROUGE (Text-Generation Metrics)
• Perplexity (LM Surprisal)
• Calibration and ECE (Probability Alignment)
• AUC: ROC vs PR (Threshold-Free Scoring)
• Drift Detection with KS and PSI
• Regression Metrics (MSE, MAE, RMSE, R-squared, MAPE)
• Business Metrics for ML (North-Star, Conversion, Retention)
• A/B Testing and Statistical Significance