Skip to Content

Calibration and ECE (Probability Alignment)

When to use

Ensure predicted probabilities actually correspond to observed frequencies — so when the model says 70 percent, the event happens about 70 percent of the time.

Analogy

Weather forecaster. If they say 70 percent chance of rain for 100 days and it rains on 30, they are overconfident.

Data-flow diagram

   predicted   actual     gap
   0.55        0.40       overconfident
   0.70        0.70       calibrated

   ECE = weighted-average |predicted - actual| across bins

Deep explanation

A model with great accuracy can be wildly miscalibrated. Calibration means: when the model says 70 percent, the event occurs 70 percent of the time over a large sample. Expected Calibration Error (ECE) bins predictions into N buckets and averages the gap. Fix with Platt scaling or isotonic regression after training.

Examples

Example 1

from sklearn.calibration import calibration_curve
import numpy as np
y_true = np.array([0,0,1,0,1,1,1,0,1,0])
y_prob = np.array([0.1,0.2,0.85,0.4,0.7,0.95,0.6,0.3,0.8,0.55])
prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=5)

calibration_curve returns true-vs-predicted binned frequencies — feed to matplotlib for a reliability diagram.

Example 2

def expected_calibration_error(y_true, y_prob, n_bins=10):
    bins = [i / n_bins for i in range(n_bins + 1)]
    ece = 0.0
    for i in range(n_bins):
        mask = (y_prob >= bins[i]) & (y_prob < bins[i+1])
        if mask.sum() == 0: continue
        ece += abs(y_prob[mask].mean() - y_true[mask].mean()) * mask.mean()
    return ece
print('ECE:', expected_calibration_error(y_true, y_prob, n_bins=5))

ECE is the weighted-average bin gap; closer to 0 is better.

Example 3

from sklearn.isotonic import IsotonicRegression
iso = IsotonicRegression(out_of_bounds='clip')
y_cal = iso.fit_transform(y_prob, y_true)
print('before ECE:', expected_calibration_error(y_true, y_prob))
print('after ECE:', expected_calibration_error(y_true, y_cal))

Isotonic regression realigns probabilities without changing accuracy.

Common mistake

Assuming calibration is preserved across deployment populations. Re-validate ECE on production splits after each data shift.

Key takeaway

Calibration matters whenever downstream decisions use probabilities as thresholds. Use ECE; fix with Platt scaling or isotonic regression.

Production Failure Playbook

Failure scenario 1: stock-confidence-on-shift

Failure scenario 2: raw-probability-as-decision