Skip to Content

Business Metrics for ML (North-Star, Conversion, Retention)

When to use

Tie model decisions to the actual money / retention / engagement outcomes the business cares about.

Analogy

A restaurant guide. Number of restaurants rated, average rating, date-night conversion rate. The model that recommends the highest-rated restaurants beats one that picks by gourmet score if customers end up eating fine dining for date night anyway.

Data-flow diagram

   North-Star     :  weekly active subscribers
   Cohort retention:  wk1   wk4   wk12
                      100%   70%   50%
   Counter-metric  :  FP_rate * predictions_per_user

Deep explanation

Business metrics are the bridge from ML metrics to revenue. North-Star stars everything else: weekly active subscribers, GMV, NPS. Supporting metrics break north-star into levers. Counter-metrics are the costs of bad predictions. Map@k and recall-at-target-precision are common bridges.

Examples

Example 1

import pandas as pd
df = pd.DataFrame({
    'cohort':   ['2026-01-01', '2026-01-08', '2026-01-15'],
    'wk1_ret':  [1.00, 1.00, 1.00],
    'wk4_ret':  [0.70, 0.65, 0.68],
    'wk12_ret': [0.50, 0.48, 0.51],
})
print(df.set_index('cohort'))

Cohort retention tables expose time-decay that one-shot KPIs hide.

Example 2

fp_per_user = 0.05 * 12  # FP rate times predictions per user per day
print('annoyance events per active user per day:', fp_per_user)

Counter-metric = FP rate times predictions per user: turns model accuracy into UX cost.

Example 3

def conversion_at_k(recommended, purchased, k):
    return len(set(recommended[:k]) & set(purchased)) / k
print('conv@5:', conversion_at_k(['a','b','c','d','e'], ['b','x','y'], 5))

Conversion-at-K bridges ranking ML to revenue.

Common mistake

Optimising for an ML metric that does NOT move the North-Star. A 99 percent accurate spam filter may cost 1 percent of legitimate mail lost.

Key takeaway

Pick a North-Star, derive supporting metrics, define counter-metrics. A/B test every change.

Production Failure Playbook

Failure scenario 1: auc-up-conversion-flat

Failure scenario 2: counter-metric-ignored