Skip to Content

MRR and NDCG (Rank-Aware Quality)

When to use

Score a ranked list when position matters a lot — first-hit latency, search ranking, recommendation order.

Analogy

MRR is finding the first correct answer in a trivia game and stopping. NDCG grades your whole essay — correct facts in the intro are worth more than in the conclusion.

Data-flow diagram

   ranks of FIRST relevant item across queries: [1, 4, 2, 9]
   MRR = mean(1/rank) = 0.465

   NDCG = DCG / IDCG
   DCG  = sum(rel_i / log2(i+1)) for i in 1..K

Deep explanation

Mean Reciprocal Rank: for each query find the rank of the first relevant item and average 1 over rank. Strong signal for ‘did the right answer show up high enough’. NDCG (Normalized Discounted Cumulative Gain) generalises MRR with graded relevance — use it when documents have multiple levels of relevance. Use log base 2 to match the original Burges paper.

Examples

Example 1

def mrr(ranks):
    return sum(1.0/r for r in ranks) / len(ranks)
print('MRR:', mrr([1, 2, 4, 3, 5]))

MRR rewards moving the first relevant item up.

Example 2

import math
def dcg(rel, k):
    return sum(r / math.log2(i + 2) for i, r in enumerate(rel[:k]))
def ndcg(rel_true, rel_pred, k):
    ideal = sorted(rel_true, reverse=True)
    return dcg(rel_pred, k) / max(dcg(ideal, k), 1e-9)
print('NDCG@5:', ndcg([3, 2, 1, 0], [3, 0, 2, 1], 5))

NDCG handles graded relevance (0..3) and discounts by rank with log base 2.

Example 3

from sklearn.metrics import ndcg_score
import numpy as np
true_rel = np.array([[3, 2, 1, 0, 0]])
scores   = np.array([[0.9, 0.1, 0.5, 0.3, 0.2]])
print('NDCG@5:', ndcg_score(true_rel, scores, k=5))

sklearn ndcg_score reorders by its score argument — no need to rank first.

Common mistake

Computing MRR when there is no relevant item in top-K — the reciprocal is undefined. Skip the query, set a sentinel, or treat as zero.

Key takeaway

MRR for first-hit latency; NDCG for graded relevance + position. Always run at fixed K.

Production Failure Playbook

Failure scenario 1: irrelevant-query-skips

Failure scenario 2: log-base-mistmatch