Why does near-duplicate detection matter for RAG retrieval?

Category: Database & RAG

Answer

Ten thousand identical forum headers and footers in your corpus dominate retrieval — every query lands on the same noisy page. De-duplication removes exact and fuzzy duplicates during ingest, so unique chunks spread the index evenly. Without it, recall@K collapses because the top-K fills with repeats.

Concrete examples from the fca project context

Example 1

Hash each chunk (SHA256 of normalized text) and drop near-duplicates via MinHash/LSH.

Example 2

Headers/footers show up thousands of times across one site — once you de-dup, your index drops 30-50%.

Example 3

De-dup happens BEFORE heuristic filtering so the filter sees unique docs only.

Data flow / flow chart

corpus -> normalize -> exact-dedup -> fuzzy-dedup (MinHash) -> heuristic filter -> embed -> index

Takeaway

De-dup is unglamorous but it’s often the single biggest recall@K win you’ll find.