How do you detect a batch-of-vectors with the wrong dimension before it poisons the index?

Category: Database & RAG

Answer

Add a check constraint on the column (Postgres allows CHECK on dimension via a function) AND a Pydantic validator at write time. Two layers: the column refuses bad inserts; the wrapper catches the bug upstream so nobody else has to chase it.

Concrete examples from the fca project context

Example 1

CHECK (vector_dims(embedding) = 1536) on the column.

Example 2

Embedding model returns an array; Pydantic v2 validates length matches expected dim.

Example 3

CI test: insert a wrong-dim vector and assert ValueError.

Data flow / flow chart

embed(text) -> vector dim N -> Pydantic check dim matches
  INSERT -> Postgres CHECK (vector_dims() = N)

Takeaway

Two layers of dim check (Pydantic + Postgres) — one bug caught at app, one at DB.