Retrieval crash test: when similar means wrong

Top-k is not a product outcome

A similarity ranking tells us which vectors are close to a query. It does not tell us whether a document concerns the right party, applicable law, procedural stage, or user question.

Recall and precision therefore need to connect to the decision the system supports. One ranking may be acceptable for exploration and unacceptable for drafting a justification.

A dataset with negative cases

The test set should include documents that look relevant but are wrong for one specific reason:

• an outdated version;
• another jurisdiction;
• a similar person or organisation name;
• a different procedural stage;
• a quotation whose missing context changes its meaning.

The evaluation unit

Passage-level evaluation is insufficient when the user needs a whole document, metadata, and relationships to other sources. The retrieval unit and evaluation unit must reflect the final artifact.

Verification after retrieval

Retrieval should be followed by a citation check: exact text, scope, source, version, and relationship to the claim. A model should not be the only verifier of material it previously generated.

Reading the benchmark

A score without corpus, query set, ground truth, and tolerated failure is only a number. There is no universal vector-database ranking; there are better or worse specified tests of a particular task.