How can we measure not what a legal rule says, but how it is actually applied?

Research question

Can a team move from a handful of retrieved cases to a reproducible account of how a rule is applied across a large corpus without losing the path to each document? NormaLab treats this as research-infrastructure design, not as the problem of building one chatbot.

Scale is necessary to identify patterns in legal practice. At the same time, every conclusion must still lead to the right matter, document version, passage, and classification rule.

Problem

A fluent answer can cite the wrong case, an outdated document, or a sentence stripped of context. In legal research, retrieval, citation, and interpretation failures are different errors; they should not disappear inside one answer-quality score.

My role

• data engineering and corpus versioning;
• agent-task and control-point design;
• reproducibility and configuration records;
• a citation engine connecting conclusions to documents.

Research protocol

The process starts with a question and explicit corpus criteria. Acquisition, normalisation, deduplication, and version control remain deterministic. An LLM appears only in tasks that require semantic work, and its output passes citation checking and QA.

question → corpus → document selection → classification → citation
→ counterexample → legal review → report

Code assembles the corpus, normalises documents, and records versions. The model helps classify material and prepare hypotheses. A lawyer checks the legal characterisation. Every transition preserves a link to the input document and the outcome of verification.

A model change must not silently alter the source set or evaluation rules. Every stage receives an identifiable input and records the version of its intermediate result.

What is measured separately

• corpus completeness and cleanliness;
• retrieval relevance;
• citation fidelity;
• classification-rule stability;
• package completeness and run reproducibility.

This separation makes it possible to repair a specific layer instead of “improving the prompt” without a diagnosis.

The audit that remained visible

External findings, including a return for revision, remain part of project history. A negative review outcome is useful when it identifies the failure and leads to a new artifact version. NormaLab therefore demonstrates a procedure for finding and correcting errors, not model infallibility.

What was built

The public platform presents the method, publications, infographics, and report examples. A research finding can be connected to its document, processing version, classification rule, and evaluation record. This provides a practical pattern for corpus research in which evidence remains visible after automation.

Why not one prompt

The process deliberately avoids a single-prompt design. Corpus construction, retrieval, semantic analysis, and citation checking have different failure profiles, so they need separate stages and criteria. An LLM is not used where a deterministic rule produces a more reproducible result.

Limitations

NormaLab remains a research pilot, not a production system. Experimental figures are published only with the methodology and a specific report. The material is not legal advice.