Eval

report.md in this directory is a committed artifact. It is the output of running the matcher on the seeded synthetic fixtures in examples/intake-demo/ and scoring the result against the planted ground truth in ground_truth.json. There is no real personal data in the fixtures.

Regenerate it with:

make eval

CI regenerates it and fails if the committed copy is stale, so the numbers in the repo always match the code.

What is measured, and why this shape

Correctness is asymmetric. A false merge joins two different people and can corrupt a record irreversibly. A missed match leaves a duplicate, which a later pass can still catch. The gated metric is therefore the false-merge rate: of the pairs the system merged without a human, how many were wrong. It is reported with a Wilson interval because the denominator (auto-merged pairs) is small and a normal-approximation interval would understate the uncertainty.

The fixtures deliberately include cases that should not auto-merge: two people with the same common name and different dates of birth, and one real duplicate whose dates of birth differ by a typo. The matcher cannot tell those apart from the data alone, so both land in the review band. That is the point: the review queue exists for exactly the pairs a confidence threshold should not decide on its own. Auto-level recall is below 100% on purpose, while coverage recall (auto plus review) captures every true duplicate.

The large-corpus report

27 records and 7 true pairs give a real gate, but a weak one: the demo's false-merge Wilson interval is [0%, 39%]. large-corpus-report.md in this directory is the same eval run against a much larger seeded synthetic corpus (10^3-10^5 records, default 50,000) generated by tools/corpusgen/, with planted duplicates across several error channels (name typo, nickname, transliteration variant, hyphenated/compound surname, date-format drift, DOB typo, address variant) and planted same-name/ different-DOB decoys that must not auto-merge. Every planted duplicate is tagged with a name-origin class, so the report also breaks recall down by class (feeding the bias question: does the matcher work as well on Vietnamese or Arabic names as on Anglo ones) and by error channel, each with a Wilson interval, plus wall-clock and peak-memory numbers for the run.

Regenerate it with:

make eval-large

This is not part of make verify or CI: a 10^4-10^5 record corpus through Splink/DuckDB takes materially longer than the 27-record demo, so it runs on release instead of on every push. The generated corpus itself (eval/large-corpus/) is not committed — it is fully reproducible from the seed baked into tools/corpusgen/run_large_eval.py — only the derived report is.

The error-model assumptions in tools/corpusgen/ (which typo shapes are common, which nicknames map to which given names, and so on) are the generator author's best approximation, not measured from real intake data. See tools/corpusgen/__init__.py and docs/ideation/02-large-scale-fixes.md (FIX-11) for what this corpus does and does not claim, and for the plan to calibrate it against real, consented pilot data once one exists (EXP-08 / E8).