Eval

report.md in this directory is a committed artifact. It is the output of running the matcher on the seeded synthetic fixtures in examples/intake-demo/ and scoring the result against the planted ground truth in ground_truth.json. There is no real personal data in the fixtures.

Regenerate it with:

make eval

CI regenerates it and fails if the committed copy is stale, so the numbers in the repo always match the code.

What is measured, and why this shape

Correctness is asymmetric. A false merge joins two different people and can corrupt a record irreversibly. A missed match leaves a duplicate, which a later pass can still catch. The gated metric is therefore the false-merge rate: of the pairs the system merged without a human, how many were wrong. It is reported with a Wilson interval because the denominator (auto-merged pairs) is small and a normal-approximation interval would understate the uncertainty.

The fixtures deliberately include cases that should not auto-merge: two people with the same common name and different dates of birth, and one real duplicate whose dates of birth differ by a typo. The matcher cannot tell those apart from the data alone, so both land in the review band. That is the point: the review queue exists for exactly the pairs a confidence threshold should not decide on its own. Auto-level recall is below 100% on purpose, while coverage recall (auto plus review) captures every true duplicate.

The external benchmark reports

febrl4-report.md is the original eval here that does not score fixtures this project wrote. FEBRL4 is a published record-linkage benchmark: the corpus, the corruptions, and the ground truth are all third-party, so the difficulty cannot be tuned to flatter the result. Regenerate it with make eval-benchmark. The corpus is downloaded on demand into gitignored benchmarks/, verified against pinned SHA-256 digests, and never committed.

Measured on 10,000 records and 5,000 known pairs: 100% precision and 74.3% recall at the auto band, 99.3% precision and 87.9% recall (F1 93.3%) counting the review queue. febrl1-report.md, febrl2-report.md, and febrl3-report.md score three more corruption levels from the same pinned upstream commit (make eval-benchmark-multi DATASET=N); unlike FEBRL4, an original in those datasets can carry more than one duplicate, so they also cover clustering across three or more records of the same person. See ../docs/BENCHMARK.md for what the benchmarks are, why a real-person corpus was declined, the normalizer gap FEBRL4 exposed that no fixture here could, and each dataset's own threshold sweep.

The large-corpus report

27 records and 7 true pairs give a real gate, but a weak one: the demo's false-merge Wilson interval is [0%, 39%]. large-corpus-report.md in this directory is the same eval run against a much larger seeded synthetic corpus (10^3-10^5 records, default 50,000) generated by tools/corpusgen/, with planted duplicates across several error channels (name typo, nickname, transliteration variant, hyphenated/compound surname, date-format drift, DOB typo, address variant) and planted same-name/ different-DOB decoys that must not auto-merge. Every planted duplicate is tagged with a name-origin class, so the report also breaks recall down by class (feeding the bias question: does the matcher work as well on Vietnamese or Arabic names as on Anglo ones) and by error channel, each with a Wilson interval, plus wall-clock and peak-memory numbers for the run.

Regenerate it with:

make eval-large

This is not part of make verify or CI: a 10^4-10^5 record corpus through Splink/DuckDB takes materially longer than the 27-record demo, so it runs on release instead of on every push. The generated corpus itself (eval/large-corpus/) is not committed — it is fully reproducible from the seed baked into tools/corpusgen/run_large_eval.py — only the derived report is.

The error-model assumptions in tools/corpusgen/ (which typo shapes are common, which nicknames map to which given names, and so on) are the generator author's best approximation, not measured from real intake data. See tools/corpusgen/__init__.py and docs/ideation/02-large-scale-fixes.md (FIX-11) for what this corpus does and does not claim, and for the plan to calibrate it against real, consented pilot data once one exists (EXP-08 / E8).

First measurement against real records (2026-08-04)

A first check of the paragraph above: how does the matcher score when the variation is real rather than generated? North Carolina publishes its full voter file, and ncid is a stable per-voter identifier, so two statewide snapshots give same-person pairs whose differences are real edits made by county staff. Orange County, active and inactive registrations, 2024-01-01 against 2026-03-03: 176,932 records and 88,466 true pairs. ncid was held out of the matcher, which saw only name and address.

	seeded synthetic corpus	real NCVR pairs
true pairs	12,438	88,466
false-merge rate	0.6%	2.6%
missed-match rate	0.0%	0.7%

Read the comparison with care: it moves two variables at once. NCVR publishes birth year rather than full date, so the real run had no dob, which is both the strongest comparison in defaults.py and its first blocking rule. Some unknown share of the gap is the missing field rather than real-versus- synthetic, and this dataset cannot separate them because it will never carry a dob column. What the run does establish is a name-and-address-only figure, which is a common intake shape: a 2.6% false-merge rate, above this project's own 1% gate.

By risk class, the picture is uneven in a way the aggregate hides:

segment	true pairs	coverage recall
unchanged	79,602	100.0%
moved	6,660	99.9%
name changed	1,642	72.5%
name and address changed	562	68.9%

Blocking is not the constraint, which was the first guess and it was wrong. Of the 452 missed name-change pairs, only 31 were never scored; the other 421 were scored and fell below the review threshold. Adding address as a fifth blocking key was tried directly: it generated 2,341 more candidate pairs, pulled 12 pairs out of the blocking-miss column, and moved no segment's recall by a single pair. It also left docs/audits/bias-report.md byte-identical. The change was reverted because it costs comparisons and buys nothing measurable.

The likelier reading is that this is close to the information limit for these fields. With name and address alone, a person who changed their surname and stayed put is not distinguishable from two people who share an address. That is the spouse, roommate, and adult-child case, which this project deliberately treats as a separate concern in household.py behind reviewed suggestions that are off by default. Merging on that evidence is the error the fail-closed band exists to prevent, so a 72.5% recall on name changes may be the matcher declining correctly rather than failing.

Known limits of this measurement: one county, one state, one 26-month window, no dob, and a public voter roll rather than nonprofit intake. Voter registration records are also cleaner than the intake documents this tool is built for. It is one real datapoint against a synthetic error model, not the consented pilot calibration EXP-08 describes.

The stage-timing baseline

large-corpus-stage-baseline-2026-08-03.md, with a JSON companion of the same name, records where the large-corpus run spends its time and memory, stage by stage: ingest, extract, normalize, score, review artifact, write. It is the committed "before" side of the UC-01 stage-cache work in docs/NOVEL-USE-CASES-PLAN.md. When the cache lands, the same command rerun on the same machine class produces the "after" side, and the two JSON files are the diff.

Regenerate with:

make perf-baseline

Like make eval-large, this is a local command, not a CI job. CI proves the harness itself works by running it on a tiny corpus (tests/test_stage_baseline.py), including a byte-for-byte check that the harness's composed stages produce the same artifacts pipeline.run and pipeline.export produce. The committed numbers were measured once on the one machine class the report names. They describe that run honestly and are not a performance promise; different hardware produces different absolute values, so before/after comparisons belong on a single machine.

The mixed CSV and PDF variant

The committed baseline's corpus is CSV-only, so its extract row reads 0.0s. That is honest and it is also useless as a before number for the half of the stage cache that caches extraction. large-corpus-stage-baseline-pdf-2026-08-04.md, with a JSON companion of the same name, measures the same six stages over a corpus where 15% of the incoming rows arrive as digitally created text-layer PDF intake documents instead of CSV rows, one record per page, generated by the same seeded generator. On the machine class that report names, extraction took 36.9s over 3,756 pages in 151 documents, against 0.0s in the CSV-only baseline. Regenerate with:

make perf-baseline-pdf

The extract row then reports the time the ingest walk spent inside the pipeline's own pdfplumber reader, sandboxed parse included, over the page count the generated manifest accounts for. Ingest reports its wall clock with that time removed, so the two rows partition the walk instead of double-counting it. The corpus lands in eval/large-corpus-pdf/, separate from the CSV-only one and equally uncommitted; the report and its JSON companion are named with a -pdf- infix.

Read the two reports side by side for the extract row, not for the rest. A record read from a PDF page carries only what the extractor recovers from a labeled line: name, a numeric date of birth, and email or phone when the form has one. Address and consent have no extraction pattern, and a date written in prose ("26 November 1942", which the date-drift channel plants) does not match the numeric date pattern, so PDF-carried people reach the matcher with fewer comparison fields than the same people as CSV rows. Candidate pairs, clusters, and written counts therefore differ from the CSV-only run at the same seed by construction. The report says so in its own text, and tests/test_corpusgen_pdf.py pins the behaviour so it stays a documented property rather than a surprise.

The PDF writer (tools/corpusgen/pdfwrite.py) is dev tooling built on the standard library. It adds no dependency to the package, and nothing in the package's runtime imports it. Its output is deterministic byte for byte, and the harness refuses a corpus whose PDFs, CSVs, or manifest were edited after generation, so a measured run always matches the parameters it records.