Every other eval in this repository scores the matcher against fixtures this
repository also wrote. The corpus, the error channels, and the ground truth all
come from one hand, so a good number partly measures the fixture author's
imagination rather than the matcher. This page covers the evals where that is
not true: FEBRL4, the original external benchmark this project scored, plus
FEBRL datasets 1 through 3 (## Widening to FEBRL datasets 1-3 (#68) below),
three more corruption levels from the same third-party corpus and third-party
ground truth.
Regenerate the FEBRL4 numbers with:
make eval-benchmarkThat downloads the corpus, verifies it against pinned digests, converts it into a
recipe, runs pipeline.run (the same entry point the CLI uses), scores the
result against third-party ground truth, and writes
../eval/febrl4-report.md. The corpus lands in
gitignored benchmarks/ and is never committed. The run takes a few seconds and
needs network access on first use; afterwards --offline works against the
cached copy. make eval-benchmark-multi DATASET=N (N in 1, 2, 3) does the same
for the widened datasets.
Measured on 10,000 records with 5,000 published ground-truth pairs.
| Metric | Value |
|---|---|
| Precision, auto-merge band | 100.0% (0 false merges in 3,714) |
| Recall, auto-merge band | 74.3% |
| F1, auto-merge band | 85.2% |
| Precision, auto + review coverage | 99.3% |
| Recall, auto + review coverage | 87.9% |
| F1, auto + review coverage | 93.3% |
| Missed-match rate | 12.1% (603 / 5,000) |
| True pairs never scored | 61 |
Read honestly: precision is still the strong half. Nothing was merged that should not have been, which is the behaviour the fail-closed design is built for. Published FEBRL4 results from tuned academic systems still sit above this. These defaults are set for small nonprofit batches where a false merge is the expensive error, and that choice costs recall; the number is measured rather than asserted.
The recall figures above are the second measurement. The first run of this benchmark is recorded below, along with what it was wrong about.
The first FEBRL4 run scored 77.1% coverage recall and reported 344 "blocking
misses", read at the time as pairs "unreachable by threshold tuning" that "only
the blocking rules in defaults.py reach". That reading was wrong, and the label
is what made it wrong.
EvalReport.blocking_misses counts true pairs that never appear in the scorer's
output. A pair lands there two ways: blocking never generated it, or blocking did
and the matcher scored it below the 0.001 floor that keeps near-zero pairs out of
the result. Recomputing the blocked candidate set directly from the rules in
defaults.blocking_rules_for separates them:
| Count | |
|---|---|
| Pairs generated by blocking | 384,499 |
| Pairs kept above the 0.001 floor | 21,295 |
| True pairs generated by blocking | 4,943 / 5,000 |
| Genuinely never blocked | 57 |
| Blocked, scored, dropped by the floor | 287 |
So 287 of the 344 were reachable by the comparison model all along, and the row
labelled "candidate pairs after blocking" was counting kept pairs, not blocked
ones, understating blocking by a factor of eighteen. Both labels are corrected in
report.py, and the field carries a comment saying what it actually counts. A
metric name that points a fix at the wrong module is worth more than a cosmetic
fix.
Characterising the 1,144 missed pairs before changing anything, the largest single class was one error mode: the given name and the family name entered in opposite fields. 298 missed pairs were crossed name pairs (220 exactly crossed, 78 crossed with a typo on one side), including 230 of the 344 that were never scored at all.
Crossed pairs were not merely unsupported by the model, they were penalised twice. Both name comparisons saw values that disagreed, each fired its "different" level at a weight near 1/95, and the product vetoed everything the other fields had to say. A pair agreeing exactly on date of birth and closely on address still scored around 8e-6.
That is worth fixing beyond this corpus. Transposition is not a typo, it is a
structural intake error: a form labels one box "first name" and the other "last
name", and the ordering that assumes is not universal. Family-name-first is the
written convention in Chinese, Korean, Japanese, Hungarian and Vietnamese naming,
so a constituent writing their name the way they always write it lands the values
in the opposite boxes from the worker who typed the earlier record. FEBRL's
dsgen injects field swaps deliberately, which is why the class is large here,
but the mechanism it is standing in for is real.
Three changes, in normalize.py and defaults.py:
- A transposition comparison level in both name comparisons. It reads all
four name values, because to either comparison alone a crossed pair simply
disagrees. Jaro-Winkler on each side at the same
_NAME_CLOSEthe same-field close level uses, so a transposition carrying a typo is still recognised. The given-name comparison carries the evidence at a weight of 20; the surname comparison abstains at a weight of exactly 1, so one mistake is counted once rather than twice. - A
name_pair_keyblocking rule, the two normalized names sorted and joined. Every other rule compares one column against itself, so a crossed duplicate agrees with itself on none of them. It is also the cheapest rule in the set: 3,179 pairs against 150,000 to 230,000 for the per-name rules. _NAME_DIFFERENT_Mraised from 0.01 to 0.02, the m_probability for a name comparison's "different" level.
| Metric | Before | After |
|---|---|---|
| Precision, auto-merge band | 100.0% (0 / 3,365) | 100.0% (0 / 3,714) |
| Recall, auto-merge band | 67.3% | 74.3% |
| F1, auto-merge band | 80.5% | 85.2% |
| Precision, auto + review coverage | 99.25% | 99.34% |
| Recall, auto + review coverage | 77.1% | 87.9% |
| F1, auto + review coverage | 86.8% | 93.3% |
| Missed-match rate | 22.9% (1,144) | 12.1% (603) |
| True pairs never scored | 344 | 61 |
| Pairs sent to review | 520 | 712 |
| Non-matches sent to review | 29 | 29 |
No auto-merge precision was traded. The false-merge count stayed at zero over a larger auto band, and coverage precision rose. The review queue grew by 192 pairs and every one of them is a true duplicate: the count of non-matches reaching a reviewer is unchanged at 29. Reviewer workload is up about a third, all of it productive.
The third change is the only one that is a weight rather than a structure, so it deserves its own justification. One percent is not a defensible reading of how often two records of the same person carry names that do not agree, are not a known nickname pair, and are not even Jaro-Winkler close. A legal name against a chosen or preferred one, an anglicized given name on one form, a marriage or divorce, a name changed after leaving an abusive partner, a transliteration from a non-Latin script, a nickname the vendored table does not carry (it maps under a twentieth of the given names here), a typo worse than one character: any one of those plausibly clears one percent on its own. 0.02 leaves the level strong evidence against a match at a weight near 1/48. What it stops doing is vetoing every other field.
One corroboration is worth naming, because it comes from a fixture this
benchmark had nothing to do with. docs/audits/bias-report.md plants one true
pair for each of five documented equity risk classes, written long before this
run. Its non-Western name order class had been at 0% coverage recall since
the audit was first committed. The transposition level takes it to 100%, and
lifts that report's auto-band recall from 60% to 80%, with precision unchanged
at 100%. The change was found on an Australian machine-generated corpus and it
moved a hand-written fixture about Chinese, Korean and Hungarian name order,
which is the kind of transfer a corpus-specific fix does not make.
Because the weight change lets a shared address be heard, the household case is
now a test rather than an argument. test_matching.py asserts that two people at one
address with the same surname and different given names do not auto-merge, both
when their dates of birth disagree and when neither has one. Families and shelter
residents share addresses, and under the DV pack a wrong merge there is a safety
failure, not a quality one.
Over all 384,563 blocked candidate pairs, every pair whose standardized addresses
are Jaro-Winkler similar at 0.90 or above is a true match. Not almost all: all of
them, and the same holds for exact address agreement. Read literally, the
u_probability on the address comparison's close level is wrong by four orders of
magnitude, and correcting it toward what the corpus shows lifts coverage recall
past 89% with no measured precision cost.
That correction is declined, and the reason matters more than the number. FEBRL's
dsgen never generates two different people at one address. Real constituent
files do, constantly: families, roommates, and shelter residents. Fitting the
address weight to a corpus that structurally cannot contain the failure mode would
produce a matcher that treats a shared address as near-conclusive, which is the
single worst assumption to carry into a domestic-violence shelter's data. The
hand-set weight stays, and _address_comparison keeps saying why.
This is the clearest example of the general rule for this benchmark: a change is adopted when it has a justification that survives the corpus being taken away.
- Two files of 5,000 person records:
dataset4a(originals) anddataset4b(one corrupted duplicate of each). Exactly 5,000 true pairs, no duplicates within either file. - Ground truth is carried in the upstream record ids rather than asserted here:
rec-N-organdrec-N-dup-0are the same person.tools/benchmark/febrl4.pyderives clusters from those ids and nothing else. - Fields offered: given name, surname, date of birth, and a split address. There
is no email, no phone, and no consent column, so the recipe maps four canonical
fields and omits the
[consent]section rather than inventing values. - The records are generated, not collected. FEBRL's
dsgensamples names, addresses, and dates from real Australian frequency tables, then applies typographic, phonetic, and field-swap corruption.
So this benchmark does not make the project's demo "real data". What changes is narrower and still worth having: the corpus, the difficulty, and the ground truth are now fixed by a third party, years before this repository existed, and cannot be tuned to flatter the result. Self-authored fixtures cannot make that claim.
The obvious upgrade is a benchmark built from real person records, and the standard one is the North Carolina voter registry (the Leipzig NCVR sets, CC licensed and freely downloadable). This project declines it.
A public voter file is a recognised locating vector for exactly the people the DV policy pack exists to protect. Pulling one onto a contributor's disk, and wiring a public repository to fetch it on demand, to make a portfolio number look better is not a trade this project should make.
That constraint is not incidental to this benchmark. It is the reason open person-linkage corpora with real ground truth are rare in the first place: real identity data with known matches is confidential nearly everywhere it exists, for the same reasons this pipeline has a no-egress mode. A reader who wants the project measured on real personal records is asking for something the field mostly cannot supply openly, and that is worth saying plainly rather than papering over.
Open question for the maintainer. If NCVR is judged acceptable after all, it would raise the realism of the corpus at a real privacy cost, and it is a judgement call rather than an engineering one. Nothing here forecloses it.
The FEBRL datasets originate with the Febrl project by Peter Christen
(Australian National University). They are redistributed inside the
recordlinkage Python package by
Jonathan de Bruin under a 3-clause BSD licence.
Nothing is vendored. tools/benchmark/febrl4.py fetches the two files at run
time from a pinned upstream commit (b93d9764, recordlinkage v0.16) and verifies
each against a recorded SHA-256 digest. A mismatch aborts the run rather than
scoring whatever arrived, so a changed upstream cannot quietly move the published
numbers. The digests are in the report as well as the source.
FEBRL4 is one corruption level. The same recordlinkage package bundles three
more datasets at the same pinned commit, at three increasing corruption
levels, and tools/benchmark/febrl_multi.py gives them the same treatment:
fetched at run time, verified against recorded SHA-256 digests, converted,
scored with the pipeline's own evaluate(), gated on the false-merge rate,
and swept across six auto-merge thresholds. Full reports:
../eval/febrl1-report.md,
../eval/febrl2-report.md,
../eval/febrl3-report.md. Regenerate any of
them with make eval-benchmark-multi DATASET=N.
Unlike dataset4 (two files, exactly one duplicate per original), datasets 1-3
ship as one file mixing originals and duplicates, and an original can have
more than one duplicate: dataset1 always exactly one, dataset2 up to five
(428 of 4,000 originals matched), dataset3 up to five with a much larger
matched share (1,165 of 2,000). This closes the gap this document's "known
gaps" section named below: FEBRL4 alone tested nothing about clustering
across three or more records of the same person, and dataset3 in particular
exercises exactly that, since every duplicate of one person is itself a true
pair of every other duplicate under Splink's dedupe_only matching, not only
a pair of the original.
| Dataset | Corruption | Records | Matched persons | True pairs | Precision (auto) | Recall (auto) | F1 (auto) | Precision (coverage) | Recall (coverage) | F1 (coverage) | False merges |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | low | 1,000 | 500 | 500 | 100.0% | 74.4% | 85.3% | 100.0% | 85.4% | 92.1% | 0 |
| 2 | medium | 5,000 | 428 | 1,934 | 100.0% | 59.7% | 74.8% | 99.5% | 78.7% | 87.9% | 0 |
| 3 | high | 5,000 | 1,165 | 6,538 | 100.0% | 56.3% | 72.0% | 99.9% | 74.5% | 85.4% | 0 |
| 4 (already reported above) | highest | 10,000 | 5,000 | 5,000 | 100.0% | 74.3% | 85.2% | 99.3% | 87.9% | 93.3% | 0 |
Read plainly: the false-merge gate holds at zero across every corruption
level and every dataset size measured, which is the property the fail-closed
design is built for. Recall falls as corruption rises from dataset1 to
dataset3, as expected; dataset4 does not continue that curve because it is a
different, harder corruption profile from dsgen, not simply "more of the
same knob" as 1 through 3, so its position in the table is a sibling
measurement, not the next point on a line.
Each report's own threshold sweep (threshold_sweep in
tools/benchmark/febrl_multi.py) re-bands the same scored candidates at six
auto-merge thresholds without re-running the matcher, so the precision/recall
trade-off at a stricter or looser cutoff is measured directly rather than
guessed. dataset3 at threshold 0.80 (looser than this project's 0.97
default) shows the trade-off concretely: precision drops to 99.9% (5 false
merges) in exchange for recall rising from 56.3% to 74.5%.
A benchmark harness can produce entirely plausible metrics while the corpus it claims to have scored never reached the resolver. That failure is invisible in the metrics themselves, so the report carries a flow-through section instead of asking to be trusted: the SHA-256 of the exact input bytes, the record counts the pipeline ingested, per-field population before and after normalization, and named example pairs that can be looked up in the source files. The harness also fails the run outright if the converter and the scorer disagree about how many ground-truth pairs exist, which is what a stale truth file looks like.
The date-of-birth normalizer silently discarded every date in the corpus.
FEBRL4 writes dates in ISO 8601 basic format (19151111). normalize_dob
handled the extended form (1915-11-11) and eight other layouts, but not the
compact one, so all 9,707 populated dates normalized to the empty string. Nothing
errored. The matcher simply scored 10,000 records as though no one had a date of
birth, and every fixture in this repository writes dates in a format the
normalizer already knew, so no test could see it.
Measured effect of teaching the normalizer that one format. Both columns predate the recall work above, so the "after" column is that section's "before":
| Before | After | |
|---|---|---|
| DOB values parsed | 0 / 9,707 (0.0%) | 9,643 / 9,707 (99.3%) |
| Precision, coverage | 84.7% | 99.3% |
| Recall, coverage | 58.5% | 77.1% |
| F1, coverage | 69.2% | 86.8% |
| Missed-match rate | 41.5% | 22.9% |
| True pairs never scored | 676 | 344 |
The remaining 64 unparsed dates are FEBRL corruptions that are not calendar dates
at all (19960094, 19450493). Rejecting those is correct: the normalizer
returns empty rather than rolling an impossible date over into a valid one.
The compact format is deliberately gated on a plausible leading year, so
12041990 and 04121990 still normalize to empty. A registry exporting
DDMMYYYY should produce a missing date, not a confidently wrong one.
Claims in this repository that could be checked against 10,000 external records were checked. None was contradicted; two were imprecise enough to be worth quantifying.
| Claim | Where | Measured |
|---|---|---|
| Address tables follow USPS Publication 28 | address.py, adr/0004 |
Accurate, and costly off-shore: 71.8% of street lines end in a token the table knows. 10.9% are real Commonwealth street types it does not carry — CIRCUIT (698), CLOSE (331), GARDEN (31), GROVE (15), RETREAT (15). The other 17.3% are FEBRL's injected typos, which should not normalize. |
| Nickname table is "small, curated, English-centric" | nicknames.py, adr/0009 |
Accurate. 249 variants; 4.9% of first names here map to a different canonical key. |
| Soundex blocking costs "a few more comparisons" | defaults.py |
Understated. 3,018 distinct surnames fall into 1,429 buckets (674 hold more than one, mean 2.11, largest 14), but the rule generates 229,324 candidate pairs, the most of any rule in the set. It is still cheap in wall clock (blocking runs in about 0.1s over 10,000 records) and it recovers 3,846 true pairs, so the rule stays; the phrase was measuring bucket sizes rather than pair counts. |
| Compound-surname heuristic takes the last two tokens | normalize.py, adr/0009 |
Rarely engaged on this corpus: 1.29% of records have a multi-token surname. |
The address result is a scope limit rather than a defect. The standardizer is documented as CASS-style and US-oriented, and this corpus is Australian, so the number measures how much that scope costs outside its intended deployment. It is recorded here so the recall figures above are read with it in mind: some share of the missed matches is address normalization declining to canonicalize a street type it was never given.
- The corpus is Australian, and both the address standardizer and the nickname table are US and English oriented. Recall here is a floor for a US deployment, not an estimate of one.
- No email and no phone means two of the strongest matching signals are absent. The defaults weight those heavily, so this corpus exercises a weaker feature set than a typical intake batch.
- FEBRL4's own duplicates are one-to-one; nothing in this section tests clustering across three or more records of the same person. Datasets 1-3 (above) close that gap: dataset3 alone has 1,165 matched persons carrying up to five duplicates apiece, so three-or-more-record clustering is now measured, not merely assumed to work by extension from the pairwise case.
- No two distinct people in this corpus share an address, because
dsgengenerates each person independently. That is the single most important thing this benchmark cannot measure, since households, roommates, and shelter residents are the normal case in constituent data and the reason the address comparison is weighted the way it is. Any address-related number here reads as an upper bound, and the section above declines the tuning it invites. - Transposition is over-represented relative to a typical intake batch:
dsgeninjects field swaps deliberately, so the recall gained from the transposition level here is larger than the same change would buy on most real files. The mechanism generalizes; the magnitude does not. - The LLM field judge and the whole extraction seam are out of the path: the input is structured CSV. The report says so rather than reporting a kappa failure for a component that never ran.