SCORECARD.md

nearmiss's own scorecard

nearmiss scored against its own benchmark suite (v2.0.0), regenerated by:

python benchmarks/generator.py
python benchmarks/scorer.py

The numbers below are copied from the committed benchmarks/cities/*/scorecard.json (each also reproducible standalone). Re-run the two commands above to regenerate them from source — see README.md for what each column means and what each regime tests.

The hotspot columns are measured again as of v2.0.0 (issue #196)

Through 2026-08-19 every generated city was a graph island: generator.py laid the grid out as short, mutually non-touching stubs, so the street-network adjacency graph had no edges at all and every "Gi* z" this suite ever scored was actually the plain global z-score (ADR-0015, issue #193) — the hotspot columns were unmeasured, and said so.

The generator now emits a real, connected street grid — avenue segments that share exact intersection endpoints, plus cross-streets tying consecutive rows together (see generator.py's module docstring, "The street grid") — and every one of the 161 segments in every regenerated city (107 in maup_coarse) has at least one genuine network neighbour. The numbers below are real measurements against nearmiss's default configuration, not placeholders.

Regime	Recall	Precision	Decoy FP	Bias trap	Background FP	CI coverage
baseline	80%	36%	0%	0%	5%	99%
reporting_bias	40%	67%	0%	0%	1%	99%
overdispersion	0%	0%	0%	0%	2%	91%
exposure_error	0%	n/a	0%	0%	0%	94%
maup_fine	40%	40%	0%	0%	2%	98%
maup_coarse	0%	n/a	0%	0%	0%	98%

n/a = zero segments were flagged significant, so precision (true positives / flagged) has no defined denominator — see the committed scorecard.json for the literal null.

What this says about nearmiss, plainly, including where it is NOT good:

• decoy_exposure_fp_rate is 0% in every regime. The busy-but-average-rate decoy (the classic "raw count vs. normalized rate" heat-map lie) never fools the Getis-Ord layer. That is the one property exposure normalization is specifically supposed to guarantee, and it holds — the one number in this table that was true under the old disconnected fixtures for the wrong reason (nothing could be flagged at all) and is now true for the right one.
• reporting_bias_trap_rate is also 0% here, including in the reporting_bias regime itself (2 of 3 planted decoys, 5x reporting multiplier). Read this as a property of this suite's grid scale, not a claim that nearmiss detects and rejects reporting-propensity confounds — it cannot; stats/bias.py's report-share-vs-exposure-share panel is a caveat, never a correction, and nothing in the Getis-Ord layer models reporting probability. At this grid's ~100 m block spacing and the default gi_band_m (300 m), a decoy's Gi* neighbourhood is small enough, and the background class large enough (150 of 161 segments) for the Benjamini-Hochberg correction, that these particular decoys' z-scores did not clear the bar. A denser grid, a tighter decoy cluster, or a larger decoy_reporting_multiplier could change this in either direction — this row measures these three planted decoys under nearmiss's default settings, not a general immunity claim.
• Recall is low, and zero in three of six regimes (overdispersion, exposure_error, maup_coarse). Inspecting the raw output: the planted cluster's z-scores in exposure_error are genuinely elevated (2.85-2.93 on segments whose observed rate is 2-12x baseline) but do not survive Benjamini-Hochberg across all 161 simultaneous tests. That is the intended, conservative trade-off of controlling false discoveries rather than maximizing detections — see docs/METHODOLOGY.md — but it also means this small suite (one 5-segment cluster inside a 150-segment background class) is a harder test than nearmiss's own committed davis fixture, where the equivalent planted hotspot (seg-06, tests/fixtures/davis) reliably clears significance (tests/test_hotspot.py). Read the recall numbers here as "does the signal survive this suite's scale and multiple-comparison burden," not as a general detection rate.
• overdispersion degrades both detection (0% recall) and interval coverage (91%) relative to baseline (80%, 99%) — the direction the regime is designed to show: the published confidence interval assumes Poisson variance, and a real-world overdispersed reporting process (Var > mean) both widens the true sampling variability the z-score has to overcome and makes the interval too narrow more often than the nominal 95% promises.
• MAUP: the planted hotspot's recall drops from 40% (maup_fine) to 0% (maup_coarse), from the identical underlying report locations (see maup_fine/reports.json == maup_coarse/reports.json). Two effects compound under the 3-column merge: exposure-weighted true rate is diluted everywhere the merge groups three cells, most severely for the north/south cluster mates — each absorbs two ordinary background cells alongside its one elevated one, true rate 30 -> 16.7 — while the centre's own group fares better because it absorbs its already-elevated east/west mates rather than background, true rate 60 -> 40; and cross-streets at boundaries the merge swallows lose their link to the avenue layer at that column (see generator.py's module docstring). The result: seg-04-03 (the coarse city's merged centre segment, true rate 40 against baseline 10) reaches only z ≈ 1.92, against 107 simultaneous tests. The signal did not survive this change of spatial units at this suite's scale — a real, measured MAUP effect, not the "aggregation changed which segment is significant, not whether the signal survived" conclusion the pre-#196 suite draft (and the pre-network-topology run before it) both asserted. A single merge factor (3 columns) on a 9x9 grid is not exhaustive evidence about MAUP robustness in general — see stats/maup.py's own rank-stability check (FIX-01/RR-05) for the primary, davis-scale MAUP evidence this suite complements.
• Interval coverage is the one column that behaves the same as before, and still holds up. 91-99% against the nominal 95%, degrading exactly where the overdispersion and exposure_error regimes are designed to make it degrade. It never depended on the neighbourhood graph, so the connectivity fix did not change its meaning — only its neighbours.
• background_fp_rate sits at or under the nominal 5% everywhere (0-5%), consistent with Benjamini-Hochberg controlling the false-discovery proportion at alpha=0.05 as designed. baseline's low hotspot precision (36%) is a consequence of that same 5% background rate applied to a 150-segment background class outnumbering the 5-segment true-hotspot class 30-to-1, not evidence nearmiss cries wolf more than its own design target — background_fp_rate is the correct column to read that claim off, and it does not support it.

None of this is tuned to make nearmiss look good — regenerate it yourself and, if you find nearmiss's numbers wrong, that is exactly the kind of bug report this suite exists to make possible.