This describes how OurHike tests its code, project-wide - not just the Python data pipeline, which is the only thing built so far. When the client (React/TypeScript PWA) or a backend (Python/FastAPI, Phase 2+) exist, they get their own section below following the same philosophy, not a separate one invented from scratch.
Most of the pipeline was built through manual, ad hoc verification this session: one-off scripts, print statements, eyeballing results. That caught real bugs - an ST_Transform axis-order bug that silently produced garbage geometry, a band-count mismatch between fallback and bulk raster tiles, 3 genuinely corrupted USGS source files a shallow check missed - but none of those catches were permanent. Nothing stopped any of them from silently coming back the next time that code was touched.
The core rule: every real gotcha becomes a regression test in the same change that fixes it - not just a comment. A comment explains the bug to a human reading the code later; a test catches it automatically for a machine that isn't reading comments. This project has already paid for several of these lessons the hard way - the tests exist so nobody pays for them twice.
- Pure functions - zero setup, run in milliseconds. Every pure function should have one.
- Small-synthetic-fixture tests for anything spatial/numerical/binary (geometry, rasters, etc.) - tiny fixtures generated in test code, not committed as opaque binary files. A test that builds its own "corrupted" file byte-for-byte documents exactly what "corrupted" means; a checked-in blob doesn't.
- HTTP-mocked tests for any network-touching logic (especially change-detection/skip-logic, which is easy to silently break and expensive to notice - you'd only find out via a full re-fetch that should've been skipped). Real network calls are never allowed to fire during tests.
- Real network calls to live third-party services (ArcGIS, opentrail.org, USGS, etc.) in the automated suite - slow, flaky, not reproducible, and impolite to hammer on every test run.
- Real large datasets (e.g. the full 1,654-quad USGS fetch/mosaic) - too slow/heavy for a test suite. Verifying the real pipeline end-to-end against real data stays a documented manual procedure, not a test - see each component's section below for what that looks like today.
- Pixel-perfect "golden image" comparisons for raster output - assert structural properties (band count, CRS, nodata boundary location, no unexpected NaNs) instead of byte-identical files, which are too fragile to maintain as inputs change.
Found a bug where the code did something surprising or silently wrong? Before moving on:
- Fix it.
- Write a test named for the behavior it guards against (e.g.
test_full_band_read_catches_late_strip_corruption, nottest_bug_47), in the same commit/PR as the fix. - If it's fast to check, verify the test actually fails against the pre-fix code (revert the fix locally, confirm red, restore it, confirm green) - proves the test has teeth instead of passing vacuously.
Framework: pytest, with requests-mock for HTTP isolation (any unmocked request raises rather than silently hitting the network) and pytest-cov for a visibility-only coverage report (see below - not a merge gate).
Lint/format: Ruff, pinned to a narrow rule set (pyflakes + core pycodestyle errors + import sorting - see the comment in pyproject.toml's [tool.ruff.lint]) rather than its much larger default rule set, and matching this project's "boring, low-maintenance" preference over a maximalist lint config.
Quick start:
cd pipeline
.venv/Scripts/pip install -r requirements-dev.txt # Windows; .venv/bin/pip on macOS/Linux
.venv/Scripts/python -m pytest # run everything (prints a coverage summary too)
.venv/Scripts/python -m pytest tests/test_x.py # one file
.venv/Scripts/python -m pytest -k test_name # one test
.venv/Scripts/python -m ruff check . # lint
.venv/Scripts/python -m ruff format . # auto-format
On coverage: the report is there to show what's untested, not to chase a percentage - CI doesn't fail on a coverage threshold. Padding coverage numbers pulls in the opposite direction from this file's actual rule (test real gotchas as regressions), so it's deliberately not gated.
Layout: pipeline/tests/, one file per source module, conftest.py for shared fixtures. pipeline/pyproject.toml sets pythonpath = ["."] so tests can import fetch_all, from lib import arcgis, etc. directly - these are standalone scripts, not an installed package.
Network isolation caveat: DuckDB's spatial extension (INSTALL spatial) fetches from DuckDB's own extension repository over the network on first use per machine - this is a one-time local setup step, not a per-test-run network call, but it's a real exception to "tests never touch the network" worth knowing about if a fresh environment's first test run looks like it's doing something unexpected.
Blaze normalization, still to build (see features/TRAIL_BLAZE_COLORS.md): decoding each trail-line source's raw color coding into one normalized blaze_color attribute happens here, during export - not on the client (see the Client section above, which only tests the client's use of the already-normalized value). Once built, this needs a regression test for exactly the gotcha TRAIL_BLAZE_COLORS.md already names from the real data: side_trails' Blaze field has 24 features with no value at all, 9 with the literal string "Unknown", and 3 with "Gold", none of which are in its actual 0-9 coded domain. The test should assert all three fall through to the neutral default with a warning logged - not a crash, and not a silently wrong color.
What's intentionally manual-only: fetching the real ~1,650-quad USGS corridor dataset and mosaicking it (fetch_topo_quads.py + spike_raster_mosaic.py) is a real multi-GB, multi-minute operation against live services - run it by hand to verify changes that touch fetch/mosaic logic, don't expect it in pytest.
Not scaffolded yet (see ROADMAP.md Phase 2). Framework: Vitest + React Testing Library - Vitest because it shares Vite's transform pipeline (the confirmed bundler per TECHNICAL_ARCHITECTURE.md) instead of adding a second one; same philosophy as the pipeline above (pure-function/component unit tests, no real network in the suite, mock any data/API calls), not a from-scratch set of conventions.
The test plan below is drawn from WIREFRAMES.md - it names the behaviors those v1 MVP screens need covered, written before the client itself so the first pass at each area can be built test-first per this file's core rule. Written as behaviors, not implementations, so they survive refactors.
Pure logic (fast unit tests, zero rendering):
- Blaze normalization (client's half) - the pipeline decodes raw source data into a normalized
blaze_colorstring during export (see the Pipeline section below); the client's job is just mapping that already-clean string to a MapLibre paint style via amatchexpression. Test the client's defensive fallback: anyblaze_colorvalue that isn't one of the expected strings renders as the neutral grey and emits a warning, rather than the client trusting the pipeline blindly or crashing on an unexpected value. - Naismith - 2.6 mi / 640 ft ⇒
≈1h 10m; rounds to 5-minute steps; descent never subtracts; output always carries≈; never formats an arrival time. - Download detail levels - each of Light/Standard/Fine maps to its correct zoom (z11/z12/z13) and its correct measured size (64 MB/314 MB/1.18 GB, from
pipeline/README.md) as a table-driven test, guarding against one of the three drifting out of sync with the other two. No per-section math - see WIREFRAMES.md Known Deviations #1. - Staleness tiers - boundary tests at 14 and 60 days,
never confirmed⇒ stale, and staleness is independent of the verified/unverified flag (all four combinations produce the right pin treatment). - Onboarding step counter - total derived from the live step list; skipping a step does not change the total; adding a future step (trail name) yields "of 4" without touching call sites.
- Wrong-way detector - table-driven against synthetic GPS traces: short backtrack to a spring ⇒ silent; standing still at a shelter ⇒ silent; sustained reversal past the persistence threshold ⇒ cue, then push; off-line distance under threshold ⇒ silent. False negatives are acceptable; false positives are the failure.
Component tests: 7. Legend contents - equals exactly what the viewport contains, with correct counts; recomputes on pan/zoom; closure and serious-warning rows render without a hide control. 8. Map chrome - compass + locate present and ≥44px effective target; zoom buttons absent on mobile, present on web; scale bar respects the unit preference; attribution always rendered. 9. Report status rendering - each of Waiting / Confirmed / Fixed / Not confirmed renders its own affordance; "Not confirmed" applies no penalty styling to the reporter. 10. Closure sheet - shows status, dates, marked-by, sync age; offers no detour computation; renders the barred-band line treatment distinctly from a red blaze.
Integration / E2E: 11. First run - three onboarding steps, each skippable; location asked only after the value-prop screen; no notification prompt anywhere in first run; a fresh install can reach a usable map with zero taps beyond the flow. 12. Report -> sign-in -> submit - the report survives the sign-in detour intact; the stored timestamp is the authoring time, not the send time; internal-only report types never appear in any public query result (mock the API here - see Backend below for the server-side half of the same invariant). 13. Offline outbox - write three items with the network off, restore network, all three sync with original timestamps and no duplicates; deleting a queued item deletes it for good. 14. Download + detail - choose each of Light/Standard/Fine for the single whole-corridor package; the size shown matches the measured figure. 15. Closure freshness - with a 3-day-old package, the closure sheet shows the sync age; after sync, newly-marked closures appear.
Invariants worth asserting explicitly (regression guards): 16. No route in the app exposes a closures toggle or a warnings toggle - assert on the settings schema, not the DOM. 17. Serious warnings never enqueue a push; the wrong-way alert is the only push publisher in the client codebase.
Field testing (not automatable): thresholds for off-trail distance and wrong-direction persistence need real GPS behaviour under tree canopy, ideally with NYNJTC/ATC volunteers, before the push path ships. Sunlight-glare readability and gloved one-handed use likewise (WIREFRAMES.md's 9d is the greyscale pass to test against).
Not scaffolded yet (see TECHNICAL_ARCHITECTURE.md). When it exists, it would likely reuse pytest and much of the pipeline's approach (HTTP mocking, synthetic fixtures over real data/DB where possible). Three invariants from the wireframe handoff belong here specifically, since they're only meaningfully enforceable server-side:
severity: seriouson aReportis only ever set by a user with a moderator role; a self-set attempt is rejected server-side, not just hidden client-side.- Any report type intended to stay private (
bad_hikerstoday - see WIREFRAMES.md Known Deviations #2 for the still-open question of exactly what replaces it) hasvisibility: internal-onlyset on write, and public map/search API queries filter it out at the query level, not just in client rendering. - Browsing endpoints (map, POIs, elevation, closures, warnings) require no auth token - a signed-out client can fetch all of them.
.github/workflows/pipeline-tests.yml runs ruff check, ruff format --check, and pytest on every push and on PRs targeting main. It's not (yet) a required check via branch protection - a red run doesn't currently block merging, it's just visible on the PR.