OurHike data pipeline

Companion to ../TECHNICAL_ARCHITECTURE.md. This is the Python + DuckDB (spatial) pipeline that turns upstream ATC/USGS/opentrail.org data into the packages the client app downloads.

Setup

python -m venv .venv
.venv/Scripts/pip install duckdb requests rasterio numpy   # Windows
# .venv/bin/pip install duckdb requests rasterio numpy     # macOS/Linux

Discovering source URLs

sources.json is the registry of every upstream ATC layer we pull from, and it's generated by discover_sources.py, not hand-written. The ATC doesn't publish a plain list of data downloads - their public map is an ArcGIS Experience Builder app - so this script walks the same chain a browser resolves at load time to find the real FeatureServer URLs underneath it:

Experience app item -> app config -> dataSources (WEB_MAP entries)
-> web map item -> web map data -> operationalLayers[].url

.venv/Scripts/python discover_sources.py "https://experience.arcgis.com/experience/<ITEM_ID>"

Re-run it whenever ATC's map might have changed (new layers, moved services). It's additive/safe by design:

• Existing keys get their url/title refreshed in place (and it prints a note if a url actually changed).
• Hand-added fields like notes on an entry are preserved across re-runs.
• A source that disappears from the app is kept, not deleted, with a warning - that usually means the app changed, not that the data is gone, and deleting registry entries automatically is too destructive for something this script can't fully verify.

Not everything in sources.json came from the Experience Builder app, though - bridges, privies, and at_treadway were found by listing the ANST_Facilities FeatureServer's root directly (it has more layers than the public map curates), and added by hand. discover_sources.py won't touch those unless they later show up in the app's own layer list too.

To point the script at a different Experience Builder app (e.g. if the ATC publishes a second public map, or another club/provider uses one), just pass that app's URL and a --provider label - nothing else in the script is ATC-specific.

Fetching ATC sources

fetch_all.py reads sources.json and downloads each layer (paginating past ArcGIS's per-request record cap via lib/arcgis.py) to data/raw/<key>.geojson. It's change-aware: before doing the full paginated fetch, it checks each layer's editingInfo.dataLastEditDate (one cheap metadata request) against the value recorded in data/raw/manifest.json from the last run, and skips the source entirely if unchanged. Only writes the manifest if every registered source succeeded or was confirmed up to date - if any layer fails or goes missing, it exits non-zero instead of silently producing a partial dataset.

.venv/Scripts/python fetch_all.py

Current ATC sources (as of 2026-07-25)

key	title	count	geometry	notes
trail_club_sections	A.T. Trail Club Sections	30	Polygon	One polygon per maintaining club's section (incl. NYNJTC)
centerline	A.T. Centerline	3,025	Line	The trail itself, segmented; Trail_Club/Acronym fields identify maintainer per segment
side_trails	A.T. Side Trails	1,200	Line	Blue-blazed and other connector trails
campsites	A.T. Campsites	232	Point	Detailed facility attributes (tent pads, food storage, etc.)
shelters	A.T. Shelters	280	Point	Detailed facility/construction attributes
parking	A.T. Parking	482	Point	Trailhead parking areas
viewpoints	A.T. Viewpoints	1,223	Point	Scenic overlooks
communities	A.T. Communities	59	Point	Designated "A.T. Community" towns - partial resupply proxy
half_mile_points_from_springer	A.T. Half Mile Points From Springer	4,395	Point	Mile-marker points along the trail
bridges	A.T. Bridges	409	Point	Found via the FeatureServer root, not the public map
privies	A.T. Privies	316	Point	Found via the FeatureServer root, not the public map
at_treadway	A.T. Treadway	30	?	Found via the FeatureServer root - not yet checked how this differs from centerline

Gap, now partially filled: ATC's own data has no dedicated water-source or general resupply layer. communities is a partial resupply proxy; fetch_opentrail.py (below) is the real fill for both water and resupply.

Fetching opentrail.org (water sources + resupply)

fetch_opentrail.py pulls AT waypoints from opentrail.org's public API (/api/getData?trail=AT) - 1,840 features, of which 142 are tagged water sources (w), 72 resupply (r), and 103 towns (t), the gap ATC's own data leaves. Licensing isn't formally confirmed (no LICENSE file in their repo; the maintainer reportedly called it "open data" in a Reddit post - see the ROADMAP.md Phase 1 todo to follow up directly), so this deliberately excludes their user comments (personal contributions from named individuals - a separate consent question from licensing).

.venv/Scripts/python fetch_opentrail.py

Change-aware via real HTTP conditional requests (the API documents ETag/If-None-Match support) - a 304 response means skip, no re-parsing or re-saving.

Fetching USGS US Topo background quads

fetch_topo_quads.py pulls the raster quads used for the background map. Rather than the TNM Access API (flaky pagination, multiple-editions-per-quad problems - see the script's docstring for the full story of what didn't work), it uses USGS's own metadata inventory (ustopo_current.csv) to find exactly which quads intersect the 30-mile corridor, then matches each to its real GeoTIFF file by listing each state's S3 folder directly (the CSV's own filename field is unreliable for constructing the URL).

.venv/Scripts/python fetch_topo_quads.py

Scope: 1,654 quads, ~14GB (vs. 300-500GB for a naive full-state pull across all 14 AT states) - corridor-scoping is what keeps this download/hosting-sized reasonably (value #8). Change-aware per-quad via S3 Last-Modified against data/raw/topo_quads/manifest.json.

Known data-quality issue: 3 of the 1,654 quads (NC_Glade_Valley, VA_Marion, WV_Princeton) are genuinely corrupted on USGS's own S3 bucket - confirmed via two independent codebases (rasterio/GDAL and tifffile/imagecodecs both fail to decode the same strip on a byte-exact-verified fresh download), not a truncated download on our end. fetch_topo_quads.py itself only checks HTTP presence/Last-Modified, not actual readability, so this kind of corruption goes undetected until something tries to read the file - run fix_corrupted_quads.py after any fresh fetch_topo_quads.py run to catch and work around this (see below).

Still an open item for this whole-corridor path specifically: add a lightweight read-check to fetch_topo_quads.py itself so this isn't a separate manual step. The per-cell CI path (fetch_and_mosaic_cell.py, see further down) already does this - it calls fix_corrupted_quads.py's recovery logic (fix_quad()) inline the moment a quad fails validation, rather than requiring a second command. That couldn't be backported here without fetch_topo_quads.py importing from fix_corrupted_quads.py, which already imports from it (bare_key) - a circular import. Untangling that (probably by moving bare_key somewhere both can import from) is real but small future work, not done as part of adding the per-cell path.

Fixing corrupted quads

fix_corrupted_quads.py re-downloads each known-bad quad once (in case it was a transient issue), and if still corrupted, fetches a substitute covering the same footprint from basemap.nationalmap.gov's live export service (the same one used in the raster spike below), saved to data/raw/topo_quads_fallback/. spike_raster_mosaic.py includes these fallback files alongside the bulk quads automatically.

.venv/Scripts/python fix_corrupted_quads.py

If a different quad turns out corrupted later (found via spike_raster_mosaic.py's full-band validation pass, which checks every quad), add it to the BAD_QUADS dict in this script.

Spatial spike: corridor computation (done)

spike_corridor.py proves the core Phase 1 operation: buffer the full AT centerline (3,025 segments, GA to Maine) by 30 miles, union it into one corridor polygon (~81,138 sq mi), and clip real ATC POI data (campsites, shelters) against it.

.venv/Scripts/python spike_corridor.py

Output goes to data/spike/ (corridor.geojson, campsites_clipped.geojson, shelters_clipped.geojson).

Gotcha hit and fixed - watch for this in any future ST_Transform call: EPSG:4326's authority-defined axis order is (lat, lon), but every geometry source we actually use (GeoJSON, GeoPandas, etc.) is (lon, lat). Without always_xy := true, ST_Transform silently swaps the axes instead of erroring - the buffer/union still "succeeds" but produces geometry transformed as if every point were on the wrong side of the globe, which only surfaced as ST_Area returning nan on the reprojected-back result. Always pass always_xy := true on both the forward and inverse transform.

Raster background: mosaic + clip at real scale (done)

spike_raster_mosaic.py mosaics the real 1,654 downloaded US Topo quads into the actual corridor-clipped background raster - the full-scale version of the corridor-clip method proven above.

.venv/Scripts/python spike_raster_mosaic.py

Real complications this had to handle that a single-tile test wouldn't:

• Each quad is in its own native UTM zone (they vary by longitude across the trail), so they can't be merged directly - each is lazily reprojected to EPSG:4326 via a WarpedVRT before merging.
• One mosaic sized to the corridor's full bounding rectangle would be enormous even downsampled, since the actual corridor is a thin ~60-mile-wide winding band, not a filled rectangle (GA to Maine as a rectangle is most empty space). So this processes in small geographic cells (matching the corridor-intersecting grid used elsewhere in the pipeline) and outputs one clipped tile per cell - the same reason real map tile systems don't ship one giant image.
• Corrupted source quads (see above) are validated with a full-band read (a corner-pixel read isn't reliable - it missed 2 of the 3 known-bad quads) and skipped/substituted rather than crashing the run.
• Every corridor-intersecting cell must produce a tile - a hard completeness check at the end (matching the pattern used everywhere else in this pipeline) fails the run if any cell has no output, rather than silently reporting a partial result as if it were complete.
• Every US Topo GeoTIFF is a scan of the entire printed map sheet, not just the map - a white margin plus a header/footer collar (USGS/US Topo logos, title, scale bar, legend, adjoining-quadrangle diagram), and its georeferenced raster extent covers that whole sheet. Left uncropped, the collar gets treated as real terrain - it showed up as white bands and text baked into the exported background. Fixed by cropping every quad to its real neatline (the actual mapped area) before reprojecting, using the westbc/eastbc/northbc/southbc columns already present in USGS's own metadata CSV (ustopo_current.csv, fetched by fetch_topo_quads.py) - a clean 7.5'x7.5' box per quad, confirmed noticeably smaller than the raw raster's full extent (e.g. CT_Ansonia: a 0.125x0.125deg neatline vs a ~0.17x0.16deg raster).

Output: 51 tiles in data/processed/topo_background/, ~9.5GB total, at a fixed ~11m/pixel resolution (plenty for a phone background map at this stage).

Per-cell fetch + mosaic (for CI, not local use)

fetch_and_mosaic_cell.py fetches and mosaics exactly one corridor cell - the unit of work a GitHub Actions matrix job runs so this raster pipeline can execute on a hosted runner's disk, which can't hold the whole corridor's ~14GB raw + ~9.5GB processed data at once. Peak disk for a single cell is a few hundred MB. It's built entirely from the same functions the two whole-corridor scripts above use internally (fetch_quads_for_cell, resolve_state_index, index_quads_in_dir, mosaic_one_cell), plus fix_corrupted_quads.py's fix_quad() for inline corruption recovery - unlike the whole-corridor path, a corrupted quad here is redownloaded-then-substituted in the same job, not flagged for a separate manual step.

Needs cells.json first - the corridor's cell grid plus each cell's quad list (lib/corridor_grid.py), computed once from small vector data, no rasters touched:

.venv/Scripts/python build_cells_manifest.py
.venv/Scripts/python fetch_and_mosaic_cell.py --cell-index 0

Does not replace the whole-corridor workflow above - fetch_topo_quads.py/spike_raster_mosaic.py are still how a maintainer runs the full pipeline locally. The two paths never share fetched files: the per-cell path downloads into data/raw/topo_quads_cell_NNN/, not data/raw/topo_quads/<state>/, so a local whole-corridor run and a local per-cell run can coexist on the same checkout without colliding.

Roughly 22.9% of corridor quads (378 of 1,654) bbox-overlap more than one 1-degree cell, so a quad near a cell boundary gets fetched once per owning cell rather than shared - about 3.6GB of deliberate, bounded redundancy across the whole corridor, accepted rather than adding a cross-job quad cache (which would reintroduce the disk/coordination problem this per-cell split exists to avoid).

Exporting the background as PMTiles (done)

export_pmtiles.py packages the 51 corridor-clipped cells above into a single PMTiles archive - a Web Mercator XYZ tile pyramid MapLibre GL JS reads directly in-browser, no tile server required (see TECHNICAL_ARCHITECTURE.md's "Export" pipeline step).

.venv/Scripts/python export_pmtiles.py

Optional flags override the defaults below: --min-zoom, --max-zoom, --out. All three download tiers are built this way - see "All three download tiers are built" below for the exact commands and measured sizes.

Format choices, backed by real measurements on this project's own data, not defaults:

• 512x512px tiles, WebP quality 80. WebP came out ~7-8x smaller than PNG on real sample tiles with no visible quality loss for a background/context basemap (the safety-relevant POI data is separate vector GeoJSON, untouched by this lossy step). 512px tiles (via MapLibre's tileSize: 512 raster option) compress better per unit ground area than 256px - a 512px tile at zoom z covers the same ground area as four 256px tiles at zoom z+1, so it's the same effective resolution with 4x fewer files and less per-file header overhead.
• Zoom levels 6 through 12 by default in this 512px scheme - equivalent in ground resolution to 256px zoom 7-13. Corridor-vs-tile intersection uses shapely in a plain Python loop, not one DuckDB query per candidate tile (an earlier attempt at the latter was killed after 2 minutes with zero output - loading the corridor polygon once and testing intersections directly finishes in seconds).

Real output at the default max zoom (12): 5,816 tiles, 0 empty/skipped, 314MB for the entire 2,190-mile trail's background map.

Why 12, not 13 (changed 2026-07-28): the first real run went to zoom 13 (matching the source's native ~11m/pixel resolution exactly) and came to 1.18GB. Measuring the real per-zoom breakdown of that archive found zoom 13 alone was 73% of every byte in it:

zoom	tiles	size	% of total
6-10	501	12.3 MB	1.0%
11	1,139	52.0 MB	4.4%
12	4,176	249.6 MB	21.1%
13	15,932	868.0 MB	73.4%

For a background/context layer (not the precise trail line or POI locations, which are separate vector data), that last zoom level's detail gain wasn't worth 73% of the download - so 12 is now the default, cutting the whole-corridor package by ~73% for a still-detailed ~19m/pixel result. The zoom-13 archive isn't gone, just not the default - it's kept as data/processed/background_z13.pmtiles (rebuilt via --max-zoom=13 --out=...) for ROADMAP.md's planned per-user zoom-choice setting (Phase 2) - hikers with more data/storage headroom will eventually be able to opt into more detail.

All three download tiers are built (2026-07-29)

The per-user zoom choice above is no longer a plan - the app ships it. client/src/lib/downloadDetail.ts offers Light / Standard / Fine, publish.py's BACKGROUND_ARCHIVES maps each tier to its archive, and all three now exist:

tier	zooms	build	tiles	measured	app advertises	drift
Light	6-11	--max-zoom=11 --out=data/processed/background_z11.pmtiles	1,640	64,403,146 B (64.4 MB)	64 MB	+0.6%
Standard	6-12	(default)	5,816	313,987,530 B (314 MB)	314 MB	-0.0%
Fine	6-13	--max-zoom=13 --out=data/processed/background_z13.pmtiles	21,747	1,182,024,009 B (1.18 GB)	1.18 GB	+0.2%

All three land within 0.6% of the figure the app shows before someone commits to a download - which matters because that figure is weighed against remaining phone storage at a trailhead, not read as an approximation. Worth re-measuring whenever the source rasters are refreshed; a tier drifting far from its advertised size is a real problem, not a rounding detail.

Light came out almost exactly on the estimate implied by the per-zoom table above (52.0 MB for z11 plus 12.3 MB for z6-10 = 64.3 MB), which is a useful confirmation that the breakdown is trustworthy for predicting future tiers.

A completeness check at the end (mirroring the pattern in spike_raster_mosaic.py) confirms every one of the 51 source cells contributed to at least one tile at max zoom, failing loudly rather than silently shipping a coverage gap.

Checking output quality before publish (done)

check_output_quality.py is the gate between Export and Publish. It runs after export_trails.py, export_poi.py, export_elevation.py, and export_pmtiles.py have all produced their output, and before publish.py ships any of it to R2:

.venv/Scripts/python check_output_quality.py

It's check_freshness.py's output-side sibling: check_freshness.py (run before fetching) asks whether anything upstream has changed; this asks whether this run's own output can be trusted, after everything has already run. Four checks, in priority order - see the module's own docstring for the full reasoning behind each, especially the corridor one:

1. Completeness cross-check - re-reads trails_manifest.json, poi/manifest.json, and elevation_manifest.json, re-hashes the artifact file each one points at (catching drift between a manifest and what's actually on disk), and re-checks the same non-zero feature/point-count rule each exporting script already enforces on itself (crossing excepted, same as export_poi.py's own exception) - a second, independent check for the case where a script's own gate has a bug or got bypassed.
2. Corridor cross-check - the one check no single export script can run on itself. Rebuilds the 30-mile corridor twice, independently, from data/raw/centerline.geojson, and requires both a plausible (non-degenerate) result and agreement between the two builds.
3. fetch_topo_quads.py backstop - re-verifies that every quad recorded in data/raw/topo_quads/manifest.json still exists on disk, and that a sample of them still reads as a valid raster, as defense in depth alongside that script's own exit-code gate.
4. Drop-vs-baseline detection - compares this run's counts against data/quality_baseline.json (gitignored, like everything else under data/) and flags a count dropping more than ~10% with no matching check_freshness.py-reported upstream change. Only rewrites the baseline on a fully-passing run.

Check 4 needs to be told what changed upstream, because this gate deliberately never touches the network — standing directly in front of publish.py, it should not be able to fail because an upstream host is down. Pass the sources check_freshness.py reported as STALE:

.venv/Scripts/python check_output_quality.py --changed-source atc --changed-source opentrail

Without them the check is conservative rather than wrong: a real drop that an upstream refresh fully explains still gets flagged, so a legitimate change reads as a problem until someone says otherwise.

Exits non-zero if any check finds a real problem - publish.py shouldn't run after that until the cause is fixed.

Publishing

Publishing artifacts to Cloudflare R2 is intentionally write-disabled by default. Set R2_WRITE_ENABLED=true in the trusted environment that is allowed to publish; otherwise publish.py refuses to upload anything, so developers with only read access to the bucket cannot accidentally write to it.

To serve data/processed/ locally instead - for testing the client's offline download without publishing anything - use serve_processed.py, which answers byte-range requests and sets the CORS headers a cross-origin bucket needs. See ../client/README.md.

Next steps

See ../ROADMAP.md Phase 1 for what's still open - notably the unified POI schema (joining ATC + opentrail.org + NHD into one schema) and the real Publish step (change-aware release to Cloudflare R2 so hikers don't re-download data that hasn't changed). Chunking granularity decided 2026-07-28: whole corridor, one package (not per-state/per-section) - see ROADMAP.md Phase 2's "Offline download flow" for why, and its new "quality/size tradeoff in settings" item for the zoom-11/12/13 choice this Export step now supports.