Companion to ../TECHNICAL_ARCHITECTURE.md. This is the Python + DuckDB (spatial) pipeline that turns upstream ATC/USGS/opentrail.org data into the packages the client app downloads.
python -m venv .venv
.venv/Scripts/pip install duckdb requests rasterio numpy # Windows
# .venv/bin/pip install duckdb requests rasterio numpy # macOS/Linux
sources.json is the registry of every upstream ATC layer we pull from, and it's generated by discover_sources.py, not hand-written. The ATC doesn't publish a plain list of data downloads - their public map is an ArcGIS Experience Builder app - so this script walks the same chain a browser resolves at load time to find the real FeatureServer URLs underneath it:
Experience app item -> app config -> dataSources (WEB_MAP entries)
-> web map item -> web map data -> operationalLayers[].url
.venv/Scripts/python discover_sources.py "https://experience.arcgis.com/experience/<ITEM_ID>"
Re-run it whenever ATC's map might have changed (new layers, moved services). It's additive/safe by design:
- Existing keys get their
url/titlerefreshed in place (and it prints a note if a url actually changed). - Hand-added fields like
noteson an entry are preserved across re-runs. - A source that disappears from the app is kept, not deleted, with a warning - that usually means the app changed, not that the data is gone, and deleting registry entries automatically is too destructive for something this script can't fully verify.
Not everything in sources.json came from the Experience Builder app, though - bridges, privies, and at_treadway were found by listing the ANST_Facilities FeatureServer's root directly (it has more layers than the public map curates), and added by hand. discover_sources.py won't touch those unless they later show up in the app's own layer list too.
To point the script at a different Experience Builder app (e.g. if the ATC publishes a second public map, or another club/provider uses one), just pass that app's URL and a --provider label - nothing else in the script is ATC-specific.
fetch_all.py reads sources.json and downloads each layer (paginating past ArcGIS's per-request record cap via lib/arcgis.py) to data/raw/<key>.geojson. It's change-aware: before doing the full paginated fetch, it checks each layer's editingInfo.dataLastEditDate (one cheap metadata request) against the value recorded in data/raw/manifest.json from the last run, and skips the source entirely if unchanged. Only writes the manifest if every registered source succeeded or was confirmed up to date - if any layer fails or goes missing, it exits non-zero instead of silently producing a partial dataset.
.venv/Scripts/python fetch_all.py
| key | title | count | geometry | notes |
|---|---|---|---|---|
trail_club_sections |
A.T. Trail Club Sections | 30 | Polygon | One polygon per maintaining club's section (incl. NYNJTC) |
centerline |
A.T. Centerline | 3,025 | Line | The trail itself, segmented; Trail_Club/Acronym fields identify maintainer per segment |
side_trails |
A.T. Side Trails | 1,200 | Line | Blue-blazed and other connector trails |
campsites |
A.T. Campsites | 232 | Point | Detailed facility attributes (tent pads, food storage, etc.) |
shelters |
A.T. Shelters | 280 | Point | Detailed facility/construction attributes |
parking |
A.T. Parking | 482 | Point | Trailhead parking areas |
viewpoints |
A.T. Viewpoints | 1,223 | Point | Scenic overlooks |
communities |
A.T. Communities | 59 | Point | Designated "A.T. Community" towns - partial resupply proxy |
half_mile_points_from_springer |
A.T. Half Mile Points From Springer | 4,395 | Point | Mile-marker points along the trail |
bridges |
A.T. Bridges | 409 | Point | Found via the FeatureServer root, not the public map |
privies |
A.T. Privies | 316 | Point | Found via the FeatureServer root, not the public map |
at_treadway |
A.T. Treadway | 30 | ? | Found via the FeatureServer root - not yet checked how this differs from centerline |
Gap, now partially filled: ATC's own data has no dedicated water-source or general resupply layer. communities is a partial resupply proxy; fetch_opentrail.py (below) is the real fill for both water and resupply.
fetch_opentrail.py pulls AT waypoints from opentrail.org's public API (/api/getData?trail=AT) - 1,840 features, of which 142 are tagged water sources (w), 72 resupply (r), and 103 towns (t), the gap ATC's own data leaves. Licensing isn't formally confirmed (no LICENSE file in their repo; the maintainer reportedly called it "open data" in a Reddit post - see the ROADMAP.md Phase 1 todo to follow up directly), so this deliberately excludes their user comments (personal contributions from named individuals - a separate consent question from licensing).
.venv/Scripts/python fetch_opentrail.py
Change-aware via real HTTP conditional requests (the API documents ETag/If-None-Match support) - a 304 response means skip, no re-parsing or re-saving.
fetch_topo_quads.py pulls the raster quads used for the background map. Rather than the TNM Access API (flaky pagination, multiple-editions-per-quad problems - see the script's docstring for the full story of what didn't work), it uses USGS's own metadata inventory (ustopo_current.csv) to find exactly which quads intersect the 30-mile corridor, then matches each to its real GeoTIFF file by listing each state's S3 folder directly (the CSV's own filename field is unreliable for constructing the URL).
.venv/Scripts/python fetch_topo_quads.py
Scope: 1,654 quads, ~14GB (vs. 300-500GB for a naive full-state pull across all 14 AT states) - corridor-scoping is what keeps this download/hosting-sized reasonably (value #8). Change-aware per-quad via S3 Last-Modified against data/raw/topo_quads/manifest.json.
Known data-quality issue: 3 of the 1,654 quads (NC_Glade_Valley, VA_Marion, WV_Princeton) are genuinely corrupted on USGS's own S3 bucket - confirmed via two independent codebases (rasterio/GDAL and tifffile/imagecodecs both fail to decode the same strip on a byte-exact-verified fresh download), not a truncated download on our end. fetch_topo_quads.py itself only checks HTTP presence/Last-Modified, not actual readability, so this kind of corruption goes undetected until something tries to read the file - run fix_corrupted_quads.py after any fresh fetch_topo_quads.py run to catch and work around this (see below). Open item: add a lightweight read-check to fetch_topo_quads.py itself so this isn't a separate manual step forever.
fix_corrupted_quads.py re-downloads each known-bad quad once (in case it was a transient issue), and if still corrupted, fetches a substitute covering the same footprint from basemap.nationalmap.gov's live export service (the same one used in the raster spike below), saved to data/raw/topo_quads_fallback/. spike_raster_mosaic.py includes these fallback files alongside the bulk quads automatically.
.venv/Scripts/python fix_corrupted_quads.py
If a different quad turns out corrupted later (found via spike_raster_mosaic.py's full-band validation pass, which checks every quad), add it to the BAD_QUADS dict in this script.
spike_corridor.py proves the core Phase 1 operation: buffer the full AT centerline (3,025 segments, GA to Maine) by 30 miles, union it into one corridor polygon (~81,138 sq mi), and clip real ATC POI data (campsites, shelters) against it.
.venv/Scripts/python spike_corridor.py
Output goes to data/spike/ (corridor.geojson, campsites_clipped.geojson, shelters_clipped.geojson).
Gotcha hit and fixed - watch for this in any future ST_Transform call: EPSG:4326's authority-defined axis order is (lat, lon), but every geometry source we actually use (GeoJSON, GeoPandas, etc.) is (lon, lat). Without always_xy := true, ST_Transform silently swaps the axes instead of erroring - the buffer/union still "succeeds" but produces geometry transformed as if every point were on the wrong side of the globe, which only surfaced as ST_Area returning nan on the reprojected-back result. Always pass always_xy := true on both the forward and inverse transform.
spike_raster_mosaic.py mosaics the real 1,654 downloaded US Topo quads into the actual corridor-clipped background raster - the full-scale version of the corridor-clip method proven above.
.venv/Scripts/python spike_raster_mosaic.py
Real complications this had to handle that a single-tile test wouldn't:
- Each quad is in its own native UTM zone (they vary by longitude across the trail), so they can't be merged directly - each is lazily reprojected to EPSG:4326 via a
WarpedVRTbefore merging. - One mosaic sized to the corridor's full bounding rectangle would be enormous even downsampled, since the actual corridor is a thin ~60-mile-wide winding band, not a filled rectangle (GA to Maine as a rectangle is most empty space). So this processes in small geographic cells (matching the corridor-intersecting grid used elsewhere in the pipeline) and outputs one clipped tile per cell - the same reason real map tile systems don't ship one giant image.
- Corrupted source quads (see above) are validated with a full-band read (a corner-pixel read isn't reliable - it missed 2 of the 3 known-bad quads) and skipped/substituted rather than crashing the run.
- Every corridor-intersecting cell must produce a tile - a hard completeness check at the end (matching the pattern used everywhere else in this pipeline) fails the run if any cell has no output, rather than silently reporting a partial result as if it were complete.
- Every US Topo GeoTIFF is a scan of the entire printed map sheet, not just the map - a white margin plus a header/footer collar (USGS/US Topo logos, title, scale bar, legend, adjoining-quadrangle diagram), and its georeferenced raster extent covers that whole sheet. Left uncropped, the collar gets treated as real terrain - it showed up as white bands and text baked into the exported background. Fixed by cropping every quad to its real neatline (the actual mapped area) before reprojecting, using the
westbc/eastbc/northbc/southbccolumns already present in USGS's own metadata CSV (ustopo_current.csv, fetched byfetch_topo_quads.py) - a clean 7.5'x7.5' box per quad, confirmed noticeably smaller than the raw raster's full extent (e.g.CT_Ansonia: a 0.125x0.125deg neatline vs a ~0.17x0.16deg raster).
Output: 51 tiles in data/processed/topo_background/, ~9.5GB total, at a fixed ~11m/pixel resolution (plenty for a phone background map at this stage).
export_pmtiles.py packages the 51 corridor-clipped cells above into a single PMTiles archive - a Web Mercator XYZ tile pyramid MapLibre GL JS reads directly in-browser, no tile server required (see TECHNICAL_ARCHITECTURE.md's "Export" pipeline step).
.venv/Scripts/python export_pmtiles.py
Optional flags override the defaults below: --min-zoom, --max-zoom, --out. All three download tiers are built this way - see "All three download tiers are built" below for the exact commands and measured sizes.
Format choices, backed by real measurements on this project's own data, not defaults:
- 512x512px tiles, WebP quality 80. WebP came out ~7-8x smaller than PNG on real sample tiles with no visible quality loss for a background/context basemap (the safety-relevant POI data is separate vector GeoJSON, untouched by this lossy step). 512px tiles (via MapLibre's
tileSize: 512raster option) compress better per unit ground area than 256px - a 512px tile at zoom z covers the same ground area as four 256px tiles at zoom z+1, so it's the same effective resolution with 4x fewer files and less per-file header overhead. - Zoom levels 6 through 12 by default in this 512px scheme - equivalent in ground resolution to 256px zoom 7-13. Corridor-vs-tile intersection uses shapely in a plain Python loop, not one DuckDB query per candidate tile (an earlier attempt at the latter was killed after 2 minutes with zero output - loading the corridor polygon once and testing intersections directly finishes in seconds).
Real output at the default max zoom (12): 5,816 tiles, 0 empty/skipped, 314MB for the entire 2,190-mile trail's background map.
Why 12, not 13 (changed 2026-07-28): the first real run went to zoom 13 (matching the source's native ~11m/pixel resolution exactly) and came to 1.18GB. Measuring the real per-zoom breakdown of that archive found zoom 13 alone was 73% of every byte in it:
| zoom | tiles | size | % of total |
|---|---|---|---|
| 6-10 | 501 | 12.3 MB | 1.0% |
| 11 | 1,139 | 52.0 MB | 4.4% |
| 12 | 4,176 | 249.6 MB | 21.1% |
| 13 | 15,932 | 868.0 MB | 73.4% |
For a background/context layer (not the precise trail line or POI locations, which are separate vector data), that last zoom level's detail gain wasn't worth 73% of the download - so 12 is now the default, cutting the whole-corridor package by ~73% for a still-detailed ~19m/pixel result. The zoom-13 archive isn't gone, just not the default - it's kept as data/processed/background_z13.pmtiles (rebuilt via --max-zoom=13 --out=...) for ROADMAP.md's planned per-user zoom-choice setting (Phase 2) - hikers with more data/storage headroom will eventually be able to opt into more detail.
The per-user zoom choice above is no longer a plan - the app ships it. client/src/lib/downloadDetail.ts offers Light / Standard / Fine, publish.py's BACKGROUND_ARCHIVES maps each tier to its archive, and all three now exist:
| tier | zooms | build | tiles | measured | app advertises | drift |
|---|---|---|---|---|---|---|
| Light | 6-11 | --max-zoom=11 --out=data/processed/background_z11.pmtiles |
1,640 | 64,403,146 B (64.4 MB) | 64 MB | +0.6% |
| Standard | 6-12 | (default) | 5,816 | 313,987,530 B (314 MB) | 314 MB | -0.0% |
| Fine | 6-13 | --max-zoom=13 --out=data/processed/background_z13.pmtiles |
21,747 | 1,182,024,009 B (1.18 GB) | 1.18 GB | +0.2% |
All three land within 0.6% of the figure the app shows before someone commits to a download - which matters because that figure is weighed against remaining phone storage at a trailhead, not read as an approximation. Worth re-measuring whenever the source rasters are refreshed; a tier drifting far from its advertised size is a real problem, not a rounding detail.
Light came out almost exactly on the estimate implied by the per-zoom table above (52.0 MB for z11 plus 12.3 MB for z6-10 = 64.3 MB), which is a useful confirmation that the breakdown is trustworthy for predicting future tiers.
A completeness check at the end (mirroring the pattern in spike_raster_mosaic.py) confirms every one of the 51 source cells contributed to at least one tile at max zoom, failing loudly rather than silently shipping a coverage gap.
See ../ROADMAP.md Phase 1 for what's still open - notably the unified POI schema (joining ATC + opentrail.org + NHD into one schema) and the real Publish step (change-aware release to Cloudflare R2 so hikers don't re-download data that hasn't changed). Chunking granularity decided 2026-07-28: whole corridor, one package (not per-state/per-section) - see ROADMAP.md Phase 2's "Offline download flow" for why, and its new "quality/size tradeoff in settings" item for the zoom-11/12/13 choice this Export step now supports.