Companion to ../TECHNICAL_ARCHITECTURE.md. This is the Python + DuckDB (spatial) pipeline that turns upstream ATC/USGS/opentrail.org data into the packages the client app downloads.
python -m venv .venv
.venv/Scripts/pip install duckdb requests rasterio numpy # Windows
# .venv/bin/pip install duckdb requests rasterio numpy # macOS/Linux
sources.json is the registry of every upstream ATC layer we pull from, and it's generated by discover_sources.py, not hand-written. The ATC doesn't publish a plain list of data downloads - their public map is an ArcGIS Experience Builder app - so this script walks the same chain a browser resolves at load time to find the real FeatureServer URLs underneath it:
Experience app item -> app config -> dataSources (WEB_MAP entries)
-> web map item -> web map data -> operationalLayers[].url
.venv/Scripts/python discover_sources.py "https://experience.arcgis.com/experience/<ITEM_ID>"
Re-run it whenever ATC's map might have changed (new layers, moved services). It's additive/safe by design:
- Existing keys get their
url/titlerefreshed in place (and it prints a note if a url actually changed). - Hand-added fields like
noteson an entry are preserved across re-runs. - A source that disappears from the app is kept, not deleted, with a warning - that usually means the app changed, not that the data is gone, and deleting registry entries automatically is too destructive for something this script can't fully verify.
Not everything in sources.json came from the Experience Builder app, though - bridges, privies, and at_treadway were found by listing the ANST_Facilities FeatureServer's root directly (it has more layers than the public map curates), and added by hand. discover_sources.py won't touch those unless they later show up in the app's own layer list too.
To point the script at a different Experience Builder app (e.g. if the ATC publishes a second public map, or another club/provider uses one), just pass that app's URL and a --provider label - nothing else in the script is ATC-specific.
fetch_all.py reads sources.json and downloads each layer (paginating past ArcGIS's per-request record cap via lib/arcgis.py) to data/raw/<key>.geojson. It's change-aware: before doing the full paginated fetch, it checks each layer's editingInfo.dataLastEditDate (one cheap metadata request) against the value recorded in data/raw/manifest.json from the last run, and skips the source entirely if unchanged. Only writes the manifest if every registered source succeeded or was confirmed up to date - if any layer fails or goes missing, it exits non-zero instead of silently producing a partial dataset.
.venv/Scripts/python fetch_all.py
| key | title | count | geometry | notes |
|---|---|---|---|---|
trail_club_sections |
A.T. Trail Club Sections | 30 | Polygon | One polygon per maintaining club's section (incl. NYNJTC) |
centerline |
A.T. Centerline | 3,025 | Line | The trail itself, segmented; Trail_Club/Acronym fields identify maintainer per segment |
side_trails |
A.T. Side Trails | 1,200 | Line | Blue-blazed and other connector trails |
campsites |
A.T. Campsites | 232 | Point | Detailed facility attributes (tent pads, food storage, etc.) |
shelters |
A.T. Shelters | 280 | Point | Detailed facility/construction attributes |
parking |
A.T. Parking | 482 | Point | Trailhead parking areas |
viewpoints |
A.T. Viewpoints | 1,223 | Point | Scenic overlooks |
communities |
A.T. Communities | 59 | Point | Designated "A.T. Community" towns - partial resupply proxy |
half_mile_points_from_springer |
A.T. Half Mile Points From Springer | 4,395 | Point | Mile-marker points along the trail |
bridges |
A.T. Bridges | 409 | Point | Found via the FeatureServer root, not the public map |
privies |
A.T. Privies | 316 | Point | Found via the FeatureServer root, not the public map |
at_treadway |
A.T. Treadway | 30 | ? | Found via the FeatureServer root - not yet checked how this differs from centerline |
Gap, now partially filled: ATC's own data has no dedicated water-source or general resupply layer. communities is a partial resupply proxy; fetch_opentrail.py (below) is the real fill for both water and resupply.
fetch_opentrail.py pulls AT waypoints from opentrail.org's public API (/api/getData?trail=AT) - 1,840 features, of which 142 are tagged water sources (w), 72 resupply (r), and 103 towns (t), the gap ATC's own data leaves. Licensing isn't formally confirmed (no LICENSE file in their repo; the maintainer reportedly called it "open data" in a Reddit post - see the ROADMAP.md Phase 1 todo to follow up directly), so this deliberately excludes their user comments (personal contributions from named individuals - a separate consent question from licensing).
.venv/Scripts/python fetch_opentrail.py
Change-aware via real HTTP conditional requests (the API documents ETag/If-None-Match support) - a 304 response means skip, no re-parsing or re-saving.
fetch_topo_quads.py pulls the raster quads used for the background map. Rather than the TNM Access API (flaky pagination, multiple-editions-per-quad problems - see the script's docstring for the full story of what didn't work), it uses USGS's own metadata inventory (ustopo_current.csv) to find exactly which quads intersect the 30-mile corridor, then matches each to its real GeoTIFF file by listing each state's S3 folder directly (the CSV's own filename field is unreliable for constructing the URL).
.venv/Scripts/python fetch_topo_quads.py
Scope: 1,654 quads, ~14GB (vs. 300-500GB for a naive full-state pull across all 14 AT states) - corridor-scoping is what keeps this download/hosting-sized reasonably (value #8). Change-aware per-quad via S3 Last-Modified against data/raw/topo_quads/manifest.json.
Known data-quality issue: 3 of the 1,654 quads (NC_Glade_Valley, VA_Marion, WV_Princeton) are genuinely corrupted on USGS's own S3 bucket - confirmed via two independent codebases (rasterio/GDAL and tifffile/imagecodecs both fail to decode the same strip on a byte-exact-verified fresh download), not a truncated download on our end. fetch_topo_quads.py itself only checks HTTP presence/Last-Modified, not actual readability, so this kind of corruption goes undetected until something tries to read the file - run fix_corrupted_quads.py after any fresh fetch_topo_quads.py run to catch and work around this (see below). Open item: add a lightweight read-check to fetch_topo_quads.py itself so this isn't a separate manual step forever.
fix_corrupted_quads.py re-downloads each known-bad quad once (in case it was a transient issue), and if still corrupted, fetches a substitute covering the same footprint from basemap.nationalmap.gov's live export service (the same one used in the raster spike below), saved to data/raw/topo_quads_fallback/. spike_raster_mosaic.py includes these fallback files alongside the bulk quads automatically.
.venv/Scripts/python fix_corrupted_quads.py
If a different quad turns out corrupted later (found via spike_raster_mosaic.py's full-band validation pass, which checks every quad), add it to the BAD_QUADS dict in this script.
spike_corridor.py proves the core Phase 1 operation: buffer the full AT centerline (3,025 segments, GA to Maine) by 30 miles, union it into one corridor polygon (~81,138 sq mi), and clip real ATC POI data (campsites, shelters) against it.
.venv/Scripts/python spike_corridor.py
Output goes to data/spike/ (corridor.geojson, campsites_clipped.geojson, shelters_clipped.geojson).
Gotcha hit and fixed - watch for this in any future ST_Transform call: EPSG:4326's authority-defined axis order is (lat, lon), but every geometry source we actually use (GeoJSON, GeoPandas, etc.) is (lon, lat). Without always_xy := true, ST_Transform silently swaps the axes instead of erroring - the buffer/union still "succeeds" but produces geometry transformed as if every point were on the wrong side of the globe, which only surfaced as ST_Area returning nan on the reprojected-back result. Always pass always_xy := true on both the forward and inverse transform.
spike_raster_mosaic.py mosaics the real 1,654 downloaded US Topo quads into the actual corridor-clipped background raster - the full-scale version of the corridor-clip method proven above.
.venv/Scripts/python spike_raster_mosaic.py
Real complications this had to handle that a single-tile test wouldn't:
- Each quad is in its own native UTM zone (they vary by longitude across the trail), so they can't be merged directly - each is lazily reprojected to EPSG:4326 via a
WarpedVRTbefore merging. - One mosaic sized to the corridor's full bounding rectangle would be enormous even downsampled, since the actual corridor is a thin ~60-mile-wide winding band, not a filled rectangle (GA to Maine as a rectangle is most empty space). So this processes in small geographic cells (matching the corridor-intersecting grid used elsewhere in the pipeline) and outputs one clipped tile per cell - the same reason real map tile systems don't ship one giant image.
- Corrupted source quads (see above) are validated with a full-band read (a corner-pixel read isn't reliable - it missed 2 of the 3 known-bad quads) and skipped/substituted rather than crashing the run.
- Every corridor-intersecting cell must produce a tile - a hard completeness check at the end (matching the pattern used everywhere else in this pipeline) fails the run if any cell has no output, rather than silently reporting a partial result as if it were complete.
Output: 51 tiles in data/processed/topo_background/, ~9GB total, at a fixed ~11m/pixel resolution (plenty for a phone background map at this stage - full production tiling/zoom-level generation is a separate later step, see TECHNICAL_ARCHITECTURE.md's "Export" pipeline step).
See ../ROADMAP.md Phase 1 for what's still open - notably the unified POI schema (joining ATC + opentrail.org + NHD into one schema) and the real Export/Publish steps (PMTiles packaging, change-aware release so hikers don't re-download data that hasn't changed).