OurHike data pipeline

Companion to ../TECHNICAL_ARCHITECTURE.md. This is the Python + DuckDB (spatial) pipeline that turns upstream ATC/USGS/opentrail.org data into the packages the client app downloads.

Setup

python -m venv .venv
.venv/Scripts/pip install duckdb requests rasterio numpy   # Windows
# .venv/bin/pip install duckdb requests rasterio numpy     # macOS/Linux

Discovering source URLs

sources.json is the registry of every upstream ATC layer we pull from, and it's generated by discover_sources.py, not hand-written. The ATC doesn't publish a plain list of data downloads - their public map is an ArcGIS Experience Builder app - so this script walks the same chain a browser resolves at load time to find the real FeatureServer URLs underneath it:

Experience app item -> app config -> dataSources (WEB_MAP entries)
-> web map item -> web map data -> operationalLayers[].url

.venv/Scripts/python discover_sources.py "https://experience.arcgis.com/experience/<ITEM_ID>"

Re-run it whenever ATC's map might have changed (new layers, moved services). It's additive/safe by design:

• Existing keys get their url/title refreshed in place (and it prints a note if a url actually changed).
• Hand-added fields like notes on an entry are preserved across re-runs.
• A source that disappears from the app is kept, not deleted, with a warning - that usually means the app changed, not that the data is gone, and deleting registry entries automatically is too destructive for something this script can't fully verify.

Not everything in sources.json came from the Experience Builder app, though - bridges, privies, and at_treadway were found by listing the ANST_Facilities FeatureServer's root directly (it has more layers than the public map curates), and added by hand. discover_sources.py won't touch those unless they later show up in the app's own layer list too.

To point the script at a different Experience Builder app (e.g. if the ATC publishes a second public map, or another club/provider uses one), just pass that app's URL and a --provider label - nothing else in the script is ATC-specific.

That --provider label is as far as this registry currently goes toward being multi-organization: every entry says "ATC", and adding a fourteenth source is still a code change by whoever owns this repository. The thirteenth arrived on 2026-08-12 and is the first that is not an ArcGIS layer — ATC's Trail Updates, read from their website (../features/ATC_TRAIL_UPDATES.md). It carries kind, trust, steward, licence and freshness, so several of the fields below now exist on real data rather than only in the design; lib/source_registry.py is what reads kind, and fetch_all.py fetches only the entries that are feature layers. It is hand-written rather than discovered, which works because discover_sources.py keeps entries it did not rediscover — and now keeps hand-added fields on the ones it did. ../features/SOURCE_REGISTRY.md is the design for letting an outside organization register its own layers and a contact to notify - including the fields this file would gain (steward, kind, licence, field_map, freshness, trust, state) and why the registry stays a reviewed file in git rather than becoming a database table.

What exists upstream beyond the registry - the maintaining clubs, the federal servers, the community datasets, and what each is worth - is surveyed and qualified in SOURCE_SURVEY.md (snapshot dated 2026-08-09; it also corrects who actually hosts the layers above). The water-source question that survey left open - measured against the shelters for #529 - 97% of shelters have no water source within 250 m, and the trail is not like that - has its own dated snapshot in WATER_SOURCES.md, with the three measurement spikes (spike_shelter_water.py, spike_osm_water_census.py, spike_guide_water_check.py) beside it.

Where a script's output goes

Three shelves, and picking the wrong one is the one mistake that cannot be undone — see ../CONTRIBUTING.md's "Data does not go in commits" for why a commit is a permanent publication of somebody else's data.

shelf	for	tracked?
data/raw/	anything fetched or derived — layers, extracts, derivations, caches	no, gitignored and cached between CI runs
data/processed/	export output, the artifacts publish.py uploads to R2	no
reference/	a join that encodes judgement somebody reviews row by row (shelter_capacity.json, water_distance.json)	yes, and under a line ceiling

.github/tests/test_no_committed_data.py enforces it: a tracked data/ path fails, a data-shaped file outside a stated allowlist fails, and a reference file past ~8,000 lines fails because "committed so the judgement in it can be reviewed" stops being true once nobody reads the rows.

Fetching ATC sources

fetch_all.py reads sources.json and downloads each layer (paginating past ArcGIS's per-request record cap via lib/arcgis.py) to data/raw/<key>.geojson. It's change-aware: before doing the full paginated fetch, it checks each layer's editingInfo.dataLastEditDate (one cheap metadata request) against the value recorded in data/raw/manifest.json from the last run, and skips the source entirely if unchanged. Only writes the manifest if every registered source succeeded or was confirmed up to date - if any layer fails or goes missing, it exits non-zero instead of silently producing a partial dataset.

.venv/Scripts/python fetch_all.py

Current ATC sources (as of 2026-07-25)

key	title	count	geometry	notes
trail_club_sections	A.T. Trail Club Sections	30	Polygon	One polygon per maintaining club's section (incl. NYNJTC)
centerline	A.T. Centerline	3,025	Line	The trail itself, segmented; Trail_Club/Acronym fields identify maintainer per segment
side_trails	A.T. Side Trails	1,200	Line	Blue-blazed and other connector trails
campsites	A.T. Campsites	232	Point	Detailed facility attributes (tent pads, food storage, etc.)
shelters	A.T. Shelters	280	Point	Detailed facility/construction attributes - but no capacity, see below
parking	A.T. Parking	482	Point	Trailhead parking areas
viewpoints	A.T. Viewpoints	1,223	Point	Scenic overlooks
communities	A.T. Communities	59	Point	Designated "A.T. Community" towns - partial resupply proxy
half_mile_points_from_springer	A.T. Half Mile Points From Springer	4,395	Point	Mile-marker points along the trail
bridges	A.T. Bridges	409	Point	Found via the FeatureServer root, not the public map
privies	A.T. Privies	316	Point	Found via the FeatureServer root, not the public map
at_treadway	A.T. Treadway	30	?	Found via the FeatureServer root - not yet checked how this differs from centerline

Which of these reach a hiker as waypoints: shelters, campsites, viewpoints, parking and privies each become one poi_type in export_poi.py, and communities folds into resupply at low confidence. The other six are fetched for other reasons — centerline and side_trails are the trail lines, half_mile_points_from_springer the mile markers — and bridges and at_treadway are registered but feed nothing yet. trail_club_sections was in that group until 2026-08-13, when export_club_sections.py (#594) started reading it; it supplies club names and regions, while the club attribution comes off centerline's own Acronym field, which is two years fresher and sits on the trail line (SOURCE_SURVEY.md §3e). Vistas, parking and privies were in that second group until 2026-08-09: registered on 2026-07-25 and downloaded by every run since, with nothing downstream reading them.

Gap, now partially filled: ATC's own data has no dedicated water-source or general resupply layer. communities is a partial resupply proxy; fetch_opentrail.py (below) is the real fill for resupply, and for water it is one of two — fetch_osm_water.py (below, #529) took the corridor's water layer from 174 points to 1,705.

A second gap, and no ATC source fills it: nothing says how many people a shelter sleeps. Searched rather than assumed (2026-08-09) — all twelve registered sources above, ANST_Facilities' three unregistered asset tables (a maintenance inventory in EA/LF/SF; its Sleeping Platform rows are 6 across the whole trail, all in square feet), the shelter layer's free text on all 280 features (every "sleeping" mention is a dimension, not a person count), and the sibling A.T. services in the same NPS org. The shelter layer's own 135 fields are an FMSS asset inventory, and FMSS_QTY is floor area, not people: 15.6 × 15.6 = 243.36 exactly. build_shelter_capacity.py (below) is the fill.

Shelter capacity

build_shelter_capacity.py writes reference/shelter_capacity.json: how many people each A.T. shelter sleeps, keyed to ATC's own GlobalIDs. export_poi.py reads that file and publishes capacity on shelter features, where the client's waypoint card renders it as "Sleeps 8".

.venv/Scripts/python build_shelter_capacity.py            # rebuild and review the diff
.venv/Scripts/python build_shelter_capacity.py --check    # confirm the checked-in file still matches

The output is checked in, not fetched at build time, which is the opposite of every other source here and deliberate. The join is by name between two lists that disagree about them - ATC's "Doc's Knob Shelter" against the source's "Docs Knob Shelter", ATC's "Winturri" against its "Wintturi", and ATC's "Rocky Run Shelter 1"/"2" against a single "Rocky Run Shelters" row. A fuzzy join running unsupervised inside a data build is a join nobody ever reads; a checked-in file makes each of those a reviewable line in a diff, and keeps a release build off the network for it.

262 of 280 shelters resolve; the other 18 publish nothing, on purpose. Each carries a stated reason in the file - a pair listed under one number that could be each or the total, an old and a new structure with different numbers, a capacity written "xxx" or "A lot". Capacity is a number a hiker plans an evening around, so a blank beats an invention, and the card omits the line rather than showing a zero.

One independent check exists, and it passes. GRSM_BACKCOUNTRY_SHELTERS, a park layer in the same NPS org, is the one A.T.-adjacent source with a real capacity field. It covers 15 shelters, 12 of them in ATC's — a twentieth of the trail, so not worth a second join, a second provenance and a precedence rule. It is worth comparing against, and it agrees with this file on all 12, exactly.

Licensing is unconfirmed, and worth saying plainly. The capacity numbers come from greenbelly.co's A.T. shelter list, which credits Whiteblaze, the Appalachian Trail Conservancy and TNlandforums, and which states no licence at all - the same position as opentrail.org above (#98), recorded here rather than discovered later. Two things narrow what is taken: only the capacity column, not the mileages or elevations or ordering, and it is re-keyed onto ATC GlobalIDs, so what ships is a set of facts about shelters this project already knows about rather than a copy of somebody's table. That is a better position than a scrape, not a settled one. Confirming terms with Greenbelly is the honest next step, and until then this carries the same caveat opentrail.org does.

Water distance at shelters and campsites

build_water_distance.py writes reference/water_distance.json: how far the nearest water source is from each A.T. shelter and campsite, keyed to ATC's own GlobalIDs — the same checked-in-and-reviewed shape as shelter capacity, for the same reason (the join encodes judgement calls a diff should show). export_poi.py reads it, publishes water_distance_ft on shelter and campsite features, and names water among the anchor's nearby parts where no actual water point folded into the site and the distance is within the site vocabulary's 150 m — in ATC's own feet, which the card then writes in whichever units the hiker chose. Wherever that entry fires, the export also synthesizes a water POI onto the site (#694) — a source: "atc_csi" member at the anchor's own inherited coordinates, since ATC states how far and never where — so the pin's strip and the card's chips show the water the sentence promises; its description says whose measurement it is and that the spot is unmapped, leaving the figure to the chip beside it, and a real mapped water point folding in stops the synthesis for that site.

.venv/Scripts/python build_water_distance.py            # rebuild and review the diff
.venv/Scripts/python build_water_distance.py --check    # confirm the checked-in file still matches

The distances are ATC's own, from the Campsite_Sustainability_Index layer on their ArcGIS org — official sites only; the layer's 2,333 user-created campsites are never even requested (SOURCE_SURVEY.md §3b says why their locations must not ship). 305 of 512 features publish a distance. The FarOut-measured rows (218 of those 305) first shipped held back — WATER_SOURCES.md §4 found 42% of CSI's distances derive from that commercial dataset — and were released on the maintainer's 2026-08-13 authorisation that data ATC publishes is reusable, recorded as the atc_licence block in sources.json beside photo_licence and in its shape (#688); ATC's own written answer stays the ideal (§10's combined ask). The provenance gate outlived the holdback: a Nearest_Water_Source value ATC introduces later publishes nothing until a human adds it to PUBLISHABLE_PROVENANCES deliberately. Every one of the 512 features is listed in the file either way — a blank always carries its reason (no CSI row within 150 m, a 0 ft value nobody can read, or an unknown provenance).

Water: OSM points and the stream sentence (#529)

The two halves of WATER_SOURCES.md §7's recommendation, built together and honest in different shapes — a pin where a point is true, a sentence where only proximity is:

fetch_osm_water.py scans the fourteen Geofabrik state extracts export_basemap.py already downloads for OSM's water point sources — natural=spring, amenity=drinking_water, man_made=water_tap, man_made=water_well, the census's exact clause set — 7,574 nodes on the first full scan (2026-08-13). export_poi.py folds them into poi_type water at low confidence (a mapped spring is one contributor's observation, which is what the dashed rim and the card's "Unverified" sentence say), drops each point within 25 m of an opentrail water point as the same OSM node arriving twice (measured before choosing: 41 of opentrail's 174 water points have an OSM twin inside that radius, and the tail past it is real neighbours), and composes each point's sentence from its own tags — "Spring, mapped as intermittent." — never strengthening an absent tag into a claim. ODbL; the client's credits screen already names OpenStreetMap, and each description carries "Mapped by OpenStreetMap contributors".

.venv/Scripts/python fetch_osm_water.py             # ~3.5 GB of extracts on a cold machine; skip-if-present
.venv/Scripts/python fetch_osm_water.py --refetch   # force current extracts

fetch_trail_water.py writes data/raw/trail_water.json — gitignored, cached between CI runs, and published to R2 like every other fetched layer, because 20,000 lines of derived coordinates are data and data does not go in commits: where the trail meets water, and which sites have water they can actually walk to. Two products from one derivation over both hydrographies — USGS's NHD and OpenStreetMap — merged rather than picked between, because they know different things: USGS classifies flow (perennial / intermittent / ephemeral), OSM more often carries the local name and is edited by people who walk there. A crossing deduped across the two keeps whichever half each supplied, records both in sources, and attributes the flow claim to whoever made it (flow_source) — features/POI_DEDUPLICATION.md's combine-don't-drop rule. USGS arrives as bulk staged GeoPackages, one subregion at a time, downloaded read and deleted: its query service 504s under corridor-scale load, and a derivation nobody can re-run is not one anybody can check. OSM costs no network at all — it is the same Geofabrik extracts the basemap build already downloads:

• Crossings — exact geometric intersections of ATC's centerline with the stream lines of both hydrographies. The two lines cross, so a hiker walking the trail walks through the water. These fill crossing, the poi_type declared in lib/poi_schema.py and empty since it was declared. 1,125 crossings land in the corridor, 571 of them corroborated by both databases, and 39 of 512 shelters and campsites get water they can walk to.

• Site water — for each shelter and campsite, the nearest point on a stream, published only where a hiker could reach it: within 100 ft and under a 15% grade, the second gate measured from real USGS 3DEP elevations at both ends. A stream 90 ft away and 120 ft below is not a water source however close the map says it is. The radius is deliberately tight: most A.T. shelters have had their own spring built out over decades, so the water a shelter uses is usually a piped source somebody dug rather than the nearest blue line, and ATC's own measured distance (build_water_distance.py) stays the better answer there — this fills in a real coordinate only where geometry can honestly supply one. Every rejected candidate keeps its distance, drop and grade in the file, so either gate can be re-argued from the numbers rather than re-run in the dark.

The match radius sits inside lib/poi_sites.py's 60 m proximity fold on purpose: a published point at real coordinates is folded onto the shelter's pin by the grouping that already exists, so there is no second matching rule to keep in step. Nothing composed here carries a distance — the point has coordinates, so the card measures the walk and writes it in the hiker's own units (#625).

.venv/Scripts/python fetch_trail_water.py    # 14 OSM extracts, then 21 USGS subregions one at a time

Is any of it the water a guidebook lists? (#97)

spike_guide_water_check.py is the cross-reference #97 — Validate NHD flowline stream-crossings as a water-source candidate list asks for as its second validation step: measure real overlap versus gap against a source somebody trusts, rather than comparing totals. Run 2026-08-14 against the maintainer's own copy of The A.T. Guide (980 water-tagged mile-table rows), with both put on ATC's half-mile points as a shared ruler and the two mileages aligned by an offset measured from the rows that print coordinates rather than assumed to be zero:

what the guide row describes	rows	our crossing within 0.2 mi	any OurHike water
a stream (creek/brook/river/fork/run)	460	302 (66%)	320 (70%)
other	310	75 (24%)	141 (45%)
a spring	210	20 (10%)	77 (37%)
all water rows	980	397 (41%)	538 (55%)

Two thirds of the guidebook's stream rows have one of our crossings within a fifth of a mile, from hydrography that has never seen the guidebook — which is what a crossing claims. Springs are the structural gap and cannot be otherwise: a spring does not cross the trail. And 60% of our crossings are not guidebook water, which is why they publish as crossing and never as water pins.

The guide is a personal copy and stays one. It is copyright AntiGravityGear, LLC (SOURCE_SURVEY.md §8: context only, never a source), so the parse runs in memory, the results file holds counts and percentages only, and the PDF lives in personal_reference/ — the first line of the repository's .gitignore. Every machine without one gets told the PDF is missing, which is the correct outcome; the method and the numbers are in the script's docstring.

.venv/Scripts/python spike_guide_water_check.py             # needs `pip install pypdf`
GUIDE_PDF=/path/to/at_guide.pdf .venv/Scripts/python spike_guide_water_check.py

Fetching club PDFs (review-only)

fetch_club_pdfs.py downloads the PDFs the maintaining clubs publish, as sources.json registers them (kind: "club_pdf" — #669), and parses the ones lib/club_pdfs.py has a parser for into structured rows beside the PDF under data/raw/club_pdfs/. First registrant: GATC's water-sources PDF — 65 rows of mile point + entry text covering the approach trail and all of Georgia, reliability notes included ("Typically very low or dry. Use creek at MP 2.9").

.venv/Scripts/python fetch_club_pdfs.py     # needs `pip install pypdf` - deliberately unpinned, requirements.in explains

Change-aware per entry (conditional GET against its own manifest, plus a body-hash check because WordPress does not always honour conditionals), strict on parse (a PDF whose layout changed stops the run and keeps the previous known-good state — build_shelter_capacity.py's posture applied to a fetch), and review-only by construction: no export reads data/raw/club_pdfs/. Club PDFs state no terms (SOURCE_SURVEY.md §9), so each registry entry's licence field records the ask that has to be answered before anything here reaches a hiker; WATER_SOURCES.md §4 sizes GATC's as "a pilot-state candidate after an email". The next club document is one sources.json entry and, optionally, one parser — not a new script.

POI descriptions

Every POI from one of ATC's own facility layers carries a description — one sentence the waypoint card shows under the name:

Two-storey clapboard shelter, sleeps 14, with a fireplace, a fire ring and a porch. Built 1915.

A 100° view south-east from a ridge or rock outcrop.

Gravel parking area, room for 12 cars.

Multi-seat moldering privy. Built 2019.

It is composed, not copied, because ATC has no prose description. Both text fields were read in full (2026-08-09):

field	what it actually holds
Descriptio, aliased "Description"	The club acronym followed by the feature's own name — "MATC Chairback Gap Lean-to Shelter" — on 488 of the 510 features that have it. The rest are spelling variants of the same thing, or literally "NA". Published, it would render directly under a heading already saying the name.
Comments	The real free text, and a surveyor's notebook: populated on 81 of 280 shelters and 65 of 232 campsites, ranging from useful ("Has a loft", "Not an accessible shelter") through construction detail ("Shiplap siding") to notes the survey wrote to itself — "Not sure about spatial info" on twenty-four campsites, "GIS CS629-CS635", "Added based on existing GIS data".

What ATC does have is the inventory, and it is complete: Stories, Chimneys, the fire-ring and food-storage counts, Deck_Lengt, Exterior_M and Year_Built are non-null on all 280 shelters, and Site_Num on 231 of 232 campsites. lib/poi_description.py assembles the sentence from those, so every clause is a fact ATC states and coverage is 280/280 shelters and 232/232 campsites rather than the 26% the free text manages. Which columns are worth a clause is one list, FEATURES, so disagreeing with the selection is a one-line change — the line drawn is what changes a hiker's decision (food storage, a fire, a porch), which is why the window and skylight counts are left out.

Where ATC did write a usable comment it is appended as "ATC notes: …" — attributed rather than blended in, because that half is a person's prose and the rest is assembled from columns. lib/atc_notes.py is what decides "usable": it drops the survey's own bookkeeping sentence by sentence, never rewording, so Cable Gap's "Log and mortar exterior. Majority of structure is log. Please see photos." keeps its first two sentences instead of being thrown away whole. 74 shelters and 29 campsites end up with a note.

The other three layers (2026-08-09)

Vistas, parking areas and privies compose the same way, and coverage lands at 1,194/1,223 vistas, 480/482 parking areas and 314/316 privies. Three decisions in them are worth knowing:

• A vista's direction is derived, not copied. Left_Beari and Right_Bear bound the view swept clockwise and are populated on 1,006 of 1,223; Scope claims to be that width and disagrees with the bearings on 93 of the 512 features carrying it, so the arc is computed and Scope is never published. The width is rounded to 5° because ATC's own field notes say the instrument wandered — "measured bearings 3 times, each time getting different results" — and a 62° view claims a precision that measurement does not support. Beyond 300° the sentence says "panoramic" instead of naming one edge of a view you can turn round in.
• Nothing maps a free-text value onto a code. viewpoints.Type is a code on 988 of 1,223 and free text (Unimproved, Improved) on the rest; parking.Type on 417 of 482, with Roadside/Shoulder on 53 more. Roadside/Shoulder is recognised on its own terms, because a shoulder is a different thing to arrive at than a lot. Every other unrecognised value drops its clause, so the sentence is shorter rather than wrong — deciding Unimproved "means" code 0 would be guessing at somebody's data entry.
• The vista layer brought its own bookkeeping dialect, and lib/atc_notes.py grew patterns for it: Improvements = none identified (a form saying nothing, and the most common comment on the layer), the 2021 VRI review, Preliminary Review with VARO, survey point ids like VP1058, and the surveyor's trouble with a compass. Measured against the shelter and campsite layers, those patterns change nothing there — the same 74 and 29 notes survive.

Type on a vista (Improved/Unimproved) is deliberately not in the sentence: it says whether ATC has built decking or railings at the spot, which is a maintenance distinction rather than a hiker's. Year_Built is out for vistas too — on a viewpoint it is ambiguous between when a structure went up and when the view was cleared — and out for parking, where the age of a car park changes nothing anyone does. It stays on privies, where 308 of 316 carry one and a rebuild three years ago is a different proposition from 1965.

Fetching opentrail.org (water sources + resupply)

fetch_opentrail.py pulls AT waypoints from opentrail.org's public API (/api/getData?trail=AT) - 1,840 features, of which 142 are tagged water sources (w), 72 resupply (r), and 103 towns (t), the gap ATC's own data leaves. Licensing isn't formally confirmed (no LICENSE file in their repo; the maintainer reportedly called it "open data" in a Reddit post - #98 tracks following up directly), so this deliberately excludes their user comments (personal contributions from named individuals - a separate consent question from licensing).

.venv/Scripts/python fetch_opentrail.py

Change-aware via real HTTP conditional requests (the API documents ETag/If-None-Match support) - a 304 response means skip, no re-parsing or re-saving.

Fetching POI photos from Wikimedia Commons

fetch_poi_images.py matches openly-licensed photos to corridor POIs for the waypoint card's photo slot (design and sourcing decisions in features/POI_PHOTOS.md). Per POI it geosearches Commons' File namespace around the coordinates (per-type radius: 300m shelters/campsites, 120m water, 500m resupply towns), then keeps only files that are JPEGs with an EXIF capture date inside the last four years and a licence OurHike can ship under - public domain, CC0, or CC BY / CC BY-SA at 4.0+, with an author to credit wherever the licence requires one (lib/commons.py holds the rules, including why pre-4.0 CC versions are rejected). Nearest eligible file wins. Three published types are deliberately not crawled here at all — viewpoint, parking and privy have no entry in SEARCH_RADIUS_M, which is what makes "not searched" a decision rather than an oversight: proximity is measurably the wrong matcher for a facility (0 usable photos for 280 shelters), ATC's own inventory covers exactly those three at 37%/58%/95%, and fetch_atc_photos.py wins any overlap anyway. Licensing is per photo, not per source, so each photo's licence, author, file-page URL and capture date are recorded in data/raw/poi_images.json and ride the exported features as photo_* properties (see export_poi.py), where the client renders them as the card's credit line.

Every photo, not just the first (#471). ATC's layers carry Photo1..Photo10 and 433 of the 489 features with a photo use more than one, so fetch_atc_photos.py keeps them all in ATC's own order - Photo1 is their judgement about which best shows the facility, so it stays the card photo. The export publishes the first through the flat photo_* fields, as it always has, plus the whole list as a photos property. Note the two artifact formats disagree about that field's type: the pipeline writes one JSON string (FlatGeobuf property values are scalars, so a nested array cannot be a column), and GDAL then emits it as real JSON in the .geojson while the .fgb keeps the string. The client accepts both.

The image bytes are ours, not a hotlink (#362). The chosen 640px rendering is downloaded into data/raw/poi_photos/, named by the sha256 of its own bytes, and publish.py uploads it to photos/<digest>.jpg - so a waypoint card never depends on upload.wikimedia.org being reachable, and we stop spending a nonprofit's bandwidth on our traffic. Content-addressing is what makes that safe to re-run: identical images shared by two waypoints are one object, an already-uploaded photo is skipped, and a key never needs renaming (which R2_LAYOUT.md cannot do anyway). The exported feature carries photo_key, the bucket key - never a URL, since the host is the client's own build-time base.

.venv/Scripts/python fetch_poi_images.py            # only POIs without a recorded outcome
.venv/Scripts/python fetch_poi_images.py --recheck  # re-query everything (new uploads, deleted files)

Run after fetch_all.py and fetch_opentrail.py (it derives the POI list by calling export_poi.py's own unify + corridor clip, so ids match the export exactly), before export_poi.py. Change-aware per POI: every outcome - found, with the photo record, or a recorded miss - is kept with its check date, so the first pass is thousands of throttled sequential requests (tens of minutes; Wikimedia-required User-Agent, maxlag=5 waited out politely, 429/5xx retried honoring Retry-After, progress flushed atomically every 200 queried POIs so an aborted crawl resumes from its last flush) and every later pass only queries new POIs plus found photos that have aged past the freshness window. A run that would wipe a suspicious share of still-fresh photos refuses to persist, same posture as fetch_opentrail.py's drop guard. Coverage is expected to be partial and honest - most water sources have no Commons photo at all; the card's category-glyph placeholder is the designed fallback. Measured over all 817 corridor POIs on 2026-08-08: 76 photos (9.3%), but 0 of 280 shelters and 0 of 232 campsites got one that plausibly depicts them, and 35 of the 76 are iNaturalist species observations - the numbers, the reasons, and what they mean for the source live in features/POI_PHOTOS.md. Not yet wired into publish-vector-data.yml: doing so couples every data release to Commons availability, a decision deliberately left open in POI_PHOTOS.md.

Fetching USGS US Topo background quads

fetch_topo_quads.py pulls the raster quads used for the background map. Rather than the TNM Access API (flaky pagination, multiple-editions-per-quad problems - see the script's docstring for the full story of what didn't work), it uses USGS's own metadata inventory (ustopo_current.csv) to find exactly which quads intersect the 30-mile corridor, then matches each to its real GeoTIFF file by listing each state's S3 folder directly (the CSV's own filename field is unreliable for constructing the URL).

.venv/Scripts/python fetch_topo_quads.py

Scope: 1,654 quads, ~14GB (vs. 300-500GB for a naive full-state pull across all 14 AT states) - corridor-scoping is what keeps this download/hosting-sized reasonably (value #8). Change-aware per-quad via S3 Last-Modified against data/raw/topo_quads/manifest.json.

Known data-quality issue: 3 of the 1,654 quads (NC_Glade_Valley, VA_Marion, WV_Princeton) are genuinely corrupted on USGS's own S3 bucket - confirmed via two independent codebases (rasterio/GDAL and tifffile/imagecodecs both fail to decode the same strip on a byte-exact-verified fresh download), not a truncated download on our end. fetch_topo_quads.py itself only checks HTTP presence/Last-Modified, not actual readability, so this kind of corruption goes undetected until something tries to read the file - run fix_corrupted_quads.py after any fresh fetch_topo_quads.py run to catch and work around this (see below).

Still an open item for this whole-corridor path specifically: add a lightweight read-check to fetch_topo_quads.py itself so this isn't a separate manual step. The per-cell CI path (fetch_and_mosaic_cell.py, see further down) already does this - it calls fix_corrupted_quads.py's recovery logic (fix_quad()) inline the moment a quad fails validation, rather than requiring a second command. That couldn't be backported here without fetch_topo_quads.py importing from fix_corrupted_quads.py, which already imports from it (bare_key) - a circular import. Untangling that (probably by moving bare_key somewhere both can import from) is real but small future work, not done as part of adding the per-cell path.

Fixing corrupted quads

fix_corrupted_quads.py re-downloads each known-bad quad once (in case it was a transient issue), and if still corrupted, fetches a substitute covering the same footprint from basemap.nationalmap.gov's live export service (the same one used in the raster spike below), saved to data/raw/topo_quads_fallback/. spike_raster_mosaic.py includes these fallback files alongside the bulk quads automatically.

.venv/Scripts/python fix_corrupted_quads.py

If a different quad turns out corrupted later (found via spike_raster_mosaic.py's full-band validation pass, which checks every quad), add it to the BAD_QUADS dict in this script.

Spatial spike: corridor computation (done)

spike_corridor.py proves the core Phase 1 operation: buffer the full AT centerline (3,025 segments, GA to Maine) by 30 miles, union it into one corridor polygon (~81,138 sq mi), and clip real ATC POI data (campsites, shelters) against it.

.venv/Scripts/python spike_corridor.py

Output goes to data/spike/ (corridor.geojson, campsites_clipped.geojson, shelters_clipped.geojson).

Gotcha hit and fixed - watch for this in any future ST_Transform call: EPSG:4326's authority-defined axis order is (lat, lon), but every geometry source we actually use (GeoJSON, GeoPandas, etc.) is (lon, lat). Without always_xy := true, ST_Transform silently swaps the axes instead of erroring - the buffer/union still "succeeds" but produces geometry transformed as if every point were on the wrong side of the globe, which only surfaced as ST_Area returning nan on the reprojected-back result. Always pass always_xy := true on both the forward and inverse transform.

Planning spike: can shelters carry a generated plan? (not yet run)

spike_day_planner.py answers the one question in ../features/HIKE_PLANNING.md that no amount of design settles: given where the ATC actually put shelters and campsites, can a plan whose days all end at a real site hit a target day length — and how bad is the worst day when it can't?

.venv/Scripts/python spike_day_planner.py
.venv/Scripts/python spike_day_planner.py --targets 12,15,18 --cap 25

Reads already-fetched ATC data (shelters, campsites, centerline) the same way spike_corridor.py does — no network. It positions every site along the ordered centerline using export_elevation.py's own helpers, so the miles it reports are the same measurement elevation_profile.json uses rather than a third one, then reports the real spacing distribution and runs a shortest-path day planner across a range of targets. If data/processed/elevation_profile.json exists it also plans against a time target rather than a distance one, which is the comparison that says whether planning by Naismith hours is worth the machinery.

It has not been run against real data yet — the environment it was written in has no route to ATC's servers. Everything in HIKE_PLANNING.md's Finding 3 is arithmetic over the feature counts in the source table above, not a measurement, and closing that gap is the first thing to do with this script. The planner in it is deliberately throwaway: the real one runs on the phone, and what should survive is the shape rather than the code.

Duplicate spike: how many POIs are the same place twice? (done)

spike_poi_duplicates.py is the measurement behind ../features/POI_DEDUPLICATION.md and #696 — Nothing stops two sources publishing the same place twice, and the one rule that does is a 25 m constant for a single source pair.

.venv/Scripts/python spike_poi_duplicates.py
.venv/Scripts/python spike_poi_duplicates.py --refetch

Unlike the two spikes above it needs no prior fetch_all.py run — it pulls the six ATC POI layers and opentrail.org into data/spike/poi_duplicates/ and re-reads that cache afterwards, so a re-measurement costs upstream nothing. It measures the published set rather than a second version of it, by pointing export_poi.py's own RAW_DIR at the cache and calling that module's own unify_all_sources.

Measured 2026-08-13, over all 2,837 unified points: 48 same-type pairs sit within 25 m of each other, every one of them inside a single source, and 35 of the 48 are two real places — ATC distinguishes them by a trailing sibling number, a direction token, or an outright different name. The real duplicates are 11 places holding 23 records (0.42% of the map), almost all of them the viewpoint layer carrying one overlook twice, once with a trailing "Vista". And the one cross-source overlap already shipping — ATC's Communities against opentrail's resupply points — has no pair within 1 km of each other, which is what says a radius alone cannot be the definition of a duplicate.

Raster background: mosaic + clip at real scale (done)

spike_raster_mosaic.py mosaics the real 1,654 downloaded US Topo quads into the actual corridor-clipped background raster - the full-scale version of the corridor-clip method proven above.

.venv/Scripts/python spike_raster_mosaic.py

Real complications this had to handle that a single-tile test wouldn't:

• Each quad is in its own native UTM zone (they vary by longitude across the trail), so they can't be merged directly - each is lazily reprojected to EPSG:4326 via a WarpedVRT before merging.
• One mosaic sized to the corridor's full bounding rectangle would be enormous even downsampled, since the actual corridor is a thin ~60-mile-wide winding band, not a filled rectangle (GA to Maine as a rectangle is most empty space). So this processes in small geographic cells (matching the corridor-intersecting grid used elsewhere in the pipeline) and outputs one clipped tile per cell - the same reason real map tile systems don't ship one giant image.
• Corrupted source quads (see above) are validated with a full-band read (a corner-pixel read isn't reliable - it missed 2 of the 3 known-bad quads) and skipped/substituted rather than crashing the run.
• Every corridor-intersecting cell must produce a tile - a hard completeness check at the end (matching the pattern used everywhere else in this pipeline) fails the run if any cell has no output, rather than silently reporting a partial result as if it were complete.
• Every US Topo GeoTIFF is a scan of the entire printed map sheet, not just the map - a white margin plus a header/footer collar (USGS/US Topo logos, title, scale bar, legend, adjoining-quadrangle diagram), and its georeferenced raster extent covers that whole sheet. Left uncropped, the collar gets treated as real terrain - it showed up as white bands and text baked into the exported background. Fixed by cropping every quad to its real neatline (the actual mapped area) before reprojecting, using the westbc/eastbc/northbc/southbc columns already present in USGS's own metadata CSV (ustopo_current.csv, fetched by fetch_topo_quads.py) - a clean 7.5'x7.5' box per quad, confirmed noticeably smaller than the raw raster's full extent (e.g. CT_Ansonia: a 0.125x0.125deg neatline vs a ~0.17x0.16deg raster).

Output: 51 tiles in data/processed/topo_background/, ~3.7GB total (DEFLATE-compressed, 512px-tiled), at a fixed ~11m/pixel resolution (plenty for a phone background map at this stage).

An earlier version of this line said ~9.5GB, which was wrong twice over: the tiles were written uncompressed at 14.59GB, and du -sh reported 9.5G only because it counts allocated blocks on the dev machine's volume. A Linux CI runner materialises the real figure. Compression was added in mosaic_one_cell() for exactly that reason - see the note there, including why predictor=2 is deliberately not used.

Per-cell fetch + mosaic (for CI, not local use)

fetch_and_mosaic_cell.py fetches and mosaics exactly one corridor cell - the unit of work a GitHub Actions matrix job runs so this raster pipeline can execute on a hosted runner's disk, which can't hold the whole corridor's ~14GB raw + ~3.7GB processed data at once. Peak disk for a single cell is a few hundred MB. It's built entirely from the same functions the two whole-corridor scripts above use internally (fetch_quads_for_cell, resolve_state_index, index_quads_in_dir, mosaic_one_cell), plus fix_corrupted_quads.py's fix_quad() for inline corruption recovery - unlike the whole-corridor path, a corrupted quad here is redownloaded-then-substituted in the same job, not flagged for a separate manual step.

Needs cells.json first - the corridor's cell grid plus each cell's quad list (lib/corridor_grid.py), computed once from small vector data, no rasters touched:

.venv/Scripts/python build_cells_manifest.py
.venv/Scripts/python fetch_and_mosaic_cell.py --cell-index 0

Does not replace the whole-corridor workflow above - fetch_topo_quads.py/spike_raster_mosaic.py are still how a maintainer runs the full pipeline locally. The two paths never share fetched files: the per-cell path downloads into data/raw/topo_quads_cell_NNN/, not data/raw/topo_quads/<state>/, so a local whole-corridor run and a local per-cell run can coexist on the same checkout without colliding.

Roughly 22.9% of corridor quads (378 of 1,654) bbox-overlap more than one 1-degree cell, so a quad near a cell boundary gets fetched once per owning cell rather than shared - about 3.6GB of deliberate, bounded redundancy across the whole corridor, accepted rather than adding a cross-job quad cache (which would reintroduce the disk/coordination problem this per-cell split exists to avoid).

The hiking sheet's offline archives (vector basemap + DEM, #184/#185/#186)

The vector-first offline program — design, trade-offs and build numbers in BASEMAP.md, which is their one home. The scripts, indexed:

• export_basemap.py — the periodic Planetiler build (Geofabrik state extracts, osmium pre-clip to the corridor shape, OpenMapTiles schema).
• extract_package.py — cuts a trail's download package from that build; keeps every source tile through z9 so packages carry their own context.
• export_dem.py — the corridor DEM: terrarium tiles fetched, blue channel floored to 0.5 m, lossless WebP, PMTiles.
• check_dem_archive.py — the DEM's publish gate: complete regional coverage, every tile decodes, header and metadata say what they must.
• spike_dem_banding.py — the rendered evidence behind the 0.5 m step.
• spike_package_overlap.py — what two overlapping packages would store twice on one phone (#193), measured against the published archives.

build-basemap.yml and build-dem.yml run the builds and, behind their twice-guarded publish inputs, upload the archives; package-overlap-spike.yml runs the overlap measurement. First published 2026-08-06.

Measured per-zoom, dem.pmtiles as published (z0–13, 0.5 m quantize, 21,758 tiles, none absent — 607,265,661 bytes total):

zoom	tiles	MB
0–10	511	21.2
11	1,139	49.3
12	4,176	138.9
13	15,932	397.6

at_basemap_package.pmtiles as published: 83,818 tiles, 532,459,439 bytes — per-zoom in BASEMAP.md's measured results. Its z13-capped sibling at_basemap_package_z13.pmtiles (21,721 tiles, 182,286,799 bytes) is the hiking sheet's Standard level (#276). With the DEM the sheet is ≈ 790 MB at Standard and ≈ 1.14 GB at Fine (client/src/lib/packages.ts composing lib/hikingDetail.ts, sizes exact to the byte against these artifacts).

Exporting the background as PMTiles (rebuilt from native resolution, #191)

Superseded on 2026-08-06: export_pmtiles.py and its 11 m intermediate are gone. The measured diagnosis in #191 found the old chain shipped ~1/88th of the source's pixels - US Topo GeoTIFFs are 2.032 m/px natively, and the mosaic step downsampled them to 11.13 m/px with bilinear before the tile export resampled them again.

The rebuild renders every tile in ONE warp from the native quads (lib/raster_tiles.py), with the kernel matched to the zoom: average where the zoom decimates (z<=13), cubic near native (z14). It runs as the same 51-cell fan-out (render_cell_tiles.py, one CI job per cell - each renders the z11-14 tiles it owns by tile centre, fetching every quad an owned tile touches so cell borders cannot seam, plus a 24 m overview), and assemble_raster.py reconverges the cells: verifies the ownership receipts tile the corridor exactly, renders z0-10 from the overviews, and writes the tier archives:

tier	zooms	archive
Light	0-11	background_z11.pmtiles
Standard	0-12	background.pmtiles
Fine	0-13	background_z13.pmtiles
Quad sheet	0-14	quad_sheet_z14.pmtiles (z14 within 5 mi of the trail)

The client declares the source tileSize: 256 (the @2x convention), so a 512px tile maps 1:1 to a DPR-2 phone's device pixels instead of being upscaled 2x - the free half of #191's fix, and the reason lib/archiveCoverage.ts carries a camera-vs-tile zoom offset.

Measured sizes from the first successful full-corridor run (run 31100130798, 2026-08-06, all 51 cells, receipts verified), under the same +/-0.6% honesty bar the previous build held (its 11 m-chain figures were 64.4 MB / 314 MB / 1.18 GB, measured 2026-07-29):

tier	tiles	measured
Light background_z11.pmtiles	1,650	68.9 MB
Standard background.pmtiles	5,826	300.3 MB
Fine background_z13.pmtiles	21,758	1,179.2 MB
Quad sheet quad_sheet_z14.pmtiles	31,987	1,698.1 MB

This table records what a RUN produced. It is not what the bucket currently serves, and the two are not the same fact - which is how they drifted apart unnoticed until verify_release.py check 18 compared them (#505). Measured against the published archives on 2026-08-09: Light 65.0 MB, Standard 315.1 MB, Fine 1,184.7 MB - closer to the previous build's figures above than to this run's. So either the bucket is stale or this run never shipped; #505 carries that question. downloadDetail.ts no longer copies from here, and advertises the bucket's figures instead, because what a hiker weighs against their remaining storage is the object they will actually download.

The quad sheet is not yet in the client's detail catalog - whether it ships as a fourth choice there or as its own optional package is #193's shape decision, and its size waits in this table either way.

Checking the POI sources before anything expensive

export_poi.py --check reads and clips every POI source, gates on the per-type counts, and writes nothing. The publish workflow runs it before the photo fetches, and it costs seconds.

.venv/Scripts/python export_poi.py --check

It exists because the ordering below it cannot change: export_poi.py attaches photos, so it has to run after fetches that take the better part of an hour — which meant every defect in the raw data surfaced an hour into a run. That happened twice in one release (2026-08-09): a dead Google Drive photo link, then a parking row ATC left with no geometry. Each cost a full run, and each threw away the photo cache with it.

The check runs the export's own reading code rather than a copy, so a source it passes is a source the export can read. What it cannot speak for is the enrichment, which needs the fetches — so the same completeness gate runs again inside the export proper.

Every fetcher's output is cached by its own restore/save pair now, with if: always() on the save, for the other half of the same problem: actions/cache's bundled post-step only runs on success, so a failure late in the job discarded an hour of downloading that was already on disk and already correct.

That cache used to cover the photos alone, which is why run 31592776758 — dead on a USGS 504 at step 13 — kept its photos and re-fetched all thirteen ArcGIS layers on the retry. A runner is empty every time, so fetch_all.py's change-aware skip had no manifest.json to compare against and every run was a cold run. The cache now carries every fetcher's durable output and its receipt (below), and the two must stay in step: a receipt restored without the file it describes fails the gate for drift when the only thing wrong is a gap in the path list.

Checking output quality before publish (done)

check_output_quality.py is the gate between Export and Publish. It runs after export_trails.py, export_poi.py, export_elevation.py, and the raster assemble (assemble_raster.py) have all produced their output, and before publish.py ships any of it to R2:

.venv/Scripts/python check_output_quality.py

It's check_freshness.py's output-side sibling: check_freshness.py (run before fetching) asks whether anything upstream has changed; this asks whether this run's own output can be trusted, after everything has already run. Five checks, in priority order - see the module's own docstring for the full reasoning behind each, especially the corridor one:

1. Completeness cross-check - re-reads trails_manifest.json, poi/manifest.json, and elevation_manifest.json, re-hashes the artifact file each one points at (catching drift between a manifest and what's actually on disk), and re-checks the same non-zero feature/point-count rule each exporting script already enforces on itself (crossing excepted, same as export_poi.py's own exception) - a second, independent check for the case where a script's own gate has a bug or got bypassed.
2. Corridor cross-check - the one check no single export script can run on itself. Rebuilds the 30-mile corridor twice, independently, from data/raw/centerline.geojson, and requires both a plausible (non-degenerate) result and agreement between the two builds.
3. fetch_topo_quads.py backstop - re-verifies that every quad recorded in data/raw/topo_quads/manifest.json still exists on disk, and that a sample of them still reads as a valid raster, as defense in depth alongside that script's own exit-code gate.
4. Drop-vs-baseline detection - compares this run's counts against data/quality_baseline.json (gitignored, like everything else under data/) and flags a count dropping more than ~10% with no matching check_freshness.py-reported upstream change. Only rewrites the baseline on a fully-passing run.
5. Fetch receipts - the only check here that looks upstream of data/processed/, because the four above cannot ask its question. They verify what the exports derived; none of them can tell whether the input an export derived it from was fetched on this run or left on disk by the last one. Every fetcher ends by writing data/raw/receipts/<name>.json recording when it finished and what its outputs hashed to; this re-hashes them. A week-old input is a legitimate release and a never-fetched one is not — an export reading the file cannot tell those apart, so only a record the fetcher itself left can. Staleness is printed with its age, never failed; an absent receipt for a fetcher this run needed, a corrupt one, or an output that has changed since it was fetched all fail.

Check 5 needs to be told which conditional fetchers a run asked for, since photos and elevation are workflow inputs. fetch_all and fetch_opentrail are always required and need no flag — no export can run without either:

.venv/Scripts/python check_output_quality.py --fetched fetch_atc_photos --fetched fetch_elevation

Note the asymmetry with --optional below, which is deliberate: --optional excuses an artifact that was never built, while --fetched adds a requirement. A missing receipt is never excused, because it is the finding. fetch_poi_images is the one exception the other way — its workflow step carries continue-on-error because Commons is a third party this project has no relationship with, so its receipt is reported with its age and never required.

Check 4 needs to be told what changed upstream, because this gate deliberately never touches the network — standing directly in front of publish.py, it should not be able to fail because an upstream host is down. Pass the sources check_freshness.py reported as STALE:

.venv/Scripts/python check_output_quality.py --changed-source atc --changed-source opentrail

Without them the check is conservative rather than wrong: a real drop that an upstream refresh fully explains still gets flagged, so a legitimate change reads as a problem until someone says otherwise.

Exits non-zero if any check finds a real problem - publish.py shouldn't run after that until the cause is fixed.

Publishing

Publishing artifacts to Cloudflare R2 is intentionally write-disabled by default. Set R2_WRITE_ENABLED=true in the trusted environment that is allowed to publish; otherwise publish.py refuses to upload anything, so developers with only read access to the bucket cannot accidentally write to it.

And it will not publish until it is told where: ../features/DATA_ENVIRONMENTS.md. OURHIKE_DATA_ENV names one of ../RELEASING.md §3's three environments, and every key of the run is scoped by it - production is the bucket root, everything else is environments/<name>/. There is deliberately no default, because the only value a default could take is the one that overwrites what hikers have already downloaded.

OURHIKE_DATA_ENV=ua R2_WRITE_ENABLED=true .venv/Scripts/python publish.py

publish.py is the only thing in this project that writes to the bucket, which is what makes that one variable enough: a run publishing to UA cannot name production's keys rather than merely being expected not to.

Where an artifact goes in the bucket, and what it may be called: R2_LAYOUT.md. A key is a public URL that deployed clients and app-store builds already request, and publish.py's manifest merge is additive-only, so a name that lands wrong cannot be renamed - only joined by a sibling and served alongside the mistake. lib/r2_keys.py checks every key of a run before the first upload; read the layout doc before adding an artifact, not after.

Where this is going, designed 2026-07-31: DATA_RELEASES.md. publish.py today overwrites live keys at the bucket root, which means a publish can land on top of a download already in progress and gives a hiker no way to pin a dataset or be told one changed. The plan replaces that with immutable dated release folders, a daily upstream check that only flags, a weekly incremental build, a verification battery run against the published bytes, and release only via a merged code change. Nothing in that plan is built yet - everything described below is still how publishing works.

To serve data/processed/ locally instead - for testing the client's offline download without publishing anything - use serve_processed.py, which answers byte-range requests and sets the CORS headers a cross-origin bucket needs. See ../client/README.md.

Next steps

Open pipeline work is tracked in issues - notably #99 (the unified POI schema beyond its first slice), #96 (nothing runs the freshness check on a schedule) and #100 (the dbt transform layer). Publish now ships as publish.py; the release design that supersedes it is DATA_RELEASES.md.

Chunking granularity decided 2026-07-28: whole corridor, one package (not per-state/per-section) - see ../ROADMAP.md Phase 2 for why, and the zoom-11/12/13 detail choice this Export step supports.