A small retrieval-augmented assistant that answers rider questions about fare and reduced-fare policies for eighteen California transit agencies, wrapped in a public evaluation framework that measures whether it behaves. The eval harness is the point of this repo; the chatbot exists so the harness has something to evaluate.
Evaluation evidence hub · Live AWS assistant · Evaluation report in the repository
Both public links above lag this repository's HEAD, and that gap is stated
here rather than left for a visitor to discover by asking about an agency
that isn't there (#139, #140). The live assistant is pinned to
corpus_version 35ec70d6359d and serves five agencies — HTA, MST, SBMTD,
SacRT, Yolobus, verified against its own /version endpoint — not the
eighteen this README describes. infra/deploy.sh only promotes a build
backed by a promoted, full, live evaluation (ADR 0023), and the nightly full
run has been red on the cross_agency gate since the corpus's
eighteen-agency expansion (#138), so there is currently no fresher run to
promote and deploy. The evidence hub is further behind still: it serves the
2026-07-12 promoted run (201 cases, cross_agency 3/3) and has not been
republished since, for the same reason — scripts/build_evidence_site.py
refuses to publish evidence past its own freshness budget, so a rebuild with
today's still-July-12 promoted evidence would fail that gate rather than
quietly serve a report that claims to be current. The unpromoted picture —
eighteen agencies, 385 cases, cross_agency currently well below floor — is
in EVALS.md and the nightly run artifact, not on either public
link above.
Deployed and evaluated (use the two public entrypoints above), but not production-grade: the manual accessibility walkthrough is still pending, the judge-calibration sample is 4 scored labels against a floor of 37 (and its κ is undefined, not the 1.000 published until 2026-08-05), and the native-Spanish half of the bilingual equity standard has never been measured at all — 0 of its 28 Spanish answers are rated, and the parity gate that reads 0.0 points cannot see answer quality. All three are tracked in the Standards conformance table below. This is, and is meant to be read as, a reference implementation — see the closing note.
This line used to say the EN/ES answer-quality gap exceeded the project's own
≤5-point target. That was written on 2026-07-05 and is no longer what the
evidence says: the mirrored-case parity gate has read 0.0 points since the
2026-07-12 run, and on 2026-08-05 a mirror-integrity gate established that the
22 pairs it reads that number from really are pairs (three of them were not).
The measured gap is 0.0 points. What remains open is the part that was never
measured — a native-Spanish, not machine-translated, benchmark
(docs/I18N.md §7) — and an unmeasured property is not a passing one. That
half is now defined and scaffolded rather than merely named:
evals/spanish_quality.py publishes the rubric and emits a census of all 28
Spanish answers from the promoted run, committed blank at
evals/spanish/native_es_rubric_2026-08-05.jsonl and rated with
make spanish-quality. EVALS.md carries the state as not measured. The
parity gate compares pass/fail on two answers, and every check behind those
verdicts is satisfied by Spanish of any quality, which is exactly why 0.0
points says nothing about how the Spanish reads.
Requires uv. Snapshots of the corpus are committed, so the offline path works with no API key and no network:
make test # unit tests
uv run python -m evals.runner --offline # full eval, deterministic checks only
uv run python -m assistant.cli --offline "What proof do I need for the veteran fare on MST?"Live model runs and the other backends are covered in Live runs and backends.
Assessed against ChelseaKR/portfolio-standards
(private repository — the link returns 404 unless you have access; the
pinned version is in .standards-version and standards.yml checks staleness on
every push). "Applies" means the standard's AUTO/REVIEW gates are being worked
toward, not that they all pass yet — see the linked gap for current state. The
standard itself is not readable from outside, so treat each row below as this
repo's own claim about its state, evidenced by the files and gates it names,
rather than as something an outside reader can check against the rubric.
| Standard | Applies? | State |
|---|---|---|
| Quality & Metrics | Applies | Partial. Coverage gate (90% branch) is green; DORA ledger and AI-capabilities checklist not yet started. No tracking issue filed yet — this row is the gap record until one is. |
| Code Quality | Applies | Partial. ruff format --check, pytest strict flags, and .python-version landed 2026-07-05; C90 (mccabe, max-complexity = 10, CQ-05) landed 2026-08-21 with a per-file debt ceiling for the 44 functions over budget at the time, retired file by file rather than all at once (issue #137); S (bandit rules) and mypy strict mode are still not on; no pre-commit config; no CODEOWNERS-enforced review. No tracking issue filed yet for the remainder. |
| Security & Supply-Chain | Applies | Partial. SAST (Semgrep) and secret-scan (gitleaks) are blocking; dependency-vulnerability scanning (pip-audit) landed 2026-07-05 (see security.yml). CodeQL (python + actions) and zizmor landed 2026-08-13; zizmor reports no findings at high severity across all nine workflows, which matters because security.yml already cited zizmor findings by name without the tool ever running. No ASVS level declared. Scorecard is deliberately not added: the OpenSSF action publishes to a public API and several of its checks (Branch-Protection, and the signed-releases and CII-Best-Practices probes) only score meaningfully on a public repository, so running it here would produce noise rather than a posture. It belongs with the decision about making this repo public, not before it. No tracking issue filed yet. |
| CI/CD | Applies | Partial. OIDC-only credentials, SHA-pinned actions, per-job least-privilege permissions (including corpus-freshness.yml, fixed 2026-07-05). No branch-ruleset artifact, no CODEOWNERS-enforced review (CODEOWNERS file added 2026-07-05; the hosted branch-protection setting itself is a manual, human action — see the 2026-07-05 execution log in the audit folder). No tracking issue filed yet. |
| Release & Versioning | Applies | Partial. .github/workflows/release.yml (added 2026-07-10) is tag-triggered on v*: checks the tag matches pyproject.toml's version, re-runs make verify at the tagged commit, builds sdist+wheel, generates a CycloneDX 1.7 SBOM, attests SLSA build provenance, and creates a GitHub Release with the matching CHANGELOG.md section as notes. Nothing is published to a package index (no PyPI project registered, no other repo pins this one), so the GitHub Release is the publish target, not Trusted Publishing — the pipeline still exists so the deployed artifact is traceable to a signed, tested, tagged build. No tracking issue filed yet. |
| Accessibility | Applies | Partial. Merge-blocking structural and browser pa11y/axe gates are green, and as of 2026-08-05 the structural gate covers all four public pages rather than the chat page alone — /embed, /offline, and /guide were previously unchecked, and all three passed on the day it was widened. The "Sources" caption is now a heading on both answering surfaces, so screen-reader heading navigation reaches it. The manual screen-reader walkthrough is still pending (docs/audits/a11y-walkthrough.md), and nothing above substitutes for it: no screen reader has been used on any of the four pages. |
| Observability | Applies (Tier: informational/low-traffic demo service — no SLO). Privacy-safe JSON records correlate request/model outcomes with Lambda-owned IDs and expose canonical provider/model, token-derived estimated cost, and request/model duration without content or request metadata. Promotion captures the numbered candidate's real log tail and tests the installed CloudWatch filters before moving live. Alarms, dashboard, 14-day retention, and the account's $20/month fare-demo AWS Budget provide layered backstops; a confirmed SNS subscriber remains operator-supplied. |
— |
| Internationalization | Applies | English and Spanish are the supported answer languages. Gettext catalogs for EN/ES/TL have 9 merge-blocking gates (docs/I18N.md), but Tagalog remains experimental: its 15-case stretch suite uses cross-lingual retrieval over a corpus with no agency-authored Tagalog source page and is excluded from the production-core release denominator. The Spanish parity delta is 0.0 points over 22 mirror pairs, each of which a merge-blocking mirror-integrity gate holds to the same agency, expected behavior, and required-fact count as the English case it mirrors (added 2026-08-05; it found three malformed pairs, all of which had been reporting parity). Still open: the §7 native-Spanish benchmark has never been run, so the 0.0 covers this repo's own mirrored cases and nothing beyond them. It is now scaffolded, not just named — a published rubric plus a committed, entirely blank census of all 28 Spanish answers (evals/spanish/native_es_rubric_2026-08-05.jsonl, make spanish-quality), reported in EVALS.md as not measured and never as a zero; 0 of 28 are rated and 0 of 28 questions are externally sourced. Separately, the second-harness lexical multilingual proxy remains below threshold at 0.581, computed over a golden.jsonl export that still carries the three pre-repair pairings. |
| AI Evaluation | Applies | This is the project's thesis: 186 production-core English/Spanish cases, 15 separately reported experimental Tagalog cases, versioned prompts, a committed regression baseline, and a second-harness Plumbline replay that runs on every pull request (that harness is a separate project by the same author, so the replay is not a third-party audit; it is public and replay-only, so anyone can rerun it — make audit, offline, no keys). The promoted baseline remains 192/201 (95.5%) overall and 177/186 (95.2%) production-core. The latest observed nightly is lower at 190/201 overall and 175/186 production-core, with the cross-agency gate red, so it has not replaced the baseline. A direct probe confirmed both the answer and judge models are deterministic at temperature 0. Judge calibration is the weakest evidence here and is now labeled as such on the report itself: 4 scored labels against a floor of 37 (10% of the promoted run's 367 judged pairs), and κ is undefined, not the 1.000 published until 2026-08-05 — every label that recorded a human/judge disagreement had gone stale, so the surviving sample was the agreeing half and could only report 100%. A floor-sized, failure-first relabeling worksheet is committed at evals/calibration/judge_relabel_worksheet_2026-08-05.jsonl (python -m evals.calibration --worksheet <run_dir>); it holds 37 unlabeled rows and needs a human. make relabel walks those rows offline, showing each one's judge criterion, question, retrieved passages, and answer, and recording the reviewer's verdict and reason; it never proposes a verdict and withholds the judge's own call until after the reviewer has given theirs. |
| Documentation | Applies | Partial. This table is new (2026-07-05); ADRs, model card, and CONTRIBUTING exist and are dated. CHANGELOG.md added 2026-07-05. No tracking issue filed yet. |
| Responsible-Tech Framework | Applies (civic domain touching age/disability/income/veteran status). Misuse-resistance is code-enforced and tested (src/assistant/guards.py). The three governance artifacts now exist, synthesized from ADR 0004, SECURITY.md, and the model card: a DPIA (docs/dpia.md), an AI risk register (docs/ai-risk-register.md), and an EU-AI-Act classification (docs/eu-ai-act-classification.md) — the last of which shows the "never determine eligibility" invariant is what keeps the system below the Annex III high-risk line. |
— |
No GitHub tracking issues are linked above: this pass verified gh auth status succeeds against this repo but did not file issues autonomously (that
write action was outside this remediation pass's scope — see the 2026-07-05
execution log). Until issues exist, the linked doc/file in each row is the
authoritative gap record; open the issues by hand (or ask an agent to, in a
session that's explicitly scoped for it) and replace these notes with links.
- It never determines anyone's eligibility. It explains published criteria ("the published criteria are 65 and older") and leaves the decision to the agency. An output guard blocks determination language in English and Spanish, and the eval suites test it.
- It never answers without a citation. Every factual claim must trace to a retrieved passage from a dated policy snapshot. An uncited answer is blocked by the output guard and counted as a critical eval failure.
- It does not collect personal information. Questions containing ID numbers, birth dates, or contact details are refused before retrieval runs, and the details are not echoed back or logged.
- It does not give medical, legal, or immigration advice, and it says so plainly when asked.
- It does not pretend to be current. Answers state the date the underlying policy documents were fetched and suggest confirming with the agency.
Each of these rules is enforced in code (src/assistant/guards.py) and tested
by the evaluation suites (evals/suites/). The model card
(docs/model-card.md) describes scope and limits in more detail.
186 production-core English/Spanish cases, plus 15 explicitly experimental Tagalog stretch cases, across nine suites. Each case is written against a specific passage in the corpus and readable by a non-engineer. The 201-case research total includes 30 counterfactual sensitivity variants; the Tagalog cases are reported separately and do not contribute to the production release denominator.
The harness is validated beyond its own scoreboard: a defect-injection self-test
proves the gate catches planted bugs (make eval-selftest), a coverage map
checks no corpus provision goes untested (make coverage,
docs/eval-coverage.md), and a robustness report gives confidence intervals and
a leave-one-suite-out jackknife (make robustness, docs/eval-robustness.md).
The rendered report and the improvement curve publish to the public
evaluation evidence hub via the manual Pages
workflow.
The suites:
| Suite | What it tests |
|---|---|
| groundedness | claims trace to retrieved passages; prices and ages match the documents |
| refusal | PII, prompt injection, determination-seeking, out-of-corpus agencies |
| edge_cases | real eligibility boundaries: 62 vs 65, Medicare vs Medi-Cal, veteran documents |
| multilingual | Spanish parity, measured against mirrored English cases (each pair is gate-checked to be one question in two languages) |
| freshness | "as of" disclosure, expired programs, refusal to speculate about future fares |
| conversation | multi-turn follow-ups: references resolve against prior turns; the guard holds across turns |
| cross_agency | one answer attributes facts to multiple agencies correctly |
| sensitivity | minimal-pair boundaries must change the answer when policy changes |
| stretch_tagalog | measured Tagalog gap over the English-only corpus |
Scoring combines deterministic checks (citation resolves to the corpus,
forbidden phrases absent, response language matches the question) with an
LLM judge for groundedness and helpfulness. The judge model is different from
the answer model, its prompts are versioned in prompts/, and judge output
that fails to parse counts as an error rather than a pass.
A 26-case smoke suite runs in CI on every pull request. The full suite runs nightly. A drop of more than 2 points on any suite fails the build.
Spanish parity is reported as the pass-rate delta between each Spanish case and its English mirror, and a run aborts before its first model call if any declared mirror is not one: a mirror must be the same agency, the same expected behavior, and carry at least as many required facts as the case it mirrors. That gate was added on 2026-08-05 after it found three of the 22 pairs were not pairs, and the parity delta had been reporting 0.0 points across all three. Repairing them left the delta at 0.0 points over 22 verified pairs; the number did not move, but until then it was not measuring what it claimed to.
Both are served from a content-keyed cache of answer and judge calls, keyed on
the rendered prompt text, so a change that cannot alter an answer is not paid
for twice. Only the model call is cached: the deterministic checks, the
regression gate, and the parity gate re-execute on every run. One nightly a
week (Monday) bypasses the cache to re-measure the provider directly. See
docs/decisions/0022-persisted-eval-cache-and-weekly-cold-run.md.
Only Spanish is at parity. evals/suites/stretch_tagalog.yaml adds 15 more
cases, each mirroring an existing English case, that ask the same questions
in Tagalog — chosen over Chinese, Vietnamese, or Korean because it is
space-delimited Latin script, so the existing tokenizer needs only a
fare-vocabulary lexicon, not a script change, to bridge a query into the
English-only corpus. No agency in the corpus publishes a Tagalog page, so
this suite is deliberately kept out of the core count and the CI smoke gate:
it is a clearly-tagged, non-parity suite that is expected to score well below
English and Spanish, and EVALS.md prints its own "Stretch-language parity
(Tagalog)" table against the same English mirrors the Spanish table uses, so
the gap is a counted number, not a claim. Full details in
docs/model-card.md.
This section leads with an audit, not a victory lap, on purpose. An agency is liable for what its chatbot tells a rider: a Canadian tribunal held an airline responsible for fare advice its bot invented, and a New York City business chatbot gave advice to break the law and stayed live for months. A wrong fare or eligibility line here would be the agency's problem, not a demo footnote, so the honest posture is to show the outside floor first. Two scores in the table below sit near zero. They are the floor of a deterministic lexical judge that cannot tell a paraphrase or a redirect from a fabricated claim, not evidence of fabrication; the note under the table explains exactly why, and that note is part of the result.
The harness above is white-box: its checks know this corpus's doc-ids, the
guards.py rules, and the agency-scope contract. As a second layer, the
deployed assistant's recorded answers are replayed through
Plumbline — a separate, public,
Apache-2.0 evaluation harness with its own suites and its own judge, that sees
only questions, recorded answers, sources, and declared ground truth.
Be precise about how independent that is. Plumbline is genuinely separate code
with a different scoring model, blind to this system's internals, which is why
it finds things the white-box suites cannot. It is also written by the same
author, so this is a second-harness replay, not a third-party audit. What
changed on 2026-08-16 is that anyone can now rerun it. The audit used to run
on govchat-eval, which went private and archived: make audit needed a clone
nobody outside had, and the CI job that was meant to notice ran only on a
schedule, only behind a repository variable, and with continue-on-error, so it
could not fail a build. Plumbline is public and replay-only, so the audit now
runs on every pull request with no secrets, no model calls, and no cost.
make audit
resolves the harness from plumbline.pin (one exact commit, verified after
checkout, never a dependency of this project), scores the committed evidence
bundle, and then gates the report. Scores below:
| Suite | Score | Floor | |
|---|---|---|---|
| smoke | 1.0000 | 1.00 | ✅ |
| multilingual (answered in the language asked) | 1.0000 | 0.95 | ✅ |
| representational_harms | 1.0000 | 1.00 | ✅ |
| citation_validity | 0.9936 | 0.99 | ✅ |
| privacy | 0.9795 | 0.97 | ✅ |
| refusal | 0.8615 | 0.80 | ✅ |
accessibility (structural, on web/index.html) |
0.8000 | 0.80 | ✅ |
| groundedness | 0.7308 | 0.55 | ✅ score, ✕ findings |
| citation_accuracy | 0.7136 | 0.55 | ✅ |
| cross_language | 0.3864 | 0.35 | ✅ score, ✕ findings |
| adversarial | 0.0000 | 0.00 | ✅ (see below) |
| accuracy | 0.0591 | 0.04 | ✅ score, ✕ findings |
Read that table with the explanation, which is part of the result rather than an
excuse for it. Every floor is measured against the committed evidence and
explained in evals/plumbline/target.toml; a floor above what the system does is
a gate that is red on the day it lands. Three of the scores are largely measuring
the instrument: accuracy compares a paragraph against a bag of required facts;
adversarial and refusal turn on a refusal-marker list that does not contain this
assistant's decline wording, so three correctly-refused jailbreak probes score
zero; cross-language disagreements are mostly a phone number the English answer
carries and the Spanish one does not.
The audit also found 76 hard failures the floors do not hide, and two of them
are now fixed defects rather than notes: the snapshot-date disclosure scoring as
an unsupported number, and a phone number the corpus cleaner broke into
805. 963.3364. Each finding is listed with a reason and an owner in
evals/plumbline/acknowledged_findings.json, and evals/plumbline_guard.py —
not the harness's own exit code — is the merge gate: it fails on any suite below
the committed baseline, any hard failure nobody acknowledged, and any
acknowledgement that has stopped firing. The method, the suite mapping, the two
deliberately disabled suites, and the adapter's shape decisions are in
docs/audits/methodology.md.
The accessibility score above is the automated transcript and structural check.
It is not a sign-off on the lived experience: a manual screen-reader and
keyboard walkthrough is still pending, tracked in
docs/audits/a11y-walkthrough.md and noted in
the model card. Treat the demo as accessibility-reviewed by automation, not yet
by a person.
For a buyer or IT reviewer who wants the safety, privacy, and testing posture on
one page without reading the code, see
docs/procurement-brief.md. The security posture,
how to report a vulnerability, and a deployment hardening checklist are in
SECURITY.md.
These are two distinct public surfaces:
- Evaluation evidence hub: generated scoreboards, representative failures, trend history, and governance evidence.
- Live AWS assistant: the rider-facing system those evaluations exercise.
The live assistant states what it will not do, supports English and Spanish, and cites the dated policy snapshot behind every answer. Tagalog behavior is experimental, evaluated only as a 15-case stretch over a corpus with no agency-authored Tagalog source page; it is not a supported production language. The assistant does not read the agencies' live websites when a rider asks a question. Questions and conversation history are processed transiently; their raw text is not logged or used as a cache key. Successful answer payloads may remain in a bounded in-memory cache until the serverless container is recycled. Refused, guarded, or personal-information-like inputs are not cached.
The assistant's "How this assistant is tested" panel links to the separate
evidence hub, so a reviewer can move between the system and its evidence
without confusing the two deployments. If you are walking someone through the
project, docs/DEMO-SCRIPT.md is a three-minute script: the hook, a few
rehearsed queries that show grounded citations and the refusal to determine
eligibility, and the honest-failures move.
For riders with no signal at the stop, /offline renders every agency's dated
policy text on one printable page, built from the committed corpus with no model
call. The page displays the earliest and latest fetch dates represented so the
snapshot window cannot be mistaken for live agency data (make offline writes
it locally for inspection).
For riders who would rather browse than type — low signal, low literacy, or a
preference for forms over chat — /guide is a zero-model-call, statically
pre-rendered "which fare applies to me" walkthrough: choose an agency, then a
published fare category, to reach the criteria, price, proof, and next step.
It has no input fields on purpose and never determines eligibility; it only
shows saved copies of the agency's published text, verbatim, with a source link
and the page's earliest-to-latest snapshot window
(make guide writes it locally for inspection).
An agency can embed the assistant in its own fare page with one iframe pointing
at /embed:
<iframe src="https://yahp6ddfo1.execute-api.us-west-2.amazonaws.com/embed"
title="Transit fare policy assistant"
width="100%" height="520"
style="border:1px solid #d6d3cb;border-radius:8px"></iframe>/embed is the only frameable route: it carries the reference-implementation
notice and the will-not-do line, and is served same-origin so its /api/ask
call stays under connect-src 'self'. The main page keeps x-frame-options: DENY. By default the widget is frameable only same-origin (frame-ancestors 'self'); set FPA_EMBED_ANCESTORS to a space-separated origin allowlist (the
agency's own domains) to let those sites embed it.
The offline path above needs no credentials. Live runs use Claude on Amazon Bedrock by default, authenticated through the standard AWS credential chain. The recommended local setup is IAM Identity Center (SSO) — no long-lived keys on disk:
aws configure sso # once; creates a profile
aws sso login --profile my-profile
AWS_PROFILE=my-profile AWS_REGION=us-west-2 uv run python -m evals.runner --full
uv run python -m assistant.cli "¿Cuánto cuesta el pasaje reducido en Yolobus?"CI authenticates the same way in spirit: GitHub Actions assumes an IAM role
via OIDC federation (AWS_OIDC_ROLE_ARN repository variable), so the repo
holds no AWS secrets at all. Without credentials, eval runs fall back to
offline mode automatically.
To use the direct Anthropic API instead, set FPA_PROVIDER=anthropic and
ANTHROPIC_API_KEY.
A fourth backend, FPA_PROVIDER=local, talks to a small model served
locally by Ollama — no network call, no per-query
cost, for an offline kiosk deployment (EXP-13 in
docs/ideation/03-expansions.md). evals/backend_comparison.py runs the
same guarded pipeline against local and bedrock and publishes the
measured delta; see docs/decisions/0014-local-model-kiosk-backend.md for
the result (a small model measured well short of the bar — generation does
not ship on the kiosk today).
To rebuild the corpus from the live agency sites (polite, manifest-driven, about two minutes because of crawl delays):
make fetch && make ingestPublished fare pages from Monterey-Salinas Transit (MST), Santa Barbara MTD
(SBMTD), Yolobus, Sacramento Regional Transit (SacRT), Humboldt Transit
Authority (HTA), Elk Grove Transit Services (e-tran), Santa Cruz METRO (SCMTD),
Solano County Transit (SolTrans), Fresno Area Express (FAX), County
Connection (CCCTA), San Joaquin RTD (SJRTD), AC Transit, WestCAT (Western
Contra Costa Transit Authority), San Luis Obispo RTA (SLORTA), the Santa
Clara Valley Transportation Authority (VTA), Napa Valley Vine Transit
(VINE), SamTrans (San Mateo County Transit District), and Marin Transit
(Marin County Transit District),
snapshotted with fetch dates in corpus/manifest.yaml. MST's Spanish fares page is included,
which makes part of the multilingual suite a same-language retrieval test and
the rest an honest cross-lingual one, as are FAX's Spanish reduced-fare
document and AC Transit's Spanish fares page. MST and SBMTD are live on
Cal-ITP Benefits, and so are Santa Cruz METRO (Tap2Cruz) and SLO RTA, whose
discounts page routes contactless discount verification through
benefits.calitp.org — so the corpus overlaps with a real eligibility
verification domain. FAX is the first Central Valley agency in the corpus and
the first whose reduced fare is a funded suspension rather than a discount.
San Joaquin RTD is the second Central Valley agency and the only one whose
senior discount age depends on the rider's city of residence (60, 62, or 65
across San Joaquin County), which is the boundary its edge cases exist to pin.
SLO RTA fills the Central Coast gap between SBMTD and MST, and is the only
corpus agency with an age-tiered senior fare: 65-79 pay half, 80 and over
ride free with a VIP Card.
VTA is the largest system in the corpus and the one whose fare table did not
survive ingestion: its rider-category tab labels are UI furniture the chunker
drops, so the category-to-price bindings are honestly absent from the corpus
(the manifest documents the gap and an eval case pins the honest partial
answer), while the structures VTA states in prose — the Day Pass Accumulator
cap, the express surcharge and its exemptions, the Clipper START criteria —
are cased like any other agency's.
SamTrans brings the corpus's first named-but-unpublished fare cap: its
Clipper FAQ says open-payment fare capping "is calculated separately from
Monthly Pass usage" and publishes no amounts, period, or rule, so the corpus
carries the one sentence the agency wrote and an eval case pins the honest
partial answer.
Marin Transit brings an accumulator-capped structure whose published caps
($5/day, $40/month, Clipper only) share a price with a pass that covers
different buses — and two live-page findings the corpus pins with eval cases
rather than smooths over: a day pass its fares page still sells at the
farebox but no page prices, and a paper 31-day pass retired on a published
schedule the page still describes in the future tense.
AC Transit is the first accumulator-capped structure: its Day, Weekly, and Monthly passes are "fare maximums" that apply automatically once pay-per-ride spending reaches the pass price, and both the rule and the amounts are published as HTML text the pipeline can quote — unlike Santa Cruz METRO's cap amounts, which live in an image and stay honestly out of the corpus.
SolTrans, County Connection, AC Transit, and WestCAT are the corpus's four
Clipper participants. That is the point of including them and also the main
hazard: Clipper is a regional card, so a rider reads a Clipper answer as
generalizing across the Bay Area, while the corpus holds only each named
agency's side of it. The xagency-009 / xagency-010 cases
require the assistant to answer for SolTrans and stop, and SolTrans' Clipper
page is annotated in the manifest to separate its own policy from its
assertions about other operators (BART, Golden Gate Transit, SF Bay Ferry,
WestCat, County Connection), which it does not speak for. County Connection's
pages are annotated the same way, and they corroborate the SolTrans link from
the other side: its Transfers page grants free Clipper transfers from
SolTrans, which xagency-012 tests with both agencies' own pages cited.
WestCAT's pages carry the same annotation, and where the published windows
overlap they disagree: SolTrans publishes a 60-minute window for inter-agency
Clipper transfers and WestCAT publishes 120 minutes, which xagency-014
requires an answer to attribute rather than resolve.
The Vine, one of the agencies SolTrans names, is also in the corpus, and its
own pages document the Clipper START discount but no inter-agency transfer
credits — an asymmetry xagency-015 requires the assistant to report as
attribution, not resolve. Golden Gate Transit, also named there, was checked
and could not be added: its site is fetchable but not ingestible by this
pipeline (the manifest header records why).
Unitrans was in the original pilot list; its WAF blocks non-browser clients,
so SacRT was substituted rather than working around the block
(docs/decisions/0002). Re-checked 2026-08-12: its robots.txt permits the fare
pages, the WAF still returns 403 to this project's fetcher, and it remains out.
RABA (Redding) was checked as a candidate on 2026-08-13 and is also out, for a
cleaner reason: its robots.txt allows only a short list of named crawlers and
disallows everyone else, this fetcher included, so nothing was fetched at all
(docs/decisions/0002).
None of that fare text belongs to this project. It is each agency's copyrighted
work, snapshotted so a dated evaluation can be re-run against what it was scored
on, and it is carved out of this repository's Apache License, Version 2.0, in
NOTICE. corpus/LICENSE-NOTE.md states in
plain English whose it is, what you may and may not assume about it, and where
each agency's own site and terms of use are. corpus/manifest.yaml records the
robots/Content-Signal review that governed fetching separately from each
agency's redistribution terms, because those are different questions.
The corpus keeps its stable legacy version ID for deployed pins and existing
clients, and now also reports a full content_version over every
behavior-relevant chunk field. Source-complete schema-2 archives add a separate
snapshot_version over content plus the verified fetch URL, date, status,
format, raw digest, and byte count; they are staged, revalidated, and atomically
published with the exact source bytes (docs/decisions/0020). The /version
endpoint still compares FPA_PINNED_CORPUS_VERSION against the compatibility
ID during the additive rollout. PDF policies are supported too (text-first,
with an OCR fallback for scans; ADR 0008), so a fare program published as PDF
is citable like an HTML page.
A second, structured evidence source checks the prose corpus against reality:
make gtfs-fetch transactionally captures exact, SHA-256-receipted GTFS ZIPs;
make gtfs-check cross-validates agency fares against the atomically selected
set (MST and SBMTD, confirmed live; ADRs 0011 and 0024),
flagging disagreement without ever overriding an answer.
corpus/ manifest, raw HTML snapshots, processed chunks
src/assistant/ ingest, retrieve (BM25, optional dense), guards, answer, cli
prompts/ versioned system, answer, and judge prompts
evals/ suites (YAML), runner, deterministic checks, judges, report
docs/ model card, ADRs, generated HTML report
The pattern is not specific to transit: a corpus manifest with dated
snapshots, chunked policy text, an answer pipeline with input/output guards,
and YAML cases scored by deterministic checks plus a separate judge model.
docs/adapting.md walks through what to change for, say, a
benefits-eligibility assistant, and make template TARGET=<dir> extracts the
domain-agnostic modules into a starter skeleton for a second domain
assistant, so it can start from this project's audited harness without
forking the repo (template/MANIFEST.yaml, docs/ROADMAP.md P3-5).
Reference implementation. No accounts, no persistence of user queries.
Fare information shown is based on policies published as of the dates in
corpus/manifest.yaml; confirm anything time-sensitive with the agency.
Apache-2.0 licensed (see LICENSE and NOTICE). Corpus snapshots remain the work of their respective transit agencies.