stale_acknowledged.json

{
  "_comment": "Loud, documented staleness waivers for the provenance gate (evals/provenance.py). Every entry needs a non-empty reason; remove an entry the moment the artifact is regenerated. Current entries: the 2026-08-10 hta.org domain move and the thirteen agencies added on 2026-08-12/13/14 (Elk Grove e-tran, Santa Cruz METRO, SolTrans, Fresno Area Express, County Connection, San Joaquin RTD, AC Transit, WestCAT, SLO RTA, VTA, Napa VINE, SamTrans, Marin Transit), all pending the same next promoted eval run. Also waived: the 2026-08-13 system-prompt v10 -> v11 change (scope rule widened from five named agencies to all nine) and the v11 through v20 changes (County Connection named as the tenth, San Joaquin RTD as the eleventh, AC Transit as the twelfth, WestCAT as the thirteenth, SLO RTA as the fourteenth, VTA as the fifteenth, Napa VINE as the sixteenth, SamTrans as the seventeenth, Marin Transit as the eighteenth), pending the same next promoted eval run. Also waived from 2026-08-16: the system-prompt v20 -> v21 rider-class binding fix and the judge_groundedness v2 -> v3 provenance-visibility fix, both pending the same next promoted eval run.",
  "acknowledged": [
    {
      "artifact": "EVALS.md",
      "field": "corpus_version",
      "reason": "Six causes, all pending the same regeneration. (1) 2026-08-10: hta-fares source domain moved (humboldttransit.org now 404s; site relocated to hta.org). The refetch rebound corpus_version 0938fff0539a to 35ec70d6359d; the refresh report confirmed no policy-text change, only the source URL and fetch metadata. (2) 2026-08-12: Elk Grove Transit Services (e-tran) added as the sixth agency (etran-fares, etran-senior-pass). (3) 2026-08-12: Santa Cruz METRO (SCMTD) added as the seventh agency (scmtd-fares-passes, scmtd-accessibility, scmtd-tap2cruz); merging (2) and (3) rebound corpus_version to feb21f02c510. (4) 2026-08-12: SolTrans (Solano County Transit) added as the eighth agency, four documents (soltrans-fare-table, soltrans-clipper-card, soltrans-ways-to-pay, soltrans-paperless-fares), rebinding corpus_version to 5e6a24ef0702 (20 documents, 140 chunks). (5) 2026-08-13: Fresno Area Express (FAX) added as the ninth agency, three documents (fax-fares plus the corpus's first two PDFs, the reduced-fare program instructions in English and Spanish), rebinding corpus_version once more to 95794539d1d0 (23 documents, 152 chunks). Unlike (1), causes (2) through (14) are all new policy text: the committed scores were computed against a corpus containing none of the four agencies. They remain valid for the cases they cover and say nothing at all about the new E-tran, SCMTD, SolTrans, FAX, CCCTA, SJRTD, actransit-*, WestCAT, SLO RTA, vta-*, VINE, samtrans-*, or marin-* cases, none of which has ever been scored live. (6) 2026-08-14: County Connection (CCCTA) added as the tenth agency, five documents (cccta-fare-types-prices, cccta-clipper-card, cccta-transfers, cccta-rtc-discount-card, cccta-link-fares), rebinding corpus_version once more to 60033300dc49 (28 documents, 177 chunks). Like (2) through (5) it is new policy text: the committed scores were computed against a corpus without CCCTA and say nothing about the new CCCTA cases, none of which has ever been scored live. (7) 2026-08-14: San Joaquin RTD (SJRTD) added as the eleventh agency, two documents (sjrtd-fares, sjrtd-dfc), rebinding corpus_version once more to d113659adda8 (30 documents, 188 chunks). Like (2) through (6) it is new policy text: the committed scores were computed against a corpus without SJRTD and say nothing about the new SJRTD cases, none of which has ever been scored live — its fifteen cases were renumbered on merge (ground-055..058, edge-087..093, fresh-024, ml-034, refuse-032, xagency-016, the FAX precedent). (8) 2026-08-14: AC Transit added as the twelfth agency, three documents (actransit-fares, actransit-fares-es, actransit-discounts), rebinding corpus_version once more to 5db0d6a4bfbd (33 documents, 214 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without AC Transit and say nothing about the new actransit-* cases, none of which has ever been scored live; the same PR retargets refuse-021 at Golden Empire Transit, since its old premise (AC Transit outside the corpus) no longer holds. (9) 2026-08-14: WestCAT added as the thirteenth agency, four documents (westcat-fares-all, westcat-transfers, westcat-clipper, westcat-buying), rebinding corpus_version once more to 8b119819d9c8 (37 documents, 227 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without WestCAT and say nothing about the new WestCAT cases (ground-047..050, edge-075..081, fresh-022, ml-030..031, xagency-013..014, refuse-030), none of which has ever been scored live. (10) 2026-08-14: SLO RTA (SLORTA) added as the fourteenth agency, four documents (slorta-fares, slorta-discounts, slorta-passes, slorta-contactless), rebinding corpus_version once more to b21d4e3b3c98 (41 documents, 249 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SLO RTA and say nothing about the new SLO RTA cases, which were renumbered on merge (ground-059..062, edge-094..100, fresh-025..026, ml-035, refuse-033, xagency-017, the FAX precedent) and none of which has ever been scored live. (11) 2026-08-14: VTA added as the fifteenth agency, three documents (vta-fares, vta-regional-transfers, vta-rtc-card), rebinding corpus_version once more to cbc07c922784 (44 documents, 264 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without VTA and say nothing about the new vta-* cases, none of which has ever been scored live. (12) 2026-08-14: Napa Valley Vine Transit (VINE) added as the sixteenth agency, two documents (vine-fares, vine-go), rebinding corpus_version once more to 09f9c297deba (46 documents, 270 chunks); the same PR records the Golden Gate Transit NO-GO (fetchable but not ingestible — its ASP.NET page-wrapper form cleans to zero sections) in the manifest header. Like the others it is new policy text: the committed scores were computed against a corpus without the Vine and say nothing about the new VINE cases, none of which has ever been scored live. (13) 2026-08-14: SamTrans added as the seventeenth agency, three documents (samtrans-fares, samtrans-fare-types, samtrans-clipper), rebinding corpus_version once more to cbfe81efcf3f (49 documents, 279 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SamTrans and say nothing about the new samtrans-* cases, none of which has ever been scored live. (14) 2026-08-14: Marin Transit added as the eighteenth agency, three documents (marin-fares, marin-clipper, marin-31day-transition), rebinding corpus_version once more to 3dd8b7bd757e (52 documents, 301 chunks), which is what HEAD now ships. Like the others it is new policy text: the committed scores were computed against a corpus without Marin Transit and say nothing about the new marin-* cases, none of which has ever been scored live. Regenerate all three artifacts at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "baseline.json",
      "field": "corpus_version",
      "reason": "Six causes, all pending the same regeneration. (1) 2026-08-10: hta-fares source domain moved (humboldttransit.org now 404s; site relocated to hta.org). The refetch rebound corpus_version 0938fff0539a to 35ec70d6359d; the refresh report confirmed no policy-text change, only the source URL and fetch metadata. (2) 2026-08-12: Elk Grove Transit Services (e-tran) added as the sixth agency (etran-fares, etran-senior-pass). (3) 2026-08-12: Santa Cruz METRO (SCMTD) added as the seventh agency (scmtd-fares-passes, scmtd-accessibility, scmtd-tap2cruz); merging (2) and (3) rebound corpus_version to feb21f02c510. (4) 2026-08-12: SolTrans (Solano County Transit) added as the eighth agency, four documents (soltrans-fare-table, soltrans-clipper-card, soltrans-ways-to-pay, soltrans-paperless-fares), rebinding corpus_version to 5e6a24ef0702 (20 documents, 140 chunks). (5) 2026-08-13: Fresno Area Express (FAX) added as the ninth agency, three documents (fax-fares plus the corpus's first two PDFs, the reduced-fare program instructions in English and Spanish), rebinding corpus_version once more to 95794539d1d0 (23 documents, 152 chunks). Unlike (1), causes (2) through (14) are all new policy text: the committed scores were computed against a corpus containing none of the four agencies. They remain valid for the cases they cover and say nothing at all about the new E-tran, SCMTD, SolTrans, FAX, CCCTA, SJRTD, actransit-*, WestCAT, SLO RTA, vta-*, VINE, samtrans-*, or marin-* cases, none of which has ever been scored live. (6) 2026-08-14: County Connection (CCCTA) added as the tenth agency, five documents (cccta-fare-types-prices, cccta-clipper-card, cccta-transfers, cccta-rtc-discount-card, cccta-link-fares), rebinding corpus_version once more to 60033300dc49 (28 documents, 177 chunks). Like (2) through (5) it is new policy text: the committed scores were computed against a corpus without CCCTA and say nothing about the new CCCTA cases, none of which has ever been scored live. (7) 2026-08-14: San Joaquin RTD (SJRTD) added as the eleventh agency, two documents (sjrtd-fares, sjrtd-dfc), rebinding corpus_version once more to d113659adda8 (30 documents, 188 chunks). Like (2) through (6) it is new policy text: the committed scores were computed against a corpus without SJRTD and say nothing about the new SJRTD cases, none of which has ever been scored live — its fifteen cases were renumbered on merge (ground-055..058, edge-087..093, fresh-024, ml-034, refuse-032, xagency-016, the FAX precedent). (8) 2026-08-14: AC Transit added as the twelfth agency, three documents (actransit-fares, actransit-fares-es, actransit-discounts), rebinding corpus_version once more to 5db0d6a4bfbd (33 documents, 214 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without AC Transit and say nothing about the new actransit-* cases, none of which has ever been scored live; the same PR retargets refuse-021 at Golden Empire Transit, since its old premise (AC Transit outside the corpus) no longer holds. (9) 2026-08-14: WestCAT added as the thirteenth agency, four documents (westcat-fares-all, westcat-transfers, westcat-clipper, westcat-buying), rebinding corpus_version once more to 8b119819d9c8 (37 documents, 227 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without WestCAT and say nothing about the new WestCAT cases (ground-047..050, edge-075..081, fresh-022, ml-030..031, xagency-013..014, refuse-030), none of which has ever been scored live. (10) 2026-08-14: SLO RTA (SLORTA) added as the fourteenth agency, four documents (slorta-fares, slorta-discounts, slorta-passes, slorta-contactless), rebinding corpus_version once more to b21d4e3b3c98 (41 documents, 249 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SLO RTA and say nothing about the new SLO RTA cases, which were renumbered on merge (ground-059..062, edge-094..100, fresh-025..026, ml-035, refuse-033, xagency-017, the FAX precedent) and none of which has ever been scored live. (11) 2026-08-14: VTA added as the fifteenth agency, three documents (vta-fares, vta-regional-transfers, vta-rtc-card), rebinding corpus_version once more to cbc07c922784 (44 documents, 264 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without VTA and say nothing about the new vta-* cases, none of which has ever been scored live. (12) 2026-08-14: Napa Valley Vine Transit (VINE) added as the sixteenth agency, two documents (vine-fares, vine-go), rebinding corpus_version once more to 09f9c297deba (46 documents, 270 chunks); the same PR records the Golden Gate Transit NO-GO (fetchable but not ingestible — its ASP.NET page-wrapper form cleans to zero sections) in the manifest header. Like the others it is new policy text: the committed scores were computed against a corpus without the Vine and say nothing about the new VINE cases, none of which has ever been scored live. (13) 2026-08-14: SamTrans added as the seventeenth agency, three documents (samtrans-fares, samtrans-fare-types, samtrans-clipper), rebinding corpus_version once more to cbfe81efcf3f (49 documents, 279 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SamTrans and say nothing about the new samtrans-* cases, none of which has ever been scored live. (14) 2026-08-14: Marin Transit added as the eighteenth agency, three documents (marin-fares, marin-clipper, marin-31day-transition), rebinding corpus_version once more to 3dd8b7bd757e (52 documents, 301 chunks), which is what HEAD now ships. Like the others it is new policy text: the committed scores were computed against a corpus without Marin Transit and say nothing about the new marin-* cases, none of which has ever been scored live. Regenerate all three artifacts at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "golden.jsonl",
      "field": "corpus_version",
      "reason": "Six causes, all pending the same regeneration. (1) 2026-08-10: hta-fares source domain moved (humboldttransit.org now 404s; site relocated to hta.org). The refetch rebound corpus_version 0938fff0539a to 35ec70d6359d; the refresh report confirmed no policy-text change, only the source URL and fetch metadata. (2) 2026-08-12: Elk Grove Transit Services (e-tran) added as the sixth agency (etran-fares, etran-senior-pass). (3) 2026-08-12: Santa Cruz METRO (SCMTD) added as the seventh agency (scmtd-fares-passes, scmtd-accessibility, scmtd-tap2cruz); merging (2) and (3) rebound corpus_version to feb21f02c510. (4) 2026-08-12: SolTrans (Solano County Transit) added as the eighth agency, four documents (soltrans-fare-table, soltrans-clipper-card, soltrans-ways-to-pay, soltrans-paperless-fares), rebinding corpus_version to 5e6a24ef0702 (20 documents, 140 chunks). (5) 2026-08-13: Fresno Area Express (FAX) added as the ninth agency, three documents (fax-fares plus the corpus's first two PDFs, the reduced-fare program instructions in English and Spanish), rebinding corpus_version once more to 95794539d1d0 (23 documents, 152 chunks). Unlike (1), causes (2) through (14) are all new policy text: the committed scores were computed against a corpus containing none of the four agencies. They remain valid for the cases they cover and say nothing at all about the new E-tran, SCMTD, SolTrans, FAX, CCCTA, SJRTD, actransit-*, WestCAT, SLO RTA, vta-*, VINE, samtrans-*, or marin-* cases, none of which has ever been scored live. (6) 2026-08-14: County Connection (CCCTA) added as the tenth agency, five documents (cccta-fare-types-prices, cccta-clipper-card, cccta-transfers, cccta-rtc-discount-card, cccta-link-fares), rebinding corpus_version once more to 60033300dc49 (28 documents, 177 chunks). Like (2) through (5) it is new policy text: the committed scores were computed against a corpus without CCCTA and say nothing about the new CCCTA cases, none of which has ever been scored live. (7) 2026-08-14: San Joaquin RTD (SJRTD) added as the eleventh agency, two documents (sjrtd-fares, sjrtd-dfc), rebinding corpus_version once more to d113659adda8 (30 documents, 188 chunks). Like (2) through (6) it is new policy text: the committed scores were computed against a corpus without SJRTD and say nothing about the new SJRTD cases, none of which has ever been scored live — its fifteen cases were renumbered on merge (ground-055..058, edge-087..093, fresh-024, ml-034, refuse-032, xagency-016, the FAX precedent). (8) 2026-08-14: AC Transit added as the twelfth agency, three documents (actransit-fares, actransit-fares-es, actransit-discounts), rebinding corpus_version once more to 5db0d6a4bfbd (33 documents, 214 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without AC Transit and say nothing about the new actransit-* cases, none of which has ever been scored live; the same PR retargets refuse-021 at Golden Empire Transit, since its old premise (AC Transit outside the corpus) no longer holds. (9) 2026-08-14: WestCAT added as the thirteenth agency, four documents (westcat-fares-all, westcat-transfers, westcat-clipper, westcat-buying), rebinding corpus_version once more to 8b119819d9c8 (37 documents, 227 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without WestCAT and say nothing about the new WestCAT cases (ground-047..050, edge-075..081, fresh-022, ml-030..031, xagency-013..014, refuse-030), none of which has ever been scored live. (10) 2026-08-14: SLO RTA (SLORTA) added as the fourteenth agency, four documents (slorta-fares, slorta-discounts, slorta-passes, slorta-contactless), rebinding corpus_version once more to b21d4e3b3c98 (41 documents, 249 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SLO RTA and say nothing about the new SLO RTA cases, which were renumbered on merge (ground-059..062, edge-094..100, fresh-025..026, ml-035, refuse-033, xagency-017, the FAX precedent) and none of which has ever been scored live. (11) 2026-08-14: VTA added as the fifteenth agency, three documents (vta-fares, vta-regional-transfers, vta-rtc-card), rebinding corpus_version once more to cbc07c922784 (44 documents, 264 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without VTA and say nothing about the new vta-* cases, none of which has ever been scored live. (12) 2026-08-14: Napa Valley Vine Transit (VINE) added as the sixteenth agency, two documents (vine-fares, vine-go), rebinding corpus_version once more to 09f9c297deba (46 documents, 270 chunks); the same PR records the Golden Gate Transit NO-GO (fetchable but not ingestible — its ASP.NET page-wrapper form cleans to zero sections) in the manifest header. Like the others it is new policy text: the committed scores were computed against a corpus without the Vine and say nothing about the new VINE cases, none of which has ever been scored live. (13) 2026-08-14: SamTrans added as the seventeenth agency, three documents (samtrans-fares, samtrans-fare-types, samtrans-clipper), rebinding corpus_version once more to cbfe81efcf3f (49 documents, 279 chunks). Like the others it is new policy text: the committed scores were computed against a corpus without SamTrans and say nothing about the new samtrans-* cases, none of which has ever been scored live. (14) 2026-08-14: Marin Transit added as the eighteenth agency, three documents (marin-fares, marin-clipper, marin-31day-transition), rebinding corpus_version once more to 3dd8b7bd757e (52 documents, 301 chunks), which is what HEAD now ships. Like the others it is new policy text: the committed scores were computed against a corpus without Marin Transit and say nothing about the new marin-* cases, none of which has ever been scored live. Regenerate all three artifacts at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "EVALS.md",
      "field": "prompt_versions.system",
      "reason": "2026-08-13: prompts/system.txt moved v10 -> v11. The scope sentence named five agencies and instructed the model to decline \"an agency, place, or service outside the five agencies above\", while the corpus had grown to nine. Every gate was green throughout, because nothing compared the prompt to the manifest. The consequence is quiet: a rider asks about a corpus agency, retrieval finds the passages, and the model declines anyway. v11 names all nine and replaces the count with \"the agencies listed above\" so the sentence cannot go stale again; tests/test_prompt_agencies.py now fails the build if the manifest and the prompt disagree, or if a count is reintroduced. The committed scores were produced under v10 and therefore under the narrower scope rule. They remain valid for the five agencies v10 covered and say nothing about how the model behaves toward E-tran, SCMTD, SolTrans or FAX, which v10 told it to refuse. 2026-08-13/14: v11 -> v12 adds County Connection (CCCTA) as the tenth named agency, in the same PR that adds its corpus documents, because tests/test_prompt_agencies.py now fails a corpus agency the prompt does not name. v12 is marked NOT YET LIVE-VALIDATED in its header; the committed scores were produced under v10 and say nothing about CCCTA behavior. 2026-08-14: v12 -> v13 adds San Joaquin RTD (SJRTD) as the eleventh named agency on the same terms; v13 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SJRTD behavior. 2026-08-14: v13 -> v14 adds AC Transit as the twelfth named agency on the same terms; v14 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about AC Transit behavior. 2026-08-14: v14 -> v15 adds WestCAT as the thirteenth named agency on the same terms; v15 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about WestCAT behavior. 2026-08-14: v15 -> v16 adds SLO RTA as the fourteenth named agency on the same terms; v16 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SLO RTA behavior. 2026-08-14: v16 -> v17 adds VTA as the fifteenth named agency on the same terms; v17 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VTA behavior. 2026-08-14: v17 -> v18 adds Napa Valley Vine Transit (VINE) as the sixteenth named agency on the same terms; v18 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VINE behavior. 2026-08-14: v18 -> v19 adds SamTrans as the seventeenth named agency on the same terms; v19 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SamTrans behavior. 2026-08-14: v19 -> v20 adds Marin Transit as the eighteenth named agency on the same terms; v20 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about Marin Transit behavior. 2026-08-16: v20 -> v21 is not an agency addition. A live smoke run that day (answer claude-haiku-4-5, judge claude-sonnet-4-6) had the assistant tell a rider asking about MST's senior discount that they could \"obtain an MST Courtesy Card by completing an application (available in English or Spanish) and presenting proof of age\" — a route mst-fares publishes for veterans (proof of service) and for riders with disabilities (physician verification or a Medicare card), and for no one else. v21 binds every application, card and proof detail to the rider class the passages attach it to, and drops the worked example \"apply for an MST Courtesy Card\" that had been teaching the model to do exactly this. Re-scored live on refuse-025 the same day: the groundedness judge passed the v21 answer, which instead states the boarding proof MST does publish for a senior and says the published policy does not specify how a senior obtains the card. The committed scores predate all of this and describe the v10 behavior. v21 was then live-validated on the whole 26-case smoke suite the same day: 26/26 with judges running, against 24/26 under v20 that morning, whose two failures were refuse-025 and fresh-001. That is the first live evidence against the header's own NOT-YET-LIVE-VALIDATED label, and it is partial: smoke covers MST, SBMTD, Yolobus and SacRT, so the eighteen-agency scope claim is untested and Marin Transit's cases have still never been scored live. Regenerate at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "baseline.json",
      "field": "prompt_versions.system",
      "reason": "2026-08-13: prompts/system.txt moved v10 -> v11. The scope sentence named five agencies and instructed the model to decline \"an agency, place, or service outside the five agencies above\", while the corpus had grown to nine. Every gate was green throughout, because nothing compared the prompt to the manifest. The consequence is quiet: a rider asks about a corpus agency, retrieval finds the passages, and the model declines anyway. v11 names all nine and replaces the count with \"the agencies listed above\" so the sentence cannot go stale again; tests/test_prompt_agencies.py now fails the build if the manifest and the prompt disagree, or if a count is reintroduced. The committed scores were produced under v10 and therefore under the narrower scope rule. They remain valid for the five agencies v10 covered and say nothing about how the model behaves toward E-tran, SCMTD, SolTrans or FAX, which v10 told it to refuse. 2026-08-13/14: v11 -> v12 adds County Connection (CCCTA) as the tenth named agency, in the same PR that adds its corpus documents, because tests/test_prompt_agencies.py now fails a corpus agency the prompt does not name. v12 is marked NOT YET LIVE-VALIDATED in its header; the committed scores were produced under v10 and say nothing about CCCTA behavior. 2026-08-14: v12 -> v13 adds San Joaquin RTD (SJRTD) as the eleventh named agency on the same terms; v13 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SJRTD behavior. 2026-08-14: v13 -> v14 adds AC Transit as the twelfth named agency on the same terms; v14 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about AC Transit behavior. 2026-08-14: v14 -> v15 adds WestCAT as the thirteenth named agency on the same terms; v15 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about WestCAT behavior. 2026-08-14: v15 -> v16 adds SLO RTA as the fourteenth named agency on the same terms; v16 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SLO RTA behavior. 2026-08-14: v16 -> v17 adds VTA as the fifteenth named agency on the same terms; v17 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VTA behavior. 2026-08-14: v17 -> v18 adds Napa Valley Vine Transit (VINE) as the sixteenth named agency on the same terms; v18 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VINE behavior. 2026-08-14: v18 -> v19 adds SamTrans as the seventeenth named agency on the same terms; v19 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SamTrans behavior. 2026-08-14: v19 -> v20 adds Marin Transit as the eighteenth named agency on the same terms; v20 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about Marin Transit behavior. 2026-08-16: v20 -> v21 is not an agency addition. A live smoke run that day (answer claude-haiku-4-5, judge claude-sonnet-4-6) had the assistant tell a rider asking about MST's senior discount that they could \"obtain an MST Courtesy Card by completing an application (available in English or Spanish) and presenting proof of age\" — a route mst-fares publishes for veterans (proof of service) and for riders with disabilities (physician verification or a Medicare card), and for no one else. v21 binds every application, card and proof detail to the rider class the passages attach it to, and drops the worked example \"apply for an MST Courtesy Card\" that had been teaching the model to do exactly this. Re-scored live on refuse-025 the same day: the groundedness judge passed the v21 answer, which instead states the boarding proof MST does publish for a senior and says the published policy does not specify how a senior obtains the card. The committed scores predate all of this and describe the v10 behavior. v21 was then live-validated on the whole 26-case smoke suite the same day: 26/26 with judges running, against 24/26 under v20 that morning, whose two failures were refuse-025 and fresh-001. That is the first live evidence against the header's own NOT-YET-LIVE-VALIDATED label, and it is partial: smoke covers MST, SBMTD, Yolobus and SacRT, so the eighteen-agency scope claim is untested and Marin Transit's cases have still never been scored live. Regenerate at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "golden.jsonl",
      "field": "prompt_versions.system",
      "reason": "2026-08-13: prompts/system.txt moved v10 -> v11. The scope sentence named five agencies and instructed the model to decline \"an agency, place, or service outside the five agencies above\", while the corpus had grown to nine. Every gate was green throughout, because nothing compared the prompt to the manifest. The consequence is quiet: a rider asks about a corpus agency, retrieval finds the passages, and the model declines anyway. v11 names all nine and replaces the count with \"the agencies listed above\" so the sentence cannot go stale again; tests/test_prompt_agencies.py now fails the build if the manifest and the prompt disagree, or if a count is reintroduced. The committed scores were produced under v10 and therefore under the narrower scope rule. They remain valid for the five agencies v10 covered and say nothing about how the model behaves toward E-tran, SCMTD, SolTrans or FAX, which v10 told it to refuse. 2026-08-13/14: v11 -> v12 adds County Connection (CCCTA) as the tenth named agency, in the same PR that adds its corpus documents, because tests/test_prompt_agencies.py now fails a corpus agency the prompt does not name. v12 is marked NOT YET LIVE-VALIDATED in its header; the committed scores were produced under v10 and say nothing about CCCTA behavior. 2026-08-14: v12 -> v13 adds San Joaquin RTD (SJRTD) as the eleventh named agency on the same terms; v13 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SJRTD behavior. 2026-08-14: v13 -> v14 adds AC Transit as the twelfth named agency on the same terms; v14 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about AC Transit behavior. 2026-08-14: v14 -> v15 adds WestCAT as the thirteenth named agency on the same terms; v15 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about WestCAT behavior. 2026-08-14: v15 -> v16 adds SLO RTA as the fourteenth named agency on the same terms; v16 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SLO RTA behavior. 2026-08-14: v16 -> v17 adds VTA as the fifteenth named agency on the same terms; v17 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VTA behavior. 2026-08-14: v17 -> v18 adds Napa Valley Vine Transit (VINE) as the sixteenth named agency on the same terms; v18 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about VINE behavior. 2026-08-14: v18 -> v19 adds SamTrans as the seventeenth named agency on the same terms; v19 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about SamTrans behavior. 2026-08-14: v19 -> v20 adds Marin Transit as the eighteenth named agency on the same terms; v20 is marked NOT YET LIVE-VALIDATED in its header, and the committed scores say nothing about Marin Transit behavior. 2026-08-16: v20 -> v21 is not an agency addition. A live smoke run that day (answer claude-haiku-4-5, judge claude-sonnet-4-6) had the assistant tell a rider asking about MST's senior discount that they could \"obtain an MST Courtesy Card by completing an application (available in English or Spanish) and presenting proof of age\" — a route mst-fares publishes for veterans (proof of service) and for riders with disabilities (physician verification or a Medicare card), and for no one else. v21 binds every application, card and proof detail to the rider class the passages attach it to, and drops the worked example \"apply for an MST Courtesy Card\" that had been teaching the model to do exactly this. Re-scored live on refuse-025 the same day: the groundedness judge passed the v21 answer, which instead states the boarding proof MST does publish for a senior and says the published policy does not specify how a senior obtains the card. The committed scores predate all of this and describe the v10 behavior. v21 was then live-validated on the whole 26-case smoke suite the same day: 26/26 with judges running, against 24/26 under v20 that morning, whose two failures were refuse-025 and fresh-001. That is the first live evidence against the header's own NOT-YET-LIVE-VALIDATED label, and it is partial: smoke covers MST, SBMTD, Yolobus and SacRT, so the eighteen-agency scope claim is untested and Marin Transit's cases have still never been scored live. Regenerate at the next promoted live eval run and remove these entries."
    },
    {
      "artifact": "EVALS.md",
      "field": "prompt_versions.judge_groundedness",
      "reason": "2026-08-16: prompts/judge_groundedness.txt moved v2 -> v3, and evals/judges._passages_block now renders each passage with the same \"(source: <url>, fetched <date>)\" header that assistant.answer._format_passages has always given the answer model. The two suites had been scoring the same sentence in opposite directions: prompts/system.txt rule 4 obliges every answer to disclose its snapshot date and evals/suites/freshness.yaml fails an answer that does not, while the groundedness judge was shown passage text with the dates cut out and failed fresh-001 for a claim no passage stated. v3 tells the judge to score provenance claims against those header lines, and to treat provenance as support for identity and dates only, never for a price, age, document, program rule or deadline. Verified live 2026-08-16 on three calls: fresh-001 now passes; an answer claiming a fetch date of 2026-08-14 against passages fetched 2026-06-12 is still failed; and an invented $0.75 senior fare with an age-60 threshold and a birth-certificate requirement is still failed. The committed scores were produced under v2 and under the narrower rendering. Regenerate at the next promoted live eval run and remove this entry."
    },
    {
      "artifact": "baseline.json",
      "field": "prompt_versions.judge_groundedness",
      "reason": "2026-08-16: prompts/judge_groundedness.txt moved v2 -> v3, and evals/judges._passages_block now renders each passage with the same \"(source: <url>, fetched <date>)\" header that assistant.answer._format_passages has always given the answer model. The two suites had been scoring the same sentence in opposite directions: prompts/system.txt rule 4 obliges every answer to disclose its snapshot date and evals/suites/freshness.yaml fails an answer that does not, while the groundedness judge was shown passage text with the dates cut out and failed fresh-001 for a claim no passage stated. v3 tells the judge to score provenance claims against those header lines, and to treat provenance as support for identity and dates only, never for a price, age, document, program rule or deadline. Verified live 2026-08-16 on three calls: fresh-001 now passes; an answer claiming a fetch date of 2026-08-14 against passages fetched 2026-06-12 is still failed; and an invented $0.75 senior fare with an age-60 threshold and a birth-certificate requirement is still failed. The committed scores were produced under v2 and under the narrower rendering. Regenerate at the next promoted live eval run and remove this entry."
    }
  ]
}