Every recorded eval run in evals/runs/, oldest first, on one page: per-suite pass rates, cost, duration, and each prompt-version bump. The run directories themselves are a local, gitignored archive; this rendered page (and the SVG) is the committed artifact. Regenerate with make report (or python -m evals.history).
Read within an instrument, never across. Mock/offline runs are scored by deterministic checks against a mock answer model; live runs call a real answer and judge model. They are different instruments and measure different things, so the chart and tables below group runs by instrument (mode + offline/live) and never plot mock and live scores on the same series. Smoke and full runs differ in sample size too. Only the trajectory within one instrument is a like-for-like comparison.
| Run (UTC) | Overall | edge_cases | freshness | groundedness | multilingual | refusal | conversation | cross_agency | Est USD | Duration |
|---|---|---|---|---|---|---|---|---|---|---|
2026-06-12T06:03:12+00:00 |
11.4% | 0.0% | 0.0% | 0.0% | 7.1% | 50.0% | — | — | n/a | 0.0s |
2026-06-12T06:07:25+00:00 |
11.4% | 0.0% | 0.0% | 0.0% | 7.1% | 50.0% | — | — | n/a | 0.0s |
| ↑ prompt bump: answer_user v1→v2, judge_helpfulness v1→v2, system v1→v4 | ||||||||||
2026-06-13T00:24:22+00:00 |
10.7% | 0.0% | 0.0% | 0.0% | 10.0% | 47.4% | — | — | n/a | 0.0s |
| ↑ prompt bump: answer_user v2→v3, system v4→v6 | ||||||||||
2026-06-30T17:18:26+00:00 |
9.1% | 0.0% | 0.0% | 0.0% | 9.5% | 47.4% | 0.0% | 0.0% | $0.0000 | 0.1s |
| ↑ prompt bump: answer_user v3→v4, system v6→v7 | ||||||||||
2026-06-30T17:27:28+00:00 |
8.7% | 0.0% | 0.0% | 0.0% | 9.1% | 47.4% | 0.0% | 0.0% | $0.0000 | 0.1s |
2026-06-30T17:28:31+00:00 |
8.7% | 0.0% | 0.0% | 0.0% | 9.1% | 47.4% | 0.0% | 0.0% | $0.0000 | 0.1s |
| Run (UTC) | Overall | edge_cases | freshness | groundedness | multilingual | refusal | Est USD | Duration |
|---|---|---|---|---|---|---|---|---|
2026-06-12T06:07:34+00:00 |
8.0% | 0.0% | 0.0% | 0.0% | 0.0% | 40.0% | n/a | 0.0s |
2026-06-12T06:14:41+00:00 |
8.0% | 0.0% | 0.0% | 0.0% | 0.0% | 40.0% | n/a | 0.0s |
| Run (UTC) | Overall | edge_cases | freshness | groundedness | multilingual | refusal | conversation | cross_agency | sensitivity | stretch_tagalog | Est USD | Duration |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
2026-06-12T06:39:41+00:00 |
72.9% | 83.3% | 100.0% | 87.5% | 14.3% | 85.7% | — | — | — | — | n/a | 494.7s |
| ↑ prompt bump: system v1→v2 | ||||||||||||
2026-06-12T06:53:08+00:00 |
84.3% | 83.3% | 87.5% | 93.8% | 71.4% | 85.7% | — | — | — | — | n/a | 488.2s |
2026-06-12T07:06:48+00:00 |
92.9% | 88.9% | 100.0% | 100.0% | 92.9% | 85.7% | — | — | — | — | n/a | 534.6s |
| ↑ prompt bump: system v2→v3 | ||||||||||||
2026-06-12T07:17:04+00:00 |
92.9% | 88.9% | 100.0% | 100.0% | 92.9% | 85.7% | — | — | — | — | n/a | 482.0s |
| ↑ prompt bump: system v3→v4 | ||||||||||||
2026-06-13T00:01:47+00:00 |
92.9% | 100.0% | 87.5% | 100.0% | 78.6% | 92.9% | — | — | — | — | n/a | 480.5s |
| ↑ prompt bump: answer_user v1→v2, judge_helpfulness v1→v2 | ||||||||||||
2026-06-13T00:11:12+00:00 |
95.7% | 94.4% | 87.5% | 100.0% | 92.9% | 100.0% | — | — | — | — | n/a | 486.9s |
2026-06-13T00:36:42+00:00 |
92.2% | 92.9% | 90.0% | 92.3% | 90.0% | 94.7% | — | — | — | — | n/a | 728.7s |
2026-06-13T00:50:43+00:00 |
93.2% | 92.9% | 90.0% | 92.3% | 95.0% | 94.7% | — | — | — | — | n/a | 723.0s |
2026-06-17T00:04:04+00:00 |
94.2% | 96.4% | 90.0% | 92.3% | 95.0% | 94.7% | — | — | — | — | n/a | 797.5s |
| ↑ prompt bump: system v4→v5 | ||||||||||||
2026-06-17T00:46:39+00:00 |
93.2% | 100.0% | 100.0% | 92.3% | 85.0% | 89.5% | — | — | — | — | $1.4324 | 745.8s |
| ↑ prompt bump: system v5→v4 | ||||||||||||
2026-06-17T01:00:41+00:00 |
95.1% | 96.4% | 100.0% | 92.3% | 95.0% | 94.7% | — | — | — | — | $1.4538 | 772.3s |
2026-06-17T01:38:37+00:00 |
94.5% | 96.4% | 100.0% | 92.3% | 95.0% | 94.7% | 83.3% | — | — | — | $1.5837 | 839.3s |
2026-06-17T02:22:39+00:00 |
96.6% | 97.0% | 100.0% | 96.6% | 100.0% | 94.7% | 83.3% | — | — | — | $1.6843 | 898.0s |
2026-06-17T02:39:35+00:00 |
97.5% | 100.0% | 100.0% | 96.6% | 100.0% | 94.7% | 83.3% | — | — | — | $1.6869 | 906.4s |
| ↑ prompt bump: system v4→v5 | ||||||||||||
2026-06-17T03:11:14+00:00 |
94.9% | 100.0% | 100.0% | 93.1% | 95.2% | 94.7% | 66.7% | — | — | — | $1.6856 | 922.6s |
| ↑ prompt bump: answer_user v2→v3, system v5→v6 | ||||||||||||
2026-06-30T04:04:35+00:00 |
95.8% | 93.9% | 100.0% | 100.0% | 90.5% | 94.7% | 100.0% | — | — | — | $1.6877 | 838.2s |
2026-06-30T04:35:31+00:00 |
95.8% | 100.0% | 100.0% | 100.0% | 85.7% | 100.0% | 66.7% | — | — | — | $1.7021 | 850.3s |
| ↑ prompt bump: system v6→v7 | ||||||||||||
2026-07-11T21:20:11+00:00 |
94.5% | 97.0% | 100.0% | 100.0% | 95.2% | 92.3% | 66.7% | 66.7% | — | — | $1.7814 | 216.0s |
2026-07-11T21:24:20+00:00 |
94.5% | 100.0% | 90.0% | 96.6% | 95.2% | 96.2% | 66.7% | 66.7% | — | — | $1.7730 | 176.1s |
| ↑ prompt bump: answer_user v3→v4, judge_groundedness v1→v2, judge_helpfulness v2→v3, system v7→v9 | ||||||||||||
2026-07-11T21:52:44+00:00 |
73.1% | 81.2% | 70.0% | 93.1% | 81.8% | 85.3% | 60.0% | 66.7% | 63.3% | 0.0% | $3.1030 | 368.2s |
| ↑ prompt bump: answer_user v4→v6 | ||||||||||||
2026-07-11T22:03:46+00:00 |
80.1% | 85.4% | 80.0% | 100.0% | 100.0% | 94.1% | 60.0% | 66.7% | 70.0% | 0.0% | $3.1337 | 292.9s |
2026-07-11T22:10:48+00:00 |
79.6% | 85.4% | 100.0% | 96.6% | 90.9% | 91.2% | 70.0% | 66.7% | 70.0% | 0.0% | $3.1468 | 317.0s |
2026-07-11T23:03:23+00:00 |
84.6% | 87.5% | 100.0% | 100.0% | 90.9% | 91.2% | 70.0% | 66.7% | 73.3% | 46.7% | $3.1475 | 7.1s |
2026-07-11T23:05:36+00:00 |
86.6% | 87.5% | 100.0% | 100.0% | 90.9% | 91.2% | 70.0% | 66.7% | 73.3% | 73.3% | $3.1468 | 15.9s |
| ↑ prompt bump: answer_user v6→v7, system v9→v10 | ||||||||||||
2026-07-11T23:13:18+00:00 |
90.5% | 93.8% | 90.0% | 93.1% | 100.0% | 94.1% | 60.0% | 100.0% | 76.7% | 100.0% | $3.1241 | 302.6s |
| ↑ prompt bump: system v10→v11 | ||||||||||||
2026-07-12T04:27:28+00:00 |
94.5% | 87.5% | 100.0% | 100.0% | 95.5% | 97.1% | 70.0% | 100.0% | 100.0% | 100.0% | $3.1875 | 324.8s |
| ↑ prompt bump: system v11→v12 | ||||||||||||
2026-07-12T04:40:07+00:00 |
94.0% | 93.8% | 90.0% | 100.0% | 95.5% | 97.1% | 80.0% | 66.7% | 96.7% | 86.7% | $3.2106 | 312.3s |
| ↑ prompt bump: system v12→v10 | ||||||||||||
2026-07-12T05:00:18+00:00 |
95.5% | 95.8% | 90.0% | 93.1% | 100.0% | 100.0% | 80.0% | 100.0% | 93.3% | 100.0% | $3.1580 | 10.5s |
2026-07-12T05:01:17+00:00 |
95.5% | 95.8% | 90.0% | 93.1% | 100.0% | 100.0% | 80.0% | 100.0% | 93.3% | 100.0% | $3.1580 | 0.7s |
2026-07-12T07:08:10+00:00 |
90.0% | 93.8% | 90.0% | 89.7% | 86.4% | 100.0% | 70.0% | 100.0% | 76.7% | 100.0% | $3.1798 | 214.6s |
| Run (UTC) | Overall | freshness | Est USD | Duration |
|---|---|---|---|---|
2026-06-17T00:33:30+00:00 |
0.0% | 0.0% | $0.0000 | 0.0s |
| Run (UTC) | Overall | edge_cases | freshness | groundedness | multilingual | refusal | Est USD | Duration |
|---|---|---|---|---|---|---|---|---|
2026-06-17T01:18:01+00:00 |
96.0% | 100.0% | 100.0% | 100.0% | 83.3% | 100.0% | $0.3485 | 185.1s |
| Run (UTC) | Overall | conversation | Est USD | Duration |
|---|---|---|---|---|
2026-06-17T01:24:28+00:00 |
0.0% | 0.0% | $0.0000 | 0.0s |
| Run (UTC) | Overall | edge_cases | Est USD | Duration |
|---|---|---|---|---|
2026-06-30T04:10:13+00:00 |
93.9% | 93.9% | $0.5066 | 260.2s |
| ↑ prompt bump: answer_user v3→v2, system v6→v5 | ||||
2026-06-30T04:15:38+00:00 |
100.0% | 100.0% | $0.5062 | 263.2s |
| ↑ prompt bump: answer_user v2→v3, system v5→v6 | ||||
2026-06-30T04:20:57+00:00 |
100.0% | 100.0% | $0.5110 | 260.4s |
| Run (UTC) | Overall | edge_cases | Est USD | Duration |
|---|---|---|---|---|
2026-06-30T17:27:47+00:00 |
0.0% | 0.0% | $0.0000 | 0.0s |
| Run (UTC) | Overall | freshness | Est USD | Duration |
|---|---|---|---|---|
2026-07-11T21:54:41+00:00 |
100.0% | 100.0% | $0.1556 | 19.0s |
| ↑ prompt bump: answer_user v5→v6 | ||||
2026-07-11T22:05:26+00:00 |
100.0% | 100.0% | $0.1588 | 19.9s |
| Run (UTC) | Overall | multilingual | Est USD | Duration |
|---|---|---|---|---|
2026-07-11T21:55:32+00:00 |
81.8% | 81.8% | $0.4252 | 50.2s |
2026-07-11T21:57:24+00:00 |
86.4% | 86.4% | $0.4016 | 19.0s |
| ↑ prompt bump: answer_user v5→v6 | ||||
2026-07-11T21:58:43+00:00 |
100.0% | 100.0% | $0.4060 | 43.7s |
| Run (UTC) | Overall | stretch_tagalog | Est USD | Duration |
|---|---|---|---|---|
2026-07-11T23:07:14+00:00 |
80.0% | 80.0% | $0.2331 | 27.4s |
2026-07-11T23:08:04+00:00 |
100.0% | 100.0% | $0.2331 | 0.1s |