Generated by sprout gate-inventory from docs/ROADMAP.md (FIX-02) — do not hand-edit.
Maps every ledger row to the mechanism its Measured by cell names, and mechanically
resolves that mechanism against Makefile / .github/workflows/*.yml / the repo tree.
An AUTO row that cannot be resolved is a declared-but-unenforced gate.
29 AUTO rows · 0 unresolved.
| Section | Metric | Gate | Measured by | Resolution |
|---|---|---|---|---|
Code quality and coverage — CODE-QUALITY-STANDARD |
Lint | AUTO | ruff format --check + ruff check src tests |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Type safety | AUTO | mypy (strict; py.typed shipped) |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Branch coverage | AUTO | pytest --cov=sprout --cov-fail-under=90 |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Layout | AUTO | uv build (packaging regression check, CQ-10) + tests/test_resources.py import |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
groundedness (every claim entailed by its cited passage) | AUTO | eval/suites/groundedness + judge entailment |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
safety (cite a toxicity ref, never certify "safe," route to vet/poison-control) | AUTO | eval/suites/safety + deny-list guard |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
refusal (out-of-scope, "just tell me it's fine," embedded injection) | AUTO | eval/suites/refusal |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
multilingual (ES preserves the facts + citations of its EN mirror) | AUTO | eval/suites/multilingual + judge equivalence |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
calibration (stated confidence tracks correctness) | AUTO | eval/suites/calibration (reliability diagram + ECE) |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Abstention enforced below threshold | AUTO | eval/suites/calibration invariant |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
EN/ES pass-rate parity | AUTO | eval/suites/multilingual bilingual slice |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Hallucination rate | AUTO | tests/test_rag.py citation-guard tests + eval/suites/groundedness |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge ↔ human agreement (deterministic judge, CI floor) | AUTO (gated 2026-07-08, P0-4) — probe set expanded 12 → 66 (within the 50–100 target) across all 16 corpus species, and the judge’s negation-only polarity guard gained has_antonym_conflict for antonym flips carrying no negation marker ("safe" vs "toxic"); measured on the combined set: agreement 0.955, κ 0.906, both clear threshold; the CI/make verify step passes --gate and fails the build on regression. This gates the reproducible offline judge as a coverage/polarity smoke-floor only — morphological synonyms and low-overlap paraphrase remain documented blind spots (3 known disagreements are committed, not hidden). See docs/audits/judge-calibration.md. |
sprout calibrate --gate on the dated probe set |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge ↔ human agreement (LLM judge, production gate) | not yet done — the LLM judge needs a live Anthropic credential and is deliberately never invoked in CI (see eval/llm_judge.py); calibrating and gating it (--judge llm --gate) before it backs any production judging decision is a separate, still-outstanding step, tracked here rather than conflated with the deterministic-judge CI floor above. |
sprout calibrate --judge llm |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge-calibration freshness | warn-only (wired 2026-07-05, P1-19) — re-labeled 2026-07-08 alongside the 12→66 expansion; sprout calibrate prints a warning past 30 days but does not yet fail the command (ties to P0-4: flips to a hard failure once the LLM judge is calibrated and --gate is enabled) |
labeled_date on eval/judge_probes.yaml, checked by sprout calibrate |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Fail-closed loader | AUTO | eval/dataset.py + runner.fail_closed |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Model card completeness | AUTO (wired 2026-07-05 — see P1-11; previously declared AUTO with no lint) | tests/test_model_card.py |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Card honesty / limits framing | REVIEW | owner review per release | — exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Red-team (OWASP LLM01–LLM10) | config committed 2026-07-08 (eval/redteam/promptfooconfig.yaml, eval/redteam/README.md) — covers OWASP LLM01–LLM10 against the live POST /api/chat pipeline in EN+ES; wired as an advisory, non-blocking redteam CI job (.github/workflows/ci.yml) that needs ANTHROPIC_API_KEY; gap now: the key is not yet provisioned as a repo secret and no run has completed, so it is not yet in ci-gate and "0 open critical findings" is not yet a measured number — the refusal/adversarial eval suite plus the manual dated red-team report (docs/audits/red-team-2026-06-22.md) remain the standing substitute until a run is observed clean and the job is promoted to blocking |
Promptfoo redteam on prompt/model PRs |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Garak (LLM vulnerability scanner) | N/A-with-reason — the offline deterministic default has no LLM to scan (extractive generation, no model in the loop); revisit when the Bedrock/Anthropic generator seam is activated in a production configuration. Added 2026-07-05 (previously unrecorded — AIEV-14). | — | — exempt (not an unconditional AUTO claim) |
Accessibility — ACCESSIBILITY-STANDARD |
Conformance level | AUTO | .github/workflows/ci.yml pa11y + lighthouse jobs (both ci-gate dependencies): axe + pa11y (merge-blocking) and Lighthouse accessibility (merge-blocking, threshold 0.95) on the reference question UI + HTML eval report; transcript view not yet built (see row below) |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Structural a11y check | AUTO | sprout a11y-check on web/dist/index.html + docs/audits/eval-report.html |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Non-chat alternate view | AUTO | tests/test_a11y_and_judge.py (render_transcript) |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Color independence | REVIEW | manual SR review (NVDA, VoiceOver) | — exempt (not an unconditional AUTO claim) |
Accessibility — ACCESSIBILITY-STANDARD |
ACR (VPAT 2.5 Rev 508) | REVIEW | docs/accessibility/ACR.md |
— exempt (not an unconditional AUTO claim) |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
App-security level | REVIEW | review-gate checklist | — exempt (not an unconditional AUTO claim) |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Dependency audit | AUTO | pip-audit (blocking in CI; never ` |
|
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Secret scanning | AUTO | gitleaks |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Static analysis | AUTO | semgrep + .github/workflows/codeql.yml |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Actions pinning | AUTO | zizmor workflow-SAST scan (.github/workflows/ci.yml) |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
SBOM | AUTO | .github/workflows/release.yml (Generate SBOM step; never ` |
|
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
PII in logs | AUTO (never N/A) | obs.py _ALLOWED_FIELDS + Semgrep/bandit |
— exempt (not an unconditional AUTO claim) |
Internationalization — INTERNATIONALIZATION-STANDARD |
EN/ES key + placeholder parity | AUTO (wired 2026-07-08 — see FIX-02; previously declared AUTO with no such diff implemented) | tests/test_i18n_parity.py |
✅ resolved |
Internationalization — INTERNATIONALIZATION-STANDARD |
EN/ES eval pass-rate parity | AUTO | eval/suites/multilingual (also in the AI ledger above) |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
First-token latency (offline) | AUTO (wired 2026-07-05 — previously declared AUTO with no test; see P1-10) | tests/test_latency.py |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Reproducibility | AUTO | tests/test_eval_suites.py (test_run_is_byte_identical) |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Versioning | AUTO (mechanism wired; never yet exercised — no tag has ever been cut, corrected 2026-07-05; see CHANGELOG.md) | .github/workflows/release.yml |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Publish | AUTO (wired; unexercised — same caveat) | .github/workflows/release.yml |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
CI parity | AUTO (wired 2026-07-08, closing ci-parity-no-mechanical-diff; the checker's first run also surfaced two real gaps — docs and the zizmor workflow-SAST scan were required by ci-gate but absent from make verify — now closed via new docs/workflow-lint prerequisites) |
sprout ci-parity-check (src/sprout/ci_parity.py, tests/test_ci_parity.py) mechanically diffs .github/workflows/ci.yml's required jobs against their Makefile target(s); run via make ci-parity-check (also a make verify prerequisite) and the ci-parity CI job (a ci-gate dependency) |
✅ resolved |