Tags: ghzhost/Soup
Tags
feat(reward): soup reward stress — adversarial verifier gameability p… …robe (v0.71.41) Turn the reward-hacking detector on the verifier itself: feed empty / length-padded / repetition / sentinel-spam completions and flag any the verifier accepts. Loads via the existing load_reward_fn (probes a synth .py or a builtin); a gold-requiring target with no --references is a hard error, never a false "robust". Exit 0=robust / 2=gameable / 1=error. Pure, offline, no schema change, no new deps. Also corrects the ops-docs Telemetry section (the sender exists but is wired to nothing — no data is sent). Telemetry flywheel deferred pending a public privacy policy. Tests: 16490 -> 16529 (+39). 5 sequential ECC reviews, every finding fixed.
feat(reward): soup reward synth — synthesize a deterministic reward v… …erifier (v0.71.40) Point `soup reward synth <refs.jsonl> -o reward.py` at reference (gold) outputs and it infers a deterministic verifier (numeric / json_schema / regex / tool_call), emits a readable, committable .py reward_fn that rides load_reward_fn's existing .py path (no new exec surface), and — the moat — REFUSES to emit one that can't discriminate its references from auto-perturbed negatives via a mandatory calibration report (accept refs >=90% AND reject negatives; hard floor at discrimination<=0). Nothing in TRL/Unsloth/Axolotl synthesizes a reward. Fixes MakazhanAlpamys#311: a comma-separated reward_fn ("accuracy,format") now loads as a reward ensemble (GRPOTrainer reward_funcs=[...], unlocks the rm_ensemble detector), GRPO-only and validated at config-parse; the deepseek-v3-reasoning recipe that shipped this previously crashed with "Unknown reward function". Riders: reward_fn field-validator (null-byte/blank/oversize/empty-comma-segment); comma-aware verifiable-domain check; envs/calculator + guess_number docstrings corrected. 5 sequential ECC reviews, every finding fixed (python HIGH PPO gate; code 2xHIGH per-tool arg binding + json_schema mixed-shape refuse; security HIGH rel_hint codegen injection; tdd 8xHIGH). Live smoke on RTX 3050: synth from envs/calculator agrees with math_verify; degenerate refused (exit 2); real GRPO on SmolLM2-135M with reward_fn=accuracy,format completed optimizer steps. +103 tests (tests/test_v07140.py); 16387 -> 16490.
feat(ship): close the evidence loop — emit-evidence + config + proven… …ance + PR comment (v0.71.39) soup ship's verdict is now emittable, committable, reviewable, and provenance-bound so a fine-tuning gate runs on every PR: - --emit-evidence: verdict_to_evidence re-serialises scores into the --evidence INPUT schema (output replayable as input, MakazhanAlpamys#312) - ShipConfig under eval.ship + --config: committable gate policy read with CLI > config > default precedence (Click ParameterSource) - --push owner/repo#N: verdict as a GitHub PR comment (best-effort, reuses adapter_pr.post_pr_comment; never flips the SHIP/DON'T-SHIP exit code) - provenance/staleness: --config+--emit-evidence STAMPS config_sha (eval.ship excluded from the recipe hash) + base_model + data_sha; --config+--evidence GATES, refusing drifted/absent config_sha (exit 3) - soup ci init --config binds the generated gate to the committed config Security: config_sha shape-gated before echo (ESC hygiene); data_sha via O_NOFOLLOW + symlink guard + 8 GiB cap; ci path rejects '#' + NEL/LS/PS (YAML plain-scalar run: truncation). 16330 -> 16387 tests (+57 tests/test_v07139.py).
feat(ship): make soup ship's leg-2 regression gate real (v0.71.38) soup ship's leg 2 — the catastrophic-forgetting / regression gate that carries the whole SHIP / DON'T-SHIP claim — was 15 trivia prompts scored by raw substring containment (it credited "B" for "Berlin", "3" for "13") with zero coverage for tool-calling, safety, or JSON. This makes the gate real. - forgetting.py: score_answer/extract_mcq_letter replace the substring scorer with answer-extraction (cue -> paren -> clause-terminating bare letter) + boundary-aware token match. MINI_BENCHMARKS expanded (mmlu 26 / common_sense 24 / instruction 24) + new mini_arithmetic (36) so a 1-item flip trips 0.05. BREAKING: an existing run's verdict can change (the old gate under-reported). - eval/gate_suites.py (new): bundled offline general-suite registry, no torch. DEFAULT_GENERAL_SUITE = the 4 MCQ suites + 3 behavioural JSONL suites (mini_tool_call / mini_format_json / mini_safety) scored per-model-absolute by the pure custom/diagnose scorers. _fraction_passing isolates a per-item scorer exception (deep-JSON RecursionError scores as a failed item). - ship.py: leg-2 scores bundled suites offline (base+tuned) before routing any non-bundled name to lm-eval; default general suite = the full bundled set. Exit-code taxonomy: usage errors move 2 -> 3 so exit 2 means only DON'T-SHIP (a typo'd flag was previously indistinguishable from a caught regression). - diagnose/__init__: "Six" -> "Seven" probes + re-export all 7 score_* fns; removed the dead SUPPORTED_TASK_MODES "pairwise reserved" gate. - Bundled gate fixtures ship in the wheel via the pyproject artifacts glob. Every bundled item is original, hand-authored (no MMLU/GSM8K rows copied). Test count 16288 -> 16330 (+42 in tests/test_v07138.py).
fix(cli): quote install hints so `pip install soup-cli[extra]` works …
…on cmd.exe (v0.71.37)
Every printed and documented `pip install 'soup-cli[extra]'` was bash / zsh /
PowerShell syntax and failed on Windows cmd.exe:
ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at
the start of dependency specifier
cmd.exe has no single-quote quoting, so it passes the quotes to pip verbatim
and pip rejects the requirement. Nothing in Soup can fix that once the command
is typed -- pip and the shell own it, and Soup is not installed yet when the
README line runs -- so the fix is the spelling we print.
Migrated 147 sites across 67 files to `pip install "soup-cli[extra]"`:
- 64 in src/ (Rich console hints + plain ImportError text)
- 57 in README.md + docs/
- 22 in src/soup_cli/templates/*.yaml + examples/configs/*.yaml
- 3 in examples/README.md
Double quotes are the only spelling valid in every shell (cmd, PowerShell,
bash, zsh), which is why the repo already used `pip install -e ".[dev]"`.
Measured on Windows: single quotes fail ONLY on cmd; double quotes pass
everywhere; bare passes on Windows but zsh globs `[extra]` and fails.
Method note (the PR MakazhanAlpamys#247 class): the hints sit INSIDE double-quoted Python
string literals, so a blind ' -> " sed produces SyntaxError. A tokenize-based
rewriter escaped `\"` in DQUOTE tokens and left bare `"` in TRIPLE / COMMENT
tokens; every touched .py was compile-checked. The full suite (not ruff, not
compile-check) caught two rewriter blind spots: the real YAML templates under
src/soup_cli/templates/ (byte-identical drift test) and examples/README.md.
A regression test (tests/test_v07137.py) scans the package and every docs code
block for the single-quoted form; prose may still name it so a reader from an
older tutorial recognises the error.
Also bundles MakazhanAlpamys#315 (@Sanjays2402): eval-gate benchmark tasks now run via
ForgettingDetector instead of a helper that never existed. Closes MakazhanAlpamys#310.
Test count: 16283 -> 16288 (+4 in tests/test_v07137.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs: correct the test count to the measured 16283 The self-audit caught a wrong number before it shipped. I wrote "+253" early, before the ECC review fixes and the live-smoke fixes added more tests; the file actually carries 282 (281 passed + 1 POSIX-only skip). Verified against an authoritative full run (no pipe, so the exit code is pytest's own rather than tail's): 16154 passed, 127 skipped, 2 deselected -> 16283 collected PYTEST_EXIT=0, zero FAILED/ERROR lines and 16001 + 282 = 16283 reconciles exactly with the v0.71.35 baseline. CONTRIBUTING is the only committed file affected; CLAUDE.md, the history files and plan.md are gitignored and were corrected locally.
test(v0.71.35): CI-proof the CLI output assertions (ANSI + wrap toler…
…ant)
CI went red on `TestPushCardRider::test_push_has_card_option`: Rich colourises
flag names, so at CI's terminal `--card` renders as
`\x1b[1;36m-\x1b[0m\x1b[1;36m-card\x1b[0m` and a raw substring check misses it.
Same class as the v0.71.26 / v0.71.32 CI reds (15785 tests passed; only this
assertion failed).
Add a shared `_clean()` (ANSI-strip + whitespace-collapse) and route EVERY
CLI-output text assertion through it, not just the one that broke — multi-word
phrases ("cannot write card", "not found") would wrap at a narrow CI width and
fail next.
Verified by reproducing CI locally (COLUMNS=80 FORCE_COLOR=1): the assertion
fails with the fix reverted and passes with it restored.
docs(draft): v0.71.33 — soup draft, shipped as the honest measurement… … gate Version 0.71.32 -> 0.71.33. CHANGELOG (with a Known-limitations block), README What's New, docs/commands.md, docs/serving-and-export.md, CLAUDE.md (arch + CLI + counts + history roll), CONTRIBUTING counts. The live smoke changed the pitch. The plan promised '~1.5-2x faster to serve by distilling its own draft'. Measured on SmolLM2-360M-Instruct <- 135M-Instruct: the STOCK draft already scored 69.3% acceptance; distilling it gave 69.7% at 2 epochs and 69.3% at 10 -- no gain beyond noise. A small same-family draft is already at its capacity ceiling vs the target, and logit KD cannot buy capacity it does not have. Assisted decoding also measured 0.55-0.64x -- a net SLOWDOWN. So the speedup claim is withdrawn, not shipped. The feature ships as the honest gate: soup draft measure tells you whether speculative decoding is worth enabling BEFORE you ship it, and on this pair it correctly says no. That negative result is stated in the CHANGELOG, the README, the serving docs and CLAUDE.md, and the pre-existing unverified '2-3x faster' line in the speculative-decoding docs was tempered to match. Full suite: 15680 passed / 124 skipped (15806 collected). ruff clean.