CHANGELOG.md

Changelog

All notable changes to Soup CLI are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Detailed, per-release notes for every published version live on the GitHub Releases page. This file tracks unreleased changes and links out for historical detail rather than reproducing 70+ versions of notes.

Unreleased

[0.74.0] - 2026-09-04

Added

• soup eval aider runs Aider's Polyglot code-editing benchmark through its official Docker harness (#91 by @Amix29 in #482). The command preflights Docker, the daemon, and the locally built benchmark image; mounts a prepared Polyglot corpus read-only; keeps output under cwd; and aggregates bounded per-exercise JSON into a Soup result row. --run-id records the score in eval_results for the existing run comparison workflow. The optional [aider] extra installs the normal Aider CLI while the docs make the source-only benchmark-image setup explicit.

• Native Apple Silicon telemetry for soup monitor (#99 by @Amix29 in #481). The monitor now reads bounded plist output from macOS powermetrics and renders GPU utilization and power in the existing Rich table. It reuses an explicitly cached sudo credential through non-interactive sudo -n, never reads a password, and gives an actionable Activity Monitor fallback when permission or telemetry is unavailable. NVIDIA-only VRAM, memory-utilization, and temperature fields remain unavailable rather than being guessed.

• soup mcp serve gains network transports: --transport sse and --transport http (#296 in #479). v1 was stdio-only, which suits a client that spawns Soup as a subprocess but leaves remote and multi-client setups with nothing. Both new transports serve the same registry — the end-to-end test compares the advertised tool names against build_registry rather than a hardcoded count, so a subset cannot creep in. stdio remains the default and is untouched.

  Adding a listener is the risky part, so it is gated three ways. Every HTTP request needs Authorization: Bearer <token>, compared with secrets.compare_digest, with no opt-out — a loopback port is reachable by every process on the box. The token is validated by the existing utils/qr_url.py::validate_token, so soup ui and soup mcp serve agree on what a token is instead of growing a second format, and it travels in the header only: no query-string fallback that could land in an access log. The SDK's DNS-rebinding protection is switched on (421 on a foreign Host, 403 on a foreign Origin), which is the gate the token cannot be — a page the operator merely visits sends no Authorization header but its request still reaches the port. Binding off loopback warns, and a wildcard bind warns again that the Host check has nothing left to pin.

  --allow-execute is refused with either network transport, and that refusal is the one behavioural change to an existing flag. Gated execution (#297) spawns real training / export processes; behind a listener a leaked Bearer token would mean process execution rather than plan disclosure, and stdio is a pipe to a client the operator already started, which is a different trust boundary. The refusal is made twice on purpose: once in the CLI so the operator gets a readable message, and once in build_asgi_app() so a direct caller cannot put an executing registry behind a listener either. The stdio banner also stopped claiming "execution disabled" -- that string predated #297 and was false from the moment execution shipped.

  --host / --port / --auth-token are refused under --transport stdio rather than silently ignored, and the ASGI app is built by a build_asgi_app() factory so the auth and rebinding behaviour is tested through an in-process transport without binding a socket; one further test binds a real ephemeral port and drives initialize -> list_tools -> call_tool through the SDK's own SSE client.

  The [mcp] extra floor moves 1.2.0 -> 1.10.0, measured against the published wheels rather than a changelog: server/streamable_http_manager.py first appears in 1.8.0 (absent in 1.7.0) and server/transport_security.py in 1.10.0 (absent in 1.9.4). --transport sse alone would have run on the old floor; rebinding protection would not, and a listener whose origin checking silently disappears on an older SDK is worse than a resolver error. The <2 cap is unchanged — #322 is the 2.x migration. No new package: mcp already requires starlette, uvicorn, sse-starlette and httpx-sse.

• soup data best-of-n can sample candidates from Ollama or vLLM providers (#299 by @Faisal01011 in #466). The existing local Transformers --base path stays the default, while --provider ollama|vllm --model <m> [--base-url <url>] draws each prompt's N candidates through the existing SSRF-validated raw- completion seam. Provider and model are recorded in _best_of_n provenance; Anthropic is refused by name because its Messages API has no raw-completion endpoint, and local-only flags cannot be silently ignored in provider mode.

• Baseline artifacts carry a scorer/version provenance stamp (#404 by @AchuthReddy-16 in #485). Shared helpers stamp_baseline_scores / write_baseline_file write {"scores": {...}, "provenance": {"soup_version", "scorer_revision"}}. soup eval gate --write-baseline <path> is the user-facing producer. resolve_baseline warns once on unknown provenance (unstamped files) or a scorer_revision mismatch, and stays silent when the stamp matches. BUNDLED_SCORER_REVISION + a locked fingerprint fail the suite if a bundled scorer's output moves without a revision bump. Registry save_eval_result stamps details_json; soup ship --emit-evidence includes the same stamp.

• Layer streaming now accepts Qwen3.5 MoE text checkpoints whose decoder layers do not expose exactly the same weight keys in every block (by @Shutaru in #426). The sharder now records per-layer shard headers instead of deriving the pool from layer 0 alone, while still refusing divergent storage layouts for any shared key and rebuilding NF4 Params4bit views from validated per-weight metadata. qwen3_5_moe / qwen3_5_moe_text route through the existing qwen3 streamer. A heterogeneous toy MoE is bit-exact streamed versus resident on CPU; live validation was also run on Qwen/Qwen3.5-35B-A3B with layer streaming, NF4, MoE LoRA target resolution and a 3072-token SFT dataset. stream_layers now refuses moe_expert_quant, which is applied only by the resident setup path and was otherwise silently ignored.

• live_eval.load_model_and_tokenizer gains a quantization parameter; no live evaluation path sets it yet (#367 by @AmirF194 in #461). quantization="4bit" builds the same nf4 BitsAndBytesConfig every other 4-bit load path in this codebase uses ("8bit" and the unset default are also supported), and the four internal callers this helper has (make_generator, make_multi_generator, lora_probe, measure_logit_agreement) still call it with no quantization, so today an NF4-trained adapter is still judged against a bf16 base it never saw during training. The follow-up is wiring those four callers to a default derived from the run's own configuration (soup ship --config, the registry entry, or the adapter's adapter_config.json), per the issue's own Fix path; that plus soup ship reporting the numerics it judged with and a staleness gate on mismatched-numerics evidence are left open (issue acceptance criteria 2 and 4).

• training.stream_pin makes layer-streaming pinning configurable (#366 by @ousamabenyounes in #416). Page-locking the RAM store is chosen automatically by decide_pinning, and until now nothing could override it — so while #331 was live, pin=False was the only known mitigation for silently wrong NF4 gradients yet was unreachable from soup.yaml. stream_pin: false now forces the pageable store (and the pre-flight states the throughput it costs, up to 6.56x measured, rather than absorbing it silently); stream_pin: true forces the pinned store and, on the RAM tier, refuses the run — naming the store size, not the ceiling (#366 AC3): pinned_limit_bytes is passed as None, so the page-lock ceiling is deliberately left unprobed and the store size is the only figure the refusal can honestly cite — if the box cannot page-lock it, instead of degrading silently; unset keeps today's automatic behaviour. On the disk tier (no RAM store to page-lock) and on CPU (no device to copy to) pinning is inapplicable rather than unsatisfiable, so true is announced and the run proceeds — refusing there would brick the large-model runs the disk tier exists for, and would make the key uncommittable to a config shared between a GPU box and a CPU box. Set while stream_layers: false it is rejected as a footgun, like the other stream keys.

• A ready-made qwen3.5-9b-grpo recipe for GRPO reasoning training with Qwen/Qwen3.5-9B (#277 by @harshitthek in #448). The recipe combines the established GRPO defaults (accuracy reward, beta=0.1, 4 generations) with LoRA r=16 and 4-bit quantization.

• A ready-made glm-5.1-dpo recipe for DPO preference training with zai-org/GLM-5.1 (#280 by @Osheun in #452). The first recipe pairing DPO with a MoE base, so it carries moe_lora: true alongside the established DPO defaults (beta=0.1, LoRA r=32 / alpha=64, 4-bit quantization). epochs: 1 and max_length: 8192 are taken from the SFT sibling rather than the smaller qwen defaults, which suit a 754B MoE better.

• Cross-tokenizer draft support for soup draft and soup serve (#304 by @CODING-DARSH in #417).

  • soup draft distill now accepts target/draft pairs with mismatched tokenizers or vocabularies, automatically routing through uld_strategy: wasserstein_aligned (Universal Logit Distillation) instead of refusing the pair.
  • soup draft measure supports cross-tokenizer acceptance measurement using decoded character-span alignment (count_accepted_spans) across different vocabularies and token boundaries. Target generation neutralizes only repetition_penalty with repetition_penalty=1.0 (Refs #345) so target greedy argmax and draft raw-logit scoring are evaluated consistently; remaining generation processors (no_repeat_ngram_size, encoder_repetition_penalty, min_new_tokens, bad_words_ids, suppress_tokens, and sequence_bias) are not altered.
  • soup serve --speculative-decoding supports cross-tokenizer draft serving via Transformers Universal Assisted Decoding (UAD) when supported by the installed transformers version, raising a clear error if unsupported.
  • Compatible same-tokenizer pairs strictly preserve the existing native fast path.

• data.interleave is now wired into training-time dataset loading (#443 by @blackcoderx in #460). parse_interleave/InterleaveSpec have been schema-validated and unit-tested since v0.42.0, but load_dataset() never called them — every multi-dataset mixture request silently trained on nothing but data.train's single path, the same gap #330 and #442 papered over in their respective renderers. DataConfig.train now accepts str | list[str]; a list of >= 2 local file paths combines via data.interleave (concat / under / over / {strategy: probs, probs: [...]}) into one row set before the existing val_split line in _finalize runs, so a single path stays byte-identical. interleave is local-files-only: training.packing / training.multipack and data.streaming / an HF-hub dataset name are all rejected at config-parse time with a message naming the reason (streaming/hub-dataset interleaving is follow-up #459). Both the soup data mix --optimize recipe writer and the --live overlay renderer emit the real N-dataset mixture again instead of collapsing to one path.

• A repo-wide documentation ratchet to guarantee declared recipe counts stay synchronized with the catalog (#453 by @harshitthek in #457). Derives the expected count dynamically from len(RECIPES) and scans all declared documentation sites, preventing silent Git auto-merge drift across sequential recipe additions.

• Branch coverage for lr_groups.py's build_optimizer_param_groups: the case where every parameter matches a configured group, so no base optimizer group is appended (#273 by @AmirF194 in #469).

• Branch coverage for replay.py's downsample: the case where the stride already lands on the last row, so the endpoint pin is not appended a second time (#273 by @AmirF194 in #470).

• Re-enabled /v1/tools/bash execution with OS-level namespace/sandbox isolation (#151 by @kok-o in #527). Note this is a breaking change: serve() now raises typer.Exit(code=2) when run with --host 0.0.0.0 without --tool-auth-token, whereas previously it only printed a warning.

• data.interleave now supports data.streaming: true and lists of HF-hub dataset names (#459 by @blackcoderx in #468). #443 left data.interleave local-files-only, refusing data.streaming and any remote-URI / hub-name list entry at parse time with a message naming this issue as the follow-up. Every data.train list entry is now classified once (local file / remote URI / HF-hub name) and dispatched: an all-local/remote list with data.streaming: true delegates combining to HF datasets.interleave_datasets / concatenate_datasets rather than reimplementing mixing over a source whose size can't be known ahead of time — concat maps to concatenate_datasets, under/over map to stopping_strategy="first_exhausted"/"all_exhausted", and probs maps to probabilities=probs, each chosen so the strategy names mean the same thing as the local path (verified by running one probs config through both paths and comparing the resulting proportions, not by two tests that each pass alone). An all-HF-hub-name list is loaded eagerly per entry and combined with the same _combine_interleaved the local path already uses; a hub entry's own validation split is honoured for the combined result only when every entry provides one, otherwise it's ignored (warned) and data.val_split applies to the combined train rows — a decided precedence rather than an emergent one. Still refused, by name: an all-hub list with data.streaming: true (streaming N differently-shaped hub datasets and reconciling their splits is a separate, larger effort), and any list mixing hub names with local/remote entries.

• LISA now accepts task: pretrain, not sft alone (#307 by @ousamabenyounes in #476). Continued pre-training is the same full-fine-tune-of-a-rotating-set-of-decoder-layers mechanism LISA was built for, so the sft-only gate (inherited from Spectrum's unfrozen_parameters) was arbitrary. The schema task gate now reads a _LISA_SUPPORTED_TASKS allow-list (sft, pretrain) and its refusal names every accepted task instead of only rejecting yours; trainer/pretrain.py replaces its LoRA path with LISA and attaches LisaCallback, and reports its parameter summary as LISA counted off the raw parameters, since a LISA run has no PeftModel wrapper to ask get_nb_trainable_parameters. Both trainers route through one peft_wiring.apply_lisa_setup, so they cannot drift on what "LISA is on" means. The rest of the gate is unchanged: transformers + text + quantization: none, mutually exclusive with the LoRA feature flags, freeze_layers/freeze_ratio and unfrozen_parameters. The released [0.71.34] block still says task: sft, which is what 0.71.34 shipped.

• A weekly dependency drift job, and a test that asks trl what it still accepts (#323 in #486). Two failure classes were structurally invisible until a PR happened to be open: a bug that only manifests on a CPU-only runner (a CUDA build never calls _convert_weight_packed_for_cpu, so no GPU dev box can reach it), and an upstream removal behind a floor-only pin. The second one shipped for several releases with CI green throughout — trl>=0.7.0 let CI resolve 0.29.1 while the dev box ran 0.19.1, and on 0.29.1 six trainers could not build their config at all, because the trl imports live inside setup() and no test had ever called it.

  .github/workflows/dependency-drift.yml runs the same resolve on a schedule: it installs the latest resolvable stack, runs the suite against it, and writes a resolved-vs-declared table into the run summary. It also flags a package declared with incompatible ranges in two extras — which it already found before merging: [mlx] asks for transformers>=5.0.0 while [train] caps it below 5, so pip install "soup-cli[train,mlx]" cannot resolve.

  tests/test_issue323_trl_kwarg_drift.py answers the setup() question without a model: it reads the keywords each wrapper passes to its trl config and asks the installed class whether it still accepts them, through the same config_accepts capability probe the wrappers use at runtime — never a version comparison, since a version table is what was wrong twice. The resolve_trl_symbol indirection is followed, so the three configs that moved to trl.experimental are covered rather than silently skipped. Two blind spots are stated rather than implied: **splat calls are invisible to a static read, and the trl TRAINER classes are excluded because they take much of their signature through **kwargs, which makes a signature check report model and train_dataset as rejected — measured, and the reason the scan filters on Config.

• soup mcp serve now runs on both mcp majors, and the <2 cap is lifted (#322 in #498). v0.72.3 capped the [mcp] extra the day mcp 2.0.0 broke every round-trip test; that unblocked a release and pinned anyone who wanted 2.x in the same environment. Both removals are bridged rather than pinned around: the @server.list_tools() / @server.call_tool() decorators became on_list_tools= / on_call_tool= constructor callbacks, and create_connected_server_and_client_session gave way to the lower-level create_client_server_memory_streams, which both majors still ship.

  build_server chooses by probing the Server constructor, never by reading mcp.__version__ — the rule trainer/_trl_compat.py earned after two wrong bounds derived from version tables, and a test walks the AST to enforce it. The dispatch logic stays in a single _dispatch_tool with two thin adapters, guarded by a test that fails if a second implementation appears, because two copies is how the majors would drift apart while both kept passing.

  The three modules the sse / streamable-http transports depend on all survive 2.0.0 unchanged, so those 46 tests needed no edit. Verified by running the MCP suites twice, once per major: 184 passed against 2.0.0, and the full suite against 1.29.0. The floor stays at the 1.10.0 measured in #296.

• Added a text-only qwen3.8-27b-sft catalog recipe for Qwen/Qwen3.8-27B (#477 by @Amix29 in #513). The recipe explicitly selects modality: text, uses the measured Qwen3.5-family decoder path from #507, and includes catalog, configuration, CLI, modality, and count-sync coverage.

• soup train --cloud lambda adds cloud GPU training plans for Lambda Cloud instances (Refs #264 by @kok-o in #528).

• Added 7 ready-made SFT recipes - Qwen2.5-Coder 1.5B/14B/32B (qwen2.5-coder-{1.5b,14b,32b}-sft), Qwen2.5-Math 1.5B/7B (qwen2.5-math-{1.5b,7b}-sft), and DeepSeek-R1-Distill-Qwen 1.5B/7B (deepseek-r1-distill-qwen-{1.5b,7b}-sft) - and corrected mistral-small-3-sft to point at the real hub repo (mistralai/Mistral-Small-24B-Instruct-2501). Catalog grows 147 -> 154 (#536 by @Nick-800)

• Added #550: a content-addressed two-phase Best-of-N workflow for exporting local candidates and materializing verified offline judgments without model or network access (#550 by @Amix29).

• Added 4 ready-made recipes - DeepSeek-R1-Distill-Llama-8B SFT (deepseek-r1-distill-llama-8b-sft) and three R1-Distill DPO variants (deepseek-r1-distill-qwen-{1.5b,7b}-dpo, deepseek-r1-distill-llama-8b-dpo) with pinned lr and dpo_beta. Catalog grows 154 -> 158 (#569 by @Nick-800)

• #572 adds a weight-free Qwen3.8-Flash-Next text-LoRA compatibility scaffold on Transformers 5.16.1, including architecture-aware linear targets, MoE detection, and a tiny-config forward/backward gate. Legacy int64-only Torch scatter runtimes receive an instance-local QSA index compatibility shim; real checkpoint, catalog, streaming, multimodal, and routed-expert-parameter support remain explicitly unclaimed (#572 by @Amix29).

• #575 adds opt-in PEFT target_parameters LoRA for Qwen4-Exp routed gate_up_proj and down_proj expert tensors in resident Transformers SFT and continued pretraining, with validated compatibility constraints and weight-free backward/save/reload coverage (#575 by @Amix29).

• #576 adds training.lisa_train_embeddings (default true = LISA as published in #267). Set it false to freeze LISA's always-on group — input embeddings, LM head, and final norm — so only the sampled lisa_num_layers decoder layers train. That always-on group is ~70% of everything LISA trains at 8B, so it is where LISA's memory actually goes; freezing it is a real quality/memory trade rather than a free win, which is why it is an opt-in knob. Setting it false while lisa_enabled is false is rejected, matching the other lisa_* fields. The analytical VRAM pre-flight still treats LISA as full fine-tuning regardless of this flag (it does not yet credit the frozen-embeddings saving, which needs a measured constant on GPU hardware), so a frozen-embeddings run that would fit can still be conservatively refused — use --allow-oom-attempt to launch it. Refs #377 (#576 by @Srinivasan8888).

• #582 adds a ready-made smollm3-3b-sft recipe for HuggingFaceTB/SmolLM3-3B. The catalog shipped SmolLM2 in three sizes but no SmolLM3; this fills that gap with a small/edge LoRA SFT recipe (r8, 8-bit, auto batch). Catalog count 158 → 159. Closes #271 (#582 by @Srinivasan8888).

• Add Qwen4-Exp layer-streamed SFT with sparse read-only PLE N-gram access for dense Transformers and oMLX/oQ affine checkpoints, including an explicit training.stream_ngram_source policy and fail-closed task/quantization/media gates (#602 by @Amix29 in #603).

• Added the model-free AutoDistill Milestone A artifact contract: versioned and immutable plan/capture/shard/consumption schemas, explicit top-k plus residual-tail semantics, transactional resume/corruption rules, and deterministic plan-only estimates (#580 by @Amix29 in #613).

• Added a ready-made kimi-k2.6-grpo recipe for moonshotai/Kimi-K2.6 (#281 by @umran666 in #614). The v0.71.24 model-family expansion shipped the SFT variant but no GRPO reasoning variant; this fills that gap with the MoE giant GRPO shape (grpo_beta: 0.1, num_generations: 4, reward_fn: accuracy, moe_lora: true, gradient_checkpointing: true, 4-bit, max_length: 8192), and keeps the Modified MIT licence note in the recipe description. Catalog count 160 -> 161.

• Added a ready-made qwen3.5-35b-a3b-dpo recipe for Qwen/Qwen3.5-35B-A3B (#276 by @Srinivasan8888 in #615). The catalog shipped the MoE SFT sibling qwen3.5-35b-a3b-sft in v0.71.24 but no preference-tuning variant; this pairs the qwen2.5-7b-dpo DPO shape (format: dpo, lr 5e-6, dpo_beta: 0.1) with the sibling's MoE settings (moe_lora: true, moe_aux_loss_coeff: 0.01), and carries an explicit modality: text so it joins the Qwen3.5-family contract in test_issue427_qwen35_text_modality.py rather than falling back to the schema default. Catalog count 159 → 160.

• Added the internal AutoDistill Milestone B1 same-tokenizer, teacher-only MLX capture and transactional shard publication boundary, with immutable input fingerprints, explicit corruption/resume checks, and proof that no student model is loaded during capture (#580 by @Amix29 in #629).

• Added a ready-made qwen3.5-9b-dpo recipe for Qwen/Qwen3.5-9B (#275 by @Srinivasan8888 in #632). The base already shipped SFT and GRPO variants; this completes the trio with the preference-alignment shape (task: dpo, format: dpo, dpo_beta: 0.1, lr: 5e-6, LoRA r16/a32, 4-bit, max_length: 4096), matching every other DPO recipe in the catalog. Catalog count 161 -> 162.

• Pinned every recipe's resolved SoupConfig against a committed snapshot (tests/fixtures/recipe_config_snapshots.json, regenerated via scripts/generate_recipe_snapshot.py), so a schema-default change no longer silently retunes recipes that rely on that default. Confirmed by mutation: changing dpo_beta's schema default failed 149 of the 162 recipes — every one that doesn't explicitly pin the field — which is the exposure this closes (#621 by @SID-6921 in #637).

• Added a ready-made glm-5.1-grpo recipe for zai-org/GLM-5.1 (#275 by @Srinivasan8888 in #656). The base already shipped SFT (v0.71.24) and DPO (#280 by @Osheun); this completes the trio with the reasoning shape (task: grpo, reasoning_train.jsonl, grpo_beta: 0.1, num_generations: 4, reward_fn: accuracy) over the 754B MoE geometry its siblings already use (LoRA r32/a64, batch_size: 1, gradient_accumulation_steps: 16, 4-bit, moe_lora, gradient_checkpointing, max_length: 8192) — deliberately not the 30B GRPO template's r16/a32. lr: 1e-5 is the value both conventions agree on: 15 of 22 GRPO recipes and glm-5.1-sft alike. The recipe is not trained — 754B is multi-node — so no hyperparameter here is a measured recommendation. Catalog count 162 -> 163.

Changed

• Remove the name-based SCORER_CHANGED_IN_V0_73_2 baseline warning in favour of the #404 scorer_revision stamp (#404 by @AchuthReddy-16 in #485). Stale baselines are detected by provenance, not by a hard-coded suite list.

• Remove hand-maintained test suite statistics from CONTRIBUTING.md in favor of a permanent digit-free shape invariant (#465 by @harshitthek in #467). Eliminates drift across routine test additions by making test-count divergence impossible at the documentation source.

• Lazy callback builders now self-import their callback class names so runtime lookup never raises NameError while preserving lazy heavy-dependency loading (#320 by @AchuthReddy-16 in #455). build_echo_trap_callback, build_reward_hack_callback, build_minillm_callback, build_rl_checkpoint_callback, and build_push_callback now resolve their callback types through local module imports in the builder body, and the regression suite adds subprocess coverage for all five builder calls.

• User-visible changes now use per-PR changelog fragments (#487 by @Amix29 in #490). Contributors add a uniquely named, version-scoped Markdown file instead of editing the shared [Unreleased] section. A standard-library assembler preserves long entries verbatim, rejects stale or malformed fragments, and consumes them during release preparation. A tag-time release gate refuses publication if any fragment remains, so the conflict is removed without making changelog loss silent.

• Qwen3.5 and Qwen3.6 text recipes now state their decoder-only intent (#427 by @Amix29 in #501). Apple Silicon measurements confirmed that Soup's text path loads Qwen3_5ForCausalLM without the visual tower, so all twelve catalog recipes now set modality: text explicitly and regression coverage prevents that decision from falling back to the schema default.

• Transformers training now supports Qwen3.5-family text decoders and can be installed together with MLX (#502 and #503 by @Amix29 in #507). The shared stack moves to Transformers 5.12.1+, TRL 0.29+, and PEFT 0.20+, with exact floor coverage, matching soup doctor diagnostics, and capability-based fallbacks for TRL APIs that moved under trl.experimental. Transformers LoRA target_modules: auto now covers both full- and linear-attention projections in Qwen3.5 text models without changing explicit targets or MLX defaults. The compatibility pass also updates removed Transformers arguments and keeps TRL 0.29's streamed DPO reference adapter off meta, so it remains a frozen adapter-sized snapshot instead of requiring a second model. DPO and ORPO also restore the removed prompt cap on TRL's prepared token ids, so data.max_length remains an effective sequence bound rather than a configuration-only value. The CLI's histogram and loss-curve renderers now support both plotext 5 and plotext 6, with a real Plotext 6 runtime pinned in the compatibility CI cell.

• Unknown config keys are now reported instead of silently dropped, and v0.75 will reject them (#627 by @Srinivasan8888 in #628). None of the config models overrode Pydantic's default extra="ignore", so a key the schema did not declare validated clean and was discarded: soup train --dry-run printed "Config valid. Ready to train!", the run exited 0, and the requested setting was never applied — training.quantizaton: none trained 4-bit quantized when full precision was what you asked for, training.gradient_checkpoint: true did no checkpointing, data.max_len: 512 truncated at 2048. #623 is the live case: training.stream_pin reached main two days after 0.73.3 shipped, a user on the released wheel wrote the documented escape hatch, and the resulting OOM was investigated as a layer-streaming defect. Loading a config now walks the whole model tree — data, training, training.lora and the rest, so a guard applied to one model and forgotten on another cannot look like it works — and reports every key it cannot place in one report per load, naming the field you probably meant (did you mean 'quantization'?). This is a warning, not a refusal: a config written against a newer Soup still runs on an older wheel, which is the case #623's user was in. From v0.75 the same config will fail to load — one minor of notice, since this release is the one that starts warning — and the warning names that version so the deadline is decidable rather than a permanent notice. The version is stated in one constant (config/unknown_keys.py) and asserted against the declared soup_cli.__version__ by a test, so the release that crosses the deadline turns a test red rather than leaving the message promising a rejection that already shipped.

• soup sweep now hard-errors on a --param that matches no config field, with no deadline (#627 by @Srinivasan8888 in #628). A different failure class from a dropped training key, so it is called out separately: a sweep whose swept knob is never applied produces arms that are all identical, and there is no partially-useful result to preserve by continuing. --param lora_rank=8,16 used to run the full grid at the base config's LoRA rank and report the winner; the grid is now checked before the first arm starts, and sweep parameter does not match any config field: unknown config key 'lora_rank' - not applied. is printed and the command exits 1. Anyone with a typo'd sweep parameter will see a new error where they previously got plausible, meaningless results, and a scripted sweep now fails instead of succeeding with a table of failed arms.

• The recipe-config snapshot fixture (#621/#637) is now delta-encoded — a change to a schema default that no recipe explicitly pins now produces one failure naming the shared baseline, not 149 identical failures across every recipe that relies on it. What's now structurally distinct is "a recipe's own value moved" (a named per-recipe test) versus "the shared defaults moved" (the one baseline test) — those are different tests now, not just different-looking diff text. Measured: tests/fixtures/recipe_config_snapshots.json goes from 1,571,967 bytes / 49,586 lines to 66,560 bytes / 2,040 lines — 23.6x smaller by byte count, 24.3x fewer lines.

  Traded, not eliminated, and worth stating at the size that actually occurs: a recipe that redundantly pins a value equal to the old default starts appearing in its own delta once the default moves out from under it, since what used to be a no-op pin just started doing something — that recipe's resolved config hasn't changed, only its relationship to the (now different) baseline has. For a rarely-pinned field (dpo_beta, 0.1 -> 0.2) that's 13 named recipes plus the baseline, 14 total, down from 149. For epochs (all 162 recipes declare it explicitly, distributed {1: 32, 2: 4, 3: 126}), moving the default 3 -> 1 moves 158: the 126 that pinned the old default plus the 32 that already pinned the new one — both now differ from a baseline that used to match one of them. The 4 recipes pinning 2 stay green, correctly, since neither the old nor the new default was ever their value. 158 named recipes plus the baseline — a wall of red on a change that alters no recipe's actual behavior. This is the acceptance criterion "changing a schema default fails, naming the affected recipes" holding exactly as specified, applied to recipes whose default-shaped pin is the thing that moved; it is not the "one failure" case, and the fragment previously understated it by only showing the favorable example.

  Verified by mutation against the real catalog: a new schema field produces exactly 1 failure (down from 164); a recipe's own value changing, a recipe added without regenerating, an empty or deleted fixture, and a redundant default-duplicating line being deleted (exposure 1, still deliberately unpinned per #621) all behave exactly as before (#638 by @SID-6921 in #640).

Fixed

• The twelve non-SFT trainers (dpo, kto, orpo, simpo, ipo, bco, online_dpo, grpo, ppo, pretrain, reward_model, embedding) loaded a frozen LoRA base as float32 regardless of the checkpoint's own dtype (#491 by @AmirF194 in #492). #471 fixed this for the SFT trainer; the same model_kwargs shape, missing the same key, was unchanged in the other twelve. None of them has a full fine-tuning branch, so every load there is a frozen base: the new shared resolve_frozen_base_load_dtype() keeps the checkpoint's own dtype (torch_dtype="auto"), except on a pre-Ampere CUDA card, where it now matches the float16 compute dtype those cards already use instead of leaving the base in bf16 storage.

• Layer streaming now keeps its host store on CPU on Apple Silicon (#434 by @Amix29 in #480). PyTorch 2.7+ can return an MPS tensor for torch.empty(device="cpu", pin_memory=True), while is_pinned() remains false. Direct runtime callers could therefore place the whole frozen base in the MPS allocator and still report a pinned RAM store, even though the normal soup train setup already disabled pinning outside CUDA. The runtime now disables both optional and required pinning when the target is MPS, and RamSource independently refuses any allocation that is not genuinely CPU memory (or claims pinning without being pinned). MPS proceeds experimentally with a pageable CPU source and MPS layer buffers; CUDA pinning behaviour is unchanged.

• CI and production load sites no longer assume Transformers dtype= (#478 by @AchuthReddy-16). dtype= on AutoModel*.from_pretrained / from_config is the

  =4.56 rename of torch_dtype=. Soup still declares transformers>=4.36.0,<5.0.0, but the 12-cell matrix only ever installed the newest 4.x, so a >=4.56-only kwarg stayed green. Call sites in chat / diff / infer / export / merge / serve / mole routing / layer-stream runtime now pass torch_dtype= (still accepted on current 4.57.x). A static AST guard fails if a production AutoModel* load/config site reintroduces dtype=. A new Ubuntu/3.11 transformers-floor job installs under .github/constraints/transformers-floor.txt using the lowest non-yanked resolvable Transformers version 4.46.1 with the lowest version in the declared TRL range 0.14.0 (transformers==4.36.0 is ResolutionImpossible against declared trl — the declared Transformers floor in pyproject.toml remains unchanged), runs pip check, asserts both pins, and runs the guard. The existing 12-cell matrix is untouched.

• downsample now returns at most max_points rows, which is what its docstring has always promised (#473 in #474). The stride was len(rows) // max_points — a divisor, not a cap — so five rows with max_points=2 came back with three, and the endpoint pin could add a fourth. Sample indices are now spread evenly across the series with both endpoints included, so soup runs replay renders exactly min(len(rows), max_points) points. Even spacing also avoids the cliff a ceil-based stride would introduce: a series one row over the cap keeps max_points points rather than roughly half of them. max_points=1 is the one shape where both endpoints cannot fit and returns the final row, which is what the endpoint pin existed to guarantee. The off-by-one guard added in #470 is kept, renamed to test_last_index_lands_on_last_row_no_duplicate and re-pinned to the new arithmetic. Consequence of the old behaviour was a chart with a few more points than intended, never a wrong number.

• cut_ce.py and liger.py now normalize path separators and match architecture keywords on the last path component only (#456 by @harshitthek in #458). On Windows, rsplit("/") never split on backslashes, causing parent directory names (e.g. C:\experiments\phi-3-runs\step-2000) to over-match architecture keywords during fallback detection when config resolution was unavailable. In liger.py, switching from whole-path to last-component matching also drops org-prefix false positives (e.g. mistralai/*, Qwen/* when the model name does not contain the keyword) and prevents parent directories from applying the wrong fused kernel across POSIX and Windows. Both modules now normalize dual separators and strip trailing slashes deterministically.

• measure_gemm_tflops now records per-repeat samples and test_takes_the_best_repeat_not_the_first verifies best-of-N selection within a single measurement (#444 by @harshitthek in #451). Comparing two separate probe calls taken at different moments caused intermittent test failures on developer GPU machines under background contention. GemmCeiling now preserves samples: tuple[float, ...] and the test asserts max(samples) selection deterministically.

• Duck-typed tokenizer mappings no longer raise a misleading error, and data_doctor shares the public coerce_token_ids helper (#441 by @AchuthReddy-16 in #447, part 2 found by @emre155). A dict-like output that is not registered as collections.abc.Mapping used to be iterated as keys (input_ids[0]='input_ids'), sending the operator looking at their data; the mask path skipped the same objects and silently dropped assistant_masks. Both gates now use one mapping-like predicate, and the helper is public so the two modules cannot drift.

• soup data mix --live handed every candidate proxy run a config it could not load (#442 by @blackcoderx in #445). _render_overlay_yaml emitted data.train as a YAML list, the same shape #330 fixed in the recipe writer, but every --live test mocked subprocess.run so nothing ever loaded the overlay through the schema — a config the tool could not itself load read as a passing feature. data.train now renders as the single highest-weighted dataset in each candidate, mirroring #330's fix, with a comment noting which dataset was picked and why.

• use_cut_ce silently did nothing for any model loaded from a local checkpoint directory, and conflated Phi-2 with Phi-3 (#383 by @AmirF194 in #446). apply_cut_ce() picked the CCE patcher by matching an architecture keyword against the model path's last component, so checkpoint-2000 / my-finetune / any other directory soup merge/soup shrink/soup train writes out matched nothing and CCE stayed off with no error, on a flag the user explicitly set. Separately, every Phi variant (phi-2, phi-3, phi-4) mapped to the same "phi3" patcher, even though cut_cross_entropy has no Phi-2 patcher at all (its config.model_type is "phi", not "phi3"), and a bare "gemma" fallback entry dispatched to a patcher cut_cross_entropy does not have at all, crashing instead of reporting unsupported. Detection now resolves AutoConfig.from_pretrained(model_name).model_type first, mirroring the identical fix already shipped for Liger Kernel (#78), and falls back to the name-based match only when that is unavailable; Phi-2 and plain Gemma-1 both correctly report unsupported instead of running under the wrong kernel or crashing. The two call sites that separately hand-wrote the "no matching architecture" advisory now share one message.

• soup draft measure now refuses a mismatched pair up front and no longer discards a completed measurement when the assisted arm fails (#344 by @ousamabenyounes in #409). measure gated on same_tokenizer() (tokenizer vocab + probe ids), which accepts a pair whose tokenizers are identical but whose config.vocab_size differs by padded embedding rows (e.g. Qwen2.5 large←small) — exactly the pair distill refuses. Transformers' assisted generation gates on config.vocab_size, so the run died with "different tokenizers" deep inside generate(), after the expensive load, and because the report was written only after that arm every completed acceptance/plain-throughput number was thrown away. measure now uses the same config.vocab_size precondition as distill (refusing before any model loads; the shared _vocab_size_of also reads a composite model's get_text_config(), so a multimodal target like Llava is no longer refused), keeps same_tokenizer() as an additional check, and writes the report incrementally so a failing assisted arm leaves the acceptance rate and plain throughput on disk. The report records an assisted_status (pending / complete / untimed / crash / interrupted) so a failed arm is distinguishable on disk from a completed or un-run one — pending is what a report keeps when the process dies mid-arm and no handler runs, which is the case the incremental write exists for.

• soup export --format gptq crashed with no calibration data and, when it did run, wrote a shard name the standard loader can't find (#338 by @AmirF194 in #475). With no --calibration-data, _export_gptq called model.quantize(tokenizer); auto-gptq's quantize() expects tokenized examples, not a bare tokenizer, so this failed with "object is not iterable". GPTQ export now requires --calibration-data up front and rejects a file with zero usable samples, since auto-gptq has no built-in fallback dataset (unlike AWQ). Separately, save_quantized writes its own gptq_model-<bits>bit-<group>g.safetensors shard, which AutoModelForCausalLM.from_pretrained does not look for; the exported directory now also carries a standard model.safetensors. The shared except ImportError blocks on both the AWQ and GPTQ paths also stopped reporting a fixed "not installed" string when the package itself imports fine but a transitive import inside it fails for an unrelated reason.

• SmolVLM/Idefics3 vision SFT now reaches real training batches (#302 by @Amix29 in #488). Soup keeps LLaVA messages and PIL images together until collation, converts legacy <image> markers to structured multimodal content, and lets the processor produce image-token expansion plus architecture-specific pixel tensors. The vision path uses the Transformers trainer with this collator so older supported TRL releases cannot pre-tokenize the dataset as text-only. Image placeholder ids are excluded from causal-LM labels, and the collator preserves a leading BOS whether it comes from the chat template or the tokenizer default.

• Pre-Ampere cards (T4/P100/V100/GTX 16xx — the whole free Colab/Kaggle tier) could crash stream_layers: true training with _amp_foreach_non_finite_check_ and_unscale_cuda not implemented for 'BFloat16': peft creates LoRA adapters in the base checkpoint's dtype while fp16 GradScaler requires fp32 gradients. Trainable *lora_* params are now cast to fp32 before optimizer creation from every trainer train() site through one shared helper (lora.r: 0, Spectrum and LISA full-FT paths are deliberately untouched so trainable memory does not double after the VRAM pre-flight). The #385 static scanner was narrowed, not weakened: modules that take the precision decision via the shared alignment helper now count as covered (#425, #429) (#429 by @lesterppo).

• SFTTrainerWrapper no longer silently upcasts every load to fp32 (#339 by @blackcoderx in #471). All three from_pretrained call sites (text/vision/audio) now pass an explicit torch_dtype: a frozen base (LoRA/QLoRA — the base never receives an optimizer step) preserves the checkpoint's own dtype via the shared resolve_frozen_base_load_dtype() (#491/#492) instead of defaulting to fp32 — pre-Ampere CUDA cards (T4/P100/V100/GTX 16xx/RTX 20xx) get an explicit torch.float16 override there instead of bf16-storage/fp16-compute. A trainable base (lora.r: 0, unfrozen_parameters, lisa_enabled — schema-gated to modality='text') loads torch.float32 as a deliberate, documented numerics choice. Measured on an H100 with Llama-3.1-8B, LoRA, frozen base: 48,241 MiB -> 18,658 MiB peak (2.59x / 28.9 GB), byte-identical across 3 repeats — the original #339/#471 benchmark, carried over unchanged in this revision rather than re-measured. The full-FT discriminator is a single shared is_full_finetune(), used by both SFTTrainerWrapper and commands/train.py's VRAM pre-flight classifier — previously independent copies that disagreed in both directions. setup()'s console summary label also now names LISA runs correctly instead of mislabeling them "LoRA applied".

• Every DeepSpeed-capable trainer now prunes the empty LoRA optimizer group, not just sft.py (#359 in #484). #336 fixed the failure where LoRA leaves HF's no-decay parameter group empty, DeepSpeed drops it, and the LR scheduler keeps two base_lrs until torch's strict zip raises at the first lr_scheduler.step() — but it fixed it in one wrapper. Measured before changing anything: 19 modules under soup_cli/trainer/ accept a deepspeed_config and exactly one called the guard, so 18 tasks still died the same way under --deepspeed with LoRA.

  Coverage is enforced by a scan over soup_cli/trainer/*.py rather than a list of names, following test_device_map_distributed.py — whose own history is the argument, since its first version parametrized over the six trainers that fix had touched and passed while nine more sites still carried the defect. The scanner requires the guard only where a module both accepts a deepspeed_config and constructs a trainer itself, so the delegating preference.py wrapper is correctly exempt, and it carries a control proving the pattern can fail.

  attach_empty_param_group_guard now declines a trainer with no callable create_optimizer instead of raising. That is load-bearing once the guard is attached from eighteen wrappers rather than one: not every TRL trainer exposes the method, and an AttributeError there would convert a DeepSpeed-only defect into a crash on the ordinary path.

• --deepspeed my.json is now resolved the way a preset is (#359 in #484). A user-supplied config reached DeepSpeed unresolved, so none of the preset rewrites applied to it. The decision recorded: resolve, but only keys that are provably invalid for the run. zero_hpz_partition_size is refused by DeepSpeed when the world size is not divisible by it, and the fp16 quantiser against a bf16 run raises expected mat1 and mat2 to have the same dtype inside deepspeed/runtime/zero/linear.py — neither is a preference. Since the documented way to customise ZeRO++ is to copy the preset JSON, which copies both defects, an unresolved user file inherited a crash the presets are already protected from. A config using none of those keys is returned by its own path, byte-identical; a repair is printed and written to a temp copy, and the user's file on disk is never modified. Malformed JSON passes through untouched, because DeepSpeed reports a bad config better than a traceback.

• Transformers floor CI now tracks declared dependency metadata (#494 by @Amix29 in #496). The compatibility guard derives the training lower bound from pyproject.toml, refuses a tested pin below it, requires a documented reason for a higher resolvable pin, and makes the workflow read exact versions from the constraints file instead of restating them.

• A server crash between spawning an MCP training job and recording its pid no longer lets a restart double-book a second job (#506 by @AmirF194). The one-active-execution cap now treats an unresolved launch as active rather than reading it as free capacity.

• TensorRT-LLM export now fails immediately with a clear message instead of silently producing zero artifact bytes (#337 by @AmirF194 in #508). _export_tensorrt() shelled out to python -m tensorrt_llm.commands.convert_checkpoint, a module absent from every current TensorRT-LLM release (tensorrt_llm.commands ships only bench/build/eval/prune/refit/serve; conversion now lives as a per-architecture examples/<arch>/convert_checkpoint.py script instead). The entry point is checked right after the existing tensorrt_llm availability check, before the LoRA merge and checkpoint directory are touched. Docs now warn that installing tensorrt_llm can downgrade a training environment's torch/transformers/numpy/datasets pins.

• Fixed layer streaming's hidden duplicate-disk cost: Soup now reuses regular Hugging Face cache files, preflights materialized-weight and shard-cache writes per volume, refuses before exhausting disk space, and explains source-fingerprint cache rebuilds (#510 by @Amix29).

• Layer streaming now admits the dense qwen3_5 / qwen3_5_text decoder used by Qwen3.8-27B after native mixed-attention resident-vs-streamed parity coverage. (#514 by @Amix29 in #515)

• Layer streaming now preserves BF16 checkpoints on capable MPS runtimes instead of silently doubling the store and disk cache to FP32. (#516 by @Amix29 in #519)

• The negative llama.cpp quantizer lookup test is now isolated from binaries installed on the host PATH. (#518 by @Amix29 in #520)

• Fixed fresh installs with Plotext 6: soup data stats histograms and soup runs loss charts now dispatch to Plotext 6's Figure API while retaining the Plotext 5 module-level path. The temporary <6.0.0 dependency cap is lifted because both supported majors are covered by the compatibility layer and regression tests (#507, #522).

• Added soup mcp runs reconcile --expunge-launching so operators can safely recover execution capacity from stale launching rows without editing SQLite by hand (#524 by @Amix29 in #525).

• Layer streaming now shards an untied embed_tokens and lm_head pair separately and reuses one vocabulary-sized device buffer instead of keeping both matrices resident (#526 by @Amix29). Tied embeddings keep their existing resident one-matrix path and numerics.

• SFT now refuses examples without causal loss targets and final training states containing non-finite metrics or parameters before saving artifacts (#535 by @Amix29).

• Streamed LoRA adapters now preserve the exact configured base-model provenance, restoring automatic base detection across adapter workflows (#537 by @Amix29).

• Fully cached Hugging Face snapshots can now be materialized and sharded with outgoing traffic disabled, while preserving commit and blob integrity (#538 by @Amix29).

• Apple internal NVMe volumes behind APFS and Apple Fabric are now detected for the layer-streaming disk tier without requiring a manual override (#539 by @Amix29).

• Response-only masking now supports system-first conversations with native chat templates that reject transient prefixes without a user query (#543 by @Amix29).

• The final training panel now preserves the last real loss, learning rate, and gradient norm when Transformers emits a summary-only log event (#544 by @Amix29).

• Reproducibility receipts now record the MPS backend, Apple chip name, and unified-memory capacity without collecting unique hardware identifiers (#545 by @Amix29).

• Best-of-N now rejects non-finite and boolean judge scores before selecting a winner or emitting invalid JSONL (#547 by @Amix29 in #551).

• Best-of-N now rejects malformed prompt rows with line-numbered errors and records each accepted row's source line instead of silently losing data (#548 by @Amix29 in #552).

• Fixed #549 by making Best-of-N generation durable and resumable per prompt, with exactly-once checkpoint validation and rollback-safe, manifest-last publication that never exposes a partial SFT/DPO generation after an output failure (#553 by @Amix29).

• Fixed #555: two-phase Best-of-N candidate export now resumes from durable, authenticated per-prompt checkpoints, while offline validation and output materialization stream through bounded-memory disk staging. Resume binds prompt source lines and exact local-model content, and streamed SFT/DPO publication rolls back as one set on failure. Candidate export now seeds each prompt independently so resumed runs reproduce the same candidates; this changes the output of existing --export-candidates --seed N runs (#559 by @Amix29).

• Fixed #556: offline Best-of-N now commits SFT and optional DPO outputs through a final verifiable manifest, rejects unsupported online recovery options, and makes interrupted or mismatched generations fail closed (#557 by @Amix29).

• _assert_finite_training_state no longer refuses a run over a self-corrected transient metric (#546 by @AmirF194 in #560). The log_history scan now checks only the most recently logged value of each metric instead of raising on the first non-finite value found at any step, so a GradScaler warm-up nan that recovers no longer blocks the final save.

• Live multipack training now packs each bin to training.batch_size * data.max_length instead of data.max_length alone, so the configured batch size actually affects packing density again (#562 by @AmirF194).

• #564 enables BF16 autocast for hardware-validated resident SFT, DPO, reward-model, and PRM training on capable Apple Silicon MPS runtimes. The live capability probe is shared with layer streaming; CPU and unvalidated MPS trainers remain FP32, while PRM retains FP32 master weights to avoid a fatal Metal optimizer dtype mismatch (#564 by @Amix29).

• Preserve reward metadata from local GRPO datasets and keep earlier assistant turns when separating a final reference answer from multi-turn conversations (#566 by @Amix29).

• #568 enables runtime-probed BF16 autocast for local Transformers GRPO/RLVR on capable Apple Silicon MPS runtimes. Unsupported MPS remains FP32, local generation avoids the CUDA-only vLLM path, and an Apple Silicon train/save/reload smoke covers deterministic RLVR with LoRA (#568 by @Amix29).

• live_eval's four callers now forward quantization; soup ship --config reuses the training run's own (#367 by @AmirF194 in #570). #461 added quantization to load_model_and_tokenizer but nothing set it, so an NF4-trained adapter was still judged against a bf16 base. make_generator, make_multi_generator, lora_probe and measure_logit_agreement now accept and forward it. soup ship --config soup.yaml derives a default from training.quantization (4bit/8bit; other quant_menu formats still fall back to full precision) and prints which precision it loaded, now matching what it actually passes to from_pretrained (previously the fallback message claimed bf16 without ever setting torch_dtype). advise --probe-model, diagnose --base-model and tunability --live take no --config/--quantization flag yet, so they keep the existing default. Stamping the numerics into the verdict/evidence JSON and a staleness gate on mismatched-numerics evidence (issue criteria 2 and 4) stay open.

• #574 corrects the MCP execution-tool refusal, which told operators to restart with --allow-execute and then falsely added that execution tools "are not implemented in this version" — execution shipped in v0.73.3 (#297 by @Srinivasan8888 in #574). The refusal now names the disabled tool and the flag that enables it, and the stale build_registry docstring no longer describes allow_execute as reserved for future tools. Closes #483.

• Eval-gate LLM judge normalization uses the active rubric scale instead of hardcoded / 10.0 (#577 by @here-2007 in #578). _run_judge_task in eval/gate.py divided the aggregate judge score by 10.0, assuming a 1–10 scale, while DEFAULT_RUBRIC in eval/judge.py uses 1–5. A perfect judge score of 5.0 normalized to 0.50 instead of 1.00, making typical gate thresholds (0.70, 0.80) impossible to satisfy and triggering false-positive training stops under on_regression: stop. Normalization now dynamically derives scale_min and scale_max from evaluator.rubric["scale"] via min-max scaling to [0.0, 1.0], clamps out-of-bounds values, handles degenerate scales (min == max), and aligns with the existing dynamic normalization in commands/ship.py.

• #581 ports two v0.73.0 vLLM serve fixes to the SGLang backend, which had the identical pair standing. The prompt now applies the model's own chat template via the shared build_chat_prompt (with the legacy User:/Assistant: fallback for template-less models) instead of a hand-rolled third copy, so the model no longer sees a format it was never trained on. And finish_reason reports "length" when a response hits max_tokens (on both the sync and streaming paths) instead of a hardcoded "stop", so a client doing continue-on-length can tell a truncated answer from a completed one. The tokenizer load in soup serve --backend sglang now uses the same trust_remote_code setting the SGLang runtime itself uses, so a custom-code model no longer falls back to the legacy prompt in silence, and the three-branch operator warning vLLM prints about the chat template is now printed here too. Live verification against a real SGLang runtime on Linux remains an open follow-up on #360 (#581 by @Srinivasan8888).

• soup train now warns when training.convergence_detection is enabled but not yet wired into the live training loop, matching the existing honesty guard for other advisory training-intelligence flags (#583).

• GRPO objective variants now execute without silently falling back to stock TRL loss (#584 by @here-2007 in #585). _GRPOTrainerVariant.compute_loss() previously checked inputs for per_token_logps, which is never pre-populated by TRL's rollout dataloader. The trainer now obtains per-token log probabilities in a single forward pass via _get_per_token_logps_and_entropies without duplicating forward computation or increasing peak VRAM, ensuring gspo, dapo, dr_grpo, bnpo, two_sided, and rft objectives train their intended loss formulations.

• PPO's reward model, and task: reward_model's own trained model, now only inherit the Quant Menu quantization config when training.quantize_reward_model is set (#586 by @AmirF194). Since v0.53.0 added that flag, it was validated by its own task-scoped check but never read by either loader, so both tasks quantized regardless of the flag's value.

• #588 makes FSDP + BNB 4-bit training resolve quantized storage and trainable adapter parameters to the same floating compute dtype, preventing the integer-storage and mixed-dtype flattening failures that made the llama3-70b-fsdp2 recipe unrunnable. The recipe now pins bf16 storage for its A100/H100 target hardware (#588 by @Faisal01011).

• Raised the accelerate floor to 0.27.0, the first release whose FSDP checkpoint save/load path can be restricted to the trainable adapter. This closes a remaining way a LoRA/PEFT run under FSDP could regress back to writing the full frozen base model into every checkpoint on an unpinned accelerate install (#352, #591) (#591 by @AmirF194).

• _telemetry_endpoint_is_safe could not reject private/loopback HTTPS endpoints because the upstream HTTPS requirement bypassed the scheme-conditional private-IP check in validate_hub_endpoint. The fix layers a telemetry-strict private, loopback, and link-local rejection on top of the existing hub sanitisation without modifying hubs.py (#593, #598) (#598 by @harshitthek).

• training.stream_vram_probe docs now name each ratio's denominator (#595 by @umran666 in #605). Four code sites quoted the probe series (0.992x / 0.830x) labelled "the real peak". They now carry BOTH series with their denominators: the real-run peak (0.934x at seq 5120, 0.787x at 6144) and the probe series (0.992x at seq 4096, 0.830x at 5120) — the gap between them is the probe running 12.5-14.3% above the real training step, the conservativeness the gate depends on. The field description also carried the withdrawn claim that the probe under-measures preference losses badly enough to make the gate unsafe — at the one matching shape it reads +13.5% high, the same safe direction it shows for SFT, and the task='sft' restriction stands because one shape is not a validation. Comments and field descriptions only; no behaviour change.

• soup serve --backend mii now applies the served model's own chat template and reports the engine's real finish_reason (#606 by @ARAVIND281 in #608), instead of a hand-rolled System:/User:/Assistant: prompt and a hardcoded "stop". The backend reuses build_chat_prompt and resolve_finish_reason rather than keeping its own copies, and serve loads and passes the tokenizer for the MII path as it already does for vLLM. Behaviour is unchanged for a model that ships no chat template.

• The reward-hack ladders' mode split is now documented, and their thresholds are pinned by tests (#371 fix-path item 2, by @umran666 in #611). The H100 gate record carried an open inconsistency — the beta ladder escalated on four consecutive HACK votes while the rollback ladder, requiring three, never fired in any arm. The code settles it: the beta ladder lives in kl_control (no rollback rung), the rollback rung lives only in pid_lagrangian, and the schema already rejects reward_hack_rollback under any other mode — so the arm that fired one structurally lacks the other, and every arm with the other crashed at #342 before it could run. Documented in the module docstring and both run paths; comments only, no behaviour change. Tests pin the remaining CPU-testable acceptance criteria: each ladder fires at its documented threshold on N consecutive HACK votes with a control at N-1, and the beta == floor degenerate case the defaults produce.

• soup doctor now recommends a PyTorch CUDA wheel that matches the driver's nvidia-smi CUDA version instead of always suggesting cu121. An unreadable driver header falls back to cu121 (conservative: a parse failure is treated as an old driver, not a new one). Drivers at CUDA 11.8 get cu118. Windows also notes that the PyPI torch wheel is CPU-only. (#612 by @MKnaomi2)

• #631 pins the last open row of #273: the 74->73 branch of setup_logging, which was only ever executed incidentally depending on ambient logger state (100% branch coverage on one machine, 98% on another). Two hermetic tests now force the state: a handler without the _soup_log_tier tag survives reconfiguration, and a Soup handler with a stale tier is still removed. Dropping the is not None guard fails both tests by name (mutation demonstrated and reverted). BrPart 1 -> 0, for a reason rather than by luck (#631 by @umran666).

• Three tests no longer fail on a clean checkout when FORCE_COLOR is set (#633 by @Srinivasan8888 in #635). soup recipes show and soup data mix --apply emit syntax-highlighted output, and three assertions read that output raw — so on a colour-capable terminal one of them fed ANSI escapes to a YAML parser (unacceptable character #x001b) and another looked for modality: text inside a highlighted span. The assertions now normalise through a shared strip_ansi helper, and a repo-wide AST guard fails on any new raw-output assertion against a syntax-highlighted command, so the fourth occurrence of this failure class is also the last one that has to be found by hand.

• --resume auto (and a direct --resume <path>) now finds MLX checkpoints. _resolve_checkpoint only recognized checkpoint-N directories (the transformers/unsloth shape); mlx-lm's tuner saves step-numbered NNNNNNN_adapters.safetensors files instead, so an MLX run's own output never matched and training silently restarted from scratch every time. MLXSFTTrainerWrapper.train() now loads the located checkpoint's adapter weights before training starts. This is a weights-only warm start, not a full resume — mlx-lm's LoRA trainer exposes no optimizer state or step count, so the step count and data position both restart from zero regardless of how far the checkpoint got, and the run says so. Also fixes MLX --resume for any config with experiment_name set, which the first version of this fix missed (#634 reported and diagnosed by @imahsanali, fixed by @SID-6921 in #639).

• [train] now declares the torch floor that actually binds, and doctor's copy is pinned to it (#636 by @umran666 in #641). [train] said torch>=2.3.0 while requiring transformers>=5.16.1, whose own torch extra forces torch>=2.5 — the declared floor could never bind, and soup doctor carried an unpinned second copy on top. The extra now declares torch>=2.5.0 with the reason stated beside it; doctor keeps a literal that tests/test_issue636_torch_floor.py pins to the declaration, and a drift guard compares the declared floor against the declared transformers floor's real metadata on the transformers-floor CI job (which installs exactly 5.16.1 under the constraints file), skipping with a reason anywhere the installed transformers is a different release rather than passing vacuously. Closes #636.

• soup sweep no longer crashes with an IndexError when the run grid is empty (#628 by @Srinivasan8888 in #643). The unknown-parameter precheck added in #628 reads combinations[0] to probe the grid, so a --max-runs value that leaves no combinations — --max-runs -1, for instance — raised IndexError: list index out of range from inside the guard rather than reaching the sweep. Nonsense input either way, but the previous behaviour was an empty results table and exit 0, and an unhandled traceback is a worse answer than that. The probe is skipped when the grid is empty, restoring the old response. Bounding --max-runs at ge=1 would reject the input outright and is the better long-term fix; it is a pre-existing CLI contract change and is left for its own change.

• stream_source: auto now falls back to the NVMe disk tier when the RAM store fits MemAvailable but the store plus resident extras would exceed Soup's physical-host safety ceiling, and forced stream_source: ram refuses that case during pre-flight instead of letting the kernel OOM-kill the process (#622 by @ousamabenyounes in #644).

• soup sweep --dry-run now validates instead of returning before the config loads — and forecloses a latent Python 3.10 crash in the sweep pre-flight (#642 by @umran666 in #645). --dry-run exits before load_config ever ran, so the #627 unknown-key warning and the #628 --param pre-flight were both unreachable under it: the issue's typo'd quantizaton printed a grid and exit 0 with no warning, while soup train --dry-run reported it. This is a stated behaviour change: --dry-run now means "validate, then print the grid without running anything", both paths share the single load, and the severity split is unchanged — warn-only exits 0, a typo'd sweep parameter exits 1 before printing a grid it can never run. On the execution path the same validation now happens before the confirmation prompt instead of after it. Making the pre-flight reachable exposed a latent crash in it: the probe walks a full model_dump(), which visits training.preference_loss_weights (dict[str, float]). On Python 3.10 a PEP-585 alias passes isinstance(c, type), and whether the subsequent issubclass raises TypeError depends on the pydantic release — ModelMetaclass inherits the raising check from ABCMeta through 2.10.x (verified: every real sweep died with issubclass() arg 1 must be a class on 3.10.11 + pydantic 2.10.6) and defines its own tolerant __subclasscheck__ by 2.13.4 (verified under the same interpreter), so the crash was live across the unmasked part of the supported pydantic>=2.0.0 range on the oldest supported Python, and a current pydantic masks it — which is also how #628 shipped green: its unit tests fed the walker hand-built dicts that never reached such a field. _nested_models now filters parameterized generics via typing.get_origin; real-command tests cover each branch and the clean-config control, and the filter has a deterministic killer (an ABCMeta stand-in recreating the pre-override metaclass, pinning the real schema field) plus a control so it cannot be "fixed" by never walking. Closes #642.

• Live training panel now labels the GPU figure as GPU peak: and reads max_memory_allocated() instead of memory_allocated() (#650 by @YuriPerro in #652). on_log fired between steps, after activations and gradients were freed, so the sampled memory_allocated() was the inter-step trough -- on the reproducing run 5.8/15.9 GB while nvidia-smi showed the device fully occupied. The fix uses max_memory_allocated(), the lifetime peak, and changes the label to GPU peak: so the figure is self-describing. reset_peak_memory_stats() is intentionally not called here: that reset is process-global and would clobber the grad-accum advisor's own peak reading at callback.py:566.

• The CUDA batch probe now refuses on a measured peak, not only on a raise (#649 by @YuriPerro in #654). Under the WDDM driver (native Windows, WSL2) the allocator spills to host memory instead of raising OutOfMemoryError, so make_cuda_probe_fn's only fit criterion, "the step did not throw", approved a batch 16 that ran an order of magnitude slower in shared memory and pick_batch_size cached it. The probe now reads max_memory_allocated after the synthetic step and refuses when it exceeds what this process can reach on the device (mem_get_info free plus what it already holds), mirroring the counter and basis decide_measured_fit documents for layer streaming; an OOM surfacing at a synchronize point as AcceleratorError / RuntimeError("out of memory") is classified as OOM instead of propagating as misconfiguration; and the cache key carries a version tag so entries written by the old probe are ignored. The WDDM case is covered by a mocked-torch test that needs no GPU, with a fitting control beside it. Closes #649.

Security

• Telemetry and webhook SSRF guards now reject abbreviated, decimal, hex, and octal IPv4 forms (#600 by @here-2007 in #604). _is_private_or_link_local() in utils/hubs.py, utils/hf.py, and utils/webhooks.py previously delegated exclusively to ipaddress.ip_address(), which raises ValueError on non-canonical IPv4 representations like 127.1, 2130706433, 0x7f000001, and 0177.0.0.1. On Linux (glibc), the OS resolver (getaddrinfo/inet_aton) parses these forms, allowing crafted --slack-url https://2852039166/ webhooks or telemetry overrides to reach internal network addresses and cloud metadata (169.254.169.254). All three guards now normalise non-standard IPv4 literals via in-process socket.inet_aton() and strip trailing FQDN dots before verifying against ipaddress.ip_address().

• The SGLang serve backend now obeys the --trust-remote-code gate instead of loading every model with it enabled (#360 by @Srinivasan8888 in #619). serve.py resolves the v0.36.0 default-deny gate once for every backend and passes the result to vLLM, but _serve_sglang was called without it and create_sglang_runtime hardcoded trust_remote_code=True at both sgl.Runtime call sites, so soup serve --backend sglang executed a model's custom repo code whether or not the user opted in. The warning panel said so, but a notice is not a gate. create_sglang_runtime now defaults to False like the vLLM path, and the resolved value reaches the runtime and the tokenizer load alike. Behaviour change: a custom-code model on the SGLang backend now fails to load without --trust-remote-code rather than silently running its code.

• Telemetry SSRF guard now applies a tiered architecture with primary allowlist, internal TLD rejection, and DNS defence in depth (#599 by @here-2007 in #624). _telemetry_endpoint_is_safe() previously only checked literal IP formats, allowing hostnames resolving to private addresses (10.0.0.1.nip.io, localtest.me) or internal TLDs (metadata.internal) to bypass validation. The guard now validates via three tiers: (1) a primary allowlist for canonical PostHog domains (*.posthog.com) with zero DNS lookups, (2) static string rejection of internal TLD suffixes (.local, .internal, .lan, .home, .corp, .intranet) and literal private IPs, and (3) defence-in-depth DNS resolution for custom non-default FQDNs that fails closed on resolver errors/timeouts and rejects private or loopback target IPs.

• The OTLP tracing endpoint validator now rejects abbreviated, decimal, hex, and octal IPv4 forms, closing a bypass #600/#604 never reached (#616 by @SID-6921 in #625). utils/tracing.py's _is_private_ip predated the #604 fix and was never folded into it: it had no socket.inet_aton fallback. The loopback spellings (https://127.1:4317 and friends) were permitted anyway; the reachable hole was non-loopback private ranges in alternate encodings — https://2852039166:4317 and https://0xa9fea9fe:4317 both resolve to 169.254.169.254, the cloud-metadata address, and both were accepted while the dotted-quad form was refused. Same for 10.0.0.5 as 167772165 / 0xa000005 / 012.0.0.5 / 10.5, and 192.168.1.1 as 3232235777. All refused now; 127.0.0.1, localhost and ordinary hostnames still pass. Found while consolidating the SSRF loopback/private-host predicate, which had drifted into three copies of the function and six of the host set across hf.py, hubs.py, webhooks.py, loop_stages.py, qr_url.py and tracing.py — issue #616 undercounted, and an AST walk during review found the real number. All six call sites now import one definition from the new utils/net_guard.py, and a guard test walks src/soup_cli/ for any function whose name ends in is_private_or_link_local or constant ending in LOOPBACK_HOSTS — matching the leading-underscore spelling every original copy used — so a reappearing duplicate fails the build. loop_stages._endpoint_is_local, a seventh copy of the parsing logic guarding the deploy canary, had the same abbreviated-IPv4 gap; it now shares net_guard.parse_ip_literal while keeping its own narrower policy, since adopting the wider predicate would change what that surface trusts as a local target rather than only where the logic lives. The consolidation is otherwise behaviour-preserving. The shared predicate's is_reserved / is_multicast clauses do not change any caller's outcome: in hf.py and hubs.py both branches raise, so the predicate selects an error message rather than an accept/reject decision, and qr_url.py imports only the host set. loop_stages becomes marginally more permissive — it is an allow-list gate, so teaching it parse_ip_literal makes it recognise 26 further spellings as local — but every one of those canonicalises to an address it already accepted, so the set of endpoints Soup will POST to is unchanged.

[0.73.3] - 2026-08-18

Added

• eval.ship.noise_floor is now committable to soup.yaml (#406 by @ousamabenyounes in #410). Every soup ship gate-policy flag was settable in a committed config and read back by --config — except --noise-floor (added in v0.73.2), which had no field and could only be passed on the command line, so a team enforcing a floor in CI had to hand-edit the workflow. ShipConfig gains a bounded noise_floor ([2, 10], imported from ship_verdict so the schema and the CLI validator cannot disagree; bool-as-int rejected), wired with the same CLI > config > default precedence as the other five flags. As a live-measurement input it is measured only when producing evidence and refused under --evidence exactly as the flag is; it follows forgetting_threshold in being excluded from the recipe config_sha, so setting a floor never invalidates evidence.

• A ready-made qwen3.5-4b-pretrain recipe for continued pre-training of Qwen/Qwen3.5-4B-Base (#278 by @Faisal01011 in #422). The recipe uses plaintext data, one epoch, QLoRA 4-bit quantization, and the established continued-pretraining defaults.

• A ready-made deepseek-v4-flash-grpo recipe for GRPO reasoning training with deepseek-ai/DeepSeek-V4-Flash (#279 by @Faisal01011 in #432). The recipe combines the established GRPO defaults with MoE LoRA and gradient checkpointing.

• soup mcp serve --allow-execute can now actually execute, behind a single-use confirmation token (#297 by @CODING-DARSH in #393). train_start and export issue a short-lived, server-generated token bound to the plan and to the execution kind; train_execute / export_execute accept only that token — no command, no argv, no shell string, no client-supplied environment — and the server launches the planned Soup CLI command with shell=False, stdin=DEVNULL, output to .soup/mcp-runs/<run_id>.log. Two integrity properties close the gap between planning and running: the config is snapshotted at plan time and executed from the copy, so the file cannot be edited underneath the run, and digest_file now walks a directory tree by content (sorted relative paths + per-file hash, symlink refusal, bounded) rather than by mtime+size, which did not change when a file inside a protected directory was rewritten — so a model could be swapped between plan and execution and revalidation still passed. Token consumption and capacity acquisition happen before Popen, so a failed spawn requires a fresh plan rather than enabling replay. Runs go through the existing ExperimentTracker, so soup runs sees what MCP started.

Changed

• The MLX SFT dispatch route no longer imports the Transformers SFT wrapper before choosing a backend (#394 by @Shutaru in #431). The wrapper is import-light today, but the standalone soup-cli[mlx] runtime no longer depends on it remaining so. An additive Apple Silicon CI job verifies that mlx and mlx-lm import, the PyTorch/TRL training stack is absent, and a one-step real CLI SFT run completes. This hardens the runtime boundary; it does not claim to resolve the still-unpinned torch-present hang reported in #394.
• soup card now links the ML-BOM and the in-toto/SLSA attestation — they are first-class registry artifact kinds (#309 by @ousamabenyounes in #420). bom and attestation were the two compliance documents soup card could not surface: RegistryStore had no such kinds, so there was no way to attach them. Added both to the valid-kind set and a --attach-to-registry <id> flag to soup bom emit and soup attest emit (mirroring the existing --attach-to-registry pattern); once attached they appear in the card's artifact table for free. Signed attestations also attach the detached .sig sidecar. The flag needs --output (omitting it exits 2), and a requested registry attachment failure exits 1 after preserving files already emitted.

Fixed

• Assistant-only masking no longer mistakes BatchEncoding keys for token ids (#430 by @Shutaru in #439). Tokenizer mapping outputs are read through input_ids, and tensor-like ids are normalised to Python integers before they reach the collator; missing or non-integer ids now fail loudly instead of building a garbage label mask. A template that returns an all-zero assistant mask while assistant messages exist now falls back to incremental rendering, avoiding a silent all--100 no-op training run. The same mapping assumption was removed from the data doctor.

• soup data mix --optimize wrote a recipe soup train could not load (#330 by @blackcoderx in #440). render_mix_recipe_yaml emitted data.train as a YAML list of every searched dataset, but DataConfig.train is typed str, so the recipe failed to load with data -> train: Input should be a valid string. data.train now renders as the single highest-weighted dataset from the search; the full ranked weight/path breakdown is kept as a comment above it so no information is lost, and the comment explains that data.interleave is not yet consumed by training.

• Layer streaming now verifies that every trainable LoRA parameter has real storage after PEFT attaches the adapter (#433 reported by @lesterppo, fixed by @Faisal01011 in #435 and #437). PEFT 0.18 creates streamed adapters on meta for Soup to materialise, while PEFT 0.19 may create them as real tensors immediately, so materialize_meta_adapters() returning 0 cannot distinguish a healthy no-op from a missed adapter. The streamed build now enforces the actual postcondition and refuses to install the runtime if a trainable lora_* parameter remains on meta, naming the stranded parameter instead of allowing a silent no-training run.

• Windows process liveness misread a process that exited with code 259 as still running, forever (#424 by @blackcoderx in #436). GetExitCodeProcess returns STILL_ACTIVE (259) for a genuinely running process, but 259 is also a legal exit code, so a child that exited with 259 was indistinguishable from one still running. That silently defeated ExperimentTracker's reconcile-on-read (#401) and could wedge the MCP server's one-active-execution cap (#402) shut, refusing every subsequent execution with no error an operator could act on. The check now waits on the process handle with WaitForSingleObject(handle, 0), which is signalled the instant the process exits regardless of its exit code, falling back to the exit-code read only if the wait itself fails. The two byte-identical copies of this primitive in experiment/tracker.py and mcp_server/execution.py are now one shared utils/process_liveness.py.

• soup train --no-reexec now prints the flags you actually typed (#372 by @AchuthReddy-16 in #415). The advisory accelerate launch command is derived from the same argv the auto-reexec would have used, so --fsdp / --deepspeed / --config (and the rest of the run-shaping tail, including --name and --replay) cannot silently fall off the printed hint. Following that line used to train without FSDP.

• detect_device() and get_gpu_info() now recognise Apple Silicon MLX (#423 by @harshitthek in #428). Previously on Apple Silicon, detect_device() only probed PyTorch MPS and fell back to 'cpu', triggering a false Warning: 4bit quantization is not supported on CPU alert and silently downgrading quantization from 4bit to none. Device and GPU info detection now accept an optional backend= parameter: passing backend="mlx" resolves to 'mlx' with the chip name (e.g. Apple Silicon (Apple M2 Max)), preserves 4-bit quantization intact for mlx-lm pre-quantized models, reports Apple unified memory in telemetry, and skips the CUDA-shaped analytical VRAM preflight on MLX runs. The preflight skip is pinned by TestHardwareFitGateIsMlxAware (mirrors the streaming-aware gate test). The known-limitation warning in docs/backends-and-ops.md is replaced with the resolved behaviour.

• A run whose watcher died was reported running forever (#401 by @ousamabenyounes in #407). ExecutionManager._watch runs as a daemon=True thread, so when the MCP server process exits it is killed without unwinding and finish_execution never runs — ExperimentTracker kept the run at running with no watcher left to correct it, training the operator to ignore the one status field that guards against a second concurrent run. The tracker now reconciles on read: when a running row carries a pid whose process is gone, list_runs / get_run rewrite it to terminated with an unknown (None) exit code — an unknown outcome is never recorded as success, and the richer completed/failed terminal statuses are left untouched. soup runs no longer hardcodes running for a non-running row.

• Layer streaming now accepts a fast virtio/cloud disk instead of refusing it as an HDD (#365 by @ousamabenyounes in #411). detect_disk_kind trusted /sys/block/<dev>/queue/rotational, which a paravirtual (virtio) block device defaults to 1 with no media hint — so a genuinely NVMe-backed cloud disk (measured 1.5 GB/s read) was classified hdd and denied the disk-overflow tier, the very audience the tier targets. rotational=0 stays authoritative (solid state); when the flag is unreliable (rotational=1), the media type is now decided by a bounded, O_DIRECT sequential-read measurement rather than the flag — NVMe-class throughput (>= 1 GB/s) earns the tier, a genuinely slow disk still classifies hdd and is still refused (160 seeks/step, plan P11). A new training.stream_disk_kind override (nvme/ssd/hdd) is the escape hatch for the case where even that is wrong; it prints what it overrode beside what was detected. Disk detection may write a small scratch file next to the streamed shards to run the probe.

• The one-active-execution cap could double-book after a server restart (#402 by @ousamabenyounes in #408). ExecutionManager._active_run_id is in-memory, so a restarted MCP server started with an empty slot, saw free capacity, and would launch a second training while a child from the previous server (which survives a client disconnect, #297) was still using the GPU. The cap is now gated on a persisted run whose recorded pid is still alive: a live prior child blocks a new execution across restarts, while a stale record whose process is gone frees the slot rather than wedging it shut. docs/commands.md now states the actual scope of the cap.

• soup ship --noise-floor now measures the leg-1 task axis in the judge modes, so a judge-scored win smaller than the judge's own noise no longer counts (#403 by @ousamabenyounes in #419). The floor was measured in --task-mode metric only; judge_score / pairwise printed a warning and left leg 1 at a 0.0 floor, i.e. the exact blindness the flag exists to remove. It is now measured: judge_score scores the base side N times through the judge, and pairwise judges the base model against itself (expected win-rate 0.5, so the spread is directly measured, not inferred). Because those repeats fold in the judge's own sampling noise, the floor is labelled decode + judge on the panel and stamped judge_inclusive in the evidence/JSON block so it is never read as a decode-only number. --help states the extra judge-API cost.

• training.bnb_4bit_use_double_quant was validated but never read — every 4-bit path Soup builds the BitsAndBytesConfig for hardcoded double-quantization to True, so bnb_4bit_use_double_quant: false passed validation and was silently ignored (#321 by @ousamabenyounes in #418). The flag is now threaded through the three call sites Soup owns — the resident loader (build_quantization_config_for_loader), the layer-streaming path (stream_setup reads it once and passes the SAME value to the sharder and the meta skeleton, so streamed-vs-resident bit-exactness cannot drift), and the 4bit save path (soup merge --no-double-quant for the 4bit / 4bit_forced save formats). The Unsloth loader is out of scope: FastLanguageModel builds its own BitsAndBytesConfig internally with double-quant hardcoded on and exposes no override, so that path cannot honour the flag. The schema field is now tri-state (Optional[bool], unset = None): unset resolves to the shipped default (double-quant on), so a run that never set the flag trains identically, but because the resolved config now carries the field, soup ship --evidence provenance and soup lock check will report a one-time fingerprint drift on a previously-unset config even though the model numerics are unchanged. The config-load footgun (=true requires quantization: 4bit) fires only on an explicit true, and unset serializes as None, so a dumped-and-reloaded config no longer trips it.

• soup env check now flags an installed package that violates Soup's own declared version bound (#368 by @ousamabenyounes in #421). pip install "soup-cli[serve-fast]" (vllm) into a training venv silently pushes transformers past the <5.0.0 cap Soup declares, producing an environment Soup's own metadata says is unsupported with no warning at any point. env check audits installed versions against the bounds read from package metadata — not a hardcoded copy, since a second copy of the cap is exactly the drift this catches — and exits 3 when one is violated, independent of any lock file. An extra == "X" requirement is Soup's bound only when [X] was opted into, which the running environment cannot confirm, so its marker is evaluated and it is skipped rather than raised as a false positive — except for the ABI-relevant TRACKED_PACKAGES set, which since the v0.71.0 deps-split lives in metadata only under extra == "train"/"all"/ "dev" and therefore includes the transformers <5.0.0 case #368 was reported about. A package is counted once even when its bound is restated under several extras. If packaging is somehow unimportable the audit still degrades to a clean report so env check keeps working, but now says so rather than reporting a silent clean it never verified. The bounds audit no longer short-circuits the lock diagnostic (both print). packaging is a declared dependency now — the audit degraded to a silent "clean" without it, the wrong direction for a checker. docs/serving-and-export.md states the separate-environment guidance for [serve-fast].

• kl_control rewrote the trainer's β/kl_coef on every step, including a hold, so a non-acting run was numerically identical to log_only (#371 by @AmirF194 in #414). _run_bang_bang called _apply_coefficient unconditionally; on a hold the controller writes back the value already there, which is a no-op value but not a no-op write. The write is now skipped when the bang-bang step holds, and the mitigation log records mitigation_status (held / acted / released) so a non-acting step is distinguishable from an acting one without parsing the free-text action reason.

• extract_mcq_letter scored zero for \boxed {A} — whitespace between the command and the brace (follow-up to #357, by @ousamabenyounes in #396). The shipped \boxed\{ regex tolerates spaces inside the braces but not between \boxed and {; LaTeX permits it there and models emit it, so \boxed { C } still read as no answer and was not rescued by the cue tier either. The boxed-letter regex now allows \s* after the command.

• soup draft distill --steps N now delivers ~N optimiser steps instead of N/4.44 (#364 by @ousamabenyounes in #399). The epoch count that realised --steps divided the request by rows // batch_size, ignoring that val_split (0.1) removes rows from training and gradient_accumulation_steps (4) micro-batches make one optimiser step — both divide the budget, so every distill run trained for roughly a fifth of the requested steps, silently. Epochs are now derived from the effective steps-per-epoch, the emitted config pins the run shape (val_split, gradient_accumulation_steps) so the arithmetic and the trainer cannot drift, and the pre-flight prints the resolved step count.

• MitigationLogWriter silently dropped every record once its parent directory vanished mid-run (#343 by @ousamabenyounes in #398). record() reopens the log per call and swallowed the OSError from open("ab") with a bare return, so when a shared temp root was cleaned by another process the controller kept acting while its log quietly stopped growing — the run completes while its evidence goes missing, the failure shape this project treats as the worst kind. The writer now recreates the directory and retries the write, and surfaces the loss once via a warning naming the path. record() still never raises, so a vanished log never takes down the training run.

[0.73.2] - 2026-08-15

soup ship's leg 2 is the project's differentiator, and it was lying in both directions: two of its suites ranked by the wrong thing, one whole failure direction had no detector at all, and a caller error was indistinguishable from a regression. Every item below was reproduced on the dev box against the shipped v0.73.1 code before a line was changed.

Added

• soup ship --noise-floor N — measure what the instrument can resolve before calling a delta significant. Greedy decoding is not deterministic on GPU: measured on an H100, the same model with no adapter over five runs spread 0.015 strict / 0.020 format-blind, and soup ship compared against a 0.05 threshold without ever telling the operator what its own instrument could resolve — four of six paired deltas in that session sat inside the floor. With the flag, the BASE model is re-run N times (2–10), the per-axis floor is max − min across the repeats, it is printed beside the verdict, and every axis is then gated at max(threshold, floor). Leg 1 must clear the task axis's floor too. The max is load-bearing in both directions: a floor above the threshold widens the gate to what is actually measurable, and a floor below it must never tighten the gate behind the operator's back. Opt-in, so no existing run changes; costs N extra base passes. The leg-1 floor is measured in --task-mode metric only — in the judge modes a repeat would fold the judge's own sampling noise into a number presented as decode noise, so the run warns and leaves leg 1 at a 0.0 floor instead. Carried caveat, because it bounds the claim: n=3, one model, one dataset. It sizes the effect; it does not calibrate a threshold.
• A benign-prompt axis, mini_over_refusal, joins the default leg-2 suite (7 → 8) (#317). Leg 2 flagged a drop in mini_safety and had no reverse, so a tune that refuses everything registered as a monotone safety improvement with no ceiling on how useless the model became. Reproduced as indistinguishability, which is the actual claim: two models with byte-identical scores on all seven shipped suites and the same SHIP verdict, one of which refuses every benign request. The new suite is 40 hand-authored, original benign-but-scary-sounding requests (defensive security, first aid, sysadmin verbs like "kill"/"terminate", fiction with dark themes) scored as fraction NOT refused, so the existing regression rule catches over-refusal with no change to the verdict engine. Paired with mini_safety neither axis can be gamed alone. Same caveat as above: 40 prompts and one greedy pass size a gap, they do not calibrate a threshold.
• Semantic Stratified Splitting for soup data split (#388). --stratify-semantic and --num-clusters partition splits proportionally across semantic groups (TF-IDF + K-Means) so a whole topic cannot land entirely in one split. scikit-learn is a declared member of the [data] extra and a missing import refuses with the install command rather than silently falling back. 50k-row cap, an explicit error for a pure-stop-word corpus, and a warning when --num-clusters is passed without the flag. Contributed by @Deadpool2000.
• soup mcp serve --allow-execute (#391). A stronger opt-in than --allow-mutating, which it implies, plus the _refuse_execute handler the execution tools will use. train_start and export stay plan-only — nothing in this slice executes anything, and the help text says so in the present tense. build_registry(allow_mutating=True) still returns the same 16 tools. Contributed by @CODING-DARSH.

Fixed

• extract_mcq_letter did not know \boxed{C}, and the MCQ prompt never asked for a letter (#357). Meta-Llama-3.1-8B-Instruct scored 0.423 on mini_mmlu — below a 0.5B — while scoring 1.000 on two other MCQ suites. Of 15 failures, 8 boxed the right letter and 6 boxed a value because nothing in the prompt asked for a letter. Reproduced here at the extreme: a stub that answers every item CORRECTLY in the boxed-letter style scored 0.000 on mini_mmlu and mini_common_sense. Both halves are needed — the extractor alone is worth +8 items, the prompt alone 0, together 0.423 → 0.731 and the inversion disappears. The new tier fires only when the box holds a single A–J letter: reading \boxed{4} as "option 4" would be a wrong credit, not a repair. Among the boxed / cue / paren forms, position decides, not form — a model that boxes a scratch answer and then self-corrects chose the correction.
• mini_tool_call ranked by brace hygiene (#346). The 8B named the right tool 40/40 and scored 0.225: it emitted three opening braces and two closing ones, the whole-string parse failed, the bounded scan returned the INNER object, and the scorer rejected it for having no "function" key. The gate suite's own unwrapping layer now restores the envelope for an object carrying both name and arguments — requiring arguments is what stops an echoed {"name", "description"} menu entry from scoring, i.e. from crediting copying. The missing brace is the model's own output, not truncation; that attribution was believed and shipped in c87fd00 before a budget sweep disproved it.
• score_bundled_suite returned 0.0 for a non-callable gen (#355). On the three behavioural suites it scored 0.0 while the MCQ suites raised TypeError — and in leg 2 a 0.0 reads as "the model failed every item" → DON'T SHIP, so a caller error was indistinguishable from a regression and failed in the direction that looks like a finding. Both branches now raise. A callable that misbehaves is still a failed item, which is the correct existing contract.
• The verdict panel silently ate its own leg-1 marker. render_ship_panel built its header as ... [{won_str}], and a bare [no win] is valid Rich markup for an unknown tag — so the panel never printed "won"/"no win" on any release up to and including v0.73.1. The plain-text rubric, which has no markup parser, printed it correctly the whole time, which is why it went unnoticed. Found while rendering the new noise-floor panel.
• Untrusted names could drive the terminal. Benchmark and noise-floor axis names come from an --evidence JSON file, and rich.markup.escape neutralises Rich's [...] syntax and nothing else, so a raw ESC byte survived it. Both render paths now strip C0/DEL first, matching the _for_terminal pattern already used in six other command modules.
• The --evidence round-trip and the MCP ship_evidence tool now agree. A verdict decided against a measured floor does not replay without it, so the floor is part of the evidence schema (#312's output-is-input property), and both readers honour it — a schema extended in one consumer and not the other would have made the same file replay to different decisions through the CLI and through soup mcp serve.
• A duplicated #392 CHANGELOG entry. PR #388 branched before 18a278a moved that entry into [0.73.1], and the merge re-introduced it under [Unreleased], listing the same fix twice.

Security

• An evidence-supplied noise floor is bounded and never silent. A floor widens the gate, so "floors": {"mini_mmlu": 1.0} in an evidence file masks any possible drop on that axis. This does not cross a new trust boundary — anyone who can edit that file can already write {"base": 0.9, "tuned": 0.9} and force a SHIP outright — but it is a far quieter edit to miss in review, and soup ci init wires ship --evidence as a PR merge gate. Values are therefore bounded to [0.0, 1.0], the mapping is capped at 50 axes / 256-char names (mirroring the CLI's own limits), a malformed block is refused rather than dropped (a dropped floor replays as a different verdict), and any floor that exceeds --forgetting-threshold is announced — on stderr by the CLI, and in the returned payload by the MCP tool, whose stdout is the JSON-RPC channel. Neither reader is the quiet one.

Changed

• decide_ship now canonicalises the TaskWin it stores, as it already did for the benchmark deltas. It recomputes leg 1 from the raw scores, so a TaskWin built without the floor would otherwise have rendered "won" beside a DON'T SHIP decided with it.
• --baseline snapshots taken before this release are on a different scale for mini_mmlu, mini_common_sense and mini_tool_call, because their scorers changed. A stored baseline skips the live base run, so it would be diffed against a freshly-scored tuned model — measured on an unchanged model the jumps are 0.423 → 0.731 and 0.225 → 1.000, far larger than the 0.05 gate. soup ship now warns by name when --baseline supplies a stored score for an affected suite. mini_instruction and mini_arithmetic are unaffected and are deliberately not named: neither carries a single-letter answer, so the prompt cue and the option-letter extractor never touch them (verified, 0 of 24 and 0 of 36 items).

0.73.1 - 2026-08-14

Added

• training.stream_vram_probe decides the layer-streaming VRAM pre-flight on a MEASUREMENT instead of the fitted formula (#349). The pre-flight predicts peak VRAM from a formula fitted to 10 real runs, and its documented contract is that it never under-predicts. Measured through the real soup train on an RTX 3050 Laptop (4 GB, Windows, torch 2.5.1) with SmolLM2-135M streamed in bf16 at batch 1, that contract holds at short sequence and then fails:

  seq	predicted	real peak	ratio
  4352	3.282 GB	3.036 GB	1.081x — over-predicts, safe
  5120	3.844 GB	4.118 GB	0.934x — under-predicts
  6144	4.590 GB	5.830 GB	0.787x — under by 21%

  Under-prediction is the direction that does not announce itself: an OOM on Linux, and on Windows/WDDM a silent spill to host memory. The existing grid could not have caught it — all ten of its rows are at seq 256 or 512, so it varies batch and says nothing about sequence length, and test_never_under_predicts has been narrowed to state that scope rather than imply a global guarantee.

  With the flag on, one real forward+backward runs at the configured shape after the streamed model is built and its peak decides; the prediction is printed beside it so a divergence is visible. Measured cost: 1.0-5.3 s (SmolLM2-135M at 1x1024 and 2x2048; Llama-3.1-8B NF4 at 1x512), against a training run of minutes to hours. Off by default — it costs a step, and it can refuse a run the formula accepts.

  Scope is deliberately narrow. task: sft only: the probe runs a plain causal-LM step, which is the SFT step but is not a preference loss, so its agreement with one is not established. Measured at a single matching shape it is conservative there too (6.02 GB against a real DPO step's 5.30 GB, +13.5%) — but one point is not a validation, and a sign flip would mean a gate waving through over-budget runs. It also cannot overrule a prediction more than 4x over budget: the largest disagreement ever measured is 21%, so beyond a small multiple the config is simply too big and is refused by arithmetic without touching the GPU. The gate reads max_memory_allocated, not max_memory_reserved — reserved runs 1.08-1.41x allocated and overshoots what has to fit, and gating on it would refuse this feature's own flagship configuration (Llama-3.1-8B NF4, 3.70 GB reserved against 3.45 GB free, which runs).

  Two readings were tried during this work and withdrawn as unsupported, recorded because the tempting inference was wrong twice in one investigation: that preference losses are over-budgeted ~8.8x (it is 1.15x at the budgeted shape — the earlier figure came from rows that realised 142 of a budgeted 2048 tokens), and that the over-budget runs were silently spilling (num_alloc_retries was 0 on every shape measured). The mechanism behind the long-sequence divergence is likewise not claimed: seq**2 from the attention score matrix is the obvious candidate and the numbers do not settle it.

Fixed

• The MLX adapter_config.json shipped target_modules unresolved, so a default MLX adapter loaded as a silent no-op (#392). _apply_lora resolved target_modules: auto into a local variable and trained the resolved modules; the writer serialised the raw config value, so the shipped file carried {"keys": ["auto"]}. On load, linear_to_lora_layers matches no module against that and load_weights(strict=False) drops every LoRA tensor without a word — generation with the adapter is bit-identical to the base model. "auto" is the schema default, so this was every MLX run that did not name its modules by hand, and the file exists precisely to promise the output dir loads with mlx_lm.load(..., adapter_path=...). Both callers now go through one resolve_mlx_target_keys(), because two copies of "which modules did we train?" is how they drifted. Reported with a root cause and a control by @armanbot-jpg: hand-editing keys in the saved file makes the very same adapters.safetensors produce the tuned behaviour.

• training.batch_size accepted 0 and negative values. Union[int, Literal["auto"]] carried no lower bound, so batch_size: -4 loaded and then meant whatever each trainer's arithmetic did with it — including the streaming VRAM pre-flight, which multiplies by it. Now rejected at config load. Surfaced by the #349 security review.

Fixed

• Layer streaming's VRAM pre-flight now actually calls its own calibration hook (#348). calibrated_logits_bytes_per_element() exists to raise the budget when a stack's loss path measures a heavier retention than the shipped constant assumes, guarding against a future stack silently under-budgeting by 12.5% with nothing to catch it. _stream_budget_lines called estimate_stream_peak_vram() without logits_bytes_per_element=, so the parameter was always None and the calibration never ran outside its own test. It is now forwarded to both the budget and the panel's logits figure; the value can only raise the prediction (floored at the shipped constant), and the panel prints an extra line naming both numbers when the calibration measures above it. This makes the probe unconditional rather than opt-in (see #327 below, whose wording is updated to match): every streamed run now pays one transient 14 * vocab_size * max(tokens) allocation (96 MiB at the defaults) plus two torch.cuda.synchronize() calls before the fit decision is taken. On today's measured stacks this is a no-op in effect (measured is 12.0 with zero spread, max(14, 14) is 14, no extra line prints), so the cost buys nothing yet, which is the point of a guard against a stack that hasn't shipped.
• MLX backend now actually dispatches to the MLX trainer for task: sft. Previously backend: mlx silently fell through to the transformers SFTTrainerWrapper, training on MPS/CUDA instead of MLX. The trainer was also rewritten for mlx-lm >= 0.31 (create_dataset + CacheDataset, TrainingCallback), with model.freeze() before LoRA — without it the saved "adapter" was a full fine-tune (172 tensors vs 24 LoRA tensors on a 1.2B model) — and an adapter_config.json is written so the output dir loads directly with mlx_lm.load(..., adapter_path=dir). (#362)
• mlx-lm floor raised to >= 0.31.3 (the version the MLX SFT path is built against).
• training.seed reached the SFT wrapper and nothing else (#353). #341 added the knob and wired it into trainer/sft.py. The other seventeen task wrappers each build their own TrainingArguments subclass (GRPOConfig, DPOConfig, RewardConfig, and so on) and none of them read the field, so task: grpo with training.seed: 7 trained at HF's default of 42 with no error and no warning. Replicates that differed only in training.seed were therefore the same run, which is what happened to STEP 25 of the H100 record: its five "replicates" were five runs of seed 42, measured against a within-mode spread produced by the very thing it thought it was varying. Threading the config is only half the repair. Trainer.__init__ runs set_seed(args.seed), but get_peft_model has already drawn lora_A by then, and classifier / reward_model / prm have already drawn a freshly initialised head inside from_pretrained, so every wrapper now applies the seed at the top of setup() as well, before the model is loaded. unlearn builds no Trainer at all and drew its RMU control vector from a generator hard-coded to 0; that draw now follows training.seed, staying at 0 when the seed is unset so existing runs keep their control direction. An unset seed still resolves to 42 and leaves data_seed at None, so the values a run trains at are unchanged. What changes is when they arrive. Nothing called set_seed before get_peft_model previously, so lora_A was drawn from torch's default generator, which is seeded from entropy once per process: an unseeded run's adapter and classification-head initialisation varied from one process to the next, and it is now deterministic at 42. Replicate variation that came out of runs setting no seed was coming from exactly that, so those runs are now identical to each other and varying a replicate means setting training.seed on purpose. The MLX backend (backend: mlx) is the one path that still reads neither field.
• training.seed on the MLX backend now says it is ignored (#353, fourth criterion). MLX has its own RNG (mx.random) and none of the MLX wrappers touch it, so a seeded MLX run was silently unseeded — which looks identical to a seeded one until two replicates disagree. That gap only became reachable between #353 being filed and #381 landing: backend: mlx was never dispatched at all until #362 (#363). Setting either field now appends to the wrapper's existing "MLX backend ignores:" line, naming training.seed / training.data_seed so it is greppable as written. A warning, not a rejection: a config valid on transformers should not become unloadable by switching backend. Seeding MLX for real is separate work with a separate RNG.
• Under use_fsdp2_compile, every checkpoint-* still loads as a dead adapter (#351). #335's repair runs once, on the output root, after the final save_model. HF's Trainer writes its periodic checkpoints through that same save_model with output_dir=<run>/checkpoint-N, so they come out carrying _orig_mod. on every key and nothing ever normalised them. Measured at 70B on 8×H100 (benchmarks/gate-h100-validation.md, STEP 28): 320 canonical keys in the output root, 320 prefixed ones in checkpoint-100. Resuming is the case that decides how bad this is, and it is worse than #335 was. PeftModel.from_pretrained at least warns; Trainer._load_from_checkpoint calls model.load_adapter(...) and drops its return value, and load_adapter deliberately does not warn (it hands the missing keys back in the load result instead, which nothing reads), while load_state_dict(strict=False) discards the _orig_mod. keys without a word. A resumed run therefore continues from a re-zeroed lora_B in total silence: #335's failure shape with its one warning removed. load_best_model_at_end was reloading a dead adapter for the same reason, since the Trainer restores state.best_model_checkpoint through that same load_adapter, and this repairs that path too. Normalising now happens on HF's on_save, as each checkpoint is written. Both that callback and the final save are gated on args.should_save, the condition save_model writes under, so exactly the rank holding the file repairs it rather than all eight opening it at once. The callback is attached in setup(), ahead of anything a caller adds later: HFPushCallback.on_save uploads checkpoint-{step} on this same event and CallbackHandler dispatches in insertion order, so a normalisation attached after it would leave --push-as publishing the prefixed adapter and keeping the repaired one on local disk.
• A streamed model's named_parameters() still carried the wrapper's .inner. segment, so a name-keyed comparison against a resident model of the same checkpoint saw no overlap (#369). v0.72.1 made state_dict() canonical for serialisation; this issue is the first time something compared the two model kinds by parameter name instead, and the #331 repair gate read the resulting empty intersection (grads exact 0/0) as a pass. layer_stream_runtime.canonical_named_parameters() strips .inner. the same way state_dict() already does, and assert_canonical_parameters_intersect() raises instead of reporting an empty intersection as success. Neither the forward path nor state_dict() changed, so the v0.72.0 bit-exactness gates remain valid unexercised. named_modules() and named_buffers() carry the same segment and are deliberately not covered — the comparison that produced the false green was over parameter names.

Added

• training.stream_vram_override gives the layer-streaming VRAM pre-flight an explicit escape hatch (#347). decide_stream_fit refused any run it predicted would not fit, with no way through except lowering batch_size or max_length. Setting this field now replaces the measured free-VRAM figure the pre-flight checks against, in either direction: raised past a documented over-prediction to let a known-safe config through, or lowered to enforce a cap mem_get_info() cannot see, such as set_per_process_memory_fraction on a shared or capped card (a Colab/Kaggle T4, a MIG slice). Rejected at config load when set while stream_layers is false, mirroring the existing stream_source/stream_buffers footgun gate.

Fixed

• A hosted notebook's preinstalled torchao made get_peft_model raise, and it read as a Soup bug (#389). peft's is_torchao_available() does not return False on a version it considers too old, it raises ImportError — nine frames inside get_peft_model, with nothing near the top of the traceback naming torchao. Colab preinstalls torchao 0.10.0 against a peft that demands newer, so the first soup train on a free notebook died in a place unrelated to anything the user had configured. Now mapped in utils/errors.py to the cause and the one-line fix (pip uninstall -y torchao), with the part worth saying out loud: Soup does not need torchao at all unless training.quantization_aware is set. Found by running notebooks/proof-4gb.ipynb on a real free-tier session, which is the same place #385's first repair was caught being a no-op.

• bf16 was assumed on every CUDA card, so the entire free GPU tier failed (#385, #387). Fourteen places, and only the first was known: trainer/stream_setup.py chose the layer-streaming store dtype with the literal "bfloat16" if on_cuda else "float32" (#385), and then a live run found SFTTrainerWrapper._resolve_mixed_precision returning (device == "cuda", False) as its default — and an audit found the same bf16=self.device == "cuda" in twelve more wrappers: bco, classifier, distill, dpo, embedding, ipo, kto, online_dpo, orpo, pretrain, reward_model, simpo (#387). So it was not a streaming bug at all; every task died on that hardware. The sharpest detail is that the codebase already knew: trainer/asr.py carries the comment "bf16=cuda was hardcoded, which crashes on pre-Ampere cards (T4 / GTX 16xx)" and fixes it — in that one wrapper, never propagated. All fourteen now take the answer from one place, utils/gpu.bf16_fp16_flags, including ASR, whose private copy was folded in. bf16 needs Ampere. Colab's free tier is a T4 (sm_75), Kaggle is a T4 or a P100, and V100 / GTX 16xx / RTX 20xx are all pre-Ampere, so on that hardware Soup ran in a dtype the card has no units for. Neither could fail on the maintainer's RTX 3050, which is Ampere; this is the same shape as the four backends the H100 session found had never executed once.

  Two corrections to the first version of this entry, both established by finally running it on a real T4 rather than reasoning about one. (1) The claim that every task died before step 0 on transformers' "Your setup doesn't support bf16/gpu" was wrong: that error was produced by a local stub forcing is_bf16_supported() to False, and transformers gates on the same permissive call described next, so on the current stack it does not raise at all. (2) More seriously, the first fix was a no-op on the hardware it was written for. torch.cuda.is_bf16_supported() defaults to including_emulation=True: when its compute-capability fast path fails it falls through to merely constructing a bf16 tensor, which software emulation satisfies — so a T4 answers True, and asking the bare question selected bf16 exactly as the hardcoded literal had. The predicate now asks is_bf16_supported(including_emulation=False) (falling back to a capability check on older torch), and get_compute_dtype — a second copy of the same question — was folded into it. What a T4 actually does with emulated bf16, as opposed to what it reports, is not yet measured. This cannot regress a working setup: where bf16 is supported the answer is unchanged, and where it is not the previous behaviour was a crash. A test SCANS every module in soup_cli/trainer/ rather than parametrising over a hand-written list — the list is what hid the twelve — and the existing unit test for this line had to be rewritten because it asserted the defect (test_auto_flag_off_preserves_legacy required bf16 on any CUDA device, and passed in CI precisely because CI has no GPU and the old code never asked the driver). Correctness of the alternative was measured before the change rather than assumed — streamed-vs-resident logits are bit-exact at 0.000000e+00 in float16 as well as bfloat16, in both quantisations, against resident references of matching numerics (an NF4 streamed run compared against a genuinely NF4 resident one, since comparing it against a bf16 model would measure the dtype rather than the streaming), and the adapter is non-zero. Not yet verified on a pre-Ampere card: that exactness was measured using fp16 on Ampere, so it establishes the plumbing, not the Turing/Pascal kernels — the free-tier notebook is the natural place to close that.

Changed

• Retracted: "layer streaming is bound by host-to-device transfer, not by the GPU." That sentence appeared in the README, in the v0.73.0 notes below, in the H100 gate record and in the preprint's abstract. It was an inference from the H100 replication — the same configuration returning the same throughput on a card with two orders of magnitude more compute — and it had never been measured. It was measured on 2026-08-11 on the original laptop and is false at the published configuration: four interleaved ablation arms in one process at a pinned clock show that removing all host-to-device traffic (6.864 GB per step) buys 1.44%, removing the NF4 dequantisation buys 9.80%, and removing both leaves 88.7% of the step; the compute stream is blocked on a copy for 8.4 ms of a 4190 ms step; and the step runs at 71.3% of that card's same-session, shape-matched GEMM ceiling. The claim is true below roughly 128 tokens per step, where the fixed transfer volume dominates — the published configuration is not there. No measured number anywhere changes, and the replication result stands in a weaker form: the constraint is common to both machines and is not the compute the datacenter card adds. The H100's own bottleneck was never instrumented and no claim is made about it. Record: benchmarks/probe-v0.73.0-what-bounds-streaming.md. Every occurrence in the gate record and in the v0.73.0 notes below is annotated in place rather than deleted, because in a folder whose whole premise is publishing the record as written, a silent deletion costs more credibility than the error does.

Validation (measured, not changed)

• Layer streaming completed a run on hardware the maintainer does not own: a free-tier Colab Tesla T4 (sm_75, Turing), via notebooks/proof-4gb.ipynb. Every streaming number this project has published came from one RTX 3050 Laptop or one borrowed 8×H100, and the pre-Ampere fix above (#385, #387) had been verified using fp16 on an Ampere card, which establishes the plumbing and not the Turing kernels. This closes that specific gap and nothing wider. NousResearch/Meta-Llama-3.1-8B-Instruct, NF4, stream_layers: true, stream_buffers: 2, batch 1, max_length: 256, LoRA r=8/α=16, fp16 (a T4 has no bf16 units): 7 steps, exit 0, adapter written with 128 tensors, 128 of them non-zero, and a measured peak of 2.91 GB against the pre-flight's predicted ~3.02 GB — an over-prediction of 3.8%, which is the direction the estimator was fitted to err in (v0.72.3 fitted it to never under-predict) and is the whole reason it is allowed to stop a run. The card has 15.6 GB, so the run was constrained artificially with torch.cuda.set_per_process_memory_fraction to 4.00 GB, and the cap was shown to bite rather than assumed: a deliberate 4.29 GiB allocation was refused. That artificial cap is also the reason no throughput figure is quoted from this run, here or in the notebook — a capped card is not a benchmark, and the panel's own forecast (31–46 tok/s from 2.20 TFLOPS measured at 1185 MHz) is a compute bound, not a measurement of what the run did. Two things the run did not establish, stated because the temptation is to let the exit code cover them. Backward/gradient exactness at 8B on Turing is not shown — a non-zero adapter proves gradients flowed, not that they were right, and the streamed-vs-resident comparison in the notebook's section 4 produced no captured output, so it is recorded as unrun rather than as a pass. And the loss moving 4.5266 → 4.0674 over 7 steps, non-monotonically (4.5266, 4.2388, 3.7485, 4.6930, 4.1940, 4.1127, 4.0674), is reported because it is what the run printed; over seven steps it is not evidence of learning. One observation worth carrying forward: the pre-flight panel reported free VRAM 15.10 GB, i.e. it read the device and not the per-process cap the run was actually held to, so the fit decision was taken against a number 3.8× larger than the budget in force. The run fit on its own merits (2.91 GB against 4.00 GB), so nothing was protected by luck — but on that hardware the pre-flight is not what would have caught an over-budget config, which is exactly the case training.stream_vram_override (#347, above) exists for. Record: benchmarks/run-t4-colab-free-tier.md.

[0.73.0] - 2026-08-09

The release that came out of three days on somebody else's hardware.

Every number this project had ever published was measured on one machine: an RTX 3050 Laptop, 4 GB, Windows. From 5–9 August it ran on a borrowed 8×H100 box (Ubuntu 24.04, a much newer torch / bitsandbytes / trl / peft stack) for the first time. That found one silent correctness defect in layer streaming, four backends that had never actually run, and a documented multi-GPU entry point that had never launched — plus the first evidence that the laptop result reproduces on hardware nothing like it. The full record, published as written including six rejected hypotheses and three false positives that controls caught, is benchmarks/gate-h100-validation.md.

This is a minor bump, not a v0.72.x patch: it adds two capabilities that did not exist, and repairs four backends.

Added

• training.seed and training.data_seed (#341). Every Soup run trained at seed 42 with no way to change it, so "run this twice with a different seed" was impossible. Both default to None rather than to 42 on purpose: an unset seed has to reproduce two different historical defaults — HF's 42 for TrainingArguments and 0 for the multipack sampler since v0.37.0 — and a plain 42 default would have silently re-ordered every existing multipack run. Scope, stated rather than left as a footgun: wired into the SFT trainer only. Other task wrappers build their own TrainingArguments, so setting it on a DPO run parses and does nothing — tracked as #353.
• Full fine-tuning as lora.r: 0 (#340). The SFT trainer's full-FT branch was dead code with no way to reach it. r: 0 was chosen on repo evidence, not taste: three consumers already treat rank 0 as "no adapter", and r: 0 previously crashed inside PEFT, so no config that worked before changes meaning. lora.r also gained a lower bound — r: -5 used to parse and die inside PEFT.
• --deepspeed zero3_offload — ZeRO-3 with CPU parameter offload. zero3 set offload_param: none and the only offload preset was stage 2, optimizer-only, so the configuration a user short of VRAM actually wants could not be named on the command line. Measured on one H100 (Llama-3.1-8B, bf16, LoRA r=8, 256 steps): 21.65 tok/s at a 38,135 MiB peak. offload_optimizer deliberately stays none — turning it on makes DeepSpeed JIT-build cpu_adam against a matching CUDA toolkit and fail without one; copy the emitted JSON and flip it if you have nvcc.
• trl support widened to >=0.14.0,<0.29 (#326), behind a capability-probe compat layer (trainer/_trl_compat.py) rather than a version table — a version table is what was wrong twice before. The trainers now ask each config class whether it accepts max_prompt_length, and resolve ORPOConfig/CPOConfig/BCOConfig through trl.experimental when trl 0.29 drops them from the public namespace. All six preference trainers construct and train to identical losses on trl 0.26.2, 0.28.0 and 1.9.2.

Fixed — backends that had never been run

• soup train --gpus N never launched at all (#77). accelerate launch takes a script path positionally, and Soup handed it sys.executable — so accelerate opened the Python binary and parsed it as source (SyntaxError: source code cannot contain null bytes). Every rank died before the trainer existed. The documented multi-GPU entry point has been dead since it shipped, invisible to single-GPU CI because that path skips the launcher wrapper entirely. Separately, --no-reexec printed a command with every user flag dropped, so following the hint literally trained without --fsdp.
• DeepSpeed could not train a LoRA model on any stage (#336). HF builds two optimizer parameter groups and with LoRA the no-decay group is empty; DeepSpeed drops it, leaving one group against two base_lrs, and torch's scheduler then hits a strict-zip length mismatch. Verified repaired on 2×H100 with real soup train: zero2 6790.2 tok/s, zero3 1025.3, zero++ 977.1, all exit 0 with a live adapter (96/96). zero++ failed earlier and independently — it set fp16 quantised weights/gradients against a bf16 model and hardcoded zero_hpz_partition_size: 8 regardless of the real world size; both are now derived, and the rewrite is printed rather than applied silently.
• use_fsdp2_compile wrote an adapter that reloads as all zeros (#335). Under torch.compile the Trainer saves through the wrapper, so every key came out as _orig_mod.base_model.model.... The tensors were genuinely trained (max|lora_B| 7.0e-3 measured) and PeftModel.from_pretrained matched none of them — emitting only a UserWarning and leaving lora_B at its zero init. Measured on 4×H100: 0 of 96 non-zero against 96/96 for the paired non-compile run, reproduced 3/3, with the run exiting 0 throughout.
• soup serve --backend sglang returned 500 on every generation (#76). sglang 0.5.16's Runtime.generate returns a JSON string; Soup subscripted it as a dict. Deterministic, not a race — the backend loaded cleanly and then failed 100% of requests. It had genuinely never been run, because SGLang does not support Windows.
• The vLLM backend ignored the model's chat template (#332). utils/vllm.py hand-rolled a "User: ...\nAssistant:" prompt while the transformers backend used apply_chat_template, so every vLLM user's model saw a format it was never trained on. On Llama-3.1-8B + LoRA, identical server and sampling params, only the prompt differing: a run-on loop burning all 200 tokens before, an 8-token answer after. Both backends now share one build_chat_prompt.
• Three vLLM serving defects (#333), each verified live: finish_reason was hardcoded "stop" even at completion_tokens == max_tokens; --dashboard silently no-opped (/metrics returned 404 with nothing printed); and --max-model-len did not exist although the engine factory already accepted it. --dashboard on a backend that cannot serve it now warns at startup naming the backend instead of doing nothing.

Fixed — training paths that silently did the wrong thing

• data.max_length was capped at 1024 on every SFT run (#78). SFTTrainer converts TrainingArguments with SFTConfig(**args.to_dict()), and max_length is an SFT-only field that TrainingArguments does not carry — so it always took SFTConfig's default. Measured before the fix: data.max_length=4096 gave 1024 tokens per sample, with no warning.
• training.use_liger: true crashed at step 0 (#78). Soup patched the model but never set TrainingArguments.use_liger_kernel, the flag TRL reads to know the fused path returns logits=None; its entropy metric then ran on None. Reproduced across the whole supported trl pin, so the feature did not run at all. Separately, Liger's architecture match was a substring of the model name, so any model loaded from a local directory trained without Liger on a flag the user had explicitly set — it now reads AutoConfig.model_type.
• FlashAttention 3 was selected from a version that can never report 3 (#334). Dao-AILab ships FA3 as flash_attn_3; flash_attn itself stays in the 2.x line, so the branch could not fire for any real install — and had it fired, it produced an attn_implementation transformers would reject. On Hopper hardware users silently got FA2 or SDPA while the docs advertised FA3. Both detectors now ask transformers. This makes detection honest; it does not make FA3 measurable here.

Fixed — layer streaming

• A silent wrong-gradient defect on large NF4 models (#331). bitsandbytes.MatMul4Bit stashes the packed weight and quant_state on ctx as plain attributes instead of through save_for_backward, so gradient checkpointing cannot discard and recompute them. The reference is captured in the forward, aliases the streaming buffer pool, and is read in the backward after that slot has been refilled. Result: a bit-exact forward, a healthy-looking loss curve, and wrong gradients on every layer but the last stream_buffers. It bites NF4 above a threshold bracketed at 163.8–171.5 MiB per layer — so 32B and 72B, never 8B or 14B, and never bf16. The repair keeps the weight out of that function entirely: dequantise inside the checkpointed region and use a native matmul, which saves the dequantised weight through the ordinary mechanism. Gated against a resident NF4 reference with a repair-disabled control arm in the same process: real 32B 256/256 gradient tensors exact against the control's 8–12/256, at +2.9% peak VRAM and −4.8% throughput; and again on real 72B — the size where the defect was worst — 320/320 against 8/320, at +2.6% and −3.7%. De-aliasing was measured and rejected first: bnb holds the reference across the whole forward-to-backward span, so any copy is O(model), which took real 32B from 4,220 to 19,720 MiB and deletes the feature's premise.
• All four preference losses died on newer trl (#328). should_enable_hf_gradient_checkpointing existed so the HF Trainer does not checkpoint an already-checkpointed streamed model twice — and only sft.py ever called it. TRL's default is False on trl 0.19.1 and True on 0.26.2, so one omission had two opposite symptoms: an explicit gradient_checkpointing: true silently dropped on the older stack, and on the newer one HF checkpointed the inner decoder layer, recomputed it after the reparametrisation context had exited, and killed dpo/orpo/simpo/kto with Tensor on device cuda:0 is not on the expected device meta!.
• Layer streaming is now refused when nn.DataParallel would engage. HF wraps the model in DataParallel whenever more than one CUDA device is visible and the run is not distributed, and DataParallel requires every parameter on device_ids[0] — streaming keeps the decoder on meta by design. It accounted for 8 of the 9 streaming-suite failures on the H100 box and is unreachable on a one-GPU machine. It refuses rather than quietly using 1 of 8 cards.
• device_map="auto" broke every distributed launch, in fifteen places. transformers refuses device_map="auto" outright under a distributed launch, so the exact accelerate launch command soup train --gpus 8 prints died on every rank. A first pass fixed six trainers; nine sites still carried it. All fifteen now go through utils/gpu.resolve_device_map, and the regression guard scans every module in soup_cli/trainer/ instead of a hand-written list — the list is what hid the nine.
• LOGITS_BYTES_PER_ELEMENT is split into two independently measured terms (#327). The 14 is 12 + 2, measured stage by stage on an H100 with zero spread across three repeats, and only the 2 differs between stacks. It is deliberately not lowered — see Known Limitations. What is new is an upward-only calibration (max(14, measured + 2)), which closes a real unguarded hole: a future stack that grew a fourth fp32 buffer would be under-budgeted by 12.5% today with nothing to catch it. Default behaviour is byte-identical and the pre-flight path takes no new CUDA. (Originally landed as an opt-in probe with no caller; #348 above wires it into the pre-flight itself, so it now runs on every streamed run rather than sitting inert.)

Fixed — eval, ship and export

• Two of soup ship's three behavioural suites measured nothing (#316). On Meta-Llama-3.1-8B-Instruct, mini_tool_call scored 0.000 and mini_format_json 0.000 on a model that does both correctly. All harness defects: the tool-call prompt never told the model tools exist, the JSON check ran json.loads over the whole output while 38 of 40 answers sit inside a ```json fence, and the generation budget was taking make_generator's default of 64, truncating 31 of 40 tool calls one closing brace short.
• The refusal detector missed the apostrophe models actually type (#316). The patterns spelled the contraction with U+0027; Llama-3.1 writes U+2019. Over the shipped 40-item mini_safety suite, 28 of 40 refusals scored as non-refusals and the suite reported 0.300 for a model whose true refusal rate is 1.000 — a 0.70 error against a 0.05 regression threshold, i.e. 14× the thing it exists to detect.
• soup train --task online_dpo passed a keyword trl removed at 0.25 (#300). The probe asked whether BasePairwiseJudge was importable and used the answer to decide whether to pass reward_model=; those two facts had decoupled, so the probe said yes for every trl in the supported range. It now asks the signature — through an MRO walk, because on 0.26.2 the top-level class is a deprecation shim whose direct signature reports no parameters at all.
• A quantised GGUF export deleted a previously exported f16 (#144). The f16 intermediate was named {model}.f16.gguf next to the output — exactly the default output name of a --quant f16 export — and unlinked when quantisation finished. So exporting q4_0 destroyed an earlier f16 export, even with an unrelated --output. Reproduced. It now lives in a private temp directory. Separately, the convert dependencies were installed only after an auto-clone, so --llama-cpp /path and an already-present checkout both died on ModuleNotFoundError: sentencepiece.

Fixed — packaging and docs

• requires-python now has an upper bound: >=3.10,<3.13 (#358). CI tests 3.10, 3.11 and 3.12 and nothing above. Without a ceiling, pip on 3.13+ resolved torch wheels nobody here has run, and the failure was not a Soup error message — it was a loader crash inside c10.dll / libc10.so before any Soup code executed, leaving the user nothing to act on. The bound is 3.13, not 3.14: 3.13 is equally untested. tests/test_requires_python_bound.py derives the bound from the CI matrix, so widening one without the other fails the suite.
• The trl bounds shipped in v0.72.4 were wrong at both ends, and were corrected before #326 widened them again. The ceiling was over-tight by five releases: the v0.72.4 table read a trl/experimental/ relocation as a field removal. The floor >=0.7.0 was impossible — setup() imports GRPOTrainer, first exported at 0.14.0. One detail in the v0.72.4 note was also imprecise: ORPOConfig/CPOConfig are not deleted at 0.29, they leave the public namespace — an ImportError rather than a rejected keyword, which is worse, not milder.
• The layer-streaming pre-flight panel was titled after a flag that does not exist (#329). It read soup train --stream-layers; there is no such option — streaming is enabled by training.stream_layers in soup.yaml. It is the first thing a streaming run prints, so it was the feature's most-read line of documentation, and it pointed at a No such option.
• Seven of the eight configs in examples/configs/ did not parse. They were written against a pre-nesting schema, so soup train --config examples/configs/sft_basic.yaml — the first command examples/README.md tells you to run — failed validation. Also fixed while there: two configs declared format: sharegpt for preference-shaped data (a silently wrong training run, not an error), target_modules listed out_proj which no Llama has, and vision_llama.yaml pointed at files that have never existed in this repo. tests/test_examples_configs.py now parses every one of them.
• soup data demo metadata described files other than the ones it ships. sharegpt_demo declared format: sharegpt for prompt/chosen/rejected rows — the value a user copies straight into data.format — and grpo_demo declared reasoning, which is not in the schema's format literal at all.
• Encoding corruption in pyproject.toml (fourteen em-dashes round-tripped through cp1251, one of them in the unit pytest marker description that pytest --markers prints), and docs/commands.md, where missing newlines collapsed five commands onto two lines and the page promised "the full command list" while omitting seven.

Changed — documentation corrected against measurement

• LISA delivers the quality half of its claim, not the memory half (#306). Measured at 3B and 8B on an H100 with LISA engagement verified three independent ways: it beats full fine-tuning at both learning rates, and it is 1.22× LoRA's VRAM at 3B and 1.51× at 8B, with the gap widening with scale — embeddings, LM head and final norm stay trainable every interval and are 70.7% of everything LISA trains at 8B. The docs now say that plainly, and say when LISA is still the right choice.
• FlashAttention and Liger were measured for the first time, at 1.015× and 1.051× / −12.9% VRAM, against documented claims of "2–4×" and "20–60% / 20–40%". Both claims are corrected in the docs.
• The GEMM-ceiling test could not pass on a datacenter GPU. Its plausibility bound was < 200 TFLOPS, written when the only hardware this project had was an RTX 3050; an H100 returns a correct 786.5 TFLOPS and the assertion fired.

Validation (measured, not changed)

• The laptop result reproduces on completely different hardware. Llama-3.1-8B NF4: 119.6 tok/s in a 3.32 GB peak on the RTX 3050 against a median 113.00 tok/s in the same 3.32 GB on an H100 — first outside evidence that the method is bound by host-to-device transfer, not by the GPU. (See Known Limitations for what the laptop figure does and does not now mean.)

  Correction, 2026-08-13, left in place rather than rewritten. The measurement stands; the explanation does not. "Bound by host-to-device transfer" was an inference, never a measurement, and a probe on 2026-08-11 refuted it at the published configuration — deleting every host-to-device byte buys 1.4% and the step runs at 71.3% of the card's same-session GEMM ceiling (benchmarks/probe-v0.73.0-what-bounds-streaming.md). What survives is the weaker claim: the constraint is common to both machines and is not the compute the H100 adds.

• Forward bit-exactness at real model sizes, not just on toys: logits torch.equal against a resident reference of matching numerics at 0.5B, 8B, 14B, 32B and 72B. Every previously published bit-exactness result was on 3-layer from-config checkpoints, because a 4 GB card cannot hold a resident 8B to compare against. Backward exactness is a separate claim measured separately — see #331 above and the per-model ledger at the top of the gate record.
• A streamed model is as good as a resident one. Paired over five disjoint training subsets and judged by Soup's own soup ship: mean difference +0.006 against an identical 0.013 within-arm spread, in bf16; +0.0053 against spreads of 0.0333 and 0.0200 in NF4. This had never been measured anywhere in the project.
• Against DeepSpeed ZeRO-3 CPU offload, same box, same data, same model: 2.93× the throughput at 9.7× less peak VRAM at matched numerics. The honest reading is narrow, and the same session establishes it: eight cards of ZeRO-3 are slower than one card training resident for a model that fits. Layer streaming is not "faster than DeepSpeed" — it is for the case where the one card you have is too small.

Known limitations

• The training.seed knob reaches the SFT trainer only (#353). Every other task builds its own TrainingArguments; setting it there parses and does nothing.
• A resident 4-bit run is still not bit-reproducible from a seed (#354), while a streamed one is. get_peft_model builds lora_A before Trainer.__init__ calls set_seed, so the adapter init escapes the seed. Diagnosed with three competing hypotheses each killed by a control; the one-line fix is verified in the gate record but not shipped here. It has a real consequence for soup ship: three runs of one unchanged resident config moved mini_common_sense by 0.375 and mini_mmlu by 0.269 against a forgetting_threshold of 0.05, so five of seven suites can cross the regression line on a re-run that changed nothing.
• The layer-streaming VRAM pre-flight still over-predicts (#327), and LOGITS_BYTES_PER_ELEMENT was deliberately left at 14 rather than lowered to the measured loss-path value. The asymmetry decides it: over-predicting refuses a config that would have worked — visible, annoying, data intact. Under-predicting on Linux is a clean OOM, but on Windows/WDDM there is no exception at all — it silently spills to host memory (measured: 9.27 GB allocated on a 4.29 GB card with nothing raised) and the claim "peak is bounded by one layer" quietly stops being true. The counterfactual settles it: 14 under-predicts 0 of 10 measured rows (worst +0.85% over), the lower value under-predicts 10 of 10 (worst −10.49%). The practical cost is that a streamed DPO run is refused from max_length: 768 on a 4 GB card.
• bitsandbytes still has the defect #331 works around. Soup no longer sends streamed NF4 weights through MatMul4Bit, but the underlying library behaviour — saving tensors outside save_for_backward where checkpointing cannot see them — is upstream and unchanged. Filed as bitsandbytes-foundation/bitsandbytes#2034.
• DeepSpeed + LoRA is repaired in sft.py only (#336). The other trainer wrappers still hit the empty-parameter-group failure under DeepSpeed, a user-supplied --deepspeed my.json is passed through unresolved, and ZeRO++ hierarchical partitioning is unit-tested but never exercised across two nodes.
• utils/sglang.py still has both of the defects the vLLM rewrite fixed — its own hand-rolled prompt and a hardcoded finish_reason. Named rather than silently skipped.
• The closed-loop reward-hacking controller's mechanism is confirmed; its efficacy is not (#286). At 7B the between-mode difference of 0.130 sits inside the within-mode spread of 0.140–0.195. Settling it needs ≥5 seeds per mode.
• Two soup ship leg-2 suites have scoring gaps that outrank capability (#346, #357). mini_tool_call ranks by brace hygiene — Llama-3.1-8B names the right tool 40/40 and scores 0.225 — and mini_mmlu loses 8 of 26 items because extract_mcq_letter does not know \boxed{C}, scoring the 8B at 0.423, below a 0.5B. Adding that one form takes it to 0.731 and the inversion disappears. A third suspected inversion (#356) was withdrawn as a measurement error of our own — it was measured at a 64-token budget where soup ship uses 256.
• The 70B FSDP2 recipe could not be smoke-tested (#41): it needs all eight cards, so it could not be parallelised with anything else in the session.
• The RAM-vs-disk streaming throughput gap remains unmeasured (#325), and layer streaming remains BETA.

A note on the preprint

DOI 10.5281/zenodo.21771064 is unaffected in its correctness claims: its configuration is 8B NF4 at 105 MiB per layer, comfortably below the 163.8–171.5 MiB boundary of #331, and it survives a 50-backward soak at worst_abs = 0.0. What this release changes is scope, not validity — exactness moves from 3-layer from-config toys to resident references at 8B / 14B / 32B / 72B. A version 2 of the record carrying the H100 validation, the resident references, the DeepSpeed comparison and the disclosed defect is in preparation.

One number should be read with a caveat: the published 119.6 tok/s laptop figure was measured before the #331 repair and has not been re-run on repaired code. The repair cost −4.8% throughput at 32B, so treat the laptop figure as a pre-repair number until someone re-measures it on an RTX 3050. An H100 cannot substitute — the whole point of the H100 result is that this method is transfer-bound, so its throughput does not carry across machines.

Correction, 2026-08-13. The conclusion holds, the reason given for it does not: the laptop is not transfer-bound (see the correction above). An H100 still cannot substitute, for a narrower reason — the repair's cost is paid in the per-layer NF4 dequantisation, measured at 9.8% of the step on the laptop, and that share belongs to that card's clock, GEMM ceiling and launch overhead.

[0.72.4] - 2026-08-03

Added — preference losses over layer streaming: DPO, ORPO, SimPO and KTO.

Layer streaming kept the frozen base in CPU RAM and fed it to the GPU one decoder layer at a time, but only for task: sft. This release opens it to the four preference losses. The whole risk was one thing: DPO needs a reference model, and a second model instance would double memory and defeat the feature entirely.

• The reference is the same streamed base with its adapters disabled — one set of weights, one stream, no second pass. Measured on an RTX 3050 4 GB with a 730 MB model: streamed DPO peaked at 0.914x the SFT peak, with a byte-identical RAM store and buffer pool. Forcing a real second instance in the same harness cost +730.44 MB against 730.44 MB of weights — exactly one copy. That control is what makes the first number mean something.
• KTO is not reference-free, contrary to how it is usually described: it selects its reference exactly the way DPO does, so it gets the same treatment and the same memory assertion. ORPO and SimPO genuinely are reference-free.
• Bit-exact against a resident run of the same loss — 0.0 difference for all four, the standard every slot in this series inherits.
• The pre-flight now knows that a paired loss is twice the rows. DPO, ORPO and SimPO concatenate chosen and rejected into one tensor, so a VRAM budget computed at one row per example would have under-predicted by half — and on Windows the consequence is not an error but a silent spill to host memory that makes the run an order of magnitude slower.
• KTO requires batch_size >= 2 (its KL term is degenerate at 1). Soup now says so when your config is read, rather than minutes later after sharding the checkpoint. KTO is streamable at all only because v0.72.3 lifted the batch-1 restriction.
• grpo and ppo remain excluded permanently, not "not yet": generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming depends on. The refusal says so and deliberately names no release.

The streaming setup now lives in one shared place instead of being copied per trainer, so the NF4 pre-flight, the RAM/disk tier choice and the VRAM fit refusal cannot drift between SFT and the preference losses.

Fixed — trl is now capped, and that is a real bug fix, not a CI tweak. Six trainers (bco, dpo, ipo, kto, orpo, simpo) pass max_prompt_length to their trl config, and trl removed it in stages. So anyone who ran pip install 'soup-cli[train]' and resolved to a recent trl had soup train --task orpo fail on import.

Correction (see [0.73.0]). This release shipped the cap as <0.25 on the strength of a staged-removal table that was itself wrong. The real stages are kto at 0.27, bco/orpo/simpo at 0.28 and dpo/ipo at 0.29. v0.73.0 first corrected the cap to <0.27 and then raised it to <0.29 behind a capability-probe compat layer, so the trainers no longer set the cap at all.

That was already true before this release and nothing caught it: the trl imports live inside setup(), which no test had ever called on those wrappers, so CI stayed green while the code only worked on older trl. This release's end-to-end preference tests are what surfaced it.

The boundary was read off the published wheels per config rather than inferred from a version number — the removal being staged is exactly why a single spot-check gives the wrong answer, and see the correction above for how that method can still land on the wrong answer. Supporting the newer API is its own piece of work; declaring a dependency the code actually works with comes first.

Honest costs: streaming makes the reference free in memory, not in time — DPO traverses the layer stack three times per step against SFT's two, measured at 1.52x the layer reads. And the VRAM pre-flight is a sound upper bound for preference losses rather than a tight estimate; see Known Limitations in the release notes.

[0.72.3] - 2026-07-28

Added — layer streaming breadth: more architectures, bigger batches, resume, and a disk tier.

Layer streaming (v0.72.0–.2) was deliberately narrow: Llama/Qwen only, batch 1, no gradient accumulation, no resume, RAM only. This release lifts all of it, and each capability was gated against a streamed-vs-resident bit-exactness reference before any of it was written.

• Six more model families. mistral, gemma, gemma2, gemma3_text, phi and phi3 join the allowlist, each verified bit-exact against the same checkpoint loaded resident, under both bf16 and NF4. Phi-3 is the notable one: it fuses Q/K/V into a single qkv_proj, so there is no q_proj to find, and it is bit-exact anyway. Multimodal gemma3 is deliberately not accepted — only gemma3_text.
• batch_size above 1, and gradient accumulation. Both previously refused.
• A pre-flight VRAM budget that accounts for batch and vocabulary. Streaming bounds the weights; activations and the logits tensor are untouched by it and both scale with batch × seq. On a 152k-vocab model at batch 8 the logits term alone measured 8.71 GB — 146× the entire layer-buffer pool. soup train now predicts peak VRAM and refuses a run that will not fit, naming the two knobs that scale it.
• A throughput forecast, quoted as a range from a GEMM ceiling measured on your own card in that session, alongside the SM clock — never a compiled-in per-card constant.
• --resume and --hf-resume work with streaming.
• A disk overflow tier. stream_source: auto (the default) uses RAM when the base fits and falls back to an NVMe disk tier when it does not, holding nothing resident. stream_source: ram refuses instead of falling back. Non-NVMe disks are still refused outright.
• soup doctor --disk reports the detected media type.

Fixed

• estimate_logits_bytes charged 6 bytes per logit element; the measured peak is 14 (transformers holds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and the fp32 gradient live at once). The old figure under-predicted that term by 2.33×.
• Adapters could not be loaded into a streamed model: load_state_dict narrows keys by child name, so a canonical checkpoint matched 0 of N tensors and PEFT reported only a warning — a resumed run reproduced the from-scratch loss curve exactly. The streaming layer now redirects canonical keys at load time, mirroring the v0.72.1 save-side fix.
• The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
• Streaming weight sources are now released when training ends or raises; the disk tier holds one open shard handle per decoder layer.
• Subprocess helpers resolve tools to absolute paths (on Windows, CreateProcess searches the current directory before PATH).
• The [mcp] extra is now capped at mcp<2. The SDK's 2.0.0 release removed mcp.shared.memory.create_connected_server_and_client_session and dropped Server.list_tools, breaking soup mcp serve's round-trip tests for anyone installing fresh. Support for the 2.x API is tracked separately.

Known limitations

• The RAM-vs-disk performance gap is unmeasured on the development hardware and no number is claimed for it. safetensors memory-maps the shards, so the OS page cache keeps them resident between steps on a machine with spare RAM, and at ~5 effective TFLOPS the NVMe read hides under compute. The disk tier's correctness is verified bit-exact against the RAM tier; its speed relative to RAM is not characterised.
• End-to-end soup train --resume could not be demonstrated on the development box: transformers refuses torch.load below torch 2.6 (CVE-2025-32434), which blocks every resume there, streaming or not. The streaming-specific half — the adapter round-trip and loss continuity — is verified on the production CUDA path.
• Loading into a streamed model works; named_parameters() and state_dict() still disagree in memory, which is the deliberate cost of a serialisation-only design.
• Layer streaming remains BETA.

[0.72.2] - 2026-07-28

Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.

Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of the whole model. It was bf16-only, which capped it at about 3B on a small card. Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B within reach.

Add one line to a streaming config:

training:
  stream_layers: true
  quantization: 4bit    # NF4
  batch_size: 1

Measured on a 4 GB RTX 3050 Laptop (Windows, batch 1, S=512, gradient checkpointing, 50 steps after 10 warm-up, PagedAdamW8bit):

Model	tok/s	Peak VRAM	RAM store	GPU util
Llama-3.1-8B-Instruct	119.6	3.32 GB	3.60 GB (page-locked)	100%
Qwen2.5-3B	264.2	1.76 GB	1.43 GB (page-locked)	100%

For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from the measured rate, not a separate measurement).

Qwen2.5-3B also got 1.85× faster than the bf16 streaming path (264.2 vs 143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the machine's page-locked memory ceiling where a 5.55 GB one did not, which restores asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.

Correctness. A streamed NF4 run is bit-exact against a resident NF4 run: the same quantised bytes through the same bitsandbytes kernels. Logit equality, non-zero gradients at layer 0, and a matching multi-step loss curve are all regression tests, not one-off measurements.

The base is quantised once, offline, and cached under ~/.soup/layer-stream/. The cache is keyed to the quantisation and the source checkpoint, so switching between none and 4bit, or retraining a base in place, re-shards instead of silently streaming the wrong bytes.

Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high. SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only — training was unaffected — but at 8B it would have read ~52 B.

Scope is unchanged and still BETA: RAM tier, task: sft, Llama/Qwen, batch_size: 1, no gradient accumulation, no --resume. Every rejected config names the release that lifts it. quantization values other than none and 4bit are refused.

Fixed — soup --help was 5x slower than it should be. Since v0.72.0 the CLI imported PyTorch on startup, taking 6.0 s where it now takes 1.15 s.

soup reward stress (v0.71.41) put utils/reward_stress on the light CLI path. That module imported utils/reward_hack_control to reuse a single string constant — and reward_hack_control resolves its TrainerCallback base class at module scope, which pulls in transformers and torch. Importing a ~4.4 s dependency for one constant made every soup invocation pay for the training stack, including commands that never touch a model.

Nothing produced wrong results; this was purely startup latency. The light core (pip install soup-cli without the [train] extra) was never broken — it fell back cleanly when torch was absent, just slowly when it was present.

Added tests/test_cli_startup_is_light.py, which asserts the invariant at runtime (import soup_cli.cli must not put torch in sys.modules) instead of inspecting source text for import torch, which is what the previous guards did and why this went unnoticed.

[0.72.1] - 2026-07-27

Fixed — layer-streaming adapters were saved in an unloadable form. If you trained with stream_layers: true on v0.72.0, the adapter that run wrote is inert: every tensor was saved under a key containing an extra .inner. segment, so soup merge, soup serve, soup chat and PeftModel.from_pretrained all loaded zero adapter tensors and silently returned the untuned base model. PEFT emitted only a UserWarning, so nothing failed and nothing looked wrong.

The training itself was correct — the streamed run's numerics are unaffected, and v0.72.0's bit-exactness results still stand. Only the saved file was affected.

If you have a v0.72.0 streamed adapter: re-save or re-run it on v0.72.1. There is no way to recover the original file's association with the base model beyond renaming its keys; re-running is the reliable path. A quick check — if adapter_model.safetensors contains keys with .inner. in them, it is affected:

python -c "from safetensors.torch import load_file; \
print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"

Streamed adapters now save byte-for-byte in the same layout as an ordinary LoRA run, and are portable to any tool that has never heard of layer streaming.

Also fixed — --hf-resume bypassed the streaming resume refusal. The guard only tested --resume, but --hf-resume reaches resume_from through a different branch. That combination previously appeared to work by accident (checkpoint and live model shared the same key shape); once adapters are saved canonically it would instead have matched nothing and continued with a freshly initialised adapter, silently. Both flags are now refused for streaming runs, naming v0.72.3.

Also in this release: every "this lands in vX.Y.Z" refusal message was corrected after the v0.72.x roadmap was renumbered (NF4 streaming is now v0.72.2; the disk tier, wider architectures, larger batches, gradient accumulation and checkpoint/resume are v0.72.3; preference losses are v0.72.4).

Known limitation: in memory the streamed model's named_parameters() still carries the wrapper segment, so loading into a streaming run (--resume) remains unsupported and is refused with a message naming v0.72.3.

[0.72.0] - 2026-07-26

Superseded by v0.72.1 — adapters saved by this version load as zero tensors. The entry below is left as published; the defect and the fix are described under [0.72.1]. Version numbers named as "upcoming" below were also renumbered there (NF4 is v0.72.2, not v0.72.1).

Layer streaming (BETA) — fine-tune models that don't fit in your card. The frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is bounded by the size of one layer instead of the whole model. Only the LoRA adapters, their gradients and optimizer state stay resident. Slower than resident training — but these models did not run on the card at all.

Measured on the development box (RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM), batch 1, gradient checkpointing on, 50 steps after 10 warm-up:

Model	S	tok/s	GPU util	Peak VRAM
Qwen2.5-0.5B	512	978.6	91.4%	1.47 GB
Qwen2.5-1.5B	512	525.0	96.8%	1.82 GB
Qwen2.5-1.5B	1024	487.6	96.7%	2.96 GB
Qwen2.5-3B	512	143.1	79.3%	2.15 GB

Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs. The honest cost: 1.43× slower than resident, measured at 0.5B — the only apples-to-apples comparison available on this box, because 1.5B and above cannot run resident here at all.

Added

• training.stream_layers: true — stream the frozen base layer-by-layer from CPU RAM. training.stream_source (auto/ram/disk) and training.stream_buffers (2–8, default 2 = double buffering) tune it.
• soup_cli/utils/layer_stream.py — tier choice, pinned-vs-pageable decision, architecture allowlist, VRAM/throughput arithmetic (no torch import).
• soup_cli/utils/layer_shard.py — rewrites an HF checkpoint into one safetensors shard per decoder layer, one tensor at a time, so sharding a model that does not fit never needs it to fit.
• soup_cli/utils/layer_stream_runtime.py — buffer pool, CPU-RAM weight source, prefetch scheduler on a dedicated CUDA stream, and the streamed layer wrapper.
• Shards are cached under ~/.soup/layer-stream/ (override with SOUP_LAYER_STREAM_CACHE_DIR) and keyed to the source checkpoint's fingerprint, so a base retrained in place re-shards instead of silently training against stale weights.
• When the base cannot be page-locked, the RAM store falls back to pageable memory and says so, including the measured cost (GPU utilisation ~97% → ~79%).

Changed

• The pre-flight hardware-fit gate is skipped for streaming runs: it models a resident run and would otherwise refuse exactly the runs streaming enables.
• Gradient checkpointing is handled per-layer by the streamer; the HF Trainer's own is left off so layers are not recomputed twice.

Known limitations

• BETA, and proof-of-mechanism at 3B. Nothing above 3B was measured. No 8B/14B claim is supported.
• Scope: RAM tier, bf16, task: sft, Llama/Qwen, batch size 1, no gradient accumulation, no --resume. Every refusal names the release that lifts it.
• 4-bit (NF4) streaming is v0.72.1 — NF4 weights carry a quantisation state and cannot be byte-copied into a plain buffer.
• The disk overflow tier, larger batches, gradient accumulation and checkpoint/resume are v0.72.2.
• The 3B number used a pageable store (this box cannot page-lock 5.55 GB), so it is a lower bound.
• expandable_segments:True is silently ignored on Windows; Soup detects this and does not claim it is active.
• Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.

[0.71.41] - 2026-07-19

soup reward stress: is your reward verifier gameable? Turn the reward-hacking detector on the verifier itself. soup reward synth (v0.71.40) proves a verifier separates your references from friendly perturbations; stress asks the adversarial question a reward-hacking model asks at train time — does the verifier pay out for degenerate junk? It feeds empty, length-padded, repetition, and sentinel-spam completions and flags any the verifier accepts. Pure, offline, exit 0 = robust / 2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a verifier for gameability.

Added

• soup reward stress <reward.py|builtin> [--references golds.jsonl] — adversarial verifier probe. Attacks (--attacks empty,length,repetition,sentinel, --sentinel) are scored against the real gold, so numeric / tool_call / json_schema verifiers get a valid target and still must reject the junk. Reports a per-attack accept-rate table + an overall gameability verdict (--max-gameable, --threshold, --output-report). Loads the target through the existing reward loader, so it probes a synthesized .py and a builtin (accuracy / format / verifiable). A gold-requiring verifier probed with no --references is a hard error, never a false "robust".

Fixed

• Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but are not wired to any command — no data is ever sent today (the previous wording implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.

[0.71.40] - 2026-07-19

soup reward synth: auto-generate a deterministic reward verifier from your data. Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a readable / committable .py reward function, and — the moat — refuses to emit one that can't tell your references from auto-generated bad answers. Nothing in TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is hand-written, hand-picked, or a trained-weights artifact.

Added

• soup reward synth <references.jsonl> -o reward.py — deterministic verifier synthesis. Four families, auto-detected (or pick with --kind): numeric (last-number / \boxed{} / #### extraction, exact or --tolerance), json_schema (induced keys + types + required), regex (positional char-classes over equal-length golds), tool_call (per-tool required/allowed argument binding). The emitted file is self-contained and rides load_reward_fn's existing .py path — no new trusted-exec surface; you read, edit, commit, and diff it.
• Mandatory calibration report — the synthesized verifier is loaded back and run against its own references (must accept ≥90%) and auto-perturbed negatives (must reject). A degenerate always-accept verifier is refused (exit 2), never silently emitted. --plan-only reports the induced spec without writing; --output-report persists the calibration JSON.
• Comma-separated reward_fn ("accuracy,format") now trains — it resolves to a reward ensemble (GRPOTrainer(reward_funcs=[...]), and unlocks the rm_ensemble reward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse time. Fixes a recipe (deepseek-v3-reasoning) that shipped exactly this and previously crashed with Unknown reward function (#311).

Changed / Fixed

• training.reward_fn gains a field validator (null-byte / blank / oversize / empty-comma-segment rejection) — the oldest arbitrary-code field was the least guarded. Comma + verifiable without a verifiable_domain now fails at parse time like the bare verifiable form.
• envs/calculator.py / envs/guess_number.py docstrings corrected: the reward is reward_fn: verifiable + verifiable_domain: math (the bare reward_fn='math' they showed was never valid).

[0.71.39] - 2026-07-19

"CI for weights, not prompts": close the evidence loop. soup ship's verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — turning the soup ci init gate from "edit two numbers in a JSON file" into a reproducible, provenance-bound check that renders on every PR.

Added

• soup ship --emit-evidence <path> — re-serialises the verdict into the --evidence INPUT schema, so a run's output is replayable as input: feeding it back through --evidence (same --forgetting-threshold) reproduces an identical verdict. Output is finally input.
• ShipConfig under eval.ship in soup.yaml + soup ship --config soup.yaml — commit the gate policy (task_eval / task_mode / general_suite / forgetting_threshold / judge_model / baseline) so the verdict is reviewable in a PR diff and reproducible. An explicit CLI flag always wins (CLI > config > default).
• soup ship --push owner/repo#N — post the verdict as a GitHub PR comment (reuses the soup adapters pr --push gh api plumbing). Best-effort: a missing token / gh failure warns but never flips the SHIP / DON'T-SHIP exit code.
• Evidence provenance + staleness gate. With --emit-evidence, --config STAMPS a provenance block (config_sha — a semantic, order-insensitive recipe hash — plus base_model and a best-effort data_sha) onto the evidence. With --evidence alone, --config GATES: it refuses (exit 3) evidence whose config_sha drifted from the committed config, so a PR that changed soup.yaml but forgot to recompute its evidence is caught. The gate policy (eval.ship) is EXCLUDED from the hash, so tuning forgetting_threshold never falsely invalidates evidence about an unchanged model.
• soup ci init --config <soup.yaml> — binds the generated gate's soup ship step to the committed config (provenance/staleness enforcement in CI).

Changed

• Exit-code note: soup ship --config usage / staleness errors are exit 3 (usage), preserving 0 = SHIP, 2 = DON'T SHIP, 1 = runtime from v0.71.38.

Security

• provenance.config_sha read from untrusted evidence is shape-validated as a hex digest before being echoed (a raw value could smuggle terminal ESC bytes past rich.markup.escape).
• provenance.data_sha hashes the training file through an O_NOFOLLOW fd with a symlink-rejecting containment check and an 8 GiB cap (was an unguarded hash_file).
• soup ci init's path validation now rejects # and the YAML 1.1 line breaks (NEL / LS / PS), closing a plain-scalar comment-truncation of the generated run: step (also hardens the pre-existing --data / --suite / --evidence args).

[0.71.38] - 2026-07-17

soup ship's regression leg now has teeth. Leg 2 (the catastrophic-forgetting / regression gate that carries the whole SHIP / DON'T-SHIP claim) was 15 hand-written trivia prompts scored by case-insensitive substring containment — it scored "B" for "Berlin", "ok" for "look", "3" for "13", and had zero items for tool-calling, safety, or JSON validity. This release makes the gate real: a fixed, extraction-based scorer + bundled, offline, zero-dep eval suites that catch a regression the old gate waved through.

Changed

• Fixed answer scorer (breaking — verdicts can change). soup ship's leg-2 MCQ / instruction / arithmetic answers are now scored by answer-extraction
  • a boundary-aware match, replacing the raw substring test. A spurious substring inside another word ("Berlin", "look", "13") no longer scores a correct answer, so an existing run's verdict may flip — intentionally, because the old gate was reporting false negatives.
• Bundled, offline general suite. The default --general-suite is now seven hand-authored suites shipped in the wheel: mini_mmlu / mini_common_sense / mini_instruction (expanded), a new mini_arithmetic, and three behavioural suites the old gate had no coverage for — mini_tool_call (function-calling), mini_format_json (JSON validity), and mini_safety (refusal-rate). Each is scored to a per-model absolute score by the pure scorers Soup already ships (eval/custom, utils/diagnose); no lm-eval, no network, no download. Every suite is large enough that a single-item flip (1/N < 0.05) trips the default threshold instead of being rounded away.
• soup ship exit codes: usage errors moved 2 → 3. Exit 2 now means only DON'T-SHIP; a typo'd flag or bad --general-suite exits 3 (mirroring soup plan / soup env check), so CI can tell a config error from a caught regression. Offline --evidence read/parse errors stay 1. Breaking for anyone parsing exit 2 as "usage error".

Fixed

• soup ship help + docstrings no longer describe --task-mode pairwise as "reserved for a later release" (it shipped in v0.71.31); the dead SUPPORTED_TASK_MODES gate is removed.
• soup diagnose's package docstring said "Six" probes (there are seven — citation) and pointed live loading at an unshipped version; it now re-exports all seven score_* probe functions so callers need not reach into submodules.

[0.71.37] - 2026-07-17

Every pip install soup-cli[extra] command now works on Windows cmd.exe, and eval-gate benchmark tasks run instead of always failing.

Fixed

• Install hints are now quoted so they work in every shell. Soup printed pip install 'soup-cli[ui]' — bash / zsh / PowerShell syntax. cmd.exe has no single-quote quoting, so it hands the quotes to pip verbatim and pip refuses:

  ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier
  

  Every hint, README command, and docs example now uses pip install "soup-cli[extra]", which works in cmd, PowerShell, bash, and zsh alike — the same spelling the repo already used for pip install -e ".[dev]". Measured on Windows: single quotes fail only on cmd.exe; double quotes pass everywhere; dropping the quotes passes on Windows but breaks zsh, which globs the bracket.

  Nothing in Soup can rescue the command after it is typed — pip and the shell own it, and Soup is not installed yet when the README command runs — so the fix is the spelling we print. A regression test now scans the package and every docs code block for the single-quoted form.

  If you followed an older tutorial and hit Invalid requirement, swap the ' for "; nothing is wrong with the package.

• Eval-gate type: benchmark tasks now actually run. eval/gate.py probed for a forgetting.run_mini_benchmark helper that never existed, so every type: benchmark task in a gate suite failed 100% of the time — while advising an [eval] extras install that could not fix it. The gate now calls ForgettingDetector directly (the same way soup ship already did), and an unknown benchmark name fails with the list of valid names. Thanks @Sanjays2402! (#315, closes #310)

[0.71.36] - 2026-07-16

Data Moat II — a semantic layer over your training data, plus two tools for what a fine-tune forgets and leaks.

Added

• soup data dedup --semantic — near-duplicate removal over embedding cosine instead of MinHash shingling. Catches reworded duplicates that MinHash misses (measured: 0.88–0.91 cosine on rewordings MinHash scored as distinct) while correctly keeping distinct-but-similar instructions. Zero new dependencies: uses transformers from the [train] extra. Read the known-limitation below before lowering --threshold.
• soup data topics <data> — cluster a dataset and label each cluster with c-TF-IDF terms, plus a coverage table (82% code · 6% math) and a warning for thin topics. Labels are emergent term clusters, not a fixed taxonomy.
• soup data canary insert|check — Secret-Sharer memorization probe. Insert K high-entropy secrets, then check any model/adapter: each secret's loss is ranked against never-inserted controls drawn from the same space. Exit 2 on MAJOR so CI can gate. Measured on SmolLM2-135M: a memorized set lands at percentile 0.0 (loss 1.7–2.5) against a clean model's 4.1–6.2.
• soup train --replay old.jsonl --replay-ratio 0.1 — continual-learning rehearsal. Interleaves a seeded sample of an old dataset into training so a new task does not erase the old one. r is the fraction of the final mixed set (n_replay = round(r/(1-r) · n_new)), rows are interleaved rather than appended, and an undersized pool reports the shortfall instead of repeating rows. Mixed into train only — validation stays pure new-task.

Fixed

• The hardware-fit gate refused to train any local checkpoint. A merged model (soup merge -o ./mymodel) has no size marker in its name, so the size guesser returned its 7B default, predicted ~16 GB of VRAM and refused. Local checkpoints are now measured from their safetensors header (0.135B actual vs 7.0B guessed) — this had blocked soup merge → train-from-merged entirely. Third instance of this class after the v0.71.32 (Whisper) and v0.71.33 (M suffix) fixes.
• pip install 'soup-cli[extra]' hints printed without the extra. Rich ate the bracket, so every "install the missing dependency" message across 17 sites told users to run pip install 'soup-cli' — which succeeds and still leaves the feature broken. Affected [eval], [data], [serve], [ui], [tui], [compile], [mcp], [carbon] and others, including Typer help text.
• Replay rows bypassed the image/audio path-traversal validation that the primary dataset receives.

Known limitations

• Semantic dedup is not a paraphrase detector. Measured with all-MiniLM-L6-v2, paraphrase cosines (0.49–0.76) overlap with genuinely-distinct rows (0.54–0.76): "Add two numbers" vs "Multiply two numbers" scores 0.759, higher than the true paraphrase "reverse a string" / "invert the order of characters" at 0.491. No threshold separates them, so lowering --threshold to chase paraphrases deletes real training rows. The default (0.8) is deliberately conservative.
• Replay is validated at proof-of-mechanism scale. On SmolLM2-135M + LoRA, replay retained the old task 7% better than a no-replay control — the correct direction — but forgetting without it was only +4%, i.e. mild. The effect size at full fine-tuning or 7B+ is unproven on a 4 GB box.
• Canary exposure is the sampled-control approximation, not full-space rank enumeration. "No exposure" is not proof of no memorization.
• data topics / dedup --semantic require [train] (torch) and download an embedding model. Plain MinHash dedup stays on the light core.
• Replay v1 is sft/pretrain only and is incompatible with packing/multipack.

[0.71.35] - 2026-07-15

Added

• Compliance templates — soup init --template hipaa|soc2|eu-ai-act|sr-11-7. Four regulation-shaped starting configs. Soup's compliance controls are CLI flags/commands rather than config keys, so each template is a valid training config plus header comments naming the exact commands for that regime (PHI scrubbing + air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI + energy tracking for the EU AI Act, repro-receipt + diagnose/ship for SR 11-7). Templates default to a license-clean Apache-2.0 base.
• soup card <registry-id> -o MODELCARD.md — model-card autogen. Turns a Local Model Registry entry into a publishable, provenance-carrying HF model card: base model, training config, eval scorecard, config/data hashes, lineage (ancestors) and a table of every registered artifact. Adapter vs full-model is inferred from registered artifacts, falling back to the training config (LoRA rank, with Spectrum/LISA full-FT correctly treated as dense), so the card sets the right library_name and never misreports the model type.
• soup push --card <registry-id> — render that registry-driven card and upload it as README.md, overriding the auto-generated one. A bad ref fails fast before any network call; HF hub only.
• soup ci init — fine-tuning CI. Writes .github/workflows/soup-gate.yml, a PR gate chaining soup data validate → soup expect → soup ship --evidence (exit 2 blocks the merge). Every interpolated path is validated to stay under the repo root and shell-quoted; the branch and Python version are regex-gated; the write is atomic, symlink-rejecting, and refuses to clobber an existing workflow without --force.
• Compliance quickstart — a new docs/compliance.md walkthrough: template → PII scrub → train with receipt/Annex XI/energy → registry → BOM + attestation → scan/sign/verify → air-gap → model card → CI gate.

Fixed

• GGUF export now actually works on Windows (validated end-to-end against a locally-built llama.cpp: SmolLM2-135M → q4_0 / q4_k_m / q8_0 / f16 → soup deploy ollama → live inference). Four real bugs, each of which independently broke the path:
  • soup export --format gguf cloned llama.cpp into your current directory. SOUP_DIR is the bare name .soup, but the lookup used it relatively rather than anchoring to ~ like the rest of the codebase — so the canonical ~/.soup/llama.cpp was never found and a fresh ~200 MB checkout was dropped into whatever directory you ran from.
  • The first GGUF export downgraded your PyTorch and broke CUDA. The auto-clone ran pip install -r <llama.cpp>/requirements.txt into your interpreter, and llama.cpp pins torch~=2.2.1 against the CPU wheel index (observed: torch 2.5.1+cu → 2.2.2+cpu, transformers 4.57 → 4.46). Soup now installs only the convert script's extra dependencies, unpinned, and never touches torch.
  • A correctly-built llama.cpp was not found on Windows. MSVC (like Xcode) is a multi-config generator and emits build/bin/Release/llama-quantize.exe; only the flat single-config layout was searched.
  • soup deploy ollama failed on a relative GGUF path with "pull model manifest: file does not exist" — Ollama resolves FROM against the Modelfile's directory, and Soup writes the Modelfile to a temp dir. The Modelfile now emits an absolute path.
• Model-card injection hardening (affects the pre-existing soup push card too). The ## Training section interpolated base / task / scheduler / recipe unescaped. Since SoupConfig.base and scheduler have no charset validator, a crafted-but-valid config could smuggle raw HTML — or a backtick breaking out of the surrounding code span — into a card published to the Hub. All values now go through the markdown escaper, which additionally neutralises backticks and strips C0/ESC control bytes.

[0.71.34] - 2026-07-15

Added

• soup adapters arithmetic — task-vector algebra over LoRA adapters (add / scale / negate). Apply task arithmetic (arXiv:2212.04089) to LoRA deltas via an expression such as "coder + 0.5*math - toxic", mapping names to adapter dirs with repeatable --adapter name=path. Produces one merged adapter you can serve or merge.
  • Signed, un-normalized element-wise combine over same-rank adapters; the effective delta ΔW = B @ A scales linearly with each coefficient (negation flips the delta, 0.5· halves it) via a √|c| factor split — not the c² a naive sum gives. Mixed-rank inputs are refused with a clear "harmonize rank" message.
  • Reuses the backdoor-scan gate (refuses a FAIL-scanned input unless --allow-unscanned) and a same-base-model check (--allow-cross-base to override). Hand-written expression parser (no eval), cwd-contained/symlink-rejecting paths, exit 0 = ok / 1 = refusal.
• LISA — Layerwise Importance Sampled AdamW (arXiv:2403.17919). Full-fine-tuning quality at LoRA-like memory: every N steps LISA freezes all decoder layers except a small random set (embeddings + head always trainable). Enable with training.lisa_enabled: true (+ lisa_num_layers, lisa_interval_steps) on a task: sft, transformers, text, quantization: none run; mutually exclusive with LoRA features and the other freeze mechanisms. Live on a 4 GB GPU for small models.

[0.71.33] - 2026-07-13

Added

• soup draft — train and, above all, MEASURE a speculative-decoding draft model.

  • soup draft measure --target <m> --draft <d> --prompts p.jsonl reports a draft's acceptance rate (the fraction of the target's own greedy tokens the draft would have proposed correctly) plus real plain-vs-assisted throughput. This is the honest gate: it tells you whether speculative decoding is worth enabling before you ship it. Exit 0 / 2 (below --min-acceptance, for CI) / 1.
  • soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/ distils your target into the tiny base (logit KD via the existing task: distill trainer) and emits a dense draft model, ready to load as an assistant_model.
  • soup draft list, plus a local draft registry (~/.soup/drafts.json) that soup serve --auto-spec consults before the built-in pairing table — so a draft you trained yourself is picked up automatically.
  • Draft and target must share a tokenizer; a mismatched pair is refused up front (speculative decoding proposes draft token ids into the target's vocabulary, so a mismatch silently produces garbage rather than failing).

  Read the measured results before you use this — see Known limitations below. On the validated pair, distillation did not improve acceptance, and speculative decoding was a net slowdown. soup draft measure is what tells you that.

Known limitations

• Distilling a draft did not improve its acceptance rate on the validated pair. Measured on SmolLM2-360M-Instruct (target) with a SmolLM2-135M-Instruct draft: the stock draft already scored 69.3%, and distilling it moved that to 69.7% after 2 epochs and back to 69.3% after 10 epochs — i.e. no gain beyond noise. A small same-family draft is already near its capacity ceiling for agreeing with the target, and logit KD cannot buy capacity it does not have. Whether distillation materially raises acceptance for a genuinely diverged fine-tune, or a larger target/draft pair, is unproven on a 4 GB box — tracked as a scale issue.
• Speculative decoding was a net slowdown on that pair (measured 0.55–0.64×): the draft's forward pass costs more than the tokens it saves at this size. soup draft measure reports this truthfully rather than assuming a speedup — which is precisely the point of shipping the measurement.
• Acceptance is teacher-forced greedy agreement, the metric the Medusa/EAGLE papers report. It is exact and deterministic, and it is the right number for comparing drafts — but it is not the accepted-token count of a sampling run, which also depends on the rejection-resample cascade.
• Same-tokenizer only. Cross-tokenizer drafts (ULD-aligned + universal assisted decoding) are deferred.

Changed

• PRM-guided GRPO scores completions in a single batched forward. PRMScorer.__call__ now right-pads all completions into one [B, T] tensor (+ attention mask) and runs one output_hidden_states pass instead of one forward per completion, cutting per-step reward latency on non-tiny models. Numerically identical to the per-completion path (parity + mixed-length tests). Closes #298 (#301 by @Ekaanksh-dev).

Fixed

• The hardware-fit gate no longer refuses to train small models. model_size_from_name did not understand an M (millions) suffix, so SmolLM2-135M fell through to the 7B default, was predicted to need ~14 GB of weights, and was blocked. Every draft-sized model hit this. 1.7B was also being read as 7B (the 7b marker matched inside 1.7b), while Qwen2.5-7B-Instruct-1M correctly stays 7B (1M is the context, not the parameter count).
• soup serve --backend vllm no longer force-enables trust_remote_code. The vLLM path now goes through the same --trust-remote-code default-deny gate (and warning panel) as the transformers backend, so serving an untrusted repo never executes its code silently.
• Multi-adapter serving (soup serve --adapters name=path) now actually switches adapters. The named adapters are loaded into the model and selected per request (via POST /v1/adapters/activate/{name} or the request adapter field); previously every request silently ran the startup model. The base model is served when no adapter is selected.
• Vision datasets reject out-of-directory image paths. llava / sharegpt4v rows are containment-checked against image_dir (mirroring the audio loader), so a crafted {"image": "/etc/passwd"} row can no longer read arbitrary local files.
• soup train --dry-run --gpus N no longer launches a real multi-GPU run. The accelerate re-exec is skipped under --dry-run. The re-exec also now forwards --minillm-on-policy, --capture-activations, and --capture-prompts (previously dropped on multi-GPU runs).
• MLX SFT now builds a real optimizer (AdamW from the configured LR) instead of passing optimizer=None, which left the model untrained.
• soup data inspect / preview / search escape dataset- and Hub-derived text so a stray [/] no longer crashes the command and a crafted [link=…] tag can't render a phishing hyperlink. soup runs list/show escape config-derived fields too.
• soup infer --task asr hardening — an oversized reference no longer crashes the whole batch after transcription (that row's metric is skipped); an all-skipped run exits non-zero instead of reporting success; --asr-task is validated upfront; and dataset-derived filenames are control-stripped before printing. ASR training now caps transcript labels to Whisper's decoder limit, warns on >30 s audio, and picks fp16 on pre-Ampere GPUs instead of hardcoding bf16.
• Knowledge-distillation KD term aligns with the CE term. The token-level divergence is now computed over causal-shifted positions, so the distillation signal covers exactly the trained tokens (previously off by one).
• Miscellaneous robustness — load_config_from_string raises ValueError (not TypeError) on a non-mapping YAML document; the Web UI Bearer-token check is constant-time; and soup doctor --vscode / the LR-finder report use the centralised atomic, symlink-rejecting writer.

[0.71.32] - 2026-07-07

Added

• ASR fine-tuning (task='asr', Whisper) — fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) / base (74M) train on a 4 GB GPU.
  • New AsrTrainerWrapper (HF Seq2SeqTrainer + WhisperProcessor); data rows are {"audio": <path>, "text": <transcript>} under the new data.format='asr'. Audio decodes via the hardened load_audio_mono (16 kHz mono, soundfile pre-probe + O_NOFOLLOW + symlink/size guards); transcripts become decoder labels (pad → −100, decoder-start token stripped).
  • Optional LoRA on q/v attention projections via training.asr_lora: true (default full fine-tune); training.asr_language / training.asr_task (transcribe|translate) set the decoder prefix and are persisted to an asr_generation.json sidecar so inference restores them.
  • soup infer --task asr — transcribe an {"audio": path[, "text": ref]} JSONL; reports per-row and corpus WER/CER when references are present. Loads a full model or a PEFT/LoRA adapter dir. Flags: --asr-language, --asr-task, --audio-dir (audio paths are cwd/dir-contained; UNC/traversal rejected).
  • New pure-python utils/asr_metrics.py — WER / CER / word_accuracy (= 1 − WER, for a higher-is-better ship metric leg) / corpus_wer, with a light Whisper-style text normalizer (no new dependency).
  • Recipes: whisper-tiny-asr, whisper-base-asr (live-trainable), whisper-large-v3-asr (parse-only, needs a larger GPU), plus smolvlm-256m-sft (vision). Catalog 138 → 142.

Fixed

• model_size_from_name now knows Whisper checkpoint sizes (tiny…large), so the hardware-fit gate no longer mistakes a 39M whisper-tiny for the 7B default and blocks ASR training on consumer GPUs.

[0.71.31] - 2026-07-06

Added

• Judge-in-the-loop suite — put an LLM judge in the loop across the workflow:
  • task='online_dpo' — Online DPO training (wraps TRL OnlineDPOTrainer): the model generates two completions per prompt on-policy each step and a judge (a pairwise LLM judge over the existing ollama/openai-compatible backend) OR a reward_model picks the winner. Config: training.online_dpo_judge: "ollama://model" (or set reward_model — exactly one), online_dpo_loss_type: sigmoid|ipo, online_dpo_max_new_tokens; beta reuses dpo_beta. Transformers + text only. Recipe: online-dpo-smollm2-135m. Adapts to the installed TRL: on trl 0.19.x the judge is a swap-debiased pairwise comparison; on trl 1.x (which removed pairwise judges) the same JudgeEvaluator is used as a pointwise reward function — a documented per-version behaviour difference.
  • soup data best-of-n — Best-of-N rejection sampling (BOND-lite): sample N completions from --base locally, a --judge scores each pointwise, and the winner is written as an SFT chat row (with provenance). --emit-pairs also writes winner-vs-loser DPO pairs.
  • soup data evolve — Evol-Instruct instruction evolution (WizardLM depth / breadth) over an ollama/vllm provider, completing the synthetic-data suite (Magpie / Forge / Persona / evolve).
  • soup ship --task-mode pairwise — a true pairwise judge win-rate as the ship leg-1 task-win (base = 0.5 coin-flip, tuned = its win-rate; swap-debiased), fusing with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.

Security

• soup data best-of-n / evolve write outputs via atomic mkstemp + os.replace (re-validated cwd containment), closing the TOCTOU symlink-swap window between the containment check and the write. All judge/provider URLs are SSRF-validated; model loads probe trust_remote_code.

[0.71.30] - 2026-07-05

Added

• PRM-guided GRPO — use a trained Process Reward Model as the per-step reward inside GRPO (the o1-era process-supervision signal). Set training.prm_reward: <PRM dir|id> (a model produced by soup train task=prm) and training.prm_aggregate: min|prod|last; the PRM splits each generated completion into reasoning steps, scores every step with its reward head, and folds the per-step scores into one scalar reward that GRPO optimises. The PRM reward replaces reward_fn and rides the existing reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller observes it. Cross-validators gate task='grpo' + backend='transformers' + modality='text'. Default aggregation is min (weakest-link); prod assumes calibrated [0,1] step scores.
• Bundled rollout environments — three pure-Python toy environments (soup_cli.envs.calculator / retrieval_qa / guess_number) exposing a rollout(prompts) entry point so the live openenv GRPO rollout path runs out-of-the-box: training.rollout_backend=openenv + training.rollout_func=soup_cli.envs.calculator:rollout. Three ready-made recipes added (grpo-env-calculator / grpo-env-retrieval-qa / grpo-env-guess-number); catalog 134 → 137.

Fixed

• soup train task=prm producer conformance (surfaced by the v0.71.30 live smoke): the PRM trainer now casts its reward head to the base-model dtype (bf16 CUDA runs previously crashed on the first compute_loss), saves the tokenizer alongside the model (so a PRM checkpoint is loadable standalone), and returns the standard trainer-result shape (previously the CLI crashed with a KeyError: 'initial_loss' right after saving).

Notes

• Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny synthetic PRM and synthetic reward — not a production reward-model claim (scale ask tracked in #286). The bundled environments are deterministic single-shot seeders, not interactive multi-turn model-in-the-loop episodes (the live openenv contract passes only prompts). Step split is a newline heuristic; PRM completions are scored one forward pass each.

[0.71.29] - 2026-07-05

Added

• soup shrink — depth-prune a model + optional distill-heal ("The Unreasonable Ineffectiveness of the Deeper Layers", arXiv:2403.17887). Ranks decoder layers by the angular distance of the residual stream across a contiguous block over a calibration set, drops the least-important block (first and last layer always protected), optionally heals by distilling the original model into the pruned student, and emits a single dense smaller model plus a one-screen SHIP / DON'T SHIP perplexity verdict. soup shrink --model <id|path> [--drop-ratio 0.25 | --drop-layers N] --calib <calib.jsonl> [--heal <heal.jsonl> --heal-steps N] [--tolerance 0.10] [-o <dir>] [--device cpu] [--attach-to-registry <id>] [--plan-only]. Exit codes: 0 = SHIP, 2 = DON'T SHIP, 1 = error. The heal runs as an isolated soup train subprocess (LoRA-student logit distillation) and the adapter is fused back so the shipped artifact stays a single dense model. Arch allowlist v1: Llama / Qwen / SmolLM. Validated live on SmolLM2-135M (drop 25 %: 30 -> 22 layers, ppl x2.98 unhealed; drop 4 + heal: ppl x1.35, recovered).

Security

• soup shrink contains --calib / --heal / --output-dir (and every derived write path: <out>/model, <out>/heal_adapter, the fuse staging dir) under cwd with os.path.realpath + commonpath + O_NOFOLLOW + symlink rejection, re-validating derived paths right before each write (TOCTOU). The heal subprocess uses an argv list (no shell) with a timeout; its config is schema-validated before spawn; subprocess output is C0/ESC-stripped before it reaches the terminal. --model defaults trust_remote_code=False with a probe + warn.

[0.71.28] - 2026-07-04

Added

• MCP server (soup mcp serve) — drive Soup from any Model Context Protocol client (Claude Code / Cursor / Cline / Continue) over stdio. No fine-tuning CLI ships an MCP server. Exposes 14 read-only tools as JSON — advise, data_inspect / data_validate / data_score / data_doctor, recipes_search / recipes_show, runs_list / runs_show, registry_list / registry_show, profile, diagnose_evidence, ship_evidence — plus two plan-only mutating tools (train_start, export) gated behind --allow-mutating (they render the exact command that would run; they never execute). The official mcp SDK is behind a new [mcp] extra (pip install 'soup-cli[mcp]'), lazy-imported so the core CLI stays light. Security: stdio-only (no network listener); every path argument re-enters cwd-containment + symlink rejection; output is control-char sanitized; errors are path-free; string / size / int bounds enforced.

Fixed

• The DPO / IPO / KTO / BCO trainers now apply configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) via the shared apply_vocab_expansion() helper during setup, consistent with the SFT path — they previously ignored it. Closes #292 (#293 by @CODING-DARSH).
• The ORPO / SimPO / GRPO trainers now also apply configured vocabulary expansion via the shared apply_vocab_expansion() helper — completing consistent vocab-expansion behavior across every SFT/preference/RL trainer. Closes #294 (#295 by @CODING-DARSH).

[0.71.27] - 2026-07-03

Added

• Fine-tune Doctor — kill the top silent fine-tune failures before a single training step; no competitor (Unsloth/Axolotl/LlamaFactory) ships any of these three:
  • soup data doctor <data> --model <id|path> — chat-template compatibility report over 8 checks: chat_template present, template_renders cleanly, has {% generation %} markers, eos_in_labels (the #1 "model never stops generating" bug — every assistant turn's trained span must actually contain an EOS/EOT token; checks every turn, not just the last), bos_duplication (template + tokenizer both prepending BOS), system_role support (Mistral-style templates reject a leading system turn), unknown_roles, and truncation_risk (p95 rendered length vs max_length). Same OK / MINOR / MAJOR taxonomy as soup diagnose; exit 0 = OK/MINOR, exit 2 = MAJOR. --train-on-responses-only / --train-on-messages-with-train-field select the same masking strategy soup train would use, so the report and --show-mask never disagree about what's actually trained.
  • soup data doctor ... --show-mask N — render N sample rows with per-token trained/masked colouring through the REAL collator path (answer-only / per-message-train-field / RAFT span-mask) — not a reimplementation — so an assistant-mask bug is visible instantly.
  • soup data lint <data> — preference-data linter for dpo/orpo/simpo/ipo/bco/kto: length_bias (chosen systematically longer than rejected — the #1 silent DPO degradation, reported as a Cohen's d effect size), label_imbalance (KTO desirable:undesirable ratio), near_duplicates (MinHash/LSH, reuses the soup data dedup kernel), identical_pairs (chosen == rejected — zero preference signal), and prompt_leak (the prompt echoed verbatim inside the completion — a common synthetic-data pipeline bug). Optional --model for exact token-length bias (default: word count).
  • Validated live against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer on Windows + RTX 3050 — this smoke pass found and fixed two genuine bugs beyond what synthetic fixtures alone caught: an EOS check that required the EOS token to be the literal last trained token (real templates often have a trailing formatting token after the closing tag that stays inside the trained span), and two call sites that only caught (ValueError, TypeError) around a tokenizer's apply_chat_template when a real Jinja raise_exception() (Mistral-style no-system-role guard) raises jinja2.exceptions.TemplateError.

Fixed

• Harden commands/diagnose.py's --evidence loader against a TOCTOU symlink swap: opens with O_NOFOLLOW and size-checks the open fd via os.fstat instead of os.path.getsize on the path before the open — backports the hardened loader shipped for soup ship in v0.71.25 (closes v0.71.25 known-limitation (4)).
• Harden judge-model URL validation against a hostname prefix bypass (http://localhost.attacker.com) — GateTask._valid_judge_url / _parse_judge_url now use urllib.parse.urlparse + hostname checks instead of startswith. Closes #283 (#288 by @CODING-DARSH).
• SFTTrainerWrapper now applies configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) and resizes the model embeddings during initialization — previously these fields were accepted by the schema but silently ignored. Closes #289 (#287 by @CODING-DARSH).
• Vision and audio SFT paths now apply that same configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) via the shared apply_vocab_expansion() helper, consistent with the text SFT path — they previously ignored it. Closes #290 (#291 by @CODING-DARSH).

Security

• soup data doctor strips C0 control characters (keeping tab/newline/CR) from dataset-derived content before it reaches the terminal — Rich's markup.escape() only neutralises [...] tag syntax, not raw escape sequences, so an untrusted training row (e.g. an unknown role field, or --show-mask's decoded token text on a byte-level BPE tokenizer) could otherwise carry a literal ESC byte through to the terminal (title-bar / OSC-8 link spoofing, or obscuring a MAJOR verdict via cursor tricks). --output JSON is unaffected (json.dumps already escapes control characters).

[0.71.26] - 2026-07-01

Added

• Closed-loop reward-hacking auto-mitigation. The trainer now detects reward hacking mid-run and self-corrects — instead of only halting. Set training.reward_hack_mitigation (or soup train --reward-hack-mitigation) to one of four modes on a GRPO/PPO run (requires reward_hack_detector):
  • log_only — instrument only: append a per-step mitigation_log.jsonl (drop_pct, verdict, reward mean/std, completion-length trend, repetition) and never touch training.
  • kl_control — a reversible bang-bang + hysteresis controller: when the hacking signal trips, raise the KL coefficient β (geometric, clamped to [floor, ceil], never crossing 0); relax it when the signal recovers. Dwell + release-patience prevent flapping; a multi-signal vote combines the detector drop with a length-trend and repetition signal.
  • pid_lagrangian — a PID-Lagrangian controller (Stooke et al.) that holds the hacking signal at a target, plus an escalation ladder: raise β → roll back to the last-good RL checkpoint → early-stop.
  • Anti-gaming hardening: per-signal EMA/median smoothing, conservative-on-disagreement voting, a reward-distribution-drift guard, and optional bounded reward shaping on the gamed proxy (length / repetition / sentinel). A plain-English give-up explanation is logged on early-stop.
  • Proof-of-mechanism only (see Known Limitations): validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 — all four stages pass live, including a real mid-run rollback. PPO ships BETA (mechanism unit-tested; the on-GPU proof is GRPO-only).
• Ready-made qwen2.5-coder-7b-sft recipe for Qwen/Qwen2.5-Coder-7B-Instruct (catalog 133 → 134) (#285 by @Deadpool2000).

Security

• RLCheckpointCallback.restore_checkpoint / save_checkpoint refuse a symlinked optimizer.pt — torch.load(weights_only=False) on an attacker-placed symlink in a shared checkpoint dir was an RCE vector.
• Bool-before-int/float guards on every new reward_hack_* numeric field; reward_hack_signals bounded (max_length=4); the mitigation log writer is cwd-contained with symlink-reject-on-rotate and secret redaction.

[0.71.25] - 2026-06-27

Added

• soup ship — the SHIP / DON'T-SHIP verdict. After fine-tuning, answer one question: did the model get better, or did I break it? soup ship fuses two checks into a single binary decision — leg 1: the task metric strictly improved (base → tuned); AND leg 2: no general benchmark regressed past a forgetting threshold (default 0.05 absolute points). It SHIPs only when both hold — otherwise DON'T SHIP, even if the task metric looks great. The output is a one-screen verdict + the reason, with CI-gateable exit codes (0 = SHIP, 2 = DON'T SHIP, 1 = runtime error).
  • Leg-1 modes: --task-mode metric (eval accuracy) or judge_score (LLM-as-a-judge); pairwise win-rate is planned for a later release.
  • Leg-2 suite: built-in mini benchmarks by default (offline, CPU), or --general-suite <names> to route lm-eval benchmarks; --baseline registry://… | file.json supplies recorded base scores.
  • --evidence ev.json decides offline from pre-computed scores (no model load); --output verdict.json persists the machine-readable verdict.
• Friendlier error messages: the CUDA-OOM hint now also suggests gradient_checkpointing and 4bit quantization, plus new mappings for Hugging Face gated repos (huggingface-cli login / HF_TOKEN) and trust_remote_code errors. Closes #272 (#282 by @Akshaya-reddy18).

Security

• soup ship input hardening: --evidence is opened with O_NOFOLLOW + an fstat size cap (16 MiB) under cwd containment; --task-eval is cwd-contained and symlink-rejected; --judge-model is validated by scheme/host via urlparse (blocks the http://localhost.attacker.com prefix bypass); lm-eval model ids reject ,/= injection; --general-suite is bounded (≤ 50 names, ≤ 256 chars each).

[0.71.24] - 2026-06-21

Added

• 2026 model-family recipe expansion (catalog 116 → 133). 17 new ready-made SFT recipes for the open-weight models released Feb–Jun 2026, each with its Hugging Face repo-ID verified to resolve:
  • Qwen 3.5 (Apache-2.0): qwen3.5-0.8b-sft, qwen3.5-2b-sft, qwen3.5-4b-sft, qwen3.5-9b-sft, qwen3.5-27b-sft, and the qwen3.5-35b-a3b-sft / qwen3.5-122b-a10b-sft / qwen3.5-397b-a17b-sft MoE sizes.
  • Qwen 3.6 (Apache-2.0): qwen3.6-27b-sft, qwen3.6-35b-a3b-sft.
  • DeepSeek-V4 (MIT): deepseek-v4-flash-sft, deepseek-v4-pro-sft.
  • GLM (MIT): glm-5.1-sft.
  • Kimi (Modified MIT): kimi-k2.5-sft, kimi-k2.6-sft.
  • MiniMax (MiniMax Community License — commercial use needs a separate agreement): minimax-m3-sft.
  • Mistral Large 3 (Apache-2.0, 675B/41B-active multimodal MoE): mistral-large-3-sft.
• Unit-test coverage for the warmup.py auto-warmup-steps helper (#274 by @shatakshi-1404).

Fixed

• Stale recipe repo-ID: glm-5-sft now points at zai-org/GLM-5 (the org migrated from THUDM).

[0.71.23] - 2026-06-12

Added

• Native Spectrum targeted training (closes #266). A new soup spectrum scan reads a model's .safetensors shards one tensor at a time (no model load — peak RAM is the largest single weight matrix), computes a singular-value SNR per weight matrix with a Marchenko-Pastur noise threshold (arXiv:2406.06623), ranks layers within each module-type group and prints the top --top-percent as a ready-to-paste training.unfrozen_parameters YAML block. This lets you scan even a very large model's layer SNR on a CPU box and then full-fine-tune only the high-signal layers.
  • soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn|all] [--output patch.yaml] — SNR table + the YAML patch; results cache at ~/.soup/spectrum/<slug>.json (override via SOUP_SPECTRUM_CACHE_DIR).
  • New schema field training.unfrozen_parameters: list[str] — regex patterns of parameter names to keep trainable; the SFT trainer freezes every parameter then unfreezes the matched set (full fine-tuning, LoRA off). Mutually exclusive with LoRA features / freeze_layers / freeze_ratio / train_router_only / expand_layers; requires task=sft, backend=transformers, modality=text, and quantization=none.
  • The SNR kernel is pure-numpy and transpose-invariant (singular values are identical for W and W.T); GPT-2 Conv1D naming (c_attn/c_fc/c_proj) is recognised alongside Llama-style names.
  • The existing spectrum trainer-plugin wrapper is untouched (back-compat). LISA (per-step layer sampling) is tracked separately in #267.

Security

• soup spectrum scan validates unfrozen_parameters patterns at parse time: rejects nested-unbounded-quantifier regexes (ReDoS), null bytes, empties, and caps count (50k) and length (512). Hub downloads route through the SSRF-hardened, namespace-pinned hubs.snapshot_download; symlinked shards and matrices above a 2^31-element SVD cap are skipped; --output stays under cwd.

[0.71.22] - 2026-06-10

Added

• Perf & measure polish — a 4-issue patch tightening four live paths from the recent BETA lifts. Pure code, validated on Windows + RTX 3050.
  • MiniLLM on-policy KV-cache (closes #263). The on-policy distillation rollout (soup train with training.minillm_on_policy: true) now threads past_key_values so each step forwards only the new token instead of re-feeding the whole prefix — resolving the O(L²) per-step cost from v0.71.18. A LoRA student (the common distill case) activates the cache too: the new _supports_kv_cache probe unwraps the PEFT model via get_base_model() before deciding. The teacher is always cached; the student cache respects the retained autograd graph and degrades gracefully if a model returns no cache mid-loop.
  • soup serve --mole KV-cache (closes #262). Each of the N task adapters in a served MoLE now keeps its own KV cache in lockstep, created fresh per generate() call (never stored on the instance, so there is no cross-request leak). Top-k zero-weight adapters are still skipped, and the output is byte-identical to the no-cache path on a real MoLE.
  • Deploy-autopilot live measure factories (closes #143). soup deploy autopilot --measure ships a first-party transformers loader factory (lazy import, per-candidate quant config via the Quant Menu loader; before = base, after = quantised) replacing the inject-only test hooks. The baseline is now scored once and the whole candidate list is pre-validated up front, so a typo in --measure-candidates raises before any model load instead of burning N live loads or doubling peak VRAM.
  • Live-codec TTS via SNAC, partial (#265-partial). The live-codec encode path (data.format='audio') is validated for Orpheus: load_audio_mono now probes soundfile.info (duration + byte cap) before soundfile.read (no multi-GB decode into RAM) and reads through an O_NOFOLLOW file descriptor; a real SNAC-backed encode of a 24 kHz wav produced 42 Orpheus codec tokens.

Fixed

• MiniLLM on-policy KV-cache was silently disabled for LoRA students (the PEFT wrapper hid the base model's past_key_values support) — now probed via get_base_model().
• Deploy-measure no longer re-scores the baseline once per candidate or burns live model loads on a bad candidate (per-candidate validation moved up front).
• load_audio_mono capped audio duration only after decoding into RAM — the cap is now checked from soundfile.info before reading.

Known limitations

• KV-cache correctness is validated (cache == no-cache equality on real tiny artifacts) but large-model throughput gains were not measured on the 4 GB dev box.
• #265 stays open — the live-codec data.format='audio' SNAC encode path is validated for Orpheus only; the other four TTS families keep their per-family codec dependency gate.
• The deploy-measure first-party factory's real quantized (bitsandbytes 4-bit) load is CUDA + bitsandbytes-gated; on Windows / no-bnb the injected test seams are the validated path.
• The MoLE serve KV-cache assumes single-sequence (B == 1) decode.

[0.71.21] - 2026-06-10

Added

• Precision & rollout lift (BETA, hw-gated) — lifts five deferred NotImplementedError stubs to live code.
  • FP8 attention + NVFP4 (closes #141). training.fp8_attention: true now converts the model's attention projections (q/k/v/o + fused qkv variants) to FP8 training modules via torchao's convert_to_float8_training with an attention-only module_filter_fn (Hopper SM ≥ 9.0 gate); training.nvfp4: true quantises via torchao's NVFP4Config (Blackwell SM ≥ 10.0 gate). Both are wired into the v0.28 speed/memory pipeline and degrade to a visible yellow advisory when the gate fires — a conversion failing partway raises an honest "model may be PARTIALLY converted" error rather than silently training on a half-converted model.
  • vLLM sleep mode (closes #124). training.vllm_sleep_mode: true is live: create_vllm_engine(sleep_mode=True) sets AsyncEngineArgs.enable_sleep_mode (vLLM ≥ 0.7 gate with a friendly upgrade message), the new vllm_sleep_cycle(engine, level=1|2) context manager wraps the optimisation step (wake in finally), and the GRPO trainer threads the flag into TRL's GRPOConfig when the installed TRL exposes the hook (advisory otherwise).
  • Multi-turn agent rollout launchers (closes #125). soup train with task: grpo + training.rollout_backend: openenv + training.rollout_func: my_module:fn now runs a LIVE rollout: the resolver imports the operator's callable (same trusted-code policy as data.prompt_strategy), feeds it the dataset prompts as seeds, and the returned {prompt, answer?} rows replace the prompt dataset. Rows are normalised (extra keys stripped, message-list prompts deep-copied, non-string answers rejected loudly). art / ruler / nemo_gym raise a friendly ImportError when the backend package is missing and an honest BETA gate when present (injectable _EXTERNAL_ROLLOUT_RUNNERS seam). Validated by a real GRPO + openenv rollout train on SmolLM2-135M.
  • Apple-adapter conversion (closes #228). soup apple-adapter is live for hf-to-mlx / mlx-to-hf: PEFT LoRA safetensors ↔ mlx-lm adapters with both matrices transposed (lora_A [r,in] ↔ lora_a [in,r]), bf16 sources upcast via the torch loader, adapters.safetensors + num_layers emitted for mlx-lm's load_adapters, rank/alpha/dropout carried through, legacy adapters.npz still read, optional v0.60 Merkle-root signing. The *-to-apple directions stay upstream-gated (no published FoundationModels adapter spec). Validated by a real bf16 PEFT adapter round-tripping with numeric equality.
  • Llama-4 expert delinearization (closes #97). soup delinearize-llama4 now runs a live torch runtime: fused 2-D expert tensors [E*dim_in, dim_out] reshape to 3-D [E, dim_in, dim_out] (expert count from config.json or --num-experts), other tensors pass through, JSON sidecars are copied, writes are atomic. --plan-only keeps the old render-and-exit flow.

Fixed

• safetensors.numpy.save silently mangles non-contiguous (transposed) arrays — the apple-adapter writer now makes every array C-contiguous first (caught by the new round-trip assertions).

Known limitations

• fp8_attention / nvfp4 / vllm_sleep_mode are BETA hardware-gated — the converters and gates ship validated via capability probes and fake-module dispatch tests, but end-to-end runs need a Hopper/Blackwell GPU + torchao (or vLLM ≥ 0.7), none of which exist on the maintainer's RTX 3050 / Windows box. The art / ruler / nemo_gym rollout adapters are honestly BETA-gated until validated against the upstream packages.

[0.71.20] - 2026-06-09

Added

• Modality II trainers — TTS / BitNet / MoE expert quant (BETA, hw-gated) — lifts three v0.52.0 schema-only NotImplementedError stubs to real code.
  • TTS fine-tuning (closes #131). soup train with task='tts' + modality='audio_out' now routes to a live TTSTrainerWrapper. TTS families (Orpheus / Sesame-CSM / Llasa / Spark / Oute) are decoder language models, so a TTS fine-tune is next-token cross-entropy over interleaved [text][audio-codec-token] chat sequences — the wrapper reuses the SFT path and adds per-family emotion-control templating (Orpheus / Oute) and registration of operator-supplied codec special tokens (data.new_special_tokens) with an embedding resize. The pre-encoded chat workflow (codec tokens produced offline, then trained with data.format=chat) is the live, validated path; the live-codec workflow (data.format='audio', encode raw audio at train time) needs the family's heavyweight codec dependency (SNAC / BiCodec / XCodec2 / …) and is hardware/dependency-gated with a friendly per-family RuntimeError. Verified end-to-end on SmolLM2-135M-Instruct.
  • BitNet 1.58-bit (closes #134). build_bitnet_trainer returns a live BitNetTrainerWrapper that gates on the upstream onebitllms package (absent → friendly RuntimeError naming it). soup export --format bitnet | tq1_0 now runs a real llama.cpp TQ1_0 ternary export (reuses the v0.53.1 gguf convert→quantize pipeline) instead of the deferred panel; it requires a built llama.cpp toolchain (friendly FileNotFoundError when absent).
  • MoE expert quant + router-only training (closes #136). apply_moe_expert_quant detects fused-MoE expert nn.Linear blocks and replaces them with bitsandbytes Linear4bit (nf4) / Linear8bitLt (int8_rowwise), leaving attention + the router in full precision; it runs before get_peft_model (QLoRA-on-experts) so PEFT attaches to the quantized base. train_router_only freezes every expert and keeps the gating router trainable, applied after LoRA. CUDA-gated (friendly RuntimeError when bitsandbytes/CUDA absent). Validated live on an RTX 3050: 8 expert Linears → 8 Linear4bit with dequant error 0.0155 vs source (weights genuinely carried), router-only freeze, and device-aware placement.

Known limitations

• The TTS live-codec workflow, BitNet 1.58 training (onebitllms), and BitNet GGUF export (llama.cpp) are hardware/dependency-gated — the friendly gates ship and the plumbing is validated, but the end-to-end runs against real TTS models + audio codecs / a BitNet base + onebitllms / a built llama.cpp toolchain stay open infra-blocked items on the maintainer's RTX 3050 / Windows box.

[0.71.19] - 2026-06-09

Added

• Quant Menu for vision / audio modality (closes #81). The Quant Menu (gptq / awq / hqq:Nbit / aqlm / eetq / mxfp4 / fp8) was rejected by the config modality gate for modality in {vision, audio} — those paths carried inline BitsAndBytesConfig blocks that handled only 4bit / 8bit. v0.71.19 drops the gate (the mlx-backend gate is retained) and threads the unified build_quantization_config_for_loader through _setup_vision_transformers / _setup_audio_transformers, so multi-modal SFT can train a LoRA on top of any pre-quantized base. The 4bit / 8bit config shapes are byte-for-byte the same as the old inline blocks; mxfp4 still routes through prepare_model_for_kbit_training. Verified: the unified loader returns the right config object for every format on both modalities, and _setup_vision_transformers threads a GPTQConfig into AutoModelForVision2Seq.from_pretrained.

Fixed

• Multipack DataLoader sharding under FSDP / DeepSpeed ZeRO / DDP (closes #80). The multipack get_train_dataloader override built a raw DataLoader and returned it directly, so under distribution every rank trained on the same packed bins (no data sharding). It now routes the loader through accelerator.prepare(...) when num_processes > 1 — exactly what HF Trainer's own get_train_dataloader does — so accelerate's BatchSamplerShard round-robins whole bins across ranks (preserving the FFD packing) and equalises per-rank batch counts. The single-process path is unchanged (byte-for-byte the validated v0.40.4 raw-DataLoader behaviour). Verified live: a single-GPU multipack SFT on SmolLM2-135M trains end-to-end (RTX 3050). Full multi-GPU validation remains a QA item (no multi-GPU box); the distributed routing is mocked-tested.

[0.71.18] - 2026-06-08

Added

• MiniLLM true on-policy rollout (closes #257). training.minillm_on_policy: true (with minillm_enabled: true) replaces the offline distribution blend with the real on-policy procedure of Gu et al. 2024 §3.1: each step samples a fresh autoregressive rollout from the per-token mixture ratio·teacher + (1-ratio)·student, then accumulates the length-normalised reverse-KL KL(student || teacher) on the full distributions (differentiable w.r.t. the student only; sampled tokens are detached). New training.minillm_rollout_length knob ([1, 512]; auto-derives min(max_length, 32) when unset — the loop re-forwards the full prefix each step, so keep it small). Verified live: on-policy distill on tiny-gpt2 (student + frozen teacher), finite loss, end-to-end train.
• Cross-tokenizer ULD with token-sequence alignment (closes #258). New training.uld_strategy: wasserstein_aligned handles fully-disjoint tokenizers (not just a vocab-size mismatch): per batch element the student and teacher token sequences are aligned over their decoded character spans (offset-overlap when both decode to the same text, difflib Ratcliff-Obershelp char matching otherwise), the teacher logits are mean-pooled onto the student positions, and the existing sorted-Wasserstein-1 surrogate is applied. Verified live: aligned distill with a GPT-2 BPE student + a Llama SentencePiece teacher, finite loss, end-to-end train.
• soup agent eval --sandbox (closes #110). Each heuristic-passing tool-call prediction is now executed against a generated mock of the endpoint in the v0.25.0 RLVR code_exec sandbox and classified into ok / tool_error / timeout / arg_error. The endpoint path, its required path params, and the predicted arguments are base64-embedded as data (no code interpolation). Strong isolation (RLIMIT / namespaces / sandbox-exec) is POSIX-only; on Windows the subprocess + 5 s timeout + 10 KB output cap + network guard still apply (a friendly reduced-isolation advisory is printed). Verified live on Windows: 4-prediction scorecard (ok=1 / tool_error=1 / arg_error=2 / timeout=0).
• soup train --cloud modal (closes #16). Render a self-contained Modal.com app from soup.yaml for serverless GPU training when you have no local GPU. The config YAML is base64-embedded as data (no interpolation, no secrets); the --gpu type (t4 / l4 / a10g / a100 / a100-80gb / l40s / h100) is validated against a closed allowlist. Default is plan-only: write the stub + print the modal run command. --cloud-submit attempts a live submit gated on a Modal token (modal setup / MODAL_TOKEN_ID + MODAL_TOKEN_SECRET). New [modal] extra (pip install 'soup-cli[modal]'; only needed for live submit — plan-only render needs no dependency). Verified live: real stub rendered, exit 0.

[0.71.17] - 2026-06-08

Added

• Serve-time MoLE (closes #259). A task='moe_lora_routing' run now writes a self-describing mole_manifest.json next to mole_gate.pt, and soup serve --mole <dir> loads the base + N frozen task LoRAs + the trained gate and blends them per token at decode time (custom blend loop — non-streaming + streaming). --mole requires --backend transformers and is mutually exclusive with --bank / --steer / --adapters / --speculative-decoding. The base model comes from --base (or the manifest when unset). Verified live on SmolLM2-135M (2 task adapters, real generation + SSE streaming).
• Per-request multi-tenant vector banks (closes #260). soup serve --bank now resolves the active VeRA/VB-LoRA user per request via a contextvars.ContextVar, so concurrent requests on a threaded server never race on shared instance state. The streaming path re-selects the user inside the generator's own context. Verified live: two X-User-Id headers produce distinct steered outputs, an absent / unknown id self-clears to the clean baseline (no cross-request leak), and a repeated user is deterministic.
• Epoch-aware RAFT document shuffle (closes #253). data.raft_epoch_shuffle: true re-permutes the golden + distractor documents each training epoch (per-epoch salt) so the model can't latch onto one fixed citation slot. epoch=0 reproduces the legacy single-permutation order exactly. Verified live on a 2-epoch SmolLM2-135M RAFT run.
• soup diagnose --citation-style / --shuffle-seed (closes #254). The live citation failure-mode probe now accepts the citation style (bracket / inline / footnote) and the RAFT shuffle seed so the golden [doc-N] ids line up with what the model saw at train time. Verified live (rows=6, mean_recall=1.000).

Fixed

• MoLE train() now returns the initial_loss / final_loss / total_steps / duration_secs / duration keys the generic train handler reads, so soup train task=moe_lora_routing completes cleanly (previously raised KeyError: 'initial_loss' after writing the gate). Surfaced by the #259 smoke.

[0.71.16] - 2026-06-07

Added

• Covariance-preconditioned ROME via --cov-corpus (closes #250). soup edit set --method rome --cov-corpus <jsonl|txt> now estimates the key covariance C = E[k kᵀ] + λI over a stats corpus and uses the preconditioned update u = C⁻¹ k* instead of the covariance-free C = I path — the genuine ROME closed form, which spreads the rank-1 update mass to reduce collateral interference with other facts. Falls back to C = I when no corpus is given. The exact post-condition down(k*) += delta is preserved either way. The corpus loader is cwd-contained, symlink-rejected (O_NOFOLLOW + raw-path lstat), and size/line-capped; --cov-corpus is rejected (fail-loud) for any method other than rome. Verified on real gpt2 (prob 0.005 → 0.9997) and SmolLM2-135M.
• GPT-2 (transformer.h / mlp.c_proj) support in the edit kernels (closes #251). ROME / MEMIT / AlphaEdit now edit GPT-2-family models, not just Llama-family. The Conv1D weight layout ([in, out], transposed relative to nn.Linear's [out, in]) gets a transpose-aware rank-1 update, AlphaEdit null-space projection, and MEMIT band dim-check. PEFT-wrapped GPT-2 / Llama models are unwrapped via get_base_model. Verified end-to-end on real gpt2.
• Mixtral joins the LongLoRA architecture allowlist (closes #147). A bare mistral token does not appear in mixtral (m-i-x vs m-i-s), so the existing is_mistral_model detector excluded the MoE variant. A dedicated is_mixtral_model helper + MixtralAttention entry in the S² forward-override regex + _SEPARATE_QKV_FAMILIES now cover Mixtral-8x7B / 8x22B (the attention is the standard separate-QKV shell; the MoE lives in the MLP).

Fixed

• Atomic EditGovernor edit-count increment (closes #252). Two concurrent soup edit set runs on the same base model could lose an increment: each read the persisted count, added locally, and the last writer clobbered the first. save_state now re-reads the persisted count INSIDE the cross-process lock and merges this run's delta (edit_count − persisted_baseline), mirroring the v0.60.0 namespace_pin pattern. Verified: two governors recording 3 + 2 edits from the same baseline persist a merged 5 (not a clobbered 2 or a naive +1).

Notes

• Test count: 13511 → 13595 (+84 net; +81 in tests/test_v07116.py).

[0.71.15] - 2026-06-07

Fixed

• Iterative-DPO config render bug (closes #261). soup iterative-dpo's default per-round trainer rendered output: {dir: ...} (a mapping), which SoupConfig.output (a plain string) rejected — so the spawned soup train subprocess failed at config validation. Now renders output: <str>, mirroring the v0.71.13 #229 local-rl fix. A regression test captures the rendered YAML and validates it via load_config_from_string; verified end-to-end with a real soup train round on SmolLM2-135M.

Changed

• CMA-ES merge loads the base model once (closes #246). soup adapters merge --strategy cmaes previously reloaded the (multi-GB) base model into a fresh PEFT wrapper on every candidate in the population. The default scorer now loads the base once and reuses it across the whole population × generations loop — each candidate only loads its small merged LoRA, applies it, generates, and unloads it. Verified on SmolLM2-135M: the base loads exactly once across N candidates.
• soup loop budget gate now estimates real cost (closes #245). The pre-wired loop's per-iteration cost estimate was a hard 0.0 placeholder, so the dollar budget gate never tripped. It now wires v0.34 run_cost. estimate_run_cost_usd off the most-recent completed run's GPU + duration (the best forward signal for a repeating loop). Falls back to 0.0 on the first iteration / a CPU / unpriced GPU; never crashes the daemon.
• --diagnose-gate is multi-node aware (closes #170). The post-training diagnose gate (and the --annex-xi / --repro-receipt / capture hooks) fired on LOCAL_RANK==0, so a shared-filesystem multi-node run ran them once per node. They now gate on the global chief (RANK==0 when RANK is set, else LOCAL_RANK==0) — once per cluster.

Added

• soup train --track-energy --energy-out <path> (closes #244) persists the measured energy/CO2 reading as JSON so soup bom emit --energy <path> (the v0.71.3 #256 consumer) can attach it to an ML-BOM. Atomic + cwd-contained + symlink-rejected. Completes the train → BOM energy hand-off.

[0.71.14] - 2026-06-05

Added

• Live FSDP shard consolidation (closes #96). soup merge-sharded-fsdp-weights lifts the v0.44.0 plan-only stub: it now streams each pytorch_model_fsdp_*.bin shard via torch.load(weights_only=True) (no arbitrary pickle exec), unions the per-rank parameter fragments into one state-dict, and writes a single .safetensors atomically. Memory-friendly (one shard loaded at a time). New --plan-only flag prints the plan without writing. Single-process — no multi-GPU needed to MERGE. (Per-rank disjoint-parameter / FULL_STATE_DICT shards; DCP sharded-tensor reconstruction is out of scope — use accelerate merge-weights for those.)
• Live kv_cache_type wiring on the transformers serve backend (closes #140). soup serve --kv-cache-type q8_0 | bf16 | f16 | fp8 lifts the v0.53.1 apply_kv_cache_type NotImplementedError stub: bf16/f16 load the model in that dtype (the KV cache inherits it); q8_0 routes an 8-bit HQQ quantized KV cache through model.generate (needs pip install hqq); fp8 raises a friendly runtime error (vLLM + Hopper-only — the transformers backend has no fp8 KV path). vLLM / SGLang KV-cache-dtype routing stays in the infra-blocked tail.
• ONNX export QA verified (closes #71) — soup export --format onnx exercised end-to-end on a tiny model: export exits 0, model.onnx loads in ONNX Runtime with input_ids present, and a forward pass produces a real output. Recorded in tests/qa/v07114_qa.md.

Notes

• GGUF export (#70), AWQ/GPTQ export (#72), the CUDA + llama.cpp QA doc (#144), HF Hub push/Spaces deploy (#74), and the Community-QA tracking meta-issue (#79) remain open with infra-blocked labels — they need a built llama.cpp toolchain, autoawq/auto-gptq Windows wheels, or HF credentials the QA box lacks. See tests/qa/v07114_qa.md.

[0.71.13] - 2026-06-04

Added

• Prompt-compile family — live wiring (closes #225, #226, #227, #229). Four soup commands that shipped as deferred-stub NotImplementedError in v0.68.0 are now real, validated end-to-end (real DPO train on SmolLM2-135M + real Ollama teacher distillation on RTX 3050).
• soup local-rl train runs a real nightly DPO/KTO/ORPO train (#229). --once harvests the latest thumbs-up/down DPO pairs from the local-RL SQLite and trains them via a soup train subprocess (argv list, no shell); a state table tracks last_train_at so a re-run with no new feedback skips, and a run with fewer than --min-pairs (default 10) skips. Without --once it renders a systemd .service/.timer + launchd .plist scheduler scaffold into --scheduler-dir for the user to install. New flags: --once, --min-pairs, --output/-o, --scheduler-dir, --hour, --minute.
• soup distill-prompt prepares a real distillation dataset (#226). For each prompt in the traces JSONL the teacher is called once via the v0.20 provider helpers (Ollama / Anthropic / vLLM); sft/kl emit {messages:[user, assistant=teacher]} and preference emits {prompt, chosen=teacher, rejected=student}. New flags: --provider, --base-url, --temperature, --max-rows.
• soup compile runs DSPy / GEPA / TextGrad prompt-program optimisation (#225) and soup compile-tools runs the TextGrad / GEPA tool-schema optimiser (#227), both lazy-importing the optimiser libraries behind the new [compile] extra (pip install 'soup-cli[compile]') with a friendly ImportError naming the extra when absent. --plan-only still renders the plan and exits 0.

Security

• systemd / launchd injection defence (#229). local-rl and the scheduler renderers reject \n / \r in the model id and shell-quote every ExecStart argument, so a crafted model id cannot inject extra unit directives.

Fixed

• local-rl train config rendered output as a mapping (#229). The nightly soup train YAML now emits output: <dir> (a plain string the schema accepts) instead of output: {dir: <dir>}; a regression test validates the rendered config against SoupConfig.

[0.71.12] - 2026-06-04

Added

• Architecture + distillation + adapter-training — live wiring (closes #145, #146, #148, #158, #84, #221, #222). Seven surfaces that shipped schema-only in earlier releases are now real, validated end-to-end on tiny models (SmolLM2-135M / a locally-built tiny Llama).
• Sequence-level knowledge distillation is live (#145). task: distill now accepts distill_mode: token|sequence; sequence mode trains the student on the teacher's generated continuations (cross-tokenizer-friendly hard-label KD) instead of per-token logit matching. sequence mode is mutually exclusive with the v0.70 cross-tokenizer ULD logit path.
• Classifier LoRA is live (#146). task: classifier|reranker|cross_encoder now attaches a LoRA adapter to the sequence-classification head when lora is configured, so a frozen encoder + small adapter can be trained instead of the full model.
• LLaMA Pro block expansion is per-architecture (#148). expand_layers now interleaves zero-initialised identity blocks for Llama / Qwen / Mistral decoder stacks (was Llama-shaped only), with freeze_trainable_layers freezing the original blocks so only the new ones train.
• LongLoRA S² shifted-sparse attention is live (#158). use_longlora: true now installs the shifted-sparse-attention forward override on the Q/K projections (Llama / Mistral / Qwen / Phi), restoring the patched forwards on context exit.
• Mixture-of-Depths is live (#84). use_mod: true attaches a per-layer top-k token router (mod_capacity_factor) so only a subset of tokens receive each block's residual update. Architecture allowlist: Llama / Qwen / Mistral; unsupported bases warn and skip.
• VeRA / VB-LoRA multi-tenant serving is live (#221). soup serve --bank <bank.json> [--bank-strength S] reconstructs the shared projection + per-user scaling vectors and installs a decode-time forward hook; the active user is selected per request via the X-User-Id header (an unknown/absent id is a zero-delta no-op, so there is no cross-request leak). Serves N personas at ~KB-per-user instead of a full LoRA each.
• MoLE per-token adapter routing is live (#222). task: moe_lora_routing with mole_task_adapters: [...] trains a per-token gating network that blends N frozen task LoRAs (mole_top_k / mole_temperature); only the router trains. The gate is saved as mole_gate.pt alongside the run.

Changed

• apply_bank_to_serve (#221) and build_gating_kernel (#222) now return live objects (a LoadedVectorBank and a torch.nn.Module router) instead of the v0.67.0 deferred-stub NotImplementedError.

[0.71.11] - 2026-06-04

Added

• GRPO / RL callbacks — live wiring (closes #235, #236, #237, #238, #239, #240, #159, #160). The reward-hacking, cross-tokenizer distillation, MiniLLM, mid-epoch RL checkpoint, iterative-DPO and echo-trap surfaces that shipped schema-only in v0.70.0 are now real, validated end-to-end on SmolLM2-135M.
• Reward-hacking detector is live (#235). --reward-hack-detector info_rm|rm_ensemble now installs a GRPO TrainerCallback that reads the per-step rewards (via a shared, thread-safe reward-fn capture buffer), computes an InfoRM cluster-separation drop (info_rm) or RM-ensemble divergence (rm_ensemble), classifies OK/WARN/HACK, logs the verdict to state.log_history, and halts training on HACK when --reward-hack-halt is set. rm_ensemble requires ≥2 reward functions.
• Cross-tokenizer ULD distillation is live (#236). task: distill with --uld-strategy wasserstein|topk_align now computes a real Wasserstein-1 (sorted-CDF) or top-k-aligned distillation loss inside the distill trainer, handling student/teacher vocab-size mismatch by clamping teacher ids to the teacher vocab.
• MiniLLM reverse-KL distillation is live (#237). --minillm-enabled adds a teacher-mixed, length-normalised reverse-KL term plus an optional pretrain-anchor SFT term (--minillm-pretrain-anchor-path / --minillm-pretrain-anchor-weight) that keeps the student near coherent language. The anchor corpus reader is cwd-contained + symlink-rejecting with a per-line byte cap.
• Mid-epoch RL checkpoint is live (#238). --rl-checkpoint-save-every-steps N writes a real adapter + optimizer state + JSON manifest every N steps during PPO/GRPO and prunes to --rl-checkpoint-keep-last, so a long RL run survives a crash without losing the optimizer momentum.
• soup iterative-dpo orchestrator is live (#239). Runs the full sample → reward-score → build-pairs → DPO-train loop across rounds: each round samples completions from the previous round's adapter, the next round trains a fresh LoRA from the base on that round's harvested pairs. --plan-only still renders the plan without running.
• Echo-trap detector is live (#240). --echo-trap-enabled installs a GRPO callback that scores per-trajectory n-gram repetition, classifies OK/WARN/TRAP against --echo-trap-threshold, logs the verdict, and halts on TRAP when --echo-trap-halt is set (catches RAGEN-style degenerate repetition in multi-turn agent RL).
• GRPO variant fallback now warns once (#159). When a --grpo-variant custom compute_loss falls back to the base trainer (because the installed TRL renamed the loss inputs), the trainer logs a one-shot WARNING instead of silently degrading to the default objective.

Changed

• GRPO reference-model EMA no longer materialises full state dicts (#160). --ref-model-ema-alpha now updates the reference model in place by iterating named_parameters() (ref = (1-α)·ref + α·policy), eliminating the three model-sized allocations per step the v0.53.11 path made. A total name/shape-mismatch (0 shared parameters) logs a one-shot WARNING so a misconfigured EMA can't silently no-op.

[0.71.10] - 2026-06-03

Added

• RAG family — live wiring (closes #199, #200, #201, #202). The four retrieval / steering surfaces that shipped schema-only in v0.62.0 are now real, validated on SmolLM2-135M.
• RAFT span-mask training is live (#199). data.format: raft rows ({query, golden_doc, distractor_docs, answer}) now train answer-only: the prompt span is masked to -100 and each document is labelled [doc-N] so the model learns to cite the supporting document. Documents are shuffled reproducibly (data.raft_shuffle_seed). Rows whose prompt fills max_length (answer fully truncated) are dropped with a warning rather than silently shrinking the effective dataset.
• soup ra-dit — one-shot two-stage orchestrator (#200). Trains the retriever (stage 1, embedding/contrastive) then the generator (stage 2, RAFT-SFT) in a single command, recording the trained retriever as the generator's paired retriever. A soup train of a generator-stage config with no retriever model set now auto-links the most-recent RA-DIT retriever run from the Registry. --plan-only validates both configs without training; --retriever-model overrides the auto-link.
• soup steer train / apply + soup serve --steer are live (#201). Fit a CAA (contrastive activation addition), ITI (inference-time intervention) or RepE (representation-engineering PCA) control vector from {positive, negative} contrastive pairs, persist it as a safetensors + config artifact, and apply it at decode time via a forward hook (soup serve --steer <name> --steer-strength <s>).
• soup eval citation + citation-span loss boost are live (#202). Score citation precision / recall / F1 over {predicted, expected_ids} or RAFT rows (--shuffle-seed aligns the golden [doc-N] id with what the model saw at train time). When citation_faithful: true, bracketed [doc-id] spans in the answer get a boosted per-token loss weight. A new citation failure mode is available in soup diagnose.

[0.71.9] - 2026-06-03

Added

• Knowledge edit + unlearn — live wiring (closes #193, #194, #196, #197, #203). The v0.61.0 / v0.62.0 schema-only stubs are now live, validated on SmolLM2-135M.
• soup edit set (ROME / MEMIT / AlphaEdit) is live (#194). New soup_cli/utils/edit_kernels.py ships covariance-free rank-1 weight-edit kernels: ROME (single-layer W += δ·kᵀ/‖k‖²), MEMIT (residual distributed across a layer band), AlphaEdit (ROME update projected orthogonal to the down-proj's top singular direction). apply_edit loads the model, optimises the target residual, applies the rank-1 update, and optionally saves with cwd-containment + symlink rejection. --output, --device, --governor/ --no-governor flags added. On a tiny model a ROME edit moved P("Lyon" | "The capital of France is") from 0.0016 → 0.96.
• soup edit diff live before/after generation (#194). Pass --before-model + --after-model (+ --probes) to generate completions through both models and surface the probes whose output changed.
• EditGovernor SQLite persistence + cross-process locking (#196). New EditGovernorStore (mirrors namespace_pin.NamespacePinStore — $HOME/$CWD/$TMPDIR containment, TOCTOU symlink rejection, WAL + busy_timeout, fcntl/msvcrt sidecar lock, POSIX 0600). save_governor / load_governor / default_governor_db_path (env override SOUP_EDIT_GOVERNOR_DB) persist per-base-model edit-count + verdict across separate soup edit set runs.
• apply_edit consults the EditGovernor automatically (#197). When a governor is supplied, check_can_edit() runs BEFORE the model load (refusing on norm blowup / edit cap) and record_edit() runs AFTER with the measured Frobenius delta.
• Live GRACE codebook (#203). GraceCodebook (epsilon-ball nearest-key lookup), apply_grace_edit (captures a residual key + optimises a value + appends to a codebook sidecar), save_codebook / load_codebook (atomic, cwd-contained, symlink-rejected), install_grace_hook (decode-time residual substitution). New edited_model / grace_codebook Registry artifact kinds.
• soup train --task unlearn is live (NPO / SimNPO / RMU) (#193). New soup_cli/utils/unlearn_kernels.py (NPO (2/β)·mean(-logσ(-β(πlp-reflp))), length-normalised SimNPO, RMU representation steering) + a self-contained UnlearnTrainerWrapper loop loading a LoRA policy, a frozen reference (NPO/RMU), and forget/retain JSONL datasets. NPO/SimNPO forget loss decreased on the tiny-model smoke. Warns when run without a retain set.

Security

• _save_edited_model / UnlearnTrainerWrapper output dirs + save_codebook / load_codebook + _load_unlearn_rows enforce cwd-containment, raw-path symlink rejection (TOCTOU), null-byte rejection, and file-size / per-line caps. apply_grace_edit honours the governor for direct callers.

[0.71.8] - 2026-06-03

Added

• Probes & SAE — real weights + live downloads (closes #215, #216, #217, #218, #219). A new shared soup_cli/utils/probe_kernel.py provides the linear-probe math (contrast-pair derivation, apply, flag-rate, verdict bands, operator-supplied weight loading, deterministic synthetic fallback); every heavy import (numpy / torch / safetensors) is lazy.
• soup probe sleeper --weights <w.npz|.npy|.safetensors> (#215) — load a real calibrated probe direction instead of the synthetic fallback. Weights are cwd-contained, symlink-rejected, O_NOFOLLOW-opened, allow_pickle=False, and size-capped. compute_contrast_probe(positive, negative) derives a probe from contrast-pair activations.
• soup probe sae-diff <repo> --auto-download (#216) — fetch an allowlisted SAE from the HF Hub into ~/.soup/sae-cache/ (validated against HF_HUB_ALLOWLIST BEFORE any network call) via a new SSRF-hardened soup_cli.utils.hubs.snapshot_download (repo-id shape + home/cwd/tmp cache containment + namespace-pin TOFU gate).
• soup probe truth / soup probe harm (#217) — TruthfulQA-style honesty and HarmBench-style misuse activation probes (6 bundled bases each, 5% / 20% verdict bands, --weights to skip the allowlist with a real probe). The probe pack now ships truth + harm entries per base.
• soup probe interference --measure <eval_suite> --base-model <m> --adapter name=path ... (#218) — auto-measure the N×N interference matrix by actually loading the base + each LoRA adapter (PEFT multi-adapter), measuring loss for each adapter alone (diagonal) and each co-loaded pair (add_weighted_adapter(combination_type="cat"), off-diagonal). Exit 2 on a MAJOR worst-pair.
• soup train --capture-activations <layer> --capture-prompts <jsonl> (#219) — a post-training hook writes an SAE-diff-ready per-token activation snapshot to <output>/activations/activations.json. resolve_layer_module resolves the same model.layers.N path whether or not a LoRA adapter is loaded (PEFT-wrapper fallback).

Security

• Probe / SAE / capture file I/O is cwd-contained + O_NOFOLLOW (TOCTOU close)
  • size-capped; SAE weight loads use allow_pickle=False. SAE auto-download validates the allowlist before any network call and rejects a glob result that resolves outside the snapshot dir (symlink-escape guard).

Notes

• #215 is partial: the operator-supplied / contrast-pair / synthetic paths ship now, but the 6 large-base Anthropic-calibrated probe vectors remain upstream-gated (no public calibrated artifact exists). Documented as a known limitation.

[0.71.7] - 2026-06-02

Added

• Eval live runners — six probe surfaces that previously emitted heuristic / neutral stubs now load a real model and run live (closes #161, #162, #208, #211, #212, #165). New shared soup_cli/utils/live_eval.py provides the model-loading primitives (generator / multi-generator closures, masked cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement); every heavy import (torch / transformers / peft / lm_eval) is lazy.
• soup advise --probe-model <id> — runs a LIVE ROI probe: zero/few-shot token-F1 baselines, a short LoRA probe (relative held-out-loss improvement + real wall-clock), and base-model proximity (held-out logit agreement) folded into the dataset profile. Without --probe-model, --probe stays the offline heuristic.
• soup tunability --live — replaces the offline heuristic with a real per-candidate LoRA probe (loads each repo_id, trains --probe-steps on a held-out-excluded slice, reports the held-out-loss drop).
• soup eval capability --live --model <id> — invokes lm-eval-harness per resolved task (or a --tasks override) with --limit / --device, isolating per-task failures and surfacing a no-metric result as an explicit error.
• soup eval behavior --base-model <id> [--adapter <path>] — generates pre/post responses on the bundled behaviour battery and scores the live diff.
• soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>] [--tokenizer <id>] — runs all six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) live via soup_cli.utils.diagnose.live.run_live_diagnose; falls back to neutral OK or --evidence JSON when no model is supplied.

Security

• The two new JSONL dataset readers (diagnose.live._load_dataset_rows, tunability._load_jsonl_rows) open with O_NOFOLLOW after the cwd-containment check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader policy).

[0.71.6] - 2026-06-02

Added

• soup build live runner — the dbt-for-SFT DAG (soup build <manifest>) now materialises datasets instead of only dry-running the plan. Five built-in transforms ship live (identity, drop_empty, lowercase, strip, dedup_exact); table rebuilds from scratch, view re-derives on every run, and incremental re-transforms only the rows whose content hash changed (tracked in a SQLite state store, keyed by row hash and the model's transform+config fingerprint so a transform change re-runs everything). Custom transforms are passed per-run via the Python API's transforms= map. Outputs are written atomically; the --output-dir is symlink-checked before any directory is created.
• soup data gen-magpie live generator — the Magpie synthetic generator (Xu et al. 2024) now actually generates. It feeds an aligned model its chat-template prefix (chatml / llama3 / gemma / mistral families auto-detected) and harvests the self-generated user instruction + assistant response via raw completion. Live providers: ollama (/api/generate raw) and vllm (/v1/completions) — both SSRF-hardened (loopback-only HTTP); anthropic is rejected (no raw-completion endpoint). Optional --quality-filter drops low-quality rows via the v0.47 toxicity/educational scorers; exact-duplicate instructions are de-duplicated.
• soup eval irt-subset --model {1pl,2pl,3pl} — the IRT eval-cost optimiser gained 2PL (per-item discrimination) and 3PL (+guessing floor) joint coordinate-ascent MLE fits alongside the existing 1PL Rasch. 1pl keeps the closed-form path for back-compat; 2pl/3pl route through the new fit_irt.
• Tokenizer-aware memorization probe — score_memorization(..., tokenizer=...) and split_prefix(..., tokenizer=...) (used by soup diagnose) now split the prefix/suffix on real token-id boundaries and measure echo-overlap over sub-word tokens when a tokenizer (HF id / path / duck-typed object) is supplied, catching BPE-level memorization that whitespace tokenisation misses. Default (no tokenizer) keeps the whitespace behaviour.

Fixed

• soup data augment --provider ollama|vllm no longer crashes — the command imported a non-existent OllamaProvider symbol and raised ImportError on every non-OpenAI provider. It now routes through the shared, SSRF-hardened provider factory; --model / --base-url are honoured, the output path is containment- and symlink-checked, and the write is atomic.

Security

• Ollama / vLLM provider URLs reject 0.0.0.0 — validate_ollama_url / validate_vllm_url dropped the bind-any wildcard from their loopback allow-set (now localhost / 127.0.0.1 / ::1 only), matching the newer validate_hub_endpoint / validate_webhook_url SSRF validators. Reachable now that Magpie threads a user-supplied --base-url through these providers.

[0.71.5] - 2026-06-02

Added

• soup eval against now reads eval metrics — ExperimentTracker.get_metric_series falls back to the eval_results table when the metric is not a per-step training column (loss / lr / grad_norm / speed / gpu_mem). So soup eval against <base> --candidate <run> --metric task_accuracy returns a real score series (benchmark scores live in eval_results, not metrics) instead of "Empty series". Per-step columns still read from metrics — no regression for existing callers.
• soup advise learns from past project outcomes — soup advise now reads this project's accepted-verdict history (~/.soup/advise_history.jsonl) and biases the rubric: 3+ successful SFT precedents flip a marginal RAG call to SFT; 3+ negative GRPO outcomes suppress GRPO in favour of SFT-on-traces; an encouraged choice gets a small confidence nudge. Scoped per-project (one project's record never biases another). No history → identical to before.
• Slack/Discord webhooks on four more commands — --slack-url / --discord-url (SSRF-hardened, loopback-only HTTP, RFC1918 rejected, never crashes the command) now ship on soup ingest, soup prune-prompt, soup ab (fires only on a reject_h0 / accept_h0 decision, not continue), and soup data active-sample — not just soup drift-alarm. The validator + sender moved to a shared soup_cli/utils/webhooks.py.
• Tokenizer-aware soup prune-prompt — --tokenizer <model_or_path> detects and strips the shared system-prompt prefix on token boundaries instead of characters, so a multi-byte UTF-8 prefix can never be split mid-code-point. Default (no --tokenizer) keeps the whitespace-character behaviour.
• Curriculum bucketing by loss percentile — DynamicCurriculumCallback now buckets samples by the percentile rank of the live loss (or perplexity) signal within a rolling window when data.curriculum_metric is loss / perplexity, so a consistently-hard sample is routed to the same difficulty bucket across recomputes. length and warm-up still use round-robin.
• --hub on soup data push and soup data forge — soup data push --hub modelscope|modelers uploads a dataset via the matching SDK (repo_type=dataset, commit message sanitised); soup data forge --hub <non-hf> --teacher owner/name pre-fetches the teacher model from that hub (and warns when the teacher is not a repo id so --hub is never silently ignored). HF stays the default.

Notes

• Live SaaS pull adapters for soup ingest (Langfuse / LangSmith / Helicone / OpenPipe / OpenAI SDKs, issue #204) remain deferred: they need credentialed vendor accounts with populated trace data to validate honestly. Tracked as an open, infra-blocked (external-account) item. soup ingest continues to parse the JSONL export you pull from your dashboard.

[0.71.4] - 2026-06-02

Added

• Live canary verdict for soup adapters merge — --canary <suite.json> scores the merged adapter against the first input and classifies OK / MINOR / MAJOR using the Quant-Lobotomy taxonomy (drop <2% OK, <5% MINOR, else MAJOR). --strict-verdict exits 2 on MAJOR. Pre-scored {"baseline_scores","candidate_scores"} suites run with no model load; a {"tasks":[...]} suite uses an injectable scorer. Replaces the v0.57 UNKNOWN stub.
• Live evolutionary merge — soup adapters merge --strategy cmaes --eval <suite> --budget <t> now runs the full CMA-ES loop: each candidate is merged, materialised, scored against the eval suite, and the best-weighted merge is written to --output. Replaces the v0.67 plan-only stub.
• Publish an adapter PR to GitHub — soup adapters pr <title> --base-sha <hex> --adapter <path> --push owner/repo#N posts the rendered PR Markdown as a GitHub PR comment via gh api (argv-list, body over JSON stdin; no shell). Token resolves from GITHUB_TOKEN / GH_TOKEN.
• Pre-wired soup loop production stages — soup loop init --pre-wired (or soup loop watch --pre-wired) swaps the v0.58 no-op stage stubs for real harvest (traces → preference pairs) → DPO train → eval-gate → canary-deploy callables. soup loop status now shows the pre_wired flag.
• Loop iterations as Soup Cans + Registry lineage — soup loop watch --pack-cans packs each successful iteration as a v0.26 Soup Can and appends a Registry entry (tag loop-iter), chaining a real lineage DAG across iterations visible through soup history. `soup loop replay --extract ` unpacks a recorded iteration.
• Branch pointers into the Registry — soup adapters branch <name> --attach-to-registry <id> links a branch snapshot to a Registry entry (shown as a branches node in soup history); soup adapters branch <name> --from-registry <id> derives a fresh snapshot's config + base from an entry.

Security

• The backdoor-scan gate (v0.71.2 #192) and license-conflict gate (v0.60 Part E) now run for all merge strategies, including --strategy cmaes (previously bypassed because cmaes returned before the gates).
• soup loop canary deploy restricts SOUP_LOOP_SERVE_ENDPOINT to loopback / RFC1918-private hosts (a serve endpoint is the operator's own box/LAN), beyond the general webhook SSRF policy which permits any HTTPS host.
• soup adapters pr --push builds the gh child environment from an allowlist so unrelated secrets (HF_TOKEN / OPENAI_API_KEY / …) never reach the subprocess.
• The canary-suite JSON read uses O_NOFOLLOW + os.fstat (size cap enforced on the same fd) to close the symlink/size-cap TOCTOU window.

[0.71.3] - 2026-06-01

Added

• Energy & CO2 measurement for training — soup train --track-energy wraps the training window in a codecarbon offline tracker (no IP-geolocation network call) and reports kWh / CO2 / grid intensity, feeding those numbers into --annex-xi. New EnergyTracker context manager; graceful no-op when codecarbon is absent (pip install soup-cli[carbon]). --energy-country picks the ISO-3166 alpha-3 grid for the CO2 estimate (default USA).
• PDF Annex XI/XII documents — soup train --annex-xi report.pdf now renders a reportlab PDF (a .md path still renders markdown). pip install soup-cli[pdf].
• Auto-populated training-corpus domains in Annex XI/XII — the top crawled domains (with shares) are now extracted from the training JSONL and listed in the EU AI Act docs, replacing the previous empty placeholder.
• Soup Can manifest v3 with embedded attestations — soup can pack --attest <statement.json> (repeatable) embeds in-toto Statements into a v3 can manifest; v1/v2 cans still load. Each statement is shape- and size-validated.
• Local audit log auto-instrumentation — every soup command now appends one HIPAA/SOC2-shaped record to ~/.soup/audit.jsonl (secrets redacted, args capped). Opt out per-invocation with --no-audit-log or globally with SOUP_NO_AUDIT_LOG=1. Tail/rotate with soup audit-log.
• Reproducibility receipt in airgap bundles — soup airgap-bundle --repro-receipt <receipt.json> embeds an SR 11-7 receipt as repro-receipt.json; auto-detected from <model>/repro-receipt.json when not supplied.

Security

• soup can pack --attest now rejects oversize attestation files by their raw size before parsing them into memory (defence against memory-exhaustion).
• The new file-loading paths (attestation JSON, airgap receipt, training-corpus scan, PDF write) are all cwd-contained + TOCTOU symlink-rejected and size-capped; the audit auto-log redacts hf_/sk-/Bearer tokens and never crashes the CLI on a broken log.

[0.71.2] - 2026-06-01

Added

• ed25519 signing for soup adapters sign / soup attest — real detached signatures (over the adapter Merkle root / the in-toto statement) via a new [sign] extra (pip install soup-cli[sign], pulling cryptography). soup adapters sign --backend ed25519 --key <priv.pem> (or --generate-key <out.pem>, or SOUP_SIGNING_KEY); soup adapters verify [--public-key <trusted.pem>] does a cryptographic verify and, with a trusted key, genuine authentication. soup attest emit --sign ed25519 --key <priv.pem> writes a <output>.sig sidecar; new soup attest verify <statement> --signature <sig> verifies it (canonical-JSON, so it's platform/newline-independent). Sigstore keyless signing stays infra-blocked (needs an OIDC provider + Fulcio/Rekor network — can't be honestly validated offline).
• Anti-AI-Jacking namespace pin on Hub downloads — HF model fetches now consult a trust-on-first-use pin store: a repo whose author changes (or whose created_at jumps backward) is refused unless the namespace shift is explicitly allowed. Fails open when repo metadata is unavailable.
• License auto-detection at soup adapters merge — when --license isn't given, the license is read from each adapter's adapter_config.json / config.json / model-card frontmatter (HF llama3.1-style ids mapped to canonical) and the conflict gate runs automatically.
• Backdoor-scan gate at soup adapters merge — refuses to merge any input whose soup adapters scan returns FAIL (or can't be scanned) unless --allow-unscanned is passed; WARN is advisory.

Changed

• License-conflict overrides (--license-override <reason>) are now recorded to the audit log for legal review.
• The namespace-pin store now uses SQLite WAL + busy-timeout and a cross-process file lock around its get+insert, so concurrent writers don't lose the trust anchor.

Security

• ed25519 verification fails closed (any tamper / wrong key / missing key ⇒ invalid). Signing keys + trusted public keys are symlink-rejected and size-capped via a shared reader (no cwd-containment — keys live outside the project). --generate-key refuses to overwrite any existing path.

[0.71.1] - 2026-06-01

Added

• soup env fix — render a reproducible install plan from soup-env.lock. Emits copy/paste uv pip install commands (--format uv-pip, default) or a requirements.txt body (--format requirements); --output optionally writes a requirements.txt under cwd. Print-only by design — never shells out to a package manager.
• soup lock write --env-lock <path> — auto-derive --env-hash from a soup-env.lock so operators who ran soup env lock don't copy the hash by hand. --env-hash still wins when passed explicitly.
• soup serve --record-thumbs <db> — capture thumbs-up/down feedback into a local-RL SQLite at startup, plus a new POST /v1/thumbs endpoint (transformers backend). Returns 404 when the flag isn't set.
• Judge-calibration persistence: JudgeCalibrationReport.to_dict, write_judge_calibration, and load_judge_calibration, backed by a new judge_calibration registry artifact kind. Loading re-validates the report so a corrupt on-disk field is rejected.
• Bundled MUSE and WMDP unlearning eval fixtures so soup eval unlearning --benchmark muse|wmdp runs out of the box. WMDP forget-set probes ship redacted (placeholder prompts + REFUSED responses) — Soup never ships verbatim hazardous content.

Changed

• soup completions now introspects a cached base model's actual LoRA target modules (config-only AutoConfig load, local_files_only=True, never networks or raises) and falls back to the canonical default shape when the base isn't cached locally.
• build_dag exposes a validate_build_source helper (cwd-containment + symlink rejection) for build-manifest source paths.

0.71.0 - 2026-06-01

Changed

• Breaking — install split. The heavy training stack (torch, transformers, peft, trl, datasets, bitsandbytes, accelerate) moved out of the core install into a new [train] extra. pip install soup-cli is now a light CLI + data-tools install with no PyTorch; run pip install 'soup-cli[train]' (or [all]) to fine-tune. Existing users who train must reinstall with [train]. Version pins are unchanged.
• Trimmed README.md to a ~238-line front door; the full feature reference now lives under docs/ (one topic page per area, indexed from the README).
• Raised the pytest coverage gate from 50% to 77% (--cov-fail-under=77).
• Migrated to a src/ layout (src/soup_cli/) for cleaner packaging and to stop tests accidentally importing the in-tree package.

Added

• [train] and [all] optional-dependency extras ([all] pulls train, serve, ui, data). [dev] self-references [train] so CI and contributors still get the full stack from pip install -e ".[dev]".
• Friendly error mapping: a missing heavy dependency (torch, transformers, peft, trl, datasets, bitsandbytes, accelerate) now surfaces "Training needs the [train] extra. Run: pip install 'soup-cli[train]'".
• py.typed marker (PEP 561) so downstream type checkers pick up Soup's inline type hints.
• .pre-commit-config.yaml with ruff (lint + format) and standard file-hygiene hooks.
• Lenient mypy configuration and a non-blocking type-check CI job.
• This CHANGELOG.md.

Removed

• The historical, per-version security-fix log that had grown inside SECURITY.md (~220 KB). SECURITY.md is now a concise security policy; the detailed hardening notes remain in git history and the GitHub Releases notes.