CHANGELOG.md

Changelog

All notable changes to Soup CLI are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Detailed, per-release notes for every published version live on the GitHub Releases page. This file tracks unreleased changes and links out for historical detail rather than reproducing 70+ versions of notes.

Unreleased

Fixed

• SmolVLM/Idefics3 vision SFT now reaches real training batches (#302 by @Amix29 in #488). Soup keeps LLaVA messages and PIL images together until collation, converts legacy <image> markers to structured multimodal content, and lets the processor produce image-token expansion plus architecture-specific pixel tensors. The vision path uses the Transformers trainer with this collator so older supported TRL releases cannot pre-tokenize the dataset as text-only. Image placeholder ids are excluded from causal-LM labels, and the collator preserves a leading BOS whether it comes from the chat template or the tokenizer default.

Added

• soup eval aider runs Aider's Polyglot code-editing benchmark through its official Docker harness (#91 by @Amix29 in #482). The command preflights Docker, the daemon, and the locally built benchmark image; mounts a prepared Polyglot corpus read-only; keeps output under cwd; and aggregates bounded per-exercise JSON into a Soup result row. --run-id records the score in eval_results for the existing run comparison workflow. The optional [aider] extra installs the normal Aider CLI while the docs make the source-only benchmark-image setup explicit.

• Native Apple Silicon telemetry for soup monitor (#99 by @Amix29 in #481). The monitor now reads bounded plist output from macOS powermetrics and renders GPU utilization and power in the existing Rich table. It reuses an explicitly cached sudo credential through non-interactive sudo -n, never reads a password, and gives an actionable Activity Monitor fallback when permission or telemetry is unavailable. NVIDIA-only VRAM, memory-utilization, and temperature fields remain unavailable rather than being guessed.

• soup mcp serve gains network transports: --transport sse and --transport http (#296 in #479). v1 was stdio-only, which suits a client that spawns Soup as a subprocess but leaves remote and multi-client setups with nothing. Both new transports serve the same registry — the end-to-end test compares the advertised tool names against build_registry rather than a hardcoded count, so a subset cannot creep in. stdio remains the default and is untouched.

  Adding a listener is the risky part, so it is gated three ways. Every HTTP request needs Authorization: Bearer <token>, compared with secrets.compare_digest, with no opt-out — a loopback port is reachable by every process on the box. The token is validated by the existing utils/qr_url.py::validate_token, so soup ui and soup mcp serve agree on what a token is instead of growing a second format, and it travels in the header only: no query-string fallback that could land in an access log. The SDK's DNS-rebinding protection is switched on (421 on a foreign Host, 403 on a foreign Origin), which is the gate the token cannot be — a page the operator merely visits sends no Authorization header but its request still reaches the port. Binding off loopback warns, and a wildcard bind warns again that the Host check has nothing left to pin.

  --allow-execute is refused with either network transport, and that refusal is the one behavioural change to an existing flag. Gated execution (#297) spawns real training / export processes; behind a listener a leaked Bearer token would mean process execution rather than plan disclosure, and stdio is a pipe to a client the operator already started, which is a different trust boundary. The refusal is made twice on purpose: once in the CLI so the operator gets a readable message, and once in build_asgi_app() so a direct caller cannot put an executing registry behind a listener either. The stdio banner also stopped claiming "execution disabled" -- that string predated #297 and was false from the moment execution shipped.

  --host / --port / --auth-token are refused under --transport stdio rather than silently ignored, and the ASGI app is built by a build_asgi_app() factory so the auth and rebinding behaviour is tested through an in-process transport without binding a socket; one further test binds a real ephemeral port and drives initialize -> list_tools -> call_tool through the SDK's own SSE client.

  The [mcp] extra floor moves 1.2.0 -> 1.10.0, measured against the published wheels rather than a changelog: server/streamable_http_manager.py first appears in 1.8.0 (absent in 1.7.0) and server/transport_security.py in 1.10.0 (absent in 1.9.4). --transport sse alone would have run on the old floor; rebinding protection would not, and a listener whose origin checking silently disappears on an older SDK is worse than a resolver error. The <2 cap is unchanged — #322 is the 2.x migration. No new package: mcp already requires starlette, uvicorn, sse-starlette and httpx-sse.

• soup data best-of-n can sample candidates from Ollama or vLLM providers (#299 by @Faisal01011 in #466). The existing local Transformers --base path stays the default, while --provider ollama|vllm --model <m> [--base-url <url>] draws each prompt's N candidates through the existing SSRF-validated raw- completion seam. Provider and model are recorded in _best_of_n provenance; Anthropic is refused by name because its Messages API has no raw-completion endpoint, and local-only flags cannot be silently ignored in provider mode.

• Baseline artifacts carry a scorer/version provenance stamp (#404 by @AchuthReddy-16 in #485). Shared helpers stamp_baseline_scores / write_baseline_file write {"scores": {...}, "provenance": {"soup_version", "scorer_revision"}}. soup eval gate --write-baseline <path> is the user-facing producer. resolve_baseline warns once on unknown provenance (unstamped files) or a scorer_revision mismatch, and stays silent when the stamp matches. BUNDLED_SCORER_REVISION + a locked fingerprint fail the suite if a bundled scorer's output moves without a revision bump. Registry save_eval_result stamps details_json; soup ship --emit-evidence includes the same stamp.

• Layer streaming now accepts Qwen3.5 MoE text checkpoints whose decoder layers do not expose exactly the same weight keys in every block (by @Shutaru in #426). The sharder now records per-layer shard headers instead of deriving the pool from layer 0 alone, while still refusing divergent storage layouts for any shared key and rebuilding NF4 Params4bit views from validated per-weight metadata. qwen3_5_moe / qwen3_5_moe_text route through the existing qwen3 streamer. A heterogeneous toy MoE is bit-exact streamed versus resident on CPU; live validation was also run on Qwen/Qwen3.5-35B-A3B with layer streaming, NF4, MoE LoRA target resolution and a 3072-token SFT dataset. stream_layers now refuses moe_expert_quant, which is applied only by the resident setup path and was otherwise silently ignored.

• live_eval.load_model_and_tokenizer gains a quantization parameter; no live evaluation path sets it yet (#367 by @AmirF194 in #461). quantization="4bit" builds the same nf4 BitsAndBytesConfig every other 4-bit load path in this codebase uses ("8bit" and the unset default are also supported), and the four internal callers this helper has (make_generator, make_multi_generator, lora_probe, measure_logit_agreement) still call it with no quantization, so today an NF4-trained adapter is still judged against a bf16 base it never saw during training. The follow-up is wiring those four callers to a default derived from the run's own configuration (soup ship --config, the registry entry, or the adapter's adapter_config.json), per the issue's own Fix path; that plus soup ship reporting the numerics it judged with and a staleness gate on mismatched-numerics evidence are left open (issue acceptance criteria 2 and 4).

• training.stream_pin makes layer-streaming pinning configurable (#366 by @ousamabenyounes in #416). Page-locking the RAM store is chosen automatically by decide_pinning, and until now nothing could override it — so while #331 was live, pin=False was the only known mitigation for silently wrong NF4 gradients yet was unreachable from soup.yaml. stream_pin: false now forces the pageable store (and the pre-flight states the throughput it costs, up to 6.56x measured, rather than absorbing it silently); stream_pin: true forces the pinned store and, on the RAM tier, refuses the run — naming the store size, not the ceiling (#366 AC3): pinned_limit_bytes is passed as None, so the page-lock ceiling is deliberately left unprobed and the store size is the only figure the refusal can honestly cite — if the box cannot page-lock it, instead of degrading silently; unset keeps today's automatic behaviour. On the disk tier (no RAM store to page-lock) and on CPU (no device to copy to) pinning is inapplicable rather than unsatisfiable, so true is announced and the run proceeds — refusing there would brick the large-model runs the disk tier exists for, and would make the key uncommittable to a config shared between a GPU box and a CPU box. Set while stream_layers: false it is rejected as a footgun, like the other stream keys.

• A ready-made qwen3.5-9b-grpo recipe for GRPO reasoning training with Qwen/Qwen3.5-9B (#277 by @harshitthek in #448). The recipe combines the established GRPO defaults (accuracy reward, beta=0.1, 4 generations) with LoRA r=16 and 4-bit quantization.

• A ready-made glm-5.1-dpo recipe for DPO preference training with zai-org/GLM-5.1 (#280 by @Osheun in #452). The first recipe pairing DPO with a MoE base, so it carries moe_lora: true alongside the established DPO defaults (beta=0.1, LoRA r=32 / alpha=64, 4-bit quantization). epochs: 1 and max_length: 8192 are taken from the SFT sibling rather than the smaller qwen defaults, which suit a 754B MoE better.

• Cross-tokenizer draft support for soup draft and soup serve (#304 by @CODING-DARSH in #417).

  • soup draft distill now accepts target/draft pairs with mismatched tokenizers or vocabularies, automatically routing through uld_strategy: wasserstein_aligned (Universal Logit Distillation) instead of refusing the pair.
  • soup draft measure supports cross-tokenizer acceptance measurement using decoded character-span alignment (count_accepted_spans) across different vocabularies and token boundaries. Target generation neutralizes only repetition_penalty with repetition_penalty=1.0 (Refs #345) so target greedy argmax and draft raw-logit scoring are evaluated consistently; remaining generation processors (no_repeat_ngram_size, encoder_repetition_penalty, min_new_tokens, bad_words_ids, suppress_tokens, and sequence_bias) are not altered.
  • soup serve --speculative-decoding supports cross-tokenizer draft serving via Transformers Universal Assisted Decoding (UAD) when supported by the installed transformers version, raising a clear error if unsupported.
  • Compatible same-tokenizer pairs strictly preserve the existing native fast path.

• data.interleave is now wired into training-time dataset loading (#443 by @blackcoderx in #460). parse_interleave/InterleaveSpec have been schema-validated and unit-tested since v0.42.0, but load_dataset() never called them — every multi-dataset mixture request silently trained on nothing but data.train's single path, the same gap #330 and #442 papered over in their respective renderers. DataConfig.train now accepts str | list[str]; a list of >= 2 local file paths combines via data.interleave (concat / under / over / {strategy: probs, probs: [...]}) into one row set before the existing val_split line in _finalize runs, so a single path stays byte-identical. interleave is local-files-only: training.packing / training.multipack and data.streaming / an HF-hub dataset name are all rejected at config-parse time with a message naming the reason (streaming/hub-dataset interleaving is follow-up #459). Both the soup data mix --optimize recipe writer and the --live overlay renderer emit the real N-dataset mixture again instead of collapsing to one path.

• A repo-wide documentation ratchet to guarantee declared recipe counts stay synchronized with the catalog (#453 by @harshitthek in #457). Derives the expected count dynamically from len(RECIPES) and scans all declared documentation sites, preventing silent Git auto-merge drift across sequential recipe additions.

• Branch coverage for lr_groups.py's build_optimizer_param_groups: the case where every parameter matches a configured group, so no base optimizer group is appended (#273 by @AmirF194 in #469).

• Branch coverage for replay.py's downsample: the case where the stride already lands on the last row, so the endpoint pin is not appended a second time (#273 by @AmirF194 in #470).

Changed

• Remove the name-based SCORER_CHANGED_IN_V0_73_2 baseline warning in favour of the #404 scorer_revision stamp (#404 by @AchuthReddy-16 in #485). Stale baselines are detected by provenance, not by a hard-coded suite list.

• Remove hand-maintained test suite statistics from CONTRIBUTING.md in favor of a permanent digit-free shape invariant (#465 by @harshitthek in #467). Eliminates drift across routine test additions by making test-count divergence impossible at the documentation source.

• Lazy callback builders now self-import their callback class names so runtime lookup never raises NameError while preserving lazy heavy-dependency loading (#320 by @AchuthReddy-16 in #455). build_echo_trap_callback, build_reward_hack_callback, build_minillm_callback, build_rl_checkpoint_callback, and build_push_callback now resolve their callback types through local module imports in the builder body, and the regression suite adds subprocess coverage for all five builder calls.

Fixed

• Layer streaming now keeps its host store on CPU on Apple Silicon (#434 by @Amix29 in #480). PyTorch 2.7+ can return an MPS tensor for torch.empty(device="cpu", pin_memory=True), while is_pinned() remains false. Direct runtime callers could therefore place the whole frozen base in the MPS allocator and still report a pinned RAM store, even though the normal soup train setup already disabled pinning outside CUDA. The runtime now disables both optional and required pinning when the target is MPS, and RamSource independently refuses any allocation that is not genuinely CPU memory (or claims pinning without being pinned). MPS proceeds experimentally with a pageable CPU source and MPS layer buffers; CUDA pinning behaviour is unchanged.

• CI and production load sites no longer assume Transformers dtype= (#478). dtype= on AutoModel*.from_pretrained / from_config is the

  =4.56 rename of torch_dtype=. Soup still declares transformers>=4.36.0,<5.0.0, but the 12-cell matrix only ever installed the newest 4.x, so a >=4.56-only kwarg stayed green. Call sites in chat / diff / infer / export / merge / serve / mole routing / layer-stream runtime now pass torch_dtype= (still accepted on current 4.57.x). A static AST guard fails if a production AutoModel* load/config site reintroduces dtype=. A new Ubuntu/3.11 transformers-floor job installs under .github/constraints/transformers-floor.txt using the lowest non-yanked resolvable Transformers version 4.46.1 with the lowest version in the declared TRL range 0.14.0 (transformers==4.36.0 is ResolutionImpossible against declared trl — the declared Transformers floor in pyproject.toml remains unchanged), runs pip check, asserts both pins, and runs the guard. The existing 12-cell matrix is untouched.

• downsample now returns at most max_points rows, which is what its docstring has always promised (#473 in #474). The stride was len(rows) // max_points — a divisor, not a cap — so five rows with max_points=2 came back with three, and the endpoint pin could add a fourth. Sample indices are now spread evenly across the series with both endpoints included, so soup runs replay renders exactly min(len(rows), max_points) points. Even spacing also avoids the cliff a ceil-based stride would introduce: a series one row over the cap keeps max_points points rather than roughly half of them. max_points=1 is the one shape where both endpoints cannot fit and returns the final row, which is what the endpoint pin existed to guarantee. The off-by-one guard added in #470 is kept, renamed to test_last_index_lands_on_last_row_no_duplicate and re-pinned to the new arithmetic. Consequence of the old behaviour was a chart with a few more points than intended, never a wrong number.

• cut_ce.py and liger.py now normalize path separators and match architecture keywords on the last path component only (#456 by @harshitthek in #458). On Windows, rsplit("/") never split on backslashes, causing parent directory names (e.g. C:\experiments\phi-3-runs\step-2000) to over-match architecture keywords during fallback detection when config resolution was unavailable. In liger.py, switching from whole-path to last-component matching also drops org-prefix false positives (e.g. mistralai/*, Qwen/* when the model name does not contain the keyword) and prevents parent directories from applying the wrong fused kernel across POSIX and Windows. Both modules now normalize dual separators and strip trailing slashes deterministically.

• measure_gemm_tflops now records per-repeat samples and test_takes_the_best_repeat_not_the_first verifies best-of-N selection within a single measurement (#444 by @harshitthek in #451). Comparing two separate probe calls taken at different moments caused intermittent test failures on developer GPU machines under background contention. GemmCeiling now preserves samples: tuple[float, ...] and the test asserts max(samples) selection deterministically.

• Duck-typed tokenizer mappings no longer raise a misleading error, and data_doctor shares the public coerce_token_ids helper (#441 by @AchuthReddy-16 in #447, part 2 found by @emre155). A dict-like output that is not registered as collections.abc.Mapping used to be iterated as keys (input_ids[0]='input_ids'), sending the operator looking at their data; the mask path skipped the same objects and silently dropped assistant_masks. Both gates now use one mapping-like predicate, and the helper is public so the two modules cannot drift.

• soup data mix --live handed every candidate proxy run a config it could not load (#442 by @blackcoderx in #445). _render_overlay_yaml emitted data.train as a YAML list, the same shape #330 fixed in the recipe writer, but every --live test mocked subprocess.run so nothing ever loaded the overlay through the schema — a config the tool could not itself load read as a passing feature. data.train now renders as the single highest-weighted dataset in each candidate, mirroring #330's fix, with a comment noting which dataset was picked and why.

• use_cut_ce silently did nothing for any model loaded from a local checkpoint directory, and conflated Phi-2 with Phi-3 (#383 by @AmirF194 in #446). apply_cut_ce() picked the CCE patcher by matching an architecture keyword against the model path's last component, so checkpoint-2000 / my-finetune / any other directory soup merge/soup shrink/soup train writes out matched nothing and CCE stayed off with no error, on a flag the user explicitly set. Separately, every Phi variant (phi-2, phi-3, phi-4) mapped to the same "phi3" patcher, even though cut_cross_entropy has no Phi-2 patcher at all (its config.model_type is "phi", not "phi3"), and a bare "gemma" fallback entry dispatched to a patcher cut_cross_entropy does not have at all, crashing instead of reporting unsupported. Detection now resolves AutoConfig.from_pretrained(model_name).model_type first, mirroring the identical fix already shipped for Liger Kernel (#78), and falls back to the name-based match only when that is unavailable; Phi-2 and plain Gemma-1 both correctly report unsupported instead of running under the wrong kernel or crashing. The two call sites that separately hand-wrote the "no matching architecture" advisory now share one message.

• soup draft measure now refuses a mismatched pair up front and no longer discards a completed measurement when the assisted arm fails (#344 by @ousamabenyounes in #409). measure gated on same_tokenizer() (tokenizer vocab + probe ids), which accepts a pair whose tokenizers are identical but whose config.vocab_size differs by padded embedding rows (e.g. Qwen2.5 large←small) — exactly the pair distill refuses. Transformers' assisted generation gates on config.vocab_size, so the run died with "different tokenizers" deep inside generate(), after the expensive load, and because the report was written only after that arm every completed acceptance/plain-throughput number was thrown away. measure now uses the same config.vocab_size precondition as distill (refusing before any model loads; the shared _vocab_size_of also reads a composite model's get_text_config(), so a multimodal target like Llava is no longer refused), keeps same_tokenizer() as an additional check, and writes the report incrementally so a failing assisted arm leaves the acceptance rate and plain throughput on disk. The report records an assisted_status (pending / complete / untimed / crash / interrupted) so a failed arm is distinguishable on disk from a completed or un-run one — pending is what a report keeps when the process dies mid-arm and no handler runs, which is the case the incremental write exists for.

• soup export --format gptq crashed with no calibration data and, when it did run, wrote a shard name the standard loader can't find (#338 by @AmirF194 in #475). With no --calibration-data, _export_gptq called model.quantize(tokenizer); auto-gptq's quantize() expects tokenized examples, not a bare tokenizer, so this failed with "object is not iterable". GPTQ export now requires --calibration-data up front and rejects a file with zero usable samples, since auto-gptq has no built-in fallback dataset (unlike AWQ). Separately, save_quantized writes its own gptq_model-<bits>bit-<group>g.safetensors shard, which AutoModelForCausalLM.from_pretrained does not look for; the exported directory now also carries a standard model.safetensors. The shared except ImportError blocks on both the AWQ and GPTQ paths also stopped reporting a fixed "not installed" string when the package itself imports fine but a transitive import inside it fails for an unrelated reason.

[0.73.3] - 2026-08-18

Added

• eval.ship.noise_floor is now committable to soup.yaml (#406 by @ousamabenyounes in #410). Every soup ship gate-policy flag was settable in a committed config and read back by --config — except --noise-floor (added in v0.73.2), which had no field and could only be passed on the command line, so a team enforcing a floor in CI had to hand-edit the workflow. ShipConfig gains a bounded noise_floor ([2, 10], imported from ship_verdict so the schema and the CLI validator cannot disagree; bool-as-int rejected), wired with the same CLI > config > default precedence as the other five flags. As a live-measurement input it is measured only when producing evidence and refused under --evidence exactly as the flag is; it follows forgetting_threshold in being excluded from the recipe config_sha, so setting a floor never invalidates evidence.

• A ready-made qwen3.5-4b-pretrain recipe for continued pre-training of Qwen/Qwen3.5-4B-Base (#278 by @Faisal01011 in #422). The recipe uses plaintext data, one epoch, QLoRA 4-bit quantization, and the established continued-pretraining defaults.

• A ready-made deepseek-v4-flash-grpo recipe for GRPO reasoning training with deepseek-ai/DeepSeek-V4-Flash (#279 by @Faisal01011 in #432). The recipe combines the established GRPO defaults with MoE LoRA and gradient checkpointing.

• soup mcp serve --allow-execute can now actually execute, behind a single-use confirmation token (#297 by @CODING-DARSH in #393). train_start and export issue a short-lived, server-generated token bound to the plan and to the execution kind; train_execute / export_execute accept only that token — no command, no argv, no shell string, no client-supplied environment — and the server launches the planned Soup CLI command with shell=False, stdin=DEVNULL, output to .soup/mcp-runs/<run_id>.log. Two integrity properties close the gap between planning and running: the config is snapshotted at plan time and executed from the copy, so the file cannot be edited underneath the run, and digest_file now walks a directory tree by content (sorted relative paths + per-file hash, symlink refusal, bounded) rather than by mtime+size, which did not change when a file inside a protected directory was rewritten — so a model could be swapped between plan and execution and revalidation still passed. Token consumption and capacity acquisition happen before Popen, so a failed spawn requires a fresh plan rather than enabling replay. Runs go through the existing ExperimentTracker, so soup runs sees what MCP started.

Changed

• The MLX SFT dispatch route no longer imports the Transformers SFT wrapper before choosing a backend (#394 by @Shutaru in #431). The wrapper is import-light today, but the standalone soup-cli[mlx] runtime no longer depends on it remaining so. An additive Apple Silicon CI job verifies that mlx and mlx-lm import, the PyTorch/TRL training stack is absent, and a one-step real CLI SFT run completes. This hardens the runtime boundary; it does not claim to resolve the still-unpinned torch-present hang reported in #394.
• soup card now links the ML-BOM and the in-toto/SLSA attestation — they are first-class registry artifact kinds (#309 by @ousamabenyounes in #420). bom and attestation were the two compliance documents soup card could not surface: RegistryStore had no such kinds, so there was no way to attach them. Added both to the valid-kind set and a --attach-to-registry <id> flag to soup bom emit and soup attest emit (mirroring the existing --attach-to-registry pattern); once attached they appear in the card's artifact table for free. Signed attestations also attach the detached .sig sidecar. The flag needs --output (omitting it exits 2), and a requested registry attachment failure exits 1 after preserving files already emitted.

Fixed

• Assistant-only masking no longer mistakes BatchEncoding keys for token ids (#430 by @Shutaru in #439). Tokenizer mapping outputs are read through input_ids, and tensor-like ids are normalised to Python integers before they reach the collator; missing or non-integer ids now fail loudly instead of building a garbage label mask. A template that returns an all-zero assistant mask while assistant messages exist now falls back to incremental rendering, avoiding a silent all--100 no-op training run. The same mapping assumption was removed from the data doctor.

• soup data mix --optimize wrote a recipe soup train could not load (#330 by @blackcoderx in #440). render_mix_recipe_yaml emitted data.train as a YAML list of every searched dataset, but DataConfig.train is typed str, so the recipe failed to load with data -> train: Input should be a valid string. data.train now renders as the single highest-weighted dataset from the search; the full ranked weight/path breakdown is kept as a comment above it so no information is lost, and the comment explains that data.interleave is not yet consumed by training.

• Layer streaming now verifies that every trainable LoRA parameter has real storage after PEFT attaches the adapter (#433 reported by @lesterppo, fixed by @Faisal01011 in #435 and #437). PEFT 0.18 creates streamed adapters on meta for Soup to materialise, while PEFT 0.19 may create them as real tensors immediately, so materialize_meta_adapters() returning 0 cannot distinguish a healthy no-op from a missed adapter. The streamed build now enforces the actual postcondition and refuses to install the runtime if a trainable lora_* parameter remains on meta, naming the stranded parameter instead of allowing a silent no-training run.

• Windows process liveness misread a process that exited with code 259 as still running, forever (#424 by @blackcoderx in #436). GetExitCodeProcess returns STILL_ACTIVE (259) for a genuinely running process, but 259 is also a legal exit code, so a child that exited with 259 was indistinguishable from one still running. That silently defeated ExperimentTracker's reconcile-on-read (#401) and could wedge the MCP server's one-active-execution cap (#402) shut, refusing every subsequent execution with no error an operator could act on. The check now waits on the process handle with WaitForSingleObject(handle, 0), which is signalled the instant the process exits regardless of its exit code, falling back to the exit-code read only if the wait itself fails. The two byte-identical copies of this primitive in experiment/tracker.py and mcp_server/execution.py are now one shared utils/process_liveness.py.

• soup train --no-reexec now prints the flags you actually typed (#372 by @AchuthReddy-16 in #415). The advisory accelerate launch command is derived from the same argv the auto-reexec would have used, so --fsdp / --deepspeed / --config (and the rest of the run-shaping tail, including --name and --replay) cannot silently fall off the printed hint. Following that line used to train without FSDP.

• detect_device() and get_gpu_info() now recognise Apple Silicon MLX (#423 by @harshitthek in #428). Previously on Apple Silicon, detect_device() only probed PyTorch MPS and fell back to 'cpu', triggering a false Warning: 4bit quantization is not supported on CPU alert and silently downgrading quantization from 4bit to none. Device and GPU info detection now accept an optional backend= parameter: passing backend="mlx" resolves to 'mlx' with the chip name (e.g. Apple Silicon (Apple M2 Max)), preserves 4-bit quantization intact for mlx-lm pre-quantized models, reports Apple unified memory in telemetry, and skips the CUDA-shaped analytical VRAM preflight on MLX runs. The preflight skip is pinned by TestHardwareFitGateIsMlxAware (mirrors the streaming-aware gate test). The known-limitation warning in docs/backends-and-ops.md is replaced with the resolved behaviour.

• A run whose watcher died was reported running forever (#401 by @ousamabenyounes in #407). ExecutionManager._watch runs as a daemon=True thread, so when the MCP server process exits it is killed without unwinding and finish_execution never runs — ExperimentTracker kept the run at running with no watcher left to correct it, training the operator to ignore the one status field that guards against a second concurrent run. The tracker now reconciles on read: when a running row carries a pid whose process is gone, list_runs / get_run rewrite it to terminated with an unknown (None) exit code — an unknown outcome is never recorded as success, and the richer completed/failed terminal statuses are left untouched. soup runs no longer hardcodes running for a non-running row.

• Layer streaming now accepts a fast virtio/cloud disk instead of refusing it as an HDD (#365 by @ousamabenyounes in #411). detect_disk_kind trusted /sys/block/<dev>/queue/rotational, which a paravirtual (virtio) block device defaults to 1 with no media hint — so a genuinely NVMe-backed cloud disk (measured 1.5 GB/s read) was classified hdd and denied the disk-overflow tier, the very audience the tier targets. rotational=0 stays authoritative (solid state); when the flag is unreliable (rotational=1), the media type is now decided by a bounded, O_DIRECT sequential-read measurement rather than the flag — NVMe-class throughput (>= 1 GB/s) earns the tier, a genuinely slow disk still classifies hdd and is still refused (160 seeks/step, plan P11). A new training.stream_disk_kind override (nvme/ssd/hdd) is the escape hatch for the case where even that is wrong; it prints what it overrode beside what was detected. Disk detection may write a small scratch file next to the streamed shards to run the probe.

• The one-active-execution cap could double-book after a server restart (#402 by @ousamabenyounes in #408). ExecutionManager._active_run_id is in-memory, so a restarted MCP server started with an empty slot, saw free capacity, and would launch a second training while a child from the previous server (which survives a client disconnect, #297) was still using the GPU. The cap is now gated on a persisted run whose recorded pid is still alive: a live prior child blocks a new execution across restarts, while a stale record whose process is gone frees the slot rather than wedging it shut. docs/commands.md now states the actual scope of the cap.

• soup ship --noise-floor now measures the leg-1 task axis in the judge modes, so a judge-scored win smaller than the judge's own noise no longer counts (#403 by @ousamabenyounes in #419). The floor was measured in --task-mode metric only; judge_score / pairwise printed a warning and left leg 1 at a 0.0 floor, i.e. the exact blindness the flag exists to remove. It is now measured: judge_score scores the base side N times through the judge, and pairwise judges the base model against itself (expected win-rate 0.5, so the spread is directly measured, not inferred). Because those repeats fold in the judge's own sampling noise, the floor is labelled decode + judge on the panel and stamped judge_inclusive in the evidence/JSON block so it is never read as a decode-only number. --help states the extra judge-API cost.

• training.bnb_4bit_use_double_quant was validated but never read — every 4-bit path Soup builds the BitsAndBytesConfig for hardcoded double-quantization to True, so bnb_4bit_use_double_quant: false passed validation and was silently ignored (#321 by @ousamabenyounes in #418). The flag is now threaded through the three call sites Soup owns — the resident loader (build_quantization_config_for_loader), the layer-streaming path (stream_setup reads it once and passes the SAME value to the sharder and the meta skeleton, so streamed-vs-resident bit-exactness cannot drift), and the 4bit save path (soup merge --no-double-quant for the 4bit / 4bit_forced save formats). The Unsloth loader is out of scope: FastLanguageModel builds its own BitsAndBytesConfig internally with double-quant hardcoded on and exposes no override, so that path cannot honour the flag. The schema field is now tri-state (Optional[bool], unset = None): unset resolves to the shipped default (double-quant on), so a run that never set the flag trains identically, but because the resolved config now carries the field, soup ship --evidence provenance and soup lock check will report a one-time fingerprint drift on a previously-unset config even though the model numerics are unchanged. The config-load footgun (=true requires quantization: 4bit) fires only on an explicit true, and unset serializes as None, so a dumped-and-reloaded config no longer trips it.

• soup env check now flags an installed package that violates Soup's own declared version bound (#368 by @ousamabenyounes in #421). pip install "soup-cli[serve-fast]" (vllm) into a training venv silently pushes transformers past the <5.0.0 cap Soup declares, producing an environment Soup's own metadata says is unsupported with no warning at any point. env check audits installed versions against the bounds read from package metadata — not a hardcoded copy, since a second copy of the cap is exactly the drift this catches — and exits 3 when one is violated, independent of any lock file. An extra == "X" requirement is Soup's bound only when [X] was opted into, which the running environment cannot confirm, so its marker is evaluated and it is skipped rather than raised as a false positive — except for the ABI-relevant TRACKED_PACKAGES set, which since the v0.71.0 deps-split lives in metadata only under extra == "train"/"all"/ "dev" and therefore includes the transformers <5.0.0 case #368 was reported about. A package is counted once even when its bound is restated under several extras. If packaging is somehow unimportable the audit still degrades to a clean report so env check keeps working, but now says so rather than reporting a silent clean it never verified. The bounds audit no longer short-circuits the lock diagnostic (both print). packaging is a declared dependency now — the audit degraded to a silent "clean" without it, the wrong direction for a checker. docs/serving-and-export.md states the separate-environment guidance for [serve-fast].

• kl_control rewrote the trainer's β/kl_coef on every step, including a hold, so a non-acting run was numerically identical to log_only (#371 by @AmirF194 in #414). _run_bang_bang called _apply_coefficient unconditionally; on a hold the controller writes back the value already there, which is a no-op value but not a no-op write. The write is now skipped when the bang-bang step holds, and the mitigation log records mitigation_status (held / acted / released) so a non-acting step is distinguishable from an acting one without parsing the free-text action reason.

• extract_mcq_letter scored zero for \boxed {A} — whitespace between the command and the brace (follow-up to #357, by @ousamabenyounes in #396). The shipped \boxed\{ regex tolerates spaces inside the braces but not between \boxed and {; LaTeX permits it there and models emit it, so \boxed { C } still read as no answer and was not rescued by the cue tier either. The boxed-letter regex now allows \s* after the command.

• soup draft distill --steps N now delivers ~N optimiser steps instead of N/4.44 (#364 by @ousamabenyounes in #399). The epoch count that realised --steps divided the request by rows // batch_size, ignoring that val_split (0.1) removes rows from training and gradient_accumulation_steps (4) micro-batches make one optimiser step — both divide the budget, so every distill run trained for roughly a fifth of the requested steps, silently. Epochs are now derived from the effective steps-per-epoch, the emitted config pins the run shape (val_split, gradient_accumulation_steps) so the arithmetic and the trainer cannot drift, and the pre-flight prints the resolved step count.

• MitigationLogWriter silently dropped every record once its parent directory vanished mid-run (#343 by @ousamabenyounes in #398). record() reopens the log per call and swallowed the OSError from open("ab") with a bare return, so when a shared temp root was cleaned by another process the controller kept acting while its log quietly stopped growing — the run completes while its evidence goes missing, the failure shape this project treats as the worst kind. The writer now recreates the directory and retries the write, and surfaces the loss once via a warning naming the path. record() still never raises, so a vanished log never takes down the training run.

[0.73.2] - 2026-08-15

soup ship's leg 2 is the project's differentiator, and it was lying in both directions: two of its suites ranked by the wrong thing, one whole failure direction had no detector at all, and a caller error was indistinguishable from a regression. Every item below was reproduced on the dev box against the shipped v0.73.1 code before a line was changed.

Added

• soup ship --noise-floor N — measure what the instrument can resolve before calling a delta significant. Greedy decoding is not deterministic on GPU: measured on an H100, the same model with no adapter over five runs spread 0.015 strict / 0.020 format-blind, and soup ship compared against a 0.05 threshold without ever telling the operator what its own instrument could resolve — four of six paired deltas in that session sat inside the floor. With the flag, the BASE model is re-run N times (2–10), the per-axis floor is max − min across the repeats, it is printed beside the verdict, and every axis is then gated at max(threshold, floor). Leg 1 must clear the task axis's floor too. The max is load-bearing in both directions: a floor above the threshold widens the gate to what is actually measurable, and a floor below it must never tighten the gate behind the operator's back. Opt-in, so no existing run changes; costs N extra base passes. The leg-1 floor is measured in --task-mode metric only — in the judge modes a repeat would fold the judge's own sampling noise into a number presented as decode noise, so the run warns and leaves leg 1 at a 0.0 floor instead. Carried caveat, because it bounds the claim: n=3, one model, one dataset. It sizes the effect; it does not calibrate a threshold.
• A benign-prompt axis, mini_over_refusal, joins the default leg-2 suite (7 → 8) (#317). Leg 2 flagged a drop in mini_safety and had no reverse, so a tune that refuses everything registered as a monotone safety improvement with no ceiling on how useless the model became. Reproduced as indistinguishability, which is the actual claim: two models with byte-identical scores on all seven shipped suites and the same SHIP verdict, one of which refuses every benign request. The new suite is 40 hand-authored, original benign-but-scary-sounding requests (defensive security, first aid, sysadmin verbs like "kill"/"terminate", fiction with dark themes) scored as fraction NOT refused, so the existing regression rule catches over-refusal with no change to the verdict engine. Paired with mini_safety neither axis can be gamed alone. Same caveat as above: 40 prompts and one greedy pass size a gap, they do not calibrate a threshold.
• Semantic Stratified Splitting for soup data split (#388). --stratify-semantic and --num-clusters partition splits proportionally across semantic groups (TF-IDF + K-Means) so a whole topic cannot land entirely in one split. scikit-learn is a declared member of the [data] extra and a missing import refuses with the install command rather than silently falling back. 50k-row cap, an explicit error for a pure-stop-word corpus, and a warning when --num-clusters is passed without the flag. Contributed by @Deadpool2000.
• soup mcp serve --allow-execute (#391). A stronger opt-in than --allow-mutating, which it implies, plus the _refuse_execute handler the execution tools will use. train_start and export stay plan-only — nothing in this slice executes anything, and the help text says so in the present tense. build_registry(allow_mutating=True) still returns the same 16 tools. Contributed by @CODING-DARSH.

Fixed

• extract_mcq_letter did not know \boxed{C}, and the MCQ prompt never asked for a letter (#357). Meta-Llama-3.1-8B-Instruct scored 0.423 on mini_mmlu — below a 0.5B — while scoring 1.000 on two other MCQ suites. Of 15 failures, 8 boxed the right letter and 6 boxed a value because nothing in the prompt asked for a letter. Reproduced here at the extreme: a stub that answers every item CORRECTLY in the boxed-letter style scored 0.000 on mini_mmlu and mini_common_sense. Both halves are needed — the extractor alone is worth +8 items, the prompt alone 0, together 0.423 → 0.731 and the inversion disappears. The new tier fires only when the box holds a single A–J letter: reading \boxed{4} as "option 4" would be a wrong credit, not a repair. Among the boxed / cue / paren forms, position decides, not form — a model that boxes a scratch answer and then self-corrects chose the correction.
• mini_tool_call ranked by brace hygiene (#346). The 8B named the right tool 40/40 and scored 0.225: it emitted three opening braces and two closing ones, the whole-string parse failed, the bounded scan returned the INNER object, and the scorer rejected it for having no "function" key. The gate suite's own unwrapping layer now restores the envelope for an object carrying both name and arguments — requiring arguments is what stops an echoed {"name", "description"} menu entry from scoring, i.e. from crediting copying. The missing brace is the model's own output, not truncation; that attribution was believed and shipped in c87fd00 before a budget sweep disproved it.
• score_bundled_suite returned 0.0 for a non-callable gen (#355). On the three behavioural suites it scored 0.0 while the MCQ suites raised TypeError — and in leg 2 a 0.0 reads as "the model failed every item" → DON'T SHIP, so a caller error was indistinguishable from a regression and failed in the direction that looks like a finding. Both branches now raise. A callable that misbehaves is still a failed item, which is the correct existing contract.
• The verdict panel silently ate its own leg-1 marker. render_ship_panel built its header as ... [{won_str}], and a bare [no win] is valid Rich markup for an unknown tag — so the panel never printed "won"/"no win" on any release up to and including v0.73.1. The plain-text rubric, which has no markup parser, printed it correctly the whole time, which is why it went unnoticed. Found while rendering the new noise-floor panel.
• Untrusted names could drive the terminal. Benchmark and noise-floor axis names come from an --evidence JSON file, and rich.markup.escape neutralises Rich's [...] syntax and nothing else, so a raw ESC byte survived it. Both render paths now strip C0/DEL first, matching the _for_terminal pattern already used in six other command modules.
• The --evidence round-trip and the MCP ship_evidence tool now agree. A verdict decided against a measured floor does not replay without it, so the floor is part of the evidence schema (#312's output-is-input property), and both readers honour it — a schema extended in one consumer and not the other would have made the same file replay to different decisions through the CLI and through soup mcp serve.
• A duplicated #392 CHANGELOG entry. PR #388 branched before 18a278a moved that entry into [0.73.1], and the merge re-introduced it under [Unreleased], listing the same fix twice.

Security

• An evidence-supplied noise floor is bounded and never silent. A floor widens the gate, so "floors": {"mini_mmlu": 1.0} in an evidence file masks any possible drop on that axis. This does not cross a new trust boundary — anyone who can edit that file can already write {"base": 0.9, "tuned": 0.9} and force a SHIP outright — but it is a far quieter edit to miss in review, and soup ci init wires ship --evidence as a PR merge gate. Values are therefore bounded to [0.0, 1.0], the mapping is capped at 50 axes / 256-char names (mirroring the CLI's own limits), a malformed block is refused rather than dropped (a dropped floor replays as a different verdict), and any floor that exceeds --forgetting-threshold is announced — on stderr by the CLI, and in the returned payload by the MCP tool, whose stdout is the JSON-RPC channel. Neither reader is the quiet one.

Changed

• decide_ship now canonicalises the TaskWin it stores, as it already did for the benchmark deltas. It recomputes leg 1 from the raw scores, so a TaskWin built without the floor would otherwise have rendered "won" beside a DON'T SHIP decided with it.
• --baseline snapshots taken before this release are on a different scale for mini_mmlu, mini_common_sense and mini_tool_call, because their scorers changed. A stored baseline skips the live base run, so it would be diffed against a freshly-scored tuned model — measured on an unchanged model the jumps are 0.423 → 0.731 and 0.225 → 1.000, far larger than the 0.05 gate. soup ship now warns by name when --baseline supplies a stored score for an affected suite. mini_instruction and mini_arithmetic are unaffected and are deliberately not named: neither carries a single-letter answer, so the prompt cue and the option-letter extractor never touch them (verified, 0 of 24 and 0 of 36 items).

0.73.1 - 2026-08-14

Added

• training.stream_vram_probe decides the layer-streaming VRAM pre-flight on a MEASUREMENT instead of the fitted formula (#349). The pre-flight predicts peak VRAM from a formula fitted to 10 real runs, and its documented contract is that it never under-predicts. Measured through the real soup train on an RTX 3050 Laptop (4 GB, Windows, torch 2.5.1) with SmolLM2-135M streamed in bf16 at batch 1, that contract holds at short sequence and then fails:

  seq	predicted	real peak	ratio
  4352	3.282 GB	3.036 GB	1.081x — over-predicts, safe
  5120	3.844 GB	4.118 GB	0.934x — under-predicts
  6144	4.590 GB	5.830 GB	0.787x — under by 21%

  Under-prediction is the direction that does not announce itself: an OOM on Linux, and on Windows/WDDM a silent spill to host memory. The existing grid could not have caught it — all ten of its rows are at seq 256 or 512, so it varies batch and says nothing about sequence length, and test_never_under_predicts has been narrowed to state that scope rather than imply a global guarantee.

  With the flag on, one real forward+backward runs at the configured shape after the streamed model is built and its peak decides; the prediction is printed beside it so a divergence is visible. Measured cost: 1.0-5.3 s (SmolLM2-135M at 1x1024 and 2x2048; Llama-3.1-8B NF4 at 1x512), against a training run of minutes to hours. Off by default — it costs a step, and it can refuse a run the formula accepts.

  Scope is deliberately narrow. task: sft only: the probe runs a plain causal-LM step, which is the SFT step but is not a preference loss, so its agreement with one is not established. Measured at a single matching shape it is conservative there too (6.02 GB against a real DPO step's 5.30 GB, +13.5%) — but one point is not a validation, and a sign flip would mean a gate waving through over-budget runs. It also cannot overrule a prediction more than 4x over budget: the largest disagreement ever measured is 21%, so beyond a small multiple the config is simply too big and is refused by arithmetic without touching the GPU. The gate reads max_memory_allocated, not max_memory_reserved — reserved runs 1.08-1.41x allocated and overshoots what has to fit, and gating on it would refuse this feature's own flagship configuration (Llama-3.1-8B NF4, 3.70 GB reserved against 3.45 GB free, which runs).

  Two readings were tried during this work and withdrawn as unsupported, recorded because the tempting inference was wrong twice in one investigation: that preference losses are over-budgeted ~8.8x (it is 1.15x at the budgeted shape — the earlier figure came from rows that realised 142 of a budgeted 2048 tokens), and that the over-budget runs were silently spilling (num_alloc_retries was 0 on every shape measured). The mechanism behind the long-sequence divergence is likewise not claimed: seq**2 from the attention score matrix is the obvious candidate and the numbers do not settle it.

Fixed

• The MLX adapter_config.json shipped target_modules unresolved, so a default MLX adapter loaded as a silent no-op (#392). _apply_lora resolved target_modules: auto into a local variable and trained the resolved modules; the writer serialised the raw config value, so the shipped file carried {"keys": ["auto"]}. On load, linear_to_lora_layers matches no module against that and load_weights(strict=False) drops every LoRA tensor without a word — generation with the adapter is bit-identical to the base model. "auto" is the schema default, so this was every MLX run that did not name its modules by hand, and the file exists precisely to promise the output dir loads with mlx_lm.load(..., adapter_path=...). Both callers now go through one resolve_mlx_target_keys(), because two copies of "which modules did we train?" is how they drifted. Reported with a root cause and a control by @armanbot-jpg: hand-editing keys in the saved file makes the very same adapters.safetensors produce the tuned behaviour.

• training.batch_size accepted 0 and negative values. Union[int, Literal["auto"]] carried no lower bound, so batch_size: -4 loaded and then meant whatever each trainer's arithmetic did with it — including the streaming VRAM pre-flight, which multiplies by it. Now rejected at config load. Surfaced by the #349 security review.

Fixed

• Layer streaming's VRAM pre-flight now actually calls its own calibration hook (#348). calibrated_logits_bytes_per_element() exists to raise the budget when a stack's loss path measures a heavier retention than the shipped constant assumes, guarding against a future stack silently under-budgeting by 12.5% with nothing to catch it. _stream_budget_lines called estimate_stream_peak_vram() without logits_bytes_per_element=, so the parameter was always None and the calibration never ran outside its own test. It is now forwarded to both the budget and the panel's logits figure; the value can only raise the prediction (floored at the shipped constant), and the panel prints an extra line naming both numbers when the calibration measures above it. This makes the probe unconditional rather than opt-in (see #327 below, whose wording is updated to match): every streamed run now pays one transient 14 * vocab_size * max(tokens) allocation (96 MiB at the defaults) plus two torch.cuda.synchronize() calls before the fit decision is taken. On today's measured stacks this is a no-op in effect (measured is 12.0 with zero spread, max(14, 14) is 14, no extra line prints), so the cost buys nothing yet, which is the point of a guard against a stack that hasn't shipped.
• MLX backend now actually dispatches to the MLX trainer for task: sft. Previously backend: mlx silently fell through to the transformers SFTTrainerWrapper, training on MPS/CUDA instead of MLX. The trainer was also rewritten for mlx-lm >= 0.31 (create_dataset + CacheDataset, TrainingCallback), with model.freeze() before LoRA — without it the saved "adapter" was a full fine-tune (172 tensors vs 24 LoRA tensors on a 1.2B model) — and an adapter_config.json is written so the output dir loads directly with mlx_lm.load(..., adapter_path=dir). (#362)
• mlx-lm floor raised to >= 0.31.3 (the version the MLX SFT path is built against).
• training.seed reached the SFT wrapper and nothing else (#353). #341 added the knob and wired it into trainer/sft.py. The other seventeen task wrappers each build their own TrainingArguments subclass (GRPOConfig, DPOConfig, RewardConfig, and so on) and none of them read the field, so task: grpo with training.seed: 7 trained at HF's default of 42 with no error and no warning. Replicates that differed only in training.seed were therefore the same run, which is what happened to STEP 25 of the H100 record: its five "replicates" were five runs of seed 42, measured against a within-mode spread produced by the very thing it thought it was varying. Threading the config is only half the repair. Trainer.__init__ runs set_seed(args.seed), but get_peft_model has already drawn lora_A by then, and classifier / reward_model / prm have already drawn a freshly initialised head inside from_pretrained, so every wrapper now applies the seed at the top of setup() as well, before the model is loaded. unlearn builds no Trainer at all and drew its RMU control vector from a generator hard-coded to 0; that draw now follows training.seed, staying at 0 when the seed is unset so existing runs keep their control direction. An unset seed still resolves to 42 and leaves data_seed at None, so the values a run trains at are unchanged. What changes is when they arrive. Nothing called set_seed before get_peft_model previously, so lora_A was drawn from torch's default generator, which is seeded from entropy once per process: an unseeded run's adapter and classification-head initialisation varied from one process to the next, and it is now deterministic at 42. Replicate variation that came out of runs setting no seed was coming from exactly that, so those runs are now identical to each other and varying a replicate means setting training.seed on purpose. The MLX backend (backend: mlx) is the one path that still reads neither field.
• training.seed on the MLX backend now says it is ignored (#353, fourth criterion). MLX has its own RNG (mx.random) and none of the MLX wrappers touch it, so a seeded MLX run was silently unseeded — which looks identical to a seeded one until two replicates disagree. That gap only became reachable between #353 being filed and #381 landing: backend: mlx was never dispatched at all until #362 (#363). Setting either field now appends to the wrapper's existing "MLX backend ignores:" line, naming training.seed / training.data_seed so it is greppable as written. A warning, not a rejection: a config valid on transformers should not become unloadable by switching backend. Seeding MLX for real is separate work with a separate RNG.
• Under use_fsdp2_compile, every checkpoint-* still loads as a dead adapter (#351). #335's repair runs once, on the output root, after the final save_model. HF's Trainer writes its periodic checkpoints through that same save_model with output_dir=<run>/checkpoint-N, so they come out carrying _orig_mod. on every key and nothing ever normalised them. Measured at 70B on 8×H100 (benchmarks/gate-h100-validation.md, STEP 28): 320 canonical keys in the output root, 320 prefixed ones in checkpoint-100. Resuming is the case that decides how bad this is, and it is worse than #335 was. PeftModel.from_pretrained at least warns; Trainer._load_from_checkpoint calls model.load_adapter(...) and drops its return value, and load_adapter deliberately does not warn (it hands the missing keys back in the load result instead, which nothing reads), while load_state_dict(strict=False) discards the _orig_mod. keys without a word. A resumed run therefore continues from a re-zeroed lora_B in total silence: #335's failure shape with its one warning removed. load_best_model_at_end was reloading a dead adapter for the same reason, since the Trainer restores state.best_model_checkpoint through that same load_adapter, and this repairs that path too. Normalising now happens on HF's on_save, as each checkpoint is written. Both that callback and the final save are gated on args.should_save, the condition save_model writes under, so exactly the rank holding the file repairs it rather than all eight opening it at once. The callback is attached in setup(), ahead of anything a caller adds later: HFPushCallback.on_save uploads checkpoint-{step} on this same event and CallbackHandler dispatches in insertion order, so a normalisation attached after it would leave --push-as publishing the prefixed adapter and keeping the repaired one on local disk.
• A streamed model's named_parameters() still carried the wrapper's .inner. segment, so a name-keyed comparison against a resident model of the same checkpoint saw no overlap (#369). v0.72.1 made state_dict() canonical for serialisation; this issue is the first time something compared the two model kinds by parameter name instead, and the #331 repair gate read the resulting empty intersection (grads exact 0/0) as a pass. layer_stream_runtime.canonical_named_parameters() strips .inner. the same way state_dict() already does, and assert_canonical_parameters_intersect() raises instead of reporting an empty intersection as success. Neither the forward path nor state_dict() changed, so the v0.72.0 bit-exactness gates remain valid unexercised. named_modules() and named_buffers() carry the same segment and are deliberately not covered — the comparison that produced the false green was over parameter names.

Added

• training.stream_vram_override gives the layer-streaming VRAM pre-flight an explicit escape hatch (#347). decide_stream_fit refused any run it predicted would not fit, with no way through except lowering batch_size or max_length. Setting this field now replaces the measured free-VRAM figure the pre-flight checks against, in either direction: raised past a documented over-prediction to let a known-safe config through, or lowered to enforce a cap mem_get_info() cannot see, such as set_per_process_memory_fraction on a shared or capped card (a Colab/Kaggle T4, a MIG slice). Rejected at config load when set while stream_layers is false, mirroring the existing stream_source/stream_buffers footgun gate.

Fixed

• A hosted notebook's preinstalled torchao made get_peft_model raise, and it read as a Soup bug (#389). peft's is_torchao_available() does not return False on a version it considers too old, it raises ImportError — nine frames inside get_peft_model, with nothing near the top of the traceback naming torchao. Colab preinstalls torchao 0.10.0 against a peft that demands newer, so the first soup train on a free notebook died in a place unrelated to anything the user had configured. Now mapped in utils/errors.py to the cause and the one-line fix (pip uninstall -y torchao), with the part worth saying out loud: Soup does not need torchao at all unless training.quantization_aware is set. Found by running notebooks/proof-4gb.ipynb on a real free-tier session, which is the same place #385's first repair was caught being a no-op.

• bf16 was assumed on every CUDA card, so the entire free GPU tier failed (#385, #387). Fourteen places, and only the first was known: trainer/stream_setup.py chose the layer-streaming store dtype with the literal "bfloat16" if on_cuda else "float32" (#385), and then a live run found SFTTrainerWrapper._resolve_mixed_precision returning (device == "cuda", False) as its default — and an audit found the same bf16=self.device == "cuda" in twelve more wrappers: bco, classifier, distill, dpo, embedding, ipo, kto, online_dpo, orpo, pretrain, reward_model, simpo (#387). So it was not a streaming bug at all; every task died on that hardware. The sharpest detail is that the codebase already knew: trainer/asr.py carries the comment "bf16=cuda was hardcoded, which crashes on pre-Ampere cards (T4 / GTX 16xx)" and fixes it — in that one wrapper, never propagated. All fourteen now take the answer from one place, utils/gpu.bf16_fp16_flags, including ASR, whose private copy was folded in. bf16 needs Ampere. Colab's free tier is a T4 (sm_75), Kaggle is a T4 or a P100, and V100 / GTX 16xx / RTX 20xx are all pre-Ampere, so on that hardware Soup ran in a dtype the card has no units for. Neither could fail on the maintainer's RTX 3050, which is Ampere; this is the same shape as the four backends the H100 session found had never executed once.

  Two corrections to the first version of this entry, both established by finally running it on a real T4 rather than reasoning about one. (1) The claim that every task died before step 0 on transformers' "Your setup doesn't support bf16/gpu" was wrong: that error was produced by a local stub forcing is_bf16_supported() to False, and transformers gates on the same permissive call described next, so on the current stack it does not raise at all. (2) More seriously, the first fix was a no-op on the hardware it was written for. torch.cuda.is_bf16_supported() defaults to including_emulation=True: when its compute-capability fast path fails it falls through to merely constructing a bf16 tensor, which software emulation satisfies — so a T4 answers True, and asking the bare question selected bf16 exactly as the hardcoded literal had. The predicate now asks is_bf16_supported(including_emulation=False) (falling back to a capability check on older torch), and get_compute_dtype — a second copy of the same question — was folded into it. What a T4 actually does with emulated bf16, as opposed to what it reports, is not yet measured. This cannot regress a working setup: where bf16 is supported the answer is unchanged, and where it is not the previous behaviour was a crash. A test SCANS every module in soup_cli/trainer/ rather than parametrising over a hand-written list — the list is what hid the twelve — and the existing unit test for this line had to be rewritten because it asserted the defect (test_auto_flag_off_preserves_legacy required bf16 on any CUDA device, and passed in CI precisely because CI has no GPU and the old code never asked the driver). Correctness of the alternative was measured before the change rather than assumed — streamed-vs-resident logits are bit-exact at 0.000000e+00 in float16 as well as bfloat16, in both quantisations, against resident references of matching numerics (an NF4 streamed run compared against a genuinely NF4 resident one, since comparing it against a bf16 model would measure the dtype rather than the streaming), and the adapter is non-zero. Not yet verified on a pre-Ampere card: that exactness was measured using fp16 on Ampere, so it establishes the plumbing, not the Turing/Pascal kernels — the free-tier notebook is the natural place to close that.

Changed

• Retracted: "layer streaming is bound by host-to-device transfer, not by the GPU." That sentence appeared in the README, in the v0.73.0 notes below, in the H100 gate record and in the preprint's abstract. It was an inference from the H100 replication — the same configuration returning the same throughput on a card with two orders of magnitude more compute — and it had never been measured. It was measured on 2026-08-11 on the original laptop and is false at the published configuration: four interleaved ablation arms in one process at a pinned clock show that removing all host-to-device traffic (6.864 GB per step) buys 1.44%, removing the NF4 dequantisation buys 9.80%, and removing both leaves 88.7% of the step; the compute stream is blocked on a copy for 8.4 ms of a 4190 ms step; and the step runs at 71.3% of that card's same-session, shape-matched GEMM ceiling. The claim is true below roughly 128 tokens per step, where the fixed transfer volume dominates — the published configuration is not there. No measured number anywhere changes, and the replication result stands in a weaker form: the constraint is common to both machines and is not the compute the datacenter card adds. The H100's own bottleneck was never instrumented and no claim is made about it. Record: benchmarks/probe-v0.73.0-what-bounds-streaming.md. Every occurrence in the gate record and in the v0.73.0 notes below is annotated in place rather than deleted, because in a folder whose whole premise is publishing the record as written, a silent deletion costs more credibility than the error does.

Validation (measured, not changed)

• Layer streaming completed a run on hardware the maintainer does not own: a free-tier Colab Tesla T4 (sm_75, Turing), via notebooks/proof-4gb.ipynb. Every streaming number this project has published came from one RTX 3050 Laptop or one borrowed 8×H100, and the pre-Ampere fix above (#385, #387) had been verified using fp16 on an Ampere card, which establishes the plumbing and not the Turing kernels. This closes that specific gap and nothing wider. NousResearch/Meta-Llama-3.1-8B-Instruct, NF4, stream_layers: true, stream_buffers: 2, batch 1, max_length: 256, LoRA r=8/α=16, fp16 (a T4 has no bf16 units): 7 steps, exit 0, adapter written with 128 tensors, 128 of them non-zero, and a measured peak of 2.91 GB against the pre-flight's predicted ~3.02 GB — an over-prediction of 3.8%, which is the direction the estimator was fitted to err in (v0.72.3 fitted it to never under-predict) and is the whole reason it is allowed to stop a run. The card has 15.6 GB, so the run was constrained artificially with torch.cuda.set_per_process_memory_fraction to 4.00 GB, and the cap was shown to bite rather than assumed: a deliberate 4.29 GiB allocation was refused. That artificial cap is also the reason no throughput figure is quoted from this run, here or in the notebook — a capped card is not a benchmark, and the panel's own forecast (31–46 tok/s from 2.20 TFLOPS measured at 1185 MHz) is a compute bound, not a measurement of what the run did. Two things the run did not establish, stated because the temptation is to let the exit code cover them. Backward/gradient exactness at 8B on Turing is not shown — a non-zero adapter proves gradients flowed, not that they were right, and the streamed-vs-resident comparison in the notebook's section 4 produced no captured output, so it is recorded as unrun rather than as a pass. And the loss moving 4.5266 → 4.0674 over 7 steps, non-monotonically (4.5266, 4.2388, 3.7485, 4.6930, 4.1940, 4.1127, 4.0674), is reported because it is what the run printed; over seven steps it is not evidence of learning. One observation worth carrying forward: the pre-flight panel reported free VRAM 15.10 GB, i.e. it read the device and not the per-process cap the run was actually held to, so the fit decision was taken against a number 3.8× larger than the budget in force. The run fit on its own merits (2.91 GB against 4.00 GB), so nothing was protected by luck — but on that hardware the pre-flight is not what would have caught an over-budget config, which is exactly the case training.stream_vram_override (#347, above) exists for. Record: benchmarks/run-t4-colab-free-tier.md.

[0.73.0] - 2026-08-09

The release that came out of three days on somebody else's hardware.

Every number this project had ever published was measured on one machine: an RTX 3050 Laptop, 4 GB, Windows. From 5–9 August it ran on a borrowed 8×H100 box (Ubuntu 24.04, a much newer torch / bitsandbytes / trl / peft stack) for the first time. That found one silent correctness defect in layer streaming, four backends that had never actually run, and a documented multi-GPU entry point that had never launched — plus the first evidence that the laptop result reproduces on hardware nothing like it. The full record, published as written including six rejected hypotheses and three false positives that controls caught, is benchmarks/gate-h100-validation.md.

This is a minor bump, not a v0.72.x patch: it adds two capabilities that did not exist, and repairs four backends.

Added

• training.seed and training.data_seed (#341). Every Soup run trained at seed 42 with no way to change it, so "run this twice with a different seed" was impossible. Both default to None rather than to 42 on purpose: an unset seed has to reproduce two different historical defaults — HF's 42 for TrainingArguments and 0 for the multipack sampler since v0.37.0 — and a plain 42 default would have silently re-ordered every existing multipack run. Scope, stated rather than left as a footgun: wired into the SFT trainer only. Other task wrappers build their own TrainingArguments, so setting it on a DPO run parses and does nothing — tracked as #353.
• Full fine-tuning as lora.r: 0 (#340). The SFT trainer's full-FT branch was dead code with no way to reach it. r: 0 was chosen on repo evidence, not taste: three consumers already treat rank 0 as "no adapter", and r: 0 previously crashed inside PEFT, so no config that worked before changes meaning. lora.r also gained a lower bound — r: -5 used to parse and die inside PEFT.
• --deepspeed zero3_offload — ZeRO-3 with CPU parameter offload. zero3 set offload_param: none and the only offload preset was stage 2, optimizer-only, so the configuration a user short of VRAM actually wants could not be named on the command line. Measured on one H100 (Llama-3.1-8B, bf16, LoRA r=8, 256 steps): 21.65 tok/s at a 38,135 MiB peak. offload_optimizer deliberately stays none — turning it on makes DeepSpeed JIT-build cpu_adam against a matching CUDA toolkit and fail without one; copy the emitted JSON and flip it if you have nvcc.
• trl support widened to >=0.14.0,<0.29 (#326), behind a capability-probe compat layer (trainer/_trl_compat.py) rather than a version table — a version table is what was wrong twice before. The trainers now ask each config class whether it accepts max_prompt_length, and resolve ORPOConfig/CPOConfig/BCOConfig through trl.experimental when trl 0.29 drops them from the public namespace. All six preference trainers construct and train to identical losses on trl 0.26.2, 0.28.0 and 1.9.2.

Fixed — backends that had never been run

• soup train --gpus N never launched at all (#77). accelerate launch takes a script path positionally, and Soup handed it sys.executable — so accelerate opened the Python binary and parsed it as source (SyntaxError: source code cannot contain null bytes). Every rank died before the trainer existed. The documented multi-GPU entry point has been dead since it shipped, invisible to single-GPU CI because that path skips the launcher wrapper entirely. Separately, --no-reexec printed a command with every user flag dropped, so following the hint literally trained without --fsdp.
• DeepSpeed could not train a LoRA model on any stage (#336). HF builds two optimizer parameter groups and with LoRA the no-decay group is empty; DeepSpeed drops it, leaving one group against two base_lrs, and torch's scheduler then hits a strict-zip length mismatch. Verified repaired on 2×H100 with real soup train: zero2 6790.2 tok/s, zero3 1025.3, zero++ 977.1, all exit 0 with a live adapter (96/96). zero++ failed earlier and independently — it set fp16 quantised weights/gradients against a bf16 model and hardcoded zero_hpz_partition_size: 8 regardless of the real world size; both are now derived, and the rewrite is printed rather than applied silently.
• use_fsdp2_compile wrote an adapter that reloads as all zeros (#335). Under torch.compile the Trainer saves through the wrapper, so every key came out as _orig_mod.base_model.model.... The tensors were genuinely trained (max|lora_B| 7.0e-3 measured) and PeftModel.from_pretrained matched none of them — emitting only a UserWarning and leaving lora_B at its zero init. Measured on 4×H100: 0 of 96 non-zero against 96/96 for the paired non-compile run, reproduced 3/3, with the run exiting 0 throughout.
• soup serve --backend sglang returned 500 on every generation (#76). sglang 0.5.16's Runtime.generate returns a JSON string; Soup subscripted it as a dict. Deterministic, not a race — the backend loaded cleanly and then failed 100% of requests. It had genuinely never been run, because SGLang does not support Windows.
• The vLLM backend ignored the model's chat template (#332). utils/vllm.py hand-rolled a "User: ...\nAssistant:" prompt while the transformers backend used apply_chat_template, so every vLLM user's model saw a format it was never trained on. On Llama-3.1-8B + LoRA, identical server and sampling params, only the prompt differing: a run-on loop burning all 200 tokens before, an 8-token answer after. Both backends now share one build_chat_prompt.
• Three vLLM serving defects (#333), each verified live: finish_reason was hardcoded "stop" even at completion_tokens == max_tokens; --dashboard silently no-opped (/metrics returned 404 with nothing printed); and --max-model-len did not exist although the engine factory already accepted it. --dashboard on a backend that cannot serve it now warns at startup naming the backend instead of doing nothing.

Fixed — training paths that silently did the wrong thing

• data.max_length was capped at 1024 on every SFT run (#78). SFTTrainer converts TrainingArguments with SFTConfig(**args.to_dict()), and max_length is an SFT-only field that TrainingArguments does not carry — so it always took SFTConfig's default. Measured before the fix: data.max_length=4096 gave 1024 tokens per sample, with no warning.
• training.use_liger: true crashed at step 0 (#78). Soup patched the model but never set TrainingArguments.use_liger_kernel, the flag TRL reads to know the fused path returns logits=None; its entropy metric then ran on None. Reproduced across the whole supported trl pin, so the feature did not run at all. Separately, Liger's architecture match was a substring of the model name, so any model loaded from a local directory trained without Liger on a flag the user had explicitly set — it now reads AutoConfig.model_type.
• FlashAttention 3 was selected from a version that can never report 3 (#334). Dao-AILab ships FA3 as flash_attn_3; flash_attn itself stays in the 2.x line, so the branch could not fire for any real install — and had it fired, it produced an attn_implementation transformers would reject. On Hopper hardware users silently got FA2 or SDPA while the docs advertised FA3. Both detectors now ask transformers. This makes detection honest; it does not make FA3 measurable here.

Fixed — layer streaming

• A silent wrong-gradient defect on large NF4 models (#331). bitsandbytes.MatMul4Bit stashes the packed weight and quant_state on ctx as plain attributes instead of through save_for_backward, so gradient checkpointing cannot discard and recompute them. The reference is captured in the forward, aliases the streaming buffer pool, and is read in the backward after that slot has been refilled. Result: a bit-exact forward, a healthy-looking loss curve, and wrong gradients on every layer but the last stream_buffers. It bites NF4 above a threshold bracketed at 163.8–171.5 MiB per layer — so 32B and 72B, never 8B or 14B, and never bf16. The repair keeps the weight out of that function entirely: dequantise inside the checkpointed region and use a native matmul, which saves the dequantised weight through the ordinary mechanism. Gated against a resident NF4 reference with a repair-disabled control arm in the same process: real 32B 256/256 gradient tensors exact against the control's 8–12/256, at +2.9% peak VRAM and −4.8% throughput; and again on real 72B — the size where the defect was worst — 320/320 against 8/320, at +2.6% and −3.7%. De-aliasing was measured and rejected first: bnb holds the reference across the whole forward-to-backward span, so any copy is O(model), which took real 32B from 4,220 to 19,720 MiB and deletes the feature's premise.
• All four preference losses died on newer trl (#328). should_enable_hf_gradient_checkpointing existed so the HF Trainer does not checkpoint an already-checkpointed streamed model twice — and only sft.py ever called it. TRL's default is False on trl 0.19.1 and True on 0.26.2, so one omission had two opposite symptoms: an explicit gradient_checkpointing: true silently dropped on the older stack, and on the newer one HF checkpointed the inner decoder layer, recomputed it after the reparametrisation context had exited, and killed dpo/orpo/simpo/kto with Tensor on device cuda:0 is not on the expected device meta!.
• Layer streaming is now refused when nn.DataParallel would engage. HF wraps the model in DataParallel whenever more than one CUDA device is visible and the run is not distributed, and DataParallel requires every parameter on device_ids[0] — streaming keeps the decoder on meta by design. It accounted for 8 of the 9 streaming-suite failures on the H100 box and is unreachable on a one-GPU machine. It refuses rather than quietly using 1 of 8 cards.
• device_map="auto" broke every distributed launch, in fifteen places. transformers refuses device_map="auto" outright under a distributed launch, so the exact accelerate launch command soup train --gpus 8 prints died on every rank. A first pass fixed six trainers; nine sites still carried it. All fifteen now go through utils/gpu.resolve_device_map, and the regression guard scans every module in soup_cli/trainer/ instead of a hand-written list — the list is what hid the nine.
• LOGITS_BYTES_PER_ELEMENT is split into two independently measured terms (#327). The 14 is 12 + 2, measured stage by stage on an H100 with zero spread across three repeats, and only the 2 differs between stacks. It is deliberately not lowered — see Known Limitations. What is new is an upward-only calibration (max(14, measured + 2)), which closes a real unguarded hole: a future stack that grew a fourth fp32 buffer would be under-budgeted by 12.5% today with nothing to catch it. Default behaviour is byte-identical and the pre-flight path takes no new CUDA. (Originally landed as an opt-in probe with no caller; #348 above wires it into the pre-flight itself, so it now runs on every streamed run rather than sitting inert.)

Fixed — eval, ship and export

• Two of soup ship's three behavioural suites measured nothing (#316). On Meta-Llama-3.1-8B-Instruct, mini_tool_call scored 0.000 and mini_format_json 0.000 on a model that does both correctly. All harness defects: the tool-call prompt never told the model tools exist, the JSON check ran json.loads over the whole output while 38 of 40 answers sit inside a ```json fence, and the generation budget was taking make_generator's default of 64, truncating 31 of 40 tool calls one closing brace short.
• The refusal detector missed the apostrophe models actually type (#316). The patterns spelled the contraction with U+0027; Llama-3.1 writes U+2019. Over the shipped 40-item mini_safety suite, 28 of 40 refusals scored as non-refusals and the suite reported 0.300 for a model whose true refusal rate is 1.000 — a 0.70 error against a 0.05 regression threshold, i.e. 14× the thing it exists to detect.
• soup train --task online_dpo passed a keyword trl removed at 0.25 (#300). The probe asked whether BasePairwiseJudge was importable and used the answer to decide whether to pass reward_model=; those two facts had decoupled, so the probe said yes for every trl in the supported range. It now asks the signature — through an MRO walk, because on 0.26.2 the top-level class is a deprecation shim whose direct signature reports no parameters at all.
• A quantised GGUF export deleted a previously exported f16 (#144). The f16 intermediate was named {model}.f16.gguf next to the output — exactly the default output name of a --quant f16 export — and unlinked when quantisation finished. So exporting q4_0 destroyed an earlier f16 export, even with an unrelated --output. Reproduced. It now lives in a private temp directory. Separately, the convert dependencies were installed only after an auto-clone, so --llama-cpp /path and an already-present checkout both died on ModuleNotFoundError: sentencepiece.

Fixed — packaging and docs

• requires-python now has an upper bound: >=3.10,<3.13 (#358). CI tests 3.10, 3.11 and 3.12 and nothing above. Without a ceiling, pip on 3.13+ resolved torch wheels nobody here has run, and the failure was not a Soup error message — it was a loader crash inside c10.dll / libc10.so before any Soup code executed, leaving the user nothing to act on. The bound is 3.13, not 3.14: 3.13 is equally untested. tests/test_requires_python_bound.py derives the bound from the CI matrix, so widening one without the other fails the suite.
• The trl bounds shipped in v0.72.4 were wrong at both ends, and were corrected before #326 widened them again. The ceiling was over-tight by five releases: the v0.72.4 table read a trl/experimental/ relocation as a field removal. The floor >=0.7.0 was impossible — setup() imports GRPOTrainer, first exported at 0.14.0. One detail in the v0.72.4 note was also imprecise: ORPOConfig/CPOConfig are not deleted at 0.29, they leave the public namespace — an ImportError rather than a rejected keyword, which is worse, not milder.
• The layer-streaming pre-flight panel was titled after a flag that does not exist (#329). It read soup train --stream-layers; there is no such option — streaming is enabled by training.stream_layers in soup.yaml. It is the first thing a streaming run prints, so it was the feature's most-read line of documentation, and it pointed at a No such option.
• Seven of the eight configs in examples/configs/ did not parse. They were written against a pre-nesting schema, so soup train --config examples/configs/sft_basic.yaml — the first command examples/README.md tells you to run — failed validation. Also fixed while there: two configs declared format: sharegpt for preference-shaped data (a silently wrong training run, not an error), target_modules listed out_proj which no Llama has, and vision_llama.yaml pointed at files that have never existed in this repo. tests/test_examples_configs.py now parses every one of them.
• soup data demo metadata described files other than the ones it ships. sharegpt_demo declared format: sharegpt for prompt/chosen/rejected rows — the value a user copies straight into data.format — and grpo_demo declared reasoning, which is not in the schema's format literal at all.
• Encoding corruption in pyproject.toml (fourteen em-dashes round-tripped through cp1251, one of them in the unit pytest marker description that pytest --markers prints), and docs/commands.md, where missing newlines collapsed five commands onto two lines and the page promised "the full command list" while omitting seven.

Changed — documentation corrected against measurement

• LISA delivers the quality half of its claim, not the memory half (#306). Measured at 3B and 8B on an H100 with LISA engagement verified three independent ways: it beats full fine-tuning at both learning rates, and it is 1.22× LoRA's VRAM at 3B and 1.51× at 8B, with the gap widening with scale — embeddings, LM head and final norm stay trainable every interval and are 70.7% of everything LISA trains at 8B. The docs now say that plainly, and say when LISA is still the right choice.
• FlashAttention and Liger were measured for the first time, at 1.015× and 1.051× / −12.9% VRAM, against documented claims of "2–4×" and "20–60% / 20–40%". Both claims are corrected in the docs.
• The GEMM-ceiling test could not pass on a datacenter GPU. Its plausibility bound was < 200 TFLOPS, written when the only hardware this project had was an RTX 3050; an H100 returns a correct 786.5 TFLOPS and the assertion fired.

Validation (measured, not changed)

• The laptop result reproduces on completely different hardware. Llama-3.1-8B NF4: 119.6 tok/s in a 3.32 GB peak on the RTX 3050 against a median 113.00 tok/s in the same 3.32 GB on an H100 — first outside evidence that the method is bound by host-to-device transfer, not by the GPU. (See Known Limitations for what the laptop figure does and does not now mean.)

  Correction, 2026-08-13, left in place rather than rewritten. The measurement stands; the explanation does not. "Bound by host-to-device transfer" was an inference, never a measurement, and a probe on 2026-08-11 refuted it at the published configuration — deleting every host-to-device byte buys 1.4% and the step runs at 71.3% of the card's same-session GEMM ceiling (benchmarks/probe-v0.73.0-what-bounds-streaming.md). What survives is the weaker claim: the constraint is common to both machines and is not the compute the H100 adds.

• Forward bit-exactness at real model sizes, not just on toys: logits torch.equal against a resident reference of matching numerics at 0.5B, 8B, 14B, 32B and 72B. Every previously published bit-exactness result was on 3-layer from-config checkpoints, because a 4 GB card cannot hold a resident 8B to compare against. Backward exactness is a separate claim measured separately — see #331 above and the per-model ledger at the top of the gate record.
• A streamed model is as good as a resident one. Paired over five disjoint training subsets and judged by Soup's own soup ship: mean difference +0.006 against an identical 0.013 within-arm spread, in bf16; +0.0053 against spreads of 0.0333 and 0.0200 in NF4. This had never been measured anywhere in the project.
• Against DeepSpeed ZeRO-3 CPU offload, same box, same data, same model: 2.93× the throughput at 9.7× less peak VRAM at matched numerics. The honest reading is narrow, and the same session establishes it: eight cards of ZeRO-3 are slower than one card training resident for a model that fits. Layer streaming is not "faster than DeepSpeed" — it is for the case where the one card you have is too small.

Known limitations

• The training.seed knob reaches the SFT trainer only (#353). Every other task builds its own TrainingArguments; setting it there parses and does nothing.
• A resident 4-bit run is still not bit-reproducible from a seed (#354), while a streamed one is. get_peft_model builds lora_A before Trainer.__init__ calls set_seed, so the adapter init escapes the seed. Diagnosed with three competing hypotheses each killed by a control; the one-line fix is verified in the gate record but not shipped here. It has a real consequence for soup ship: three runs of one unchanged resident config moved mini_common_sense by 0.375 and mini_mmlu by 0.269 against a forgetting_threshold of 0.05, so five of seven suites can cross the regression line on a re-run that changed nothing.
• The layer-streaming VRAM pre-flight still over-predicts (#327), and LOGITS_BYTES_PER_ELEMENT was deliberately left at 14 rather than lowered to the measured loss-path value. The asymmetry decides it: over-predicting refuses a config that would have worked — visible, annoying, data intact. Under-predicting on Linux is a clean OOM, but on Windows/WDDM there is no exception at all — it silently spills to host memory (measured: 9.27 GB allocated on a 4.29 GB card with nothing raised) and the claim "peak is bounded by one layer" quietly stops being true. The counterfactual settles it: 14 under-predicts 0 of 10 measured rows (worst +0.85% over), the lower value under-predicts 10 of 10 (worst −10.49%). The practical cost is that a streamed DPO run is refused from max_length: 768 on a 4 GB card.
• bitsandbytes still has the defect #331 works around. Soup no longer sends streamed NF4 weights through MatMul4Bit, but the underlying library behaviour — saving tensors outside save_for_backward where checkpointing cannot see them — is upstream and unchanged. Filed as bitsandbytes-foundation/bitsandbytes#2034.
• DeepSpeed + LoRA is repaired in sft.py only (#336). The other trainer wrappers still hit the empty-parameter-group failure under DeepSpeed, a user-supplied --deepspeed my.json is passed through unresolved, and ZeRO++ hierarchical partitioning is unit-tested but never exercised across two nodes.
• utils/sglang.py still has both of the defects the vLLM rewrite fixed — its own hand-rolled prompt and a hardcoded finish_reason. Named rather than silently skipped.
• The closed-loop reward-hacking controller's mechanism is confirmed; its efficacy is not (#286). At 7B the between-mode difference of 0.130 sits inside the within-mode spread of 0.140–0.195. Settling it needs ≥5 seeds per mode.
• Two soup ship leg-2 suites have scoring gaps that outrank capability (#346, #357). mini_tool_call ranks by brace hygiene — Llama-3.1-8B names the right tool 40/40 and scores 0.225 — and mini_mmlu loses 8 of 26 items because extract_mcq_letter does not know \boxed{C}, scoring the 8B at 0.423, below a 0.5B. Adding that one form takes it to 0.731 and the inversion disappears. A third suspected inversion (#356) was withdrawn as a measurement error of our own — it was measured at a 64-token budget where soup ship uses 256.
• The 70B FSDP2 recipe could not be smoke-tested (#41): it needs all eight cards, so it could not be parallelised with anything else in the session.
• The RAM-vs-disk streaming throughput gap remains unmeasured (#325), and layer streaming remains BETA.

A note on the preprint

DOI 10.5281/zenodo.21771064 is unaffected in its correctness claims: its configuration is 8B NF4 at 105 MiB per layer, comfortably below the 163.8–171.5 MiB boundary of #331, and it survives a 50-backward soak at worst_abs = 0.0. What this release changes is scope, not validity — exactness moves from 3-layer from-config toys to resident references at 8B / 14B / 32B / 72B. A version 2 of the record carrying the H100 validation, the resident references, the DeepSpeed comparison and the disclosed defect is in preparation.

One number should be read with a caveat: the published 119.6 tok/s laptop figure was measured before the #331 repair and has not been re-run on repaired code. The repair cost −4.8% throughput at 32B, so treat the laptop figure as a pre-repair number until someone re-measures it on an RTX 3050. An H100 cannot substitute — the whole point of the H100 result is that this method is transfer-bound, so its throughput does not carry across machines.

Correction, 2026-08-13. The conclusion holds, the reason given for it does not: the laptop is not transfer-bound (see the correction above). An H100 still cannot substitute, for a narrower reason — the repair's cost is paid in the per-layer NF4 dequantisation, measured at 9.8% of the step on the laptop, and that share belongs to that card's clock, GEMM ceiling and launch overhead.

[0.72.4] - 2026-08-03

Added — preference losses over layer streaming: DPO, ORPO, SimPO and KTO.

Layer streaming kept the frozen base in CPU RAM and fed it to the GPU one decoder layer at a time, but only for task: sft. This release opens it to the four preference losses. The whole risk was one thing: DPO needs a reference model, and a second model instance would double memory and defeat the feature entirely.

• The reference is the same streamed base with its adapters disabled — one set of weights, one stream, no second pass. Measured on an RTX 3050 4 GB with a 730 MB model: streamed DPO peaked at 0.914x the SFT peak, with a byte-identical RAM store and buffer pool. Forcing a real second instance in the same harness cost +730.44 MB against 730.44 MB of weights — exactly one copy. That control is what makes the first number mean something.
• KTO is not reference-free, contrary to how it is usually described: it selects its reference exactly the way DPO does, so it gets the same treatment and the same memory assertion. ORPO and SimPO genuinely are reference-free.
• Bit-exact against a resident run of the same loss — 0.0 difference for all four, the standard every slot in this series inherits.
• The pre-flight now knows that a paired loss is twice the rows. DPO, ORPO and SimPO concatenate chosen and rejected into one tensor, so a VRAM budget computed at one row per example would have under-predicted by half — and on Windows the consequence is not an error but a silent spill to host memory that makes the run an order of magnitude slower.
• KTO requires batch_size >= 2 (its KL term is degenerate at 1). Soup now says so when your config is read, rather than minutes later after sharding the checkpoint. KTO is streamable at all only because v0.72.3 lifted the batch-1 restriction.
• grpo and ppo remain excluded permanently, not "not yet": generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming depends on. The refusal says so and deliberately names no release.

The streaming setup now lives in one shared place instead of being copied per trainer, so the NF4 pre-flight, the RAM/disk tier choice and the VRAM fit refusal cannot drift between SFT and the preference losses.

Fixed — trl is now capped, and that is a real bug fix, not a CI tweak. Six trainers (bco, dpo, ipo, kto, orpo, simpo) pass max_prompt_length to their trl config, and trl removed it in stages. So anyone who ran pip install 'soup-cli[train]' and resolved to a recent trl had soup train --task orpo fail on import.

Correction (see [0.73.0]). This release shipped the cap as <0.25 on the strength of a staged-removal table that was itself wrong. The real stages are kto at 0.27, bco/orpo/simpo at 0.28 and dpo/ipo at 0.29. v0.73.0 first corrected the cap to <0.27 and then raised it to <0.29 behind a capability-probe compat layer, so the trainers no longer set the cap at all.

That was already true before this release and nothing caught it: the trl imports live inside setup(), which no test had ever called on those wrappers, so CI stayed green while the code only worked on older trl. This release's end-to-end preference tests are what surfaced it.

The boundary was read off the published wheels per config rather than inferred from a version number — the removal being staged is exactly why a single spot-check gives the wrong answer, and see the correction above for how that method can still land on the wrong answer. Supporting the newer API is its own piece of work; declaring a dependency the code actually works with comes first.

Honest costs: streaming makes the reference free in memory, not in time — DPO traverses the layer stack three times per step against SFT's two, measured at 1.52x the layer reads. And the VRAM pre-flight is a sound upper bound for preference losses rather than a tight estimate; see Known Limitations in the release notes.

[0.72.3] - 2026-07-28

Added — layer streaming breadth: more architectures, bigger batches, resume, and a disk tier.

Layer streaming (v0.72.0–.2) was deliberately narrow: Llama/Qwen only, batch 1, no gradient accumulation, no resume, RAM only. This release lifts all of it, and each capability was gated against a streamed-vs-resident bit-exactness reference before any of it was written.

• Six more model families. mistral, gemma, gemma2, gemma3_text, phi and phi3 join the allowlist, each verified bit-exact against the same checkpoint loaded resident, under both bf16 and NF4. Phi-3 is the notable one: it fuses Q/K/V into a single qkv_proj, so there is no q_proj to find, and it is bit-exact anyway. Multimodal gemma3 is deliberately not accepted — only gemma3_text.
• batch_size above 1, and gradient accumulation. Both previously refused.
• A pre-flight VRAM budget that accounts for batch and vocabulary. Streaming bounds the weights; activations and the logits tensor are untouched by it and both scale with batch × seq. On a 152k-vocab model at batch 8 the logits term alone measured 8.71 GB — 146× the entire layer-buffer pool. soup train now predicts peak VRAM and refuses a run that will not fit, naming the two knobs that scale it.
• A throughput forecast, quoted as a range from a GEMM ceiling measured on your own card in that session, alongside the SM clock — never a compiled-in per-card constant.
• --resume and --hf-resume work with streaming.
• A disk overflow tier. stream_source: auto (the default) uses RAM when the base fits and falls back to an NVMe disk tier when it does not, holding nothing resident. stream_source: ram refuses instead of falling back. Non-NVMe disks are still refused outright.
• soup doctor --disk reports the detected media type.

Fixed

• estimate_logits_bytes charged 6 bytes per logit element; the measured peak is 14 (transformers holds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and the fp32 gradient live at once). The old figure under-predicted that term by 2.33×.
• Adapters could not be loaded into a streamed model: load_state_dict narrows keys by child name, so a canonical checkpoint matched 0 of N tensors and PEFT reported only a warning — a resumed run reproduced the from-scratch loss curve exactly. The streaming layer now redirects canonical keys at load time, mirroring the v0.72.1 save-side fix.
• The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
• Streaming weight sources are now released when training ends or raises; the disk tier holds one open shard handle per decoder layer.
• Subprocess helpers resolve tools to absolute paths (on Windows, CreateProcess searches the current directory before PATH).
• The [mcp] extra is now capped at mcp<2. The SDK's 2.0.0 release removed mcp.shared.memory.create_connected_server_and_client_session and dropped Server.list_tools, breaking soup mcp serve's round-trip tests for anyone installing fresh. Support for the 2.x API is tracked separately.

Known limitations

• The RAM-vs-disk performance gap is unmeasured on the development hardware and no number is claimed for it. safetensors memory-maps the shards, so the OS page cache keeps them resident between steps on a machine with spare RAM, and at ~5 effective TFLOPS the NVMe read hides under compute. The disk tier's correctness is verified bit-exact against the RAM tier; its speed relative to RAM is not characterised.
• End-to-end soup train --resume could not be demonstrated on the development box: transformers refuses torch.load below torch 2.6 (CVE-2025-32434), which blocks every resume there, streaming or not. The streaming-specific half — the adapter round-trip and loss continuity — is verified on the production CUDA path.
• Loading into a streamed model works; named_parameters() and state_dict() still disagree in memory, which is the deliberate cost of a serialisation-only design.
• Layer streaming remains BETA.

[0.72.2] - 2026-07-28

Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.

Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of the whole model. It was bf16-only, which capped it at about 3B on a small card. Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B within reach.

Add one line to a streaming config:

training:
  stream_layers: true
  quantization: 4bit    # NF4
  batch_size: 1

Measured on a 4 GB RTX 3050 Laptop (Windows, batch 1, S=512, gradient checkpointing, 50 steps after 10 warm-up, PagedAdamW8bit):

Model	tok/s	Peak VRAM	RAM store	GPU util
Llama-3.1-8B-Instruct	119.6	3.32 GB	3.60 GB (page-locked)	100%
Qwen2.5-3B	264.2	1.76 GB	1.43 GB (page-locked)	100%

For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from the measured rate, not a separate measurement).

Qwen2.5-3B also got 1.85× faster than the bf16 streaming path (264.2 vs 143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the machine's page-locked memory ceiling where a 5.55 GB one did not, which restores asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.

Correctness. A streamed NF4 run is bit-exact against a resident NF4 run: the same quantised bytes through the same bitsandbytes kernels. Logit equality, non-zero gradients at layer 0, and a matching multi-step loss curve are all regression tests, not one-off measurements.

The base is quantised once, offline, and cached under ~/.soup/layer-stream/. The cache is keyed to the quantisation and the source checkpoint, so switching between none and 4bit, or retraining a base in place, re-shards instead of silently streaming the wrong bytes.

Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high. SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only — training was unaffected — but at 8B it would have read ~52 B.

Scope is unchanged and still BETA: RAM tier, task: sft, Llama/Qwen, batch_size: 1, no gradient accumulation, no --resume. Every rejected config names the release that lifts it. quantization values other than none and 4bit are refused.

Fixed — soup --help was 5x slower than it should be. Since v0.72.0 the CLI imported PyTorch on startup, taking 6.0 s where it now takes 1.15 s.

soup reward stress (v0.71.41) put utils/reward_stress on the light CLI path. That module imported utils/reward_hack_control to reuse a single string constant — and reward_hack_control resolves its TrainerCallback base class at module scope, which pulls in transformers and torch. Importing a ~4.4 s dependency for one constant made every soup invocation pay for the training stack, including commands that never touch a model.

Nothing produced wrong results; this was purely startup latency. The light core (pip install soup-cli without the [train] extra) was never broken — it fell back cleanly when torch was absent, just slowly when it was present.

Added tests/test_cli_startup_is_light.py, which asserts the invariant at runtime (import soup_cli.cli must not put torch in sys.modules) instead of inspecting source text for import torch, which is what the previous guards did and why this went unnoticed.

[0.72.1] - 2026-07-27

Fixed — layer-streaming adapters were saved in an unloadable form. If you trained with stream_layers: true on v0.72.0, the adapter that run wrote is inert: every tensor was saved under a key containing an extra .inner. segment, so soup merge, soup serve, soup chat and PeftModel.from_pretrained all loaded zero adapter tensors and silently returned the untuned base model. PEFT emitted only a UserWarning, so nothing failed and nothing looked wrong.

The training itself was correct — the streamed run's numerics are unaffected, and v0.72.0's bit-exactness results still stand. Only the saved file was affected.

If you have a v0.72.0 streamed adapter: re-save or re-run it on v0.72.1. There is no way to recover the original file's association with the base model beyond renaming its keys; re-running is the reliable path. A quick check — if adapter_model.safetensors contains keys with .inner. in them, it is affected:

python -c "from safetensors.torch import load_file; \
print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"

Streamed adapters now save byte-for-byte in the same layout as an ordinary LoRA run, and are portable to any tool that has never heard of layer streaming.

Also fixed — --hf-resume bypassed the streaming resume refusal. The guard only tested --resume, but --hf-resume reaches resume_from through a different branch. That combination previously appeared to work by accident (checkpoint and live model shared the same key shape); once adapters are saved canonically it would instead have matched nothing and continued with a freshly initialised adapter, silently. Both flags are now refused for streaming runs, naming v0.72.3.

Also in this release: every "this lands in vX.Y.Z" refusal message was corrected after the v0.72.x roadmap was renumbered (NF4 streaming is now v0.72.2; the disk tier, wider architectures, larger batches, gradient accumulation and checkpoint/resume are v0.72.3; preference losses are v0.72.4).

Known limitation: in memory the streamed model's named_parameters() still carries the wrapper segment, so loading into a streaming run (--resume) remains unsupported and is refused with a message naming v0.72.3.

[0.72.0] - 2026-07-26

Superseded by v0.72.1 — adapters saved by this version load as zero tensors. The entry below is left as published; the defect and the fix are described under [0.72.1]. Version numbers named as "upcoming" below were also renumbered there (NF4 is v0.72.2, not v0.72.1).

Layer streaming (BETA) — fine-tune models that don't fit in your card. The frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is bounded by the size of one layer instead of the whole model. Only the LoRA adapters, their gradients and optimizer state stay resident. Slower than resident training — but these models did not run on the card at all.

Measured on the development box (RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM), batch 1, gradient checkpointing on, 50 steps after 10 warm-up:

Model	S	tok/s	GPU util	Peak VRAM
Qwen2.5-0.5B	512	978.6	91.4%	1.47 GB
Qwen2.5-1.5B	512	525.0	96.8%	1.82 GB
Qwen2.5-1.5B	1024	487.6	96.7%	2.96 GB
Qwen2.5-3B	512	143.1	79.3%	2.15 GB

Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs. The honest cost: 1.43× slower than resident, measured at 0.5B — the only apples-to-apples comparison available on this box, because 1.5B and above cannot run resident here at all.

Added

• training.stream_layers: true — stream the frozen base layer-by-layer from CPU RAM. training.stream_source (auto/ram/disk) and training.stream_buffers (2–8, default 2 = double buffering) tune it.
• soup_cli/utils/layer_stream.py — tier choice, pinned-vs-pageable decision, architecture allowlist, VRAM/throughput arithmetic (no torch import).
• soup_cli/utils/layer_shard.py — rewrites an HF checkpoint into one safetensors shard per decoder layer, one tensor at a time, so sharding a model that does not fit never needs it to fit.
• soup_cli/utils/layer_stream_runtime.py — buffer pool, CPU-RAM weight source, prefetch scheduler on a dedicated CUDA stream, and the streamed layer wrapper.
• Shards are cached under ~/.soup/layer-stream/ (override with SOUP_LAYER_STREAM_CACHE_DIR) and keyed to the source checkpoint's fingerprint, so a base retrained in place re-shards instead of silently training against stale weights.
• When the base cannot be page-locked, the RAM store falls back to pageable memory and says so, including the measured cost (GPU utilisation ~97% → ~79%).

Changed

• The pre-flight hardware-fit gate is skipped for streaming runs: it models a resident run and would otherwise refuse exactly the runs streaming enables.
• Gradient checkpointing is handled per-layer by the streamer; the HF Trainer's own is left off so layers are not recomputed twice.

Known limitations

• BETA, and proof-of-mechanism at 3B. Nothing above 3B was measured. No 8B/14B claim is supported.
• Scope: RAM tier, bf16, task: sft, Llama/Qwen, batch size 1, no gradient accumulation, no --resume. Every refusal names the release that lifts it.
• 4-bit (NF4) streaming is v0.72.1 — NF4 weights carry a quantisation state and cannot be byte-copied into a plain buffer.
• The disk overflow tier, larger batches, gradient accumulation and checkpoint/resume are v0.72.2.
• The 3B number used a pageable store (this box cannot page-lock 5.55 GB), so it is a lower bound.
• expandable_segments:True is silently ignored on Windows; Soup detects this and does not claim it is active.
• Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.

[0.71.41] - 2026-07-19

soup reward stress: is your reward verifier gameable? Turn the reward-hacking detector on the verifier itself. soup reward synth (v0.71.40) proves a verifier separates your references from friendly perturbations; stress asks the adversarial question a reward-hacking model asks at train time — does the verifier pay out for degenerate junk? It feeds empty, length-padded, repetition, and sentinel-spam completions and flags any the verifier accepts. Pure, offline, exit 0 = robust / 2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a verifier for gameability.

Added

• soup reward stress <reward.py|builtin> [--references golds.jsonl] — adversarial verifier probe. Attacks (--attacks empty,length,repetition,sentinel, --sentinel) are scored against the real gold, so numeric / tool_call / json_schema verifiers get a valid target and still must reject the junk. Reports a per-attack accept-rate table + an overall gameability verdict (--max-gameable, --threshold, --output-report). Loads the target through the existing reward loader, so it probes a synthesized .py and a builtin (accuracy / format / verifiable). A gold-requiring verifier probed with no --references is a hard error, never a false "robust".

Fixed

• Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but are not wired to any command — no data is ever sent today (the previous wording implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.

[0.71.40] - 2026-07-19

soup reward synth: auto-generate a deterministic reward verifier from your data. Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a readable / committable .py reward function, and — the moat — refuses to emit one that can't tell your references from auto-generated bad answers. Nothing in TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is hand-written, hand-picked, or a trained-weights artifact.

Added

• soup reward synth <references.jsonl> -o reward.py — deterministic verifier synthesis. Four families, auto-detected (or pick with --kind): numeric (last-number / \boxed{} / #### extraction, exact or --tolerance), json_schema (induced keys + types + required), regex (positional char-classes over equal-length golds), tool_call (per-tool required/allowed argument binding). The emitted file is self-contained and rides load_reward_fn's existing .py path — no new trusted-exec surface; you read, edit, commit, and diff it.
• Mandatory calibration report — the synthesized verifier is loaded back and run against its own references (must accept ≥90%) and auto-perturbed negatives (must reject). A degenerate always-accept verifier is refused (exit 2), never silently emitted. --plan-only reports the induced spec without writing; --output-report persists the calibration JSON.
• Comma-separated reward_fn ("accuracy,format") now trains — it resolves to a reward ensemble (GRPOTrainer(reward_funcs=[...]), and unlocks the rm_ensemble reward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse time. Fixes a recipe (deepseek-v3-reasoning) that shipped exactly this and previously crashed with Unknown reward function (#311).

Changed / Fixed

• training.reward_fn gains a field validator (null-byte / blank / oversize / empty-comma-segment rejection) — the oldest arbitrary-code field was the least guarded. Comma + verifiable without a verifiable_domain now fails at parse time like the bare verifiable form.
• envs/calculator.py / envs/guess_number.py docstrings corrected: the reward is reward_fn: verifiable + verifiable_domain: math (the bare reward_fn='math' they showed was never valid).

[0.71.39] - 2026-07-19

"CI for weights, not prompts": close the evidence loop. soup ship's verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — turning the soup ci init gate from "edit two numbers in a JSON file" into a reproducible, provenance-bound check that renders on every PR.

Added

• soup ship --emit-evidence <path> — re-serialises the verdict into the --evidence INPUT schema, so a run's output is replayable as input: feeding it back through --evidence (same --forgetting-threshold) reproduces an identical verdict. Output is finally input.
• ShipConfig under eval.ship in soup.yaml + soup ship --config soup.yaml — commit the gate policy (task_eval / task_mode / general_suite / forgetting_threshold / judge_model / baseline) so the verdict is reviewable in a PR diff and reproducible. An explicit CLI flag always wins (CLI > config > default).
• soup ship --push owner/repo#N — post the verdict as a GitHub PR comment (reuses the soup adapters pr --push gh api plumbing). Best-effort: a missing token / gh failure warns but never flips the SHIP / DON'T-SHIP exit code.
• Evidence provenance + staleness gate. With --emit-evidence, --config STAMPS a provenance block (config_sha — a semantic, order-insensitive recipe hash — plus base_model and a best-effort data_sha) onto the evidence. With --evidence alone, --config GATES: it refuses (exit 3) evidence whose config_sha drifted from the committed config, so a PR that changed soup.yaml but forgot to recompute its evidence is caught. The gate policy (eval.ship) is EXCLUDED from the hash, so tuning forgetting_threshold never falsely invalidates evidence about an unchanged model.
• soup ci init --config <soup.yaml> — binds the generated gate's soup ship step to the committed config (provenance/staleness enforcement in CI).

Changed

• Exit-code note: soup ship --config usage / staleness errors are exit 3 (usage), preserving 0 = SHIP, 2 = DON'T SHIP, 1 = runtime from v0.71.38.

Security

• provenance.config_sha read from untrusted evidence is shape-validated as a hex digest before being echoed (a raw value could smuggle terminal ESC bytes past rich.markup.escape).
• provenance.data_sha hashes the training file through an O_NOFOLLOW fd with a symlink-rejecting containment check and an 8 GiB cap (was an unguarded hash_file).
• soup ci init's path validation now rejects # and the YAML 1.1 line breaks (NEL / LS / PS), closing a plain-scalar comment-truncation of the generated run: step (also hardens the pre-existing --data / --suite / --evidence args).

[0.71.38] - 2026-07-17

soup ship's regression leg now has teeth. Leg 2 (the catastrophic-forgetting / regression gate that carries the whole SHIP / DON'T-SHIP claim) was 15 hand-written trivia prompts scored by case-insensitive substring containment — it scored "B" for "Berlin", "ok" for "look", "3" for "13", and had zero items for tool-calling, safety, or JSON validity. This release makes the gate real: a fixed, extraction-based scorer + bundled, offline, zero-dep eval suites that catch a regression the old gate waved through.

Changed

• Fixed answer scorer (breaking — verdicts can change). soup ship's leg-2 MCQ / instruction / arithmetic answers are now scored by answer-extraction
  • a boundary-aware match, replacing the raw substring test. A spurious substring inside another word ("Berlin", "look", "13") no longer scores a correct answer, so an existing run's verdict may flip — intentionally, because the old gate was reporting false negatives.
• Bundled, offline general suite. The default --general-suite is now seven hand-authored suites shipped in the wheel: mini_mmlu / mini_common_sense / mini_instruction (expanded), a new mini_arithmetic, and three behavioural suites the old gate had no coverage for — mini_tool_call (function-calling), mini_format_json (JSON validity), and mini_safety (refusal-rate). Each is scored to a per-model absolute score by the pure scorers Soup already ships (eval/custom, utils/diagnose); no lm-eval, no network, no download. Every suite is large enough that a single-item flip (1/N < 0.05) trips the default threshold instead of being rounded away.
• soup ship exit codes: usage errors moved 2 → 3. Exit 2 now means only DON'T-SHIP; a typo'd flag or bad --general-suite exits 3 (mirroring soup plan / soup env check), so CI can tell a config error from a caught regression. Offline --evidence read/parse errors stay 1. Breaking for anyone parsing exit 2 as "usage error".

Fixed

• soup ship help + docstrings no longer describe --task-mode pairwise as "reserved for a later release" (it shipped in v0.71.31); the dead SUPPORTED_TASK_MODES gate is removed.
• soup diagnose's package docstring said "Six" probes (there are seven — citation) and pointed live loading at an unshipped version; it now re-exports all seven score_* probe functions so callers need not reach into submodules.

[0.71.37] - 2026-07-17

Every pip install soup-cli[extra] command now works on Windows cmd.exe, and eval-gate benchmark tasks run instead of always failing.

Fixed

• Install hints are now quoted so they work in every shell. Soup printed pip install 'soup-cli[ui]' — bash / zsh / PowerShell syntax. cmd.exe has no single-quote quoting, so it hands the quotes to pip verbatim and pip refuses:

  ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier
  

  Every hint, README command, and docs example now uses pip install "soup-cli[extra]", which works in cmd, PowerShell, bash, and zsh alike — the same spelling the repo already used for pip install -e ".[dev]". Measured on Windows: single quotes fail only on cmd.exe; double quotes pass everywhere; dropping the quotes passes on Windows but breaks zsh, which globs the bracket.

  Nothing in Soup can rescue the command after it is typed — pip and the shell own it, and Soup is not installed yet when the README command runs — so the fix is the spelling we print. A regression test now scans the package and every docs code block for the single-quoted form.

  If you followed an older tutorial and hit Invalid requirement, swap the ' for "; nothing is wrong with the package.

• Eval-gate type: benchmark tasks now actually run. eval/gate.py probed for a forgetting.run_mini_benchmark helper that never existed, so every type: benchmark task in a gate suite failed 100% of the time — while advising an [eval] extras install that could not fix it. The gate now calls ForgettingDetector directly (the same way soup ship already did), and an unknown benchmark name fails with the list of valid names. Thanks @Sanjays2402! (#315, closes #310)

[0.71.36] - 2026-07-16

Data Moat II — a semantic layer over your training data, plus two tools for what a fine-tune forgets and leaks.

Added

• soup data dedup --semantic — near-duplicate removal over embedding cosine instead of MinHash shingling. Catches reworded duplicates that MinHash misses (measured: 0.88–0.91 cosine on rewordings MinHash scored as distinct) while correctly keeping distinct-but-similar instructions. Zero new dependencies: uses transformers from the [train] extra. Read the known-limitation below before lowering --threshold.
• soup data topics <data> — cluster a dataset and label each cluster with c-TF-IDF terms, plus a coverage table (82% code · 6% math) and a warning for thin topics. Labels are emergent term clusters, not a fixed taxonomy.
• soup data canary insert|check — Secret-Sharer memorization probe. Insert K high-entropy secrets, then check any model/adapter: each secret's loss is ranked against never-inserted controls drawn from the same space. Exit 2 on MAJOR so CI can gate. Measured on SmolLM2-135M: a memorized set lands at percentile 0.0 (loss 1.7–2.5) against a clean model's 4.1–6.2.
• soup train --replay old.jsonl --replay-ratio 0.1 — continual-learning rehearsal. Interleaves a seeded sample of an old dataset into training so a new task does not erase the old one. r is the fraction of the final mixed set (n_replay = round(r/(1-r) · n_new)), rows are interleaved rather than appended, and an undersized pool reports the shortfall instead of repeating rows. Mixed into train only — validation stays pure new-task.

Fixed

• The hardware-fit gate refused to train any local checkpoint. A merged model (soup merge -o ./mymodel) has no size marker in its name, so the size guesser returned its 7B default, predicted ~16 GB of VRAM and refused. Local checkpoints are now measured from their safetensors header (0.135B actual vs 7.0B guessed) — this had blocked soup merge → train-from-merged entirely. Third instance of this class after the v0.71.32 (Whisper) and v0.71.33 (M suffix) fixes.
• pip install 'soup-cli[extra]' hints printed without the extra. Rich ate the bracket, so every "install the missing dependency" message across 17 sites told users to run pip install 'soup-cli' — which succeeds and still leaves the feature broken. Affected [eval], [data], [serve], [ui], [tui], [compile], [mcp], [carbon] and others, including Typer help text.
• Replay rows bypassed the image/audio path-traversal validation that the primary dataset receives.

Known limitations

• Semantic dedup is not a paraphrase detector. Measured with all-MiniLM-L6-v2, paraphrase cosines (0.49–0.76) overlap with genuinely-distinct rows (0.54–0.76): "Add two numbers" vs "Multiply two numbers" scores 0.759, higher than the true paraphrase "reverse a string" / "invert the order of characters" at 0.491. No threshold separates them, so lowering --threshold to chase paraphrases deletes real training rows. The default (0.8) is deliberately conservative.
• Replay is validated at proof-of-mechanism scale. On SmolLM2-135M + LoRA, replay retained the old task 7% better than a no-replay control — the correct direction — but forgetting without it was only +4%, i.e. mild. The effect size at full fine-tuning or 7B+ is unproven on a 4 GB box.
• Canary exposure is the sampled-control approximation, not full-space rank enumeration. "No exposure" is not proof of no memorization.
• data topics / dedup --semantic require [train] (torch) and download an embedding model. Plain MinHash dedup stays on the light core.
• Replay v1 is sft/pretrain only and is incompatible with packing/multipack.

[0.71.35] - 2026-07-15

Added

• Compliance templates — soup init --template hipaa|soc2|eu-ai-act|sr-11-7. Four regulation-shaped starting configs. Soup's compliance controls are CLI flags/commands rather than config keys, so each template is a valid training config plus header comments naming the exact commands for that regime (PHI scrubbing + air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI + energy tracking for the EU AI Act, repro-receipt + diagnose/ship for SR 11-7). Templates default to a license-clean Apache-2.0 base.
• soup card <registry-id> -o MODELCARD.md — model-card autogen. Turns a Local Model Registry entry into a publishable, provenance-carrying HF model card: base model, training config, eval scorecard, config/data hashes, lineage (ancestors) and a table of every registered artifact. Adapter vs full-model is inferred from registered artifacts, falling back to the training config (LoRA rank, with Spectrum/LISA full-FT correctly treated as dense), so the card sets the right library_name and never misreports the model type.
• soup push --card <registry-id> — render that registry-driven card and upload it as README.md, overriding the auto-generated one. A bad ref fails fast before any network call; HF hub only.
• soup ci init — fine-tuning CI. Writes .github/workflows/soup-gate.yml, a PR gate chaining soup data validate → soup expect → soup ship --evidence (exit 2 blocks the merge). Every interpolated path is validated to stay under the repo root and shell-quoted; the branch and Python version are regex-gated; the write is atomic, symlink-rejecting, and refuses to clobber an existing workflow without --force.
• Compliance quickstart — a new docs/compliance.md walkthrough: template → PII scrub → train with receipt/Annex XI/energy → registry → BOM + attestation → scan/sign/verify → air-gap → model card → CI gate.

Fixed

• GGUF export now actually works on Windows (validated end-to-end against a locally-built llama.cpp: SmolLM2-135M → q4_0 / q4_k_m / q8_0 / f16 → soup deploy ollama → live inference). Four real bugs, each of which independently broke the path:
  • soup export --format gguf cloned llama.cpp into your current directory. SOUP_DIR is the bare name .soup, but the lookup used it relatively rather than anchoring to ~ like the rest of the codebase — so the canonical ~/.soup/llama.cpp was never found and a fresh ~200 MB checkout was dropped into whatever directory you ran from.
  • The first GGUF export downgraded your PyTorch and broke CUDA. The auto-clone ran pip install -r <llama.cpp>/requirements.txt into your interpreter, and llama.cpp pins torch~=2.2.1 against the CPU wheel index (observed: torch 2.5.1+cu → 2.2.2+cpu, transformers 4.57 → 4.46). Soup now installs only the convert script's extra dependencies, unpinned, and never touches torch.
  • A correctly-built llama.cpp was not found on Windows. MSVC (like Xcode) is a multi-config generator and emits build/bin/Release/llama-quantize.exe; only the flat single-config layout was searched.
  • soup deploy ollama failed on a relative GGUF path with "pull model manifest: file does not exist" — Ollama resolves FROM against the Modelfile's directory, and Soup writes the Modelfile to a temp dir. The Modelfile now emits an absolute path.
• Model-card injection hardening (affects the pre-existing soup push card too). The ## Training section interpolated base / task / scheduler / recipe unescaped. Since SoupConfig.base and scheduler have no charset validator, a crafted-but-valid config could smuggle raw HTML — or a backtick breaking out of the surrounding code span — into a card published to the Hub. All values now go through the markdown escaper, which additionally neutralises backticks and strips C0/ESC control bytes.

[0.71.34] - 2026-07-15

Added

• soup adapters arithmetic — task-vector algebra over LoRA adapters (add / scale / negate). Apply task arithmetic (arXiv:2212.04089) to LoRA deltas via an expression such as "coder + 0.5*math - toxic", mapping names to adapter dirs with repeatable --adapter name=path. Produces one merged adapter you can serve or merge.
  • Signed, un-normalized element-wise combine over same-rank adapters; the effective delta ΔW = B @ A scales linearly with each coefficient (negation flips the delta, 0.5· halves it) via a √|c| factor split — not the c² a naive sum gives. Mixed-rank inputs are refused with a clear "harmonize rank" message.
  • Reuses the backdoor-scan gate (refuses a FAIL-scanned input unless --allow-unscanned) and a same-base-model check (--allow-cross-base to override). Hand-written expression parser (no eval), cwd-contained/symlink-rejecting paths, exit 0 = ok / 1 = refusal.
• LISA — Layerwise Importance Sampled AdamW (arXiv:2403.17919). Full-fine-tuning quality at LoRA-like memory: every N steps LISA freezes all decoder layers except a small random set (embeddings + head always trainable). Enable with training.lisa_enabled: true (+ lisa_num_layers, lisa_interval_steps) on a task: sft, transformers, text, quantization: none run; mutually exclusive with LoRA features and the other freeze mechanisms. Live on a 4 GB GPU for small models.

[0.71.33] - 2026-07-13

Added

• soup draft — train and, above all, MEASURE a speculative-decoding draft model.

  • soup draft measure --target <m> --draft <d> --prompts p.jsonl reports a draft's acceptance rate (the fraction of the target's own greedy tokens the draft would have proposed correctly) plus real plain-vs-assisted throughput. This is the honest gate: it tells you whether speculative decoding is worth enabling before you ship it. Exit 0 / 2 (below --min-acceptance, for CI) / 1.
  • soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/ distils your target into the tiny base (logit KD via the existing task: distill trainer) and emits a dense draft model, ready to load as an assistant_model.
  • soup draft list, plus a local draft registry (~/.soup/drafts.json) that soup serve --auto-spec consults before the built-in pairing table — so a draft you trained yourself is picked up automatically.
  • Draft and target must share a tokenizer; a mismatched pair is refused up front (speculative decoding proposes draft token ids into the target's vocabulary, so a mismatch silently produces garbage rather than failing).

  Read the measured results before you use this — see Known limitations below. On the validated pair, distillation did not improve acceptance, and speculative decoding was a net slowdown. soup draft measure is what tells you that.

Known limitations

• Distilling a draft did not improve its acceptance rate on the validated pair. Measured on SmolLM2-360M-Instruct (target) with a SmolLM2-135M-Instruct draft: the stock draft already scored 69.3%, and distilling it moved that to 69.7% after 2 epochs and back to 69.3% after 10 epochs — i.e. no gain beyond noise. A small same-family draft is already near its capacity ceiling for agreeing with the target, and logit KD cannot buy capacity it does not have. Whether distillation materially raises acceptance for a genuinely diverged fine-tune, or a larger target/draft pair, is unproven on a 4 GB box — tracked as a scale issue.
• Speculative decoding was a net slowdown on that pair (measured 0.55–0.64×): the draft's forward pass costs more than the tokens it saves at this size. soup draft measure reports this truthfully rather than assuming a speedup — which is precisely the point of shipping the measurement.
• Acceptance is teacher-forced greedy agreement, the metric the Medusa/EAGLE papers report. It is exact and deterministic, and it is the right number for comparing drafts — but it is not the accepted-token count of a sampling run, which also depends on the rejection-resample cascade.
• Same-tokenizer only. Cross-tokenizer drafts (ULD-aligned + universal assisted decoding) are deferred.

Changed

• PRM-guided GRPO scores completions in a single batched forward. PRMScorer.__call__ now right-pads all completions into one [B, T] tensor (+ attention mask) and runs one output_hidden_states pass instead of one forward per completion, cutting per-step reward latency on non-tiny models. Numerically identical to the per-completion path (parity + mixed-length tests). Closes #298 (#301 by @Ekaanksh-dev).

Fixed

• The hardware-fit gate no longer refuses to train small models. model_size_from_name did not understand an M (millions) suffix, so SmolLM2-135M fell through to the 7B default, was predicted to need ~14 GB of weights, and was blocked. Every draft-sized model hit this. 1.7B was also being read as 7B (the 7b marker matched inside 1.7b), while Qwen2.5-7B-Instruct-1M correctly stays 7B (1M is the context, not the parameter count).
• soup serve --backend vllm no longer force-enables trust_remote_code. The vLLM path now goes through the same --trust-remote-code default-deny gate (and warning panel) as the transformers backend, so serving an untrusted repo never executes its code silently.
• Multi-adapter serving (soup serve --adapters name=path) now actually switches adapters. The named adapters are loaded into the model and selected per request (via POST /v1/adapters/activate/{name} or the request adapter field); previously every request silently ran the startup model. The base model is served when no adapter is selected.
• Vision datasets reject out-of-directory image paths. llava / sharegpt4v rows are containment-checked against image_dir (mirroring the audio loader), so a crafted {"image": "/etc/passwd"} row can no longer read arbitrary local files.
• soup train --dry-run --gpus N no longer launches a real multi-GPU run. The accelerate re-exec is skipped under --dry-run. The re-exec also now forwards --minillm-on-policy, --capture-activations, and --capture-prompts (previously dropped on multi-GPU runs).
• MLX SFT now builds a real optimizer (AdamW from the configured LR) instead of passing optimizer=None, which left the model untrained.
• soup data inspect / preview / search escape dataset- and Hub-derived text so a stray [/] no longer crashes the command and a crafted [link=…] tag can't render a phishing hyperlink. soup runs list/show escape config-derived fields too.
• soup infer --task asr hardening — an oversized reference no longer crashes the whole batch after transcription (that row's metric is skipped); an all-skipped run exits non-zero instead of reporting success; --asr-task is validated upfront; and dataset-derived filenames are control-stripped before printing. ASR training now caps transcript labels to Whisper's decoder limit, warns on >30 s audio, and picks fp16 on pre-Ampere GPUs instead of hardcoding bf16.
• Knowledge-distillation KD term aligns with the CE term. The token-level divergence is now computed over causal-shifted positions, so the distillation signal covers exactly the trained tokens (previously off by one).
• Miscellaneous robustness — load_config_from_string raises ValueError (not TypeError) on a non-mapping YAML document; the Web UI Bearer-token check is constant-time; and soup doctor --vscode / the LR-finder report use the centralised atomic, symlink-rejecting writer.

[0.71.32] - 2026-07-07

Added

• ASR fine-tuning (task='asr', Whisper) — fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) / base (74M) train on a 4 GB GPU.
  • New AsrTrainerWrapper (HF Seq2SeqTrainer + WhisperProcessor); data rows are {"audio": <path>, "text": <transcript>} under the new data.format='asr'. Audio decodes via the hardened load_audio_mono (16 kHz mono, soundfile pre-probe + O_NOFOLLOW + symlink/size guards); transcripts become decoder labels (pad → −100, decoder-start token stripped).
  • Optional LoRA on q/v attention projections via training.asr_lora: true (default full fine-tune); training.asr_language / training.asr_task (transcribe|translate) set the decoder prefix and are persisted to an asr_generation.json sidecar so inference restores them.
  • soup infer --task asr — transcribe an {"audio": path[, "text": ref]} JSONL; reports per-row and corpus WER/CER when references are present. Loads a full model or a PEFT/LoRA adapter dir. Flags: --asr-language, --asr-task, --audio-dir (audio paths are cwd/dir-contained; UNC/traversal rejected).
  • New pure-python utils/asr_metrics.py — WER / CER / word_accuracy (= 1 − WER, for a higher-is-better ship metric leg) / corpus_wer, with a light Whisper-style text normalizer (no new dependency).
  • Recipes: whisper-tiny-asr, whisper-base-asr (live-trainable), whisper-large-v3-asr (parse-only, needs a larger GPU), plus smolvlm-256m-sft (vision). Catalog 138 → 142.

Fixed

• model_size_from_name now knows Whisper checkpoint sizes (tiny…large), so the hardware-fit gate no longer mistakes a 39M whisper-tiny for the 7B default and blocks ASR training on consumer GPUs.

[0.71.31] - 2026-07-06

Added

• Judge-in-the-loop suite — put an LLM judge in the loop across the workflow:
  • task='online_dpo' — Online DPO training (wraps TRL OnlineDPOTrainer): the model generates two completions per prompt on-policy each step and a judge (a pairwise LLM judge over the existing ollama/openai-compatible backend) OR a reward_model picks the winner. Config: training.online_dpo_judge: "ollama://model" (or set reward_model — exactly one), online_dpo_loss_type: sigmoid|ipo, online_dpo_max_new_tokens; beta reuses dpo_beta. Transformers + text only. Recipe: online-dpo-smollm2-135m. Adapts to the installed TRL: on trl 0.19.x the judge is a swap-debiased pairwise comparison; on trl 1.x (which removed pairwise judges) the same JudgeEvaluator is used as a pointwise reward function — a documented per-version behaviour difference.
  • soup data best-of-n — Best-of-N rejection sampling (BOND-lite): sample N completions from --base locally, a --judge scores each pointwise, and the winner is written as an SFT chat row (with provenance). --emit-pairs also writes winner-vs-loser DPO pairs.
  • soup data evolve — Evol-Instruct instruction evolution (WizardLM depth / breadth) over an ollama/vllm provider, completing the synthetic-data suite (Magpie / Forge / Persona / evolve).
  • soup ship --task-mode pairwise — a true pairwise judge win-rate as the ship leg-1 task-win (base = 0.5 coin-flip, tuned = its win-rate; swap-debiased), fusing with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.

Security

• soup data best-of-n / evolve write outputs via atomic mkstemp + os.replace (re-validated cwd containment), closing the TOCTOU symlink-swap window between the containment check and the write. All judge/provider URLs are SSRF-validated; model loads probe trust_remote_code.

[0.71.30] - 2026-07-05

Added

• PRM-guided GRPO — use a trained Process Reward Model as the per-step reward inside GRPO (the o1-era process-supervision signal). Set training.prm_reward: <PRM dir|id> (a model produced by soup train task=prm) and training.prm_aggregate: min|prod|last; the PRM splits each generated completion into reasoning steps, scores every step with its reward head, and folds the per-step scores into one scalar reward that GRPO optimises. The PRM reward replaces reward_fn and rides the existing reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller observes it. Cross-validators gate task='grpo' + backend='transformers' + modality='text'. Default aggregation is min (weakest-link); prod assumes calibrated [0,1] step scores.
• Bundled rollout environments — three pure-Python toy environments (soup_cli.envs.calculator / retrieval_qa / guess_number) exposing a rollout(prompts) entry point so the live openenv GRPO rollout path runs out-of-the-box: training.rollout_backend=openenv + training.rollout_func=soup_cli.envs.calculator:rollout. Three ready-made recipes added (grpo-env-calculator / grpo-env-retrieval-qa / grpo-env-guess-number); catalog 134 → 137.

Fixed

• soup train task=prm producer conformance (surfaced by the v0.71.30 live smoke): the PRM trainer now casts its reward head to the base-model dtype (bf16 CUDA runs previously crashed on the first compute_loss), saves the tokenizer alongside the model (so a PRM checkpoint is loadable standalone), and returns the standard trainer-result shape (previously the CLI crashed with a KeyError: 'initial_loss' right after saving).

Notes

• Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny synthetic PRM and synthetic reward — not a production reward-model claim (scale ask tracked in #286). The bundled environments are deterministic single-shot seeders, not interactive multi-turn model-in-the-loop episodes (the live openenv contract passes only prompts). Step split is a newline heuristic; PRM completions are scored one forward pass each.

[0.71.29] - 2026-07-05

Added

• soup shrink — depth-prune a model + optional distill-heal ("The Unreasonable Ineffectiveness of the Deeper Layers", arXiv:2403.17887). Ranks decoder layers by the angular distance of the residual stream across a contiguous block over a calibration set, drops the least-important block (first and last layer always protected), optionally heals by distilling the original model into the pruned student, and emits a single dense smaller model plus a one-screen SHIP / DON'T SHIP perplexity verdict. soup shrink --model <id|path> [--drop-ratio 0.25 | --drop-layers N] --calib <calib.jsonl> [--heal <heal.jsonl> --heal-steps N] [--tolerance 0.10] [-o <dir>] [--device cpu] [--attach-to-registry <id>] [--plan-only]. Exit codes: 0 = SHIP, 2 = DON'T SHIP, 1 = error. The heal runs as an isolated soup train subprocess (LoRA-student logit distillation) and the adapter is fused back so the shipped artifact stays a single dense model. Arch allowlist v1: Llama / Qwen / SmolLM. Validated live on SmolLM2-135M (drop 25 %: 30 -> 22 layers, ppl x2.98 unhealed; drop 4 + heal: ppl x1.35, recovered).

Security

• soup shrink contains --calib / --heal / --output-dir (and every derived write path: <out>/model, <out>/heal_adapter, the fuse staging dir) under cwd with os.path.realpath + commonpath + O_NOFOLLOW + symlink rejection, re-validating derived paths right before each write (TOCTOU). The heal subprocess uses an argv list (no shell) with a timeout; its config is schema-validated before spawn; subprocess output is C0/ESC-stripped before it reaches the terminal. --model defaults trust_remote_code=False with a probe + warn.

[0.71.28] - 2026-07-04

Added

• MCP server (soup mcp serve) — drive Soup from any Model Context Protocol client (Claude Code / Cursor / Cline / Continue) over stdio. No fine-tuning CLI ships an MCP server. Exposes 14 read-only tools as JSON — advise, data_inspect / data_validate / data_score / data_doctor, recipes_search / recipes_show, runs_list / runs_show, registry_list / registry_show, profile, diagnose_evidence, ship_evidence — plus two plan-only mutating tools (train_start, export) gated behind --allow-mutating (they render the exact command that would run; they never execute). The official mcp SDK is behind a new [mcp] extra (pip install 'soup-cli[mcp]'), lazy-imported so the core CLI stays light. Security: stdio-only (no network listener); every path argument re-enters cwd-containment + symlink rejection; output is control-char sanitized; errors are path-free; string / size / int bounds enforced.

Fixed

• The DPO / IPO / KTO / BCO trainers now apply configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) via the shared apply_vocab_expansion() helper during setup, consistent with the SFT path — they previously ignored it. Closes #292 (#293 by @CODING-DARSH).
• The ORPO / SimPO / GRPO trainers now also apply configured vocabulary expansion via the shared apply_vocab_expansion() helper — completing consistent vocab-expansion behavior across every SFT/preference/RL trainer. Closes #294 (#295 by @CODING-DARSH).

[0.71.27] - 2026-07-03

Added

• Fine-tune Doctor — kill the top silent fine-tune failures before a single training step; no competitor (Unsloth/Axolotl/LlamaFactory) ships any of these three:
  • soup data doctor <data> --model <id|path> — chat-template compatibility report over 8 checks: chat_template present, template_renders cleanly, has {% generation %} markers, eos_in_labels (the #1 "model never stops generating" bug — every assistant turn's trained span must actually contain an EOS/EOT token; checks every turn, not just the last), bos_duplication (template + tokenizer both prepending BOS), system_role support (Mistral-style templates reject a leading system turn), unknown_roles, and truncation_risk (p95 rendered length vs max_length). Same OK / MINOR / MAJOR taxonomy as soup diagnose; exit 0 = OK/MINOR, exit 2 = MAJOR. --train-on-responses-only / --train-on-messages-with-train-field select the same masking strategy soup train would use, so the report and --show-mask never disagree about what's actually trained.
  • soup data doctor ... --show-mask N — render N sample rows with per-token trained/masked colouring through the REAL collator path (answer-only / per-message-train-field / RAFT span-mask) — not a reimplementation — so an assistant-mask bug is visible instantly.
  • soup data lint <data> — preference-data linter for dpo/orpo/simpo/ipo/bco/kto: length_bias (chosen systematically longer than rejected — the #1 silent DPO degradation, reported as a Cohen's d effect size), label_imbalance (KTO desirable:undesirable ratio), near_duplicates (MinHash/LSH, reuses the soup data dedup kernel), identical_pairs (chosen == rejected — zero preference signal), and prompt_leak (the prompt echoed verbatim inside the completion — a common synthetic-data pipeline bug). Optional --model for exact token-length bias (default: word count).
  • Validated live against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer on Windows + RTX 3050 — this smoke pass found and fixed two genuine bugs beyond what synthetic fixtures alone caught: an EOS check that required the EOS token to be the literal last trained token (real templates often have a trailing formatting token after the closing tag that stays inside the trained span), and two call sites that only caught (ValueError, TypeError) around a tokenizer's apply_chat_template when a real Jinja raise_exception() (Mistral-style no-system-role guard) raises jinja2.exceptions.TemplateError.

Fixed

• Harden commands/diagnose.py's --evidence loader against a TOCTOU symlink swap: opens with O_NOFOLLOW and size-checks the open fd via os.fstat instead of os.path.getsize on the path before the open — backports the hardened loader shipped for soup ship in v0.71.25 (closes v0.71.25 known-limitation (4)).
• Harden judge-model URL validation against a hostname prefix bypass (http://localhost.attacker.com) — GateTask._valid_judge_url / _parse_judge_url now use urllib.parse.urlparse + hostname checks instead of startswith. Closes #283 (#288 by @CODING-DARSH).
• SFTTrainerWrapper now applies configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) and resizes the model embeddings during initialization — previously these fields were accepted by the schema but silently ignored. Closes #289 (#287 by @CODING-DARSH).
• Vision and audio SFT paths now apply that same configured vocabulary expansion (data.add_new_tokens / data.new_special_tokens) via the shared apply_vocab_expansion() helper, consistent with the text SFT path — they previously ignored it. Closes #290 (#291 by @CODING-DARSH).

Security

• soup data doctor strips C0 control characters (keeping tab/newline/CR) from dataset-derived content before it reaches the terminal — Rich's markup.escape() only neutralises [...] tag syntax, not raw escape sequences, so an untrusted training row (e.g. an unknown role field, or --show-mask's decoded token text on a byte-level BPE tokenizer) could otherwise carry a literal ESC byte through to the terminal (title-bar / OSC-8 link spoofing, or obscuring a MAJOR verdict via cursor tricks). --output JSON is unaffected (json.dumps already escapes control characters).

[0.71.26] - 2026-07-01

Added

• Closed-loop reward-hacking auto-mitigation. The trainer now detects reward hacking mid-run and self-corrects — instead of only halting. Set training.reward_hack_mitigation (or soup train --reward-hack-mitigation) to one of four modes on a GRPO/PPO run (requires reward_hack_detector):
  • log_only — instrument only: append a per-step mitigation_log.jsonl (drop_pct, verdict, reward mean/std, completion-length trend, repetition) and never touch training.
  • kl_control — a reversible bang-bang + hysteresis controller: when the hacking signal trips, raise the KL coefficient β (geometric, clamped to [floor, ceil], never crossing 0); relax it when the signal recovers. Dwell + release-patience prevent flapping; a multi-signal vote combines the detector drop with a length-trend and repetition signal.
  • pid_lagrangian — a PID-Lagrangian controller (Stooke et al.) that holds the hacking signal at a target, plus an escalation ladder: raise β → roll back to the last-good RL checkpoint → early-stop.
  • Anti-gaming hardening: per-signal EMA/median smoothing, conservative-on-disagreement voting, a reward-distribution-drift guard, and optional bounded reward shaping on the gamed proxy (length / repetition / sentinel). A plain-English give-up explanation is logged on early-stop.
  • Proof-of-mechanism only (see Known Limitations): validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 — all four stages pass live, including a real mid-run rollback. PPO ships BETA (mechanism unit-tested; the on-GPU proof is GRPO-only).
• Ready-made qwen2.5-coder-7b-sft recipe for Qwen/Qwen2.5-Coder-7B-Instruct (catalog 133 → 134) (#285 by @Deadpool2000).

Security

• RLCheckpointCallback.restore_checkpoint / save_checkpoint refuse a symlinked optimizer.pt — torch.load(weights_only=False) on an attacker-placed symlink in a shared checkpoint dir was an RCE vector.
• Bool-before-int/float guards on every new reward_hack_* numeric field; reward_hack_signals bounded (max_length=4); the mitigation log writer is cwd-contained with symlink-reject-on-rotate and secret redaction.

[0.71.25] - 2026-06-27

Added

• soup ship — the SHIP / DON'T-SHIP verdict. After fine-tuning, answer one question: did the model get better, or did I break it? soup ship fuses two checks into a single binary decision — leg 1: the task metric strictly improved (base → tuned); AND leg 2: no general benchmark regressed past a forgetting threshold (default 0.05 absolute points). It SHIPs only when both hold — otherwise DON'T SHIP, even if the task metric looks great. The output is a one-screen verdict + the reason, with CI-gateable exit codes (0 = SHIP, 2 = DON'T SHIP, 1 = runtime error).
  • Leg-1 modes: --task-mode metric (eval accuracy) or judge_score (LLM-as-a-judge); pairwise win-rate is planned for a later release.
  • Leg-2 suite: built-in mini benchmarks by default (offline, CPU), or --general-suite <names> to route lm-eval benchmarks; --baseline registry://… | file.json supplies recorded base scores.
  • --evidence ev.json decides offline from pre-computed scores (no model load); --output verdict.json persists the machine-readable verdict.
• Friendlier error messages: the CUDA-OOM hint now also suggests gradient_checkpointing and 4bit quantization, plus new mappings for Hugging Face gated repos (huggingface-cli login / HF_TOKEN) and trust_remote_code errors. Closes #272 (#282 by @Akshaya-reddy18).

Security

• soup ship input hardening: --evidence is opened with O_NOFOLLOW + an fstat size cap (16 MiB) under cwd containment; --task-eval is cwd-contained and symlink-rejected; --judge-model is validated by scheme/host via urlparse (blocks the http://localhost.attacker.com prefix bypass); lm-eval model ids reject ,/= injection; --general-suite is bounded (≤ 50 names, ≤ 256 chars each).

[0.71.24] - 2026-06-21

Added

• 2026 model-family recipe expansion (catalog 116 → 133). 17 new ready-made SFT recipes for the open-weight models released Feb–Jun 2026, each with its Hugging Face repo-ID verified to resolve:
  • Qwen 3.5 (Apache-2.0): qwen3.5-0.8b-sft, qwen3.5-2b-sft, qwen3.5-4b-sft, qwen3.5-9b-sft, qwen3.5-27b-sft, and the qwen3.5-35b-a3b-sft / qwen3.5-122b-a10b-sft / qwen3.5-397b-a17b-sft MoE sizes.
  • Qwen 3.6 (Apache-2.0): qwen3.6-27b-sft, qwen3.6-35b-a3b-sft.
  • DeepSeek-V4 (MIT): deepseek-v4-flash-sft, deepseek-v4-pro-sft.
  • GLM (MIT): glm-5.1-sft.
  • Kimi (Modified MIT): kimi-k2.5-sft, kimi-k2.6-sft.
  • MiniMax (MiniMax Community License — commercial use needs a separate agreement): minimax-m3-sft.
  • Mistral Large 3 (Apache-2.0, 675B/41B-active multimodal MoE): mistral-large-3-sft.
• Unit-test coverage for the warmup.py auto-warmup-steps helper (#274 by @shatakshi-1404).

Fixed

• Stale recipe repo-ID: glm-5-sft now points at zai-org/GLM-5 (the org migrated from THUDM).

[0.71.23] - 2026-06-12

Added

• Native Spectrum targeted training (closes #266). A new soup spectrum scan reads a model's .safetensors shards one tensor at a time (no model load — peak RAM is the largest single weight matrix), computes a singular-value SNR per weight matrix with a Marchenko-Pastur noise threshold (arXiv:2406.06623), ranks layers within each module-type group and prints the top --top-percent as a ready-to-paste training.unfrozen_parameters YAML block. This lets you scan even a very large model's layer SNR on a CPU box and then full-fine-tune only the high-signal layers.
  • soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn|all] [--output patch.yaml] — SNR table + the YAML patch; results cache at ~/.soup/spectrum/<slug>.json (override via SOUP_SPECTRUM_CACHE_DIR).
  • New schema field training.unfrozen_parameters: list[str] — regex patterns of parameter names to keep trainable; the SFT trainer freezes every parameter then unfreezes the matched set (full fine-tuning, LoRA off). Mutually exclusive with LoRA features / freeze_layers / freeze_ratio / train_router_only / expand_layers; requires task=sft, backend=transformers, modality=text, and quantization=none.
  • The SNR kernel is pure-numpy and transpose-invariant (singular values are identical for W and W.T); GPT-2 Conv1D naming (c_attn/c_fc/c_proj) is recognised alongside Llama-style names.
  • The existing spectrum trainer-plugin wrapper is untouched (back-compat). LISA (per-step layer sampling) is tracked separately in #267.

Security

• soup spectrum scan validates unfrozen_parameters patterns at parse time: rejects nested-unbounded-quantifier regexes (ReDoS), null bytes, empties, and caps count (50k) and length (512). Hub downloads route through the SSRF-hardened, namespace-pinned hubs.snapshot_download; symlinked shards and matrices above a 2^31-element SVD cap are skipped; --output stays under cwd.

[0.71.22] - 2026-06-10

Added

• Perf & measure polish — a 4-issue patch tightening four live paths from the recent BETA lifts. Pure code, validated on Windows + RTX 3050.
  • MiniLLM on-policy KV-cache (closes #263). The on-policy distillation rollout (soup train with training.minillm_on_policy: true) now threads past_key_values so each step forwards only the new token instead of re-feeding the whole prefix — resolving the O(L²) per-step cost from v0.71.18. A LoRA student (the common distill case) activates the cache too: the new _supports_kv_cache probe unwraps the PEFT model via get_base_model() before deciding. The teacher is always cached; the student cache respects the retained autograd graph and degrades gracefully if a model returns no cache mid-loop.
  • soup serve --mole KV-cache (closes #262). Each of the N task adapters in a served MoLE now keeps its own KV cache in lockstep, created fresh per generate() call (never stored on the instance, so there is no cross-request leak). Top-k zero-weight adapters are still skipped, and the output is byte-identical to the no-cache path on a real MoLE.
  • Deploy-autopilot live measure factories (closes #143). soup deploy autopilot --measure ships a first-party transformers loader factory (lazy import, per-candidate quant config via the Quant Menu loader; before = base, after = quantised) replacing the inject-only test hooks. The baseline is now scored once and the whole candidate list is pre-validated up front, so a typo in --measure-candidates raises before any model load instead of burning N live loads or doubling peak VRAM.
  • Live-codec TTS via SNAC, partial (#265-partial). The live-codec encode path (data.format='audio') is validated for Orpheus: load_audio_mono now probes soundfile.info (duration + byte cap) before soundfile.read (no multi-GB decode into RAM) and reads through an O_NOFOLLOW file descriptor; a real SNAC-backed encode of a 24 kHz wav produced 42 Orpheus codec tokens.

Fixed

• MiniLLM on-policy KV-cache was silently disabled for LoRA students (the PEFT wrapper hid the base model's past_key_values support) — now probed via get_base_model().
• Deploy-measure no longer re-scores the baseline once per candidate or burns live model loads on a bad candidate (per-candidate validation moved up front).
• load_audio_mono capped audio duration only after decoding into RAM — the cap is now checked from soundfile.info before reading.

Known limitations

• KV-cache correctness is validated (cache == no-cache equality on real tiny artifacts) but large-model throughput gains were not measured on the 4 GB dev box.
• #265 stays open — the live-codec data.format='audio' SNAC encode path is validated for Orpheus only; the other four TTS families keep their per-family codec dependency gate.
• The deploy-measure first-party factory's real quantized (bitsandbytes 4-bit) load is CUDA + bitsandbytes-gated; on Windows / no-bnb the injected test seams are the validated path.
• The MoLE serve KV-cache assumes single-sequence (B == 1) decode.

[0.71.21] - 2026-06-10

Added

• Precision & rollout lift (BETA, hw-gated) — lifts five deferred NotImplementedError stubs to live code.
  • FP8 attention + NVFP4 (closes #141). training.fp8_attention: true now converts the model's attention projections (q/k/v/o + fused qkv variants) to FP8 training modules via torchao's convert_to_float8_training with an attention-only module_filter_fn (Hopper SM ≥ 9.0 gate); training.nvfp4: true quantises via torchao's NVFP4Config (Blackwell SM ≥ 10.0 gate). Both are wired into the v0.28 speed/memory pipeline and degrade to a visible yellow advisory when the gate fires — a conversion failing partway raises an honest "model may be PARTIALLY converted" error rather than silently training on a half-converted model.
  • vLLM sleep mode (closes #124). training.vllm_sleep_mode: true is live: create_vllm_engine(sleep_mode=True) sets AsyncEngineArgs.enable_sleep_mode (vLLM ≥ 0.7 gate with a friendly upgrade message), the new vllm_sleep_cycle(engine, level=1|2) context manager wraps the optimisation step (wake in finally), and the GRPO trainer threads the flag into TRL's GRPOConfig when the installed TRL exposes the hook (advisory otherwise).
  • Multi-turn agent rollout launchers (closes #125). soup train with task: grpo + training.rollout_backend: openenv + training.rollout_func: my_module:fn now runs a LIVE rollout: the resolver imports the operator's callable (same trusted-code policy as data.prompt_strategy), feeds it the dataset prompts as seeds, and the returned {prompt, answer?} rows replace the prompt dataset. Rows are normalised (extra keys stripped, message-list prompts deep-copied, non-string answers rejected loudly). art / ruler / nemo_gym raise a friendly ImportError when the backend package is missing and an honest BETA gate when present (injectable _EXTERNAL_ROLLOUT_RUNNERS seam). Validated by a real GRPO + openenv rollout train on SmolLM2-135M.
  • Apple-adapter conversion (closes #228). soup apple-adapter is live for hf-to-mlx / mlx-to-hf: PEFT LoRA safetensors ↔ mlx-lm adapters with both matrices transposed (lora_A [r,in] ↔ lora_a [in,r]), bf16 sources upcast via the torch loader, adapters.safetensors + num_layers emitted for mlx-lm's load_adapters, rank/alpha/dropout carried through, legacy adapters.npz still read, optional v0.60 Merkle-root signing. The *-to-apple directions stay upstream-gated (no published FoundationModels adapter spec). Validated by a real bf16 PEFT adapter round-tripping with numeric equality.
  • Llama-4 expert delinearization (closes #97). soup delinearize-llama4 now runs a live torch runtime: fused 2-D expert tensors [E*dim_in, dim_out] reshape to 3-D [E, dim_in, dim_out] (expert count from config.json or --num-experts), other tensors pass through, JSON sidecars are copied, writes are atomic. --plan-only keeps the old render-and-exit flow.

Fixed

• safetensors.numpy.save silently mangles non-contiguous (transposed) arrays — the apple-adapter writer now makes every array C-contiguous first (caught by the new round-trip assertions).

Known limitations

• fp8_attention / nvfp4 / vllm_sleep_mode are BETA hardware-gated — the converters and gates ship validated via capability probes and fake-module dispatch tests, but end-to-end runs need a Hopper/Blackwell GPU + torchao (or vLLM ≥ 0.7), none of which exist on the maintainer's RTX 3050 / Windows box. The art / ruler / nemo_gym rollout adapters are honestly BETA-gated until validated against the upstream packages.

[0.71.20] - 2026-06-09

Added

• Modality II trainers — TTS / BitNet / MoE expert quant (BETA, hw-gated) — lifts three v0.52.0 schema-only NotImplementedError stubs to real code.
  • TTS fine-tuning (closes #131). soup train with task='tts' + modality='audio_out' now routes to a live TTSTrainerWrapper. TTS families (Orpheus / Sesame-CSM / Llasa / Spark / Oute) are decoder language models, so a TTS fine-tune is next-token cross-entropy over interleaved [text][audio-codec-token] chat sequences — the wrapper reuses the SFT path and adds per-family emotion-control templating (Orpheus / Oute) and registration of operator-supplied codec special tokens (data.new_special_tokens) with an embedding resize. The pre-encoded chat workflow (codec tokens produced offline, then trained with data.format=chat) is the live, validated path; the live-codec workflow (data.format='audio', encode raw audio at train time) needs the family's heavyweight codec dependency (SNAC / BiCodec / XCodec2 / …) and is hardware/dependency-gated with a friendly per-family RuntimeError. Verified end-to-end on SmolLM2-135M-Instruct.
  • BitNet 1.58-bit (closes #134). build_bitnet_trainer returns a live BitNetTrainerWrapper that gates on the upstream onebitllms package (absent → friendly RuntimeError naming it). soup export --format bitnet | tq1_0 now runs a real llama.cpp TQ1_0 ternary export (reuses the v0.53.1 gguf convert→quantize pipeline) instead of the deferred panel; it requires a built llama.cpp toolchain (friendly FileNotFoundError when absent).
  • MoE expert quant + router-only training (closes #136). apply_moe_expert_quant detects fused-MoE expert nn.Linear blocks and replaces them with bitsandbytes Linear4bit (nf4) / Linear8bitLt (int8_rowwise), leaving attention + the router in full precision; it runs before get_peft_model (QLoRA-on-experts) so PEFT attaches to the quantized base. train_router_only freezes every expert and keeps the gating router trainable, applied after LoRA. CUDA-gated (friendly RuntimeError when bitsandbytes/CUDA absent). Validated live on an RTX 3050: 8 expert Linears → 8 Linear4bit with dequant error 0.0155 vs source (weights genuinely carried), router-only freeze, and device-aware placement.

Known limitations

• The TTS live-codec workflow, BitNet 1.58 training (onebitllms), and BitNet GGUF export (llama.cpp) are hardware/dependency-gated — the friendly gates ship and the plumbing is validated, but the end-to-end runs against real TTS models + audio codecs / a BitNet base + onebitllms / a built llama.cpp toolchain stay open infra-blocked items on the maintainer's RTX 3050 / Windows box.

[0.71.19] - 2026-06-09

Added

• Quant Menu for vision / audio modality (closes #81). The Quant Menu (gptq / awq / hqq:Nbit / aqlm / eetq / mxfp4 / fp8) was rejected by the config modality gate for modality in {vision, audio} — those paths carried inline BitsAndBytesConfig blocks that handled only 4bit / 8bit. v0.71.19 drops the gate (the mlx-backend gate is retained) and threads the unified build_quantization_config_for_loader through _setup_vision_transformers / _setup_audio_transformers, so multi-modal SFT can train a LoRA on top of any pre-quantized base. The 4bit / 8bit config shapes are byte-for-byte the same as the old inline blocks; mxfp4 still routes through prepare_model_for_kbit_training. Verified: the unified loader returns the right config object for every format on both modalities, and _setup_vision_transformers threads a GPTQConfig into AutoModelForVision2Seq.from_pretrained.

Fixed

• Multipack DataLoader sharding under FSDP / DeepSpeed ZeRO / DDP (closes #80). The multipack get_train_dataloader override built a raw DataLoader and returned it directly, so under distribution every rank trained on the same packed bins (no data sharding). It now routes the loader through accelerator.prepare(...) when num_processes > 1 — exactly what HF Trainer's own get_train_dataloader does — so accelerate's BatchSamplerShard round-robins whole bins across ranks (preserving the FFD packing) and equalises per-rank batch counts. The single-process path is unchanged (byte-for-byte the validated v0.40.4 raw-DataLoader behaviour). Verified live: a single-GPU multipack SFT on SmolLM2-135M trains end-to-end (RTX 3050). Full multi-GPU validation remains a QA item (no multi-GPU box); the distributed routing is mocked-tested.

[0.71.18] - 2026-06-08

Added

• MiniLLM true on-policy rollout (closes #257). training.minillm_on_policy: true (with minillm_enabled: true) replaces the offline distribution blend with the real on-policy procedure of Gu et al. 2024 §3.1: each step samples a fresh autoregressive rollout from the per-token mixture ratio·teacher + (1-ratio)·student, then accumulates the length-normalised reverse-KL KL(student || teacher) on the full distributions (differentiable w.r.t. the student only; sampled tokens are detached). New training.minillm_rollout_length knob ([1, 512]; auto-derives min(max_length, 32) when unset — the loop re-forwards the full prefix each step, so keep it small). Verified live: on-policy distill on tiny-gpt2 (student + frozen teacher), finite loss, end-to-end train.
• Cross-tokenizer ULD with token-sequence alignment (closes #258). New training.uld_strategy: wasserstein_aligned handles fully-disjoint tokenizers (not just a vocab-size mismatch): per batch element the student and teacher token sequences are aligned over their decoded character spans (offset-overlap when both decode to the same text, difflib Ratcliff-Obershelp char matching otherwise), the teacher logits are mean-pooled onto the student positions, and the existing sorted-Wasserstein-1 surrogate is applied. Verified live: aligned distill with a GPT-2 BPE student + a Llama SentencePiece teacher, finite loss, end-to-end train.
• soup agent eval --sandbox (closes #110). Each heuristic-passing tool-call prediction is now executed against a generated mock of the endpoint in the v0.25.0 RLVR code_exec sandbox and classified into ok / tool_error / timeout / arg_error. The endpoint path, its required path params, and the predicted arguments are base64-embedded as data (no code interpolation). Strong isolation (RLIMIT / namespaces / sandbox-exec) is POSIX-only; on Windows the subprocess + 5 s timeout + 10 KB output cap + network guard still apply (a friendly reduced-isolation advisory is printed). Verified live on Windows: 4-prediction scorecard (ok=1 / tool_error=1 / arg_error=2 / timeout=0).
• soup train --cloud modal (closes #16). Render a self-contained Modal.com app from soup.yaml for serverless GPU training when you have no local GPU. The config YAML is base64-embedded as data (no interpolation, no secrets); the --gpu type (t4 / l4 / a10g / a100 / a100-80gb / l40s / h100) is validated against a closed allowlist. Default is plan-only: write the stub + print the modal run command. --cloud-submit attempts a live submit gated on a Modal token (modal setup / MODAL_TOKEN_ID + MODAL_TOKEN_SECRET). New [modal] extra (pip install 'soup-cli[modal]'; only needed for live submit — plan-only render needs no dependency). Verified live: real stub rendered, exit 0.

[0.71.17] - 2026-06-08

Added

• Serve-time MoLE (closes #259). A task='moe_lora_routing' run now writes a self-describing mole_manifest.json next to mole_gate.pt, and soup serve --mole <dir> loads the base + N frozen task LoRAs + the trained gate and blends them per token at decode time (custom blend loop — non-streaming + streaming). --mole requires --backend transformers and is mutually exclusive with --bank / --steer / --adapters / --speculative-decoding. The base model comes from --base (or the manifest when unset). Verified live on SmolLM2-135M (2 task adapters, real generation + SSE streaming).
• Per-request multi-tenant vector banks (closes #260). soup serve --bank now resolves the active VeRA/VB-LoRA user per request via a contextvars.ContextVar, so concurrent requests on a threaded server never race on shared instance state. The streaming path re-selects the user inside the generator's own context. Verified live: two X-User-Id headers produce distinct steered outputs, an absent / unknown id self-clears to the clean baseline (no cross-request leak), and a repeated user is deterministic.
• Epoch-aware RAFT document shuffle (closes #253). data.raft_epoch_shuffle: true re-permutes the golden + distractor documents each training epoch (per-epoch salt) so the model can't latch onto one fixed citation slot. epoch=0 reproduces the legacy single-permutation order exactly. Verified live on a 2-epoch SmolLM2-135M RAFT run.
• soup diagnose --citation-style / --shuffle-seed (closes #254). The live citation failure-mode probe now accepts the citation style (bracket / inline / footnote) and the RAFT shuffle seed so the golden [doc-N] ids line up with what the model saw at train time. Verified live (rows=6, mean_recall=1.000).

Fixed

• MoLE train() now returns the initial_loss / final_loss / total_steps / duration_secs / duration keys the generic train handler reads, so soup train task=moe_lora_routing completes cleanly (previously raised KeyError: 'initial_loss' after writing the gate). Surfaced by the #259 smoke.

[0.71.16] - 2026-06-07

Added

• Covariance-preconditioned ROME via --cov-corpus (closes #250). soup edit set --method rome --cov-corpus <jsonl|txt> now estimates the key covariance C = E[k kᵀ] + λI over a stats corpus and uses the preconditioned update u = C⁻¹ k* instead of the covariance-free C = I path — the genuine ROME closed form, which spreads the rank-1 update mass to reduce collateral interference with other facts. Falls back to C = I when no corpus is given. The exact post-condition down(k*) += delta is preserved either way. The corpus loader is cwd-contained, symlink-rejected (O_NOFOLLOW + raw-path lstat), and size/line-capped; --cov-corpus is rejected (fail-loud) for any method other than rome. Verified on real gpt2 (prob 0.005 → 0.9997) and SmolLM2-135M.
• GPT-2 (transformer.h / mlp.c_proj) support in the edit kernels (closes #251). ROME / MEMIT / AlphaEdit now edit GPT-2-family models, not just Llama-family. The Conv1D weight layout ([in, out], transposed relative to nn.Linear's [out, in]) gets a transpose-aware rank-1 update, AlphaEdit null-space projection, and MEMIT band dim-check. PEFT-wrapped GPT-2 / Llama models are unwrapped via get_base_model. Verified end-to-end on real gpt2.
• Mixtral joins the LongLoRA architecture allowlist (closes #147). A bare mistral token does not appear in mixtral (m-i-x vs m-i-s), so the existing is_mistral_model detector excluded the MoE variant. A dedicated is_mixtral_model helper + MixtralAttention entry in the S² forward-override regex + _SEPARATE_QKV_FAMILIES now cover Mixtral-8x7B / 8x22B (the attention is the standard separate-QKV shell; the MoE lives in the MLP).

Fixed

• Atomic EditGovernor edit-count increment (closes #252). Two concurrent soup edit set runs on the same base model could lose an increment: each read the persisted count, added locally, and the last writer clobbered the first. save_state now re-reads the persisted count INSIDE the cross-process lock and merges this run's delta (edit_count − persisted_baseline), mirroring the v0.60.0 namespace_pin pattern. Verified: two governors recording 3 + 2 edits from the same baseline persist a merged 5 (not a clobbered 2 or a naive +1).

Notes

• Test count: 13511 → 13595 (+84 net; +81 in tests/test_v07116.py).

[0.71.15] - 2026-06-07

Fixed

• Iterative-DPO config render bug (closes #261). soup iterative-dpo's default per-round trainer rendered output: {dir: ...} (a mapping), which SoupConfig.output (a plain string) rejected — so the spawned soup train subprocess failed at config validation. Now renders output: <str>, mirroring the v0.71.13 #229 local-rl fix. A regression test captures the rendered YAML and validates it via load_config_from_string; verified end-to-end with a real soup train round on SmolLM2-135M.

Changed

• CMA-ES merge loads the base model once (closes #246). soup adapters merge --strategy cmaes previously reloaded the (multi-GB) base model into a fresh PEFT wrapper on every candidate in the population. The default scorer now loads the base once and reuses it across the whole population × generations loop — each candidate only loads its small merged LoRA, applies it, generates, and unloads it. Verified on SmolLM2-135M: the base loads exactly once across N candidates.
• soup loop budget gate now estimates real cost (closes #245). The pre-wired loop's per-iteration cost estimate was a hard 0.0 placeholder, so the dollar budget gate never tripped. It now wires v0.34 run_cost. estimate_run_cost_usd off the most-recent completed run's GPU + duration (the best forward signal for a repeating loop). Falls back to 0.0 on the first iteration / a CPU / unpriced GPU; never crashes the daemon.
• --diagnose-gate is multi-node aware (closes #170). The post-training diagnose gate (and the --annex-xi / --repro-receipt / capture hooks) fired on LOCAL_RANK==0, so a shared-filesystem multi-node run ran them once per node. They now gate on the global chief (RANK==0 when RANK is set, else LOCAL_RANK==0) — once per cluster.

Added

• soup train --track-energy --energy-out <path> (closes #244) persists the measured energy/CO2 reading as JSON so soup bom emit --energy <path> (the v0.71.3 #256 consumer) can attach it to an ML-BOM. Atomic + cwd-contained + symlink-rejected. Completes the train → BOM energy hand-off.

[0.71.14] - 2026-06-05

Added

• Live FSDP shard consolidation (closes #96). soup merge-sharded-fsdp-weights lifts the v0.44.0 plan-only stub: it now streams each pytorch_model_fsdp_*.bin shard via torch.load(weights_only=True) (no arbitrary pickle exec), unions the per-rank parameter fragments into one state-dict, and writes a single .safetensors atomically. Memory-friendly (one shard loaded at a time). New --plan-only flag prints the plan without writing. Single-process — no multi-GPU needed to MERGE. (Per-rank disjoint-parameter / FULL_STATE_DICT shards; DCP sharded-tensor reconstruction is out of scope — use accelerate merge-weights for those.)
• Live kv_cache_type wiring on the transformers serve backend (closes #140). soup serve --kv-cache-type q8_0 | bf16 | f16 | fp8 lifts the v0.53.1 apply_kv_cache_type NotImplementedError stub: bf16/f16 load the model in that dtype (the KV cache inherits it); q8_0 routes an 8-bit HQQ quantized KV cache through model.generate (needs pip install hqq); fp8 raises a friendly runtime error (vLLM + Hopper-only — the transformers backend has no fp8 KV path). vLLM / SGLang KV-cache-dtype routing stays in the infra-blocked tail.
• ONNX export QA verified (closes #71) — soup export --format onnx exercised end-to-end on a tiny model: export exits 0, model.onnx loads in ONNX Runtime with input_ids present, and a forward pass produces a real output. Recorded in tests/qa/v07114_qa.md.

Notes

• GGUF export (#70), AWQ/GPTQ export (#72), the CUDA + llama.cpp QA doc (#144), HF Hub push/Spaces deploy (#74), and the Community-QA tracking meta-issue (#79) remain open with infra-blocked labels — they need a built llama.cpp toolchain, autoawq/auto-gptq Windows wheels, or HF credentials the QA box lacks. See tests/qa/v07114_qa.md.

[0.71.13] - 2026-06-04

Added

• Prompt-compile family — live wiring (closes #225, #226, #227, #229). Four soup commands that shipped as deferred-stub NotImplementedError in v0.68.0 are now real, validated end-to-end (real DPO train on SmolLM2-135M + real Ollama teacher distillation on RTX 3050).
• soup local-rl train runs a real nightly DPO/KTO/ORPO train (#229). --once harvests the latest thumbs-up/down DPO pairs from the local-RL SQLite and trains them via a soup train subprocess (argv list, no shell); a state table tracks last_train_at so a re-run with no new feedback skips, and a run with fewer than --min-pairs (default 10) skips. Without --once it renders a systemd .service/.timer + launchd .plist scheduler scaffold into --scheduler-dir for the user to install. New flags: --once, --min-pairs, --output/-o, --scheduler-dir, --hour, --minute.
• soup distill-prompt prepares a real distillation dataset (#226). For each prompt in the traces JSONL the teacher is called once via the v0.20 provider helpers (Ollama / Anthropic / vLLM); sft/kl emit {messages:[user, assistant=teacher]} and preference emits {prompt, chosen=teacher, rejected=student}. New flags: --provider, --base-url, --temperature, --max-rows.
• soup compile runs DSPy / GEPA / TextGrad prompt-program optimisation (#225) and soup compile-tools runs the TextGrad / GEPA tool-schema optimiser (#227), both lazy-importing the optimiser libraries behind the new [compile] extra (pip install 'soup-cli[compile]') with a friendly ImportError naming the extra when absent. --plan-only still renders the plan and exits 0.

Security

• systemd / launchd injection defence (#229). local-rl and the scheduler renderers reject \n / \r in the model id and shell-quote every ExecStart argument, so a crafted model id cannot inject extra unit directives.

Fixed

• local-rl train config rendered output as a mapping (#229). The nightly soup train YAML now emits output: <dir> (a plain string the schema accepts) instead of output: {dir: <dir>}; a regression test validates the rendered config against SoupConfig.

[0.71.12] - 2026-06-04

Added

• Architecture + distillation + adapter-training — live wiring (closes #145, #146, #148, #158, #84, #221, #222). Seven surfaces that shipped schema-only in earlier releases are now real, validated end-to-end on tiny models (SmolLM2-135M / a locally-built tiny Llama).
• Sequence-level knowledge distillation is live (#145). task: distill now accepts distill_mode: token|sequence; sequence mode trains the student on the teacher's generated continuations (cross-tokenizer-friendly hard-label KD) instead of per-token logit matching. sequence mode is mutually exclusive with the v0.70 cross-tokenizer ULD logit path.
• Classifier LoRA is live (#146). task: classifier|reranker|cross_encoder now attaches a LoRA adapter to the sequence-classification head when lora is configured, so a frozen encoder + small adapter can be trained instead of the full model.
• LLaMA Pro block expansion is per-architecture (#148). expand_layers now interleaves zero-initialised identity blocks for Llama / Qwen / Mistral decoder stacks (was Llama-shaped only), with freeze_trainable_layers freezing the original blocks so only the new ones train.
• LongLoRA S² shifted-sparse attention is live (#158). use_longlora: true now installs the shifted-sparse-attention forward override on the Q/K projections (Llama / Mistral / Qwen / Phi), restoring the patched forwards on context exit.
• Mixture-of-Depths is live (#84). use_mod: true attaches a per-layer top-k token router (mod_capacity_factor) so only a subset of tokens receive each block's residual update. Architecture allowlist: Llama / Qwen / Mistral; unsupported bases warn and skip.
• VeRA / VB-LoRA multi-tenant serving is live (#221). soup serve --bank <bank.json> [--bank-strength S] reconstructs the shared projection + per-user scaling vectors and installs a decode-time forward hook; the active user is selected per request via the X-User-Id header (an unknown/absent id is a zero-delta no-op, so there is no cross-request leak). Serves N personas at ~KB-per-user instead of a full LoRA each.
• MoLE per-token adapter routing is live (#222). task: moe_lora_routing with mole_task_adapters: [...] trains a per-token gating network that blends N frozen task LoRAs (mole_top_k / mole_temperature); only the router trains. The gate is saved as mole_gate.pt alongside the run.

Changed

• apply_bank_to_serve (#221) and build_gating_kernel (#222) now return live objects (a LoadedVectorBank and a torch.nn.Module router) instead of the v0.67.0 deferred-stub NotImplementedError.

[0.71.11] - 2026-06-04

Added

• GRPO / RL callbacks — live wiring (closes #235, #236, #237, #238, #239, #240, #159, #160). The reward-hacking, cross-tokenizer distillation, MiniLLM, mid-epoch RL checkpoint, iterative-DPO and echo-trap surfaces that shipped schema-only in v0.70.0 are now real, validated end-to-end on SmolLM2-135M.
• Reward-hacking detector is live (#235). --reward-hack-detector info_rm|rm_ensemble now installs a GRPO TrainerCallback that reads the per-step rewards (via a shared, thread-safe reward-fn capture buffer), computes an InfoRM cluster-separation drop (info_rm) or RM-ensemble divergence (rm_ensemble), classifies OK/WARN/HACK, logs the verdict to state.log_history, and halts training on HACK when --reward-hack-halt is set. rm_ensemble requires ≥2 reward functions.
• Cross-tokenizer ULD distillation is live (#236). task: distill with --uld-strategy wasserstein|topk_align now computes a real Wasserstein-1 (sorted-CDF) or top-k-aligned distillation loss inside the distill trainer, handling student/teacher vocab-size mismatch by clamping teacher ids to the teacher vocab.
• MiniLLM reverse-KL distillation is live (#237). --minillm-enabled adds a teacher-mixed, length-normalised reverse-KL term plus an optional pretrain-anchor SFT term (--minillm-pretrain-anchor-path / --minillm-pretrain-anchor-weight) that keeps the student near coherent language. The anchor corpus reader is cwd-contained + symlink-rejecting with a per-line byte cap.
• Mid-epoch RL checkpoint is live (#238). --rl-checkpoint-save-every-steps N writes a real adapter + optimizer state + JSON manifest every N steps during PPO/GRPO and prunes to --rl-checkpoint-keep-last, so a long RL run survives a crash without losing the optimizer momentum.
• soup iterative-dpo orchestrator is live (#239). Runs the full sample → reward-score → build-pairs → DPO-train loop across rounds: each round samples completions from the previous round's adapter, the next round trains a fresh LoRA from the base on that round's harvested pairs. --plan-only still renders the plan without running.
• Echo-trap detector is live (#240). --echo-trap-enabled installs a GRPO callback that scores per-trajectory n-gram repetition, classifies OK/WARN/TRAP against --echo-trap-threshold, logs the verdict, and halts on TRAP when --echo-trap-halt is set (catches RAGEN-style degenerate repetition in multi-turn agent RL).
• GRPO variant fallback now warns once (#159). When a --grpo-variant custom compute_loss falls back to the base trainer (because the installed TRL renamed the loss inputs), the trainer logs a one-shot WARNING instead of silently degrading to the default objective.

Changed

• GRPO reference-model EMA no longer materialises full state dicts (#160). --ref-model-ema-alpha now updates the reference model in place by iterating named_parameters() (ref = (1-α)·ref + α·policy), eliminating the three model-sized allocations per step the v0.53.11 path made. A total name/shape-mismatch (0 shared parameters) logs a one-shot WARNING so a misconfigured EMA can't silently no-op.

[0.71.10] - 2026-06-03

Added

• RAG family — live wiring (closes #199, #200, #201, #202). The four retrieval / steering surfaces that shipped schema-only in v0.62.0 are now real, validated on SmolLM2-135M.
• RAFT span-mask training is live (#199). data.format: raft rows ({query, golden_doc, distractor_docs, answer}) now train answer-only: the prompt span is masked to -100 and each document is labelled [doc-N] so the model learns to cite the supporting document. Documents are shuffled reproducibly (data.raft_shuffle_seed). Rows whose prompt fills max_length (answer fully truncated) are dropped with a warning rather than silently shrinking the effective dataset.
• soup ra-dit — one-shot two-stage orchestrator (#200). Trains the retriever (stage 1, embedding/contrastive) then the generator (stage 2, RAFT-SFT) in a single command, recording the trained retriever as the generator's paired retriever. A soup train of a generator-stage config with no retriever model set now auto-links the most-recent RA-DIT retriever run from the Registry. --plan-only validates both configs without training; --retriever-model overrides the auto-link.
• soup steer train / apply + soup serve --steer are live (#201). Fit a CAA (contrastive activation addition), ITI (inference-time intervention) or RepE (representation-engineering PCA) control vector from {positive, negative} contrastive pairs, persist it as a safetensors + config artifact, and apply it at decode time via a forward hook (soup serve --steer <name> --steer-strength <s>).
• soup eval citation + citation-span loss boost are live (#202). Score citation precision / recall / F1 over {predicted, expected_ids} or RAFT rows (--shuffle-seed aligns the golden [doc-N] id with what the model saw at train time). When citation_faithful: true, bracketed [doc-id] spans in the answer get a boosted per-token loss weight. A new citation failure mode is available in soup diagnose.

[0.71.9] - 2026-06-03

Added

• Knowledge edit + unlearn — live wiring (closes #193, #194, #196, #197, #203). The v0.61.0 / v0.62.0 schema-only stubs are now live, validated on SmolLM2-135M.
• soup edit set (ROME / MEMIT / AlphaEdit) is live (#194). New soup_cli/utils/edit_kernels.py ships covariance-free rank-1 weight-edit kernels: ROME (single-layer W += δ·kᵀ/‖k‖²), MEMIT (residual distributed across a layer band), AlphaEdit (ROME update projected orthogonal to the down-proj's top singular direction). apply_edit loads the model, optimises the target residual, applies the rank-1 update, and optionally saves with cwd-containment + symlink rejection. --output, --device, --governor/ --no-governor flags added. On a tiny model a ROME edit moved P("Lyon" | "The capital of France is") from 0.0016 → 0.96.
• soup edit diff live before/after generation (#194). Pass --before-model + --after-model (+ --probes) to generate completions through both models and surface the probes whose output changed.
• EditGovernor SQLite persistence + cross-process locking (#196). New EditGovernorStore (mirrors namespace_pin.NamespacePinStore — $HOME/$CWD/$TMPDIR containment, TOCTOU symlink rejection, WAL + busy_timeout, fcntl/msvcrt sidecar lock, POSIX 0600). save_governor / load_governor / default_governor_db_path (env override SOUP_EDIT_GOVERNOR_DB) persist per-base-model edit-count + verdict across separate soup edit set runs.
• apply_edit consults the EditGovernor automatically (#197). When a governor is supplied, check_can_edit() runs BEFORE the model load (refusing on norm blowup / edit cap) and record_edit() runs AFTER with the measured Frobenius delta.
• Live GRACE codebook (#203). GraceCodebook (epsilon-ball nearest-key lookup), apply_grace_edit (captures a residual key + optimises a value + appends to a codebook sidecar), save_codebook / load_codebook (atomic, cwd-contained, symlink-rejected), install_grace_hook (decode-time residual substitution). New edited_model / grace_codebook Registry artifact kinds.
• soup train --task unlearn is live (NPO / SimNPO / RMU) (#193). New soup_cli/utils/unlearn_kernels.py (NPO (2/β)·mean(-logσ(-β(πlp-reflp))), length-normalised SimNPO, RMU representation steering) + a self-contained UnlearnTrainerWrapper loop loading a LoRA policy, a frozen reference (NPO/RMU), and forget/retain JSONL datasets. NPO/SimNPO forget loss decreased on the tiny-model smoke. Warns when run without a retain set.

Security

• _save_edited_model / UnlearnTrainerWrapper output dirs + save_codebook / load_codebook + _load_unlearn_rows enforce cwd-containment, raw-path symlink rejection (TOCTOU), null-byte rejection, and file-size / per-line caps. apply_grace_edit honours the governor for direct callers.

[0.71.8] - 2026-06-03

Added

• Probes & SAE — real weights + live downloads (closes #215, #216, #217, #218, #219). A new shared soup_cli/utils/probe_kernel.py provides the linear-probe math (contrast-pair derivation, apply, flag-rate, verdict bands, operator-supplied weight loading, deterministic synthetic fallback); every heavy import (numpy / torch / safetensors) is lazy.
• soup probe sleeper --weights <w.npz|.npy|.safetensors> (#215) — load a real calibrated probe direction instead of the synthetic fallback. Weights are cwd-contained, symlink-rejected, O_NOFOLLOW-opened, allow_pickle=False, and size-capped. compute_contrast_probe(positive, negative) derives a probe from contrast-pair activations.
• soup probe sae-diff <repo> --auto-download (#216) — fetch an allowlisted SAE from the HF Hub into ~/.soup/sae-cache/ (validated against HF_HUB_ALLOWLIST BEFORE any network call) via a new SSRF-hardened soup_cli.utils.hubs.snapshot_download (repo-id shape + home/cwd/tmp cache containment + namespace-pin TOFU gate).
• soup probe truth / soup probe harm (#217) — TruthfulQA-style honesty and HarmBench-style misuse activation probes (6 bundled bases each, 5% / 20% verdict bands, --weights to skip the allowlist with a real probe). The probe pack now ships truth + harm entries per base.
• soup probe interference --measure <eval_suite> --base-model <m> --adapter name=path ... (#218) — auto-measure the N×N interference matrix by actually loading the base + each LoRA adapter (PEFT multi-adapter), measuring loss for each adapter alone (diagonal) and each co-loaded pair (add_weighted_adapter(combination_type="cat"), off-diagonal). Exit 2 on a MAJOR worst-pair.
• soup train --capture-activations <layer> --capture-prompts <jsonl> (#219) — a post-training hook writes an SAE-diff-ready per-token activation snapshot to <output>/activations/activations.json. resolve_layer_module resolves the same model.layers.N path whether or not a LoRA adapter is loaded (PEFT-wrapper fallback).

Security

• Probe / SAE / capture file I/O is cwd-contained + O_NOFOLLOW (TOCTOU close)
  • size-capped; SAE weight loads use allow_pickle=False. SAE auto-download validates the allowlist before any network call and rejects a glob result that resolves outside the snapshot dir (symlink-escape guard).

Notes

• #215 is partial: the operator-supplied / contrast-pair / synthetic paths ship now, but the 6 large-base Anthropic-calibrated probe vectors remain upstream-gated (no public calibrated artifact exists). Documented as a known limitation.

[0.71.7] - 2026-06-02

Added

• Eval live runners — six probe surfaces that previously emitted heuristic / neutral stubs now load a real model and run live (closes #161, #162, #208, #211, #212, #165). New shared soup_cli/utils/live_eval.py provides the model-loading primitives (generator / multi-generator closures, masked cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement); every heavy import (torch / transformers / peft / lm_eval) is lazy.
• soup advise --probe-model <id> — runs a LIVE ROI probe: zero/few-shot token-F1 baselines, a short LoRA probe (relative held-out-loss improvement + real wall-clock), and base-model proximity (held-out logit agreement) folded into the dataset profile. Without --probe-model, --probe stays the offline heuristic.
• soup tunability --live — replaces the offline heuristic with a real per-candidate LoRA probe (loads each repo_id, trains --probe-steps on a held-out-excluded slice, reports the held-out-loss drop).
• soup eval capability --live --model <id> — invokes lm-eval-harness per resolved task (or a --tasks override) with --limit / --device, isolating per-task failures and surfacing a no-metric result as an explicit error.
• soup eval behavior --base-model <id> [--adapter <path>] — generates pre/post responses on the bundled behaviour battery and scores the live diff.
• soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>] [--tokenizer <id>] — runs all six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) live via soup_cli.utils.diagnose.live.run_live_diagnose; falls back to neutral OK or --evidence JSON when no model is supplied.

Security

• The two new JSONL dataset readers (diagnose.live._load_dataset_rows, tunability._load_jsonl_rows) open with O_NOFOLLOW after the cwd-containment check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader policy).

[0.71.6] - 2026-06-02

Added

• soup build live runner — the dbt-for-SFT DAG (soup build <manifest>) now materialises datasets instead of only dry-running the plan. Five built-in transforms ship live (identity, drop_empty, lowercase, strip, dedup_exact); table rebuilds from scratch, view re-derives on every run, and incremental re-transforms only the rows whose content hash changed (tracked in a SQLite state store, keyed by row hash and the model's transform+config fingerprint so a transform change re-runs everything). Custom transforms are passed per-run via the Python API's transforms= map. Outputs are written atomically; the --output-dir is symlink-checked before any directory is created.
• soup data gen-magpie live generator — the Magpie synthetic generator (Xu et al. 2024) now actually generates. It feeds an aligned model its chat-template prefix (chatml / llama3 / gemma / mistral families auto-detected) and harvests the self-generated user instruction + assistant response via raw completion. Live providers: ollama (/api/generate raw) and vllm (/v1/completions) — both SSRF-hardened (loopback-only HTTP); anthropic is rejected (no raw-completion endpoint). Optional --quality-filter drops low-quality rows via the v0.47 toxicity/educational scorers; exact-duplicate instructions are de-duplicated.
• soup eval irt-subset --model {1pl,2pl,3pl} — the IRT eval-cost optimiser gained 2PL (per-item discrimination) and 3PL (+guessing floor) joint coordinate-ascent MLE fits alongside the existing 1PL Rasch. 1pl keeps the closed-form path for back-compat; 2pl/3pl route through the new fit_irt.
• Tokenizer-aware memorization probe — score_memorization(..., tokenizer=...) and split_prefix(..., tokenizer=...) (used by soup diagnose) now split the prefix/suffix on real token-id boundaries and measure echo-overlap over sub-word tokens when a tokenizer (HF id / path / duck-typed object) is supplied, catching BPE-level memorization that whitespace tokenisation misses. Default (no tokenizer) keeps the whitespace behaviour.

Fixed

• soup data augment --provider ollama|vllm no longer crashes — the command imported a non-existent OllamaProvider symbol and raised ImportError on every non-OpenAI provider. It now routes through the shared, SSRF-hardened provider factory; --model / --base-url are honoured, the output path is containment- and symlink-checked, and the write is atomic.

Security

• Ollama / vLLM provider URLs reject 0.0.0.0 — validate_ollama_url / validate_vllm_url dropped the bind-any wildcard from their loopback allow-set (now localhost / 127.0.0.1 / ::1 only), matching the newer validate_hub_endpoint / validate_webhook_url SSRF validators. Reachable now that Magpie threads a user-supplied --base-url through these providers.

[0.71.5] - 2026-06-02

Added

• soup eval against now reads eval metrics — ExperimentTracker.get_metric_series falls back to the eval_results table when the metric is not a per-step training column (loss / lr / grad_norm / speed / gpu_mem). So soup eval against <base> --candidate <run> --metric task_accuracy returns a real score series (benchmark scores live in eval_results, not metrics) instead of "Empty series". Per-step columns still read from metrics — no regression for existing callers.
• soup advise learns from past project outcomes — soup advise now reads this project's accepted-verdict history (~/.soup/advise_history.jsonl) and biases the rubric: 3+ successful SFT precedents flip a marginal RAG call to SFT; 3+ negative GRPO outcomes suppress GRPO in favour of SFT-on-traces; an encouraged choice gets a small confidence nudge. Scoped per-project (one project's record never biases another). No history → identical to before.
• Slack/Discord webhooks on four more commands — --slack-url / --discord-url (SSRF-hardened, loopback-only HTTP, RFC1918 rejected, never crashes the command) now ship on soup ingest, soup prune-prompt, soup ab (fires only on a reject_h0 / accept_h0 decision, not continue), and soup data active-sample — not just soup drift-alarm. The validator + sender moved to a shared soup_cli/utils/webhooks.py.
• Tokenizer-aware soup prune-prompt — --tokenizer <model_or_path> detects and strips the shared system-prompt prefix on token boundaries instead of characters, so a multi-byte UTF-8 prefix can never be split mid-code-point. Default (no --tokenizer) keeps the whitespace-character behaviour.
• Curriculum bucketing by loss percentile — DynamicCurriculumCallback now buckets samples by the percentile rank of the live loss (or perplexity) signal within a rolling window when data.curriculum_metric is loss / perplexity, so a consistently-hard sample is routed to the same difficulty bucket across recomputes. length and warm-up still use round-robin.
• --hub on soup data push and soup data forge — soup data push --hub modelscope|modelers uploads a dataset via the matching SDK (repo_type=dataset, commit message sanitised); soup data forge --hub <non-hf> --teacher owner/name pre-fetches the teacher model from that hub (and warns when the teacher is not a repo id so --hub is never silently ignored). HF stays the default.

Notes

• Live SaaS pull adapters for soup ingest (Langfuse / LangSmith / Helicone / OpenPipe / OpenAI SDKs, issue #204) remain deferred: they need credentialed vendor accounts with populated trace data to validate honestly. Tracked as an open, infra-blocked (external-account) item. soup ingest continues to parse the JSONL export you pull from your dashboard.

[0.71.4] - 2026-06-02

Added

• Live canary verdict for soup adapters merge — --canary <suite.json> scores the merged adapter against the first input and classifies OK / MINOR / MAJOR using the Quant-Lobotomy taxonomy (drop <2% OK, <5% MINOR, else MAJOR). --strict-verdict exits 2 on MAJOR. Pre-scored {"baseline_scores","candidate_scores"} suites run with no model load; a {"tasks":[...]} suite uses an injectable scorer. Replaces the v0.57 UNKNOWN stub.
• Live evolutionary merge — soup adapters merge --strategy cmaes --eval <suite> --budget <t> now runs the full CMA-ES loop: each candidate is merged, materialised, scored against the eval suite, and the best-weighted merge is written to --output. Replaces the v0.67 plan-only stub.
• Publish an adapter PR to GitHub — soup adapters pr <title> --base-sha <hex> --adapter <path> --push owner/repo#N posts the rendered PR Markdown as a GitHub PR comment via gh api (argv-list, body over JSON stdin; no shell). Token resolves from GITHUB_TOKEN / GH_TOKEN.
• Pre-wired soup loop production stages — soup loop init --pre-wired (or soup loop watch --pre-wired) swaps the v0.58 no-op stage stubs for real harvest (traces → preference pairs) → DPO train → eval-gate → canary-deploy callables. soup loop status now shows the pre_wired flag.
• Loop iterations as Soup Cans + Registry lineage — soup loop watch --pack-cans packs each successful iteration as a v0.26 Soup Can and appends a Registry entry (tag loop-iter), chaining a real lineage DAG across iterations visible through soup history. `soup loop replay --extract ` unpacks a recorded iteration.
• Branch pointers into the Registry — soup adapters branch <name> --attach-to-registry <id> links a branch snapshot to a Registry entry (shown as a branches node in soup history); soup adapters branch <name> --from-registry <id> derives a fresh snapshot's config + base from an entry.

Security

• The backdoor-scan gate (v0.71.2 #192) and license-conflict gate (v0.60 Part E) now run for all merge strategies, including --strategy cmaes (previously bypassed because cmaes returned before the gates).
• soup loop canary deploy restricts SOUP_LOOP_SERVE_ENDPOINT to loopback / RFC1918-private hosts (a serve endpoint is the operator's own box/LAN), beyond the general webhook SSRF policy which permits any HTTPS host.
• soup adapters pr --push builds the gh child environment from an allowlist so unrelated secrets (HF_TOKEN / OPENAI_API_KEY / …) never reach the subprocess.
• The canary-suite JSON read uses O_NOFOLLOW + os.fstat (size cap enforced on the same fd) to close the symlink/size-cap TOCTOU window.

[0.71.3] - 2026-06-01

Added

• Energy & CO2 measurement for training — soup train --track-energy wraps the training window in a codecarbon offline tracker (no IP-geolocation network call) and reports kWh / CO2 / grid intensity, feeding those numbers into --annex-xi. New EnergyTracker context manager; graceful no-op when codecarbon is absent (pip install soup-cli[carbon]). --energy-country picks the ISO-3166 alpha-3 grid for the CO2 estimate (default USA).
• PDF Annex XI/XII documents — soup train --annex-xi report.pdf now renders a reportlab PDF (a .md path still renders markdown). pip install soup-cli[pdf].
• Auto-populated training-corpus domains in Annex XI/XII — the top crawled domains (with shares) are now extracted from the training JSONL and listed in the EU AI Act docs, replacing the previous empty placeholder.
• Soup Can manifest v3 with embedded attestations — soup can pack --attest <statement.json> (repeatable) embeds in-toto Statements into a v3 can manifest; v1/v2 cans still load. Each statement is shape- and size-validated.
• Local audit log auto-instrumentation — every soup command now appends one HIPAA/SOC2-shaped record to ~/.soup/audit.jsonl (secrets redacted, args capped). Opt out per-invocation with --no-audit-log or globally with SOUP_NO_AUDIT_LOG=1. Tail/rotate with soup audit-log.
• Reproducibility receipt in airgap bundles — soup airgap-bundle --repro-receipt <receipt.json> embeds an SR 11-7 receipt as repro-receipt.json; auto-detected from <model>/repro-receipt.json when not supplied.

Security

• soup can pack --attest now rejects oversize attestation files by their raw size before parsing them into memory (defence against memory-exhaustion).
• The new file-loading paths (attestation JSON, airgap receipt, training-corpus scan, PDF write) are all cwd-contained + TOCTOU symlink-rejected and size-capped; the audit auto-log redacts hf_/sk-/Bearer tokens and never crashes the CLI on a broken log.

[0.71.2] - 2026-06-01

Added

• ed25519 signing for soup adapters sign / soup attest — real detached signatures (over the adapter Merkle root / the in-toto statement) via a new [sign] extra (pip install soup-cli[sign], pulling cryptography). soup adapters sign --backend ed25519 --key <priv.pem> (or --generate-key <out.pem>, or SOUP_SIGNING_KEY); soup adapters verify [--public-key <trusted.pem>] does a cryptographic verify and, with a trusted key, genuine authentication. soup attest emit --sign ed25519 --key <priv.pem> writes a <output>.sig sidecar; new soup attest verify <statement> --signature <sig> verifies it (canonical-JSON, so it's platform/newline-independent). Sigstore keyless signing stays infra-blocked (needs an OIDC provider + Fulcio/Rekor network — can't be honestly validated offline).
• Anti-AI-Jacking namespace pin on Hub downloads — HF model fetches now consult a trust-on-first-use pin store: a repo whose author changes (or whose created_at jumps backward) is refused unless the namespace shift is explicitly allowed. Fails open when repo metadata is unavailable.
• License auto-detection at soup adapters merge — when --license isn't given, the license is read from each adapter's adapter_config.json / config.json / model-card frontmatter (HF llama3.1-style ids mapped to canonical) and the conflict gate runs automatically.
• Backdoor-scan gate at soup adapters merge — refuses to merge any input whose soup adapters scan returns FAIL (or can't be scanned) unless --allow-unscanned is passed; WARN is advisory.

Changed

• License-conflict overrides (--license-override <reason>) are now recorded to the audit log for legal review.
• The namespace-pin store now uses SQLite WAL + busy-timeout and a cross-process file lock around its get+insert, so concurrent writers don't lose the trust anchor.

Security

• ed25519 verification fails closed (any tamper / wrong key / missing key ⇒ invalid). Signing keys + trusted public keys are symlink-rejected and size-capped via a shared reader (no cwd-containment — keys live outside the project). --generate-key refuses to overwrite any existing path.

[0.71.1] - 2026-06-01

Added

• soup env fix — render a reproducible install plan from soup-env.lock. Emits copy/paste uv pip install commands (--format uv-pip, default) or a requirements.txt body (--format requirements); --output optionally writes a requirements.txt under cwd. Print-only by design — never shells out to a package manager.
• soup lock write --env-lock <path> — auto-derive --env-hash from a soup-env.lock so operators who ran soup env lock don't copy the hash by hand. --env-hash still wins when passed explicitly.
• soup serve --record-thumbs <db> — capture thumbs-up/down feedback into a local-RL SQLite at startup, plus a new POST /v1/thumbs endpoint (transformers backend). Returns 404 when the flag isn't set.
• Judge-calibration persistence: JudgeCalibrationReport.to_dict, write_judge_calibration, and load_judge_calibration, backed by a new judge_calibration registry artifact kind. Loading re-validates the report so a corrupt on-disk field is rejected.
• Bundled MUSE and WMDP unlearning eval fixtures so soup eval unlearning --benchmark muse|wmdp runs out of the box. WMDP forget-set probes ship redacted (placeholder prompts + REFUSED responses) — Soup never ships verbatim hazardous content.

Changed

• soup completions now introspects a cached base model's actual LoRA target modules (config-only AutoConfig load, local_files_only=True, never networks or raises) and falls back to the canonical default shape when the base isn't cached locally.
• build_dag exposes a validate_build_source helper (cwd-containment + symlink rejection) for build-manifest source paths.

0.71.0 - 2026-06-01

Changed

• Breaking — install split. The heavy training stack (torch, transformers, peft, trl, datasets, bitsandbytes, accelerate) moved out of the core install into a new [train] extra. pip install soup-cli is now a light CLI + data-tools install with no PyTorch; run pip install 'soup-cli[train]' (or [all]) to fine-tune. Existing users who train must reinstall with [train]. Version pins are unchanged.
• Trimmed README.md to a ~238-line front door; the full feature reference now lives under docs/ (one topic page per area, indexed from the README).
• Raised the pytest coverage gate from 50% to 77% (--cov-fail-under=77).
• Migrated to a src/ layout (src/soup_cli/) for cleaner packaging and to stop tests accidentally importing the in-tree package.

Added

• [train] and [all] optional-dependency extras ([all] pulls train, serve, ui, data). [dev] self-references [train] so CI and contributors still get the full stack from pip install -e ".[dev]".
• Friendly error mapping: a missing heavy dependency (torch, transformers, peft, trl, datasets, bitsandbytes, accelerate) now surfaces "Training needs the [train] extra. Run: pip install 'soup-cli[train]'".
• py.typed marker (PEP 561) so downstream type checkers pick up Soup's inline type hints.
• .pre-commit-config.yaml with ruff (lint + format) and standard file-hygiene hooks.
• Lenient mypy configuration and a non-blocking type-check CI job.
• This CHANGELOG.md.

Removed

• The historical, per-version security-fix log that had grown inside SECURITY.md (~220 KB). SECURITY.md is now a concise security policy; the detailed hardening notes remain in git history and the GitHub Releases notes.