Fine-tune and post-train LLMs in one command. No SSH, no config hell.
Website · Quick Start · Config · Docs · Commands · Models
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
pip install "soup-cli[train]" # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup trainTraining LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.
- Zero SSH. Never SSH into a broken GPU box again.
- One config. A simple YAML file is all you need.
- Auto everything. Batch size, GPU detection, quantization — handled.
- Works locally. Train on your own GPU with QLoRA. No cloud required.
v0.72.2 — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU. Layer streaming keeps the frozen base in CPU RAM and feeds it to the GPU one decoder layer at a time. Quantising that base to NF4 shrinks it ~4×, which is what puts an 8B model within reach of a card that cannot hold even a quarter of it.
- Measured on a 4 GB RTX 3050 Laptop (batch 1, S=512, gradient checkpointing, 50 steps after 10 warm-up): Llama-3.1-8B-Instruct at 119.6 tok/s, peak VRAM 3.32 GB, base page-locked at 3.60 GB, GPU 100% busy. Qwen2.5-3B: 264.2 tok/s, 1.76 GB.
- Just add
quantization: 4bitto a streaming config. The base is quantised once, offline, and cached; the cache re-shards by itself if the checkpoint changes underneath it. - Correctness is not traded away. A streamed NF4 run is bit-exact against a resident NF4 run — same quantised bytes, same bitsandbytes kernels — and that is a CI test, not a one-off measurement.
- Why 3B got 1.85× faster too (264.2 vs 143.1 tok/s in bf16): not arithmetic. A 1.43 GB store page-locks where a 5.55 GB one did not, which restores asynchronous copies and takes GPU utilisation from 79.3% to 100%.
- Still BETA, and the scope is unchanged: RAM tier,
task: sft, Llama/Qwen, batch size 1, no gradient accumulation, no--resume. Every refusal names the release that lifts it.
# soup.yaml — then just `soup train --config soup.yaml`
training:
stream_layers: true # base streams from RAM; only the adapter trains
quantization: 4bit # NF4 — ~4x smaller store, so 8B fits a 4 GB card
batch_size: 1Trained with
stream_layers: trueon v0.72.0? That adapter is inert — its tensors were saved under keys with an extra.inner.segment, so every loader returned the untuned base. Fixed in v0.72.1; re-run or re-save. Check with:python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"
Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)
Point soup reward synth at a JSONL of reference outputs and it infers a deterministic verifier,
writes a readable / committable .py reward function, and — the part nobody else does — refuses to
emit one that can't tell your references from bad answers (four families: numeric / json_schema /
regex / tool_call; a mandatory calibration report is the moat). Reward ensembles
(reward_fn: "accuracy,format") also train now. (#311)
soup reward synth references.jsonl -o reward.py --output-report calib.jsonPrevious release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)
soup ship's verdict became emittable, committable, and provenance-bound: --emit-evidence makes a
run replay into an identical verdict, eval.ship in soup.yaml + --config makes the gate policy
reviewable, and --config binds evidence to the exact recipe that produced it (stale evidence → exit 3).
soup ship --push owner/repo#N posts the SHIP / DON'T-SHIP card on the PR.
Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)
soup ship's regression leg became real: a fixed, extraction-based scorer over seven bundled,
offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that
wins your task but quietly breaks tool-calling now gets a DON'T SHIP. Zero new deps.
soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
# exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime errorPrevious release — v0.71.33, soup draft (measure speculative decoding)
soup draft measure reports a draft model's acceptance rate + real plain-vs-assisted tok/s
(exit 0/2/1 for CI); soup draft distill distils your target into a dense tiny draft, auto-wired
into soup serve --auto-spec. The honest result on a small same-family pair: distillation didn't
move acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the
number you want before shipping speculative decoding.
soup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \
--prompts prod-prompts.jsonl # -> acceptance %, real tok/s, ship-or-notFull history: CHANGELOG.md · GitHub Releases.
# Light core: CLI + config + data tools, no PyTorch
pip install soup-cli
# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"
# Everything (train + serve + ui + data) in one shot
pip install "soup-cli[all]"
# Or from GitHub (latest dev)
pip install git+https://github.com/MakazhanAlpamys/Soup.gitThe full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in
docs/models.md.
Use double quotes around the extra. They are the only spelling that works in every shell —
cmd.exe, PowerShell, bash, and zsh.Older tutorials and videos (including some of ours) show the single-quoted
pip install 'soup-cli[train]'. That is bash / zsh / PowerShell syntax, and it fails on Windowscmd.exe, which has no single-quote quoting and hands the quotes straight to pip:ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifierIf you hit that, swap the
'for"— pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads[train]as a glob and fails.)
soup init, soup data …, and the other data/inspection commands work on the light install.
Fine-tuning (soup train) needs the [train] extra.
soup init # interactive wizard
soup init --template chat # or start from a templateTemplates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo,
simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.
soup train --config soup.yaml # LoRA, quantization, batching — all handled
soup chat --model ./output # talk to your model
soup push --model ./output --repo you/my-model
soup merge --adapter ./output # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m # GGUF for Ollama / llama.cppMore export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in
docs/serving-and-export.md.
A complete soup.yaml:
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth # 2-5x faster, pip install "soup-cli[fast]"
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
quantization: 4bit
output: ./outputconfig/schema.py is the single source of truth for every field. Advanced data, training,
and PEFT options are documented under Documentation.
The full feature reference lives in docs/. Start here:
| Guide | Covers |
|---|---|
| Training tasks & methods | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
| PEFT, long context & efficiency | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
| Performance & quantization | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
| Data engineering | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
| Evaluation & probes | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, soup advise |
| Serving & export | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
| Adapters, registry & governance | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
| Compliance & governance quickstart | HIPAA/SOC2/EU-AI-Act/SR-11-7 init templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (soup card), CI gate (soup ci init) |
| Backends, platform & ops | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
| Command reference | The full soup command list |
| Supported models & extras | Recommended model families, the VRAM size guide, the pip extras matrix |
All formats are auto-detected from JSONL, JSON, CSV, Parquet, or TXT:
- alpaca —
{"instruction": ..., "input": ..., "output": ...} - sharegpt —
{"conversations": [{"from": "human", "value": ...}, ...]} - chatml —
{"messages": [{"role": "user", "content": ...}, ...]} - dpo / orpo / simpo / ipo —
{"prompt": ..., "chosen": ..., "rejected": ...} - kto —
{"prompt": ..., "completion": ..., "label": true} - llava / sharegpt4v (vision), audio, plaintext (pre-training), embedding, prm, pre_tokenized, video, multimodal
Full schemas and the Axolotl/LlamaFactory-parity data pipeline (remote URIs, streaming,
sharding, interleaving, vocab expansion, document ingestion) are in
docs/data.md.
soup train --config soup.yaml # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer --model ./output --input prompts.jsonl # batch inference
soup chat --model ./output # interactive chat
soup serve --model ./output # OpenAI-compatible API server
soup merge --adapter ./output # merge LoRA into the base model
soup export --model ./output --format gguf # export for deployment
soup eval benchmark --model ./output # evaluate
soup data inspect ./data/train.jsonl # dataset stats
soup recipes list # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat # zero-config
soup doctor # check GPU / deps / environmentThe complete command list is in docs/commands.md.
Soup works with any text-generation model on the
HuggingFace Hub — if it loads with
AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral,
Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).
| VRAM | Max model (QLoRA 4-bit) | Example |
|---|---|---|
| 8 GB | ~7B | Llama-3.1-8B, Mistral-7B |
| 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B |
| 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B |
| 48 GB | ~70B | Llama-3.3-70B |
| 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 |
Full model + vision tables and the optional-extras matrix are in docs/models.md.
Run Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):
docker pull ghcr.io/makazhanalpamys/soup:latest
docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml
docker compose up # or build locally- Python 3.10+
- GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
- 8 GB+ VRAM for 7B models with QLoRA
All training tasks run on CPU for testing (quantization auto-disabled). Optional extras
(train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed,
liger, mlx, onnx, tensorrt, …) are listed in
docs/models.md.
soup doctor # GPU, system resources, dependencies, and version in one placeImportError: DLL load failed while importing _C(Windows) — reinstall PyTorch for your CUDA version:pip install torch --index-url https://download.pytorch.org/whl/cu121.soup version≠pip show soup-cli— multiple Python installs; use a virtualenv.
git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"
ruff check src/soup_cli/ tests/ # lint
pytest tests/ -v # unit tests (fast, no GPU)
pytest tests/ -m smoke -v # smoke tests (downloads a tiny model, trains)
pre-commit install # optional: ruff lint+format on commitSee CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability.
Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a single 4 GB laptop, which is why every performance number in these docs is measured rather than claimed.
If Soup saved you a training run, starring the repo helps most, and it costs nothing. If you would like to fund the work directly:
❤️ Donate — one-off, any amount (use Change amount on the checkout page). Payments are processed by Stripe under the maintainer's registered business, MePlay, Inc. — that name, not "Soup", is what appears on the checkout page and on your card statement.
Donations fund GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple
Silicon — that a single 4 GB laptop cannot reach. See the
help wanted
issues for exactly what is blocked on hardware today.
Built by the community ❤️ — thank you to everyone who has contributed. See CONTRIBUTORS.md.
Apache-2.0. Copyright © the Soup contributors.