Skip to content

backends-and-ops.md

Latest commit

 

History

History
1126 lines (788 loc) · 43.3 KB

backends-and-ops.md

File metadata and controls

1126 lines (788 loc) · 43.3 KB
Raw

Backends, Platform & Ops

← Back to the Soup README

MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, shell completions, the plugin system, and the standalone utility commands.

Contents:

Autopilot (Zero-Config)

Skip the YAML entirely. Give Autopilot a base model, a dataset, and a goal — it analyzes your data, model, and hardware, then picks the task, quantization, LoRA rank, learning rate, epochs, and performance flags for you.

# Zero-config: pick everything automatically
soup autopilot --model meta-llama/Llama-3.1-8B-Instruct \
               --data ./data/train.jsonl \
               --goal chat

# Other goals: chat | code | reasoning | instruct | vision
soup autopilot --model Qwen/Qwen2.5-7B --data ./data/math.jsonl --goal reasoning

# Constrain to a GPU budget (1GB to 1TB)
soup autopilot --model <id> --data d.jsonl --goal chat --gpu-budget 24GB

# Preview the generated config without running
soup autopilot --model <id> --data d.jsonl --goal chat --dry-run

Autopilot writes a ready-to-run soup.yaml. Edit it by hand if needed, then soup train.

Apple Silicon (MLX Backend)

Fine-tune on M1-M4 Macs via Apple's MLX framework — no CUDA, no emulation.

# Install MLX support
pip install "soup-cli[mlx]"

For SFT with local JSONL, JSON, or CSV data, [mlx] is sufficient on its own. It can also be installed with [train] when the same environment needs the PyTorch/TRL Transformers backend: both extras now share the supported transformers>=5.16.1,<6 range. A Hugging Face datasets source or streaming dataset still needs datasets because that data source owns the dependency; the local file path below does not.

detect_device() and get_gpu_info() recognise Apple Silicon when backend: mlx is set, preserving training.quantization: 4bit for mlx-community pre-quantized checkpoints instead of silently downgrading to none (#423). The CUDA-shaped analytical VRAM preflight is skipped on the MLX path because Apple unified memory is managed by Metal, not a fixed CUDA VRAM pool.

base: mlx-community/Llama-3.2-3B-Instruct-4bit
task: sft
backend: mlx  # Apple Silicon only

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  lora:
    r: 16
    alpha: 32

MLX backend supports SFT. backend: mlx with task: dpo or task: grpo is refused when the config is loaded, with an error naming the task — upstream mlx-lm ships no DPO/GRPO training helper, so those wrappers exist only as a backstop for callers that bypass config validation. Requires mlx-lm >= 0.31.3. Use soup recipes search --tag mlx for ready-made Apple Silicon configs.

Optimizers and schedules

training.optimizer, training.scheduler, training.warmup_ratio and training.weight_decay are honoured on the MLX backend (#686). Before that they were validated, accepted and dropped — every run built a bare AdamW at a constant learning rate, whatever the recipe said.

Because they are honoured rather than ignored, a setting MLX cannot express is now refused when the optimizer is built, rather than silently substituted:

setting MLX accepts otherwise
optimizer adamw_torch, adamw_hf, adamw_torch_fused, sgd, adafactor, adagrad, rmsprop, muon refused by name, listing what is available
scheduler cosine, linear, constant, constant_with_warmup refused, naming the old constant-rate behaviour
weight_decay any value on every optimizer above except adagrad and rmsprop a non-zero value on adagrad / rmsprop is refused — those MLX constructors take no weight_decay, and dropping it silently is the defect above

Soup's optimizer allowlist (utils.optimizer_zoo) is far wider than anything MLX ships, so most valid values have no MLX equivalent. Run those recipes on the transformers backend.

The schedule counts optimizer updates, not iterations: mlx-lm calls optimizer.update() once every gradient_accumulation_steps, so a warmup of warmup_ratio × (iters // gradient_accumulation_steps) is what actually runs. The effective plan is written to adapter_config.json (optimizer, scheduler, warmup_updates, total_updates, weight_decay, peak_lr), so what ran is recoverable from the output directory.

--resume auto finds mlx-lm's step-numbered NNNNNNN_adapters.safetensors checkpoints and warm-starts the LoRA weights from them (#634). This restores adapter weights only, not training state: mlx-lm's LoRA trainer exposes no optimizer state or step count, so training restarts from step 0 regardless of how far the checkpoint got. See Resume Training below for the MLX-specific checkpoint shape.

Transformers on MPS

The regular backend: transformers path can run more than MLX's SFT-only surface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocast for the hardware-validated text trainers: SFT, DPO, GRPO/RLVR, reward modelling, and PRM. The decision comes from a live MPS allocation probe rather than a macOS version guess. CPU and an unavailable or older MPS runtime remain in FP32; FP16 is not selected on MPS.

For BF16 checkpoints, resident SFT, DPO, GRPO, and reward-model runs preserve the frozen base weights in BF16 while keeping LoRA parameters in FP32. PRM uses BF16 autocast but deliberately retains FP32 master weights: loading its trainable base in BF16 makes the Metal optimizer abort because its accumulator and destination matrix dtypes differ. This policy was validated with one-step runs on Apple Silicon for all five tasks. GRPO validation uses local Transformers generation (use_vllm: false) and deterministic RLVR; vLLM remains a CUDA-oriented optional path. Other Transformers trainers remain FP32 on MPS until their task-specific kernels receive equivalent hardware coverage.

Unsloth Backend (2-5x Faster Training)

Use the Unsloth backend for significantly faster training and up to 80% less VRAM:

# Install unsloth support
pip install "soup-cli[fast]"

Then add one line to your config:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: unsloth  # 2-5x faster, -80% VRAM

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Works with all training tasks: SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, and Pretrain. If unsloth is installed but not enabled, Soup will suggest it automatically.

Tip: Soup auto-detects unsloth. When installed, you'll see a hint during soup train if you haven't enabled it yet.

Cloud GPU Training

No local GPU? soup train --cloud modal|lambda renders a provider-specific controller from your soup.yaml. The config YAML is base64-embedded as data; credentials are read from the environment only when a live submission starts.

pip install "soup-cli[modal]"   # only needed for live submit

# Plan-only (default): write the stub + print the `modal run` command.
soup train --config soup.yaml --cloud modal --gpu a100

# Submit live (authenticate once with `modal setup`, or set
# MODAL_TOKEN_ID + MODAL_TOKEN_SECRET).
soup train --config soup.yaml --cloud modal --gpu a100 --cloud-submit

--gpu accepts: t4 / l4 / a10g / a100 / a100-80gb / l40s / h100. The rendered soup_modal_app.py builds an image with soup-cli[train] pinned to your running version, writes the embedded config inside the container, and runs soup train on the chosen GPU.

RunPod (Planned)

RunPod support is currently in development and descoped from live CLI dispatch pending automated lifecycle and termination safeguards. Running soup train --cloud runpod informs the operator that RunPod is not yet live and points to active cloud backends (--cloud modal and --cloud lambda).

Lambda Cloud

Lambda uses an instance rather than a serverless function. The generated local controller sends a secret-free cloud-init script as API user_data, waits for it over SSH, copies the configured output back, and requests instance termination in a finally block. Keep the controller running until it reports that termination succeeded; shutting down the guest does not terminate billing.

Register the public half of an SSH key with Lambda first, then set:

export LAMBDA_API_KEY=...
export LAMBDA_SSH_KEY_NAME=my-lambda-key
export LAMBDA_SSH_PRIVATE_KEY=/path/to/private-key
export LAMBDA_REGION=us-tx-1  # optional; defaults to us-tx-1
soup train --config soup.yaml --cloud lambda --gpu a100 --cloud-submit

--gpu accepts: a10 / a100 / a6000 / h100. Lambda output paths must be relative so the controller can copy the artifact back safely. The API key stays on the caller and is never embedded in cloud-init or instance logs.

The Lambda submission path still requires the paid live-validation checklist in #264 before it can be described as provider-validated. Plan-only rendering and the lifecycle boundaries are covered by offline tests.

Chat with your model

# Chat with a LoRA adapter (auto-detects base model)
soup chat --model ./output

# Specify base model explicitly
soup chat --model ./output --base meta-llama/Llama-3.1-8B-Instruct

# Adjust generation
soup chat --model ./output --temperature 0.3 --max-tokens 256

Push to HuggingFace

# Upload model to HF Hub
soup push --model ./output --repo your-username/my-model

# Make it private
soup push --model ./output --repo your-username/my-model --private

# Group into a Collection
soup push --model ./output --repo your-username/my-model \
    --collection your-username/my-collection-abc123

HuggingFace Hub Deep Integration

Soup treats HF Hub as a first-class artifact backend. One env var, one flag, no token flags to plumb — all operations respect huggingface-cli login credentials by default.

# Self-hosted Hub: set once, every command routes there.
export HF_ENDPOINT=https://hf.internal.example.com

# Auto-push each save_steps checkpoint to HF as a 'checkpoint-<N>' branch.
soup train -c soup.yaml --push-as your-username/my-model

# Resume from the latest branch pushed above.
soup train -c soup.yaml --push-as your-username/my-model --hf-resume

# Upload a local JSONL file as an HF dataset repo.
soup data push --input train.jsonl --hf-dataset your-username/my-dataset

# Wrap your fine-tuned model in a Gradio chat Space in one command.
soup deploy hf-space \
    --model your-username/my-model \
    --space your-username/my-chat-space \
    --template gradio-chat

# Or a Streamlit app:
soup deploy hf-space \
    --model your-username/my-model \
    --space your-username/my-chat-space \
    --template streamlit-chat

Auto-resume workflow: if training crashes, the next soup train ... --push-as ... --hf-resume call picks up the latest checkpoint-<N> branch from your HF repo and downloads it back to output_dir, then resumes — no manual copy / paste of checkpoint paths. Cwd containment and local_dir_use_symlinks=False prevent filesystem escape from a crafted repo.

Auth follows standard HF conventions: HF_TOKEN env var > HUGGINGFACE_HUB_TOKEN

~/.cache/huggingface/token (set by huggingface-cli login) > ~/.huggingface/token. No custom token flags. The deprecated --token on soup push still works but emits a warning.

Model card v2 is auto-generated on first push: it reads sidecar training_config.yaml / soup.yaml to surface task / base / lr / optimizer, and accepts an optional eval scorecard (markdown table). Markdown-active chars in task names and scores are neutralised for safe rendering on HF Hub.

Resume Training

Resume a training run from a checkpoint:

# Auto-detect latest checkpoint in output directory
soup train --config soup.yaml --resume auto

# Resume from a specific checkpoint
soup train --config soup.yaml --resume ./output/checkpoint-500

backend: mlx writes and resumes a different checkpoint shape: a step-numbered NNNNNNN_adapters.safetensors file (or the final adapters.safetensors) directly under output, not a checkpoint-N directory. --resume auto and --resume ./output/0011800_adapters.safetensors both work; --resume ./output/checkpoint-500 does not, because MLX never writes that shape. This is a weights-only warm start — mlx-lm's LoRA trainer exposes no optimizer state or step count, so the resumed run starts counting from step 0 regardless of how far the checkpoint got.

Run Management & Cleanup

LLM training generates massive checkpoint files. Soup automatically manages an SQLite database of your training loss and metrics, empowering you to safely reclaim disk space once training is complete.

# List all historical training runs
soup runs list

# Compare two differing experiments side-by-side
soup runs compare run_202611... run_202612...

# Intelligently clean up redundant checkpoints
# (Preserves the final model and the checkpoint with the lowest loss)
soup runs clean run_202611...

# Preview space that would be reclaimed across ALL experiments
soup runs clean --all --dry-run

By default, the clean command operates in "surgical mode" (--keep-weights), deleting huge optimizer state files (optimizer.pt) from lesser checkpoints to save gigabytes, but keeping their lightweight evaluation weights just in case you want to load them later.

Alternative Model Hubs

Set training.hub in your soup.yaml to download from / push to a non-HuggingFace hub. Useful in regions where HF Hub is unreachable or blocked.

training:
  hub: modelscope   # or 'modelers' (Openmind), default 'hf'

Override the endpoint via env var:

export MODELSCOPE_ENDPOINT=https://my-mirror.example.com
export MODELERS_ENDPOINT=https://corp-modelers.internal   # HTTPS only for non-loopback
soup train --config soup.yaml

The endpoint validator follows the same SSRF rules as HF_ENDPOINT: only http/https schemes; plain HTTP allowed only for localhost / 127.0.0.1 / ::1; private and link-local IPs (RFC1918, 169.254/16, etc.) rejected on plain HTTP. backend: mlx is incompatible with non-HF hubs (mlx-lm only downloads from HF Hub).

The hub adapter is schema-only in this release; the live downloader and uploader land in v0.51.1.

TensorBoard Integration

Log training metrics to TensorBoard for local visualization:

# Enable TensorBoard logging (requires: pip install tensorboard)
soup train --config soup.yaml --tensorboard

# View logs
tensorboard --logdir ./output/runs/

Note: --tensorboard and --wandb cannot be used together. Pick one.

Weights & Biases Integration

Send training metrics to W&B for cloud-based experiment tracking:

# Enable W&B logging (requires: pip install wandb)
soup train --config soup.yaml --wandb

Make sure WANDB_API_KEY is set or run wandb login first.

Ready-Made Recipes

80 pre-built configs for popular models — no guessing hyperparameters:

# List all recipes
soup recipes list

# Preview a recipe
soup recipes show llama3.1-8b-sft

# Use a recipe (writes soup.yaml)
soup recipes use llama3.1-8b-sft

# Search by task or keyword
soup recipes search --task grpo
soup recipes search "reasoning"
soup recipes search --size 7b
soup recipes search "medical"
soup recipes search "vision"

What's covered:

Category Models
General SFT / DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / Embedding / Pretrain Llama 3.1 / 3.2 / 4, Qwen 2.5 / 3, Mistral, Gemma 3, Phi-4, DeepSeek R1 / V3
Vision (multimodal) Llama-3.2-Vision (11B + 90B), Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6
Audio (speech) Qwen2-Audio, SeamlessM4T v2 (translation), Whisper-large-v3 (ASR)
Reasoning All 6 DeepSeek-R1-Distill sizes (Qwen 1.5B / 7B / 14B / 32B + Llama 8B / 70B), Qwen3-Coder 30B, Qwen3-30B-A3B reasoning, Phi-4 reasoning
Small / edge / mobile SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini, Llama-3.2 (1B / 3B)
Domain specialists BioMistral 7B, Meditron 7B (medical) — CodeLlama (13B / 70B), Magicoder 6.7B (code) — Mathstral 7B (math) — Llama-2-13b-finance (FinGPT-style starter) — Nemotron-4 340B
Multimodal reasoning Llama-3.2-Vision GRPO, Pixtral DPO
Multi-GPU llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline
Apple Silicon (MLX) llama3.1-8b / qwen3-8b / gemma3-9b SFT-MLX
Tool-calling / agentic qwen3-8b-tools, llama4-scout-tools

Hyperparameter Sweep

Search for the best hyperparameters:

# Grid search over learning rate and LoRA rank
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --param lora_r=8,16,32

# Random search with max runs
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --strategy random --max-runs 5

# Preview without running — validates first (#642): unknown config keys emit
# the same warning block `train --dry-run` does, and a --param naming no config
# field exits non-zero before any grid is printed
soup sweep --config soup.yaml --param lr=1e-5,2e-5 --param epochs=2,3 --dry-run

# Early stopping: skip remaining runs if loss exceeds 1.5x best
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --early-stop 1.5

Model Comparison

Compare outputs of two models side-by-side:

# Compare with inline prompts
soup diff --model-a ./model_v1 --model-b ./model_v2 --prompt "Explain gravity"

# Compare with a prompts file
soup diff --model-a ./base --model-b ./finetuned --prompts test_prompts.jsonl

# Save results
soup diff --model-a ./a --model-b ./b --prompts prompts.txt --output results.jsonl

Quickstart Demo

Run a complete demo in one command — creates sample data, config, and trains a tiny model:

# Full demo (creates data + config + trains TinyLlama)
soup quickstart

# Just create files without training
soup quickstart --dry-run

# Skip confirmation
soup quickstart --yes

Health Check

Check your environment for compatibility issues:

soup doctor [--nccl]

Shows: Python version, GPU availability, system resources (RAM/Disk), all dependency versions, and fix suggestions. Use --nccl to measure and check multi-GPU communication bandwidth against expected hardware ceilings.

Version Info

# Basic version
soup version

# Machine-readable output
soup version --json
# -> {"version": "0.26.0", "python": "3.11.5", "platform": "linux"}

# Full system info (useful for bug reports)
soup version --full
# -> soup v0.26.0 | Python 3.11.5 | CUDA 12.1 | extras: serve, data

# Full system info in JSON
soup version --full --json
# -> {"version": "0.26.0", "python": "3.11.5", "platform": "linux", "torch": "2.2.0", ...}

Error Handling

Soup shows friendly error messages by default (2-3 lines with a fix suggestion). For full tracebacks:

# Global flag goes BEFORE the command
soup --verbose train --config soup.yaml

# Works with any command
soup --verbose eval --model ./output --benchmarks mmlu

Note: --verbose is a global flag — it must go before the command name, not after.

Unknown config keys

Every config model used to run with Pydantic's default extra="ignore", so a key the schema did not declare was dropped without a word. soup train --dry-run printed "Config valid. Ready to train!", the run exited 0, and the requested setting was simply never applied — quantizaton: none trained 4-bit quantized when full precision was what you asked for, gradient_checkpoint: true did no checkpointing, max_len: 512 truncated at 2048.

Loading a config now reports every key it cannot place, in one report per load, with the field you probably meant:

Warning: unknown config key 'data.max_len' - did you mean 'max_length' or 'video_maxlen'? Not applied.
unknown config key 'training.quantizaton' - did you mean 'quantization' or 'quantization_aware'? Not applied.
Soup v0.75 will reject unknown config keys instead of warning.

The deadline is real: v0.75 refuses to load a config with an unknown key. The warning ships in v0.74 and the refusal one minor later, so there is exactly one release of notice — deliberately, because a config written against a newer Soup has to keep running on an older wheel for at least one release. Until v0.75 the key is ignored, not defaulted, and the run proceeds as if you had not written it. Treat the warning as work to do, not as a note.

A config that names a key your installed Soup does not have usually means one of two things: a typo (take the suggestion), or a field added after your version shipped (soup version against the changelog will say which).

soup sweep is stricter, and has no deadline. A --param that matches no config field is a hard error from this release, not in v0.75:

soup sweep --config soup.yaml --param lora_rank=8,16   # the field is training.lora.r
# sweep parameter does not match any config field: unknown config key 'lora_rank' - not applied.
echo $?   # 1

The whole sweep is refused before the first arm starts, and the command exits non-zero, so a scripted or CI-driven sweep fails rather than reporting a grid of arms that each failed for the same reason. A sweep whose swept knob is never applied produces arms that are all identical, so there is no partially-useful result to preserve by continuing.

Experiment Tracking

Every soup train run is automatically tracked in a local SQLite database (~/.soup/experiments.db).

# List all training runs
soup runs

# Show detailed info + loss curve for a run
soup runs show run_20260223_143052_a1b2

# Compare two runs side by side
soup runs compare run_1 run_2

# Delete a run
soup runs delete run_1

# Replay an old run's summary + loss curve from history
soup runs replay run_1

Every completed run also stores an estimated cost ($ per run) computed from the captured GPU device name and duration. soup runs show renders — for CPU / MPS / unknown GPUs (no fabricated zeros).

As of v0.71.5, the metric-series lookup that powers replay (ExperimentTracker.get_metric_series) transparently falls back to the eval_results table when a metric has no per-step rows — so you can plot a benchmark-score curve (e.g. mmlu, gsm8k) the same way you plot loss, without caring which table holds the series.

Tracker integrations (--tracker mlflow / swanlab / trackio)

# Stream metrics to MLflow (set MLFLOW_TRACKING_URI to your server URL)
soup train --config soup.yaml --tracker mlflow

# Or SwanLab (cloud or local)
soup train --config soup.yaml --tracker swanlab

# Or Trackio (offline-friendly batched upload)
soup train --config soup.yaml --tracker trackio

--tracker is mutually exclusive with --wandb and --tensorboard. Soup validates the tracker name against a closed allowlist (mlflow / swanlab / trackio / wandb / tensorboard / none); the upstream package itself is loaded by HF Trainer at run time, so install the one you need separately:

pip install mlflow      # or: swanlab / trackio

Telemetry (opt-in)

Soup ships a hardware-info-only telemetry payload (Soup version + command + Python major.minor + OS + arch + duration + anonymous distinct ID). It is off by default and never sends model names, dataset paths, or config contents.

To opt in, set the environment variable:

SOUP_TELEMETRY=1 soup train --config soup.yaml

When SOUP_TELEMETRY is unset, 0, or any value other than 1/true/yes/on, Soup performs zero telemetry network requests.

You can also explicitly disable telemetry for a specific invocation using the --no-telemetry flag:

soup train --config soup.yaml --no-telemetry

When enabled, telemetry performs a synchronous fire-and-forget HTTP POST with a 1-second connect and read timeout (DNS resolution excluded) on command exit. The anonymous identifier is stored at ~/.soup/telemetry_id; deleting ~/.soup/telemetry_id regenerates it on the next opt-in. See Privacy Policy for details.

Profiling Extras

CUDA memory snapshots, anomaly tracing, and an NCCL bandwidth reference table:

from soup_cli.utils.profiling_v0_43 import (
    memory_snapshot_context, detect_anomaly_context, nccl_bandwidth_check,
)

with memory_snapshot_context("run-123") as path:
    train_step()
    # On CUDA, dumps profiles/run-123.snapshot.pickle on exit.

with detect_anomaly_context():
    train_step()
    # torch.autograd.set_detect_anomaly(True)

result = nccl_bandwidth_check(
    gpu="h100", link="nvlink", measured_gb_per_sec=400.0,
)
# {'expected_gb_per_sec': 450.0, 'measured_gb_per_sec': 400.0,
#  'ratio': 0.8889, 'status': 'OK'}

VS Code Setup (.vscode/launch.json)

One-shot writer for a sane debugger config:

from soup_cli.utils.vscode_setup import write_vscode_launch
write_vscode_launch(config_path="soup.yaml")
# Writes ./.vscode/launch.json with `soup train` + pytest entries.

Symlink-rejected at the target path regardless of force=True to defend against pre-placed symlinks redirecting the write outside cwd.

Observability & Dev UX

Tools that explain why a run misbehaved instead of dumping a stack trace.

soup why

Heuristic explainer — reads the most recent (or named) run and surfaces plain-English diagnoses with concrete next steps.

soup why                 # most recent run
soup why run_2026_abc    # specific run id (or prefix)

Detects: NaN/Inf loss, plateau (≥30 steps with <0.5% change), divergence (loss > 3× initial), persistent high gradient norm, learning rate outside the typical [1e-6, 5e-3] band. Pure rule-based — no model calls.

soup tui

Full-screen Textual dashboard. Two-pane: run list (left) + selected-run detail (right). r refreshes, q quits.

pip install "soup-cli[tui]"
soup tui --refresh 1.0 --limit 50

Auto-profiling — soup train --profile

Records a torch.profiler Chrome-trace over an early-steps window (default wait=1, warmup=1, active=5, repeat=1). Output: <output>/profiles/<run_id>.trace.json. Open in chrome://tracing or Perfetto.

Crash bundles — .crash files

When training fails, Soup auto-writes a self-contained .crash JSON to ./.soup-crashes/crash_<utc>_<hex>.crash containing: redacted error trace, classified failure kind (oom / nan / cuda / dataloader / nccl / other), GPU state at crash time, env summary, last-50 metric rows, and the config (recursively redacted of hf_* / sk-* / Bearer … tokens). The output_dir is reduced to os.path.basename so $HOME doesn't leak.

--log-level quiet|normal|verbose|debug

Global flag on the root soup command. Wires a Rich-formatted logger on the soup namespace; debug enables timestamps + module paths.

soup --log-level verbose train --config soup.yaml
soup --log-level debug runs show <id>

GPU Live Monitor

soup monitor                # 2s refresh, Util / Mem / VRAM / Temp / Power per GPU
soup monitor --refresh 0.5  # faster polling
soup monitor --once         # single snapshot, no Live panel

On NVIDIA systems, Soup calls nvidia-smi via a list-args subprocess (no shell) with a 5-second timeout. On Apple Silicon, it reads GPU utilization and power from /usr/bin/powermetrics --samplers gpu_power --format plist. Run sudo -v in a terminal before starting the monitor: Soup uses sudo -n, so it can reuse the cached credential without ever prompting for or reading a password. If the credential or utility is unavailable, the command exits with an Activity Monitor fallback rather than reporting an NVIDIA error.

macOS does not expose NVIDIA-style dedicated VRAM, memory-utilization, or GPU temperature fields through this sampler. Those columns therefore remain — instead of guessing values from unified memory or unrelated thermal sensors.

Soup Fetch — Bundled Examples

soup fetch examples                          # list bundled entries
soup fetch examples llama-3.1-8b-lora        # write to ./llama-3.1-8b-lora.yaml
soup fetch examples qwen2.5-7b-dpo -o ./my-config.yaml --force
soup fetch deepspeed_configs zero3-cpu-offload

Closed catalog (MappingProxyType) of ready-to-edit YAML / JSON. Output path cwd-contained, bundled-source os.path.commonpath check (defends against catalog escape), os.lstat + S_ISLNK symlink-reject at the write target.

Llama 4 Delinearizer

soup delinearize-llama4 ./llama4-checkpoint --target ./out-delinearized [--num-experts N] [--plan-only]

LIVE (v0.71.21): reshapes fused Llama-4 expert weights [E*din, dout] → [E, din, dout] shard-by-shard (atomic writes, per-shard 16 GiB cap, cwd containment) and copies the JSON sidecars so the target stays loadable. The expert count defaults from config.json (text_config.num_local_experts); pass --num-experts when the config doesn't carry it (exit 2 otherwise). --plan-only keeps the original preview flow and writes nothing. is_llama4_model uses a word-boundary regex matching the is_gemma4_model pattern — ungemma-llama-4ish is rejected.

Ctrl+C Graceful Save

First SIGINT → trainer writes a checkpoint and continues. Second SIGINT → trainer stops cleanly after the next save. No-state fallback raises KeyboardInterrupt so the user never gets stuck. GracefulSaveHandler.install() is idempotent and swallows signal.signal failures on non-main threads.

Checkpoint-Now Trigger File

touch ./out/.checkpoint_now    # trainer saves on the next step, then deletes the trigger

Path containment via is_under_cwd; os.lstat + S_ISLNK rejection at the trigger target so a pre-placed symlink can't redirect the write.

Onboarding Wizard Helper

from soup_cli.utils.onboarding import render_onboarding_yaml

text = render_onboarding_yaml({
    "base": "meta-llama/Llama-3.2-1B",
    "dataset": "./train.jsonl",
    "task": "sft",
    "quantization": "4bit",
    "epochs": 3,
})

Five-question wizard input → fully-validated soup.yaml. Literal allowlists on task (sft / dpo / kto / orpo / simpo / ipo / bco / preference) and quantization (4bit / 8bit / none); epochs ∈ [1, 10]; output cwd-contained; null-byte rejection on every string.

Standalone Sweep Config

soup sweep --config sweep.yaml
# sweep.yaml
strategy: random
n_runs: 20
seed: 42
params:
  lr: [0.0001, 0.0005, 0.001]
  epochs: [1, 3, 5]

Strict scalar allowlist on values (str / int / float / bool); _MAX_FILE_BYTES=256KB, _MAX_PARAM_KEYS=32, _MAX_VALUES_PER_KEY=64; SweepSpec.params is MappingProxyType[str, Tuple[Any, ...]] for genuine immutability.

Alternative Model Hubs (ModelScope / Modelers)

Set training.hub to fetch the base model from a non-HF Hub:

base: baichuan-inc/Baichuan2-7B
task: sft
training:
  hub: modelscope     # or "modelers"

soup train pre-fetches the model into ./.soup_hub_cache/<sanitized-slug>/ via the matching SDK (modelscope.snapshot_download / openmind_hub.snapshot_download) and rewrites cfg.base to the local snapshot. Re-runs reuse the cached snapshot. Both huggingface-hub, modelscope, and openmind-hub are lazy-imported — install only what you need.

Programmatic API:

from soup_cli.utils.hubs import download_repo, upload_repo

local_path = download_repo("modelscope", "baichuan-inc/Baichuan2-7B", local_dir="./snap")
upload_repo("modelers", "my-org/my-model", folder_path="./output", commit_message="Soup v0.53.8")

The dispatcher enforces shape validation on every input (bool / null-byte / leading-slash / .. segments / control characters / oversize all rejected) and runs cwd-containment on local_dir / folder_path.

Experiment Trackers (MLflow / SwanLab / Trackio)

Pick a tracker on the CLI; Soup threads it into HF Trainer's report_to:

soup train --tracker mlflow
soup train --tracker swanlab
soup train --tracker trackio

If the package is not installed, Soup now surfaces a friendly advisory before training starts instead of a mid-run ImportError:

--tracker mlflow requires the 'mlflow' package. Install with: pip install soup-cli[trackers] (or pip install mlflow)

pip install soup-cli[trackers]   # mlflow + swanlab + trackio

Telemetry & Privacy Policy

Soup contains opt-in, hardware-info-only telemetry in utils/trackers.py (build_telemetry_payload / send_telemetry_payload).

Privacy Policy

Soup's telemetry is strictly anonymous and hardware-focused. When opted in via SOUP_TELEMETRY=1, we collect only the following fields to understand what environments we need to support:

  • soup_version: the version of Soup being run
  • command: the top-level command executed (e.g. train, data, validated against known commands; unknown commands or paths are masked as (unknown))
  • python: Python major.minor version
  • os: OS platform name (platform.system())
  • arch: System architecture (platform.machine())
  • duration_seconds: Command execution duration in seconds
  • distinct_id: Anonymous UUID4 generated locally on first run and stored at ~/.soup/telemetry_id to deduplicate events. Deleting ~/.soup/telemetry_id regenerates it on the next opt-in.

We NEVER collect:

  • Dataset paths or contents
  • Model names or architectures
  • Config file contents or hyperparameters
  • Usernames, local file paths, or directory names
  • IP addresses, tokens, or credentials

All uploads use HTTPS, a 1-second connect and read timeout (DNS resolution excluded), and defensive SSRF validation. Any network or filesystem exception is silently swallowed so telemetry can never fail or interrupt your work.

Plugin System

Drop a Python module under src/soup_cli/plugins/ (or any package importable by Soup) and register at import time:

from soup_cli.plugins import register_plugin

class MyPlugin:
    def pre_train(self, ctx):
        ...
    def post_train(self, ctx):
        ...

register_plugin(
    name="my-plugin",
    version="1.0.0",
    plugin=MyPlugin(),
    description="Hooks into pre/post-train",
    templates=["my-template"],         # optional
    model_groups=["my-arch-family"],   # optional
)
soup plugins              # list registered plugins
soup plugins enable foo
soup plugins disable foo

Plugin names are kebab-case (^[a-z0-9][a-z0-9-]{0,39}$); versions are semver-ish (MAJOR.MINOR.PATCH); registry caps _MAX_PLUGINS=64, _MAX_TEMPLATES_PER_PLUGIN=32, _MAX_MODEL_GROUPS_PER_PLUGIN=32. Re-registering the same (name, version, plugin, templates, model_groups, description) is idempotent; any field mismatch is rejected with a clear error. Trainer-callback wiring of pre_train / post_train / pre_step / post_step lands in v0.45.1.

External Integrations Catalog

from soup_cli.utils.integrations import list_integrations, get_integration

list_integrations()                       # 15 entries
get_integration("lm-studio").target_artifacts   # ("gguf",)

15 ecosystem targets covered: lm-studio, comfyui, stable-diffusion-cpp, open-webui, ollama, tei, pgvector, faiss, weaviate, sentence-transformers, claude-code, cursor, continue, cline, sillytavern. Auto-detect + launch wiring lands with v0.46.0 Deploy Autopilot.

Advanced Trainer Plugins

from soup_cli.utils.trainer_plugins import validate_trainer_plugin_list

validate_trainer_plugin_list(["grokfast", "spectrum"])
# returns ("grokfast", "spectrum") — canonical lowercase, dedup, ≤ 8 entries

6-entry allowlist (cce_plugin, grokfast, spectrum, llmcompressor, sonicmoe, math_verify) so a future training.trainer_plugins: [...] schema field has a stable surface. Live callbacks in v0.45.1.

Soup Plugin Callbacks

Register a plugin once via the v0.45.0 registry API; v0.53.6 wires it into every transformer-backend trainer as a real HF TrainerCallback:

# src/soup_cli/plugins/my_plugin.py — auto-discovered at `soup` startup
from soup_cli.plugins import register_plugin

class MyPlugin:
    def pre_train(self, ctx):
        print("training about to start, args =", ctx["args"])

    def post_step(self, ctx):
        if ctx["state"].global_step % 100 == 0:
            print(f"step {ctx['state'].global_step}")

register_plugin(name="my-plugin", version="0.1.0", plugin=MyPlugin())

A misbehaving plugin hook is swallowed at WARNING — one bad plugin must never crash a multi-hour training run. The hook snapshot is taken at callback-construction time, so a plugin registered MID-run does not retroactively receive events.

Terraform-Style Plan & Apply (soup plan / soup apply)

A training run is a one-shot infrastructure-shaped operation: spot price, expected cost, base SHA, dataset SHA, peak VRAM. v0.64 borrows Terraform's plan-apply split so you can review the numbers before committing money.

# Render a pre-flight summary + write soup.tfstate
soup plan --config soup.yaml

# Apply — refuses on drift (exit 3) if the YAML changed since `plan`
soup apply --config soup.yaml

# Validate without actually running anything
soup apply --config soup.yaml --dry-run

The state file is a thin JSON envelope; the actual training is still driven by soup train. The gate prevents the "wait, why did I spend another $0.50 on the wrong config" surprise.

Hermetic Env Lockfile (soup env)

The "CUDA hell" problem: a fine-tune that worked on Friday breaks on Monday because PyPI silently upgraded transformers past the trainer's compat band. v0.34 soup doctor surfaces some of this; v0.64 makes it lockable.

# Snapshot the current env into soup-env.lock
soup env lock

# Print the locked env summary
soup env status

# Compare current env to the lock — exit 3 on ABI-sensitive drift
soup env check

soup-env.lock captures Python + platform + CUDA + 15 ABI-sensitive packages (torch / transformers / peft / trl / accelerate / bitsandbytes / flash-attn / xformers / deepspeed / unsloth / vllm / sentencepiece / tokenizers / datasets / huggingface-hub). Wire soup env check into your CI to refuse silent ABI breakage.

Hardware-Fit Calculator

Given (params, seq_len, batch_size, optimizer, quant, peft, gradient_checkpointing), the analytical predictor returns a 5-bucket peak-VRAM breakdown (weights / optimizer / gradients / activations / overhead) and an OK/OOM verdict with a 10% safety margin.

from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit

inp = HardwareFitInput(
    params_b=7.0, seq_len=2048, batch_size=4,
    optimizer="adamw_torch", quant="4bit", peft="lora",
    gradient_checkpointing=True,
)
report = decide_hardware_fit(inp, available_vram_gb=24.0)
print(report.ok, report.reason)
# True | 'fits: peak 7.76 GB + 10% margin <= 24.00 GB available'

When it doesn't fit, the report names actionable knobs: --batch-size halve, --quantization 4bit, --gradient-checkpointing auto. Composes with v0.40.3 live CUDA OOM probe (make_cuda_probe_fn) which still runs when auto_batch_size_strategy: probe.

The weights bucket assumes 2 bytes/param under quant="none" (a frozen base now really does load at the checkpoint's own dtype, typically bf16/fp16 — #339), except peft="full" (full fine-tuning), which explicitly loads fp32 master weights and so assumes 4 bytes/param instead.

Shell Completions (soup completions)

Tab-completion for soup + every subcommand. The generated script is Click/Typer-backed so new commands are picked up automatically.

# Bash
eval "$(soup completions bash)"        # current shell
soup completions bash >> ~/.bashrc     # permanent

# Zsh
soup completions zsh > "${fpath[1]}/_soup"

# Fish
soup completions fish > ~/.config/fish/completions/soup.fish

Recipe names auto-complete from the 115+ catalogue; --target-modules falls back to canonical Llama-shape defaults (q_proj / k_proj / v_proj / etc.). Live HF-config introspection per base lands in v0.64.1.

License Advisor (soup license-advisor)

Picking a license-clean base for a specific deployment target is a recurring legal-review pain point. v0.64 captures the three most common deploy contexts as a closed allowlist and surfaces the per-license downstream risk.

# What licenses are safe for a B2C consumer product?
soup license-advisor --target b2c

# Defense — restricted-use community licenses forbidden
soup license-advisor --target defense

# Per-license check: Llama community license + 800M MAU = block (exit 3)
soup license-advisor --target b2c --license llama-3 --monthly-active-users 800000000

The Llama-family allowlist is tight (no .startswith over-match), so a hypothetical future llama-permissive-2030 won't false-trigger the 700M-MAU gate. Composes with v0.60 soup adapters merge --license <id> for the merge-time conflict gate.

Troubleshooting

soup doctor    # GPU, system resources, dependencies, and version in one place
  • ImportError: DLL load failed while importing _C (Windows). PyPI's torch wheel is CPU-only. Reinstall a CUDA build; soup doctor prints the pip install command for the wheel your driver can run.
  • soup version ≠ pip show soup-cli — multiple Python installs; use a virtualenv.