Soup Examples

Real-world configuration examples and sample datasets to get you running quickly.

Quick Start with Examples

1. Basic SFT (Supervised Fine-Tuning)

Fine-tune TinyLlama on a small instruction-following dataset:

soup train --config examples/configs/sft_basic.yaml

What it does:

• Trains TinyLlama-1.1B for 1 epoch
• Uses LoRA for efficient memory usage
• Outputs to ./output_sft_basic/
• Takes ~2-3 minutes on a consumer GPU

2. Chat Assistant (DPO)

Train a chat model with preference learning:

soup train --config examples/configs/dpo_chat.yaml

What it does:

• Uses Llama 2 7B base model
• Trains with DPO (Direct Preference Optimization) on chat preferences
• Better alignment than SFT alone
• Outputs to ./output_dpo_chat/

DPO with QLoRA (Llama 3.1)

Train a preference-aligned model using DPO with 4-bit quantization:

soup train --config examples/configs/dpo_example.yaml

What it does:

• Uses Llama 3.1 8B Instruct as the base model
• Trains with DPO on simple prompt/chosen/rejected preference pairs
• Uses QLoRA (4-bit quantization) for memory-efficient training
• dpo_beta: 0.1 controls the KL divergence penalty strength
• Outputs to ./output_dpo_example/

3. Reasoning Model (GRPO)

Fine-tune a reasoning model with step-by-step answer verification:

soup train --config examples/configs/grpo_reasoning.yaml

What it does:

• Trains on reasoning tasks (math, logic)
• Uses GRPO (Group Relative Policy Optimization) to optimize for correctness
• Generates multiple outputs per prompt and selects the best
• Outputs to ./output_reasoning/

4. Vision Model

Fine-tune LLaMA-Vision on image-caption pairs:

soup train --config examples/configs/vision_llama.yaml

What it does:

• Trains LLaMA-3.2-Vision-90B on image-text data
• Uses LLaVA format for images + text
• Outputs to ./output_vision/

5. Alignment Methods (KTO / ORPO / SimPO / IPO)

Train with alternative preference optimization:

# KTO — unpaired preference (only needs thumbs up/down labels)
soup init --template kto
soup train

# ORPO — reference-free alignment (no reference model needed)
soup init --template orpo
soup train

# SimPO — length-normalized preference optimization
soup init --template simpo
soup train

# IPO — regularized preference (squared hinge loss)
soup init --template ipo
soup train

6. Continued Pre-training

Continue training on raw text corpora:

soup init --template pretrain
soup train

What it does:

• Trains on plain text (.txt files or JSONL with text field)
• No instruction format needed — just raw text
• Useful for domain adaptation (legal, medical, code)

7. MoE Models

Fine-tune Mixture-of-Experts models (Qwen3, Mixtral, DeepSeek V3):

soup init --template moe
soup train

What it does:

• Auto-detects MoE architecture (ScatterMoE / SwitchTransformers)
• moe_lora: true targets expert-specific LoRA modules
• Optional moe_aux_loss_coeff for load balancing

8. Long-Context Fine-Tuning (128k+)

Extend context windows for long-document understanding:

soup init --template longcontext
soup train

What it does:

• Uses RoPE scaling (dynamic) to extend context to 128k tokens
• Enables gradient checkpointing and FlashAttention for memory efficiency
• Supports linear, dynamic, yarn, longrope scaling types
• Optional Liger Kernel for fused ops: pip install 'soup-cli[liger]'

9. Embedding Model Fine-Tuning

Fine-tune sentence embedding models (BGE, E5, GTE) with contrastive or triplet loss:

soup init --template embedding
soup train

What it does:

• Supports contrastive, triplet, and cosine loss functions
• Configurable pooling: mean, CLS, or last token
• Works with pair data (anchor + positive) or triplets (+ negative)
• Compatible with BGE, E5, GTE, INSTRUCTOR, and any HuggingFace model

10. Audio / Speech Model

Fine-tune audio-language models (Qwen2-Audio, Whisper):

pip install 'soup-cli[audio]'
soup init --template audio
soup train

What it does:

• Trains on audio+text pairs (WAV/MP3 files + conversation)
• Supported models: Qwen2-Audio, Whisper (via transformers)
• Uses modality: audio with format: audio data

11. Batch Inference

Run inference on a batch of prompts:

soup infer --model ./output_sft_basic/ --input prompts.jsonl --output results.jsonl

13. Synthetic-data workflow

End-to-end recipe that generates training data from a local LLM, filters

• scores + decontaminates it, then trains on the cleaned set. See synthetic_workflow.md for the walkthrough and synthetic_workflow.yaml for the bundled config.

soup data generate --provider ollama --output ./synth_raw.jsonl
soup data filter --input ./synth_raw.jsonl --output ./synth_filtered.jsonl
soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl
soup data decontaminate --input ./synth_scored.jsonl \
    --output ./synth_clean.jsonl --benchmarks mmlu,gsm8k
soup train --config examples/synthetic_workflow.yaml --yes

12. Full RLHF Pipeline

Complete reinforcement learning from human feedback:

# Step 1: Pre-train with SFT
soup train --config examples/configs/rlhf_step1_sft.yaml

# Step 2: Train a reward model
soup train --config examples/configs/rlhf_step2_reward.yaml

# Step 3: PPO with reward model
soup train --config examples/configs/rlhf_step3_ppo.yaml

Dataset Formats

Datasets are included in JSONL format. Soup auto-detects and normalizes:

• Alpaca: instruction, input, output fields
• ShareGPT: conversations with from/value fields
• ChatML: OpenAI-style messages with role/content
• DPO/ORPO/SimPO/IPO: prompt + chosen + rejected fields
• KTO: prompt + completion + label fields
• LLaVA / ShareGPT4V: Vision format with image + conversations
• Plaintext: Raw .txt files or JSONL with text field (for pre-training)
• Audio: audio path + messages (for audio/speech models)

Example: Inspect a Dataset

soup data inspect examples/data/alpaca_tiny.jsonl

Output:

📊 Dataset Statistics

Format detected: alpaca
Total entries: 50
Sample 1:
  instruction: "Identify the odd one out"
  input: "twitter, instagram, skype"
  output: "skype"

Example: Convert Between Formats

# Convert Alpaca to ChatML
soup data convert examples/data/alpaca_tiny.jsonl \
  --from alpaca --to chatml \
  --output alpaca_as_chatml.jsonl

Directory Structure

examples/
  configs/              # YAML configuration files
    sft_basic.yaml
    dpo_chat.yaml
    dpo_example.yaml
    grpo_reasoning.yaml
    vision_llama.yaml
    rlhf_step1_sft.yaml
    rlhf_step2_reward.yaml
    rlhf_step3_ppo.yaml
  
  data/                 # Sample datasets (JSONL)
    alpaca_tiny.jsonl
    chat_preferences.jsonl
    dpo_sample.jsonl
    reasoning_math.jsonl

Using Your Own Data

1. Prepare data in one of the supported formats
2. Update the config with your data path:

data:
  path: /path/to/your/data.jsonl
  format: alpaca  # or sharegpt, chatml, llava

3. Run training:

soup train --config your_config.yaml

Tips & Tricks

Save Space: Use Quantization

Add quantization to reduce model size:

quantization: int8  # Reduces memory by 4x

Speed Up Training: Use Unsloth Backend

Unsloth is 2-5x faster training:

pip install 'soup-cli[fast]'

Then in your config:

backend: unsloth

Monitor Training: Use Weights & Biases

Enable W&B logging:

pip install wandb
soup train --config your_config.yaml --wandb

Export for Inference: Convert to GGUF

After training, convert for Ollama/llama.cpp:

soup export output_sft_basic/ --output model.gguf --quant q8_0

Then use with Ollama:

ollama create my-model -f Ollama.modelfile

Merge LoRA Adapter

Merge your LoRA adapter into a standalone model:

soup merge output_sft_basic/ --output merged_model/

Common Issues

"CUDA out of memory"

• Reduce batch_size in config
• Enable quantization: quantization: int8
• Use smaller model: Mistral-7B instead of Llama-70B

"Dataset not found"

• Check file path in config (use absolute path if unsure)
• Verify format is correct: soup data inspect your_data.jsonl

"Model not found on Hugging Face"

• Check model ID spelling
• Ensure you have HuggingFace token: huggingface-cli login
• Or use a different model that's publicly available

Creating Your Own Configs

Minimal Config Template

model: tinyllama-1.1b
data:
  path: ./your_data.jsonl
  format: alpaca
task: sft
lora_r: 16
lora_alpha: 32
batch_size: 32
num_epochs: 3
learning_rate: 5e-4
output_dir: ./output/

Advanced Config Template

model: llama-2-7b
data:
  path: ./dataset.jsonl
  format: sharegpt
task: dpo
backend: unsloth
quantization: int8
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
batch_size: 16
gradient_accumulation_steps: 4
num_epochs: 2
learning_rate: 1e-4
warmup_ratio: 0.1
max_seq_length: 2048
output_dir: ./output_advanced/

See the config schema (the single source of truth) for all available options.

Reward-hacking mitigation demo

reward_hacking/rewards.py provides synthetic reward functions for the closed-loop reward-hacking mitigation feature (soup train --reward-hack-mitigation): a gameable length_hack_reward / sentinel_reward proxy decoupled from a held-out true_score. Point a GRPO config's training.reward_fn at a .py that re-exports one as reward_fn, enable reward_hack_detector: info_rm + reward_hack_mitigation: kl_control, and watch mitigation_log.jsonl under the run's output dir. See docs/training.md.

Learn More

• README: Main documentation
• Docs: Full feature reference
• CONTRIBUTING: How to contribute

Questions?

• Check the GitHub Discussions
• Open an Issue
• Read SECURITY.md for security questions

Happy training! 🍲