synthetic_workflow.md

Synthetic data workflow (end-to-end)

Generate, filter, score, and train on synthetic data — all with soup. Pairs with the bundled synthetic_workflow.yaml recipe.

1. Generate

Spin up a local Ollama model and ask it for 200 instruction/response pairs around a topic:

soup data generate \
  --prompt "You write concise Python instruction/response pairs for developers." \
  --provider ollama \
  --model llama3.2:3b \
  --topic "Python error handling" \
  --count 200 \
  --output ./synth_raw.jsonl

For Anthropic / vLLM / server providers, swap --provider and follow the soup data generate --help matrix.

2. Filter for quality

Drop low-perplexity / low-coherence rows:

soup data filter ./synth_raw.jsonl \
  --output ./synth_filtered.jsonl \
  --min-coherence 0.5

3. Score for safety + diversity

Fingerprint PII / toxicity / language / educational value, then decontaminate against your downstream evals. soup data score reports a scorecard to the terminal and does not write a file, so the decontamination step reads the filtered set:

soup data score --input ./synth_filtered.jsonl
soup data decontaminate \
  --input ./synth_filtered.jsonl \
  --output ./synth_clean.jsonl \
  --benchmarks mmlu,gsm8k

4. Train

Point soup train at synthetic_workflow.yaml:

soup train --config examples/synthetic_workflow.yaml --yes

That recipe references ./synth_clean.jsonl, picks TinyLlama-1.1B-Chat as the base, and runs an SFT job with LoRA r=8.

5. Watch progress live (v0.53.9)

In another terminal:

soup ui --public --no-browser

Scan the printed QR code from your phone to monitor the loss curve and live SSE training stream while the job runs.

---

This is a thin walkthrough — for deeper coverage see examples/README.md and examples/configs/.