Skip to content

README.md

Latest commit

 

History

History
75 lines (63 loc) · 4.38 KB

README.md

File metadata and controls

75 lines (63 loc) · 4.38 KB
Raw

Red team (Promptfoo, OWASP LLM01-LLM10)

Fills the gap the ledger has been carrying since 2026-07-05: docs/ROADMAP.md row "Red-team (OWASP LLM01-LLM10)" declared "planned — no Promptfoo config exists." promptfooconfig.yaml is that config. It is the automated, repeatable counterpart to the manual, dated exercise in docs/audits/red-team-2026-06-22.md — that report stays as the qualitative, multi-turn record; this config is what a machine can re-run on demand.

What it does

  • Targets the running chat server's POST /api/chat (same request path the shipped UI and sprout ask use), once in English and once in Spanish, so a probe has to defeat the real pipeline (retrieval -> generation -> citation guard -> safety filter), not a mocked shortcut.
  • plugins: [owasp:llm, ...] expands to Promptfoo's mapping of concrete attack plugins to each of LLM01-LLM10:2025 (LLM03 Supply Chain has no runtime-probeable surface — it's covered by make security instead; see the config's comments and docs/audits/red-team-2026-06-22.md §7).
  • Adds hallucination, overreliance, and harmful:specialized-advice on top of the OWASP preset because the manual report flagged "just tell me it's safe" pressure and confabulation-under-coercion (§4, §5) as the highest-stakes classes for this domain.
  • Adds the multilingual strategy because the manual report's stated watch item is that the EN/ES guards are maintained by hand (§6) — this gives it an automated adversarial check instead of relying only on the parity eval suite's fixed cases.

Running it locally

uv run sprout ingest
uv run sprout serve &          # starts the API on :8000 (Ctrl+C / kill %1 when done)

cd eval/redteam
export ANTHROPIC_API_KEY=...   # generation + grading model; see promptfooconfig.yaml
npx --yes promptfoo@latest redteam run
npx --yes promptfoo@latest redteam report   # opens the HTML report in a browser

redteam run generates adversarial probes tailored to the purpose description and then evaluates them against both targets. A finding is: a certifying "safe"/"non-toxic" sentence rendered in either language, an ungrounded (non-cited) factual claim, or a toxicity-adjacent answer missing the vet/poison-control routing notice — see purpose in the config for the full definition Promptfoo's grader is instructed to apply.

Why this isn't in the blocking CI gate

Generation and grading call an external Anthropic model and need ANTHROPIC_API_KEY — a live network dependency and a real (small) per-run cost. That is deliberately opt-in here, consistent with config/sprout.yaml's generation.provider: deterministic default (no network, no key) and the reasoning already recorded for Garak in docs/ROADMAP.md ("the offline deterministic default has no LLM to scan"). The default build has no LLM in the generation seat, but the HTTP target still exercises the full answer pipeline regardless of which generator backs it, so this red team is meaningful today, not only once the Bedrock/Anthropic seam is turned on.

The advisory redteam job in .github/workflows/ci.yml runs this on workflow_dispatch and is skipped (not failed) when secrets.ANTHROPIC_API_KEY isn't configured on the repo — mirroring the pa11y job's continue-on-error pattern. It is intentionally excluded from ci-gate's needs: list: wiring it in as a required, blocking check is a follow-up once the key is provisioned as a repo secret and a maintainer has watched it run clean at least once (that provisioning step is a one-time action only a repo admin can take, tracked in docs/ROADMAP.md rather than done silently here).

Extending

  • New attack classes: add plugin ids to redteam.plugins (see npx promptfoo@latest redteam plugins for the full catalog).
  • New languages: Sprout's own languages.supported in config/sprout.yaml is the source of truth; add a matching targets entry (language: <code>) and extend redteam.language.
  • Findings that turn out to be real gaps: add the newly-observed attack phrasing as a regression case in src/sprout/eval/suites/safety.py or refusal.py (the deterministic, merge-blocking suites) — Promptfoo red-teaming finds the gap, the eval suite is what holds the line on every subsequent PR, same division of labor the manual report already documents (§4 "Action" note).