Skip to content

adapting.md

Latest commit

 

History

History
81 lines (66 loc) · 4.31 KB

adapting.md

File metadata and controls

81 lines (66 loc) · 4.31 KB
Raw

Adapting this harness to another domain

The harness generalizes to any assistant that answers questions from published policy documents: benefits eligibility, licensing rules, housing programs. This page lists what changes and what carries over unchanged.

Two ways to use it, in increasing order of commitment:

  • Read this page and hand-copy. Fine for a one-off experiment.
  • Run make template TARGET=../new-domain-assistant. Generates a starter skeleton in an empty directory, built from template/MANIFEST.yaml — the same claim this page makes, but as data a script and a test both check against the actual repo tree, so it can't silently drift the way prose can (see docs/ROADMAP.md P3-5, "Generalize the harness"). It copies the domain-agnostic modules verbatim, flags the handful that need a domain-specific edit (each with a marker to grep for), writes a stubbed src/assistant/domain.py, and drops a GETTING_STARTED.md in the new tree pointing back at the checklist below. It does not touch corpus content, eval case content, or prompts — see items 1-4.

What carries over unchanged

  • The runner, deterministic check framework, judge plumbing, report generator, regression gate, and CI wiring (evals/).
  • The guard architecture: input checks before retrieval, output checks that block and substitute rather than merely log (src/assistant/guards.py).
  • The corpus discipline: a manifest with URLs, fetch dates, hashes, and license notes; committed snapshots; "as of" disclosure in every answer.

What you change

  1. The domain profile. The transit-specific knobs are isolated in one object, DomainProfile in src/assistant/domain.py: the scopes (agencies), the aliases users type for them, the adjacent topics to redirect, and the fallback contact. A new domain writes a new profile and registers it; the retrieval, guard, and config code reads the active profile unchanged. The test_a_new_domain_is_just_a_new_profile case shows a housing-voucher profile reusing the whole pipeline. The active profile is late-bound: retrieve, guards, and config read it at call time, not import time, so FPA_DOMAIN may be set any time before a request is handled and the switch takes effect immediately (default_retriever()'s cache is keyed on the profile, so it switches too). What is deliberately not in the profile, because it is cross-domain safety rather than domain content, is the PII, injection, and eligibility-determination detectors in guards.py; those bind in every domain.

  2. Corpus manifest. Point corpus/manifest.yaml at your documents. Check robots.txt and content signals; record your reading of them in the manifest, not just in your head. Re-run make fetch && make ingest.

  3. The will-not-do list. Decide what your assistant must never do (for a benefits assistant: determine eligibility, advise on appeals, handle case numbers). Encode each rule three times: in the system prompt, in guards.py, and as eval cases. The repetition is the design.

  4. Forbidden-language patterns. The determination-language detector is a phrase list with hedge awareness. Rewrite the phrases for your domain ("you are approved", "your claim will succeed") in every language you serve.

  5. Eval cases. Author cases from your actual documents, during or right after ingest, while the boundary conditions are in front of you. The pattern to copy from evals/suites/:

    • groundedness: facts a reader can verify against a named passage;
    • refusal: PII, injection, determination-seeking, out-of-corpus topics;
    • edge cases: the boundaries your documents actually publish (ages, income cutoffs, document alternatives, what stacks with what);
    • multilingual: mirrored cases so parity is a number, not a hope;
    • freshness: expired programs and "as of" behavior.

  6. Agency/entity aliases. Set on the DomainProfile (item 0): whatever your users call the programs or offices in your corpus.

The two habits that matter

Commit your first bad scoreboard. The improvement curve across commits is evidence your evals connect to reality.

Change prompts only against failing cases. Every prompt edit should name the case IDs it is trying to fix, and the regression gate catches what it broke.