Project-specific findings under the portfolio Responsible-Tech Framework. This artifact is reviewed on release; generic thresholds remain in the portfolio standards. Last reviewed: 2026-08-14 (initial).
- A Ethics: applies. The harm surface is false assurance about a security authorization.
- B Bias: minimal surface; the tool applies published structural rules uniformly to every document and does not rank, score, or profile. It grades documents, never the organizations that publish them.
- C Privacy / DPIA: applies in a narrow form. The default command processes only the local files it is pointed at, entirely in process, with no network calls, no telemetry, and no persistence beyond its printed report. The four opt-in commands of ADR-0005 send the findings, passages of NIST's published text, and excerpts of the document around each finding to a model provider; see section H below.
- D Transparency: applies and is the design center: every finding carries the rule citation, source URL, and retrieval date it enforces, and every unevaluated rule is listed.
- E Accessibility: N/A today; no graphical or web surface. Output is plain
text (screen-reader-friendly terminal output) plus
--format json. - F Security: applies; see
SECURITY.md. Input is untrusted JSON. - G Effect on third parties: applies to the survey harness only, which fetches other people's servers.
- AI Evaluation: applies, since ADR-0005, to the four opt-in model-backed
commands. The validator itself still has no model call. The evaluation
harness and its results are in
evals/anddocs/evals/README.md, and section H below is the audit of the new surface. AI-assisted authoring of the code is disclosed in the README.
What could go wrong? The documents this tool reads support decisions about whether a system is allowed to operate. The dangerous failure is not a spurious error, which a human investigates, but a clean report that a reader takes as evidence of compliance. Three specific ways that could happen:
(a) A reader takes "no ERROR findings" as "this package conforms to OSCAL" when the tool evaluates 78 of NIST's 340 published constraints. (b) A reader takes structural conformance as evidence that a control is implemented. (c) A reference that could not be checked, because the imported document was not supplied, is read as one that was checked and passed.
Controls. The README opens with the structural-conformance boundary stated
before anything else, and the Limits section enumerates every check the tool
does not perform. The severity contract makes UNVERIFIABLE a first-class
outcome that is never rendered as a pass (ADR-0002), and every run emits
UNVERIFIABLE findings naming what was not checked and how much of it there was,
including runs with no other findings. docs/CONSTRAINT-COVERAGE.md lists all
340 published constraints with whether each is evaluated, is generated from the
vendored files, and is guarded by a test so it cannot drift.
tests/test_break_the_gate.py proves each claimed check actually catches its
target. SECURITY.md classifies both a false clean report and an UNVERIFIABLE
collapsed into a pass as security bugs.
Review gate. Any change to severity semantics, or to what a check claims, requires an ADR and a README scope update in the same PR.
What could go wrong? A tool that reports defects in other organizations' published files can slide into a scoreboard, or into a claim about the security of the systems those files describe.
Controls. Findings are stated as conformance of a published document to the
published specification, on a stated date, and nothing more. The survey records
metadata and finding codes only, never a value read from anyone's document.
This project does not open issues or pull requests on NIST, FedRAMP, or any
vendor's repositories, and does not characterize anyone's software; that rule is
written into CONTRIBUTING.md. Where a finding could be read as a claim about a
tool that produced a document, the write-up describes the document instead.
Accepted cost. Some findings are less satisfying to read because they stop short of naming a cause. That is the correct place to stop.
The rules come verbatim from NIST's published schema, constraint layer, and documentation; the tool adds no discretionary judgment per document. Constraint severity is NIST's declared level rather than one this tool assigns. Where the tool makes an interpretive choice, there is exactly one, it is documented in the README's Limits section and in the code, and it is the reading under which NIST's own published catalogs are not spuriously invalid.
No collection, no transmission, no retention. The DPIA-style answer is short
because the data flow is short: files in, findings out, process ends. This
matters more than usual here, because system security plans and assessment
results describe live systems' boundaries, components, and unremediated
weaknesses. Nothing leaves the machine the tool runs on, and there is no code
path that could send it anywhere. Findings quote document values back to the
operator, which is necessary for a usable report; operators handling sensitive
packages should treat the report with the same care as the package.
SECURITY.md asks reporters never to attach a real SSP, SAR, or POA&M.
The methodology, severity definitions, scope limits, and the one interpretive
choice are documented in the README. The vendored sources and their hashes are
in src/oscal_validate/vendor/SOURCES.md. A finding without a citation cannot
be constructed in the code path: the Finding model requires a Rule.
See SECURITY.md for the threat cases and the commitments. The validator runs
offline by design, which removes the largest attack classes, and that is
enforced by a test that removes socket and by a test that reads the package
source for network imports. The residual surface is the JSON parser, the
vendored XML parse (hash-checked input, never user-supplied), and the supply
chain of the dev toolchain. Documents nested past 200 levels are refused rather
than read partially.
What could go wrong? A tool that fetches other people's repositories can cost them money and attention: ignoring robots.txt, hammering a host, hiding behind a browser's User-Agent, or wandering off the documents it was pointed at.
Controls. robots.txt is fetched before anything else and obeyed, with a
Disallow as a hard stop and no flag to override it; an unreachable robots.txt is
treated as a complete disallow per RFC 9309 2.3.1.4 rather than as permission. A
blocked host is recorded in the evidence and skipped, never circumvented. The
User-Agent carries the product token and a link to this repository. One
invocation fetches robots.txt plus the documents on its target list, with a
two-second minimum interval per host that a declared Crawl-delay can lengthen.
Redirects are capped at five with robots re-checked at every hop. Every one of
these is tested against a server on localhost in tests/test_survey_fetch.py,
including the case where the document is never requested because robots.txt said
not to. Fetched bytes are cached so a re-run needs no network at all.
Residual risk. The operator chooses the target list, and a tool cannot know whether they had a reason to. What it can do is behave the same way whoever is driving it, and leave a log entry that says who it was.
What could go wrong? The harm in section A, false assurance, has a new route: a language model that reads a structural finding and writes back that a control is implemented, a system is secure, or a package would be authorized. Three further failures are specific to the model: an invented quotation presented as NIST's definition; a finding described that the validator never produced, or one it did produce left out; and a proposed edit that silently breaks something else.
Controls, each enforced in code rather than by prompt wording alone.
- The validator decides. Every model-backed command starts from the deterministic validator's findings list and works only from it. The model never assigns a severity, never adds a finding, never removes one. The walkthrough checker strikes any label the validator did not produce and appends any group the narrative omitted, with the full index printed last.
- NIST's text is the only evidence. The corpus is committed and hash-pinned
(
docs/data/nist-documentation-corpus.md). Every quote is looked up verbatim in the named source before display; every sentence-shaped quotation written inline is looked up across all sources; what is not found is withheld and counted. Measured on twelve NIST documents: 61 verified, 20 withheld, every withheld one naming a source outside the corpus. - Repairs are verified, not claimed. A draft patch is re-validated by the same validator before it is shown, and the report is what that run found. The original file is never written. Measured: 59 of 62 resolved with nothing introduced anywhere.
- The boundary is guarded twice. The system prompt refuses implementation, security, and authorization questions; a deterministic sentence-level guard then withholds any such judgment that would be shown and prints how many it withheld. Measured on 100 adversarial phrasings: 80 of 80 refuse-cases held by both the guard and an independent judge call, 80 of 80 explicit refusals, 20 of 20 structural questions answered. The guard errs toward withholding and sometimes removes a sentence that was not a judgment; that is the chosen direction of error.
- Honest refusals. A document the validator cannot parse is refused before any model call; a finding whose rule is tool policy is never attributed to NIST; a document declaring another OSCAL version carries the issue #8 note; an unparseable reply shows nothing.
- Opt-in, labeled, separate. The commands are reached only by name, import the SDK lazily, need an optional extra, and print in their first line that what follows is AI-generated and what was checked before it was shown. The default command's bytes are pinned to goldens captured before any of this existed.
Residual risks, stated.
- A document sent through these commands leaves the machine and is subject to the model provider's handling. An SSP can describe a real system in detail. The README says so beside the commands; nothing in this repository can make that decision for an operator, and the GitHub Action never runs these commands.
- The guard is lexical. A judgment phrased in a way no pattern matches would be shown. The judge call in the boundary suite is the check on that, and on the last run it found none; it is the same model family as the one under test, which the write-up says.
- The numbers are for Amazon Bedrock
claude-sonnet-4-6, not for the code defaultclaude-sonnet-5, which the account that ran them could not invoke. A change of model or of prompt wording (PROMPT_VERSION) is a change to what was measured, and the results files say which they measured.