Skip to content

Latest commit

 

History

History
440 lines (340 loc) · 19.2 KB

File metadata and controls

440 lines (340 loc) · 19.2 KB

Implemented use cases beyond one funder report

Planning date: 2026-07-22; implementation date: 2026-07-22

These are novel applications of this repository's trust chain, not claims of a new verification primitive. Each use case stays inside the product boundary: compute a bounded set of aggregate figures, attach receipts, enforce privacy, require human approval, and export evidence. None turns Outcome Receipts into a case-management system, data warehouse, or general BI product.

All six bounded workflows now have deterministic CLI implementations, passing and failing fixtures, and the versioned artifact envelope at docs/schema/workflow-artifact.schema.json. The last two remain experimental for real data: implementation is available for synthetic and controlled evaluation, while privacy and multi-organization evidence are still required before a production claim.

receipts verify-workflow --artifact PATH validates an artifact before a consumer interprets it. Version-1.0 examples for every kind are frozen under tests/fixtures/compat/v1/ and regenerated by make verify.

Commands

Use case Command Required human or policy input
Restatement receipts restate Verified prior bundle, reason, approver
Migration equivalence receipts migrate-check Reviewed before/after specs, approver
Requirement change receipts requirements-diff Stable requirement IDs
Contract evidence receipts contract-check Controlling text, citation, approver
Partner rollup receipts rollup Verified partner bundles, overlap declaration, policy ID, approver
Equity review receipts equity-review Allowlist, purpose, policy, consent basis, category provenance, approver

The detailed designs and evidence gates below remain the implementation and validation plan. Where the first plan proposed new report-spec sections, the shipped first cut uses explicit JSON input plans. This keeps existing report spec version 1.0 stable and makes operator-reviewed workflow input separate from metric SQL.

Operator input shapes

Contract thresholds and financial amounts are metric IDs, never numeric literals in the contract JSON:

{
  "contract_id": "housing-services",
  "controlling_text": "Operator transcription of contract section A.",
  "policy_citation": "Contract section A",
  "milestones": [{
    "milestone_id": "m1",
    "observed_metric_id": "people_served",
    "threshold_metric_id": "contract_threshold",
    "financial_metric_id": "payment_amount",
    "comparison": "gte"
  }]
}

A rollup plan points to partner configs and bundles so each aggregate can be re-derived and the complete export can be verified:

{
  "period": "2026-Q2",
  "population_overlap": "not_deduplicated",
  "suppression_policy_id": "cms-small-cell-v1",
  "inputs": [{
    "partner": "Partner A",
    "config": "partner-a/report.toml",
    "bundle": "partner-a/out",
    "metric_id": "people_served",
    "period": "2026-Q2",
    "suppression_policy_id": "cms-small-cell-v1"
  }, {
    "partner": "Partner B",
    "config": "partner-b/report.toml",
    "bundle": "partner-b/out",
    "metric_id": "people_served",
    "period": "2026-Q2",
    "suppression_policy_id": "cms-small-cell-v1"
  }]
}

An equity plan selects previously reviewed metric IDs. It does not accept SQL, column names, or free-form category values:

{
  "dimension": "operator-approved demographic grouping",
  "purpose": "Review access differences without causal inference.",
  "controlling_policy": "Named disclosure policy and source.",
  "consent_basis": "Documented program consent decision.",
  "category_provenance": "Program and privacy review record.",
  "groups": [
    {"label": "Group A", "metric_id": "outcome_group_a"},
    {"label": "Group B", "metric_id": "outcome_group_b"}
  ]
}

Priority and status

Order Use case Status Real-world release gate
1 Restatement and definition-change package Implemented Auditor workflow validation
2 Data-system migration equivalence check Implemented Second schema and organization evidence
3 Funder-requirement change impact review Implemented Grant-manager usability review
4 Contract milestone evidence package Implemented Contracts and finance review
5 Federated subrecipient rollup Experimental implementation Partner privacy review and two independent bundles
6 Suppression-aware equity slice review Experimental implementation Primary-policy, evaluator, and privacy review

No workflow adds a model feature. The last two require primary-policy review and real-user discovery before they are described as production-ready.

UC-1: Restatement and definition-change package

Job

A previously submitted number must be corrected, or a funder changes the definition of a metric. The organization needs to show what changed, why it changed, and which source slice produced each value without overwriting the original report.

Implementation

receipts restate takes --prior-config, --prior-bundle, --config, --reason, --approved-by, and --out. It verifies and re-runs the prior bundle, computes the current spec, compares the two manifests, and exports:

  • the prior and current displays, definitions, queries, row counts, and hashes;
  • a receipted delta computed by SQL, never arithmetic in prose;
  • a required reason and named approval;
  • the immutable prior bundle digest plus current spec and manifest digests.

The artifact records a typed supersedes relationship. It never deletes or rewrites the superseded ledger entry.

Fail-closed gates

  • The prior manifest must pass schema validation and bundle verification.
  • Metric identity must be explicit. A renamed metric is an add/remove pair unless a human supplies a mapping.
  • A changed definition or query is always material even if the value is equal.
  • A restatement with an unbound narrative, missing reason, or missing approval writes nothing.

Tests and acceptance

Passing fixtures cover changed data, changed SQL with the same value, and a definition-only correction. Failing fixtures cover a forged prior bundle, unsupported schema, ambiguous metric rename, ungrounded delta, and missing approval. Acceptance means an auditor can reproduce both values and the ledger shows the supersession without losing history.

UC-2: Data-system migration equivalence check

Job

An organization moves from one case-management export to another. Before cutover, it needs to know whether the new schema produces the same outcome figures under the same definitions.

Implementation

receipts migrate-check takes one reviewed report spec for each source. It computes each metric twice and emits a paired receipt plus a SQL-grounded delta. The report classifies each metric as equivalent, changed, or indeterminate; it does not assert that a changed value is wrong.

indeterminate is the classification for a metric small-cell suppression withholds on either side. Neither side publishes its value, so neither equivalence nor change can be asserted about it, and the record carries delta_status: "suppressed" in place of a delta receipt rather than a composed number derived from a cell the report declines to state. It is the same word receipts contract-check uses for the same reason. Classifying is what lets the command answer the question for the metrics it can compare; any real human-services export has at least one small cell, so aborting the artifact meant reporting nothing at all.

Reuse the mapping queue to propose field aliases, but require approval of every source-specific query before execution. Record source labels and spec hashes in the manifest without emitting client identifiers.

Fail-closed gates

  • Definitions must be byte-identical or explicitly approved as a definition change outside the equivalence result.
  • Duplicate keys, missing dates, and row-loss tolerances are author-declared data checks, not inferred defaults.
  • Empty, ambiguous, or unreviewed mappings block the entire equivalence claim.

Tests and acceptance

Use two synthetic schema variants with planted equal metrics, planted row loss, and an ambiguous destination field. Acceptance means every equality claim has two receipts, every difference is visible, and ambiguity cannot be labeled equivalent.

UC-3: Funder-requirement change impact review

Job

A funder publishes a revised reporting template. Staff need an inventory of which requirements are already computable, which definitions changed, and which new source fields are needed before the reporting deadline.

Implementation

receipts requirements-diff performs a deterministic requirement diff. It uses stable requirement IDs, canonical text hashes, and statuses: unchanged, definition_changed, added, removed, and blocked. Candidate SQL remains unexecuted until reviewed.

Export a planning artifact with no program figures. It may contain counts of requirements only if those counts are computed and receipted from the two requirement documents.

Fail-closed gates

  • Requirements without stable IDs enter manual matching.
  • Text similarity may order review candidates but cannot approve a match.
  • A changed denominator, period, population, or deduplication phrase is material.
  • No client data or source rows enter a model-assisted matching seam.

Tests and acceptance

Fixtures plant renamed, split, merged, and materially redefined requirements. Acceptance means no changed requirement is reported as unchanged and every automatic match remains pending human review.

UC-4: Contract milestone evidence package

Job

A government or foundation contract links payment to service milestones. Program and finance staff need one approval package connecting a receipted outcome figure to a receipted financial line and the controlling definition.

Implementation

receipts contract-check reads an operator-authored contract JSON document: contract ID, controlling text, policy citation, milestone IDs, observed metric IDs, threshold metric IDs, financial metric IDs, and comparison operators. The metrics remain ordinary report-spec metrics with row-backed receipts. The tool records evidence and explicitly records that no legal determination was made.

Fail-closed gates

  • Contract thresholds come only from a cited operator-supplied contract field.
  • Threshold comparisons are deterministic expressions with their own receipts.
  • Missing controlling text, policy citation, approval, or financial definition blocks export.
  • Client-level supporting documents remain outside the bundle.

Tests and acceptance

Fixtures cover met, unmet, and indeterminate milestones. Acceptance means indeterminate never collapses into unmet or met, all displayed amounts and thresholds bind to receipts, and finance can reproduce the package offline.

UC-5: Federated subrecipient rollup

Job

Several partner organizations report to a lead agency but cannot share client-level rows. The lead needs a combined result whose lineage reaches each partner's approved aggregate bundle.

Implementation

The implementation defines a receipt-composed rollup record whose inputs are verified partner bundle digests, metric definitions, periods, units, suppression policy IDs, and approved aggregate displays. receipts rollup accepts bundles, never CSVs, and computes rollup figures from compatible unsuppressed aggregates. The output records a Merkle-style ordered input digest and links to each partner manifest.

Do not add suppressed partner cells as if they were zero. If a safe combined value cannot be derived without revealing or estimating a partner cell, the rollup remains suppressed.

Fail-closed gates

  • Every input bundle and approval must verify.
  • Metric definitions, units, periods, and population overlap rules must match.
  • Duplicate-client risk across partners must be resolved by an operator-supplied non-client-level method or the metric is labeled non-deduplicated.
  • A plan declaring disjoint populations is rejected when two partner receipts carry the same non-empty slice hash. Equal slice hashes mean the same rows were counted twice, which a disjoint population cannot produce, so the lead agency can falsify the declaration without holding a client row. A zero-row slice hashes to the canonical empty value for every partner, so two partners both reporting a true zero are not a collision; a receipt that reports a non-zero count while carrying that same empty value can be compared against no one, so it is rejected rather than exempted. A plan already labeled not_deduplicated keeps its label. Only a byte-identical slice is falsifiable this way: the same people recorded under different identifiers or columns, and every partial overlap, remain operator declarations. ADR docs/adr/0004-fail-closed-disjoint-rollup-slice-check.md records the decision and the residual risk.
  • The most protective input suppression policy controls unless a primary policy source establishes another rule.

Tests and acceptance

Adversarial fixtures cover one forged bundle, incompatible definitions, recoverable partner cells, overlapping populations, and reordered inputs. Acceptance requires no path from the rollup artifacts to a partner's suppressed cell and deterministic output independent of input file order.

tests/test_rollup_adversarial.py holds that fixture set: a re-sealed bundle with an inflated receipt and a bundle with a swapped narrative, mismatched definitions, periods and suppression policies, an undeclared overlap value, a suppressed partner cell, identical partner slices under both overlap declarations, two partners who both report a true zero, a partner whose non-zero count is computed over an empty slice, one partner submitting the same rows under two bundles, and every ordering of three partners. Two assertions carry the acceptance property: the artifact republishes no partner value, row count, or slice hash, and all six orderings produce the same artifact apart from the digest of the plan file itself. Independently rebuilt partner fixtures reproduce the artifact byte for byte.

UC-6: Suppression-aware equity slice review

Job

An evaluator wants to see whether an outcome differs across operator-approved groups while avoiding publication of small or recoverable intersections.

Implementation

receipts equity-review reads an explicit allowlist with purpose, controlling disclosure policy, consent basis, category provenance, and reviewed metric IDs. Subgroup metrics remain ordinary SQL-backed report figures and suppression runs over the whole report before the selected receipts enter the artifact. The artifact carries interpretation limits and does not rank groups, infer causality, or generate a fairness conclusion.

Fail-closed gates

  • Free-form slicing and high-cardinality identifiers are prohibited.
  • Missing consent, category provenance, or controlling policy blocks the slice.
  • Complementary suppression analyzes totals, margins, percentages, deltas, and cross-table recovery together.
  • The optional drafter cannot receive protected-category labels or unsuppressed small aggregates without a separately reviewed provider policy.

Tests and acceptance

Use planted intersection attacks and category sparsity. Acceptance requires exhaustive recovery tests over the complete artifact set, review by a privacy specialist, and a real evaluator confirming that definitions and caveats prevent overinterpretation.

Shared architecture work

The use cases need a small set of common capabilities:

  1. Add a stable spec_hash and bundle_digest reference to every derived artifact.
  2. Generalize manifest diff into typed change records without changing existing CLI output.
  3. Add receipt composition where an aggregate is computed from verified manifests rather than rows. Keep row-backed and receipt-backed provenance visibly distinct.
  4. Extend the ledger with typed relationships such as supersedes, compares_to, and rolls_up, preserving the existing hash chain.
  5. Version each new schema from its first release and publish JSON Schema beside the report-spec and receipts schemas.

No shared capability may weaken the existing gate. A composed receipt is not a substitute for a row-backed receipt; it is a separate claim whose inputs must all verify.

Completed delivery sequence

Wave 1: replay and comparison

UC-1 and UC-2 ship in the deterministic workflow module. ADR 0003 records supersession, paired-source identity, and receipt composition. identity. Add the typed diff model, ledger relationships, CLI commands, passing and failing synthetic fixtures, docs, and migration notes.

Exit gate: full verification remains green, both commands are deterministic under --reproducible, and no prior manifest or ledger format breaks.

Wave 2: requirement and contract workflows

UC-3 and UC-4 ship with operator-authored inputs and no automated legal interpretation. Grant-manager and contracts-reviewer interviews remain evidence gates.

Exit gate: reviewers complete the workflow without editing JSON, and ambiguous requirements or milestones remain visibly blocked.

Wave 3: aggregate composition

UC-5 ships as an experimental workflow under artifact schema 1.0. Privacy and subrecipient-user review remain required before accepting real partner bundles.

Exit gate: adversarial recovery tests pass, incompatible inputs fail closed, and at least two independent partner fixtures reproduce the same rollup.

Status on 2026-08-04: the three testable conditions are met by tests/test_rollup_adversarial.py, which also closed the one hole the fixture set found. A plan could declare disjoint partner populations while two partners submitted identical rows, and the rollup summed them into a combined figure larger than the number of people served. The rollup now fails closed on that declaration. Review of the gate itself found a second way past it: a receipt whose non-zero count is computed over an empty slice carries the empty-slice hash every partner shares, and the exemption for that hash let it skip the check entirely. The exemption is now keyed on the receipt reporting nothing counted, and a non-zero count over an empty slice is refused. The privacy-specialist and subrecipient reviews are unchanged and still block a production claim; they need people, not tests.

Wave 4: equity slices

UC-6 ships as a research-gated experimental workflow. Confirm the controlling disclosure policy from a primary source for a specific report and obtain real evaluator and privacy review. Do not ship a generic subgroup explorer.

Exit gate: the policy, purpose, category provenance, consent decision, recovery analysis, and human interpretation limits are all recorded in the bundle.

Measures

Track outcomes that test the workflow rather than repository activity:

  • time to produce and verify a restatement;
  • share of migration metrics classified without manual spreadsheet comparison;
  • requirement changes correctly routed to human review;
  • blocked or indeterminate contract claims that remain blocked;
  • rollup inputs verified and incompatible inputs rejected;
  • privacy attacks detected before an equity artifact is written.

The invariant metric remains unchanged: zero unbound numeric spans may survive to an exported artifact.