Skip to content

CHANGELOG.md

Latest commit

 

History

History
314 lines (302 loc) · 19.9 KB

CHANGELOG.md

File metadata and controls

314 lines (302 loc) · 19.9 KB
Raw

Changelog

All notable changes will be documented here.

[Unreleased]

Added

  • .github/workflows/codeql.yml: a CodeQL actions-language scan of every workflow file, generally available since 2025-04-22. It is a second, independent engine from zizmor (different rule set, results in the Security tab) and is now a required status check alongside verify.
  • Portfolio standards conformance documentation: a Standards Conformance table and an AI-assisted development disclosure in the README, a roadmap with the milestone gates and the Quality & Metrics ledger (docs/ROADMAP.md), and a reasoned internationalization N/A declaration (docs/I18N.md).
  • .github/allowed_signers, the SSH allowed-signers file the release workflow uses to verify a signed annotated release tag.
  • docs/RELEASE.md (release posture and the publication-blocking checklist) and docs/INCIDENT-RESPONSE.md (the incident process the Standards Conformance table already cited) are now committed and linked from the README's Documentation list.
  • compare_snapshots and verify_comparison_document now load the packaged receipt-comparison-v0.1.schema.json at runtime and validate every comparison document against it, matching the self-check pattern civicrm_target_canary.py already used for its own result schemas. The public schema previously validated only in tests, against the repo copy; the installed package never opened it.
  • examples/directus-11.17.4-civic-case/README.md now documents the adversarial derivative the top-level README cites: the six mutations, that row and file counts are preserved by construction, that the derivative is never committed (generated per run into a TemporaryDirectory), the per-dimension observed-loss-signal table, and a claim-limits paragraph scoped to the derivative (issue #32). Matches the shape the CiviCRM example README already used.
  • scripts/check_browser_capture_bindings.mjs: a real, offline reproducibility gate for issue #31. Nothing previously bound the nine committed browser-*.json files to the four civicrm_browser_*.mjs scripts that produced them; make lint-lab only parses them. Each of those four scripts writes a hardcoded literal unconditionally once every live browser/DOM assertion above it passes, so the committed file should always equal that literal. This gate extracts the literal directly from each script's source and requires canonical equality with the corresponding committed file -- catching either side drifting from the other, without a live CiviCRM, Playwright, or Docker. It explicitly excludes the handful of fields only a live page can produce (axe-core's rule counts and version, one measured keyboard tab-count) rather than silently trusting them; those are listed by name in the script. Verified against three real scenarios: a mutated script (caught), a hand-edited committed file (caught), and a change to only a live-only field (correctly not flagged, since that case is out of this gate's scope). Wired into tests/test_gates.py (skips cleanly without Node, like the existing lab-syntax gate) and into make demo-civicrm-target-canary, both of which run in the required verify CI job.
  • examples/civicrm-6.16.2-target-roundtrip/README.md: a new "Recapturing this profile" section documents the actual manual re-capture procedure (the civicrm_target_roundtrip_lab.mjs orchestrator's CLI entry point and prerequisites) and reports, honestly, that a real attempt at it during this work completed CiviCRM provisioning but failed at the first browser step on a 15-second visibility-wait timeout roughly four minutes in -- a harness reliability question, not evidence against the scripts' determinism once a run completes. No automated or scheduled live recapture exists; this stays a documented manual procedure, matching the project's paused feature scope.

Fixed

  • build_directus_lossy_canary.py's six adversarial-mutation labels were a separately maintained constant, never re-derived from the mutations the script actually applies (issue #32). _mutate now returns the label for each mutation as it applies it, and that returned list -- not a parallel hardcoded one -- is what the derivative's adversarial-derivative.json statement declares. check_directus_canary_demo.py now also asserts the exact mutation list on every run, so a future edit that adds, drops, or reorders a mutation without updating the label is caught immediately.

  • The top-level README said the Directus derivative produces "six declared loss signals"; the evaluator observes them, the six mutation labels are what is declared, and those are two different sixes that happen to match by arithmetic today, not by construction (issue #32). Now reads "six observed loss signals."

  • The wheel force-included 25 schemas; 12 were never loaded by any code (issue #33): the six superseded civicrm-evidence-index versions (only v0.7 is read), and all six civicrm-evidence-verification versions (nothing loads that family at all). pyproject.toml's force-include block is trimmed to the 13 schemas src/exitdrill/ actually references. scripts/check_wheel.py's committed_schemas now derives the required set from a scan of src/exitdrill/ for schema-filename literals instead of globbing every file under schemas/, so the gate enforces "referenced by real code," not "exists in the tree." receipt-comparison-v0.1 is the one schema that moved from "referenced by tests only" to "referenced by real code" rather than being dropped -- see the runtime self-check added above. New tests in tests/test_gates.py assert the force-include table matches exactly what source code references, and pin the concrete regression: a superseded schema that still exists on disk must not ship even if it sneaks back into the packaged entries. The 12 superseded files stay in schemas/ and git history; nothing is deleted.

  • Thirteen paths — twelve documents and docs/adr/ — were hidden from every git status by .git/info/exclude, a per-clone file that is never pushed and was itself untracked. Nothing in the repository recorded that these paths existed or why they were unpublished, and because docs/adr/ was one of them, a new ADR written there (the portfolio's canonical discovery path per docs/adr/0000-record-architecture-decisions.md) would silently fail to appear in git status or git add. Each of the twelve documents was read in full and judged individually. Two were purely technical and public-appropriate with no existing committed equivalent (docs/RELEASE.md, docs/INCIDENT-RESPONSE.md) and are now committed. Two were stale, non-sensitive duplicates already fully superseded by a committed doc (docs/I18N.LOCAL-DRAFT.md by docs/I18N.md; docs/OBSERVABILITY.md by the Observability scope section of docs/ROADMAP.md) and stay out to avoid two disagreeing sources of truth. Eight were private product-strategy, competitive-intelligence, brand, or legal material — buyer/kill-gate economics, a rejected private codename and competitor scan, brand-positioning strategy, a trademark clearance memo, and customer-discovery scripts (docs/PRD.md, docs/RESEARCH.md, docs/NAMING.md, docs/NAMING-CLEARANCE.md, docs/RED-TEAM.md, docs/DISCOVERY-PACK.md, docs/USER-RESEARCH.md, docs/ROADMAP.LOCAL-DRAFT.md) — and stay out. .git/info/exclude is emptied back to the git default template, the decision to keep the eight private and two superseded documents unpublished is now recorded in a tracked, commented .gitignore, and docs/adr/ is no longer excluded anywhere so a future ADR left there is visible instead of vanishing. CONTRIBUTING.md no longer directs contributors to read "the PRD," a file that would not exist in a fresh public clone.

  • The attachments dimension no longer hides one class of loss behind another. An exported attachment can fail byte verification, be refused by the reference model's foreign key, or both, and those are disjoint populations. The evaluator reported max() of the two population sizes, so whichever set was smaller became invisible: an export carrying an unrestorable attachment could newly corrupt a restorable attachment's bytes and still publish an identical invalid_count, identical restored_count, and identical observed_remediation_signals. exitdrill compare --fail-on-loss-signal-increase then exited 0 and recorded no_observed_loss_signal_change for attachments — a silent-loss false negative in the one dimension the tool exists to watch. The evaluator now tracks which attachments failed each check and reports the size of their union, so overlapping failures still count once while disjoint ones both count. The restoration shortfall remains a fail-closed floor. No other dimension was affected: each has only one reachable failure mode, so max() was already exact there. Every existing fixture, demo, and canary summary is unchanged.

  • Every command that reads uv.lock now uses --locked instead of --frozen, so a pyproject.toml dependency change that was never relocked fails the build instead of passing it. uv sync --frozen and uv export --frozen install and export whatever the lockfile already says and exit 0 on a drifted lockfile, which meant a newly declared runtime dependency was neither installed for the merge gate nor present in the requirement set handed to pip-audit. Two gates cover the change: one asserts the real flag behaviour against uv, and one keeps --frozen out of the Makefile and both workflows.

  • The offline CiviCRM acceptance gate now requires the empty-target precondition control to be rejected for that precondition. It previously accepted any CiviCRMTargetCanaryError, so a derivative that stopped exercising the precondition but broke in some unrelated way still counted toward adversarial_controls_detected.

Changed

  • The release workflow is now dispatch-only and split-authority: a shared read-only authorization job verifies a signed annotated tag against trusted main, the build job re-runs make verify and both declared demo outcomes at the verified commit before packaging, and a checkout-free publish job rechecks the immutable tag object before creating the GitHub Release. It replaces the tag-push candidate build, which had never run because no tag exists; there is still no tag, no release, and no package-registry publication.
  • The wheel-content gate now derives its expected schema set from the committed schemas/ directory instead of a hand-maintained constant list. It requires the wheel to carry exactly that set, byte for byte, and rejects an unexpected packaged schema. Each schema stays pinned to exactly one $id, as before: the two schemas published under the repository URL keep that form, and every other schema must use https://exitdrill.example/schemas/<name>, so a schema added later is $id-pinned without editing the gate.
  • Strict mypy now covers scripts/ as well as src/ and tests/, so the offline acceptance gates, fixture builders, and the wheel checker are type checked like the package.
  • CI syntax-checks every committed browser-lab script through a new make lint-lab target instead of two individually named files.
  • Advanced the CiviCRM evidence index to v0.7 and verification result to v0.6 for a twelfth indexed artifact: a bounded case-search failure result.
  • Advanced the CiviCRM evidence index to v0.6 and verification result to v0.5 for an eleventh indexed artifact: a same-object browser allow control.
  • Advanced the CiviCRM evidence index to v0.5 and the closed verification result to v0.4 for a tenth indexed artifact: a separate authenticated browser access-denial result.
  • Advanced the CiviCRM evidence index to v0.4 and the closed verification result to v0.3 for a ninth indexed artifact: a separate target-generated case-client browser-workflow result.
  • Advanced the CiviCRM evidence index to v0.3 and the closed verification result to v0.2 for an eighth indexed artifact: a separate bounded contact-summary browser-workflow result.
  • Advanced the CiviCRM evidence index to v0.2, binding each catalog entry to the exact emitted artifact bytes and length while explicitly withholding any authenticity claim.
  • Advanced the independent baseline to v0.3; every declared required entity field now binds an exact expected scalar value as well as its type.
  • Advanced drill-result and receipt contracts to v0.3 so their limitations accurately state that field-value equivalence is bounded to declared required fields.
  • Advanced the independent baseline to v0.2, binding audit action and occurrence time as well as event identity.
  • Renamed the aggregate remediation field to observed_remediation_signals; it no longer implies a minimum task count.
  • Advanced result and receipt contracts to v0.2 for the closed semantic payload.

Added

  • A top-level --version flag that reports the installed exitdrill package version and exits before any subcommand is required, so an operator can confirm what they installed without running a drill.
  • Gate-completeness regression tests: the merge gate now fails when a committed schema is missing from the wheel force-include map, when a committed schema departs from its pinned $id, including by adopting the other published form, when a packaged schema is not a JSON object, when a committed browser-lab script escapes the syntax gate or fails node --check, and when the lab or type gates go back to naming individual files.
  • A negative test for the synthetic-demo summary parser, which now rejects a receipt that parses as JSON but is not an object.
  • A make lint-lab target that syntax-checks every committed browser-lab script and fails when none is found.
  • An eleventh CiviCRM evidence family that observes both synthetic cases through Case Summary, then records HTTP 500 from one exact-subject filter submission without claiming root cause or general search behavior.
  • A tenth CiviCRM evidence family that confirms the allow principal can render the same protected Contact Summary used by the browser denial probe.
  • A ninth CiviCRM evidence family that records one deny-principal browser redirect and protected-content absence while withholding universal UI/API authorization and principal-equivalence claims.
  • An eighth CiviCRM evidence family that follows the target-generated case client through Contact Summary and Cases back into Manage Case, while explicitly withholding source case-client equivalence, editing, accessibility, and operational-equivalence claims.
  • A seventh CiviCRM evidence family that reopens the case dashboard, follows the exact synthetic contact into Contact Summary, verifies the contact-page region and Cases affordance, and explicitly withholds contact-editing, case-navigation, accessibility, and operational-equivalence claims.
  • A fail-closed verify-civicrm-evidence-index command that checks the exact v0.2 catalog, bounded artifact bytes, packaged result schemas, normalized export contract, and declared attachment bytes without producing a composite or structural verdict, then emits a separate closed v0.1 verification result with machine-readable limitations.
  • A closed evidence-index.json catalog for the normalized CiviCRM export and six independent result artifacts, with per-entry schemas and decision scopes but no composite status, score, pass count, or inferred conclusion.
  • A sixth CiviCRM evidence family that follows a supported read-only activity View action, verifies one generated Open Case activity's exact bounded markers, and records its additional known runtime error without relabeling target scaffolding as restored source history.
  • A fifth CiviCRM evidence family that records one bounded keyboard interaction: the Roles disclosure is reached after 69 Tab presses, closes with Enter, and reopens with Space, without claiming complete keyboard accessibility.
  • A fourth CiviCRM evidence family: pinned axe-core 4.12.1 scans the isolated synthetic Manage Case document, retains only aggregate rule counts and sanitized violation IDs/impacts/node counts, reports two serious findings, and explicitly does not establish WCAG conformance.
  • Canonical docs/adr/ compatibility index linking all accepted ADRs without moving or rewriting their durable docs/decisions/ history.
  • A real-process, synthetic-only Directus 11.17.4 API-response canary captured from a pinned local sandbox, with schema, content, relationships, attachment bytes, permissions, activity, and a closed hash manifest.
  • A bounded source-specific Directus normalizer that verifies the custom capture bundle and atomically emits the existing normalized export and attachment contracts without entering the evaluator's trust algebra.
  • A one-command Directus-canary acceptance demonstration: deterministic normalization, clean replay, an equal-row-and-file-count six-mutation derivative, exact five-dimension loss assertions, aggregate-only reports, receipt comparison, and comparison-policy exit verification.
  • Fail-closed detection and replay evidence for same-type critical-field value loss without placing raw field values in aggregate receipts.
  • Strict independent baseline and normalized-export contracts.
  • Structural comparison across entities, relationships, attachments, permissions, and audit history.
  • Neutral foreign-key-enforced reference restore.
  • Aggregate deterministic receipt and offline replay.
  • Synthetic CRM/case-management demonstration and adversarial fault suite.
  • Shared bounded JSON decoder with duplicate-key, non-finite-number, invalid UTF-8, and excessive-nesting rejection.
  • Exact closed receipt and untrusted-envelope field validation.
  • Descriptor-stable attachment size checks and hashing.
  • Pinned CI, security, packaging, and build-only release-candidate workflows.
  • Project conformance, data-governance, incident-response, observability, release, and internationalization declarations.
  • Cumulative 128 MiB attachment hashing budget.
  • Per-row SQLite restore with read-back counts and foreign-key check.
  • Offset-aware timestamp and chronology validation.
  • Closed receipt payload/dimension arithmetic and result-algebra validation.
  • Collision-resistant atomic receipt writing.
  • Synthetic-only target-exercise preflight plan and five-probe validator.
  • PEP 561 typing marker and local wheel-content gate.
  • Deterministic offline comparison of two validated receipts with exact scope comparability gates, per-dimension signed count deltas, nonordinal status and extra-record transitions, and separate observed loss-signal directions.
  • Closed JSON Schema for receipt-comparison output and explicit duplicate, incomparable, uncertain, mixed, and no-observed-loss-signal states.
  • Wheel-packaged comparison schema with Draft 2020-12 semantic consistency tests and explicit export-generation/evaluator-version limitation.
  • Opt-in comparison CI policy exit 3 for directly observed missing/invalid increases, without changing JSON or ranking statuses, extras, or totals.
  • JSON regular-file and 200,000-node bounds, attachment size-change detection, and descriptor-relative parent traversal on supported platforms.
  • Stronger comparison-schema consistency for scope reasons, observed signal direction, assessments, status transitions, and extra-count transitions.
  • CI demonstration of ordinary comparison and the expected opt-in policy exit.
  • Pre-write semantic verification and encoded 2 MiB receipt bound before any output-directory or temporary-file mutation.
  • Installed source-bound comparison verifier that recomputes every derived field from two fully verified receipts; JSON Schema remains the structural and locally expressible contract.
  • Deterministic, accessible, script-free offline HTML evidence reports generated only from semantically verified aggregate receipts, with mandatory trust limitations and no operational-equivalence claim.