Every field this project classifies has documentation published by the agency that collects it. This directory keeps that documentation as it was published, so that when a reader asks what IPEDS means by a net price calculator, the answer is the words IPEDS used and not a paraphrase that sounds like them.
| File | What it is |
|---|---|
manifest.json |
One entry per document: publisher, title, URL, retrieval date (UTC), byte count and SHA-256 of the bytes as fetched, and byte count and SHA-256 of the bytes as stored. |
raw/ |
The bytes that arrived, committed so the hashes can be re-checked. One exception, stated in the manifest as stored: without-scripts: the Scorecard glossary page is a rendered single-page application whose script bundle embeds the website's own client-side keys and analytics tags, so it is stored with every <script> element removed. The glossary prose is server-rendered and survives; the as-fetched hash is still recorded so a re-fetch can be compared. |
passages.json |
The quotable units extracted from raw/ by disclosed.ask.corpus.extract: one per glossary entry or dictionary variable, with a locator in the document's own terms. Regenerated by disclosed corpus; the test suite re-runs the extraction and fails if the committed file differs. |
Which passage defines which graded field is written down in
src/disclosed/ask/definitions.py, with a distinction the corpus itself forces: a passage either
defines the exact variable behind a field, or is related (the same concept, as the publisher
presents it to the public, through a different variable). The Scorecard glossary's "Graduation
Rate" is the website's eight-year measure; this project grades C150_4. Both are quoted when
asked, and the difference is stated.
Quotes are verified verbatim (whitespace collapsed, nothing else) against passages.json before
they are shown. A quote that does not verify is withheld.
To refresh: disclosed corpus --fetch. It rewrites the manifest's dates and hashes, so the diff
shows exactly what the publisher changed.