Skip to content

I18N.md

Latest commit

 

History

History
205 lines (171 loc) · 12.5 KB

I18N.md

File metadata and controls

205 lines (171 loc) · 12.5 KB
Raw

Internationalization & localization status: gettext (EN/ES)

nearmiss renders end-user-facing text in English and Spanish through GNU gettext message catalogs, per STANDARDS/INTERNATIONALIZATION-STANDARD.md §3/§4. This replaces the previous bespoke _EN/_ES Python dicts (which had no extraction tooling, no plural handling, and no key-parity check).

Declared: 2026-06-30 · Reviewer: nearmiss maintainers

What is (and is not) translated

Translated — the advocacy brief only. The brief is nearmiss's single end-user-facing, natural-language surface (city councils, advocates, residents). Every fixed string in it is wrapped in gettext _() / ngettext() and rendered in the requested language:

nearmiss brief --config config/davis-demo.toml --lang es
nearmiss run   --config config/davis-demo.toml --lang es --out build/brief.es.md

Not translated — operator/developer text (English-only by design). The standard scopes i18n to end-user text, not operator logs, so these are deliberately left in English and carry no _() calls:

  • the CLI status/notice output (__main__.py) and its argparse help;
  • the public-submission moderation output;
  • the read-only server's HTTP status reasons and JSON health bodies (server.py);
  • every structured JSON log line (obs.py).

Translated — the web/ static UI, single-sourced from the same catalogs. The map/data view (web/app.js) and the submission form's status text (web/submit.js) render in English and Spanish. They used to carry their own hand-maintained I18N object (a second, drift-prone copy of every string); that is gone. The web strings are now registered as gettext ids under a web. namespace (src/nearmiss/web_i18n.py), translated in the same en/es .po catalogs, and compiled into committed per-locale JSON catalogs the browser fetches — see The web domain below. The static HTML body strings (headings, form labels) remain bilingual by their own markup and data-i18n hooks, driven by the same JSON at runtime.

The seam (how it works)

  • src/nearmiss/i18n.py — get_translation(lang) loads messages.mo for the language with fallback=True (unknown tags return the English source text, never an error). negotiate_lang() implements the §6 Accept-Language fallback chain (<requested> → <primary subtag> → en) for a future negotiated HTTP surface; the brief is CLI-driven (--lang) today.
  • The source string is the English text itself — e.g. _("Where the danger actually is — {city}") — so extraction needs no key map.
  • Catalogs live inside the package at src/nearmiss/locales/{messages.pot, <lang>/LC_MESSAGES/messages.{po,mo}}, so a checkout or an installed wheel resolves them with no separate install step.
  • babel.cfg tells pybabel extract to scan the package sources.
  • The web ids are keys, not English text — e.g. N_("web.app.title") in web_i18n.py. N_ is gettext's no-op marker (a default pybabel keyword), so the web ids are extracted into the same messages.pot as the brief; their English lives in the en catalog like every other locale. This is the one place a msgid is a key rather than source text, because the browser looks strings up by a stable id.

The web domain: single-sourced web/locales/<lang>.json

The static site is served as flat files from canonical CloudFront and the legacy GitHub Pages mirror and cannot call gettext at runtime, so tools/po2json.py compiles the web.* subset of each .po into a committed, deterministic JSON catalog at web/locales/<lang>.json (sorted keys, trailing newline) — the same "commit the compiled artifact, no deploy-time build" pattern as the .mo files. The shared loader web/i18n.js fetch()es the catalog for the active language (?lang=xx, or the language buttons on the map page) with English always loaded as the fallback; app.js and submit.js keep calling t("key") with short keys via a web.app. / web.submit. namespace, so no call site changed. JSON keys are the full msgids, so a string can never silently diverge from the catalog.

Spanish translation status — complete (no MT, no backlog)

Spanish is fully translated. It is pre-existing human translation carried over verbatim from the retired bespoke _ES dict; the two strings the old dict never covered (the Gi* glossary's final sentence and the significant-hotspot bullet, which used to render English even in the Spanish brief) were translated by hand during migration. No machine translation was introduced.

Because real Spanish exists, the G5 completeness gate is enforced as a hard merge-blocker (every msgid has a non-empty msgstr in every locale), not downgraded to a structural-only check with an untranslated-ES follow-on. There is no REVIEW-GATE R3 translation backlog for the current string set. Any new user-facing string is added in English first and must have its Spanish msgstr filled before merge (the gates below block otherwise).

Live merge-blocking gates (make i18n, wired into make verify and CI)

Local make i18n and the CI i18n job run the identical target (no drift):

Gate What it checks Mechanism
G2-lite The extraction template is committed & current pybabel extract → normalize → git diff --exit-code on messages.pot. A new/changed user-facing string without a re-extract fails.
G7 Every PO compiles cleanly msgfmt --check --check-format --check-domain on each messages.po
G6 EN/ES key-parity tools/check_catalog_parity.py: msgid sets of en and es are identical (empty symmetric difference)
G5 Completeness + placeholder parity same script: every msgstr (each plural form) non-empty; {...} field set identical source↔target
Web Web catalog completeness + JSON match tools/check_catalog_parity.py: every web.* id non-empty in en+es with matching {...} fields, and web/locales/*.json keys == the web.* msgid set. tools/po2json.py --check: the committed JSON matches what the PO would generate (drift gate).
G3 BCP 47 / RFC 5646 tag validity tools/check_bcp47.py: babel.Locale.parse on every authored locale tag
G9 No gettext bypass / hardcoded string (+ placeholder survival, ~30% expansion) tools/make_pseudolocale.py builds a build-only xx pseudo catalog; tests/test_pseudolocale.py renders the brief through it and fails on any user-facing string that shows up as raw English. Wired into make i18n via make i18n-pseudo.
G10 RTL layout smoke on web/ web/rtl_check.mjs (jsdom) loads each page with html[dir="rtl"] and rejects direction-unsafe inline styles. make rtl / cd web && npm run rtl, run in CI next to make axe.

msgfmt is a system gettext binary (like gitleaks, not a pip dependency); CI installs it with apt-get install -y gettext. pybabel/babel.Locale come from the babel>=2.16 dev dependency (pure Python — no PyICU/native ICU build, so nearmiss keeps its "runs anywhere Python runs, no native build step" property).

The POT normalizer

pybabel extract emits volatile tokens (creation timestamp, Babel version, the current year in the boilerplate comment) and false-positive python-format flags on strings containing a literal % (e.g. 95%, {pct}%) that are actually str.format (python-brace-format) strings. tools/i18n_normalize_pot.py freezes those tokens and drops the spurious flag so the G2-lite diff is meaningful, not flaky, and so msgfmt --check-format does not wrongly reject the literal percent signs. The gate runs the normalizer on both authoring and CI, keeping the committed POT byte-stable.

Compiled .mo files are committed

The compiled messages.mo catalogs are committed (not gitignored). Rationale: nearmiss's CI runs single make targets per job with no packaging/build step, and both the test job (the brief tests assert Spanish output) and the reproducibility job (which renders the brief) need the catalogs present at runtime. Committing .mo keeps every job green with local == CI and no extra per-job compile, and ships the translations in the wheel. .po ↔ .mo drift is guarded end-to-end by the brief tests (they render and assert the real Spanish); regenerate .mo after editing a .po with make i18n-compile.

The web JSON catalogs (web/locales/<lang>.json) are committed for the same reason and regenerated by the same make i18n-compile target (which now also runs tools/po2json.py); make i18n fails if they drift from the .po.

Workflow: adding or changing a user-facing string

  1. Wrap the English text in _("…") (or ngettext("…one…", "…many…", n) for counts) in brief.py / the label helpers in i18n.py. Use .format() for placeholders (_("… {n} …").format(n=n)).
  2. make i18n — re-extracts messages.pot; the G2-lite diff shows the new msgid.
  3. Merge the new msgid into the catalogs: pybabel update -i src/nearmiss/locales/messages.pot -d src/nearmiss/locales --no-fuzzy-matching, then fill the English msgstr (= msgid) and the Spanish msgstr.
  4. make i18n-compile to rebuild the .mo, then make i18n (and make verify).
  5. Commit the changed .py, messages.pot, both messages.po, and both messages.mo.

For a web string (map or submission form), register its id in web_i18n.py as N_("web.app.<key>") / N_("web.submit.<key>") and call t("<key>") at the JS call site, then follow the same steps 2–4; step 5 also commits the regenerated web/locales/*.json.

Adding a locale (community-translation runbook — no Python needed)

nearmiss's source string is the English text, so adding locale N+1 is a translate-only step. A contributor never edits Python:

  1. Copy the English catalog as the starting point for the new tag (de shown; use the real BCP 47 tag): 
    mkdir -p src/nearmiss/locales/de/LC_MESSAGES
cp src/nearmiss/locales/en/LC_MESSAGES/messages.po \
   src/nearmiss/locales/de/LC_MESSAGES/messages.po
    (Or pybabel init -i src/nearmiss/locales/messages.pot -d src/nearmiss/locales -l de.)
  2. Translate the msgstr values — brief and web.* ids alike; leave every msgid untouched, keep every {placeholder} and printf token exactly as it appears in the source, and set the header Language: de. This is the only file a translator edits.
  3. Register the tag in SUPPORTED_LANGUAGES in src/nearmiss/i18n.py (a one-line change; the only code touched).
  4. Validate — all merge-blocking, all runnable locally:
    • tag validity: python tools/check_bcp47.py (G3),
    • key-parity + completeness + placeholder parity: python tools/check_catalog_parity.py (G5/G6 — note CATALOGS there enumerates the checked locales; add the new tag),
    • PO compiles: msgfmt --check --check-format --check-domain -o /dev/null <po> (G7),
    • no gettext bypass: make i18n-pseudo (G9 — passes regardless of the new locale; it exercises the shared brief through the xx pseudo catalog).
  5. Compile + commit: make i18n-compile (regenerates the .mo and web/locales/<lang>.json), then make i18n and make verify. Commit the new .po, its compiled .mo, the regenerated web/locales/<lang>.json, and the one-line i18n.py change.

The web pages pick the locale up from ?lang=<lang>; the map page's language buttons are the only markup that needs a new entry to expose it in the UI chrome.

Scope not yet covered (honest boundaries)

This delivers the catalog, the PO-specific gates (G2/G3/G5/G6/G7), the pseudo-locale no-bypass gate (G9, make i18n-pseudo), and the RTL smoke (G10, make rtl). Other standard gates are not implemented here and remain future work: G1 (UTF-8 byte gate), G4 (html-has-lang on web/), G11 (Vary: Accept-Language — the server serves static files and negotiates no response body today), and G12 (CLDR/tzdata freshness pin). CLDR data is whatever the pinned babel bundles; revisit on the next Babel/CLDR major per the standard's cadence.