Skip to content

I18N.md

Latest commit

 

History

History
122 lines (103 loc) · 6.82 KB

I18N.md

File metadata and controls

122 lines (103 loc) · 6.82 KB
Raw

Internationalization: scope and flip conditions

This is the declaration the Standards Conformance table in README.md used to say was missing: what "supporting a language" means in Cairn, at each of three tiers, and what has to be true before a language moves up one.

Nothing here is a promise about a future language. It is a description of three tiers that already exist in the code, so a contributor proposing a fourth interface language — or wondering why a corpus document in a language Cairn has never heard of still works — has one place to read the actual rule instead of inferring it from language.py and messages.py separately.

The three tiers

1. Corpus language — no code change, ever

Any language at all. A corpus document declares its language in front matter (lang: vi) and that is the whole requirement — see cairn/corpus.py and "Corpus document format" in DESIGN.md. Retrieval, tokenization, and scoring are script-conditioned, not language-configured: cairn/text.py decides how to normalize a token from the Unicode script its characters are written in, not from a declared language code, so a corpus in a language nobody has added an interface catalogue for is still indexed, still scored, and still answerable — just not through a language-specific UI selector, and Cairn's own refusal/notice text about it is still spoken in whichever interface language (tier 2) the question was asked or answered in.

Flip condition to leave this tier alone: none. This tier requires no flip; it is the default and the floor. The one thing worth checking when a corpus gains a language it has few passages of is reachability — see LanguageStats.dilution_exempt in cairn/index.py and cairn lint, which warns when a language has too few passages for the document-frequency floor to suppress anything (DESIGN.md, "The document-frequency floor has one exemption, and it is narrow").

2. Interface language — a messages.py catalogue plus a LANGUAGES entry

The four languages Cairn currently ships system strings for: English, Spanish, Arabic, French (cairn/language.py, LANGUAGES). An interface language can be selected explicitly (--lang, the web selector), gets its own refusal and notice wording, and its own row in the accessible interface's language mirroring.

What moving a language into this tier requires:

  • A full entry in cairn/language.py's LANGUAGES table (code, endonym, English name, direction).
  • A full catalogue in cairn/messages.py's CATALOGUE, with every key the reference language (en) has. tests/test_multilingual.py's TestMessageCatalogue enforces three things about it, and a new language has to pass all three: every key present (test_every_language_carries_every_key), no value merely copied from English (test_no_translation_is_left_as_the_english_string), and the same {placeholder} set as every other language (test_placeholders_match_across_languages).
  • At least one same-language corpus probe, so the multilingual audit suite has real evidence to score for the language rather than none.

Flip condition: the tests above pass, and a same-language evidence item exists in the recorded bundle. Nothing about cross-language behavior is required to add an interface language — see the next section for why that is a separate, harder question.

French, added without that last step, and why. fr has the full LANGUAGES entry and messages.py catalogue, passes every test named above, and is fully answerable — with no French corpus content and no French evidence item, a French question falls back across languages exactly the way a Spanish question about the (English-only) GoPass document already does, or refuses in French. What it does not have is the "at least one same-language corpus probe" step: that requires adding a French corpus document, recording it with cairn record, and regenerating plumbline/baseline.json against the pinned Plumbline harness — a network-dependent step (plumbline-gate.sh resolves the harness at run time) that was not available when this language was added. cairn record --diff-against plumbline/bundle confirms the committed evidence bundle is unchanged by this addition — French added nothing to score and moved nothing that was already scored — which is a narrower, weaker claim than "the audit suite has evidence for French," and this document says so rather than leaving the gap implicit.

3. RTL-table language — an RTL_CODES entry

Writing direction is derived from the language code alone (cairn/language.py, direction_of), from one table, deliberately not configurable (DESIGN.md, "Configuration": "two places to state the direction of Arabic is one place for it to be wrong"). RTL_CODES already lists several right-to-left ISO 639-1 codes beyond the four interface languages, so a corpus document in, say, Farsi or Urdu is already laid out right-to-left and bidi-isolated correctly even though neither is an interface language yet.

Flip condition: a language's own writing system is right-to-left. This is a fact about the language, not a design decision Cairn makes, so adding a code here needs no measurement — only checking the correct ISO code against a authoritative source and a test that direction resolves to "rtl" for it (see TestDirection in tests/test_multilingual.py).

What is explicitly not on this path

  • No machine translation, ever, at any tier. Corpus content is quoted verbatim in whatever language it was written in; translating it would produce an unsourced policy fact (README, "Three languages, one of them right to left"). This is a hard line, not a missing feature.
  • No per-language dictionaries or stopword lists. Tokenization and document-frequency suppression are script-conditioned and corpus-statistical, not backed by a word list for any language (cairn/text.py, cairn/index.py). A language reaching tier 1 needs none of that machinery updated.
  • No promise about cross-language fallback reach. An interface language (tier 2) answering from another language's document when nothing in its own language clears the threshold is lexical only, and DESIGN.md's "What is still open" measures exactly how far that reaches today. Moving a language into tier 2 does not change or extend that mechanism.

Why this file exists now and did not before

The three-tier structure above was true of the code before this file existed; it was documented only as scattered prose across language.py, messages.py, and DESIGN.md. README's own Standards Conformance table named the gap directly: "there is no docs/I18N.md declaring the scope and the flip conditions, so what is committed to beyond these three languages is undeclared." This file is that declaration, and it changes nothing about what Cairn does — it is documentation of an existing boundary, not a new one.