# Localization Pages are served in the visitor's language based on a `?lang=` query parameter or the `Accept-Language` header. - **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering plumbing. Translations are consumed through a stub interface; the database still holds only the original language. - **Phase 2 (implemented):** gettext-style fragment storage in the database — machine-translated chunks plus user override patches, assembled at render time. Storage details in `docs/migrate.md`. ## Phase 1: negotiation and URLs ### Language selection Deliberately simple — **q-values are ignored**: - All known `Accept-Language` implementations send the header **in order of preference**, so we parse it as an ordered list and never reorder. - Selection rule (`select_language` in `pagerite/i18n.py`): 1. If `?lang=` is present, use it (if a translation exists; otherwise fall through to header logic). 2. If the article's original language appears anywhere in the header list, use the **original**. Rationale: an AI translation is strictly worse than the original for anyone who has that language configured at all (e.g. `fi-FI, fi, en-US, en` gets English, not machine-translated Finnish). 3. Otherwise walk the header list in order and use the first language for which a translation exists. 4. Fall back to the original. Region tags normalize to their base subtag (`fi-FI` → `fi`). ### URLs: pretty for users, indexable for search engines - Canonical URLs stay pretty (`/some-page`). Each language version is addressable as `/some-page?lang=fi` so search engines can index them. - `` names the **actually served language**: the plain URL when serving the original (for SEO the non-query URL means the article's own language), `?lang=xx` when serving a translation — however the language was arrived at (query or header). - `` entries follow the canonical directly (before the social meta tags) and are the same set on every page — the site-wide configured languages (`translate_langs`, which the translator works to fill in): `x-default` first, pointing at the plain autodetecting URL, then every language explicitly with `?lang=`, the default language included. - The override sticks for the session of clicks: a page requested with `?lang=` replicates the query onto the navigation links it renders (nav, sidebar, cards, brand — in-article links are content and stay as authored), so plain clicks and no-JS navigation keep the language. pagerite.js additionally strips the query from the address bar via `history.replaceState` (pretty, shareable URLs), remembers the language, and adds it to every internal fetch that lacks one (preloads, fetch-navigations, history traversals); history entries stay query-less. - A full page refresh or a shared link resets to automatic selection (header only). This gives a clean one-time override without cookies. ### Response correctness - Content responses carry `Vary: accept-language` (added to the existing `accept-encoding` vary). - `_cached_body` and the page ETag include the **selected language** (not the raw header, which would blow up the cache key space) and the **replicated link language**: a `?lang=fi` render and a header-selected Finnish render of the same page differ in their navigation links, so they are cached as separate variants. - `` reflects the served language. ### Rendering - The translated Markdown goes through the same `markdown.render` pipeline. - Navigation/sidebar titles come from the translation's title map, with per-node fallback to the original title (a partially translated tree must still render). - Category placeholder pages (the 404s for content-less labels) select a language like content pages, but over the **subtree's** combined availability (`subtree_languages`) — they have no chunks of their own; the heading, navigation and card text localize from the title map and the target articles' translations. - Card descriptions and cover picks run on the target article's hybrid Markdown where that page is available in the served language, with per-card fallback to the original. - Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now. - The markdown typographer (SmartyPants) is English-centric; per-language typographer options are a possible follow-up, not blocking. ## Phase 2: fragment-based translation storage (implemented) Phase 1 assumed whole-page translated Markdown delivered from outside. The refined model is gettext-style: an article has **one primary version** (its `content`, in its own language) plus, per target language, **machine fragments** (translated chunks of Markdown) and **user patches** (minimal editor overrides). Both are stored in the database and assembled into the served Markdown at render time. ### The scenario this must handle 1. Article written in English. 2. Machine-translated into Spanish → fragments stored. 3. Editor fixes one Spanish paragraph and changes a link elsewhere to point at a Spanish resource → user patch hunks stored. 4. English article edited → the edited chunk's key changes; its Spanish fragment no longer matches. 5. Page requested before the machine translation refreshes → served as a **hybrid**: old fragments for unchanged chunks, plain English for the edited chunk. User patches are attempted against this hybrid, best effort, each hunk independently: the text fix is stale (its search text no longer exists) and silently skipped; the link change still applies even though the link sits in the now-English paragraph. 6. Machine translation refreshes → full Spanish again, with both patch hunks applying. ### Chunks `chunk_markdown(markdown)` splits the source into block-level chunks — blank-line-separated blocks: headings, paragraphs, code fences (kept whole), list blocks, tables, HTML blocks. A chunk's identity is its **source text**, gettext-msgid style: ```python chunk_key = blake3(normalize(chunk_text)).digest(9) # bytes; base64 at the JSON level ``` (`normalize`: strip trailing whitespace per line, collapse surrounding blank lines — so whitespace-only source edits don't invalidate translations.) Consequences: - Editing the English source invalidates exactly the edited chunks; all other fragments keep applying. Stale fragments are simply never referenced again and can be garbage-collected lazily (or left; they are tiny). - No explicit "source version" bookkeeping is needed — staleness falls out of the keys. ### User patches Editors always edit **full Markdown** in the existing editor UX — never fragments. When editing a translated view (`?lang=es`), the editor is loaded with the *current hybrid Markdown*; on save, the server computes a minimal diff against that hybrid and stores it as a patch: ```python class Patch(msgspec.Struct, omit_defaults=True): """One editing session's overrides, applied independently per hunk.""" hunks: list[tuple[str, str]] = [] # (search, replace) on hybrid Markdown ``` Hunks are produced from `difflib.SequenceMatcher` on the hybrid vs. the edited text at block granularity: each `replace`/`delete`/`insert` opcode becomes one `(search, replace)` pair, with the preceding block's tail as left context for `insert` (pure inserts have empty search context otherwise). Application is dead simple: ```python def apply_patch(hybrid: str, patch: Patch) -> str: for search, replace in patch.hunks: if search and search in hybrid: hybrid = hybrid.replace(search, replace, 1) # missing search text = stale hunk -> silently skipped return hybrid ``` Per-hunk independence is the robustness property from the scenario: a stale text fix does not block a still-valid link change. Patches are stored as an ordered list and applied in order. ### Storage Full storage design and the `migrate_v3` restructuring live in `docs/migrate.md`. The short version, as it concerns this document: - Originals **and** translations are content-addressed text chunks in flat stores: `Data.chunks: dict[bytes, str]` and `Data.trans: dict[bytes, dict[str, str]]` (chunk hash → lang → text) — path-independent, so repeated paragraphs and menu titles are translated once and article moves touch nothing. `Node.chunks: list[bytes]` gives each article its order. - `Node` gains **`language: str = ""`**, inherited down the tree like `banner` (empty = nearest ancestor, front page last, site default `en` final). `select_language` and `` use the resolved value instead of the global `ORIGINAL_LANGUAGE` constant. - **Known weakness:** changing a page's (or subtree's) `language` after translations exist mis-keys everything — translations are keyed by *source* chunks, so old entries silently stop matching and user patches (searching for old-hybrid text) mostly go stale. That is acceptable: the orphaned data is harmless and translations regenerate. We do not migrate translations across a language change. - Article paths are stored and keyed **without leading slashes** (`"docs/setup"`, front page `""`); slashes are added only in hrefs. ### Render pipeline (the phase-1 `get_translation` stub, now real) ```python def get_translation(path, lang, data) -> Translation | None: if lang not in node.langs: return None hybrid = "\n\n".join( chunks[h] if h in node.no_trans else trans.get(h, {}).get(lang, chunks[h]) for h in node.chunks ) for patch in data.patches.get(f"{path}:{lang}", []): hybrid = apply_patch(hybrid, patch) return Translation(markdown=hybrid, titles=title_map(data, lang)) ``` - Availability is an article-level index: `node.langs: dict[lang, True]`, maintained by the translation writers (translator job, patch saves) in the same transaction as their data writes — rendering and language selection never probe the `trans` store chunk by chunk. A stale key is benign (the "translation" just renders as the original). - `titles` for nav/sidebar/cards: each node's translated title is `trans.get(hash(node.title), {}).get(lang)` with per-node fallback — one dict lookup per nav item at render time. - Cache invalidation: writes to `chunks` / `trans` / `patches` (translator, editor saves) call `_invalidate_pages()`, same as content writes. ### Editor flow The page editor has a language picker (flag + name; the same country-flag-icons set as the analytics visitor cells) listing the primary language and the union of the page's translations (`node.langs`) and the site-wide `translate_langs`. It opens in the language the page was served in (``). A note under the toolbar states the blast radius: edits to the primary language re-chunk the original (invalidating the affected translation fragments everywhere); edits to a translation stay local to that language. - WS `open` with a `lang` returns the effective **hybrid** Markdown and title for that language (ungated by `node.langs` — a language without any fragments yet starts from the original text), plus the language metadata (`lang`, `primary_lang`, `langs`, `translate_langs`). - The editor keeps a **shadow copy** of the Markdown it opened. WS `save` with `lang` sends it as `base`; the server diffs `base` → submitted text (`make_patch`) and appends a `Patch`. Diffing against the shadow (rather than the current hybrid) keeps hunks correct when the original or the machine translation moved under an open editor; application against the then-current hybrid stays best-effort per hunk, as designed. - A changed **title** on a translated save becomes a fragment in `Data.trans` keyed by the original title's chunk hash — the same storage as machine title translations. An untouched title field (holding the served translation) is not sent, so saving never freezes a stale machine title into an override. - Saving never deletes; a translation additionally cannot be emptied (that would render as a blank page in that language). - The live preview renders the version being edited, whichever language the page itself was loaded in (the render is just the edited Markdown + title). A translated save keeps that preview in place — re-fetching the page would come back in the header-selected language. - Saving the primary-language version re-chunks the submitted Markdown and updates `Data.chunks` / `node.chunks` — only genuinely new text lands in the kanta change diff (see docs/migrate.md). ### Translator service API An external machine-translation service connects over WebSocket at `/_translate/{key}` — deliberately **not** under `/_api`: the SSO forward-auth does not cover that route, and the key in the path is the access control. Keys live in `Data.translate_keys` (key -> display name) — 12 lowercase alphanumeric characters each, the first one generated at database bootstrap and multiple keys reserved for future management (e.g. a web UI). The full WS URL(s) are printed in the startup log (`ws://localhost:{port}/_translate/{key}` locally, `wss://{hostname}/_translate/{key}` on a public hostname) and the keys are surfaced to the admin in `GET /_api/settings` as `translate_keys`. An unknown or empty key rejects the handshake (close-before-accept → HTTP 403). Transactions storing results record the connecting key as the kanta transaction `user`. Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; `bytes` fields ride as base64): - `{"type": "hello", "langs": [...]}` — client greeting announcing its **capabilities**: the language codes its model can produce (normalized to base subtags; `en`/empty dropped). - `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push: ONE fragment to translate (an article title or a chunk), its text **masked** (see Masking below). - `{"type": "result", "lang", "key", "text"}` — client reply: the translation of the connection's current job, matching it by (lang, key). Which languages get translated is **server-configured**: `Data.translate_langs` (presence-key dict, bootstrapped to Spanish and Chinese — the original language is never a target — edited in the editor shell's localization tab or set via `/_api/settings` as `translate_langs`). The dispatcher offers a connection jobs only in `wanted ∩ capable`; a connection without overlap simply stays idle. Dispatch semantics (all in app.py): - **One job at a time per connection** — the next job is sent only after the current one's result. Clients wanting parallelism open multiple connections (e.g. several `scripts/translator.py` instances). - Pending work is derived from the `trans` store (`translate.pending_items`) minus the items in flight on any connection, so a **disconnect requeues** that connection's in-flight item and it is offered to any free capable connection. - Dispatch re-runs on every relevant event: Hello, result, disconnect and content change (`_invalidate_pages()` schedules it, so the pass runs after the writing transaction commits). - A result with no job in flight, a mismatched (lang, key), a duplicate hello, or any malformed frame closes the socket with a protocol error. Results are stored into `trans` in one transaction and set `node.langs[lang]` on every article they touch (shared chunks make several pages gain a language from one fragment). Unknown keys are stored anyway and re-storing overwrites — results are idempotent. #### Masking Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before dispatch and restored by number from the result. Masked: code spans, container-fence names, link and image *destinations* (link text, alt text and captions stay visible for translation), reference and footnote labels, `{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not masked — it carries no lexical content and models preserve it. Chunks with no prose left after masking (a lone `{dates}`, container fences, pure code/HTML) are never dispatched at all (`needs_translation`); every language renders them from the original chunk. A result is accepted only if every sentinel survived exactly once, in any order (translations legitimately reorder spans). A mangled result is dropped and logged, and the (lang, key) pair is skipped for the rest of the server run — generation is near-deterministic, so an immediate retry would re-fail the same way; the fragment stays pending and gets another chance on restart or a model/masking change. `Data.trans` therefore only ever holds clean, unmasked text. ### Explicitly out of scope for phase 2 - The machine translation itself: the API above moves fragments in and out; the translating is external. `scripts/translator.py` is the reference client (Seed-X-PPO-7B only — its 28 languages are the ceiling). - Garbage collection of orphaned chunks/translations (see docs/migrate.md). - sitemap.xml per-language entries; translated UI chrome; per-language typographer options; multi-locale date/number formatting.