Phase 1: content negotiation (?lang=, Accept-Language) + localization/migration design docs

- pagerite/i18n.py: parse_accept_language, select_language, Translation struct, get_translation/available_languages stubs - app.py: language threaded through show_page/_render_html/_cached_body, ETag and Vary: accept-language - views.py: <html lang>, translated nav titles with fallback, canonical with ?lang=, hreflang alternates - pagerite.js: strip ?lang= via replaceState, send it as Accept-Language on fetch-navigation/preloads, lang-aware page cache - docs/localization.md: negotiation + URL scheme + phase-2 fragment model - docs/migrate.md: migrate_v3 content-addressed chunk storage plan
2026-09-02 01:14:10 +00:00
parent b6e6e46cfb
commit 218313ba7a
6 changed files with 584 additions and 50 deletions
@@ -0,0 +1,219 @@
+# Localization
+
+Pages are served in the visitor's language based on a `?lang=` query
+parameter or the `Accept-Language` header.
+
+- **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering
+  plumbing. Translations are consumed through a stub interface; the database
+  still holds only the original language.
+- **Phase 2 (final plan):** gettext-style fragment storage in the
+  database — machine-translated chunks plus user override patches, assembled
+  at render time. Storage details in `docs/migrate.md`.
+
+## Phase 1: negotiation and URLs
+
+### Language selection
+
+Deliberately simple — **q-values are ignored**:
+
+- All known `Accept-Language` implementations send the header **in order of
+  preference**, so we parse it as an ordered list and never reorder.
+- Selection rule (`select_language` in `pagerite/i18n.py`):
+  1. If `?lang=<tag>` is present, use it (if a translation exists; otherwise
+     fall through to header logic).
+  2. If the article's original language appears anywhere in the header list,
+     use the **original**. Rationale: an AI translation is strictly worse
+     than the original for anyone who has that language configured at all
+     (e.g. `fi-FI, fi, en-US, en` gets English, not machine-translated
+     Finnish).
+  3. Otherwise walk the header list in order and use the first language for
+     which a translation exists.
+  4. Fall back to the original.
+
+Region tags normalize to their base subtag (`fi-FI` → `fi`).
+
+### URLs: pretty for users, indexable for search engines
+
+- Canonical URLs stay pretty (`/some-page`). Each language version is
+  addressable as `/some-page?lang=fi` so search engines can index them.
+- `<link rel="canonical">` points to the page **itself including the query**
+  (each language version is its own canonical).
+- `<link rel="alternate" hreflang="…">` entries point to every other language
+  version (with `?lang=`), plus `x-default` for the plain URL.
+- On page load, pagerite.js removes the `?lang=` query via
+  `history.replaceState` (restoring the pretty URL) but remembers the language
+  in a JS variable. All fetch-navigation and preloads it performs afterwards
+  send that language in the `Accept-Language` header, so the chosen language
+  sticks for the session of clicks.
+- A full page refresh or a shared link resets to automatic selection (header
+  only). This gives a clean one-time override without cookies.
+
+### Response correctness
+
+- Content responses carry `Vary: accept-language` (added to the existing
+  `accept-encoding` vary).
+- `_cached_body` and the page ETag include the **selected language** (not the
+  raw header, which would blow up the cache key space).
+- `<html lang="…">` reflects the served language.
+
+### Rendering
+
+- The translated Markdown goes through the same `markdown.render` pipeline.
+- Navigation/sidebar titles come from the translation's title map, with
+  per-node fallback to the original title (a partially translated tree must
+  still render).
+- Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
+- The markdown typographer (SmartyPants) is English-centric; per-language
+  typographer options are a possible follow-up, not blocking.
+
+## Phase 2: fragment-based translation storage (draft)
+
+Phase 1 assumed whole-page translated Markdown delivered from outside. The
+refined model is gettext-style: an article has **one primary version** (its
+`content`, in its own language) plus, per target language, **machine
+fragments** (translated chunks of Markdown) and **user patches** (minimal
+editor overrides). Both are stored in the database and assembled into the
+served Markdown at render time.
+
+### The scenario this must handle
+
+1. Article written in English.
+2. Machine-translated into Spanish → fragments stored.
+3. Editor fixes one Spanish paragraph and changes a link elsewhere to point
+   at a Spanish resource → user patch hunks stored.
+4. English article edited → the edited chunk's key changes; its Spanish
+   fragment no longer matches.
+5. Page requested before the machine translation refreshes → served as a
+   **hybrid**: old fragments for unchanged chunks, plain English for the
+   edited chunk. User patches are attempted against this hybrid, best effort,
+   each hunk independently: the text fix is stale (its search text no longer
+   exists) and silently skipped; the link change still applies even though
+   the link sits in the now-English paragraph.
+6. Machine translation refreshes → full Spanish again, with both patch hunks
+   applying.
+
+### Chunks
+
+`chunk_markdown(markdown)` splits the source into block-level chunks —
+blank-line-separated blocks: headings, paragraphs, code fences (kept whole),
+list blocks, tables, HTML blocks. A chunk's identity is its **source text**,
+gettext-msgid style:
+
+```python
+chunk_key = blake3(normalize(chunk_text), digest_size=16).hex()
+```
+
+(`normalize`: strip trailing whitespace per line, collapse surrounding blank
+lines — so whitespace-only source edits don't invalidate translations.)
+
+Consequences:
+
+- Editing the English source invalidates exactly the edited chunks; all
+  other fragments keep applying. Stale fragments are simply never referenced
+  again and can be garbage-collected lazily (or left; they are tiny).
+- No explicit "source version" bookkeeping is needed — staleness falls out
+  of the keys.
+
+### User patches
+
+Editors always edit **full Markdown** in the existing editor UX — never
+fragments. When editing a translated view (`?lang=es`), the editor is loaded
+with the *current hybrid Markdown*; on save, the server computes a minimal
+diff against that hybrid and stores it as a patch:
+
+```python
+class Patch(msgspec.Struct, omit_defaults=True):
+    """One editing session's overrides, applied independently per hunk."""
+
+    hunks: list[tuple[str, str]] = []  # (search, replace) on hybrid Markdown
+```
+
+Hunks are produced from `difflib.SequenceMatcher` on the hybrid vs. the
+edited text at block granularity: each `replace`/`delete`/`insert` opcode
+becomes one `(search, replace)` pair, with the preceding block's tail as
+left context for `insert` (pure inserts have empty search context otherwise).
+Application is dead simple:
+
+```python
+def apply_patch(hybrid: str, patch: Patch) -> str:
+    for search, replace in patch.hunks:
+        if search and search in hybrid:
+            hybrid = hybrid.replace(search, replace, 1)
+        # missing search text = stale hunk -> silently skipped
+    return hybrid
+```
+
+Per-hunk independence is the robustness property from the scenario: a stale
+text fix does not block a still-valid link change. Patches are stored as an
+ordered list and applied in order.
+
+### Storage
+
+Full storage design and the `migrate_v3` restructuring live in
+`docs/migrate.md`. The short version, as it concerns this document:
+
+- Originals **and** translations are content-addressed text chunks in flat
+  stores: `Data.chunks: dict[hash, str]` and
+  `Data.trans: dict[f"{chunk_hash}:{lang}", str]` — path-independent, so
+  repeated paragraphs and menu titles are translated once and article moves
+  touch nothing. `Node.chunks: list[hash]` gives each article its order.
+- `Node` gains **`language: str = ""`**, inherited down the tree like
+  `banner` (empty = nearest ancestor, front page last, site default `en`
+  final). `select_language` and `<html lang>` use the resolved value instead
+  of the global `ORIGINAL_LANGUAGE` constant.
+  - **Known weakness:** changing a page's (or subtree's) `language` after
+    translations exist mis-keys everything — translations are keyed by
+    *source* chunks, so old entries silently stop matching and user patches
+    (searching for old-hybrid text) mostly go stale. That is acceptable:
+    the orphaned data is harmless and translations regenerate. We do not
+    migrate translations across a language change.
+- Article paths are stored and keyed **without leading slashes**
+  (`"docs/setup"`, front page `""`); slashes are added only in hrefs.
+
+### Render pipeline (replaces the phase-1 `get_translation` stub)
+
+```python
+def get_translation(path, lang, data) -> Translation | None:
+    if lang not in node.langs:
+        return None
+    hybrid = "\n\n".join(
+        chunks[h] if h in node.no_trans else trans.get(f"{h}:{lang}", chunks[h])
+        for h in node.chunks
+    )
+    for patch in data.patches.get(f"{path}:{lang}", []):
+        hybrid = apply_patch(hybrid, patch)
+    return Translation(markdown=hybrid, titles=title_map(data, lang))
+```
+
+- Availability is an article-level index: `node.langs: dict[lang, True]`,
+  maintained by the translation writers (translator job, patch saves) in the
+  same transaction as their data writes — rendering, language selection and
+  hreflang never probe the `trans` store chunk by chunk. A stale key is
+  benign (the "translation" just renders as the original).
+- `titles` for nav/sidebar/cards: each node's translated title is
+  `trans.get(f"{hash(node.title)}:{lang}")` with per-node fallback — one dict
+  lookup per nav item at render time.
+- Cache invalidation: writes to `chunks` / `trans` / `patches` (translator,
+  editor saves) call `_invalidate_pages()`, same as content writes.
+
+### Editor flow
+
+- `GET` of page Markdown for editing with a `lang` parameter returns the
+  hybrid (not the raw original) when the article has that language.
+- `PUT`/WS save with `lang` does **not** touch `node.chunks`; it diffs
+  against the hybrid that was served and appends a `Patch`. (Serve a hybrid
+  generation token with the editor payload so a save based on a stale hybrid
+  can be rebased or rejected — simplest: recompute the diff against the
+  *current* hybrid and accept best-effort, matching the patch philosophy.)
+- Saving the primary-language version re-chunks the submitted Markdown and
+  updates `Data.chunks` / `node.chunks` — only genuinely new text lands in
+  the kanta change diff (see docs/migrate.md).
+
+### Explicitly out of scope for phase 2
+
+- The machine translation itself: chunking output goes in, translated chunks
+  come back. A background job writes `trans` entries; this doc only defines
+  the storage key (`chunk_hash:lang`) and the merge semantics.
+- Garbage collection of orphaned chunks/translations (see docs/migrate.md).
+- sitemap.xml per-language entries; translated UI chrome; per-language
+  typographer options; multi-locale date/number formatting.