224 lines
10 KiB
Markdown
224 lines
10 KiB
Markdown
# Localization
|
|
|
|
Pages are served in the visitor's language based on a `?lang=` query
|
|
parameter or the `Accept-Language` header.
|
|
|
|
- **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering
|
|
plumbing. Translations are consumed through a stub interface; the database
|
|
still holds only the original language.
|
|
- **Phase 2 (implemented):** gettext-style fragment storage in the
|
|
database — machine-translated chunks plus user override patches, assembled
|
|
at render time. Storage details in `docs/migrate.md`.
|
|
|
|
## Phase 1: negotiation and URLs
|
|
|
|
### Language selection
|
|
|
|
Deliberately simple — **q-values are ignored**:
|
|
|
|
- All known `Accept-Language` implementations send the header **in order of
|
|
preference**, so we parse it as an ordered list and never reorder.
|
|
- Selection rule (`select_language` in `pagerite/i18n.py`):
|
|
1. If `?lang=<tag>` is present, use it (if a translation exists; otherwise
|
|
fall through to header logic).
|
|
2. If the article's original language appears anywhere in the header list,
|
|
use the **original**. Rationale: an AI translation is strictly worse
|
|
than the original for anyone who has that language configured at all
|
|
(e.g. `fi-FI, fi, en-US, en` gets English, not machine-translated
|
|
Finnish).
|
|
3. Otherwise walk the header list in order and use the first language for
|
|
which a translation exists.
|
|
4. Fall back to the original.
|
|
|
|
Region tags normalize to their base subtag (`fi-FI` → `fi`).
|
|
|
|
### URLs: pretty for users, indexable for search engines
|
|
|
|
- Canonical URLs stay pretty (`/some-page`). Each language version is
|
|
addressable as `/some-page?lang=fi` so search engines can index them.
|
|
- `<link rel="canonical">` points to the page **itself including the query**
|
|
(each language version is its own canonical).
|
|
- `<link rel="alternate" hreflang="…">` entries point to every other language
|
|
version (with `?lang=`), plus `x-default` for the plain URL.
|
|
- On page load, pagerite.js removes the `?lang=` query via
|
|
`history.replaceState` (restoring the pretty URL) but remembers the language
|
|
in a JS variable. All fetch-navigation and preloads it performs afterwards
|
|
send that language in the `Accept-Language` header, so the chosen language
|
|
sticks for the session of clicks.
|
|
- A full page refresh or a shared link resets to automatic selection (header
|
|
only). This gives a clean one-time override without cookies.
|
|
|
|
### Response correctness
|
|
|
|
- Content responses carry `Vary: accept-language` (added to the existing
|
|
`accept-encoding` vary).
|
|
- `_cached_body` and the page ETag include the **selected language** (not the
|
|
raw header, which would blow up the cache key space).
|
|
- `<html lang="…">` reflects the served language.
|
|
|
|
### Rendering
|
|
|
|
- The translated Markdown goes through the same `markdown.render` pipeline.
|
|
- Navigation/sidebar titles come from the translation's title map, with
|
|
per-node fallback to the original title (a partially translated tree must
|
|
still render).
|
|
- Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
|
|
- The markdown typographer (SmartyPants) is English-centric; per-language
|
|
typographer options are a possible follow-up, not blocking.
|
|
|
|
## Phase 2: fragment-based translation storage (implemented)
|
|
|
|
Phase 1 assumed whole-page translated Markdown delivered from outside. The
|
|
refined model is gettext-style: an article has **one primary version** (its
|
|
`content`, in its own language) plus, per target language, **machine
|
|
fragments** (translated chunks of Markdown) and **user patches** (minimal
|
|
editor overrides). Both are stored in the database and assembled into the
|
|
served Markdown at render time.
|
|
|
|
### The scenario this must handle
|
|
|
|
1. Article written in English.
|
|
2. Machine-translated into Spanish → fragments stored.
|
|
3. Editor fixes one Spanish paragraph and changes a link elsewhere to point
|
|
at a Spanish resource → user patch hunks stored.
|
|
4. English article edited → the edited chunk's key changes; its Spanish
|
|
fragment no longer matches.
|
|
5. Page requested before the machine translation refreshes → served as a
|
|
**hybrid**: old fragments for unchanged chunks, plain English for the
|
|
edited chunk. User patches are attempted against this hybrid, best effort,
|
|
each hunk independently: the text fix is stale (its search text no longer
|
|
exists) and silently skipped; the link change still applies even though
|
|
the link sits in the now-English paragraph.
|
|
6. Machine translation refreshes → full Spanish again, with both patch hunks
|
|
applying.
|
|
|
|
### Chunks
|
|
|
|
`chunk_markdown(markdown)` splits the source into block-level chunks —
|
|
blank-line-separated blocks: headings, paragraphs, code fences (kept whole),
|
|
list blocks, tables, HTML blocks. A chunk's identity is its **source text**,
|
|
gettext-msgid style:
|
|
|
|
```python
|
|
chunk_key = blake3(normalize(chunk_text)).digest(9) # bytes; base64 at the JSON level
|
|
```
|
|
|
|
(`normalize`: strip trailing whitespace per line, collapse surrounding blank
|
|
lines — so whitespace-only source edits don't invalidate translations.)
|
|
|
|
Consequences:
|
|
|
|
- Editing the English source invalidates exactly the edited chunks; all
|
|
other fragments keep applying. Stale fragments are simply never referenced
|
|
again and can be garbage-collected lazily (or left; they are tiny).
|
|
- No explicit "source version" bookkeeping is needed — staleness falls out
|
|
of the keys.
|
|
|
|
### User patches
|
|
|
|
Editors always edit **full Markdown** in the existing editor UX — never
|
|
fragments. When editing a translated view (`?lang=es`), the editor is loaded
|
|
with the *current hybrid Markdown*; on save, the server computes a minimal
|
|
diff against that hybrid and stores it as a patch:
|
|
|
|
```python
|
|
class Patch(msgspec.Struct, omit_defaults=True):
|
|
"""One editing session's overrides, applied independently per hunk."""
|
|
|
|
hunks: list[tuple[str, str]] = [] # (search, replace) on hybrid Markdown
|
|
```
|
|
|
|
Hunks are produced from `difflib.SequenceMatcher` on the hybrid vs. the
|
|
edited text at block granularity: each `replace`/`delete`/`insert` opcode
|
|
becomes one `(search, replace)` pair, with the preceding block's tail as
|
|
left context for `insert` (pure inserts have empty search context otherwise).
|
|
Application is dead simple:
|
|
|
|
```python
|
|
def apply_patch(hybrid: str, patch: Patch) -> str:
|
|
for search, replace in patch.hunks:
|
|
if search and search in hybrid:
|
|
hybrid = hybrid.replace(search, replace, 1)
|
|
# missing search text = stale hunk -> silently skipped
|
|
return hybrid
|
|
```
|
|
|
|
Per-hunk independence is the robustness property from the scenario: a stale
|
|
text fix does not block a still-valid link change. Patches are stored as an
|
|
ordered list and applied in order.
|
|
|
|
### Storage
|
|
|
|
Full storage design and the `migrate_v3` restructuring live in
|
|
`docs/migrate.md`. The short version, as it concerns this document:
|
|
|
|
- Originals **and** translations are content-addressed text chunks in flat
|
|
stores: `Data.chunks: dict[bytes, str]` and
|
|
`Data.trans: dict[bytes, dict[str, str]]` (chunk hash → lang → text) —
|
|
path-independent, so repeated paragraphs and menu titles are translated
|
|
once and article moves touch nothing. `Node.chunks: list[bytes]` gives
|
|
each article its order.
|
|
- `Node` gains **`language: str = ""`**, inherited down the tree like
|
|
`banner` (empty = nearest ancestor, front page last, site default `en`
|
|
final). `select_language` and `<html lang>` use the resolved value instead
|
|
of the global `ORIGINAL_LANGUAGE` constant.
|
|
- **Known weakness:** changing a page's (or subtree's) `language` after
|
|
translations exist mis-keys everything — translations are keyed by
|
|
*source* chunks, so old entries silently stop matching and user patches
|
|
(searching for old-hybrid text) mostly go stale. That is acceptable:
|
|
the orphaned data is harmless and translations regenerate. We do not
|
|
migrate translations across a language change.
|
|
- Article paths are stored and keyed **without leading slashes**
|
|
(`"docs/setup"`, front page `""`); slashes are added only in hrefs.
|
|
|
|
### Render pipeline (the phase-1 `get_translation` stub, now real)
|
|
|
|
```python
|
|
def get_translation(path, lang, data) -> Translation | None:
|
|
if lang not in node.langs:
|
|
return None
|
|
hybrid = "\n\n".join(
|
|
chunks[h] if h in node.no_trans else trans.get(h, {}).get(lang, chunks[h])
|
|
for h in node.chunks
|
|
)
|
|
for patch in data.patches.get(f"{path}:{lang}", []):
|
|
hybrid = apply_patch(hybrid, patch)
|
|
return Translation(markdown=hybrid, titles=title_map(data, lang))
|
|
```
|
|
|
|
- Availability is an article-level index: `node.langs: dict[lang, True]`,
|
|
maintained by the translation writers (translator job, patch saves) in the
|
|
same transaction as their data writes — rendering, language selection and
|
|
hreflang never probe the `trans` store chunk by chunk. A stale key is
|
|
benign (the "translation" just renders as the original).
|
|
- `titles` for nav/sidebar/cards: each node's translated title is
|
|
`trans.get(hash(node.title), {}).get(lang)` with per-node fallback — one
|
|
dict lookup per nav item at render time.
|
|
- Cache invalidation: writes to `chunks` / `trans` / `patches` (translator,
|
|
editor saves) call `_invalidate_pages()`, same as content writes.
|
|
|
|
### Editor flow
|
|
|
|
- `GET` of page Markdown for editing with a `lang` parameter returns the
|
|
hybrid (not the raw original) when the article has that language.
|
|
- `PUT`/WS save with `lang` does **not** touch `node.chunks`; it diffs
|
|
against the hybrid that was served and appends a `Patch`. (Serve a hybrid
|
|
generation token with the editor payload so a save based on a stale hybrid
|
|
can be rebased or rejected — simplest: recompute the diff against the
|
|
*current* hybrid and accept best-effort, matching the patch philosophy.)
|
|
- Saving the primary-language version re-chunks the submitted Markdown and
|
|
updates `Data.chunks` / `node.chunks` — only genuinely new text lands in
|
|
the kanta change diff (see docs/migrate.md).
|
|
|
|
### Explicitly out of scope for phase 2
|
|
|
|
- The machine translation itself: chunking output goes in, translated chunks
|
|
come back. The **service API exists** — `GET /_api/translate/{lang}` lists
|
|
pending items (`{"key", "text", "path", "kind"}`, key = base64 chunk hash),
|
|
`POST /_api/translate/{lang}` stores a batch (`{"items": [{key, text}]}`)
|
|
into `trans` and maintains `node.langs`; an external service does the
|
|
actual translating (gated by the /_api forward-auth like everything else).
|
|
- Garbage collection of orphaned chunks/translations (see docs/migrate.md).
|
|
- sitemap.xml per-language entries; translated UI chrome; per-language
|
|
typographer options; multi-locale date/number formatting.
|