Files

T

LeoVasanko 4c886eda86 Masked translation round-trip; named translator keys

- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).

2026-09-02 19:56:54 +00:00

17 KiB

Raw Blame History

Localization

Pages are served in the visitor's language based on a ?lang= query parameter or the Accept-Language header.

Phase 1 (implemented): negotiation, URL scheme, caching, rendering plumbing. Translations are consumed through a stub interface; the database still holds only the original language.
Phase 2 (implemented): gettext-style fragment storage in the database — machine-translated chunks plus user override patches, assembled at render time. Storage details in docs/migrate.md.

Phase 1: negotiation and URLs

Language selection

Deliberately simple — q-values are ignored:

All known Accept-Language implementations send the header in order of preference, so we parse it as an ordered list and never reorder.
Selection rule (select_language in pagerite/i18n.py):
1. If ?lang=<tag> is present, use it (if a translation exists; otherwise fall through to header logic).
2. If the article's original language appears anywhere in the header list, use the original. Rationale: an AI translation is strictly worse than the original for anyone who has that language configured at all (e.g. fi-FI, fi, en-US, en gets English, not machine-translated Finnish).
3. Otherwise walk the header list in order and use the first language for which a translation exists.
4. Fall back to the original.

Region tags normalize to their base subtag (fi-FI → fi).

URLs: pretty for users, indexable for search engines

Canonical URLs stay pretty (/some-page). Each language version is addressable as /some-page?lang=fi so search engines can index them.
<link rel="canonical"> names the actually served language: the plain URL when serving the original (for SEO the non-query URL means the article's own language), ?lang=xx when serving a translation — however the language was arrived at (query or header).
<link rel="alternate" hreflang="…"> entries follow the canonical directly (before the social meta tags) and are the same set on every page — the site-wide configured languages (translate_langs, which the translator works to fill in): x-default first, pointing at the plain autodetecting URL, then every language explicitly with ?lang=, the default language included.
The override sticks for the session of clicks: a page requested with ?lang= replicates the query onto the navigation links it renders (nav, sidebar, cards, brand — in-article links are content and stay as authored), so plain clicks and no-JS navigation keep the language. pagerite.js additionally strips the query from the address bar via history.replaceState (pretty, shareable URLs), remembers the language, and adds it to every internal fetch that lacks one (preloads, fetch-navigations, history traversals); history entries stay query-less.
A full page refresh or a shared link resets to automatic selection (header only). This gives a clean one-time override without cookies.

Response correctness

Content responses carry Vary: accept-language (added to the existing accept-encoding vary).
_cached_body and the page ETag include the selected language (not the raw header, which would blow up the cache key space) and the replicated link language: a ?lang=fi render and a header-selected Finnish render of the same page differ in their navigation links, so they are cached as separate variants.
<html lang="…"> reflects the served language.

Rendering

The translated Markdown goes through the same markdown.render pipeline.
Navigation/sidebar titles come from the translation's title map, with per-node fallback to the original title (a partially translated tree must still render).
Category placeholder pages (the 404s for content-less labels) select a language like content pages, but over the subtree's combined availability (subtree_languages) — they have no chunks of their own; the heading, navigation and card text localize from the title map and the target articles' translations.
Card descriptions and cover picks run on the target article's hybrid Markdown where that page is available in the served language, with per-card fallback to the original.
Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
The markdown typographer (SmartyPants) is English-centric; per-language typographer options are a possible follow-up, not blocking.

Phase 2: fragment-based translation storage (implemented)

Phase 1 assumed whole-page translated Markdown delivered from outside. The refined model is gettext-style: an article has one primary version (its content, in its own language) plus, per target language, machine fragments (translated chunks of Markdown) and user patches (minimal editor overrides). Both are stored in the database and assembled into the served Markdown at render time.

The scenario this must handle

Article written in English.
Machine-translated into Spanish → fragments stored.
Editor fixes one Spanish paragraph and changes a link elsewhere to point at a Spanish resource → user patch hunks stored.
English article edited → the edited chunk's key changes; its Spanish fragment no longer matches.
Page requested before the machine translation refreshes → served as a hybrid: old fragments for unchanged chunks, plain English for the edited chunk. User patches are attempted against this hybrid, best effort, each hunk independently: the text fix is stale (its search text no longer exists) and silently skipped; the link change still applies even though the link sits in the now-English paragraph.
Machine translation refreshes → full Spanish again, with both patch hunks applying.

Chunks

chunk_markdown(markdown) splits the source into block-level chunks — blank-line-separated blocks: headings, paragraphs, code fences (kept whole), list blocks, tables, HTML blocks. A chunk's identity is its source text, gettext-msgid style:

chunk_key = blake3(normalize(chunk_text)).digest(9)  # bytes; base64 at the JSON level

(normalize: strip trailing whitespace per line, collapse surrounding blank lines — so whitespace-only source edits don't invalidate translations.)

Consequences:

Editing the English source invalidates exactly the edited chunks; all other fragments keep applying. Stale fragments are simply never referenced again and can be garbage-collected lazily (or left; they are tiny).
No explicit "source version" bookkeeping is needed — staleness falls out of the keys.

User patches

Editors always edit full Markdown in the existing editor UX — never fragments. When editing a translated view (?lang=es), the editor is loaded with the current hybrid Markdown; on save, the server computes a minimal diff against that hybrid and stores it as a patch:

class Patch(msgspec.Struct, omit_defaults=True):
    """One editing session's overrides, applied independently per hunk."""

    hunks: list[tuple[str, str]] = []  # (search, replace) on hybrid Markdown

Hunks are produced from difflib.SequenceMatcher on the hybrid vs. the edited text at block granularity: each replace/delete/insert opcode becomes one (search, replace) pair, with the preceding block's tail as left context for insert (pure inserts have empty search context otherwise). Application is dead simple:

def apply_patch(hybrid: str, patch: Patch) -> str:
    for search, replace in patch.hunks:
        if search and search in hybrid:
            hybrid = hybrid.replace(search, replace, 1)
        # missing search text = stale hunk -> silently skipped
    return hybrid

Per-hunk independence is the robustness property from the scenario: a stale text fix does not block a still-valid link change. Patches are stored as an ordered list and applied in order.

Storage

Full storage design and the migrate_v3 restructuring live in docs/migrate.md. The short version, as it concerns this document:

Originals and translations are content-addressed text chunks in flat stores: Data.chunks: dict[bytes, str] and Data.trans: dict[bytes, dict[str, str]] (chunk hash → lang → text) — path-independent, so repeated paragraphs and menu titles are translated once and article moves touch nothing. Node.chunks: list[bytes] gives each article its order.
Node gains language: str = "", inherited down the tree like banner (empty = nearest ancestor, front page last, site default en final). select_language and <html lang> use the resolved value instead of the global ORIGINAL_LANGUAGE constant.
- Known weakness: changing a page's (or subtree's) language after translations exist mis-keys everything — translations are keyed by source chunks, so old entries silently stop matching and user patches (searching for old-hybrid text) mostly go stale. That is acceptable: the orphaned data is harmless and translations regenerate. We do not migrate translations across a language change.
Article paths are stored and keyed without leading slashes ("docs/setup", front page ""); slashes are added only in hrefs.

Render pipeline (the phase-1 `get_translation` stub, now real)

def get_translation(path, lang, data) -> Translation | None:
    if lang not in node.langs:
        return None
    hybrid = "\n\n".join(
        chunks[h] if h in node.no_trans else trans.get(h, {}).get(lang, chunks[h])
        for h in node.chunks
    )
    for patch in data.patches.get(f"{path}:{lang}", []):
        hybrid = apply_patch(hybrid, patch)
    return Translation(markdown=hybrid, titles=title_map(data, lang))

Availability is an article-level index: node.langs: dict[lang, True], maintained by the translation writers (translator job, patch saves) in the same transaction as their data writes — rendering and language selection never probe the trans store chunk by chunk. A stale key is benign (the "translation" just renders as the original).
titles for nav/sidebar/cards: each node's translated title is trans.get(hash(node.title), {}).get(lang) with per-node fallback — one dict lookup per nav item at render time.
Cache invalidation: writes to chunks / trans / patches (translator, editor saves) call _invalidate_pages(), same as content writes.

Editor flow

The page editor has a language picker (flag + name; the same country-flag-icons set as the analytics visitor cells) listing the primary language and the union of the page's translations (node.langs) and the site-wide translate_langs. It opens in the language the page was served in (<html lang>). A note under the toolbar states the blast radius: edits to the primary language re-chunk the original (invalidating the affected translation fragments everywhere); edits to a translation stay local to that language.

WS open with a lang returns the effective hybrid Markdown and title for that language (ungated by node.langs — a language without any fragments yet starts from the original text), plus the language metadata (lang, primary_lang, langs, translate_langs).
The editor keeps a shadow copy of the Markdown it opened. WS save with lang sends it as base; the server diffs base → submitted text (make_patch) and appends a Patch. Diffing against the shadow (rather than the current hybrid) keeps hunks correct when the original or the machine translation moved under an open editor; application against the then-current hybrid stays best-effort per hunk, as designed.
A changed title on a translated save becomes a fragment in Data.trans keyed by the original title's chunk hash — the same storage as machine title translations. An untouched title field (holding the served translation) is not sent, so saving never freezes a stale machine title into an override.
Saving never deletes; a translation additionally cannot be emptied (that would render as a blank page in that language).
The live preview renders the version being edited, whichever language the page itself was loaded in (the render is just the edited Markdown + title). A translated save keeps that preview in place — re-fetching the page would come back in the header-selected language.
Saving the primary-language version re-chunks the submitted Markdown and updates Data.chunks / node.chunks — only genuinely new text lands in the kanta change diff (see docs/migrate.md).

Translator service API

An external machine-translation service connects over WebSocket at /_translate/{key} — deliberately not under /_api: the SSO forward-auth does not cover that route, and the key in the path is the access control. Keys live in Data.translate_keys (key -> display name) — 12 lowercase alphanumeric characters each, the first one generated at database bootstrap and multiple keys reserved for future management (e.g. a web UI). The full WS URL(s) are printed in the startup log (ws://localhost:{port}/_translate/{key} locally, wss://{hostname}/_translate/{key} on a public hostname) and the keys are surfaced to the admin in GET /_api/settings as translate_keys. An unknown or empty key rejects the handshake (close-before-accept → HTTP 403). Transactions storing results record the connecting key as the kanta transaction user.

Frames are JSON-encoded tagged msgspec structs (pagerite/translate.py; bytes fields ride as base64):

{"type": "hello", "langs": [...]} — client greeting announcing its capabilities: the language codes its model can produce (normalized to base subtags; en/empty dropped).
{"type": "job", "lang", "key", "text", "path", "kind"} — server push: ONE fragment to translate (an article title or a chunk), its text masked (see Masking below).
{"type": "result", "lang", "key", "text"} — client reply: the translation of the connection's current job, matching it by (lang, key).

Which languages get translated is server-configured: Data.translate_langs (presence-key dict, read/set via /_api/settings as translate_langs; no editing UI yet). The dispatcher offers a connection jobs only in wanted ∩ capable; a connection without overlap simply stays idle.

Dispatch semantics (all in app.py):

One job at a time per connection — the next job is sent only after the current one's result. Clients wanting parallelism open multiple connections (e.g. several scripts/translator.py instances).
Pending work is derived from the trans store (translate.pending_items) minus the items in flight on any connection, so a disconnect requeues that connection's in-flight item and it is offered to any free capable connection.
Dispatch re-runs on every relevant event: Hello, result, disconnect and content change (_invalidate_pages() schedules it, so the pass runs after the writing transaction commits).
A result with no job in flight, a mismatched (lang, key), a duplicate hello, or any malformed frame closes the socket with a protocol error.

Results are stored into trans in one transaction and set node.langs[lang] on every article they touch (shared chunks make several pages gain a language from one fragment). Unknown keys are stored anyway and re-storing overwrites — results are idempotent.

Masking

Fragments cross the wire masked (pagerite/masking.py): spans the model must copy byte-identically are replaced with numbered ⟦N⟧ sentinels before dispatch and restored by number from the result. Masked: code spans, container-fence names, link and image destinations (link text, alt text and captions stay visible for translation), reference and footnote labels, {...} spans (placeholders like {dates} as well as attrs), inline HTML tags and bare URLs. Markdown punctuation (*, |, [](), :::) is not masked — it carries no lexical content and models preserve it. Chunks with no prose left after masking (a lone {dates}, container fences, pure code/HTML) are never dispatched at all (needs_translation); every language renders them from the original chunk.

A result is accepted only if every sentinel survived exactly once, in any order (translations legitimately reorder spans). A mangled result is dropped and logged, and the (lang, key) pair is skipped for the rest of the server run — generation is near-deterministic, so an immediate retry would re-fail the same way; the fragment stays pending and gets another chance on restart or a model/masking change. Data.trans therefore only ever holds clean, unmasked text.

Explicitly out of scope for phase 2

The machine translation itself: the API above moves fragments in and out; the translating is external. scripts/translator.py is the reference client (Seed-X-PPO-7B only — its 28 languages are the ceiling).
Garbage collection of orphaned chunks/translations (see docs/migrate.md).
sitemap.xml per-language entries; translated UI chrome; per-language typographer options; multi-locale date/number formatting.

17 KiB Raw Blame History