LeoVasanko/pagerite

Fork 0

Files

T

LeoVasanko 10c4c9c4f9 Translator service API: GET/POST /_api/translate/{lang}

2026-09-02 02:16:28 +00:00

10 KiB

Raw Blame History

Localization

Pages are served in the visitor's language based on a ?lang= query parameter or the Accept-Language header.

Phase 1 (implemented): negotiation, URL scheme, caching, rendering plumbing. Translations are consumed through a stub interface; the database still holds only the original language.
Phase 2 (implemented): gettext-style fragment storage in the database — machine-translated chunks plus user override patches, assembled at render time. Storage details in docs/migrate.md.

Phase 1: negotiation and URLs

Language selection

Deliberately simple — q-values are ignored:

All known Accept-Language implementations send the header in order of preference, so we parse it as an ordered list and never reorder.
Selection rule (select_language in pagerite/i18n.py):
1. If ?lang=<tag> is present, use it (if a translation exists; otherwise fall through to header logic).
2. If the article's original language appears anywhere in the header list, use the original. Rationale: an AI translation is strictly worse than the original for anyone who has that language configured at all (e.g. fi-FI, fi, en-US, en gets English, not machine-translated Finnish).
3. Otherwise walk the header list in order and use the first language for which a translation exists.
4. Fall back to the original.

Region tags normalize to their base subtag (fi-FI → fi).

URLs: pretty for users, indexable for search engines

Canonical URLs stay pretty (/some-page). Each language version is addressable as /some-page?lang=fi so search engines can index them.
<link rel="canonical"> points to the page itself including the query (each language version is its own canonical).
<link rel="alternate" hreflang="…"> entries point to every other language version (with ?lang=), plus x-default for the plain URL.
On page load, pagerite.js removes the ?lang= query via history.replaceState (restoring the pretty URL) but remembers the language in a JS variable. All fetch-navigation and preloads it performs afterwards send that language in the Accept-Language header, so the chosen language sticks for the session of clicks.
A full page refresh or a shared link resets to automatic selection (header only). This gives a clean one-time override without cookies.

Response correctness

Content responses carry Vary: accept-language (added to the existing accept-encoding vary).
_cached_body and the page ETag include the selected language (not the raw header, which would blow up the cache key space).
<html lang="…"> reflects the served language.

Rendering

The translated Markdown goes through the same markdown.render pipeline.
Navigation/sidebar titles come from the translation's title map, with per-node fallback to the original title (a partially translated tree must still render).
Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
The markdown typographer (SmartyPants) is English-centric; per-language typographer options are a possible follow-up, not blocking.

Phase 2: fragment-based translation storage (implemented)

Phase 1 assumed whole-page translated Markdown delivered from outside. The refined model is gettext-style: an article has one primary version (its content, in its own language) plus, per target language, machine fragments (translated chunks of Markdown) and user patches (minimal editor overrides). Both are stored in the database and assembled into the served Markdown at render time.

The scenario this must handle

Article written in English.
Machine-translated into Spanish → fragments stored.
Editor fixes one Spanish paragraph and changes a link elsewhere to point at a Spanish resource → user patch hunks stored.
English article edited → the edited chunk's key changes; its Spanish fragment no longer matches.
Page requested before the machine translation refreshes → served as a hybrid: old fragments for unchanged chunks, plain English for the edited chunk. User patches are attempted against this hybrid, best effort, each hunk independently: the text fix is stale (its search text no longer exists) and silently skipped; the link change still applies even though the link sits in the now-English paragraph.
Machine translation refreshes → full Spanish again, with both patch hunks applying.

Chunks

chunk_markdown(markdown) splits the source into block-level chunks — blank-line-separated blocks: headings, paragraphs, code fences (kept whole), list blocks, tables, HTML blocks. A chunk's identity is its source text, gettext-msgid style:

chunk_key = blake3(normalize(chunk_text)).digest(9)  # bytes; base64 at the JSON level

(normalize: strip trailing whitespace per line, collapse surrounding blank lines — so whitespace-only source edits don't invalidate translations.)

Consequences:

Editing the English source invalidates exactly the edited chunks; all other fragments keep applying. Stale fragments are simply never referenced again and can be garbage-collected lazily (or left; they are tiny).
No explicit "source version" bookkeeping is needed — staleness falls out of the keys.

User patches

Editors always edit full Markdown in the existing editor UX — never fragments. When editing a translated view (?lang=es), the editor is loaded with the current hybrid Markdown; on save, the server computes a minimal diff against that hybrid and stores it as a patch:

class Patch(msgspec.Struct, omit_defaults=True):
    """One editing session's overrides, applied independently per hunk."""

    hunks: list[tuple[str, str]] = []  # (search, replace) on hybrid Markdown

Hunks are produced from difflib.SequenceMatcher on the hybrid vs. the edited text at block granularity: each replace/delete/insert opcode becomes one (search, replace) pair, with the preceding block's tail as left context for insert (pure inserts have empty search context otherwise). Application is dead simple:

def apply_patch(hybrid: str, patch: Patch) -> str:
    for search, replace in patch.hunks:
        if search and search in hybrid:
            hybrid = hybrid.replace(search, replace, 1)
        # missing search text = stale hunk -> silently skipped
    return hybrid

Per-hunk independence is the robustness property from the scenario: a stale text fix does not block a still-valid link change. Patches are stored as an ordered list and applied in order.

Storage

Full storage design and the migrate_v3 restructuring live in docs/migrate.md. The short version, as it concerns this document:

Originals and translations are content-addressed text chunks in flat stores: Data.chunks: dict[bytes, str] and Data.trans: dict[bytes, dict[str, str]] (chunk hash → lang → text) — path-independent, so repeated paragraphs and menu titles are translated once and article moves touch nothing. Node.chunks: list[bytes] gives each article its order.
Node gains language: str = "", inherited down the tree like banner (empty = nearest ancestor, front page last, site default en final). select_language and <html lang> use the resolved value instead of the global ORIGINAL_LANGUAGE constant.
- Known weakness: changing a page's (or subtree's) language after translations exist mis-keys everything — translations are keyed by source chunks, so old entries silently stop matching and user patches (searching for old-hybrid text) mostly go stale. That is acceptable: the orphaned data is harmless and translations regenerate. We do not migrate translations across a language change.
Article paths are stored and keyed without leading slashes ("docs/setup", front page ""); slashes are added only in hrefs.

Render pipeline (the phase-1 `get_translation` stub, now real)

def get_translation(path, lang, data) -> Translation | None:
    if lang not in node.langs:
        return None
    hybrid = "\n\n".join(
        chunks[h] if h in node.no_trans else trans.get(h, {}).get(lang, chunks[h])
        for h in node.chunks
    )
    for patch in data.patches.get(f"{path}:{lang}", []):
        hybrid = apply_patch(hybrid, patch)
    return Translation(markdown=hybrid, titles=title_map(data, lang))

Availability is an article-level index: node.langs: dict[lang, True], maintained by the translation writers (translator job, patch saves) in the same transaction as their data writes — rendering, language selection and hreflang never probe the trans store chunk by chunk. A stale key is benign (the "translation" just renders as the original).
titles for nav/sidebar/cards: each node's translated title is trans.get(hash(node.title), {}).get(lang) with per-node fallback — one dict lookup per nav item at render time.
Cache invalidation: writes to chunks / trans / patches (translator, editor saves) call _invalidate_pages(), same as content writes.

Editor flow

GET of page Markdown for editing with a lang parameter returns the hybrid (not the raw original) when the article has that language.
PUT/WS save with lang does not touch node.chunks; it diffs against the hybrid that was served and appends a Patch. (Serve a hybrid generation token with the editor payload so a save based on a stale hybrid can be rebased or rejected — simplest: recompute the diff against the current hybrid and accept best-effort, matching the patch philosophy.)
Saving the primary-language version re-chunks the submitted Markdown and updates Data.chunks / node.chunks — only genuinely new text lands in the kanta change diff (see docs/migrate.md).

Explicitly out of scope for phase 2

The machine translation itself: chunking output goes in, translated chunks come back. The service API exists — GET /_api/translate/{lang} lists pending items ({"key", "text", "path", "kind"}, key = base64 chunk hash), POST /_api/translate/{lang} stores a batch ({"items": [{key, text}]}) into trans and maintains node.langs; an external service does the actual translating (gated by the /_api forward-auth like everything else).
Garbage collection of orphaned chunks/translations (see docs/migrate.md).
sitemap.xml per-language entries; translated UI chrome; per-language typographer options; multi-locale date/number formatting.

10 KiB Raw Blame History