Localization (#1)
Implement comprehensive content localization, admin panels for editing each language, AI translation interface with automatic updates when base language version is changed. - SEO tags for all language URLs - Uses accept-language by default, ?lang=en overrides temporarily - User edits patched on top of translations - RTL language supportReviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
+8
-3
@@ -4,17 +4,22 @@ The Vue editor is a single tabbed `EditorShell.vue` mounted in a host div create
|
||||
|
||||
## Tabs
|
||||
|
||||
The shell hosts four kept-alive tabs (ordered site-wide first — site, structure — then, after a visual break, the per-page tabs — article, banner):
|
||||
The shell hosts five kept-alive tabs (ordered site-wide first — site, structure, localization — then, after a visual break, the per-page tabs — article, banner):
|
||||
|
||||
- `PageEditor.vue` — CodeMirror + server-rendered preview over WebSocket `/_api/ws/editor`, previewing into the visible article; editor and article scrolls are linked piecewise-linearly, keyed on the section anchors' `data-line` (markdown source line the backend stamps on top-level anchored h1/h2s): the page follows the cursor (fractional, wrap-aware, scrolling only when the cursor's page position leaves the viewport, with an edge margin), the editor follows page scroll with a progress-based viewport anchor, applied instantly (the window keeps scrolling normally while any editor is open — the panel is fixed to the viewport's left edge, its top tracking the banner's bottom edge until the banner scrolls away — and the panel scrolls internally); anchored h2s carry their own edit pens that open the editor scrolled to that section; a format bar offers Markdown helpers — bold/italic/code/link/table/image upload (always block-level on a fresh blank-separated line of its own — a cursor on a non-empty line, e.g. inside an existing image tag, inserts after that line, never into it; always with an empty `""` caption, cursor inside the quotes), toggling fences (` ``` ` code blocks and `::: aside` containers share the same machinery: clicked inside one they remove it and select the content, otherwise they wrap the selection or the cursor's line, keeping it selected), and `.left`/`.right`/`.wide`/`.margin` placement toggles plus `.small`/`.large`/`.huge` text-size toggles (brace attributes on the block at the cursor, mutually exclusive within each group; on `:::` containers a placement class replaces the container name instead — `::: aside` → `::: margin`), with Ctrl/Cmd-B/I/S bindings — for the hard-to-remember syntax. Edits content and title only, never the path.
|
||||
- `BannerEditor.vue` — per-page banner HTML + banner design selector, previewed into `#page-banner`.
|
||||
- `SiteEditor.vue` — site brand + optional custom brand HTML with image/video upload + theme selector + page-transition selector + font picker + favicon upload — clicking the preview tile picks a new one — + site-wide custom CSS, CSS injected into `<head id="pagerite-user">`.
|
||||
- `StructureEditor.vue` — the vue-draggable structure tree with always-editable title/slug inputs per row.
|
||||
- `StructureEditor.vue` — the vue-draggable structure tree with always-editable title/slug inputs per row, plus a per-row flag dropdown setting the page's primary language (`Node.language`, inherited by the subtree).
|
||||
- `LocalizationEditor.vue` — the site-wide translation settings: target languages as a flag grid (toggles, grouped in geographic rows; see docs/localization.md), the refresh-all-translations button, and the translator service WebSocket URL(s) to connect `scripts/translator.py` to.
|
||||
|
||||
Media uploads everywhere use the image icon buttons (pasting into the editor works too). The article, banner and site-settings pens are shorthands that open the shell on the matching tab; once open, clicking a pen switches tabs (and retargets the editors to the current page) instead of closing/remounting. The close button in the tab bar closes the shell (deliberately NOT Escape — it fired too easily by accident); tabs have no close buttons of their own. Closing only HIDES the shell — the Vue app stays mounted, so page-editor state (unsaved text included) survives until a real page reload; the editor always follows the URL, so fetch-navigating with the shell open (or before re-opening it) retargets it to the new page — unsaved text is stashed per path for the session and restored when returning, cleared on save. Saving there is explicit (Ctrl+S) and refreshes the page regions in place. Admin panels never reload the page.
|
||||
|
||||
In-place page re-rendering shared by the banner/site/structure tabs lives in `swapdoc.js` (`runScripts`/`loadPlain`: fetch a page, swap the dynamic regions, replaceState). It also exports `dropPageCache`, which the editor tabs call after any save that can alter the rendered HTML of other pages (theme, headings, structure, banners, site brand/CSS, favicon). Dropping the cache while editing avoids re-fetching every page immediately; the public runtime re-preloads visible links once the editor panel closes.
|
||||
|
||||
The page and structure tabs share one language selector: `LangSelect.vue` (small flag + dropdown) v-modeled on the shell-wide selection in `editorLang.js` (`''` = primary). While the panel is open that selection overrides the page's normal language preferences: EditorShell calls `swapdoc.setLangOverride`, which pins every `loadPlain` fetch (`?lang=`, the primary by its own code) and pagerite.js's own fetches/prefetches (`pagerite:session-lang`), until the panel closes and the override clears.
|
||||
|
||||
All WebSockets (page/banner editors, analytics view, the pagerite.js activity channel) pace their connections through `reconnect.js`: new sockets are created a staggered slot apart (a page load opens Vite's HMR socket plus several of ours at the same moment, and such bursts — like rapid retries — trip the browser's WebSocket throttling, leaving every socket to the host "pending" for minutes), a watchdog closes sockets stuck CONNECTING so they reschedule instead of hanging forever, and retries follow an exponential backoff with jitter that only a healthy connection resets. While a socket is connecting or waiting to reconnect the panel says so (`ConnNote.vue`), and the CodeMirror editors stay locked until their document arrives (typing before the doc accept would be clobbered by it).
|
||||
|
||||
## Saving behavior
|
||||
|
||||
Everything saves immediately as you edit (brand/title/CSS debounced, slug on commit since it renames the path), theme change swaps the stylesheet in place, tree rows navigate in place without transitions when focused, and the front page is a root-only row whose empty slug is editable like any other. Saves that can affect other pages drop the prefetch cache; the cache is rebuilt when the editor panel closes so navigation stays instant.
|
||||
@@ -25,6 +30,6 @@ Dropping ON the lower part of a row moves the page under that row (the child lis
|
||||
|
||||
The shell is dynamic-imported onto the content page by pagerite.js when an edit pen is clicked (the pens are injected by pagerite.js after the session validates; they carry `data-editor-src`/`data-editor-css`/`data-editor-mode`). In dev, modules load from the Vite dev server (`PAGERITE_VITE_URL`), in prod from the hashed build assets resolved via `frontend-build/.vite/manifest.json`.
|
||||
|
||||
`vite.config.js` sets `appType: 'mpa'` (no SPA fallback) and builds with `manifest: true`, `assetsDir: '_/assets'` (so the build mirrors the URL space; `frontend/public/favicon.ico` lands at the build root and is served at `/favicon.ico`). JS inputs are `src/main.js` and `src/pagerite.js`, plus `src/assets/pagerite.css` as a separate stylesheet entry; theme, banner-design and transition CSS are NOT built — they live in `pagerite/themes/{name}/` and are served by the backend. There is no `index.html` source (it would shadow `/` and turn missing dev paths into an empty Vue shell). All outputs are ES modules. The build sets `preserveEntrySignatures: 'exports-only'` because main.js is consumed via dynamic `import()` for its `openEditor`/`closeEditor` exports — Vite app builds otherwise strip unused entry exports, leaving dead edit pens. In dev the backend links theme/banner-design stylesheets like in prod (`/_themes/...`); only the base CSS is Vite-injected from JS, and pagerite.js then re-appends the `#pagerite-theme`/`#pagerite-banner`/`#pagerite-transition`/`#pagerite-user` elements to restore the canonical order (base < theme < design < transition < custom CSS). In production all page assets are inlined instead (styles as `<style id="pagerite-…">` in `<head>`, scripts at the end of the body). Theme switches in the site editor swap the `#pagerite-theme` element in place — the link href in dev, the inline style's text (fetched from `/_themes/...`) in prod.
|
||||
`vite.config.js` sets `appType: 'mpa'` (no SPA fallback) and builds with `manifest: true`, `assetsDir: '_assets'` (so the build mirrors the URL space; `frontend/public/favicon.ico` lands at the build root and is served at `/favicon.ico`). JS inputs are `src/main.js`, `src/pagerite.js` and `src/analytics-main.js`, plus `src/assets/pagerite.css` as a separate stylesheet entry; theme, banner-design and transition CSS are NOT built — they live in `pagerite/themes/{name}/` and are served by the backend. There is no `index.html` source (it would shadow `/` and turn missing dev paths into an empty Vue shell). All outputs are ES modules. The build sets `preserveEntrySignatures: 'exports-only'` because main.js is consumed via dynamic `import()` for its `openEditor`/`closeEditor` exports — Vite app builds otherwise strip unused entry exports, leaving dead edit pens. In dev the backend links theme/banner-design stylesheets like in prod (`/_themes/...`); only the base CSS is Vite-injected from JS, and pagerite.js then re-appends the `#pagerite-theme`/`#pagerite-banner`/`#pagerite-transition`/`#pagerite-user` elements to restore the canonical order (base < theme < design < transition < custom CSS). In production all page assets are inlined instead (styles as `<style id="pagerite-…">` in `<head>`, scripts at the end of the body). Theme switches in the site editor swap the `#pagerite-theme` element in place — the link href in dev, the inline style's text (fetched from `/_themes/...`) in prod.
|
||||
|
||||
`vite-plugin-fastapi.js` has an auto-upgrade marker — edit `vite.config.js`, not the plugin.
|
||||
|
||||
@@ -0,0 +1,474 @@
|
||||
# Localization
|
||||
|
||||
Pages are served in the visitor's language based on a `?lang=` query
|
||||
parameter or the `Accept-Language` header.
|
||||
|
||||
- **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering
|
||||
plumbing. Translations are consumed through a stub interface; the database
|
||||
still holds only the original language.
|
||||
- **Phase 2 (implemented):** gettext-style fragment storage in the
|
||||
database — machine-translated chunks plus user override patches, assembled
|
||||
at render time. Storage details in `docs/migrate.md`.
|
||||
|
||||
## Phase 1: negotiation and URLs
|
||||
|
||||
### The primary language
|
||||
|
||||
Each article has a primary (original) language: `Node.language`, inherited
|
||||
down the tree like `banner` — "" = the nearest ancestor's, the front page
|
||||
last (it doubles as the site default), with `en` as the final fallback
|
||||
(`ORIGINAL_LANGUAGE`, `primary_lang()` in `pagerite/i18n.py`). It is
|
||||
configured per row in the structure editor. Everything per-article keys
|
||||
off the resolved value: language selection, `<html lang>`, canonical URLs,
|
||||
what counts as a translation, and the translation targets (a node's own
|
||||
primary is never one — so the target set may include the site default, and
|
||||
a page in another language can be translated into it).
|
||||
|
||||
### Language selection
|
||||
|
||||
Deliberately simple — **q-values are ignored**:
|
||||
|
||||
- All known `Accept-Language` implementations send the header **in order of
|
||||
preference**, so we parse it as an ordered list and never reorder.
|
||||
- Selection rule (`select_language` in `pagerite/i18n.py`):
|
||||
1. If `?lang=<tag>` is present, use it (if a translation exists; otherwise
|
||||
fall through to header logic).
|
||||
2. If the article's original language appears anywhere in the header list,
|
||||
use the **original**. Rationale: an AI translation is strictly worse
|
||||
than the original for anyone who has that language configured at all
|
||||
(e.g. `fi-FI, fi, en-US, en` gets English, not machine-translated
|
||||
Finnish).
|
||||
3. Otherwise walk the header list in order and use the first language for
|
||||
which a translation exists.
|
||||
4. Fall back to the original.
|
||||
|
||||
Region tags normalize to their base subtag (`fi-FI` → `fi`).
|
||||
|
||||
### URLs: pretty for users, indexable for search engines
|
||||
|
||||
- Canonical URLs stay pretty (`/some-page`). Each language version is
|
||||
addressable as `/some-page?lang=fi` so search engines can index them.
|
||||
- `<link rel="canonical">` names the **actually served language**: the plain
|
||||
URL when serving the original (for SEO the non-query URL means the
|
||||
article's own language), `?lang=xx` when serving a translation — however
|
||||
the language was arrived at (query or header).
|
||||
- `<link rel="alternate" hreflang="…">` entries follow the canonical
|
||||
directly (before the social meta tags) and are the same set on every
|
||||
page — the site-wide configured languages (`translate_langs`, which the
|
||||
translator works to fill in): `x-default` first, pointing at the plain
|
||||
autodetecting URL, then every language explicitly with `?lang=`, the
|
||||
page's own primary language included.
|
||||
- The override sticks for the session of clicks: a page requested with
|
||||
`?lang=` replicates the query onto the navigation links it renders (nav,
|
||||
sidebar, cards, brand — in-article links are content and stay as
|
||||
authored), so plain clicks and no-JS navigation keep the language.
|
||||
pagerite.js additionally strips the query from the address bar via
|
||||
`history.replaceState` (pretty, shareable URLs), remembers the language,
|
||||
and adds it to every internal fetch that lacks one (preloads,
|
||||
fetch-navigations, history traversals); history entries stay query-less.
|
||||
- A full page refresh or a shared link resets to automatic selection (header
|
||||
only). This gives a clean one-time override without cookies.
|
||||
|
||||
### Response correctness
|
||||
|
||||
- Content responses carry `Vary: accept-language` (added to the existing
|
||||
`accept-encoding` vary).
|
||||
- `_cached_body` and the page ETag include the **selected language** (not the
|
||||
raw header, which would blow up the cache key space) and the **replicated
|
||||
link language**: a `?lang=fi` render and a header-selected Finnish render
|
||||
of the same page differ in their navigation links, so they are cached as
|
||||
separate variants.
|
||||
- `<html lang="…">` reflects the served language, and an RTL language
|
||||
(`i18n.RTL_LANGUAGES` — ar, fa, he, ur) also sets `dir="rtl"` on `<html>`
|
||||
(the editor panel carries its own `lang="en" dir="ltr"` so it stays LTR).
|
||||
Client-side page swaps (fetch navigation in pagerite.js, editor re-renders
|
||||
in swapdoc.js) copy both attributes from the fetched document, so a hot
|
||||
switch into or out of an RTL page flips the layout without a reload.
|
||||
|
||||
### Rendering
|
||||
|
||||
- The translated Markdown goes through the same `markdown.render` pipeline.
|
||||
- Navigation/sidebar titles come from the translation's title map, with
|
||||
per-node fallback to the original title (a partially translated tree must
|
||||
still render).
|
||||
- Category placeholder pages (the 404s for content-less labels) select a
|
||||
language like content pages, but over the **subtree's** combined
|
||||
availability (`subtree_languages`) — they have no chunks of their own;
|
||||
the heading, navigation and card text localize from the title map and
|
||||
the target articles' translations.
|
||||
- Card descriptions and cover picks run on the target article's hybrid
|
||||
Markdown where that page is available in the served language, with
|
||||
per-card fallback to the original.
|
||||
- Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
|
||||
- The markdown typographer (SmartyPants) is English-centric; per-language
|
||||
typographer options are a possible follow-up, not blocking.
|
||||
|
||||
## Phase 2: fragment-based translation storage (implemented)
|
||||
|
||||
Phase 1 assumed whole-page translated Markdown delivered from outside. The
|
||||
refined model is gettext-style: an article has **one primary version** (its
|
||||
`content`, in its own language) plus, per target language, **machine
|
||||
fragments** (translated chunks of Markdown) and **user patches** (minimal
|
||||
editor overrides). Both are stored in the database and assembled into the
|
||||
served Markdown at render time.
|
||||
|
||||
### The scenario this must handle
|
||||
|
||||
1. Article written in English.
|
||||
2. Machine-translated into Spanish → fragments stored.
|
||||
3. Editor fixes one Spanish paragraph and changes a link elsewhere to point
|
||||
at a Spanish resource → user patch hunks stored.
|
||||
4. English article edited → the edited chunk's key changes; its Spanish
|
||||
fragment no longer matches.
|
||||
5. Page requested before the machine translation refreshes → served as a
|
||||
**hybrid**: old fragments for unchanged chunks, plain English for the
|
||||
edited chunk. User patches are attempted against this hybrid, best effort,
|
||||
each hunk independently: the text fix is stale (its search text no longer
|
||||
exists) and silently skipped; the link change still applies even though
|
||||
the link sits in the now-English paragraph.
|
||||
6. Machine translation refreshes → full Spanish again, with both patch hunks
|
||||
applying.
|
||||
|
||||
### Chunks
|
||||
|
||||
`chunk_markdown(markdown)` splits the source into block-level chunks —
|
||||
blank-line-separated blocks: headings, paragraphs, code fences (kept whole),
|
||||
list blocks, tables, HTML blocks. A chunk's identity is its **source text**,
|
||||
gettext-msgid style:
|
||||
|
||||
```python
|
||||
chunk_key = blake3(normalize(chunk_text)).digest(9) # bytes; base64 at the JSON level
|
||||
```
|
||||
|
||||
(`normalize`: strip trailing whitespace per line, collapse surrounding blank
|
||||
lines — so whitespace-only source edits don't invalidate translations.)
|
||||
|
||||
Consequences:
|
||||
|
||||
- Editing the English source invalidates exactly the edited chunks; all
|
||||
other fragments keep applying. Stale fragments are simply never referenced
|
||||
again and can be garbage-collected lazily (or left; they are tiny).
|
||||
- No explicit "source version" bookkeeping is needed — staleness falls out
|
||||
of the keys.
|
||||
|
||||
### User patches
|
||||
|
||||
Editors always edit **full Markdown** in the existing editor UX — never
|
||||
fragments. When editing a translated view (`?lang=es`), the editor is loaded
|
||||
with the *current hybrid Markdown*; on save, the server computes a minimal
|
||||
diff against that hybrid and stores it as a patch:
|
||||
|
||||
```python
|
||||
class Patch(msgspec.Struct, omit_defaults=True):
|
||||
"""One editing session's overrides, applied independently per hunk."""
|
||||
|
||||
hunks: list[tuple[str, str]] = [] # (search, replace) on hybrid Markdown
|
||||
```
|
||||
|
||||
Hunks are produced from `difflib.SequenceMatcher` on the hybrid vs. the
|
||||
edited text at block granularity: each `replace`/`delete`/`insert` opcode
|
||||
becomes one `(search, replace)` pair, with the preceding block's tail as
|
||||
left context for `insert` (pure inserts have empty search context otherwise).
|
||||
Application is dead simple:
|
||||
|
||||
```python
|
||||
def apply_patch(hybrid: str, patch: Patch) -> str:
|
||||
for search, replace in patch.hunks:
|
||||
if search and search in hybrid:
|
||||
hybrid = hybrid.replace(search, replace, 1)
|
||||
# missing search text = stale hunk -> silently skipped
|
||||
return hybrid
|
||||
```
|
||||
|
||||
Per-hunk independence is the robustness property from the scenario: a stale
|
||||
text fix does not block a still-valid link change. Patches are stored as an
|
||||
ordered list and applied in order.
|
||||
|
||||
### Storage
|
||||
|
||||
Full storage design and the `migrate_v3` restructuring live in
|
||||
`docs/migrate.md`. The short version, as it concerns this document:
|
||||
|
||||
- Originals **and** translations are content-addressed text chunks in flat
|
||||
stores: `Data.chunks: dict[bytes, str]` and
|
||||
`Data.trans: dict[bytes, dict[str, str]]` (chunk hash → lang → text) —
|
||||
path-independent, so repeated paragraphs and menu titles are translated
|
||||
once and article moves touch nothing. `Node.chunks: list[bytes]` gives
|
||||
each article its order.
|
||||
- `Node` gains **`language: str = ""`**, inherited down the tree like
|
||||
`banner` (empty = nearest ancestor, front page last, site default `en`
|
||||
final). `select_language` and `<html lang>` use the resolved value instead
|
||||
of the global `ORIGINAL_LANGUAGE` constant.
|
||||
- **Known weakness:** changing a page's (or subtree's) `language` after
|
||||
translations exist mis-keys everything — translations are keyed by
|
||||
*source* chunks, so old entries silently stop matching and user patches
|
||||
(searching for old-hybrid text) mostly go stale. That is acceptable:
|
||||
the orphaned data is harmless and translations regenerate. We do not
|
||||
migrate translations across a language change.
|
||||
- Article paths are stored and keyed **without leading slashes**
|
||||
(`"docs/setup"`, front page `""`); slashes are added only in hrefs.
|
||||
|
||||
### Render pipeline (the phase-1 `get_translation` stub, now real)
|
||||
|
||||
```python
|
||||
def get_translation(data, path, lang) -> Translation | None:
|
||||
if lang not in node.langs:
|
||||
return None
|
||||
hybrid = "\n\n".join(
|
||||
chunks[h] if h in node.no_trans else trans.get(h, {}).get(lang, chunks[h])
|
||||
for h in node.chunks
|
||||
)
|
||||
for patch in data.patches.get(f"{path}:{lang}", []):
|
||||
hybrid = apply_patch(hybrid, patch)
|
||||
return Translation(markdown=hybrid, titles=title_map(data, lang))
|
||||
```
|
||||
|
||||
- Availability is an article-level index: `node.langs: dict[lang, True]`,
|
||||
maintained by the translation writers (translator job, patch saves) in the
|
||||
same transaction as their data writes — rendering and language selection
|
||||
never probe the `trans` store chunk by chunk. A stale key is benign (the
|
||||
"translation" just renders as the original).
|
||||
- `titles` for nav/sidebar/cards: each node's translated title is
|
||||
`trans.get(hash(node.title), {}).get(lang)` with per-node fallback — one
|
||||
dict lookup per nav item at render time.
|
||||
- Cache invalidation: writes to `chunks` / `trans` / `patches` (translator,
|
||||
editor saves) call `_invalidate_pages()`, same as content writes.
|
||||
|
||||
### Editor flow
|
||||
|
||||
The page and structure editors share one language selector (`LangSelect.vue`:
|
||||
a small flag button opening a dropdown; the same country-flag-icons set as
|
||||
the analytics visitor cells), v-modeled on one shell-wide selection
|
||||
(`editorLang.js`, `''` = the primary language). The page editor lists the
|
||||
page's own primary language (`Node.language`, resolved through the
|
||||
hierarchy and echoed in the WS doc as `primary_lang`) plus the union of
|
||||
the page's translations (`node.langs`) and
|
||||
the site-wide `translate_langs`; it always opens in the primary language,
|
||||
even when the page itself was served in a translation. A note under the
|
||||
toolbar states the blast radius:
|
||||
edits to the primary language re-chunk the original (invalidating the
|
||||
affected translation fragments everywhere); edits to a translation stay
|
||||
local to that language.
|
||||
|
||||
While the editor panel is open, its language selection **overrides the
|
||||
normal language preferences** for the page preview: EditorShell pins every
|
||||
in-place re-render and pagerite.js fetch/prefetch to it (`?lang=` — a
|
||||
primary selection pins by the current page's own resolved primary, which
|
||||
`select_language` honors), and closing the panel restores the normal
|
||||
preferences.
|
||||
|
||||
- WS `open` with a `lang` returns the effective **hybrid** Markdown and
|
||||
title for that language (ungated by `node.langs` — a language without
|
||||
any fragments yet starts from the original text), plus the language
|
||||
metadata (`lang`, `primary_lang`, `langs`, `translate_langs`).
|
||||
- The editor keeps a **shadow copy** of the Markdown it opened. WS `save`
|
||||
with `lang` sends it as `base`; the server diffs `base` → submitted text
|
||||
(`make_patch`) and appends a `Patch`. Diffing against the shadow (rather
|
||||
than the current hybrid) keeps hunks correct when the original or the
|
||||
machine translation moved under an open editor; application against the
|
||||
then-current hybrid stays best-effort per hunk, as designed.
|
||||
- A changed **title** on a translated save becomes a fragment in
|
||||
`Data.trans` keyed by the original title's chunk hash — the same storage
|
||||
as machine title translations. An untouched title field (holding the
|
||||
served translation) is not sent, so saving never freezes a stale machine
|
||||
title into an override.
|
||||
- Saving never deletes; a translation additionally cannot be emptied (that
|
||||
would render as a blank page in that language).
|
||||
- The live preview renders the version being edited, whichever language
|
||||
the page itself was loaded in (the render is just the edited Markdown +
|
||||
title). A translated save keeps that preview in place — re-fetching the
|
||||
page would come back in the header-selected language.
|
||||
- Saving the primary-language version re-chunks the submitted Markdown and
|
||||
updates `Data.chunks` / `node.chunks` — only genuinely new text lands in
|
||||
the kanta change diff (see docs/migrate.md).
|
||||
|
||||
The **structure editor** selects from the same languages with the same
|
||||
`LangSelect` (the selection is shared — switching in either tab switches
|
||||
both, and the preview). It is also where a page's **primary language** is
|
||||
configured: each row carries a small flag dropdown (the resolved flag,
|
||||
dimmed while inherited) that sets `Node.language` via a structure op —
|
||||
'' = inherit, so setting it on a section covers the whole subtree. The
|
||||
tree it
|
||||
lists (`GET /_api/pages?lang=`) comes back with per-language titles where a
|
||||
translation exists (`translated` marks those rows; untranslated rows show
|
||||
the original title, dimmed). Retitling in a non-primary language posts the
|
||||
structure op with a `lang` and writes a per-language title fragment in
|
||||
`Data.trans` (keyed by the original title's chunk hash, exactly like a
|
||||
machine title translation — a user edit simply overwrites it); sending the
|
||||
original's text drops the override. The structure itself — slugs,
|
||||
hierarchy, order — is language-independent, so pending rows, slug edits,
|
||||
drag-and-drop and deletes work identically in every language.
|
||||
|
||||
### Translator service API
|
||||
|
||||
An external machine-translation service connects over WebSocket at
|
||||
`/_translate/{key}` — deliberately **not** under `/_api`: the SSO
|
||||
forward-auth does not cover that route, and the key in the path is the
|
||||
access control. Keys live in `Data.translate_keys` (key -> display name) —
|
||||
12 lowercase alphanumeric characters each, the first one generated at
|
||||
database bootstrap and multiple keys reserved for future management (e.g.
|
||||
a web UI). The full WS URL(s) are printed in the startup log
|
||||
(`ws://localhost:{port}/_translate/{key}` locally,
|
||||
`wss://{hostname}/_translate/{key}` on a public hostname) and the keys are
|
||||
surfaced to the admin in `GET /_api/settings` as `translate_keys`. An
|
||||
unknown or empty key rejects the handshake (close-before-accept → HTTP
|
||||
403). Transactions storing results record the connecting key as the kanta
|
||||
transaction `user`.
|
||||
|
||||
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
|
||||
`bytes` fields ride as base64):
|
||||
|
||||
- `{"type": "hello", "langs": [...]}` — client greeting announcing its
|
||||
**capabilities**: the language codes its model can produce (normalized
|
||||
to base subtags; `en`/empty dropped).
|
||||
- `{"type": "job", "lang", "key", "texts", "path", "kind", "contexts"}` —
|
||||
server push: ONE fragment to translate (an article title or a chunk), as
|
||||
a list of **prose segments** (see Segmentation below). `contexts` is
|
||||
parallel to `texts` ("" = none): the surround to translate the segment
|
||||
in — for clients that translate better with context (see below).
|
||||
Contexts are not part of the result.
|
||||
- `{"type": "result", "lang", "key", "texts"}` — client reply: the
|
||||
segments translated, same order and count, matching its job by (lang, key).
|
||||
|
||||
Which languages get translated is **server-configured**:
|
||||
`Data.translate_langs` (presence-key dict, bootstrapped to Spanish and
|
||||
Chinese — edited in the editor shell's localization tab, whose flag grid
|
||||
lists every language including English, or set via `/_api/settings` as
|
||||
`translate_langs`). A target equal to an article's own primary language is
|
||||
skipped per article (its original already is that language), so the set
|
||||
may freely contain the site default. The dispatcher offers a
|
||||
connection jobs only in `wanted ∩ capable`; a connection without overlap
|
||||
simply stays idle.
|
||||
|
||||
`DELETE /_api/translations` (the localization tab's "refresh all
|
||||
translations" button) drops every machine translation (`Data.trans`) and
|
||||
rebuilds the availability index (`node.langs`) from the surviving user
|
||||
patches, so the dispatcher re-translates everything from scratch; the
|
||||
run's validation skip-list is cleared with it, giving rejected fragments
|
||||
another chance.
|
||||
|
||||
Dispatch semantics (the `Dispatcher` in `pagerite/translate.py`; app.py only
|
||||
registers the route):
|
||||
|
||||
- **One job at a time per connection** — the next job is sent only after
|
||||
the current one's result. Clients wanting parallelism open multiple
|
||||
connections (e.g. several `scripts/translator.py` instances).
|
||||
- Pending work is derived from the `trans` store
|
||||
(`translate.pending_items`) minus the items in flight on any connection,
|
||||
so a **disconnect requeues** that connection's in-flight item and it is
|
||||
offered to any free capable connection.
|
||||
- Dispatch re-runs on every relevant event: Hello, result, disconnect and
|
||||
content change (`_invalidate_pages()` schedules it, so the pass runs
|
||||
after the writing transaction commits).
|
||||
- A result with no job in flight, a mismatched (lang, key), a duplicate
|
||||
hello, or any malformed frame closes the socket with a protocol error.
|
||||
|
||||
Results are stored into `trans` in one transaction and set
|
||||
`node.langs[lang]` on every article they touch (shared chunks make several
|
||||
pages gain a language from one fragment). Unknown keys are stored anyway
|
||||
and re-storing overwrites — results are idempotent.
|
||||
|
||||
#### Segmentation
|
||||
|
||||
Fragments cross the wire as **prose segments** (`pagerite/segments.py`): the
|
||||
fragment is parsed with the project's own markdown-it setup
|
||||
(`markdown.make_md(verbatim=True)` — all extensions, but no typographer or
|
||||
tasklist label wrapping, so token text stays byte-identical to the source)
|
||||
and split into the runs a model may touch: paragraph/heading/table-cell text
|
||||
(merged across soft line breaks), image alt texts and captions, footnote
|
||||
bodies. A block of plain text, inline **links and paired text formatting**
|
||||
(strong/em/s) **stays whole** — link and formatted texts cross inline, in
|
||||
sentence context, with the Markdown stripped (see below). Everything else
|
||||
never leaves the server: code spans and
|
||||
fences, URLs and autolinks, link/image *destinations*, `{...}` spans
|
||||
(placeholders like `{dates}` as well as attrs), reference and footnote
|
||||
labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and the
|
||||
remaining markup punctuation (`|`, `:::`), which is a run boundary.
|
||||
Chunks with no segments (a lone `{dates}`, container fences, pure
|
||||
code/HTML) are never dispatched at all (`needs_translation`); every
|
||||
language renders them from the original chunk. Each segment is accompanied
|
||||
by a context string (a segment carved out of a larger block carries the
|
||||
block's plain text; a whole-block segment carries "") — context is a
|
||||
prompt aid only, never spliced into the result.
|
||||
|
||||
Reassembly is offset splicing, not text the model produced: each segment's
|
||||
source span was located at dispatch (sequential search; a run that is not a
|
||||
verbatim source substring — entity-decoded text, backslash escapes — is
|
||||
skipped and stays in the original language), and the returned translations
|
||||
are swapped in by offset. Markup corruption is therefore impossible by
|
||||
construction; the failure modes that remain are a wrong segment count, an
|
||||
empty segment, or markup injected INTO a segment (a `<br>` in a title
|
||||
translation would splice live HTML) — each returned segment must parse as
|
||||
pure prose, or the whole result is dropped and logged, and the (lang, key)
|
||||
pair is skipped for the rest of the server run (generation is
|
||||
near-deterministic, so an immediate retry would re-fail; the fragment stays
|
||||
pending and gets another chance on restart or `DELETE /_api/translations`).
|
||||
`Data.trans` therefore only ever holds clean translated Markdown.
|
||||
|
||||
Link- and formatting-carrying blocks are the one place a segment is not
|
||||
spliced verbatim: a label translated apart from its sentence comes back
|
||||
grammatically incompatible with it (case government, particles, word
|
||||
order), and shown the Markdown the model mangles it (Seed-X dropped the
|
||||
`**` and the glued-on colon in `**Pagerite**: …`), so the block crosses
|
||||
whole — all Markdown stripped — and the server re-inserts the link and
|
||||
formatting syntax into the translated block. The boundaries are found by
|
||||
**text processing alone** —
|
||||
markers on the wire are hopeless (an earlier sentinel-masking design let
|
||||
the model see and mangle exactly that punctuation: Seed-X renumbered the
|
||||
tokens and turned `` or `**`. Blocks mixing in any other inline
|
||||
markup (code spans, images, raw HTML) don't qualify and still split into
|
||||
runs at those boundaries.
|
||||
|
||||
Punctuation is the translator's own job: Seed-X tends to "finish" short
|
||||
labels (titles, nav items) with a comma or period the source never had.
|
||||
Prompt wording is NOT the fix — a punctuation-instruction clause made
|
||||
Seed-X slip into its `[COT]` reasoning mode (minutes-long generations with
|
||||
reasoning text in the output, observed for Chinese). The reference client
|
||||
enforces punctuation deterministically instead (`match_punctuation` in
|
||||
scripts/translator.py): a translation of a segment without terminal
|
||||
punctuation gets any added trailing marks (and a newly opened Spanish ¡/¿)
|
||||
stripped before the result goes back.
|
||||
|
||||
The same client-side enforcement covers markup bleed as a CLASS, not per
|
||||
artifact: `<` is the prose/markup boundary on the wire and never appears in
|
||||
a segment in either direction. Source pieces containing `<` are never
|
||||
dispatched (they stay in the original language — segments.py), and the
|
||||
reference client cuts the model's output at the first `<`
|
||||
(scripts/translator.py) — echoed language tags, stray `<br>`s and any
|
||||
future variant are one handled case. (The cut is post-decode, not a
|
||||
generation stop string: Seed-X opens every generation with its `<s>`
|
||||
framing token, which would trip a `<` stop immediately.)
|
||||
|
||||
Short fragments get more than a bare prompt: each segment may carry its
|
||||
surround in `Job.contexts` — a title carries the article's opening prose
|
||||
(its own block is just the title word), a segment carved out of a larger
|
||||
block (a partial run; a link text whose block didn't qualify for the
|
||||
whole-block treatment) carries the block's plain text, and a
|
||||
whole-block segment (a plain paragraph) is self-contextualizing and carries
|
||||
"". The reference client translates segment and surround together, stops
|
||||
generation at the blank line separating them, and keeps the segment's own
|
||||
part of the output (its line resp. paragraph; a hard-break `␣␣\n` separator
|
||||
works too). If the model merged them (no separator, or an empty first
|
||||
part), it falls back to translating the segment alone. The surround fixes
|
||||
context-free readings ("About" as "approximately" — with the opening it
|
||||
becomes "Tietoa"/"Acerca de"; "here" as "就在这里" → the idiomatic
|
||||
"点击这里") and, as a side effect, most stray trailing punctuation.
|
||||
|
||||
### Explicitly out of scope for phase 2
|
||||
|
||||
- The machine translation itself: the API above moves fragments in and out;
|
||||
the translating is external. `scripts/translator.py` is the reference
|
||||
client (Seed-X-PPO-7B only — its 28 languages are the ceiling).
|
||||
- Garbage collection of orphaned chunks/translations (see docs/migrate.md).
|
||||
- sitemap.xml per-language entries; translated UI chrome; per-language
|
||||
typographer options; multi-locale date/number formatting.
|
||||
+178
@@ -0,0 +1,178 @@
|
||||
# migrate_v3: content-addressed chunk storage
|
||||
|
||||
Status: **implemented**. `migrate_v3` restructures how article text and
|
||||
translations are stored, motivated by the localization model in
|
||||
`docs/localization.md` (phase 2). Since it is a full migration, it is free to
|
||||
break the current `Node.content: str | None` layout.
|
||||
|
||||
## Goals
|
||||
|
||||
- **Minimal change diffs.** kanta persists change diffs; editing one
|
||||
paragraph of a long article must not rewrite the whole article string, and
|
||||
a translation refresh must touch only the re-translated chunks.
|
||||
- **Fast, simple lookup.** Everything heavy lives in flat
|
||||
`dict[hash, content]` stores; ordering lives in `list[hash]`. No large
|
||||
nested structures, no deep paths.
|
||||
- **Path-independent text.** Chunks and their translations are keyed by
|
||||
content hash, not by article path — the same paragraph (or menu title)
|
||||
appearing in several articles is stored and translated once. Moving or
|
||||
renaming an article touches nothing.
|
||||
|
||||
## Design (chosen: global content-addressed stores)
|
||||
|
||||
Original articles are *also* stored as chunks; everything — originals and
|
||||
translations — lives in flat hash-keyed dicts. Costs accepted: rendering does
|
||||
one dict lookup per chunk (trivial), orphaned hashes need occasional garbage
|
||||
collection, and the editor save path re-chunks server-side (it already
|
||||
diffs). The rejected alternatives: per-article nested `LangVersion`
|
||||
structures (churn, duplication, whole-string originals) and a hybrid with
|
||||
whole originals plus global translations (keeps the worst change-diff
|
||||
property).
|
||||
|
||||
## Target layout
|
||||
|
||||
```python
|
||||
class Node(msgspec.Struct, omit_defaults=True):
|
||||
...
|
||||
#: Replaces `content: str | None`. None = pure category label;
|
||||
#: a list (possibly empty) = a page, as ordered chunk hashes.
|
||||
chunks: list[bytes] | None = None
|
||||
#: Primary language of the article (BCP-47 base tag). "" = inherit
|
||||
#: (nearest ancestor, front page last, site default "en" final).
|
||||
language: str = ""
|
||||
#: Chunk hashes the editor marked "do not translate" (always served
|
||||
#: from the original). Presence-keys, value always True.
|
||||
no_trans: dict[bytes, True] = {}
|
||||
#: Languages this article is available in (besides its primary
|
||||
#: language). Presence-keys, value always True — rendering, language
|
||||
#: selection and hreflang alternates read this set instead of probing
|
||||
#: the trans store chunk by chunk. Maintained by the writers (see
|
||||
#: "Language index maintenance" below).
|
||||
langs: dict[str, True] = {}
|
||||
|
||||
class Data(msgspec.Struct):
|
||||
...
|
||||
#: API keys gating the translator service WebSocket (/_translate/{key}):
|
||||
#: key -> display name; the first is generated at bootstrap (app.py).
|
||||
translate_keys: dict[str, str] = {}
|
||||
#: Wanted target languages for the translator service (presence-keys);
|
||||
#: jobs are offered only in these ∩ a connection's capabilities.
|
||||
translate_langs: dict[str, True] = {}
|
||||
#: All original-language text, content-addressed: blake3(normalized)
|
||||
#: digest[:9] -> Markdown chunk. Shared by every article. Keys are
|
||||
#: bytes; kanta/msgspec base64-encode them at the JSON level.
|
||||
chunks: dict[bytes, str] = {}
|
||||
#: Machine translations: chunk hash -> lang -> translated Markdown
|
||||
#: (nested, not tuple keys: msgspec's JSON serializer rejects them).
|
||||
#: Also used for node titles (hash of the title text).
|
||||
trans: dict[bytes, dict[str, str]] = {}
|
||||
#: User override patches per article and language:
|
||||
#: f"{path}:{lang}" -> ordered patches (see localization.md).
|
||||
patches: dict[str, list[Patch]] = {}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- **Article paths never carry a leading slash** in the DB or in lookup keys
|
||||
(`"docs/setup"`, front page `""`); the leading slash is added only when
|
||||
building hrefs. `migrate_v3` audits existing stored paths (translation
|
||||
keys, analytics references, any path-valued fields) and normalizes them.
|
||||
- **Titles are chunks too**, by hash only: the nav renderer looks up
|
||||
`trans.get(hash(node.title), {}).get(lang)`. No separate title storage;
|
||||
editing a title invalidates its translations automatically.
|
||||
- **Per-hunk options** live in two places: *inherent* options are derived at
|
||||
chunking time (code fences, HTML blocks and prose-free chunks are
|
||||
no-translate without storing anything — `needs_translation`, see
|
||||
docs/localization.md "Masking"); *editor-set* flags are `node.no_trans`
|
||||
(keyed by chunk
|
||||
hash, so a heavy edit silently drops the flag — acceptable and
|
||||
self-healing).
|
||||
- **Patch payloads stay inline** in `Patch.hunks` — patches are small by
|
||||
construction (minimal server-computed diffs). If a pathological case shows
|
||||
up, hunks can be hash-stored later without schema pain.
|
||||
|
||||
## Language index maintenance (`node.langs`)
|
||||
|
||||
`node.langs` is a denormalized index over the `trans`/`patches` stores so
|
||||
that article rendering, `select_language`'s availability check, and hreflang
|
||||
alternate links never enumerate chunks. It is written by whoever writes
|
||||
translation data, in the same transaction:
|
||||
|
||||
- **Translator service:** the WebSocket API at `/_translate/{key}` (see
|
||||
docs/localization.md) offers pending fragments (titles + translatable
|
||||
chunks lacking an entry for the language) as single-item jobs — one at
|
||||
a time per connection, in `Data.translate_langs` ∩ the connection's
|
||||
announced capabilities — and receives the matching result; storing it
|
||||
writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on
|
||||
every article that gained one and invalidates the page cache — all in
|
||||
one transaction.
|
||||
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
|
||||
sets `node.langs[lang] = True` (patches alone make the version exist).
|
||||
- **Removals:** deleting a patch or GC'ing translations re-derives the key:
|
||||
keep `lang` if any `trans` entry for the article's current chunks/title or
|
||||
any patch remains, otherwise drop it. Stale `langs` keys are benign (an
|
||||
advertised language that renders as the original), so removal can lag.
|
||||
|
||||
## Render / save pipeline (summary)
|
||||
|
||||
- **Render:** `text = "\n\n".join(chunks[h] for h in node.chunks)` for the
|
||||
original; for language `L` (only ever attempted when `L in node.langs`),
|
||||
per chunk `trans.get(h, {}).get(L)` unless missing or `h in node.no_trans`,
|
||||
falling back to `chunks[h]`; then apply `patches.get(f"{path}:{L}", [])`
|
||||
in order (per-hunk, best effort); then `markdown.render` as today. All of
|
||||
this assembles the `Translation` the phase-1 plumbing already consumes.
|
||||
- **Availability:** `node.langs` is the availability index; `?lang=`
|
||||
handling uses exactly this set. (hreflang alternates are site-wide from
|
||||
`translate_langs` instead — see docs/localization.md.)
|
||||
- **Save (primary language):** server re-chunks the submitted Markdown,
|
||||
inserts new hashes into `Data.chunks`, replaces `node.chunks`. Unchanged
|
||||
chunks keep their hashes — only genuinely new text lands in the diff.
|
||||
- **Save (translated view):** diff against the served hybrid, append a
|
||||
`Patch` under `patches[f"{path}:{lang}"]`; `node.chunks` untouched.
|
||||
- **Invalidate:** any write to `chunks` / `trans` / `patches` calls
|
||||
`_invalidate_pages()`.
|
||||
|
||||
## migrate_v3 steps
|
||||
|
||||
1. Walk `menu`; for every node with a string `content`:
|
||||
`chunks = chunk_markdown(content)`; write each into the new `chunks`
|
||||
store; replace the field with the hash list (`None` stays `None`).
|
||||
2. Initialize empty `chunks` / `trans` / `patches` stores.
|
||||
3. Normalize stored paths: strip leading slashes anywhere paths are keys or
|
||||
values.
|
||||
4. `language`, `no_trans` and `langs` need nothing — struct defaults cover
|
||||
them (`langs` starts empty; the translator job fills it as translations
|
||||
land).
|
||||
|
||||
Chunking must be deterministic and shared with render/save, so
|
||||
`chunk_markdown` + `chunk_key` live in `pagerite/i18n.py` (or a small
|
||||
`pagerite/chunks.py`) and are imported by both `migrations.py` and
|
||||
`views.py`/`app.py`.
|
||||
|
||||
## Implementation notes (deviations from the plan above)
|
||||
|
||||
- Chunking lives in `pagerite/chunks.py`; hashing uses the `blake3` package
|
||||
(already a dependency), truncated to a 9-byte `bytes` digest (kanta's
|
||||
JSON persistence base64-encodes bytes keys to 12-char strings).
|
||||
- `trans` is keyed `hash -> lang -> text` (nested dict), not by
|
||||
`f"{hash}:{lang}"` tuples: msgspec's JSON serializer only supports
|
||||
str-like/number-like dict keys, and kanta persists as JSON lines.
|
||||
- `Translation.titles` stayed keyed by node path (phase-1 shape, views
|
||||
untouched): `get_translation` builds it by walking the menu with the same
|
||||
per-title `trans.get(chunk_key(node.title), {}).get(lang)` lookups.
|
||||
- Insert hunks anchor on the whole preceding block (not just its tail) —
|
||||
a stronger, simpler search context.
|
||||
- `make_patch` diffs with `SequenceMatcher(autojunk=False)` so patches are
|
||||
deterministic (popular lines like blank separators never become junk).
|
||||
- Step 3's path normalization is a no-op in practice: the only path-keyed
|
||||
store (`patches`) starts empty at v3; analytics paths live outside the
|
||||
kantadb. The code still strips leading slashes defensively.
|
||||
|
||||
## Garbage collection (later, manual or idle-time)
|
||||
|
||||
Orphaned entries accumulate: chunks no longer referenced by any
|
||||
`node.chunks`/`node.title`, translations whose chunk hash is orphaned, patch
|
||||
hunks that never match. All are harmless (never read). A GC pass is a single
|
||||
tree walk collecting live hashes, then deleting the rest from `chunks` and
|
||||
`trans`; patches whose every hunk is stale get pruned. Not part of
|
||||
migrate_v3.
|
||||
Reference in New Issue
Block a user