diff --git a/AGENTS.md b/AGENTS.md index b81ce83..e0b41ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,8 +13,9 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `app.py` — FastAPI app and route registration. - `data.py` — msgspec Structs for the kanta database. - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md). - - `i18n.py` — language selection, translation assembly (chunks + patches). - - `translate.py` — translator service protocol (msgspec structs) and pending/store core for the `/_translate/{key}` WebSocket (docs/localization.md). + - `i18n.py` — language selection, translation assembly (chunks + patches) and translated-edit recording (user patches, per-language title overrides, refresh). + - `translate.py` — translator service protocol (msgspec structs), the connected-client `Dispatcher` (job pipeline, result validation) and pending/store core for the `/_translate/{key}` WebSocket (docs/localization.md); app.py only registers the route. + - `segments.py` — the translation round trip: fragments split into pure-prose wire segments (via markdown.make_md's verbatim parser) and translations spliced back by source offset (docs/localization.md). - `migrations.py` — kanta migrations (`migrate_vN`); ALL schema/storage upgrades live here (raw state dict before struct decoding), never in the app lifespan: v1 moves legacy in-db file blobs to the on-disk store and rebuilds the legacy flat `pages` as the menu tree, v2 rewrites `/_f/{hash}.ext` image links to the extension-less form, backfills AVIF/WebP/JPEG derivatives on disk and drops the obsolete `version` field. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering; theme/user-font resolution across `THEME_DIRS` / `FONT_DIRS` (cwd, site, platform data roots, then built-in `pagerite/themes/`, see `docs/themes-and-assets.md`). diff --git a/docs/localization.md b/docs/localization.md index 3880779..ca6191a 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -194,7 +194,7 @@ Full storage design and the `migrate_v3` restructuring live in ### Render pipeline (the phase-1 `get_translation` stub, now real) ```python -def get_translation(path, lang, data) -> Translation | None: +def get_translation(data, path, lang) -> Translation | None: if lang not in node.langs: return None hybrid = "\n\n".join( @@ -222,8 +222,9 @@ def get_translation(path, lang, data) -> Translation | None: The page editor has a language picker (flag + name; the same country-flag-icons set as the analytics visitor cells) listing the primary language and the union of the page's translations (`node.langs`) and the -site-wide `translate_langs`. It opens in the language the page was served -in (``). A note under the toolbar states the blast radius: +site-wide `translate_langs`. It always opens in the primary language, even +when the page itself was served in a translation. A note under the toolbar +states the blast radius: edits to the primary language re-chunk the original (invalidating the affected translation fragments everywhere); edits to a translation stay local to that language. @@ -253,6 +254,17 @@ local to that language. updates `Data.chunks` / `node.chunks` — only genuinely new text lands in the kanta change diff (see docs/migrate.md). +The **structure editor** has the same flag strip for titles. The tree it +lists (`GET /_api/pages?lang=`) comes back with per-language titles where a +translation exists (`translated` marks those rows; untranslated rows show +the original title, dimmed). Retitling in a non-primary language posts the +structure op with a `lang` and writes a per-language title fragment in +`Data.trans` (keyed by the original title's chunk hash, exactly like a +machine title translation — a user edit simply overwrites it); sending the +original's text drops the override. The structure itself — slugs, +hierarchy, order — is language-independent, so pending rows, slug edits, +drag-and-drop and deletes work identically in every language. + ### Translator service API An external machine-translation service connects over WebSocket at @@ -275,11 +287,14 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; - `{"type": "hello", "langs": [...]}` — client greeting announcing its **capabilities**: the language codes its model can produce (normalized to base subtags; `en`/empty dropped). -- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push: - ONE fragment to translate (an article title or a chunk), its text - **masked** (see Masking below). -- `{"type": "result", "lang", "key", "text"}` — client reply: the - translation of the connection's current job, matching it by (lang, key). +- `{"type": "job", "lang", "key", "texts", "path", "kind", "contexts"}` — + server push: ONE fragment to translate (an article title or a chunk), as + a list of **prose segments** (see Segmentation below). `contexts` is + parallel to `texts` ("" = none): the surround to translate the segment + in — for clients that translate better with context (see below). + Contexts are not part of the result. +- `{"type": "result", "lang", "key", "texts"}` — client reply: the + segments translated, same order and count, matching its job by (lang, key). Which languages get translated is **server-configured**: `Data.translate_langs` (presence-key dict, bootstrapped to Spanish and @@ -289,7 +304,15 @@ The dispatcher offers a connection jobs only in `wanted ∩ capable`; a connection without overlap simply stays idle. -Dispatch semantics (all in app.py): +`DELETE /_api/translations` (the localization tab's "refresh all +translations" button) drops every machine translation (`Data.trans`) and +rebuilds the availability index (`node.langs`) from the surviving user +patches, so the dispatcher re-translates everything from scratch; the +run's validation skip-list is cleared with it, giving rejected fragments +another chance. + +Dispatch semantics (the `Dispatcher` in `pagerite/translate.py`; app.py only +registers the route): - **One job at a time per connection** — the next job is sent only after the current one's result. Clients wanting parallelism open multiple @@ -309,27 +332,79 @@ Results are stored into `trans` in one transaction and set pages gain a language from one fragment). Unknown keys are stored anyway and re-storing overwrites — results are idempotent. -#### Masking +#### Segmentation -Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model -must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before -dispatch and restored by number from the result. Masked: code spans, -container-fence names, link and image *destinations* (link text, alt text -and captions stay visible for translation), reference and footnote labels, -`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML -tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not -masked — it carries no lexical content and models preserve it. Chunks with -no prose left after masking (a lone `{dates}`, container fences, pure -code/HTML) are never dispatched at all (`needs_translation`); every language -renders them from the original chunk. +Fragments cross the wire as **prose segments** (`pagerite/segments.py`): the +fragment is parsed with the project's own markdown-it setup +(`markdown.make_md(verbatim=True)` — all extensions, but no typographer or +tasklist label wrapping, so token text stays byte-identical to the source) +and split into the runs a model may touch: paragraph/heading/table-cell text +(merged across soft line breaks), link text, image alt texts and captions, +footnote bodies. Everything else never leaves the server: code spans and +fences, URLs and autolinks, link/image *destinations*, `{...}` spans +(placeholders like `{dates}` as well as attrs), reference and footnote +labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all +markup punctuation (`*`, `|`, `[]()`, `:::`), which is a run boundary. +Chunks with no segments (a lone `{dates}`, container fences, pure +code/HTML) are never dispatched at all (`needs_translation`); every +language renders them from the original chunk. Each segment is accompanied +by a context string (a segment carved out of a larger block carries the +block's plain text; a whole-block segment carries "") — context is a +prompt aid only, never spliced into the result. -A result is accepted only if every sentinel survived exactly once, in any -order (translations legitimately reorder spans). A mangled result is dropped -and logged, and the (lang, key) pair is skipped for the rest of the server -run — generation is near-deterministic, so an immediate retry would re-fail -the same way; the fragment stays pending and gets another chance on restart -or a model/masking change. `Data.trans` therefore only ever holds clean, -unmasked text. +Reassembly is offset splicing, not text the model produced: each segment's +source span was located at dispatch (sequential search; a run that is not a +verbatim source substring — entity-decoded text, backslash escapes — is +skipped and stays in the original language), and the returned translations +are swapped in by offset. Markup corruption is therefore impossible by +construction; the failure modes that remain are a wrong segment count, an +empty segment, or markup injected INTO a segment (a `
` in a title +translation would splice live HTML) — each returned segment must parse as +pure prose, or the whole result is dropped and logged, and the (lang, key) +pair is skipped for the rest of the server run (generation is +near-deterministic, so an immediate retry would re-fail; the fragment stays +pending and gets another chance on restart or `DELETE /_api/translations`). +`Data.trans` therefore only ever holds clean translated Markdown. + +The trade-off: segments splice back at fixed positions, so a translation +cannot move a link or image within a sentence — word order around inline +markup follows the original. That is the price for never feeding the model +markup (an earlier sentinel-masking design let the model see and mangle +exactly that punctuation: Seed-X turned `![` into `¡¡…!!`). + +Punctuation is the translator's own job: Seed-X tends to "finish" short +labels (titles, nav items) with a comma or period the source never had. +Prompt wording is NOT the fix — a punctuation-instruction clause made +Seed-X slip into its `[COT]` reasoning mode (minutes-long generations with +reasoning text in the output, observed for Chinese). The reference client +enforces punctuation deterministically instead (`match_punctuation` in +scripts/translator.py): a translation of a segment without terminal +punctuation gets any added trailing marks (and a newly opened Spanish ¡/¿) +stripped before the result goes back. + +The same client-side enforcement covers markup bleed as a CLASS, not per +artifact: `<` is the prose/markup boundary on the wire and never appears in +a segment in either direction. Source pieces containing `<` are never +dispatched (they stay in the original language — segments.py), and the +reference client cuts the model's output at the first `<` +(scripts/translator.py) — echoed language tags, stray `
`s and any +future variant are one handled case. (The cut is post-decode, not a +generation stop string: Seed-X opens every generation with its `` +framing token, which would trip a `<` stop immediately.) + +Short fragments get more than a bare prompt: each segment may carry its +surround in `Job.contexts` — a title carries the article's opening prose +(its own block is just the title word), a segment carved out of a larger +block (a link text, a partial run) carries the block's plain text, and a +whole-block segment (a plain paragraph) is self-contextualizing and carries +"". The reference client translates segment and surround together, stops +generation at the blank line separating them, and keeps the segment's own +part of the output (its line resp. paragraph; a hard-break `␣␣\n` separator +works too). If the model merged them (no separator, or an empty first +part), it falls back to translating the segment alone. The surround fixes +context-free readings ("About" as "approximately" — with the opening it +becomes "Tietoa"/"Acerca de"; "here" as "就在这里" → the idiomatic +"点击这里") and, as a side effect, most stray trailing punctuation. ### Explicitly out of scope for phase 2 diff --git a/frontend/src/LocalizationEditor.vue b/frontend/src/LocalizationEditor.vue index da1cc7f..39e4897 100644 --- a/frontend/src/LocalizationEditor.vue +++ b/frontend/src/LocalizationEditor.vue @@ -4,7 +4,9 @@ // language is configured per site hierarchy, not here. Flag clicks toggle // and save immediately; the settings round-trip re-reads the payload, so // this tab only ever changes translate_langs. The settings write's -// invalidation hook kicks the translation dispatcher. +// invalidation hook kicks the translation dispatcher. The refresh button +// drops all machine translations (user patches are kept), making the +// dispatcher re-translate everything. import { computed, onActivated, onMounted, onUnmounted, ref } from 'vue' import { TRANSLATABLE, flagFor, langName } from './langs' import { dropPageCache } from './swapdoc' @@ -75,6 +77,24 @@ async function toggle(code) { saveError.value = '⚠️ changes could not be saved' } } + +// Delete all machine translations server-side; the dispatcher re-fills +// them (a connected translator starts getting jobs right away). User +// patches survive — they are edits, not machine output. +const refreshing = ref(false) +async function refresh() { + if (refreshing.value) return + refreshing.value = true + try { + const res = await fetch('/_api/translations', { method: 'DELETE' }) + saveError.value = res.ok ? '' : '⚠️ translations could not be refreshed' + if (res.ok) dropPageCache() + } catch { + saveError.value = '⚠️ translations could not be refreshed' + } finally { + refreshing.value = false + } +}