LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Segments, not sentinels: prose-only translation wire protocol

Browse Source 
- pagerite/segments.py replaces masking.py: a fragment is parsed with the
  project's own markdown-it (markdown.make_md(verbatim=True), byte-identical
  tokens) and split into pure-prose segments with source spans; only the
  segments plus per-segment context surrounds cross the wire (Job.texts /
  contexts / Result.texts) and translations splice back by offset — markup
  can no longer break, it never leaves the server. Count/empty/non-prose
  results are rejected and skipped for the run.
- Localization machinery out of app.py: the translator dispatcher (clients,
  job pipeline, validation skip-list) moves into translate.Dispatcher;
  translated-edit recording moves into i18n (add_patch, set_title_translation,
  clear_translations). app.py keeps only the routes.
- Structure editor localized: flag strip switches the language titles are
  shown/edited in (GET /_api/pages?lang= flags translated rows, originals
  dimmed); retitling in a translation writes a per-language title fragment
  via StructureOp.lang — slugs, order and hierarchy stay language-independent.
- Localization tab: refresh-all button (DELETE /_api/translations) drops
  machine translations, keeps user patches and clears the skip-list so the
  dispatcher re-translates everything.
- PageEditor always opens in the primary language; editor socket gets
  reconnect/doc-mismatch logging. Reference translator: per-segment calls
  with context prompts, deterministic punctuation matching and the "<"
  markup-bleed cut.
This commit is contained in:
LeoVasanko
2026-09-03 00:24:07 +00:00
parent b1e8f0b454
commit b0866fc4f7
16 changed files with 1136 additions and 494 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/localization.md
+103 -28
View File
@@ -194,7 +194,7 @@ Full storage design and the `migrate_v3` restructuring live in
### Render pipeline (the phase-1 `get_translation` stub, now real)
```python
def get_translation(path, lang, data) -> Translation | None:
def get_translation(data, path, lang) -> Translation | None:
if lang not in node.langs:
return None
hybrid = "\n\n".join(
@@ -222,8 +222,9 @@ def get_translation(path, lang, data) -> Translation | None:
The page editor has a language picker (flag + name; the same
country-flag-icons set as the analytics visitor cells) listing the primary
language and the union of the page's translations (`node.langs`) and the
site-wide `translate_langs`. It opens in the language the page was served
in (`<html lang>`). A note under the toolbar states the blast radius:
site-wide `translate_langs`. It always opens in the primary language, even
when the page itself was served in a translation. A note under the toolbar
states the blast radius:
edits to the primary language re-chunk the original (invalidating the
affected translation fragments everywhere); edits to a translation stay
local to that language.
@@ -253,6 +254,17 @@ local to that language.
updates `Data.chunks` / `node.chunks` — only genuinely new text lands in
the kanta change diff (see docs/migrate.md).
The **structure editor** has the same flag strip for titles. The tree it
lists (`GET /_api/pages?lang=`) comes back with per-language titles where a
translation exists (`translated` marks those rows; untranslated rows show
the original title, dimmed). Retitling in a non-primary language posts the
structure op with a `lang` and writes a per-language title fragment in
`Data.trans` (keyed by the original title's chunk hash, exactly like a
machine title translation — a user edit simply overwrites it); sending the
original's text drops the override. The structure itself — slugs,
hierarchy, order — is language-independent, so pending rows, slug edits,
drag-and-drop and deletes work identically in every language.
### Translator service API
An external machine-translation service connects over WebSocket at
@@ -275,11 +287,14 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
- `{"type": "hello", "langs": [...]}` — client greeting announcing its
**capabilities**: the language codes its model can produce (normalized
to base subtags; `en`/empty dropped).
- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
ONE fragment to translate (an article title or a chunk), its text
**masked** (see Masking below).
- `{"type": "result", "lang", "key", "text"}` — client reply: the
translation of the connection's current job, matching it by (lang, key).
- `{"type": "job", "lang", "key", "texts", "path", "kind", "contexts"}` —
server push: ONE fragment to translate (an article title or a chunk), as
a list of **prose segments** (see Segmentation below). `contexts` is
parallel to `texts` ("" = none): the surround to translate the segment
in — for clients that translate better with context (see below).
Contexts are not part of the result.
- `{"type": "result", "lang", "key", "texts"}` — client reply: the
segments translated, same order and count, matching its job by (lang, key).
Which languages get translated is **server-configured**:
`Data.translate_langs` (presence-key dict, bootstrapped to Spanish and
@@ -289,7 +304,15 @@ The dispatcher offers a
connection jobs only in `wanted ∩ capable`; a connection without overlap
simply stays idle.
Dispatch semantics (all in app.py):
`DELETE /_api/translations` (the localization tab's "refresh all
translations" button) drops every machine translation (`Data.trans`) and
rebuilds the availability index (`node.langs`) from the surviving user
patches, so the dispatcher re-translates everything from scratch; the
run's validation skip-list is cleared with it, giving rejected fragments
another chance.
Dispatch semantics (the `Dispatcher` in `pagerite/translate.py`; app.py only
registers the route):
- **One job at a time per connection** — the next job is sent only after
the current one's result. Clients wanting parallelism open multiple
@@ -309,27 +332,79 @@ Results are stored into `trans` in one transaction and set
pages gain a language from one fragment). Unknown keys are stored anyway
and re-storing overwrites — results are idempotent.
#### Masking
#### Segmentation
Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model
must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before
dispatch and restored by number from the result. Masked: code spans,
container-fence names, link and image *destinations* (link text, alt text
and captions stay visible for translation), reference and footnote labels,
`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML
tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not
masked — it carries no lexical content and models preserve it. Chunks with
no prose left after masking (a lone `{dates}`, container fences, pure
code/HTML) are never dispatched at all (`needs_translation`); every language
renders them from the original chunk.
Fragments cross the wire as **prose segments** (`pagerite/segments.py`): the
fragment is parsed with the project's own markdown-it setup
(`markdown.make_md(verbatim=True)` — all extensions, but no typographer or
tasklist label wrapping, so token text stays byte-identical to the source)
and split into the runs a model may touch: paragraph/heading/table-cell text
(merged across soft line breaks), link text, image alt texts and captions,
footnote bodies. Everything else never leaves the server: code spans and
fences, URLs and autolinks, link/image *destinations*, `{...}` spans
(placeholders like `{dates}` as well as attrs), reference and footnote
labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all
markup punctuation (`*`, `|`, `[]()`, `:::`), which is a run boundary.
Chunks with no segments (a lone `{dates}`, container fences, pure
code/HTML) are never dispatched at all (`needs_translation`); every
language renders them from the original chunk. Each segment is accompanied
by a context string (a segment carved out of a larger block carries the
block's plain text; a whole-block segment carries "") — context is a
prompt aid only, never spliced into the result.
A result is accepted only if every sentinel survived exactly once, in any
order (translations legitimately reorder spans). A mangled result is dropped
and logged, and the (lang, key) pair is skipped for the rest of the server
run — generation is near-deterministic, so an immediate retry would re-fail
the same way; the fragment stays pending and gets another chance on restart
or a model/masking change. `Data.trans` therefore only ever holds clean,
unmasked text.
Reassembly is offset splicing, not text the model produced: each segment's
source span was located at dispatch (sequential search; a run that is not a
verbatim source substring — entity-decoded text, backslash escapes — is
skipped and stays in the original language), and the returned translations
are swapped in by offset. Markup corruption is therefore impossible by
construction; the failure modes that remain are a wrong segment count, an
empty segment, or markup injected INTO a segment (a `<br>` in a title
translation would splice live HTML) — each returned segment must parse as
pure prose, or the whole result is dropped and logged, and the (lang, key)
pair is skipped for the rest of the server run (generation is
near-deterministic, so an immediate retry would re-fail; the fragment stays
pending and gets another chance on restart or `DELETE /_api/translations`).
`Data.trans` therefore only ever holds clean translated Markdown.
The trade-off: segments splice back at fixed positions, so a translation
cannot move a link or image within a sentence — word order around inline
markup follows the original. That is the price for never feeding the model
markup (an earlier sentinel-masking design let the model see and mangle
exactly that punctuation: Seed-X turned `![` into `¡¡…!!`).
Punctuation is the translator's own job: Seed-X tends to "finish" short
labels (titles, nav items) with a comma or period the source never had.
Prompt wording is NOT the fix — a punctuation-instruction clause made
Seed-X slip into its `[COT]` reasoning mode (minutes-long generations with
reasoning text in the output, observed for Chinese). The reference client
enforces punctuation deterministically instead (`match_punctuation` in
scripts/translator.py): a translation of a segment without terminal
punctuation gets any added trailing marks (and a newly opened Spanish ¡/¿)
stripped before the result goes back.
The same client-side enforcement covers markup bleed as a CLASS, not per
artifact: `<` is the prose/markup boundary on the wire and never appears in
a segment in either direction. Source pieces containing `<` are never
dispatched (they stay in the original language — segments.py), and the
reference client cuts the model's output at the first `<`
(scripts/translator.py) — echoed language tags, stray `<br>`s and any
future variant are one handled case. (The cut is post-decode, not a
generation stop string: Seed-X opens every generation with its `<s>`
framing token, which would trip a `<` stop immediately.)
Short fragments get more than a bare prompt: each segment may carry its
surround in `Job.contexts` — a title carries the article's opening prose
(its own block is just the title word), a segment carved out of a larger
block (a link text, a partial run) carries the block's plain text, and a
whole-block segment (a plain paragraph) is self-contextualizing and carries
"". The reference client translates segment and surround together, stops
generation at the blank line separating them, and keeps the segment's own
part of the output (its line resp. paragraph; a hard-break `␣␣\n` separator
works too). If the model merged them (no separator, or an empty first
part), it falls back to translating the segment alone. The surround fixes
context-free readings ("About" as "approximately" — with the opening it
becomes "Tietoa"/"Acerca de"; "here" as "就在这里" → the idiomatic
"点击这里") and, as a side effect, most stray trailing punctuation.
### Explicitly out of scope for phase 2