Masked translation round-trip; named translator keys

- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).
This commit is contained in:
2026-09-02 19:56:54 +00:00
parent af76277d68
commit 4c886eda86
11 changed files with 329 additions and 46 deletions
+34 -4
View File
@@ -258,9 +258,16 @@ local to that language.
An external machine-translation service connects over WebSocket at
`/_translate/{key}` — deliberately **not** under `/_api`: the SSO
forward-auth does not cover that route, and the key in the path is the
access control. The key is `Data.translate_key`, generated once at startup
and surfaced to the admin in `GET /_api/settings` as `translate_key`. A
wrong or empty key rejects the handshake (close-before-accept → HTTP 403).
access control. Keys live in `Data.translate_keys` (key -> display name) —
12 lowercase alphanumeric characters each, the first one generated at
database bootstrap and multiple keys reserved for future management (e.g.
a web UI). The full WS URL(s) are printed in the startup log
(`ws://localhost:{port}/_translate/{key}` locally,
`wss://{hostname}/_translate/{key}` on a public hostname) and the keys are
surfaced to the admin in `GET /_api/settings` as `translate_keys`. An
unknown or empty key rejects the handshake (close-before-accept → HTTP
403). Transactions storing results record the connecting key as the kanta
transaction `user`.
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
`bytes` fields ride as base64):
@@ -269,7 +276,8 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
**capabilities**: the language codes its model can produce (normalized
to base subtags; `en`/empty dropped).
- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
ONE fragment to translate (an article title or a chunk).
ONE fragment to translate (an article title or a chunk), its text
**masked** (see Masking below).
- `{"type": "result", "lang", "key", "text"}` — client reply: the
translation of the connection's current job, matching it by (lang, key).
@@ -299,6 +307,28 @@ Results are stored into `trans` in one transaction and set
pages gain a language from one fragment). Unknown keys are stored anyway
and re-storing overwrites — results are idempotent.
#### Masking
Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model
must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before
dispatch and restored by number from the result. Masked: code spans,
container-fence names, link and image *destinations* (link text, alt text
and captions stay visible for translation), reference and footnote labels,
`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML
tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not
masked — it carries no lexical content and models preserve it. Chunks with
no prose left after masking (a lone `{dates}`, container fences, pure
code/HTML) are never dispatched at all (`needs_translation`); every language
renders them from the original chunk.
A result is accepted only if every sentinel survived exactly once, in any
order (translations legitimately reorder spans). A mangled result is dropped
and logged, and the (lang, key) pair is skipped for the rest of the server
run — generation is near-deterministic, so an immediate retry would re-fail
the same way; the fragment stays pending and gets another chance on restart
or a model/masking change. `Data.trans` therefore only ever holds clean,
unmasked text.
### Explicitly out of scope for phase 2
- The machine translation itself: the API above moves fragments in and out;
+7 -5
View File
@@ -52,9 +52,9 @@ class Node(msgspec.Struct, omit_defaults=True):
class Data(msgspec.Struct):
...
#: API key gating the translator service WebSocket (/_translate/{key});
#: generated lazily at startup (see the lifespan in app.py).
translate_key: str = ""
#: API keys gating the translator service WebSocket (/_translate/{key}):
#: key -> display name; the first is generated at bootstrap (app.py).
translate_keys: dict[str, str] = {}
#: Wanted target languages for the translator service (presence-keys);
#: jobs are offered only in these ∩ a connection's capabilities.
translate_langs: dict[str, True] = {}
@@ -81,8 +81,10 @@ Notes:
`trans.get(hash(node.title), {}).get(lang)`. No separate title storage;
editing a title invalidates its translations automatically.
- **Per-hunk options** live in two places: *inherent* options are derived at
chunking time (code fences and HTML blocks are marked no-translate without
storing anything); *editor-set* flags are `node.no_trans` (keyed by chunk
chunking time (code fences, HTML blocks and prose-free chunks are
no-translate without storing anything — `needs_translation`, see
docs/localization.md "Masking"); *editor-set* flags are `node.no_trans`
(keyed by chunk
hash, so a heavy edit silently drops the flag — acceptable and
self-healing).
- **Patch payloads stay inline** in `Patch.hunks` — patches are small by