Masked translation round-trip; named translator keys
- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
footnote/link labels, container names, HTML tags) become numbered sentinels
for the LLM round trip; results are restored by number and rejected when a
sentinel is mangled (skipped for the rest of the run, stays pending).
Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
generated at bootstrap, result transactions record the key as user=, and
startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).
This commit is contained in:
+34
-4
@@ -258,9 +258,16 @@ local to that language.
|
||||
An external machine-translation service connects over WebSocket at
|
||||
`/_translate/{key}` — deliberately **not** under `/_api`: the SSO
|
||||
forward-auth does not cover that route, and the key in the path is the
|
||||
access control. The key is `Data.translate_key`, generated once at startup
|
||||
and surfaced to the admin in `GET /_api/settings` as `translate_key`. A
|
||||
wrong or empty key rejects the handshake (close-before-accept → HTTP 403).
|
||||
access control. Keys live in `Data.translate_keys` (key -> display name) —
|
||||
12 lowercase alphanumeric characters each, the first one generated at
|
||||
database bootstrap and multiple keys reserved for future management (e.g.
|
||||
a web UI). The full WS URL(s) are printed in the startup log
|
||||
(`ws://localhost:{port}/_translate/{key}` locally,
|
||||
`wss://{hostname}/_translate/{key}` on a public hostname) and the keys are
|
||||
surfaced to the admin in `GET /_api/settings` as `translate_keys`. An
|
||||
unknown or empty key rejects the handshake (close-before-accept → HTTP
|
||||
403). Transactions storing results record the connecting key as the kanta
|
||||
transaction `user`.
|
||||
|
||||
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
|
||||
`bytes` fields ride as base64):
|
||||
@@ -269,7 +276,8 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
|
||||
**capabilities**: the language codes its model can produce (normalized
|
||||
to base subtags; `en`/empty dropped).
|
||||
- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
|
||||
ONE fragment to translate (an article title or a chunk).
|
||||
ONE fragment to translate (an article title or a chunk), its text
|
||||
**masked** (see Masking below).
|
||||
- `{"type": "result", "lang", "key", "text"}` — client reply: the
|
||||
translation of the connection's current job, matching it by (lang, key).
|
||||
|
||||
@@ -299,6 +307,28 @@ Results are stored into `trans` in one transaction and set
|
||||
pages gain a language from one fragment). Unknown keys are stored anyway
|
||||
and re-storing overwrites — results are idempotent.
|
||||
|
||||
#### Masking
|
||||
|
||||
Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model
|
||||
must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before
|
||||
dispatch and restored by number from the result. Masked: code spans,
|
||||
container-fence names, link and image *destinations* (link text, alt text
|
||||
and captions stay visible for translation), reference and footnote labels,
|
||||
`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML
|
||||
tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not
|
||||
masked — it carries no lexical content and models preserve it. Chunks with
|
||||
no prose left after masking (a lone `{dates}`, container fences, pure
|
||||
code/HTML) are never dispatched at all (`needs_translation`); every language
|
||||
renders them from the original chunk.
|
||||
|
||||
A result is accepted only if every sentinel survived exactly once, in any
|
||||
order (translations legitimately reorder spans). A mangled result is dropped
|
||||
and logged, and the (lang, key) pair is skipped for the rest of the server
|
||||
run — generation is near-deterministic, so an immediate retry would re-fail
|
||||
the same way; the fragment stays pending and gets another chance on restart
|
||||
or a model/masking change. `Data.trans` therefore only ever holds clean,
|
||||
unmasked text.
|
||||
|
||||
### Explicitly out of scope for phase 2
|
||||
|
||||
- The machine translation itself: the API above moves fragments in and out;
|
||||
|
||||
Reference in New Issue
Block a user