Masked translation round-trip; named translator keys

- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).
This commit is contained in:
2026-09-02 19:56:54 +00:00
parent af76277d68
commit 4c886eda86
11 changed files with 329 additions and 46 deletions
+8 -5
View File
@@ -13,8 +13,9 @@
"""Pagerite translator service: translate site content with Seed-X-PPO-7B.
Connects to a Pagerite server's translator WebSocket — the full URL
including the access key (the admin finds it in the site settings,
GET /_api/settings -> ``translate_key``) — and announces the languages the
including the access key (printed at server startup; the admin also finds
the key in the site settings, GET /_api/settings -> ``translate_keys``) —
and announces the languages the
model CAN translate (capabilities). The server dispatches one single-item
job at a time per connection, offered only in its configured target
languages (``Data.translate_langs``) ∩ the announced capabilities; a
@@ -55,8 +56,10 @@ SEED_X_TAGS = {
}
SEED_X_NAMES = {v: k for k, v in SEED_X_TAGS.items()}
#: The fragments are Markdown; Seed-X has no system prompt, so it goes in-line.
NOTE = ", preserving all Markdown formatting, URLs and code exactly unchanged"
#: The fragments are masked Markdown (pagerite/masking.py: ⟦N⟧ sentinels
#: stand in for code, URLs, placeholders...); Seed-X has no system prompt,
#: so the instruction goes in-line.
NOTE = ", preserving all Markdown formatting and keeping every ⟦N⟧ token exactly unchanged"
# The wire structs below duplicate pagerite/translate.py: this script runs
@@ -76,7 +79,7 @@ class Job(msgspec.Struct, tag="job"):
lang: str
key: bytes #: 9-byte chunk hash (base64 in the JSON frame)
text: str
text: str #: masked (pagerite/masking.py): the ⟦N⟧ tokens must survive verbatim
path: str #: article it came from ("" = front page), no leading slash
kind: str #: "chunk" | "title"