diff --git a/AGENTS.md b/AGENTS.md index 7a15ffb..b81ce83 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,5 +57,5 @@ Server run by CLI entry point `uv run pagerite` (no auto reloads, build needed). - Keep dependencies minimal; add via `uv add` and mention it. - The public URL space belongs to content (pretty slugs at root). Reserve only `/_` for the machinery (`/_api/`, `/_f/`, `/_assets/`), plus `/favicon.ico` from the build. Slugs are lowercase ASCII letters, digits, hyphens and underscores `[a-z0-9_-]` (the site editor filters input live via `slugify.js`, built on the `transliteration` npm package — unicode folds to ASCII, spaces become hyphens; an empty slug on a new page is derived from its title), may not begin with `_` or `.`, and such URLs are never looked up as content. -- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. The one keyed exception is `/_translate/{key}` (translator service; `Data.translate_key`, see docs/localization.md). +- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. The one keyed exception is `/_translate/{key}` (translator service; `Data.translate_keys`, see docs/localization.md). - Update the relevant MarkDown files when architecture, tooling, or conventions change. diff --git a/docs/localization.md b/docs/localization.md index 4fc231d..c97506f 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -258,9 +258,16 @@ local to that language. An external machine-translation service connects over WebSocket at `/_translate/{key}` — deliberately **not** under `/_api`: the SSO forward-auth does not cover that route, and the key in the path is the -access control. The key is `Data.translate_key`, generated once at startup -and surfaced to the admin in `GET /_api/settings` as `translate_key`. A -wrong or empty key rejects the handshake (close-before-accept → HTTP 403). +access control. Keys live in `Data.translate_keys` (key -> display name) — +12 lowercase alphanumeric characters each, the first one generated at +database bootstrap and multiple keys reserved for future management (e.g. +a web UI). The full WS URL(s) are printed in the startup log +(`ws://localhost:{port}/_translate/{key}` locally, +`wss://{hostname}/_translate/{key}` on a public hostname) and the keys are +surfaced to the admin in `GET /_api/settings` as `translate_keys`. An +unknown or empty key rejects the handshake (close-before-accept → HTTP +403). Transactions storing results record the connecting key as the kanta +transaction `user`. Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; `bytes` fields ride as base64): @@ -269,7 +276,8 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; **capabilities**: the language codes its model can produce (normalized to base subtags; `en`/empty dropped). - `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push: - ONE fragment to translate (an article title or a chunk). + ONE fragment to translate (an article title or a chunk), its text + **masked** (see Masking below). - `{"type": "result", "lang", "key", "text"}` — client reply: the translation of the connection's current job, matching it by (lang, key). @@ -299,6 +307,28 @@ Results are stored into `trans` in one transaction and set pages gain a language from one fragment). Unknown keys are stored anyway and re-storing overwrites — results are idempotent. +#### Masking + +Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model +must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before +dispatch and restored by number from the result. Masked: code spans, +container-fence names, link and image *destinations* (link text, alt text +and captions stay visible for translation), reference and footnote labels, +`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML +tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not +masked — it carries no lexical content and models preserve it. Chunks with +no prose left after masking (a lone `{dates}`, container fences, pure +code/HTML) are never dispatched at all (`needs_translation`); every language +renders them from the original chunk. + +A result is accepted only if every sentinel survived exactly once, in any +order (translations legitimately reorder spans). A mangled result is dropped +and logged, and the (lang, key) pair is skipped for the rest of the server +run — generation is near-deterministic, so an immediate retry would re-fail +the same way; the fragment stays pending and gets another chance on restart +or a model/masking change. `Data.trans` therefore only ever holds clean, +unmasked text. + ### Explicitly out of scope for phase 2 - The machine translation itself: the API above moves fragments in and out; diff --git a/docs/migrate.md b/docs/migrate.md index 78ebef5..687287a 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -52,9 +52,9 @@ class Node(msgspec.Struct, omit_defaults=True): class Data(msgspec.Struct): ... - #: API key gating the translator service WebSocket (/_translate/{key}); - #: generated lazily at startup (see the lifespan in app.py). - translate_key: str = "" + #: API keys gating the translator service WebSocket (/_translate/{key}): + #: key -> display name; the first is generated at bootstrap (app.py). + translate_keys: dict[str, str] = {} #: Wanted target languages for the translator service (presence-keys); #: jobs are offered only in these ∩ a connection's capabilities. translate_langs: dict[str, True] = {} @@ -81,8 +81,10 @@ Notes: `trans.get(hash(node.title), {}).get(lang)`. No separate title storage; editing a title invalidates its translations automatically. - **Per-hunk options** live in two places: *inherent* options are derived at - chunking time (code fences and HTML blocks are marked no-translate without - storing anything); *editor-set* flags are `node.no_trans` (keyed by chunk + chunking time (code fences, HTML blocks and prose-free chunks are + no-translate without storing anything — `needs_translation`, see + docs/localization.md "Masking"); *editor-set* flags are `node.no_trans` + (keyed by chunk hash, so a heavy edit silently drops the flag — acceptable and self-healing). - **Patch payloads stay inline** in `Patch.hunks` — patches are small by diff --git a/frontend/vite.config.js b/frontend/vite.config.js index 265bd75..e0549de 100644 --- a/frontend/vite.config.js +++ b/frontend/vite.config.js @@ -16,7 +16,7 @@ const CONTENT_PROXY = '^(?!/_|/@|/src|/node_modules|/__).*$' // https://vite.dev/config/ export default defineConfig({ plugins: [ - fastapiVue({ paths: ["/_api", "/_f", "/_themes", "/_fonts", "/_a"] }), + fastapiVue({ paths: ["/_api", "/_f", "/_themes", "/_fonts", "/_a", "/_translate"] }), vue(), vueDevTools(), ], diff --git a/pagerite/__main__.py b/pagerite/__main__.py index eab9e67..a5b3ce5 100644 --- a/pagerite/__main__.py +++ b/pagerite/__main__.py @@ -9,6 +9,7 @@ from pathlib import Path import httpx from fastapi_vue import server +from fastapi_vue.hostutil import parse_endpoints DEFAULT_PORT = 8100 DEVMODE = os.getenv("PAGERITE_DEV") == "1" @@ -96,6 +97,12 @@ def main() -> None: # Export the hostname before pagerite.app is imported: it derives the # data directory and public origin from it at import time. os.environ["PAGERITE_HOSTNAME"] = args.hostname + # And the listen port: the app prints the translator WS URL at startup, + # which for localhost includes the actual port. + for endpoint in parse_endpoints(args.listen, DEFAULT_PORT): + if "port" in endpoint: + os.environ["PAGERITE_PORT"] = str(endpoint["port"]) + break if args.dbip: _download_dbip() server.run( diff --git a/pagerite/app.py b/pagerite/app.py index ed40513..3868031 100644 --- a/pagerite/app.py +++ b/pagerite/app.py @@ -15,6 +15,7 @@ walking the tree (``resolve``), moves are slot detach/attach import asyncio import gzip import ipaddress +import logging import mimetypes import os import re @@ -48,7 +49,7 @@ from mediapreview import dispatch from pydantic import BaseModel from zstandard import ZstdCompressor -from pagerite import analytics, i18n, seed, translate, views +from pagerite import analytics, i18n, masking, seed, translate, views from pagerite.__main__ import DEVMODE from pagerite.chunks import store_chunks from pagerite.data import ( @@ -63,6 +64,8 @@ from pagerite.data import ( ) from pagerite.markdown import render, toggle_task +logger = logging.getLogger(__name__) + # Site identity: the hostname comes from the CLI (first positional argument, # exported as PAGERITE_HOSTNAME) and names the per-site data directory # ``/{content.kantadb, analytics.json, files}`` under the cwd. @@ -283,14 +286,26 @@ def _seed(data: Data) -> None: node.order = order +#: Translator key format: 12 lowercase alphanumeric characters — not +#: brute-forceable over a WebSocket handshake, still human-manageable. +_KEY_ALPHABET = "abcdefghijklmnopqrstuvwxyz0123456789" + + +@kanta.bootstrap +def _translate_key(data: Data) -> None: + """Generate the first translator service key on database creation. + + Keys are a dict (key -> display name) with the future reservation that + multiple keys could be managed (e.g. via a web interface).""" + key = "".join(secrets.choice(_KEY_ALPHABET) for _ in range(12)) + data.translate_keys[key] = "default" + + @asynccontextmanager async def lifespan(_app: FastAPI) -> AsyncIterator[None]: """Open the database (migrations run inside kanta.open), load assets, load GeoIP.""" await kanta.open() - # The translator service key is persisted; generated once on first boot. - if not data.translate_key: - with kanta.transaction("generate translate key"): - data.translate_key = secrets.token_urlsafe(24) + translate.log_service_urls(data.translate_keys, HOSTNAME) await asyncio.to_thread(file_store.load) await frontend.load() # Decompress/open the DB-IP MMDB once at startup. Lookups are then @@ -633,7 +648,7 @@ async def update_structure(op: StructureOp) -> None: async def get_settings() -> dict: """Site-wide settings (brand, theme, custom CSS and favicon URL), plus the themes, banner designs and user fonts available on disk for the - selectors, the translator service key and the wanted translation + selectors, the translator service keys and the wanted translation languages (for the /_translate socket).""" return { "brand": data.brand, @@ -646,7 +661,7 @@ async def get_settings() -> dict: "fonts": views._user_fonts(), "transition": data.transition, "transitions": views._transition_names(), - "translate_key": data.translate_key, + "translate_keys": data.translate_keys, "translate_langs": sorted(data.translate_langs), } @@ -994,7 +1009,7 @@ async def delete_page(path: str) -> None: # WebSocket API for external translation services (not under /_api: it is keyed -# with Data.translate_key instead of the SSO forward-auth). The server is a +# with Data.translate_keys instead of the SSO forward-auth). The server is a # dispatcher: one single-item job at a time per connection, offered in the # intersection of the wanted languages (Data.translate_langs) and the # connection's announced capabilities. Results are matched to content by @@ -1002,7 +1017,9 @@ async def delete_page(path: str) -> None: class _TranslatorState: """One connected translator socket: the language codes it announced as capabilities (Hello) and the (lang, chunk-key) job currently in flight - on it — one at a time, the next is sent only after its Result. + on it, with the mask spans to restore into its Result + (pagerite/masking.py) — one at a time, the next is sent only after its + Result. Per-connection only: in-flight lives solely here, so on disconnect the item simply becomes pending again and is re-offered to any free capable @@ -1011,11 +1028,17 @@ class _TranslatorState: def __init__(self, capable: set[str]) -> None: self.capable = capable self.inflight: tuple[str, bytes] | None = None + self.spans: list[str] = [] # mask spans of the in-flight job #: Connected translator sockets and their per-connection state. _translator_clients: dict[WebSocket, _TranslatorState] = {} +#: (lang, chunk key) of fragments whose result failed sentinel validation +#: (masking.unmask): skipped on later dispatches this run — generation is +#: near-deterministic, so an immediate retry would just re-fail. +_mask_failures: set[tuple[str, bytes]] = set() + def _schedule_translation_dispatch() -> None: """Schedule a dispatch pass, if any translator is connected. @@ -1057,19 +1080,23 @@ async def _dispatch_translations() -> None: continue inflight = {s.inflight for s in _translator_clients.values() if s.inflight} job = None + spans: list[str] = [] for lang in sorted(langs): for item in translate.pending_items(data, lang): - if (lang, item.key) not in inflight: - job = translate.Job( - lang=lang, key=item.key, text=item.text, - path=item.path, kind=item.kind, - ) - break + if (lang, item.key) in inflight or (lang, item.key) in _mask_failures: + continue + masked, spans = masking.mask(item.text) + job = translate.Job( + lang=lang, key=item.key, text=masked, + path=item.path, kind=item.kind, + ) + break if job is not None: break if job is None: continue state.inflight = (job.lang, job.key) # before the await: no double-assign + state.spans = spans try: await ws.send_text(msgspec.json.encode(job).decode()) except Exception: # send failed: the receive loop cleans up @@ -1082,7 +1109,8 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None: Deliberately NOT under /_api/: the external forward-auth is skipped; the server-generated client key in the path is the access control - (``Data.translate_key``, generated at startup, shown in the admin's + (``Data.translate_keys``: key -> display name; the first is generated + at bootstrap, all are shown in the admin's /_api/settings). A wrong/empty key rejects the handshake — closing before accept makes Starlette answer HTTP 403. @@ -1095,7 +1123,7 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None: different (lang, key), a duplicate Hello, or any malformed frame closes the socket with a protocol error. """ - if not data.translate_key or clientkey != data.translate_key: + if clientkey not in data.translate_keys: await ws.close(code=1008) # policy violation; pre-accept = HTTP 403 return await ws.accept() @@ -1128,12 +1156,22 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None: ): await ws.close(code=1002) return - with kanta.transaction("translator results", extra=lang): + text = masking.unmask(msg.text, state.spans) + state.inflight = None + state.spans = [] + if text is None: + # The model mangled the sentinels: drop the result and + # skip the fragment for this run (it stays pending; a + # restart or a masking/prompt change gets another chance). + _mask_failures.add((lang, msg.key)) + print(f"[{lang}] result for chunk {msg.key.hex()} rejected: sentinels mangled") + _schedule_translation_dispatch() + continue + with kanta.transaction("translator results", user=clientkey, extra=lang): paths = translate.store_results( - data, lang, [translate.TransResult(key=msg.key, text=msg.text)] + data, lang, [translate.TransResult(key=msg.key, text=text)] ) _invalidate_pages() # schedules the next dispatch - state.inflight = None if paths: print(f"[{lang}] now available for {len(paths)} page(s): {', '.join(sorted(paths))}") except WebSocketDisconnect: diff --git a/pagerite/chunks.py b/pagerite/chunks.py index 604bf8a..227b4e2 100644 --- a/pagerite/chunks.py +++ b/pagerite/chunks.py @@ -11,6 +11,8 @@ import re import blake3 +from pagerite.masking import has_prose + #: Fenced code block opener/closer: up to 3 spaces indent, then 3+ #: backticks or tildes (CommonMark). _FENCE_OPEN = re.compile(r"^ {0,3}(`{3,}|~{3,})") @@ -123,17 +125,22 @@ def chunk_key(text: str) -> bytes: def needs_translation(chunk: str) -> bool: - """False for chunks without prose: pure code fences and HTML blocks. + """False for chunks without prose: pure code fences, HTML blocks, and + anything whose masked form (pagerite/masking.py) has no letters left — + container fences, lone {placeholders}, reference definitions. These are inherently no-translate (docs/migrate.md): derived from the - chunk text itself, nothing is stored. + chunk text itself, nothing is stored. Every language renders them from + the original chunk via the hybrid fallback. """ if _FENCE_OPEN.match(chunk): return False first = chunk.split("\n", 1)[0] if any(open_re.match(first) for open_re, _ in _HTML_ATOMIC): return False - return not _HTML_TAG.match(first) + if _HTML_TAG.match(first): + return False + return has_prose(chunk) def join_chunks(chunks: list[str]) -> str: diff --git a/pagerite/data.py b/pagerite/data.py index 138cbc0..6d2a206 100644 --- a/pagerite/data.py +++ b/pagerite/data.py @@ -101,10 +101,12 @@ class Data(msgspec.Struct): #: linked as on every page. Empty = the build's #: /favicon.ico. favicon: str = "" - #: API key gating the translator service WebSocket (/_translate/{key}; - #: the external forward-auth does not cover that route). Generated - #: lazily at startup when empty (see lifespan in app.py). - translate_key: str = "" + #: API keys gating the translator service WebSocket (/_translate/{key}; + #: the external forward-auth does not cover that route): key -> display + #: name. Keys are 12 lowercase alphanumeric characters; the first is + #: generated at database bootstrap (see app.py), multiple keys are a + #: future reservation (e.g. managed via a web interface). + translate_keys: dict[str, str] = {} #: Wanted target languages for the translator service (presence-keys, #: value always True). The dispatcher offers jobs only in the #: intersection of these and a connection's announced capabilities. diff --git a/pagerite/masking.py b/pagerite/masking.py new file mode 100644 index 0000000..deec748 --- /dev/null +++ b/pagerite/masking.py @@ -0,0 +1,170 @@ +"""Masking of non-translatable spans for the machine-translation round trip. + +A translator model must copy technical spans (code, URLs, {placeholders}, +attrs, footnote and link labels, container names, HTML tags) byte-identically +while translating the prose around them — and small models translate anything +that looks like a word (a {dates} placeholder once came back as +{päivämäärät}). So before a fragment is dispatched, each such span is +replaced with a numbered sentinel (``⟦1⟧``, ``⟦2⟧``, ...) — the model only +ever sees prose — and on the way back the sentinels are restored by number +(``unmask``). A result whose sentinels did not all survive — missing, +duplicated or out of range — is rejected and the fragment stays pending. + +Punctuation structure (*, |, [], (), :::) is not masked: it carries no +lexical content and models preserve it. Link and image text — including alt +text and captions — stays visible for translation; only the destination is +masked. Rule order matters: earlier rules consume syntax later ones would +misread, and no rule may match across or inside an already emitted sentinel +(the container-fence rule runs before the brace rule for that reason). +""" + +import re + +#: A masked span marker: the span's 1-based number in brackets that never +#: appear in content and are atomic enough for a model to copy verbatim. +#: unmask() validates survival, so a model that mangles them only loses its +#: own result. +_SENTINEL = re.compile(r"⟦(\d+)⟧") + +#: URL-ish span: an destination, or a whitespace-free run +#: allowing one level of balanced parens (Wikipedia-style). +_URLISH = r"<[^<>\n]*>|[^\s()]*(?:\([^()\n]*\)[^\s()]*)*" + +#: Inline code: matching backtick runs, whole span masked. The content may +#: not cross a paragraph break, so a stray backtick cannot swallow the rest +#: of the chunk. +_CODE = re.compile(r"(`+)((?:(?!\n\n).)+?)\1(?!`)", re.DOTALL) + +#: Container fence line (`:::: aside {.x}`): the name-and-attrs tail is +#: masked; a bare `:::` has nothing to mask. Runs before the brace rule so +#: fence-line attrs are masked together with the name. +_FENCE = re.compile(r"^( {0,3}:{3,})[ \t]*(\S[^\n]*)", re.MULTILINE) + +#: Link/image destination: `[text](url "title")` -> `[text](⟦N⟧ "title")`. +_DEST = re.compile(r"(\]\(\s*)(" + _URLISH + r")") + +#: Autolinks and inline HTML (, , , ). +#: A `<` followed by whitespace (a prose "a < b") is not matched. +_TAG = re.compile(r"<[A-Za-z/!?][^<>\n]*>") + +#: Footnote definition `[^label]: text...` — label masked; the text after +#: the colon is prose. +_FOOTDEF = re.compile(r"^( {0,3}\[\^)([^\]\n]+)(\]:)", re.MULTILINE) + +#: Reference-style link definition `[label]: url "title"` — label and +#: destination masked, title stays visible. +_LINKDEF = re.compile(r"^( {0,3}\[)(?!\^)([^\]\n]+)(\]:[ \t]*)(" + _URLISH + r")", re.MULTILINE) + +#: Footnote reference `[^label]` ((?!:) — definitions are _FOOTDEF's). +_FOOTREF = re.compile(r"\[\^([^\]\n]+)\](?!:)") + +#: Reference-style link usage `[text][label]` — the label. +_REFPAIR = re.compile(r"(\][ \t]?\[)([^\]\n]+)(\])") + +#: Any {...} span: {placeholders} and {#id .class} attrs alike. +_BRACES = re.compile(r"\{[^{}\n]*\}") + +#: Bare URLs in prose (GFM autolinks); trailing sentence punctuation stays +#: outside the mask. +_BARE_URL = re.compile(r"(?()\[\]]+") + +#: Any Unicode letter (digits and underscore are not prose). +_LETTER = re.compile(r"[^\W\d_]") + + +def mask(text: str) -> tuple[str, list[str]]: + """Replace every non-translatable span with a ⟦N⟧ sentinel; return the + masked text and the original spans in sentinel order.""" + spans: list[str] = [] + + def emit(original: str) -> str: + if not original: + return original + spans.append(original) + return f"⟦{len(spans)}⟧" + + for sub in (_code, _fence, _dest, _tag, _footdef, _linkdef, _footref, + _refpair, _braces, _bare_url): + text = sub(text, emit) + return text, spans + + +def unmask(text: str, spans: list[str]) -> str | None: + """Restore the masked spans into a translated fragment; None when the + sentinels did not all survive intact (missing, duplicated or out of + range) — the caller drops the result and the fragment stays pending. + + Order is not checked: translations legitimately reorder spans. + """ + if not spans: + return text + counts: dict[int, int] = {} + + def repl(m: re.Match) -> str: + n = int(m.group(1)) + counts[n] = counts.get(n, 0) + 1 + return spans[n - 1] if 0 < n <= len(spans) else m.group(0) + + restored = _SENTINEL.sub(repl, text) + if counts != dict.fromkeys(range(1, len(spans) + 1), 1): + return None + return restored + + +def has_prose(text: str) -> bool: + """True when the masked form still contains a letter — i.e. there is + something for a translator to translate. Chunks that are all markup, + placeholders or code (a lone {dates}, container fences, reference + definitions) have no business reaching the model: every language + renders them from the original chunk.""" + return bool(_LETTER.search(mask(text)[0])) + + +def _code(text: str, emit) -> str: + return _CODE.sub(lambda m: emit(m.group(0)), text) + + +def _fence(text: str, emit) -> str: + return _FENCE.sub(lambda m: m.group(1) + emit(m.group(2)), text) + + +def _dest(text: str, emit) -> str: + return _DEST.sub(lambda m: m.group(1) + emit(m.group(2)), text) + + +def _tag(text: str, emit) -> str: + return _TAG.sub(lambda m: emit(m.group(0)), text) + + +def _footdef(text: str, emit) -> str: + return _FOOTDEF.sub(lambda m: m.group(1) + emit(m.group(2)) + m.group(3), text) + + +def _linkdef(text: str, emit) -> str: + return _LINKDEF.sub( + lambda m: m.group(1) + emit(m.group(2)) + m.group(3) + emit(m.group(4)), text + ) + + +def _footref(text: str, emit) -> str: + return _FOOTREF.sub(lambda m: "[^" + emit(m.group(1)) + "]", text) + + +def _refpair(text: str, emit) -> str: + return _REFPAIR.sub(lambda m: m.group(1) + emit(m.group(2)) + m.group(3), text) + + +def _braces(text: str, emit) -> str: + return _BRACES.sub(lambda m: emit(m.group(0)), text) + + +def _bare_url(text: str, emit) -> str: + def repl(m: re.Match) -> str: + core = m.group(0) + tail = "" + while core and core[-1] in ".,;:!?": + tail = core[-1] + tail + core = core[:-1] + return emit(core) + tail + + return _BARE_URL.sub(repl, text) diff --git a/pagerite/translate.py b/pagerite/translate.py index 08ba109..64471b8 100644 --- a/pagerite/translate.py +++ b/pagerite/translate.py @@ -4,17 +4,28 @@ The external machine-translation service connects over WebSocket (``/_translate/``, see app.py) and exchanges JSON frames decoded into the tagged msgspec structs below (``bytes`` fields ride as base64 — no manual encoding anywhere). This module holds the message structs plus the -two computations shared by the WS handler: which fragments are pending for -a language (``pending_items``) and storing a result (``store_results``). +shared computations around the WS handler: which fragments are pending for +a language (``pending_items``), storing a result (``store_results``) and +the startup URL listing (``log_service_urls``). The dispatcher itself (one job at a time per connection, wanted ∩ capable language matching, requeue on disconnect) lives in app.py. + +Fragments cross the wire masked: non-translatable spans (code, URLs, +{placeholders}, tags, ...) are numbered ⟦N⟧ sentinels in ``Job.text``, +restored and validated before storage (``pagerite/masking.py``). """ +import logging +import os + import msgspec +from pagerite.__main__ import DEFAULT_PORT from pagerite.chunks import chunk_key, needs_translation from pagerite.data import Data, Node, sorted_nodes +logger = logging.getLogger(__name__) + class Hello(msgspec.Struct, tag="hello"): """Client greeting on connect: the language codes its model CAN produce @@ -42,7 +53,7 @@ class Job(msgspec.Struct, tag="job"): lang: str key: bytes #: 9-byte chunk hash (base64 in the JSON frame) - text: str + text: str #: masked Markdown (pagerite/masking.py) path: str #: article it came from ("" = front page), no leading slash kind: str #: "chunk" | "title" @@ -136,3 +147,16 @@ def store_results(data: Data, lang: str, items: list[TransResult]) -> list[str]: walk(data.menu, "") return pages + + +def log_service_urls(keys: dict[str, str], hostname: str) -> None: + """Log the translator WebSocket URL(s) for the admin at startup, one + line: "…/_translate/ ()", comma-joined — ws:// with the port + on localhost, wss:// without a port on a public hostname (the port comes + from the CLI via the PAGERITE_PORT env). No-op without keys.""" + if not keys: + return + port = os.getenv("PAGERITE_PORT", DEFAULT_PORT) + base = f"ws://localhost:{port}" if hostname == "localhost" else f"wss://{hostname}" + urls = ", ".join(f"{base}/_translate/{key} ({name})" for key, name in keys.items()) + logger.info("Translator %s", urls) diff --git a/scripts/translator.py b/scripts/translator.py index 0f12bb9..94c42d7 100644 --- a/scripts/translator.py +++ b/scripts/translator.py @@ -13,8 +13,9 @@ """Pagerite translator service: translate site content with Seed-X-PPO-7B. Connects to a Pagerite server's translator WebSocket — the full URL -including the access key (the admin finds it in the site settings, -GET /_api/settings -> ``translate_key``) — and announces the languages the +including the access key (printed at server startup; the admin also finds +the key in the site settings, GET /_api/settings -> ``translate_keys``) — +and announces the languages the model CAN translate (capabilities). The server dispatches one single-item job at a time per connection, offered only in its configured target languages (``Data.translate_langs``) ∩ the announced capabilities; a @@ -55,8 +56,10 @@ SEED_X_TAGS = { } SEED_X_NAMES = {v: k for k, v in SEED_X_TAGS.items()} -#: The fragments are Markdown; Seed-X has no system prompt, so it goes in-line. -NOTE = ", preserving all Markdown formatting, URLs and code exactly unchanged" +#: The fragments are masked Markdown (pagerite/masking.py: ⟦N⟧ sentinels +#: stand in for code, URLs, placeholders...); Seed-X has no system prompt, +#: so the instruction goes in-line. +NOTE = ", preserving all Markdown formatting and keeping every ⟦N⟧ token exactly unchanged" # The wire structs below duplicate pagerite/translate.py: this script runs @@ -76,7 +79,7 @@ class Job(msgspec.Struct, tag="job"): lang: str key: bytes #: 9-byte chunk hash (base64 in the JSON frame) - text: str + text: str #: masked (pagerite/masking.py): the ⟦N⟧ tokens must survive verbatim path: str #: article it came from ("" = front page), no leading slash kind: str #: "chunk" | "title"