Masked translation round-trip; named translator keys

- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).

AGENTS.md

@@ -57,5 +57,5 @@ Server run by CLI entry point `uv run pagerite` (no auto reloads, build needed).
 
 - Keep dependencies minimal; add via `uv add` and mention it.
 - The public URL space belongs to content (pretty slugs at root). Reserve only `/_` for the machinery (`/_api/`, `/_f/`, `/_assets/`), plus `/favicon.ico` from the build. Slugs are lowercase ASCII letters, digits, hyphens and underscores `[a-z0-9_-]` (the site editor filters input live via `slugify.js`, built on the `transliteration` npm package — unicode folds to ASCII, spaces become hyphens; an empty slug on a new page is derived from its title), may not begin with `_` or `.`, and such URLs are never looked up as content.
-- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. The one keyed exception is `/_translate/{key}` (translator service; `Data.translate_key`, see docs/localization.md).
+- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. The one keyed exception is `/_translate/{key}` (translator service; `Data.translate_keys`, see docs/localization.md).
 - Update the relevant MarkDown files when architecture, tooling, or conventions change.

docs/localization.md

@@ -258,9 +258,16 @@ local to that language.
 An external machine-translation service connects over WebSocket at
 `/_translate/{key}` — deliberately **not** under `/_api`: the SSO
 forward-auth does not cover that route, and the key in the path is the
-access control. The key is `Data.translate_key`, generated once at startup
+access control. Keys live in `Data.translate_keys` (key -> display name) —
-and surfaced to the admin in `GET /_api/settings` as `translate_key`. A
+12 lowercase alphanumeric characters each, the first one generated at
-wrong or empty key rejects the handshake (close-before-accept → HTTP 403).
+database bootstrap and multiple keys reserved for future management (e.g.
+a web UI). The full WS URL(s) are printed in the startup log
+(`ws://localhost:{port}/_translate/{key}` locally,
+`wss://{hostname}/_translate/{key}` on a public hostname) and the keys are
+surfaced to the admin in `GET /_api/settings` as `translate_keys`. An
+unknown or empty key rejects the handshake (close-before-accept → HTTP
+403). Transactions storing results record the connecting key as the kanta
+transaction `user`.
 
 Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
 `bytes` fields ride as base64):
@@ -269,7 +276,8 @@ Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
   **capabilities**: the language codes its model can produce (normalized
   to base subtags; `en`/empty dropped).
 - `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
-  ONE fragment to translate (an article title or a chunk).
+  ONE fragment to translate (an article title or a chunk), its text
+  **masked** (see Masking below).
 - `{"type": "result", "lang", "key", "text"}` — client reply: the
   translation of the connection's current job, matching it by (lang, key).
 
@@ -299,6 +307,28 @@ Results are stored into `trans` in one transaction and set
 pages gain a language from one fragment). Unknown keys are stored anyway
 and re-storing overwrites — results are idempotent.
 
+#### Masking
+
+Fragments cross the wire **masked** (`pagerite/masking.py`): spans the model
+must copy byte-identically are replaced with numbered `⟦N⟧` sentinels before
+dispatch and restored by number from the result. Masked: code spans,
+container-fence names, link and image *destinations* (link text, alt text
+and captions stay visible for translation), reference and footnote labels,
+`{...}` spans (placeholders like `{dates}` as well as attrs), inline HTML
+tags and bare URLs. Markdown punctuation (`*`, `|`, `[]()`, `:::`) is not
+masked — it carries no lexical content and models preserve it. Chunks with
+no prose left after masking (a lone `{dates}`, container fences, pure
+code/HTML) are never dispatched at all (`needs_translation`); every language
+renders them from the original chunk.
+
+A result is accepted only if every sentinel survived exactly once, in any
+order (translations legitimately reorder spans). A mangled result is dropped
+and logged, and the (lang, key) pair is skipped for the rest of the server
+run — generation is near-deterministic, so an immediate retry would re-fail
+the same way; the fragment stays pending and gets another chance on restart
+or a model/masking change. `Data.trans` therefore only ever holds clean,
+unmasked text.
+
 ### Explicitly out of scope for phase 2
 
 - The machine translation itself: the API above moves fragments in and out;

docs/migrate.md

@@ -52,9 +52,9 @@ class Node(msgspec.Struct, omit_defaults=True):
 
 class Data(msgspec.Struct):
     ...
-    #: API key gating the translator service WebSocket (/_translate/{key});
+    #: API keys gating the translator service WebSocket (/_translate/{key}):
-    #: generated lazily at startup (see the lifespan in app.py).
+    #: key -> display name; the first is generated at bootstrap (app.py).
-    translate_key: str = ""
+    translate_keys: dict[str, str] = {}
     #: Wanted target languages for the translator service (presence-keys);
     #: jobs are offered only in these ∩ a connection's capabilities.
     translate_langs: dict[str, True] = {}
@@ -81,8 +81,10 @@ Notes:
   `trans.get(hash(node.title), {}).get(lang)`. No separate title storage;
   editing a title invalidates its translations automatically.
 - **Per-hunk options** live in two places: *inherent* options are derived at
-  chunking time (code fences and HTML blocks are marked no-translate without
+  chunking time (code fences, HTML blocks and prose-free chunks are
-  storing anything); *editor-set* flags are `node.no_trans` (keyed by chunk
+  no-translate without storing anything — `needs_translation`, see
+  docs/localization.md "Masking"); *editor-set* flags are `node.no_trans`
+  (keyed by chunk
   hash, so a heavy edit silently drops the flag — acceptable and
   self-healing).
 - **Patch payloads stay inline** in `Patch.hunks` — patches are small by

frontend/vite.config.js

@@ -16,7 +16,7 @@ const CONTENT_PROXY = '^(?!/_|/@|/src|/node_modules|/__).*$'
 // https://vite.dev/config/
 export default defineConfig({
   plugins: [
-    fastapiVue({ paths: ["/_api", "/_f", "/_themes", "/_fonts", "/_a"] }),
+    fastapiVue({ paths: ["/_api", "/_f", "/_themes", "/_fonts", "/_a", "/_translate"] }),
     vue(),
     vueDevTools(),
   ],

pagerite/__main__.py

@@ -9,6 +9,7 @@ from pathlib import Path
 
 import httpx
 from fastapi_vue import server
+from fastapi_vue.hostutil import parse_endpoints
 
 DEFAULT_PORT = 8100
 DEVMODE = os.getenv("PAGERITE_DEV") == "1"
@@ -96,6 +97,12 @@ def main() -> None:
     # Export the hostname before pagerite.app is imported: it derives the
     # data directory and public origin from it at import time.
     os.environ["PAGERITE_HOSTNAME"] = args.hostname
+    # And the listen port: the app prints the translator WS URL at startup,
+    # which for localhost includes the actual port.
+    for endpoint in parse_endpoints(args.listen, DEFAULT_PORT):
+        if "port" in endpoint:
+            os.environ["PAGERITE_PORT"] = str(endpoint["port"])
+            break
     if args.dbip:
         _download_dbip()
     server.run(

pagerite/app.py

@@ -15,6 +15,7 @@ walking the tree (``resolve``), moves are slot detach/attach
 import asyncio
 import gzip
 import ipaddress
+import logging
 import mimetypes
 import os
 import re
@@ -48,7 +49,7 @@ from mediapreview import dispatch
 from pydantic import BaseModel
 from zstandard import ZstdCompressor
 
-from pagerite import analytics, i18n, seed, translate, views
+from pagerite import analytics, i18n, masking, seed, translate, views
 from pagerite.__main__ import DEVMODE
 from pagerite.chunks import store_chunks
 from pagerite.data import (
@@ -63,6 +64,8 @@ from pagerite.data import (
 )
 from pagerite.markdown import render, toggle_task
 
+logger = logging.getLogger(__name__)
+
 # Site identity: the hostname comes from the CLI (first positional argument,
 # exported as PAGERITE_HOSTNAME) and names the per-site data directory
 # ``<hostname>/{content.kantadb, analytics.json, files}`` under the cwd.
@@ -283,14 +286,26 @@ def _seed(data: Data) -> None:
         node.order = order
 
 
+#: Translator key format: 12 lowercase alphanumeric characters — not
+#: brute-forceable over a WebSocket handshake, still human-manageable.
+_KEY_ALPHABET = "abcdefghijklmnopqrstuvwxyz0123456789"
+
+
+@kanta.bootstrap
+def _translate_key(data: Data) -> None:
+    """Generate the first translator service key on database creation.
+
+    Keys are a dict (key -> display name) with the future reservation that
+    multiple keys could be managed (e.g. via a web interface)."""
+    key = "".join(secrets.choice(_KEY_ALPHABET) for _ in range(12))
+    data.translate_keys[key] = "default"
+
+
 @asynccontextmanager
 async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
     """Open the database (migrations run inside kanta.open), load assets, load GeoIP."""
     await kanta.open()
-    # The translator service key is persisted; generated once on first boot.
+    translate.log_service_urls(data.translate_keys, HOSTNAME)
-    if not data.translate_key:
-        with kanta.transaction("generate translate key"):
-            data.translate_key = secrets.token_urlsafe(24)
     await asyncio.to_thread(file_store.load)
     await frontend.load()
     # Decompress/open the DB-IP MMDB once at startup.  Lookups are then
@@ -633,7 +648,7 @@ async def update_structure(op: StructureOp) -> None:
 async def get_settings() -> dict:
     """Site-wide settings (brand, theme, custom CSS and favicon URL), plus
     the themes, banner designs and user fonts available on disk for the
-    selectors, the translator service key and the wanted translation
+    selectors, the translator service keys and the wanted translation
     languages (for the /_translate socket)."""
     return {
         "brand": data.brand,
@@ -646,7 +661,7 @@ async def get_settings() -> dict:
         "fonts": views._user_fonts(),
         "transition": data.transition,
         "transitions": views._transition_names(),
-        "translate_key": data.translate_key,
+        "translate_keys": data.translate_keys,
         "translate_langs": sorted(data.translate_langs),
     }
 
@@ -994,7 +1009,7 @@ async def delete_page(path: str) -> None:
 
 
 # WebSocket API for external translation services (not under /_api: it is keyed
-# with Data.translate_key instead of the SSO forward-auth). The server is a
+# with Data.translate_keys instead of the SSO forward-auth). The server is a
 # dispatcher: one single-item job at a time per connection, offered in the
 # intersection of the wanted languages (Data.translate_langs) and the
 # connection's announced capabilities. Results are matched to content by
@@ -1002,7 +1017,9 @@ async def delete_page(path: str) -> None:
 class _TranslatorState:
     """One connected translator socket: the language codes it announced as
     capabilities (Hello) and the (lang, chunk-key) job currently in flight
-    on it — one at a time, the next is sent only after its Result.
+    on it, with the mask spans to restore into its Result
+    (pagerite/masking.py) — one at a time, the next is sent only after its
+    Result.
 
     Per-connection only: in-flight lives solely here, so on disconnect the
     item simply becomes pending again and is re-offered to any free capable
@@ -1011,11 +1028,17 @@ class _TranslatorState:
     def __init__(self, capable: set[str]) -> None:
         self.capable = capable
         self.inflight: tuple[str, bytes] | None = None
+        self.spans: list[str] = []  # mask spans of the in-flight job
 
 
 #: Connected translator sockets and their per-connection state.
 _translator_clients: dict[WebSocket, _TranslatorState] = {}
 
+#: (lang, chunk key) of fragments whose result failed sentinel validation
+#: (masking.unmask): skipped on later dispatches this run — generation is
+#: near-deterministic, so an immediate retry would just re-fail.
+_mask_failures: set[tuple[str, bytes]] = set()
+
 
 def _schedule_translation_dispatch() -> None:
     """Schedule a dispatch pass, if any translator is connected.
@@ -1057,19 +1080,23 @@ async def _dispatch_translations() -> None:
             continue
         inflight = {s.inflight for s in _translator_clients.values() if s.inflight}
         job = None
+        spans: list[str] = []
         for lang in sorted(langs):
             for item in translate.pending_items(data, lang):
-                if (lang, item.key) not in inflight:
+                if (lang, item.key) in inflight or (lang, item.key) in _mask_failures:
-                    job = translate.Job(
+                    continue
-                        lang=lang, key=item.key, text=item.text,
+                masked, spans = masking.mask(item.text)
-                        path=item.path, kind=item.kind,
+                job = translate.Job(
-                    )
+                    lang=lang, key=item.key, text=masked,
-                    break
+                    path=item.path, kind=item.kind,
+                )
+                break
             if job is not None:
                 break
         if job is None:
             continue
         state.inflight = (job.lang, job.key)  # before the await: no double-assign
+        state.spans = spans
         try:
             await ws.send_text(msgspec.json.encode(job).decode())
         except Exception:  # send failed: the receive loop cleans up
@@ -1082,7 +1109,8 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None:
 
     Deliberately NOT under /_api/: the external forward-auth is skipped;
     the server-generated client key in the path is the access control
-    (``Data.translate_key``, generated at startup, shown in the admin's
+    (``Data.translate_keys``: key -> display name; the first is generated
+    at bootstrap, all are shown in the admin's
     /_api/settings). A wrong/empty key rejects the handshake — closing
     before accept makes Starlette answer HTTP 403.
 
@@ -1095,7 +1123,7 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None:
     different (lang, key), a duplicate Hello, or any malformed frame
     closes the socket with a protocol error.
     """
-    if not data.translate_key or clientkey != data.translate_key:
+    if clientkey not in data.translate_keys:
         await ws.close(code=1008)  # policy violation; pre-accept = HTTP 403
         return
     await ws.accept()
@@ -1128,12 +1156,22 @@ async def translate_ws(ws: WebSocket, clientkey: str) -> None:
                 ):
                     await ws.close(code=1002)
                     return
-                with kanta.transaction("translator results", extra=lang):
+                text = masking.unmask(msg.text, state.spans)
+                state.inflight = None
+                state.spans = []
+                if text is None:
+                    # The model mangled the sentinels: drop the result and
+                    # skip the fragment for this run (it stays pending; a
+                    # restart or a masking/prompt change gets another chance).
+                    _mask_failures.add((lang, msg.key))
+                    print(f"[{lang}] result for chunk {msg.key.hex()} rejected: sentinels mangled")
+                    _schedule_translation_dispatch()
+                    continue
+                with kanta.transaction("translator results", user=clientkey, extra=lang):
                     paths = translate.store_results(
-                        data, lang, [translate.TransResult(key=msg.key, text=msg.text)]
+                        data, lang, [translate.TransResult(key=msg.key, text=text)]
                     )
                     _invalidate_pages()  # schedules the next dispatch
-                state.inflight = None
                 if paths:
                     print(f"[{lang}] now available for {len(paths)} page(s): {', '.join(sorted(paths))}")
     except WebSocketDisconnect:

pagerite/chunks.py

@@ -11,6 +11,8 @@ import re
 
 import blake3
 
+from pagerite.masking import has_prose
+
 #: Fenced code block opener/closer: up to 3 spaces indent, then 3+
 #: backticks or tildes (CommonMark).
 _FENCE_OPEN = re.compile(r"^ {0,3}(`{3,}|~{3,})")
@@ -123,17 +125,22 @@ def chunk_key(text: str) -> bytes:
 
 
 def needs_translation(chunk: str) -> bool:
-    """False for chunks without prose: pure code fences and HTML blocks.
+    """False for chunks without prose: pure code fences, HTML blocks, and
+    anything whose masked form (pagerite/masking.py) has no letters left —
+    container fences, lone {placeholders}, reference definitions.
 
     These are inherently no-translate (docs/migrate.md): derived from the
-    chunk text itself, nothing is stored.
+    chunk text itself, nothing is stored. Every language renders them from
+    the original chunk via the hybrid fallback.
     """
     if _FENCE_OPEN.match(chunk):
         return False
     first = chunk.split("\n", 1)[0]
     if any(open_re.match(first) for open_re, _ in _HTML_ATOMIC):
         return False
-    return not _HTML_TAG.match(first)
+    if _HTML_TAG.match(first):
+        return False
+    return has_prose(chunk)
 
 
 def join_chunks(chunks: list[str]) -> str:

pagerite/data.py

@@ -101,10 +101,12 @@ class Data(msgspec.Struct):
     #: linked as <link rel="icon"> on every page. Empty = the build's
     #: /favicon.ico.
     favicon: str = ""
-    #: API key gating the translator service WebSocket (/_translate/{key};
+    #: API keys gating the translator service WebSocket (/_translate/{key};
-    #: the external forward-auth does not cover that route). Generated
+    #: the external forward-auth does not cover that route): key -> display
-    #: lazily at startup when empty (see lifespan in app.py).
+    #: name. Keys are 12 lowercase alphanumeric characters; the first is
-    translate_key: str = ""
+    #: generated at database bootstrap (see app.py), multiple keys are a
+    #: future reservation (e.g. managed via a web interface).
+    translate_keys: dict[str, str] = {}
     #: Wanted target languages for the translator service (presence-keys,
     #: value always True). The dispatcher offers jobs only in the
     #: intersection of these and a connection's announced capabilities.

pagerite/masking.py

@@ -0,0 +1,170 @@
+"""Masking of non-translatable spans for the machine-translation round trip.
+
+A translator model must copy technical spans (code, URLs, {placeholders},
+attrs, footnote and link labels, container names, HTML tags) byte-identically
+while translating the prose around them — and small models translate anything
+that looks like a word (a {dates} placeholder once came back as
+{päivämäärät}). So before a fragment is dispatched, each such span is
+replaced with a numbered sentinel (``⟦1⟧``, ``⟦2⟧``, ...) — the model only
+ever sees prose — and on the way back the sentinels are restored by number
+(``unmask``). A result whose sentinels did not all survive — missing,
+duplicated or out of range — is rejected and the fragment stays pending.
+
+Punctuation structure (*, |, [], (), :::) is not masked: it carries no
+lexical content and models preserve it. Link and image text — including alt
+text and captions — stays visible for translation; only the destination is
+masked. Rule order matters: earlier rules consume syntax later ones would
+misread, and no rule may match across or inside an already emitted sentinel
+(the container-fence rule runs before the brace rule for that reason).
+"""
+
+import re
+
+#: A masked span marker: the span's 1-based number in brackets that never
+#: appear in content and are atomic enough for a model to copy verbatim.
+#: unmask() validates survival, so a model that mangles them only loses its
+#: own result.
+_SENTINEL = re.compile(r"⟦(\d+)⟧")
+
+#: URL-ish span: an <angle-bracketed> destination, or a whitespace-free run
+#: allowing one level of balanced parens (Wikipedia-style).
+_URLISH = r"<[^<>\n]*>|[^\s()]*(?:\([^()\n]*\)[^\s()]*)*"
+
+#: Inline code: matching backtick runs, whole span masked. The content may
+#: not cross a paragraph break, so a stray backtick cannot swallow the rest
+#: of the chunk.
+_CODE = re.compile(r"(`+)((?:(?!\n\n).)+?)\1(?!`)", re.DOTALL)
+
+#: Container fence line (`:::: aside {.x}`): the name-and-attrs tail is
+#: masked; a bare `:::` has nothing to mask. Runs before the brace rule so
+#: fence-line attrs are masked together with the name.
+_FENCE = re.compile(r"^( {0,3}:{3,})[ \t]*(\S[^\n]*)", re.MULTILINE)
+
+#: Link/image destination: `[text](url "title")` -> `[text](⟦N⟧ "title")`.
+_DEST = re.compile(r"(\]\(\s*)(" + _URLISH + r")")
+
+#: Autolinks and inline HTML (<http://...>, <b>, <!-- ... -->, <? ... ?>).
+#: A `<` followed by whitespace (a prose "a < b") is not matched.
+_TAG = re.compile(r"<[A-Za-z/!?][^<>\n]*>")
+
+#: Footnote definition `[^label]: text...` — label masked; the text after
+#: the colon is prose.
+_FOOTDEF = re.compile(r"^( {0,3}\[\^)([^\]\n]+)(\]:)", re.MULTILINE)
+
+#: Reference-style link definition `[label]: url "title"` — label and
+#: destination masked, title stays visible.
+_LINKDEF = re.compile(r"^( {0,3}\[)(?!\^)([^\]\n]+)(\]:[ \t]*)(" + _URLISH + r")", re.MULTILINE)
+
+#: Footnote reference `[^label]` ((?!:) — definitions are _FOOTDEF's).
+_FOOTREF = re.compile(r"\[\^([^\]\n]+)\](?!:)")
+
+#: Reference-style link usage `[text][label]` — the label.
+_REFPAIR = re.compile(r"(\][ \t]?\[)([^\]\n]+)(\])")
+
+#: Any {...} span: {placeholders} and {#id .class} attrs alike.
+_BRACES = re.compile(r"\{[^{}\n]*\}")
+
+#: Bare URLs in prose (GFM autolinks); trailing sentence punctuation stays
+#: outside the mask.
+_BARE_URL = re.compile(r"(?<![\w/])(?:https?://|www\.)[^\s<>()\[\]]+")
+
+#: Any Unicode letter (digits and underscore are not prose).
+_LETTER = re.compile(r"[^\W\d_]")
+
+
+def mask(text: str) -> tuple[str, list[str]]:
+    """Replace every non-translatable span with a ⟦N⟧ sentinel; return the
+    masked text and the original spans in sentinel order."""
+    spans: list[str] = []
+
+    def emit(original: str) -> str:
+        if not original:
+            return original
+        spans.append(original)
+        return f"⟦{len(spans)}⟧"
+
+    for sub in (_code, _fence, _dest, _tag, _footdef, _linkdef, _footref,
+                _refpair, _braces, _bare_url):
+        text = sub(text, emit)
+    return text, spans
+
+
+def unmask(text: str, spans: list[str]) -> str | None:
+    """Restore the masked spans into a translated fragment; None when the
+    sentinels did not all survive intact (missing, duplicated or out of
+    range) — the caller drops the result and the fragment stays pending.
+
+    Order is not checked: translations legitimately reorder spans.
+    """
+    if not spans:
+        return text
+    counts: dict[int, int] = {}
+
+    def repl(m: re.Match) -> str:
+        n = int(m.group(1))
+        counts[n] = counts.get(n, 0) + 1
+        return spans[n - 1] if 0 < n <= len(spans) else m.group(0)
+
+    restored = _SENTINEL.sub(repl, text)
+    if counts != dict.fromkeys(range(1, len(spans) + 1), 1):
+        return None
+    return restored
+
+
+def has_prose(text: str) -> bool:
+    """True when the masked form still contains a letter — i.e. there is
+    something for a translator to translate. Chunks that are all markup,
+    placeholders or code (a lone {dates}, container fences, reference
+    definitions) have no business reaching the model: every language
+    renders them from the original chunk."""
+    return bool(_LETTER.search(mask(text)[0]))
+
+
+def _code(text: str, emit) -> str:
+    return _CODE.sub(lambda m: emit(m.group(0)), text)
+
+
+def _fence(text: str, emit) -> str:
+    return _FENCE.sub(lambda m: m.group(1) + emit(m.group(2)), text)
+
+
+def _dest(text: str, emit) -> str:
+    return _DEST.sub(lambda m: m.group(1) + emit(m.group(2)), text)
+
+
+def _tag(text: str, emit) -> str:
+    return _TAG.sub(lambda m: emit(m.group(0)), text)
+
+
+def _footdef(text: str, emit) -> str:
+    return _FOOTDEF.sub(lambda m: m.group(1) + emit(m.group(2)) + m.group(3), text)
+
+
+def _linkdef(text: str, emit) -> str:
+    return _LINKDEF.sub(
+        lambda m: m.group(1) + emit(m.group(2)) + m.group(3) + emit(m.group(4)), text
+    )
+
+
+def _footref(text: str, emit) -> str:
+    return _FOOTREF.sub(lambda m: "[^" + emit(m.group(1)) + "]", text)
+
+
+def _refpair(text: str, emit) -> str:
+    return _REFPAIR.sub(lambda m: m.group(1) + emit(m.group(2)) + m.group(3), text)
+
+
+def _braces(text: str, emit) -> str:
+    return _BRACES.sub(lambda m: emit(m.group(0)), text)
+
+
+def _bare_url(text: str, emit) -> str:
+    def repl(m: re.Match) -> str:
+        core = m.group(0)
+        tail = ""
+        while core and core[-1] in ".,;:!?":
+            tail = core[-1] + tail
+            core = core[:-1]
+        return emit(core) + tail
+
+    return _BARE_URL.sub(repl, text)

pagerite/translate.py

@@ -4,17 +4,28 @@ The external machine-translation service connects over WebSocket
 (``/_translate/<key>``, see app.py) and exchanges JSON frames decoded into
 the tagged msgspec structs below (``bytes`` fields ride as base64 — no
 manual encoding anywhere). This module holds the message structs plus the
-two computations shared by the WS handler: which fragments are pending for
+shared computations around the WS handler: which fragments are pending for
-a language (``pending_items``) and storing a result (``store_results``).
+a language (``pending_items``), storing a result (``store_results``) and
+the startup URL listing (``log_service_urls``).
 The dispatcher itself (one job at a time per connection, wanted ∩ capable
 language matching, requeue on disconnect) lives in app.py.
+
+Fragments cross the wire masked: non-translatable spans (code, URLs,
+{placeholders}, tags, ...) are numbered ⟦N⟧ sentinels in ``Job.text``,
+restored and validated before storage (``pagerite/masking.py``).
 """
 
+import logging
+import os
+
 import msgspec
 
+from pagerite.__main__ import DEFAULT_PORT
 from pagerite.chunks import chunk_key, needs_translation
 from pagerite.data import Data, Node, sorted_nodes
 
+logger = logging.getLogger(__name__)
+
 
 class Hello(msgspec.Struct, tag="hello"):
     """Client greeting on connect: the language codes its model CAN produce
@@ -42,7 +53,7 @@ class Job(msgspec.Struct, tag="job"):
 
     lang: str
     key: bytes  #: 9-byte chunk hash (base64 in the JSON frame)
-    text: str
+    text: str  #: masked Markdown (pagerite/masking.py)
     path: str  #: article it came from ("" = front page), no leading slash
     kind: str  #: "chunk" | "title"
 
@@ -136,3 +147,16 @@ def store_results(data: Data, lang: str, items: list[TransResult]) -> list[str]:
 
     walk(data.menu, "")
     return pages
+
+
+def log_service_urls(keys: dict[str, str], hostname: str) -> None:
+    """Log the translator WebSocket URL(s) for the admin at startup, one
+    line: "…/_translate/<key> (<name>)", comma-joined — ws:// with the port
+    on localhost, wss:// without a port on a public hostname (the port comes
+    from the CLI via the PAGERITE_PORT env). No-op without keys."""
+    if not keys:
+        return
+    port = os.getenv("PAGERITE_PORT", DEFAULT_PORT)
+    base = f"ws://localhost:{port}" if hostname == "localhost" else f"wss://{hostname}"
+    urls = ", ".join(f"{base}/_translate/{key} ({name})" for key, name in keys.items())
+    logger.info("Translator %s", urls)

scripts/translator.py

@@ -13,8 +13,9 @@
 """Pagerite translator service: translate site content with Seed-X-PPO-7B.
 
 Connects to a Pagerite server's translator WebSocket — the full URL
-including the access key (the admin finds it in the site settings,
+including the access key (printed at server startup; the admin also finds
-GET /_api/settings -> ``translate_key``) — and announces the languages the
+the key in the site settings, GET /_api/settings -> ``translate_keys``) —
+and announces the languages the
 model CAN translate (capabilities). The server dispatches one single-item
 job at a time per connection, offered only in its configured target
 languages (``Data.translate_langs``) ∩ the announced capabilities; a
@@ -55,8 +56,10 @@ SEED_X_TAGS = {
 }
 SEED_X_NAMES = {v: k for k, v in SEED_X_TAGS.items()}
 
-#: The fragments are Markdown; Seed-X has no system prompt, so it goes in-line.
+#: The fragments are masked Markdown (pagerite/masking.py: ⟦N⟧ sentinels
-NOTE = ", preserving all Markdown formatting, URLs and code exactly unchanged"
+#: stand in for code, URLs, placeholders...); Seed-X has no system prompt,
+#: so the instruction goes in-line.
+NOTE = ", preserving all Markdown formatting and keeping every ⟦N⟧ token exactly unchanged"
 
 
 # The wire structs below duplicate pagerite/translate.py: this script runs
@@ -76,7 +79,7 @@ class Job(msgspec.Struct, tag="job"):
 
     lang: str
     key: bytes  #: 9-byte chunk hash (base64 in the JSON frame)
-    text: str
+    text: str  #: masked (pagerite/masking.py): the ⟦N⟧ tokens must survive verbatim
     path: str  #: article it came from ("" = front page), no leading slash
     kind: str  #: "chunk" | "title"