From 2a144a7dd684c32fb29372cf35630aa0d86427a0 Mon Sep 17 00:00:00 2001 From: Leo Vasanko Date: Sun, 20 Sep 2026 23:20:36 +0000 Subject: [PATCH] translate: job modes (segments/markdown/article), mode-scoped validation, article decomposition Hello gains model + modes; Job gains mode. Dispatch routes per connection capability: article jobs only to article-capable connections and only while a page is mostly pending; titles/chunks go as markdown (whole fragment, hybrid-neighbor contexts) or segments jobs. Validation skip list is now (lang, key, mode). align_article decomposes a whole-article result into per-chunk stores: non-translatable blocks (fences, HTML, containers) are verbatim anchors, regions between anchors pair positionally, mismatched regions and blocks with altered link destinations/placeholders stay pending. Verified offline against /tmp/llmtrial outputs: qwen3.8:27b runs pair 105/105 and 43/43 blocks; qwen3:30b-instruct's translated code comments, qwen3-next's degenerate runs and the structurally broken seedx trial files (a dropped ::: fence) are all rejected. --- docs/llm-translation.md | 193 ++++++++++++++++ pagerite/translate.py | 473 ++++++++++++++++++++++++++++++---------- 2 files changed, 550 insertions(+), 116 deletions(-) create mode 100644 docs/llm-translation.md diff --git a/docs/llm-translation.md b/docs/llm-translation.md new file mode 100644 index 0000000..c5dce5c --- /dev/null +++ b/docs/llm-translation.md @@ -0,0 +1,193 @@ +# Whole-article and scoped LLM translation + +Design for augmenting the fragment-based machine translation +(docs/localization.md) with general-purpose instruct LLMs that understand +Markdown natively — as opposed to pure text-to-text models like Seed-X. + +## Motivation + +The chunk + segment pipeline (`chunks.py` → `segments.py` → Seed-X) exists +because Seed-X mangles Markdown: links, formatting, fences and placeholders +must be stripped before dispatch and re-inserted into the result. The +re-insertion of link and formatting markup is the imprecise part: when +word-alignment by form similarity finds no anchor (always for CJK targets), +positions fall back to word-weight ratios, which land a word or so off. +All of `segments.py` — segmentation, offset splicing, `_find_mark`, +weight-ratio fallback, `_NEUTRAL` punctuation swaps, `<` encoding — is +defensive scaffolding around that one limitation. + +An instruct LLM translates Markdown natively: `[text](url)` stays intact +and moves as a unit, fences, container markers, attrs and `{...}` +placeholders are preserved, and link texts translate in sentence context. +For such a translator the entire segments layer is unnecessary. + +## Trial evidence (2026-09, RTX 4090 24 GB + 128 GB RAM, ollama 0.34) + +Whole-article translation of two real articles (5.5 KB marketing, 18.3 KB +technical with code fences and `{dates}`) into fi/es/zh: + +- **qwen3.8:27b** (dense, 17 GB Q4 — fits VRAM): structure-perfect on all + runs — URLs, placeholders, heading/block counts preserved, fenced code + byte-identical. es/zh excellent; fi fluent with occasional lexical slips + (covered by the human patch layer). ~30 s per short article, ~2.5 min + for 18 KB. **The reference model for article and markdown modes.** +- **qwen3:30b-instruct**: 3× faster, good prose, but rewrote comments and + docstrings inside code fences despite explicit instructions — fails + anchor validation (see below). +- **qwen3-next:80b** (MoE): best Finnish word choice on short documents, + but degenerates on longer input in every configuration tried — runaway + thinking loops (293k tokens), empty responses, 3× length output with + hallucinated URLs, and ~10× blowup even in 2 KB scoped chunks. Unusable + on current ollama builds. +- **CPU-only** (i7-14700, 14 threads): MoE 3B-active 10.6 t/s generation + (viable for batch), dense 27B 2.5 t/s (not viable). Hybrid GPU+CPU + splits bottleneck prompt evaluation (~52 t/s vs 62 t/s pure CPU) — + dense GPU-resident or MoE CPU-resident are the sane configurations; + mixing hurts. + +Operational requirements established by the trials (all client-side): + +- **Always disable thinking** for hybrid models (`think: false` on + ollama): the reasoning phase adds minutes per article and can loop + unbounded. +- **Always cap generation** (`num_predict` ≈ 2–3× input tokens): a + runaway on a whole-article job burns hours, vs. seconds for a + Seed-X segment. +- temperature 0.2 with the strict structure prompt works well for + qwen3.8. + +## What carries over unchanged + +The valuable parts of the current design are the **storage and staleness +model**, not the segmentation — and none of them require the machine +translation to be produced chunk by chunk. The chunk store is a +storage/diffing format; LLM output at any granularity is *projected +into* it: + +- Content-addressed source chunks (`Data.chunks`, `chunk_key`) — staleness + still falls out of source-hash keys: editing the original invalidates + exactly the edited chunks, all other translations keep applying. +- Per-chunk machine translations (`Data.trans[hash][lang]`) and the hybrid + render with per-chunk fallback to the original. +- User patches (`Data.patches`) — search/replace hunks over the assembled + hybrid, per-hunk independent and best-effort. Patches are orthogonal to + how `Data.trans` entries were produced. +- `pending_items`: after a source edit, exactly the changed (lang, hash) + pairs are pending — **focused retranslation of edits falls out of the + existing bookkeeping**, no whole-article reruns. + +## Protocol: capabilities and job modes + +The `/_translate/{key}` WebSocket stays the single channel; Seed-X +clients work unchanged. The client→server `Hello` gains two optional +fields: + +```python +class Hello(msgspec.Struct, tag="hello"): + langs: list[str] # as today: languages the model can produce + model: str = "" # free-form model string (logging, debugging) + modes: list[str] = ["segments"] # job granularities accepted +``` + +Three job modes, in increasing granularity: + +- **`segments`** — the current protocol, unchanged: `Job.texts` carries + prose segments (markup never crosses the wire), `Result.texts` returns + them, the server splices by offset (`segments.py`). For text-to-text + models (Seed-X). Default when a client omits `modes`. +- **`markdown`** (scoped instruct mode) — one fragment as full Markdown: + a body chunk or a title. `Job.texts` carries a single element, the + chunk's Markdown; `Job.contexts` carries up to two context strings + (previous and next block of the **served hybrid** in the target + language — current machine translation with user patches applied), + "" where none. The client is instructed to output ONLY the translation + of the target block; the context is terminology/tone reference. + Using the *patched* hybrid as context propagates human corrections + into fresh machine translations without the LLM ever touching patch + storage. `Result.texts` carries one element, the translated block. + The server validates: exactly one block after re-chunking, anchor + constructs (URLs, image destinations, code fence content, `{...}` + placeholders) preserved where the source block has them, then stores + to `Data.trans` as usual. +- **`article`** — a whole page. `Job.texts` carries one element, the full + original Markdown (the chunk sequence is recoverable server-side via + `node.chunks`); `Result.texts` carries one element, the full translated + Markdown. The server decomposes (below) and stores per chunk. + +Titles are jobs like any other in all modes (`kind="title"` keeps its +article-opening context rule; in `markdown` mode a title crosses as +plain text, since it carries no markup by construction). + +### Dispatch and validation + +- Routing is per connection as today (wanted ∩ capable, one job in + flight, requeue on disconnect), extended by mode: the smallest + suitable unit goes to each free connection — `article` jobs only to + article-capable connections, and only while a page is *mostly* + pending (a whole new article or a full refresh); steady-state edit + follow-up is `markdown`/`segments` jobs. Mixed translator fleets (a + Seed-X instance, a local qwen, an API-backed client) run concurrently + and share the work by capability. +- The validation skip-list becomes **mode-scoped** (`(lang, key, mode)`): + a fragment a Seed-X client rejects stays offerable to instruct clients + (and vice versa) — near-deterministic re-failure applies per model, + not across approaches. +- `Result` matching is unchanged (lang, key); article results match on + the key of the article's first chunk. + +## Article result decomposition + +1. Re-chunk the translated article with the same `chunk_markdown`. +2. Align translated blocks to source blocks. A well-behaved model does + not reorder paragraphs, so positional / `SequenceMatcher` alignment + at block granularity suffices. Blocks that must not change — code + fences, container fence lines, `{...}` placeholders, image + destinations, raw HTML — are matched verbatim and serve as alignment + anchors, like diff context lines. +3. Store each translated block in `Data.trans[source_chunk_hash][lang]`. + +Validation happens *before* anything is stored, same spirit as the +`pure_prose` segment checks but structural: + +- Anchor blocks must appear verbatim and in order (this is what rejects + qwen3:30b-instruct's translated code comments automatically). +- Per anchor-bounded region, source and translated block counts must + match 1:1; regions that don't align store nothing and their chunks + stay pending (they fall back to `markdown`-mode scoped jobs). + +## Reference client + +A second client script next to `scripts/translator.py` speaking the +`markdown` and `article` modes. Internally it targets the **OpenAI +Chat Completions API shape** (`POST /v1/chat/completions`): ollama +serves it at `:11434/v1`, llama.cpp's server likewise, and hosted APIs +(OpenAI and compatible providers) natively — `base_url` + `model` + +optional API key in the client's config selects local GPU, local CPU or +a remote model, with backend quirks (ollama's `think: false`, +`num_predict` cap, per-model sampling) in a per-model config section. +How the client drives its LLM is its internal matter; the wire protocol +above is the contract. + +The client announces in `Hello`: + +- `model`: the model string it is actually serving (e.g. `qwen3.8:27b`) +- `langs`: from its per-model language table — for the shipped qwen3.8 + configuration the site languages as configured server-side + (de, es, fi, pt, zh; Finnish flagged as the weakest, patch-covered) +- `modes`: `["markdown", "article"]` for a structure-proven model, + `["markdown"]` for one that is only trusted in scoped mode + +The Seed-X client is untouched and announces `["segments"]` (implicitly, +by omitting `modes`). + +## Importing human-made full translations + +The decomposition function doubles as an import path for translations +produced outside the pipeline — e.g. an article translated with ChatGPT +and pasted back. Today such a paste lands in the translation editor and +is stored as one giant user patch; feeding it through the same +decomposition instead writes proper `Data.trans` fragments, so later +source edits invalidate and re-translate per chunk rather than letting +the monolithic patch silently go stale hunk by hunk. This import path is +also the natural testbed for the decomposition and validation logic +before any live LLM client uses it. diff --git a/pagerite/translate.py b/pagerite/translate.py index d057524..1ed9df2 100644 --- a/pagerite/translate.py +++ b/pagerite/translate.py @@ -7,29 +7,48 @@ as base64 — no manual encoding anywhere). This module holds everything else: the message structs, the connected-client dispatcher (``Dispatcher`` — one job at a time per connection, wanted ∩ capable language matching, requeue on disconnect), which fragments are pending for a language -(``pending_items``) and storing a result (``store_results``). +(``pending_items``) and storing results (``store_results``). -Fragments cross the wire as **prose segments**: the model only ever -receives plain text runs (Job.texts) plus per-segment context surrounds -(Job.contexts) and returns their translations (Result.texts, same order); -markup never leaves the server — reassembly is offset splicing -(``pagerite/segments.py``). +Three job modes (Hello.modes announces which a connection accepts; +docs/llm-translation.md): + +- ``segments`` (default) — fragments cross as prose segments; markup never + leaves the server and translations are spliced back by offset + (``pagerite/segments.py``). For text-to-text models (Seed-X). +- ``markdown`` — one fragment as full Markdown (a body chunk or a title), + with the surrounding blocks of the served hybrid as context. For + Markdown-native instruct LLMs; the result must re-chunk to exactly one + block with anchor constructs (link destinations, placeholders) intact. +- ``article`` — a whole page's Markdown at once (only while a page is + mostly pending); the result is decomposed back into per-chunk + translations (``align_article``), anchor-aligned and validated. """ import asyncio +import difflib +import itertools import logging +import re import msgspec from fastapi import WebSocket, WebSocketDisconnect from kanta import Kanta from pagerite import i18n -from pagerite.chunks import chunk_key, needs_translation -from pagerite.data import Data, Node, sorted_nodes -from pagerite.segments import Span, join, split +from pagerite.chunks import ( + chunk_key, + chunk_markdown, + join_chunks, + needs_translation, +) +from pagerite.data import Data, Node, node_markdown, resolve, sorted_nodes +from pagerite.segments import Span, join, pure_prose, split logger = logging.getLogger(__name__) +#: Job granularities a translator connection may announce (Hello.modes). +MODES = frozenset({"segments", "markdown", "article"}) + class Hello(msgspec.Struct, tag="hello"): """Client greeting on connect: the language codes its model CAN produce @@ -37,6 +56,9 @@ class Hello(msgspec.Struct, tag="hello"): the wanted target languages (``Data.translate_langs``).""" langs: list[str] + model: str = "" #: free-form model string (logging, debugging) + #: Job granularities accepted (default: segments only). + modes: list[str] = msgspec.field(default_factory=lambda: ["segments"]) class TransItem(msgspec.Struct): @@ -60,19 +82,19 @@ class Job(msgspec.Struct, tag="job"): lang: str key: bytes #: 9-byte chunk hash (base64 in the JSON frame) - #: The fragment's prose segments (pagerite/segments.py): plain text - #: runs only — no markup, URLs, code or placeholders ever cross the - #: wire. Translate each element independently. + #: segments mode: the fragment's prose segments (pagerite/segments.py) + #: — plain text runs only, no markup. markdown/article modes: a single + #: element, the fragment's resp. the whole page's Markdown. texts: list[str] path: str #: article it came from ("" = front page), no leading slash - kind: str #: "chunk" | "title" - #: Per segment (parallel to texts; "" = none): the surround to - #: translate it in — a carved-out segment (link text, partial run) - #: carries its block's plain text, a title the article's opening. - #: Reference client behavior (scripts/translator.py): translate - #: segment+context together, keep the segment's part (its own line / - #: paragraph); fall back to the segment alone when the output holds no - #: separator. Contexts are not part of the result. + kind: str #: "chunk" | "title" | "article" + #: The job granularity (the connection's mode this job was built for). + mode: str = "segments" + #: segments mode: per segment (parallel to texts; "" = none) the + #: surround to translate it in. markdown mode: for chunks the previous + #: and next block of the served hybrid (target language, patches + #: applied; "" where none), for titles the article's opening. + #: Contexts are reference only, never part of the result. contexts: list[str] = msgspec.field(default_factory=list) @@ -89,14 +111,82 @@ class Result(msgspec.Struct, tag="result"): lang: str key: bytes - #: The job's segments, translated, same order and count. Each must be - #: pure prose — the server rejects the result otherwise. + #: The job's texts, translated: same order and count for segments jobs + #: (each pure prose, or the result is rejected); a single element — + #: the translated block resp. the whole translated article — for + #: markdown/article jobs. texts: list[str] #: Union of the client -> server frames (the "type" tag selects). ClientMsg = Hello | Result +#: Constructs a translation must preserve verbatim inside a prose block: +#: link/image destinations and {...} placeholders (sorted multisets are +#: compared, so additions and drops both fail validation). +_DEST = re.compile(r"\]\(([^)\s]+)") +_BRACES = re.compile(r"\{[^{}\n]*\}") + +#: Cap for a markdown-mode context block (previous/next hybrid block). +_CONTEXT_CHARS = 1500 + + +def _marks(text: str) -> list[str]: + return sorted(_DEST.findall(text) + _BRACES.findall(text)) + + +def clean_block(source: str, translated: str, kind: str) -> str | None: + """The translated block of a markdown-mode result, or None when it + fails validation: the result must re-chunk to exactly one block with + the source's anchor constructs (destinations, placeholders) intact; + titles must stay a single prose line.""" + blocks = chunk_markdown(translated) + if len(blocks) != 1: + return None + block = blocks[0] + if kind == "title" and ("\n" in block or not pure_prose(block)): + return None + return block if _marks(source) == _marks(block) else None + + +def align_article(source: str, translated: str) -> list[tuple[bytes, str]] | None: + """Decompose a whole-article translation into (source chunk key, + translated block) pairs (also the import path for human-made + translations, scripts/import_translation.py). + + Blocks that must not change (code fences, container fences, raw HTML — + everything ``needs_translation`` rejects) anchor the alignment: they + must appear verbatim (chunk_key equality) and in order, or the whole + result is rejected. Between two anchors the regions pair positionally; + a region whose block count changed stores nothing (its chunks stay + pending and fall back to scoped jobs), as does a paired block whose + anchor constructs did not survive. + """ + src, tgt = chunk_markdown(source), chunk_markdown(translated) + tgt_keys = [chunk_key(c) for c in tgt] + locs: list[tuple[int, int]] = [] # (source index, target index) of anchors + pos = 0 + for i, chunk in enumerate(src): + if needs_translation(chunk): + continue + want = chunk_key(chunk) + while pos < len(tgt) and tgt_keys[pos] != want: + pos += 1 + if pos == len(tgt): + return None + locs.append((i, pos)) + pos += 1 + pairs: list[tuple[bytes, str]] = [] + ends = [(-1, -1), *locs, (len(src), len(tgt))] + for (s0, t0), (s1, t1) in itertools.pairwise(ends): + sregion, tregion = src[s0 + 1 : s1], tgt[t0 + 1 : t1] + if len(sregion) != len(tregion): + continue + pairs.extend( + (chunk_key(s), t) for s, t in zip(sregion, tregion) if _marks(s) == _marks(t) + ) + return pairs + def pending_items(data: Data, lang: str) -> list[TransItem]: """Fragments of the site still untranslated for ``lang``, deduped by key. @@ -195,39 +285,57 @@ def store_results(data: Data, lang: str, items: list[TransResult]) -> list[str]: class _Connection: - """One connected translator socket: the language codes it announced as - capabilities (Hello) and the (lang, chunk-key) job currently in flight - on it, with the segment spans to splice its Result into - (pagerite/segments.py) — one at a time, the next is sent only after its - Result. + """One connected translator socket: the language codes and job modes it + announced (Hello), its model string, and the job currently in flight on + it — one at a time, the next is sent only after its Result. Per-connection only: in-flight lives solely here, so on disconnect the item simply becomes pending again and is re-offered to any free capable connection.""" - def __init__(self, capable: set[str]) -> None: + def __init__(self, capable: set[str], modes: set[str], model: str) -> None: self.capable = capable + self.modes = modes + self.model = model self.inflight: tuple[str, bytes] | None = None - #: Source spans of the in-flight job's segments (splice offsets - #: and link marks). + self.mode: str = "" #: the in-flight job's mode + #: (lang, chunk key) pairs the in-flight job covers (an article job + #: covers its page's pending chunks). + self.items: set[tuple[str, bytes]] = set() + #: segments mode: source spans of the in-flight job's segments + #: (splice offsets and link marks). self.spans: list[Span] = [] - self.original: str = "" # its full source text (for the splicing) - self.kind: str = "" # "chunk" | "title" (for the transaction action) + self.original: str = "" # its full source text (splicing / alignment) + self.kind: str = "" # "chunk" | "title" | "article" + + def take(self) -> tuple[str, str, str, list[Span]]: + """Snapshot and clear the in-flight job's working state.""" + mode, kind, spans, original = self.mode, self.kind, self.spans, self.original + self.inflight = None + self.items = set() + self.mode = self.kind = "" + self.spans = [] + self.original = "" + return mode, kind, spans, original class Dispatcher: """The translator dispatcher: connected client sockets and the job - pipeline (docs/localization.md). + pipeline (docs/localization.md, docs/llm-translation.md). One single-item job at a time per connection, offered in the - intersection of the wanted languages (``Data.translate_langs``) and the - connection's announced capabilities. Pending work is derived from the + intersection of the wanted languages (``Data.translate_langs``), the + connection's announced capabilities and its accepted job modes: + ``article`` jobs (a whole page) only to article-capable connections and + only while a page is mostly pending, steady-state follow-up as scoped + ``markdown``/``segments`` jobs. Pending work is derived from the ``trans`` store (``pending_items``) minus the items in flight on any connection, so a dropped connection's in-flight item is simply - re-offered. Results are matched to content by chunk key alone. A - (lang, key) whose Result fails segment validation is skipped for the - rest of the run — generation is near-deterministic, so an immediate - retry would just re-fail. + re-offered. Results are matched to content by chunk key alone (an + article job's key is its page's first chunk). A (lang, key, mode) + whose Result fails validation is skipped for the rest of the run — + generation is near-deterministic per model, so an immediate retry in + the same mode would just re-fail, while other modes stay offerable. """ def __init__(self, data: Data, db: Kanta, invalidate) -> None: @@ -238,14 +346,13 @@ class Dispatcher: self.invalidate = invalidate #: Connected translator sockets and their per-connection state. self.clients: dict[WebSocket, _Connection] = {} - #: (lang, chunk key) of fragments whose result failed validation - #: (segment count, empty or non-prose segments, segments.py) this run. - self.validation_failures: set[tuple[str, bytes]] = set() + #: (lang, chunk key, mode) of jobs whose result failed validation + #: this run. + self.validation_failures: set[tuple[str, bytes, str]] = set() def reset_validation_failures(self) -> None: - """Clear the skip list of fragments rejected this run (segment - validation): a translations refresh is precisely the "another - chance" for them.""" + """Clear the skip list of fragments rejected this run: a + translations refresh is precisely the "another chance" for them.""" self.validation_failures.clear() def schedule(self) -> None: @@ -263,6 +370,149 @@ class Dispatcher: return asyncio.create_task(self._dispatch()) + def _scoped_job( + self, item: TransItem, lang: str, mode: str + ) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None: + """A title/chunk job for one pending item, in segments or markdown + mode: (job, spans, original, covered (lang, key) pairs).""" + if mode == "segments": + spans, texts, contexts = split(item.text) + if not texts: + return None # prose that could not be located for splicing + if item.kind == "title" and item.context: + # A title's surround is the article's opening prose + # (TransItem.context), not its own one-word block. + contexts = [item.context] * len(texts) + job = Job( + lang=lang, + key=item.key, + texts=texts, + path=item.path, + kind=item.kind, + contexts=contexts, + ) + else: # markdown: the fragment crosses whole, as Markdown + if item.kind == "title": + contexts = [item.context] if item.context else [] + else: + contexts = self._block_contexts(lang, item) + job = Job( + lang=lang, + key=item.key, + texts=[item.text], + path=item.path, + kind=item.kind, + mode="markdown", + contexts=contexts, + ) + return job, spans if mode == "segments" else [], item.text, {(lang, item.key)} + + def _block_contexts(self, lang: str, item: TransItem) -> list[str]: + """The previous and next block of the served hybrid around a pending + chunk (current machine translation with user patches applied, so + human corrections propagate into fresh translations).""" + chain = resolve(self.data.menu, item.path) + node = chain[-1] if chain else None + if node is None or not node.chunks or item.key not in node.chunks: + return [] + served = [ + self.data.trans.get(h, {}).get(lang) or self.data.chunks.get(h, "") + for h in node.chunks + ] + hybrid = join_chunks(served) + for patch in self.data.patches.get(f"{item.path}:{lang}", []): + hybrid = i18n.apply_patch(hybrid, patch) + blocks = chunk_markdown(hybrid) + i = node.chunks.index(item.key) + # Map the chunk's served-list position onto the patched block list + # (patches may merge, split or drop blocks). + j = min(i, len(blocks)) + for tag, i1, i2, j1, j2 in difflib.SequenceMatcher( + None, served, blocks, autojunk=False + ).get_opcodes(): + if i1 <= i < i2: + j = j1 + (i - i1) if tag == "equal" else j1 + break + prev = blocks[j - 1] if 0 < j <= len(blocks) else "" + next_ = blocks[j + 1] if j + 1 < len(blocks) else "" + return [prev[-_CONTEXT_CHARS:], next_[:_CONTEXT_CHARS]] + + def _article_job( + self, lang: str, items: list[TransItem], inflight: set[tuple[str, bytes]] + ) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None: + """A whole-page job for the first page that is mostly pending for + ``lang`` (a whole new article or a full refresh; steady-state edits + stay scoped jobs). The job's key is the page's first chunk.""" + by_path: dict[str, list[TransItem]] = {} + for item in items: + if item.kind == "chunk": + by_path.setdefault(item.path, []).append(item) + for path, page_items in by_path.items(): + chain = resolve(self.data.menu, path) + node = chain[-1] if chain else None + if node is None or not node.chunks: + continue + total = { + h + for h in node.chunks + if h not in node.no_trans + and (text := self.data.chunks.get(h)) is not None + and needs_translation(text) + } + pend = {item.key for item in page_items} + if ( + not pend + or len(pend) * 2 < len(total) + or any((lang, key) in inflight for key in pend) + ): + continue + key = node.chunks[0] + if (lang, key, "article") in self.validation_failures: + continue + md = node_markdown(self.data, node) or "" + job = Job( + lang=lang, key=key, texts=[md], path=path, kind="article", mode="article" + ) + return job, [], md, {(lang, k) for k in pend} + return None + + def _pick( + self, state: _Connection, langs: list[str], inflight: set[tuple[str, bytes]] + ) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None: + """The next job for a free connection: titles before articles before + chunks — across languages too, so every menu is named before any + article body is worked on (a page's name is its most visible + string). pending_items emits in menu order, a page's title before + its chunks.""" + pending = {lang: pending_items(self.data, lang) for lang in langs} + scoped = ( + "markdown" + if "markdown" in state.modes + else "segments" + if "segments" in state.modes + else "" + ) + for kind in ("title", "article", "chunk"): + for lang in langs: + if kind == "article": + if "article" in state.modes and ( + offer := self._article_job(lang, pending[lang], inflight) + ): + return offer + continue + if not scoped: + continue + for item in pending[lang]: + if ( + item.kind != kind + or (lang, item.key) in inflight + or (lang, item.key, scoped) in self.validation_failures + ): + continue + if offer := self._scoped_job(item, lang, scoped): + return offer + return None + async def _dispatch(self) -> None: """Offer one pending item to every free capable connection.""" wanted = { @@ -273,71 +523,60 @@ class Dispatcher: for ws, state in list(self.clients.items()): if state.inflight is not None: continue - langs = wanted & state.capable + langs = sorted(wanted & state.capable) if not langs: continue - inflight = {s.inflight for s in self.clients.values() if s.inflight} - job = None - spans: list[Span] = [] - original = "" - # Titles before articles — across languages too, so every menu - # is named before any article body is worked on (a page's name - # is its most visible string). pending_items emits in menu - # order, a page's title before its chunks; filtering by kind - # keeps that stable order within each kind. - pending = {lang: pending_items(self.data, lang) for lang in sorted(langs)} - for kind in ("title", "chunk"): - for lang in sorted(langs): - for item in pending[lang]: - if ( - item.kind != kind - or (lang, item.key) in inflight - or (lang, item.key) in self.validation_failures - ): - continue - spans, texts, contexts = split(item.text) - if not texts: - continue # prose that could not be located for splicing - original = item.text - if item.kind == "title" and item.context: - # A title's surround is the article's opening prose - # (TransItem.context), not its own one-word block. - contexts = [item.context] * len(texts) - job = Job( - lang=lang, - key=item.key, - texts=texts, - path=item.path, - kind=item.kind, - contexts=contexts, - ) - break - if job is not None: - break - if job is not None: - break - if job is None: + inflight = {item for s in self.clients.values() for item in s.items} + offer = self._pick(state, langs, inflight) + if offer is None: continue + job, spans, original, items = offer state.inflight = (job.lang, job.key) # before the await: no double-assign + state.mode = job.mode + state.kind = job.kind state.spans = spans state.original = original - state.kind = job.kind + state.items = items try: await ws.send_text(msgspec.json.encode(job).decode()) except Exception: # send failed: the receive loop cleans up self.clients.pop(ws, None) + def _results( + self, + mode: str, + kind: str, + key: bytes, + original: str, + spans: list[Span], + texts: list[str], + ) -> list[TransResult] | None: + """Validate a Result against its in-flight job and turn it into + storable fragments; None when it fails validation (the caller skips + the (lang, key, mode) for this run and the work stays pending).""" + if mode == "segments": + text = join(original, spans, texts) if len(texts) == len(spans) else None + return [TransResult(key=key, text=text)] if text is not None else None + if mode == "markdown": + block = clean_block(original, texts[0], kind) if len(texts) == 1 else None + return [TransResult(key=key, text=block)] if block else None + pairs = align_article(original, texts[0]) if len(texts) == 1 else None + if not pairs: + return None + return [TransResult(key=k, text=t) for k, t in pairs] + async def handle_ws(self, ws: WebSocket, clientkey: str) -> None: """The /_translate/ channel (docs/localization.md). A wrong/empty key rejects the handshake (closing before accept makes Starlette answer HTTP 403). Protocol (JSON frames): the - client opens with Hello(langs) announcing its CAPABILITIES — the - language codes its model can produce (normalized to translation - tags; "en"/empty dropped) — then answers each Job with its - Result(lang, key, texts). A Result without an in-flight job or with - a different (lang, key), a duplicate Hello, or any malformed frame - closes the socket with a protocol error. + client opens with Hello(langs, model, modes) announcing its + CAPABILITIES — the language codes its model can produce (normalized + to translation tags; "en"/empty dropped) and the job modes it + accepts — then answers each Job with its Result(lang, key, texts). + A Result without an in-flight job or with a different (lang, key), + a duplicate Hello, or any malformed frame closes the socket with a + protocol error. """ if clientkey not in self.data.translate_keys: await ws.close(code=1008) # policy violation; pre-accept = HTTP 403 @@ -357,9 +596,17 @@ class Dispatcher: await ws.close(code=1002) return state = _Connection( - {tag for lang in msg.langs if (tag := i18n.base_tag(lang))} + {tag for lang in msg.langs if (tag := i18n.base_tag(lang))}, + set(msg.modes) & MODES or {"segments"}, + msg.model, ) self.clients[ws] = state + logger.info( + "translator connected: model=%r, modes=%s, langs=%s", + state.model, + sorted(state.modes), + sorted(state.capable), + ) self.schedule() else: # Result lang = i18n.base_tag(msg.lang) @@ -370,37 +617,31 @@ class Dispatcher: ): await ws.close(code=1002) return - texts, spans, original = msg.texts, state.spans, state.original - kind, state.kind = state.kind, "" - state.inflight = None - state.spans = [] - state.original = "" - text = ( - join(original, spans, texts) - if len(texts) == len(spans) - else None + mode, kind, spans, original = state.take() + results = self._results( + mode, kind, msg.key, original, spans, msg.texts ) - if text is None: - # The model broke the segment contract (count - # mismatch, empty or non-prose segment): drop the - # result and skip the fragment for this run (it - # stays pending; a restart, a refresh or a model - # change gets another chance). - self.validation_failures.add((lang, msg.key)) + if results is None: + # The model broke the contract (bad segment count, + # markup in a segment, a merged/split block, a lost + # anchor): drop the result and skip the (lang, key, + # mode) for this run — the work stays pending and a + # restart, a refresh, another mode or a model change + # gets another chance. + self.validation_failures.add((lang, msg.key, mode)) logger.warning( - "[%s] result for chunk %s rejected: invalid segments", + "[%s] %s result for %s rejected: failed validation", lang, + mode, msg.key.hex(), ) self.schedule() continue with self.db.transaction( - f"translate:{lang}{':title' if kind == 'title' else ''}", + f"translate:{lang}{':' + kind if kind != 'chunk' else ''}", user=clientkey, ): - paths = store_results( - self.data, lang, [TransResult(key=msg.key, text=text)] - ) + paths = store_results(self.data, lang, results) self.invalidate() # schedules the next dispatch if paths: logger.info(