translate: job modes (segments/markdown/article), mode-scoped validation, article decomposition
Hello gains model + modes; Job gains mode. Dispatch routes per connection capability: article jobs only to article-capable connections and only while a page is mostly pending; titles/chunks go as markdown (whole fragment, hybrid-neighbor contexts) or segments jobs. Validation skip list is now (lang, key, mode). align_article decomposes a whole-article result into per-chunk stores: non-translatable blocks (fences, HTML, containers) are verbatim anchors, regions between anchors pair positionally, mismatched regions and blocks with altered link destinations/placeholders stay pending. Verified offline against /tmp/llmtrial outputs: qwen3.8:27b runs pair 105/105 and 43/43 blocks; qwen3:30b-instruct's translated code comments, qwen3-next's degenerate runs and the structurally broken seedx trial files (a dropped ::: fence) are all rejected.
This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
# Whole-article and scoped LLM translation
|
||||
|
||||
Design for augmenting the fragment-based machine translation
|
||||
(docs/localization.md) with general-purpose instruct LLMs that understand
|
||||
Markdown natively — as opposed to pure text-to-text models like Seed-X.
|
||||
|
||||
## Motivation
|
||||
|
||||
The chunk + segment pipeline (`chunks.py` → `segments.py` → Seed-X) exists
|
||||
because Seed-X mangles Markdown: links, formatting, fences and placeholders
|
||||
must be stripped before dispatch and re-inserted into the result. The
|
||||
re-insertion of link and formatting markup is the imprecise part: when
|
||||
word-alignment by form similarity finds no anchor (always for CJK targets),
|
||||
positions fall back to word-weight ratios, which land a word or so off.
|
||||
All of `segments.py` — segmentation, offset splicing, `_find_mark`,
|
||||
weight-ratio fallback, `_NEUTRAL` punctuation swaps, `<` encoding — is
|
||||
defensive scaffolding around that one limitation.
|
||||
|
||||
An instruct LLM translates Markdown natively: `[text](url)` stays intact
|
||||
and moves as a unit, fences, container markers, attrs and `{...}`
|
||||
placeholders are preserved, and link texts translate in sentence context.
|
||||
For such a translator the entire segments layer is unnecessary.
|
||||
|
||||
## Trial evidence (2026-09, RTX 4090 24 GB + 128 GB RAM, ollama 0.34)
|
||||
|
||||
Whole-article translation of two real articles (5.5 KB marketing, 18.3 KB
|
||||
technical with code fences and `{dates}`) into fi/es/zh:
|
||||
|
||||
- **qwen3.8:27b** (dense, 17 GB Q4 — fits VRAM): structure-perfect on all
|
||||
runs — URLs, placeholders, heading/block counts preserved, fenced code
|
||||
byte-identical. es/zh excellent; fi fluent with occasional lexical slips
|
||||
(covered by the human patch layer). ~30 s per short article, ~2.5 min
|
||||
for 18 KB. **The reference model for article and markdown modes.**
|
||||
- **qwen3:30b-instruct**: 3× faster, good prose, but rewrote comments and
|
||||
docstrings inside code fences despite explicit instructions — fails
|
||||
anchor validation (see below).
|
||||
- **qwen3-next:80b** (MoE): best Finnish word choice on short documents,
|
||||
but degenerates on longer input in every configuration tried — runaway
|
||||
thinking loops (293k tokens), empty responses, 3× length output with
|
||||
hallucinated URLs, and ~10× blowup even in 2 KB scoped chunks. Unusable
|
||||
on current ollama builds.
|
||||
- **CPU-only** (i7-14700, 14 threads): MoE 3B-active 10.6 t/s generation
|
||||
(viable for batch), dense 27B 2.5 t/s (not viable). Hybrid GPU+CPU
|
||||
splits bottleneck prompt evaluation (~52 t/s vs 62 t/s pure CPU) —
|
||||
dense GPU-resident or MoE CPU-resident are the sane configurations;
|
||||
mixing hurts.
|
||||
|
||||
Operational requirements established by the trials (all client-side):
|
||||
|
||||
- **Always disable thinking** for hybrid models (`think: false` on
|
||||
ollama): the reasoning phase adds minutes per article and can loop
|
||||
unbounded.
|
||||
- **Always cap generation** (`num_predict` ≈ 2–3× input tokens): a
|
||||
runaway on a whole-article job burns hours, vs. seconds for a
|
||||
Seed-X segment.
|
||||
- temperature 0.2 with the strict structure prompt works well for
|
||||
qwen3.8.
|
||||
|
||||
## What carries over unchanged
|
||||
|
||||
The valuable parts of the current design are the **storage and staleness
|
||||
model**, not the segmentation — and none of them require the machine
|
||||
translation to be produced chunk by chunk. The chunk store is a
|
||||
storage/diffing format; LLM output at any granularity is *projected
|
||||
into* it:
|
||||
|
||||
- Content-addressed source chunks (`Data.chunks`, `chunk_key`) — staleness
|
||||
still falls out of source-hash keys: editing the original invalidates
|
||||
exactly the edited chunks, all other translations keep applying.
|
||||
- Per-chunk machine translations (`Data.trans[hash][lang]`) and the hybrid
|
||||
render with per-chunk fallback to the original.
|
||||
- User patches (`Data.patches`) — search/replace hunks over the assembled
|
||||
hybrid, per-hunk independent and best-effort. Patches are orthogonal to
|
||||
how `Data.trans` entries were produced.
|
||||
- `pending_items`: after a source edit, exactly the changed (lang, hash)
|
||||
pairs are pending — **focused retranslation of edits falls out of the
|
||||
existing bookkeeping**, no whole-article reruns.
|
||||
|
||||
## Protocol: capabilities and job modes
|
||||
|
||||
The `/_translate/{key}` WebSocket stays the single channel; Seed-X
|
||||
clients work unchanged. The client→server `Hello` gains two optional
|
||||
fields:
|
||||
|
||||
```python
|
||||
class Hello(msgspec.Struct, tag="hello"):
|
||||
langs: list[str] # as today: languages the model can produce
|
||||
model: str = "" # free-form model string (logging, debugging)
|
||||
modes: list[str] = ["segments"] # job granularities accepted
|
||||
```
|
||||
|
||||
Three job modes, in increasing granularity:
|
||||
|
||||
- **`segments`** — the current protocol, unchanged: `Job.texts` carries
|
||||
prose segments (markup never crosses the wire), `Result.texts` returns
|
||||
them, the server splices by offset (`segments.py`). For text-to-text
|
||||
models (Seed-X). Default when a client omits `modes`.
|
||||
- **`markdown`** (scoped instruct mode) — one fragment as full Markdown:
|
||||
a body chunk or a title. `Job.texts` carries a single element, the
|
||||
chunk's Markdown; `Job.contexts` carries up to two context strings
|
||||
(previous and next block of the **served hybrid** in the target
|
||||
language — current machine translation with user patches applied),
|
||||
"" where none. The client is instructed to output ONLY the translation
|
||||
of the target block; the context is terminology/tone reference.
|
||||
Using the *patched* hybrid as context propagates human corrections
|
||||
into fresh machine translations without the LLM ever touching patch
|
||||
storage. `Result.texts` carries one element, the translated block.
|
||||
The server validates: exactly one block after re-chunking, anchor
|
||||
constructs (URLs, image destinations, code fence content, `{...}`
|
||||
placeholders) preserved where the source block has them, then stores
|
||||
to `Data.trans` as usual.
|
||||
- **`article`** — a whole page. `Job.texts` carries one element, the full
|
||||
original Markdown (the chunk sequence is recoverable server-side via
|
||||
`node.chunks`); `Result.texts` carries one element, the full translated
|
||||
Markdown. The server decomposes (below) and stores per chunk.
|
||||
|
||||
Titles are jobs like any other in all modes (`kind="title"` keeps its
|
||||
article-opening context rule; in `markdown` mode a title crosses as
|
||||
plain text, since it carries no markup by construction).
|
||||
|
||||
### Dispatch and validation
|
||||
|
||||
- Routing is per connection as today (wanted ∩ capable, one job in
|
||||
flight, requeue on disconnect), extended by mode: the smallest
|
||||
suitable unit goes to each free connection — `article` jobs only to
|
||||
article-capable connections, and only while a page is *mostly*
|
||||
pending (a whole new article or a full refresh); steady-state edit
|
||||
follow-up is `markdown`/`segments` jobs. Mixed translator fleets (a
|
||||
Seed-X instance, a local qwen, an API-backed client) run concurrently
|
||||
and share the work by capability.
|
||||
- The validation skip-list becomes **mode-scoped** (`(lang, key, mode)`):
|
||||
a fragment a Seed-X client rejects stays offerable to instruct clients
|
||||
(and vice versa) — near-deterministic re-failure applies per model,
|
||||
not across approaches.
|
||||
- `Result` matching is unchanged (lang, key); article results match on
|
||||
the key of the article's first chunk.
|
||||
|
||||
## Article result decomposition
|
||||
|
||||
1. Re-chunk the translated article with the same `chunk_markdown`.
|
||||
2. Align translated blocks to source blocks. A well-behaved model does
|
||||
not reorder paragraphs, so positional / `SequenceMatcher` alignment
|
||||
at block granularity suffices. Blocks that must not change — code
|
||||
fences, container fence lines, `{...}` placeholders, image
|
||||
destinations, raw HTML — are matched verbatim and serve as alignment
|
||||
anchors, like diff context lines.
|
||||
3. Store each translated block in `Data.trans[source_chunk_hash][lang]`.
|
||||
|
||||
Validation happens *before* anything is stored, same spirit as the
|
||||
`pure_prose` segment checks but structural:
|
||||
|
||||
- Anchor blocks must appear verbatim and in order (this is what rejects
|
||||
qwen3:30b-instruct's translated code comments automatically).
|
||||
- Per anchor-bounded region, source and translated block counts must
|
||||
match 1:1; regions that don't align store nothing and their chunks
|
||||
stay pending (they fall back to `markdown`-mode scoped jobs).
|
||||
|
||||
## Reference client
|
||||
|
||||
A second client script next to `scripts/translator.py` speaking the
|
||||
`markdown` and `article` modes. Internally it targets the **OpenAI
|
||||
Chat Completions API shape** (`POST /v1/chat/completions`): ollama
|
||||
serves it at `:11434/v1`, llama.cpp's server likewise, and hosted APIs
|
||||
(OpenAI and compatible providers) natively — `base_url` + `model` +
|
||||
optional API key in the client's config selects local GPU, local CPU or
|
||||
a remote model, with backend quirks (ollama's `think: false`,
|
||||
`num_predict` cap, per-model sampling) in a per-model config section.
|
||||
How the client drives its LLM is its internal matter; the wire protocol
|
||||
above is the contract.
|
||||
|
||||
The client announces in `Hello`:
|
||||
|
||||
- `model`: the model string it is actually serving (e.g. `qwen3.8:27b`)
|
||||
- `langs`: from its per-model language table — for the shipped qwen3.8
|
||||
configuration the site languages as configured server-side
|
||||
(de, es, fi, pt, zh; Finnish flagged as the weakest, patch-covered)
|
||||
- `modes`: `["markdown", "article"]` for a structure-proven model,
|
||||
`["markdown"]` for one that is only trusted in scoped mode
|
||||
|
||||
The Seed-X client is untouched and announces `["segments"]` (implicitly,
|
||||
by omitting `modes`).
|
||||
|
||||
## Importing human-made full translations
|
||||
|
||||
The decomposition function doubles as an import path for translations
|
||||
produced outside the pipeline — e.g. an article translated with ChatGPT
|
||||
and pasted back. Today such a paste lands in the translation editor and
|
||||
is stored as one giant user patch; feeding it through the same
|
||||
decomposition instead writes proper `Data.trans` fragments, so later
|
||||
source edits invalidate and re-translate per chunk rather than letting
|
||||
the monolithic patch silently go stale hunk by hunk. This import path is
|
||||
also the natural testbed for the decomposition and validation logic
|
||||
before any live LLM client uses it.
|
||||
+357
-116
@@ -7,29 +7,48 @@ as base64 — no manual encoding anywhere). This module holds everything
|
||||
else: the message structs, the connected-client dispatcher (``Dispatcher``
|
||||
— one job at a time per connection, wanted ∩ capable language matching,
|
||||
requeue on disconnect), which fragments are pending for a language
|
||||
(``pending_items``) and storing a result (``store_results``).
|
||||
(``pending_items``) and storing results (``store_results``).
|
||||
|
||||
Fragments cross the wire as **prose segments**: the model only ever
|
||||
receives plain text runs (Job.texts) plus per-segment context surrounds
|
||||
(Job.contexts) and returns their translations (Result.texts, same order);
|
||||
markup never leaves the server — reassembly is offset splicing
|
||||
(``pagerite/segments.py``).
|
||||
Three job modes (Hello.modes announces which a connection accepts;
|
||||
docs/llm-translation.md):
|
||||
|
||||
- ``segments`` (default) — fragments cross as prose segments; markup never
|
||||
leaves the server and translations are spliced back by offset
|
||||
(``pagerite/segments.py``). For text-to-text models (Seed-X).
|
||||
- ``markdown`` — one fragment as full Markdown (a body chunk or a title),
|
||||
with the surrounding blocks of the served hybrid as context. For
|
||||
Markdown-native instruct LLMs; the result must re-chunk to exactly one
|
||||
block with anchor constructs (link destinations, placeholders) intact.
|
||||
- ``article`` — a whole page's Markdown at once (only while a page is
|
||||
mostly pending); the result is decomposed back into per-chunk
|
||||
translations (``align_article``), anchor-aligned and validated.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import difflib
|
||||
import itertools
|
||||
import logging
|
||||
import re
|
||||
|
||||
import msgspec
|
||||
from fastapi import WebSocket, WebSocketDisconnect
|
||||
from kanta import Kanta
|
||||
|
||||
from pagerite import i18n
|
||||
from pagerite.chunks import chunk_key, needs_translation
|
||||
from pagerite.data import Data, Node, sorted_nodes
|
||||
from pagerite.segments import Span, join, split
|
||||
from pagerite.chunks import (
|
||||
chunk_key,
|
||||
chunk_markdown,
|
||||
join_chunks,
|
||||
needs_translation,
|
||||
)
|
||||
from pagerite.data import Data, Node, node_markdown, resolve, sorted_nodes
|
||||
from pagerite.segments import Span, join, pure_prose, split
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Job granularities a translator connection may announce (Hello.modes).
|
||||
MODES = frozenset({"segments", "markdown", "article"})
|
||||
|
||||
|
||||
class Hello(msgspec.Struct, tag="hello"):
|
||||
"""Client greeting on connect: the language codes its model CAN produce
|
||||
@@ -37,6 +56,9 @@ class Hello(msgspec.Struct, tag="hello"):
|
||||
the wanted target languages (``Data.translate_langs``)."""
|
||||
|
||||
langs: list[str]
|
||||
model: str = "" #: free-form model string (logging, debugging)
|
||||
#: Job granularities accepted (default: segments only).
|
||||
modes: list[str] = msgspec.field(default_factory=lambda: ["segments"])
|
||||
|
||||
|
||||
class TransItem(msgspec.Struct):
|
||||
@@ -60,19 +82,19 @@ class Job(msgspec.Struct, tag="job"):
|
||||
|
||||
lang: str
|
||||
key: bytes #: 9-byte chunk hash (base64 in the JSON frame)
|
||||
#: The fragment's prose segments (pagerite/segments.py): plain text
|
||||
#: runs only — no markup, URLs, code or placeholders ever cross the
|
||||
#: wire. Translate each element independently.
|
||||
#: segments mode: the fragment's prose segments (pagerite/segments.py)
|
||||
#: — plain text runs only, no markup. markdown/article modes: a single
|
||||
#: element, the fragment's resp. the whole page's Markdown.
|
||||
texts: list[str]
|
||||
path: str #: article it came from ("" = front page), no leading slash
|
||||
kind: str #: "chunk" | "title"
|
||||
#: Per segment (parallel to texts; "" = none): the surround to
|
||||
#: translate it in — a carved-out segment (link text, partial run)
|
||||
#: carries its block's plain text, a title the article's opening.
|
||||
#: Reference client behavior (scripts/translator.py): translate
|
||||
#: segment+context together, keep the segment's part (its own line /
|
||||
#: paragraph); fall back to the segment alone when the output holds no
|
||||
#: separator. Contexts are not part of the result.
|
||||
kind: str #: "chunk" | "title" | "article"
|
||||
#: The job granularity (the connection's mode this job was built for).
|
||||
mode: str = "segments"
|
||||
#: segments mode: per segment (parallel to texts; "" = none) the
|
||||
#: surround to translate it in. markdown mode: for chunks the previous
|
||||
#: and next block of the served hybrid (target language, patches
|
||||
#: applied; "" where none), for titles the article's opening.
|
||||
#: Contexts are reference only, never part of the result.
|
||||
contexts: list[str] = msgspec.field(default_factory=list)
|
||||
|
||||
|
||||
@@ -89,14 +111,82 @@ class Result(msgspec.Struct, tag="result"):
|
||||
|
||||
lang: str
|
||||
key: bytes
|
||||
#: The job's segments, translated, same order and count. Each must be
|
||||
#: pure prose — the server rejects the result otherwise.
|
||||
#: The job's texts, translated: same order and count for segments jobs
|
||||
#: (each pure prose, or the result is rejected); a single element —
|
||||
#: the translated block resp. the whole translated article — for
|
||||
#: markdown/article jobs.
|
||||
texts: list[str]
|
||||
|
||||
|
||||
#: Union of the client -> server frames (the "type" tag selects).
|
||||
ClientMsg = Hello | Result
|
||||
|
||||
#: Constructs a translation must preserve verbatim inside a prose block:
|
||||
#: link/image destinations and {...} placeholders (sorted multisets are
|
||||
#: compared, so additions and drops both fail validation).
|
||||
_DEST = re.compile(r"\]\(([^)\s]+)")
|
||||
_BRACES = re.compile(r"\{[^{}\n]*\}")
|
||||
|
||||
#: Cap for a markdown-mode context block (previous/next hybrid block).
|
||||
_CONTEXT_CHARS = 1500
|
||||
|
||||
|
||||
def _marks(text: str) -> list[str]:
|
||||
return sorted(_DEST.findall(text) + _BRACES.findall(text))
|
||||
|
||||
|
||||
def clean_block(source: str, translated: str, kind: str) -> str | None:
|
||||
"""The translated block of a markdown-mode result, or None when it
|
||||
fails validation: the result must re-chunk to exactly one block with
|
||||
the source's anchor constructs (destinations, placeholders) intact;
|
||||
titles must stay a single prose line."""
|
||||
blocks = chunk_markdown(translated)
|
||||
if len(blocks) != 1:
|
||||
return None
|
||||
block = blocks[0]
|
||||
if kind == "title" and ("\n" in block or not pure_prose(block)):
|
||||
return None
|
||||
return block if _marks(source) == _marks(block) else None
|
||||
|
||||
|
||||
def align_article(source: str, translated: str) -> list[tuple[bytes, str]] | None:
|
||||
"""Decompose a whole-article translation into (source chunk key,
|
||||
translated block) pairs (also the import path for human-made
|
||||
translations, scripts/import_translation.py).
|
||||
|
||||
Blocks that must not change (code fences, container fences, raw HTML —
|
||||
everything ``needs_translation`` rejects) anchor the alignment: they
|
||||
must appear verbatim (chunk_key equality) and in order, or the whole
|
||||
result is rejected. Between two anchors the regions pair positionally;
|
||||
a region whose block count changed stores nothing (its chunks stay
|
||||
pending and fall back to scoped jobs), as does a paired block whose
|
||||
anchor constructs did not survive.
|
||||
"""
|
||||
src, tgt = chunk_markdown(source), chunk_markdown(translated)
|
||||
tgt_keys = [chunk_key(c) for c in tgt]
|
||||
locs: list[tuple[int, int]] = [] # (source index, target index) of anchors
|
||||
pos = 0
|
||||
for i, chunk in enumerate(src):
|
||||
if needs_translation(chunk):
|
||||
continue
|
||||
want = chunk_key(chunk)
|
||||
while pos < len(tgt) and tgt_keys[pos] != want:
|
||||
pos += 1
|
||||
if pos == len(tgt):
|
||||
return None
|
||||
locs.append((i, pos))
|
||||
pos += 1
|
||||
pairs: list[tuple[bytes, str]] = []
|
||||
ends = [(-1, -1), *locs, (len(src), len(tgt))]
|
||||
for (s0, t0), (s1, t1) in itertools.pairwise(ends):
|
||||
sregion, tregion = src[s0 + 1 : s1], tgt[t0 + 1 : t1]
|
||||
if len(sregion) != len(tregion):
|
||||
continue
|
||||
pairs.extend(
|
||||
(chunk_key(s), t) for s, t in zip(sregion, tregion) if _marks(s) == _marks(t)
|
||||
)
|
||||
return pairs
|
||||
|
||||
|
||||
def pending_items(data: Data, lang: str) -> list[TransItem]:
|
||||
"""Fragments of the site still untranslated for ``lang``, deduped by key.
|
||||
@@ -195,39 +285,57 @@ def store_results(data: Data, lang: str, items: list[TransResult]) -> list[str]:
|
||||
|
||||
|
||||
class _Connection:
|
||||
"""One connected translator socket: the language codes it announced as
|
||||
capabilities (Hello) and the (lang, chunk-key) job currently in flight
|
||||
on it, with the segment spans to splice its Result into
|
||||
(pagerite/segments.py) — one at a time, the next is sent only after its
|
||||
Result.
|
||||
"""One connected translator socket: the language codes and job modes it
|
||||
announced (Hello), its model string, and the job currently in flight on
|
||||
it — one at a time, the next is sent only after its Result.
|
||||
|
||||
Per-connection only: in-flight lives solely here, so on disconnect the
|
||||
item simply becomes pending again and is re-offered to any free capable
|
||||
connection."""
|
||||
|
||||
def __init__(self, capable: set[str]) -> None:
|
||||
def __init__(self, capable: set[str], modes: set[str], model: str) -> None:
|
||||
self.capable = capable
|
||||
self.modes = modes
|
||||
self.model = model
|
||||
self.inflight: tuple[str, bytes] | None = None
|
||||
#: Source spans of the in-flight job's segments (splice offsets
|
||||
#: and link marks).
|
||||
self.mode: str = "" #: the in-flight job's mode
|
||||
#: (lang, chunk key) pairs the in-flight job covers (an article job
|
||||
#: covers its page's pending chunks).
|
||||
self.items: set[tuple[str, bytes]] = set()
|
||||
#: segments mode: source spans of the in-flight job's segments
|
||||
#: (splice offsets and link marks).
|
||||
self.spans: list[Span] = []
|
||||
self.original: str = "" # its full source text (for the splicing)
|
||||
self.kind: str = "" # "chunk" | "title" (for the transaction action)
|
||||
self.original: str = "" # its full source text (splicing / alignment)
|
||||
self.kind: str = "" # "chunk" | "title" | "article"
|
||||
|
||||
def take(self) -> tuple[str, str, str, list[Span]]:
|
||||
"""Snapshot and clear the in-flight job's working state."""
|
||||
mode, kind, spans, original = self.mode, self.kind, self.spans, self.original
|
||||
self.inflight = None
|
||||
self.items = set()
|
||||
self.mode = self.kind = ""
|
||||
self.spans = []
|
||||
self.original = ""
|
||||
return mode, kind, spans, original
|
||||
|
||||
|
||||
class Dispatcher:
|
||||
"""The translator dispatcher: connected client sockets and the job
|
||||
pipeline (docs/localization.md).
|
||||
pipeline (docs/localization.md, docs/llm-translation.md).
|
||||
|
||||
One single-item job at a time per connection, offered in the
|
||||
intersection of the wanted languages (``Data.translate_langs``) and the
|
||||
connection's announced capabilities. Pending work is derived from the
|
||||
intersection of the wanted languages (``Data.translate_langs``), the
|
||||
connection's announced capabilities and its accepted job modes:
|
||||
``article`` jobs (a whole page) only to article-capable connections and
|
||||
only while a page is mostly pending, steady-state follow-up as scoped
|
||||
``markdown``/``segments`` jobs. Pending work is derived from the
|
||||
``trans`` store (``pending_items``) minus the items in flight on any
|
||||
connection, so a dropped connection's in-flight item is simply
|
||||
re-offered. Results are matched to content by chunk key alone. A
|
||||
(lang, key) whose Result fails segment validation is skipped for the
|
||||
rest of the run — generation is near-deterministic, so an immediate
|
||||
retry would just re-fail.
|
||||
re-offered. Results are matched to content by chunk key alone (an
|
||||
article job's key is its page's first chunk). A (lang, key, mode)
|
||||
whose Result fails validation is skipped for the rest of the run —
|
||||
generation is near-deterministic per model, so an immediate retry in
|
||||
the same mode would just re-fail, while other modes stay offerable.
|
||||
"""
|
||||
|
||||
def __init__(self, data: Data, db: Kanta, invalidate) -> None:
|
||||
@@ -238,14 +346,13 @@ class Dispatcher:
|
||||
self.invalidate = invalidate
|
||||
#: Connected translator sockets and their per-connection state.
|
||||
self.clients: dict[WebSocket, _Connection] = {}
|
||||
#: (lang, chunk key) of fragments whose result failed validation
|
||||
#: (segment count, empty or non-prose segments, segments.py) this run.
|
||||
self.validation_failures: set[tuple[str, bytes]] = set()
|
||||
#: (lang, chunk key, mode) of jobs whose result failed validation
|
||||
#: this run.
|
||||
self.validation_failures: set[tuple[str, bytes, str]] = set()
|
||||
|
||||
def reset_validation_failures(self) -> None:
|
||||
"""Clear the skip list of fragments rejected this run (segment
|
||||
validation): a translations refresh is precisely the "another
|
||||
chance" for them."""
|
||||
"""Clear the skip list of fragments rejected this run: a
|
||||
translations refresh is precisely the "another chance" for them."""
|
||||
self.validation_failures.clear()
|
||||
|
||||
def schedule(self) -> None:
|
||||
@@ -263,6 +370,149 @@ class Dispatcher:
|
||||
return
|
||||
asyncio.create_task(self._dispatch())
|
||||
|
||||
def _scoped_job(
|
||||
self, item: TransItem, lang: str, mode: str
|
||||
) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None:
|
||||
"""A title/chunk job for one pending item, in segments or markdown
|
||||
mode: (job, spans, original, covered (lang, key) pairs)."""
|
||||
if mode == "segments":
|
||||
spans, texts, contexts = split(item.text)
|
||||
if not texts:
|
||||
return None # prose that could not be located for splicing
|
||||
if item.kind == "title" and item.context:
|
||||
# A title's surround is the article's opening prose
|
||||
# (TransItem.context), not its own one-word block.
|
||||
contexts = [item.context] * len(texts)
|
||||
job = Job(
|
||||
lang=lang,
|
||||
key=item.key,
|
||||
texts=texts,
|
||||
path=item.path,
|
||||
kind=item.kind,
|
||||
contexts=contexts,
|
||||
)
|
||||
else: # markdown: the fragment crosses whole, as Markdown
|
||||
if item.kind == "title":
|
||||
contexts = [item.context] if item.context else []
|
||||
else:
|
||||
contexts = self._block_contexts(lang, item)
|
||||
job = Job(
|
||||
lang=lang,
|
||||
key=item.key,
|
||||
texts=[item.text],
|
||||
path=item.path,
|
||||
kind=item.kind,
|
||||
mode="markdown",
|
||||
contexts=contexts,
|
||||
)
|
||||
return job, spans if mode == "segments" else [], item.text, {(lang, item.key)}
|
||||
|
||||
def _block_contexts(self, lang: str, item: TransItem) -> list[str]:
|
||||
"""The previous and next block of the served hybrid around a pending
|
||||
chunk (current machine translation with user patches applied, so
|
||||
human corrections propagate into fresh translations)."""
|
||||
chain = resolve(self.data.menu, item.path)
|
||||
node = chain[-1] if chain else None
|
||||
if node is None or not node.chunks or item.key not in node.chunks:
|
||||
return []
|
||||
served = [
|
||||
self.data.trans.get(h, {}).get(lang) or self.data.chunks.get(h, "")
|
||||
for h in node.chunks
|
||||
]
|
||||
hybrid = join_chunks(served)
|
||||
for patch in self.data.patches.get(f"{item.path}:{lang}", []):
|
||||
hybrid = i18n.apply_patch(hybrid, patch)
|
||||
blocks = chunk_markdown(hybrid)
|
||||
i = node.chunks.index(item.key)
|
||||
# Map the chunk's served-list position onto the patched block list
|
||||
# (patches may merge, split or drop blocks).
|
||||
j = min(i, len(blocks))
|
||||
for tag, i1, i2, j1, j2 in difflib.SequenceMatcher(
|
||||
None, served, blocks, autojunk=False
|
||||
).get_opcodes():
|
||||
if i1 <= i < i2:
|
||||
j = j1 + (i - i1) if tag == "equal" else j1
|
||||
break
|
||||
prev = blocks[j - 1] if 0 < j <= len(blocks) else ""
|
||||
next_ = blocks[j + 1] if j + 1 < len(blocks) else ""
|
||||
return [prev[-_CONTEXT_CHARS:], next_[:_CONTEXT_CHARS]]
|
||||
|
||||
def _article_job(
|
||||
self, lang: str, items: list[TransItem], inflight: set[tuple[str, bytes]]
|
||||
) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None:
|
||||
"""A whole-page job for the first page that is mostly pending for
|
||||
``lang`` (a whole new article or a full refresh; steady-state edits
|
||||
stay scoped jobs). The job's key is the page's first chunk."""
|
||||
by_path: dict[str, list[TransItem]] = {}
|
||||
for item in items:
|
||||
if item.kind == "chunk":
|
||||
by_path.setdefault(item.path, []).append(item)
|
||||
for path, page_items in by_path.items():
|
||||
chain = resolve(self.data.menu, path)
|
||||
node = chain[-1] if chain else None
|
||||
if node is None or not node.chunks:
|
||||
continue
|
||||
total = {
|
||||
h
|
||||
for h in node.chunks
|
||||
if h not in node.no_trans
|
||||
and (text := self.data.chunks.get(h)) is not None
|
||||
and needs_translation(text)
|
||||
}
|
||||
pend = {item.key for item in page_items}
|
||||
if (
|
||||
not pend
|
||||
or len(pend) * 2 < len(total)
|
||||
or any((lang, key) in inflight for key in pend)
|
||||
):
|
||||
continue
|
||||
key = node.chunks[0]
|
||||
if (lang, key, "article") in self.validation_failures:
|
||||
continue
|
||||
md = node_markdown(self.data, node) or ""
|
||||
job = Job(
|
||||
lang=lang, key=key, texts=[md], path=path, kind="article", mode="article"
|
||||
)
|
||||
return job, [], md, {(lang, k) for k in pend}
|
||||
return None
|
||||
|
||||
def _pick(
|
||||
self, state: _Connection, langs: list[str], inflight: set[tuple[str, bytes]]
|
||||
) -> tuple[Job, list[Span], str, set[tuple[str, bytes]]] | None:
|
||||
"""The next job for a free connection: titles before articles before
|
||||
chunks — across languages too, so every menu is named before any
|
||||
article body is worked on (a page's name is its most visible
|
||||
string). pending_items emits in menu order, a page's title before
|
||||
its chunks."""
|
||||
pending = {lang: pending_items(self.data, lang) for lang in langs}
|
||||
scoped = (
|
||||
"markdown"
|
||||
if "markdown" in state.modes
|
||||
else "segments"
|
||||
if "segments" in state.modes
|
||||
else ""
|
||||
)
|
||||
for kind in ("title", "article", "chunk"):
|
||||
for lang in langs:
|
||||
if kind == "article":
|
||||
if "article" in state.modes and (
|
||||
offer := self._article_job(lang, pending[lang], inflight)
|
||||
):
|
||||
return offer
|
||||
continue
|
||||
if not scoped:
|
||||
continue
|
||||
for item in pending[lang]:
|
||||
if (
|
||||
item.kind != kind
|
||||
or (lang, item.key) in inflight
|
||||
or (lang, item.key, scoped) in self.validation_failures
|
||||
):
|
||||
continue
|
||||
if offer := self._scoped_job(item, lang, scoped):
|
||||
return offer
|
||||
return None
|
||||
|
||||
async def _dispatch(self) -> None:
|
||||
"""Offer one pending item to every free capable connection."""
|
||||
wanted = {
|
||||
@@ -273,71 +523,60 @@ class Dispatcher:
|
||||
for ws, state in list(self.clients.items()):
|
||||
if state.inflight is not None:
|
||||
continue
|
||||
langs = wanted & state.capable
|
||||
langs = sorted(wanted & state.capable)
|
||||
if not langs:
|
||||
continue
|
||||
inflight = {s.inflight for s in self.clients.values() if s.inflight}
|
||||
job = None
|
||||
spans: list[Span] = []
|
||||
original = ""
|
||||
# Titles before articles — across languages too, so every menu
|
||||
# is named before any article body is worked on (a page's name
|
||||
# is its most visible string). pending_items emits in menu
|
||||
# order, a page's title before its chunks; filtering by kind
|
||||
# keeps that stable order within each kind.
|
||||
pending = {lang: pending_items(self.data, lang) for lang in sorted(langs)}
|
||||
for kind in ("title", "chunk"):
|
||||
for lang in sorted(langs):
|
||||
for item in pending[lang]:
|
||||
if (
|
||||
item.kind != kind
|
||||
or (lang, item.key) in inflight
|
||||
or (lang, item.key) in self.validation_failures
|
||||
):
|
||||
continue
|
||||
spans, texts, contexts = split(item.text)
|
||||
if not texts:
|
||||
continue # prose that could not be located for splicing
|
||||
original = item.text
|
||||
if item.kind == "title" and item.context:
|
||||
# A title's surround is the article's opening prose
|
||||
# (TransItem.context), not its own one-word block.
|
||||
contexts = [item.context] * len(texts)
|
||||
job = Job(
|
||||
lang=lang,
|
||||
key=item.key,
|
||||
texts=texts,
|
||||
path=item.path,
|
||||
kind=item.kind,
|
||||
contexts=contexts,
|
||||
)
|
||||
break
|
||||
if job is not None:
|
||||
break
|
||||
if job is not None:
|
||||
break
|
||||
if job is None:
|
||||
inflight = {item for s in self.clients.values() for item in s.items}
|
||||
offer = self._pick(state, langs, inflight)
|
||||
if offer is None:
|
||||
continue
|
||||
job, spans, original, items = offer
|
||||
state.inflight = (job.lang, job.key) # before the await: no double-assign
|
||||
state.mode = job.mode
|
||||
state.kind = job.kind
|
||||
state.spans = spans
|
||||
state.original = original
|
||||
state.kind = job.kind
|
||||
state.items = items
|
||||
try:
|
||||
await ws.send_text(msgspec.json.encode(job).decode())
|
||||
except Exception: # send failed: the receive loop cleans up
|
||||
self.clients.pop(ws, None)
|
||||
|
||||
def _results(
|
||||
self,
|
||||
mode: str,
|
||||
kind: str,
|
||||
key: bytes,
|
||||
original: str,
|
||||
spans: list[Span],
|
||||
texts: list[str],
|
||||
) -> list[TransResult] | None:
|
||||
"""Validate a Result against its in-flight job and turn it into
|
||||
storable fragments; None when it fails validation (the caller skips
|
||||
the (lang, key, mode) for this run and the work stays pending)."""
|
||||
if mode == "segments":
|
||||
text = join(original, spans, texts) if len(texts) == len(spans) else None
|
||||
return [TransResult(key=key, text=text)] if text is not None else None
|
||||
if mode == "markdown":
|
||||
block = clean_block(original, texts[0], kind) if len(texts) == 1 else None
|
||||
return [TransResult(key=key, text=block)] if block else None
|
||||
pairs = align_article(original, texts[0]) if len(texts) == 1 else None
|
||||
if not pairs:
|
||||
return None
|
||||
return [TransResult(key=k, text=t) for k, t in pairs]
|
||||
|
||||
async def handle_ws(self, ws: WebSocket, clientkey: str) -> None:
|
||||
"""The /_translate/<key> channel (docs/localization.md).
|
||||
|
||||
A wrong/empty key rejects the handshake (closing before accept
|
||||
makes Starlette answer HTTP 403). Protocol (JSON frames): the
|
||||
client opens with Hello(langs) announcing its CAPABILITIES — the
|
||||
language codes its model can produce (normalized to translation
|
||||
tags; "en"/empty dropped) — then answers each Job with its
|
||||
Result(lang, key, texts). A Result without an in-flight job or with
|
||||
a different (lang, key), a duplicate Hello, or any malformed frame
|
||||
closes the socket with a protocol error.
|
||||
client opens with Hello(langs, model, modes) announcing its
|
||||
CAPABILITIES — the language codes its model can produce (normalized
|
||||
to translation tags; "en"/empty dropped) and the job modes it
|
||||
accepts — then answers each Job with its Result(lang, key, texts).
|
||||
A Result without an in-flight job or with a different (lang, key),
|
||||
a duplicate Hello, or any malformed frame closes the socket with a
|
||||
protocol error.
|
||||
"""
|
||||
if clientkey not in self.data.translate_keys:
|
||||
await ws.close(code=1008) # policy violation; pre-accept = HTTP 403
|
||||
@@ -357,9 +596,17 @@ class Dispatcher:
|
||||
await ws.close(code=1002)
|
||||
return
|
||||
state = _Connection(
|
||||
{tag for lang in msg.langs if (tag := i18n.base_tag(lang))}
|
||||
{tag for lang in msg.langs if (tag := i18n.base_tag(lang))},
|
||||
set(msg.modes) & MODES or {"segments"},
|
||||
msg.model,
|
||||
)
|
||||
self.clients[ws] = state
|
||||
logger.info(
|
||||
"translator connected: model=%r, modes=%s, langs=%s",
|
||||
state.model,
|
||||
sorted(state.modes),
|
||||
sorted(state.capable),
|
||||
)
|
||||
self.schedule()
|
||||
else: # Result
|
||||
lang = i18n.base_tag(msg.lang)
|
||||
@@ -370,37 +617,31 @@ class Dispatcher:
|
||||
):
|
||||
await ws.close(code=1002)
|
||||
return
|
||||
texts, spans, original = msg.texts, state.spans, state.original
|
||||
kind, state.kind = state.kind, ""
|
||||
state.inflight = None
|
||||
state.spans = []
|
||||
state.original = ""
|
||||
text = (
|
||||
join(original, spans, texts)
|
||||
if len(texts) == len(spans)
|
||||
else None
|
||||
mode, kind, spans, original = state.take()
|
||||
results = self._results(
|
||||
mode, kind, msg.key, original, spans, msg.texts
|
||||
)
|
||||
if text is None:
|
||||
# The model broke the segment contract (count
|
||||
# mismatch, empty or non-prose segment): drop the
|
||||
# result and skip the fragment for this run (it
|
||||
# stays pending; a restart, a refresh or a model
|
||||
# change gets another chance).
|
||||
self.validation_failures.add((lang, msg.key))
|
||||
if results is None:
|
||||
# The model broke the contract (bad segment count,
|
||||
# markup in a segment, a merged/split block, a lost
|
||||
# anchor): drop the result and skip the (lang, key,
|
||||
# mode) for this run — the work stays pending and a
|
||||
# restart, a refresh, another mode or a model change
|
||||
# gets another chance.
|
||||
self.validation_failures.add((lang, msg.key, mode))
|
||||
logger.warning(
|
||||
"[%s] result for chunk %s rejected: invalid segments",
|
||||
"[%s] %s result for %s rejected: failed validation",
|
||||
lang,
|
||||
mode,
|
||||
msg.key.hex(),
|
||||
)
|
||||
self.schedule()
|
||||
continue
|
||||
with self.db.transaction(
|
||||
f"translate:{lang}{':title' if kind == 'title' else ''}",
|
||||
f"translate:{lang}{':' + kind if kind != 'chunk' else ''}",
|
||||
user=clientkey,
|
||||
):
|
||||
paths = store_results(
|
||||
self.data, lang, [TransResult(key=msg.key, text=text)]
|
||||
)
|
||||
paths = store_results(self.data, lang, results)
|
||||
self.invalidate() # schedules the next dispatch
|
||||
if paths:
|
||||
logger.info(
|
||||
|
||||
Reference in New Issue
Block a user