Improved auto translation support (#2)
- Added full article / with markdown mode for advanced LLMs, with local ollama and API support, new llm_translator script. - Translate navigational titles as a whole (preserving site structure), and then retranslate as article headings with article context - potentially changing nav label - Improved human overrides to translations, in particular when structure is affected Reviewed-on: #2
This commit was merged in pull request #2.
This commit is contained in:
@@ -0,0 +1,234 @@
|
||||
# Whole-article and scoped LLM translation
|
||||
|
||||
**Status: implemented** (protocol modes and validation in
|
||||
`pagerite/translate.py`, reference client `scripts/llm_translator.py`,
|
||||
human-translation import `scripts/import_translation.py`; wire-level docs
|
||||
in docs/localization.md). One deviation from the text below: ollama's
|
||||
OpenAI-compatible `/v1/chat/completions` silently ignores `think: false`
|
||||
(verified on 0.34.2), so the client's `api` config selects ollama's native
|
||||
`/api/chat` for ollama backends; the OpenAI shape serves llama.cpp and
|
||||
hosted APIs.
|
||||
|
||||
Design for augmenting the fragment-based machine translation
|
||||
(docs/localization.md) with general-purpose instruct LLMs that understand
|
||||
Markdown natively — as opposed to pure text-to-text models like Seed-X.
|
||||
|
||||
## Motivation
|
||||
|
||||
The chunk + segment pipeline (`chunks.py` → `segments.py` → Seed-X) exists
|
||||
because Seed-X mangles Markdown: links, formatting, fences and placeholders
|
||||
must be stripped before dispatch and re-inserted into the result. The
|
||||
re-insertion of link and formatting markup is the imprecise part: when
|
||||
word-alignment by form similarity finds no anchor (always for CJK targets),
|
||||
positions fall back to word-weight ratios, which land a word or so off.
|
||||
All of `segments.py` — segmentation, offset splicing, `_find_mark`,
|
||||
weight-ratio fallback, `_NEUTRAL` punctuation swaps, `<` encoding — is
|
||||
defensive scaffolding around that one limitation.
|
||||
|
||||
An instruct LLM translates Markdown natively: `[text](url)` stays intact
|
||||
and moves as a unit, fences, container markers, attrs and `{...}`
|
||||
placeholders are preserved, and link texts translate in sentence context.
|
||||
For such a translator the entire segments layer is unnecessary.
|
||||
|
||||
## Trial evidence (2026-09, RTX 4090 24 GB + 128 GB RAM, ollama 0.34)
|
||||
|
||||
Whole-article translation of two real articles (5.5 KB marketing, 18.3 KB
|
||||
technical with code fences and `{dates}`) into fi/es/zh:
|
||||
|
||||
- **qwen3.8:27b** (dense, 17 GB Q4 — fits VRAM): structure-perfect on all
|
||||
runs — URLs, placeholders, heading/block counts preserved, fenced code
|
||||
byte-identical. es/zh excellent; fi fluent with occasional lexical slips
|
||||
(covered by the human patch layer). ~30 s per short article, ~2.5 min
|
||||
for 18 KB. **The reference model for article and markdown modes.**
|
||||
- **qwen3:30b-instruct**: 3× faster, good prose, but rewrote comments and
|
||||
docstrings inside code fences despite explicit instructions — fails
|
||||
anchor validation (see below).
|
||||
- **qwen3-next:80b** (MoE): best Finnish word choice on short documents,
|
||||
but degenerates on longer input in every configuration tried — runaway
|
||||
thinking loops (293k tokens), empty responses, 3× length output with
|
||||
hallucinated URLs, and ~10× blowup even in 2 KB scoped chunks. Unusable
|
||||
on current ollama builds.
|
||||
- **CPU-only** (i7-14700, 14 threads): MoE 3B-active 10.6 t/s generation
|
||||
(viable for batch), dense 27B 2.5 t/s (not viable). Hybrid GPU+CPU
|
||||
splits bottleneck prompt evaluation (~52 t/s vs 62 t/s pure CPU) —
|
||||
dense GPU-resident or MoE CPU-resident are the sane configurations;
|
||||
mixing hurts.
|
||||
|
||||
Operational requirements established by the trials (all client-side):
|
||||
|
||||
- **Always disable thinking** for hybrid models (`think: false` on
|
||||
ollama): the reasoning phase adds minutes per article and can loop
|
||||
unbounded.
|
||||
- **Always cap generation** (`num_predict` ≈ 2–3× input tokens): a
|
||||
runaway on a whole-article job burns hours, vs. seconds for a
|
||||
Seed-X segment.
|
||||
- temperature 0.2 with the strict structure prompt works well for
|
||||
qwen3.8.
|
||||
|
||||
## What carries over unchanged
|
||||
|
||||
The valuable parts of the current design are the **storage and staleness
|
||||
model**, not the segmentation — and none of them require the machine
|
||||
translation to be produced chunk by chunk. The chunk store is a
|
||||
storage/diffing format; LLM output at any granularity is *projected
|
||||
into* it:
|
||||
|
||||
- Content-addressed source chunks (`Data.chunks`, `chunk_key`) — staleness
|
||||
still falls out of source-hash keys: editing the original invalidates
|
||||
exactly the edited chunks, all other translations keep applying.
|
||||
- Per-chunk machine translations (`Data.trans[hash][lang]`) and the hybrid
|
||||
render with per-chunk fallback to the original.
|
||||
- User patches (`Data.patches`) — search/replace hunks over the assembled
|
||||
hybrid, per-hunk independent and best-effort. Patches are orthogonal to
|
||||
how `Data.trans` entries were produced.
|
||||
- `pending_items`: after a source edit, exactly the changed (lang, hash)
|
||||
pairs are pending — **focused retranslation of edits falls out of the
|
||||
existing bookkeeping**, no whole-article reruns.
|
||||
|
||||
## Protocol: capabilities and job modes
|
||||
|
||||
The `/_translate/{key}` WebSocket stays the single channel; Seed-X
|
||||
clients work unchanged. The client→server `Hello` gains two optional
|
||||
fields:
|
||||
|
||||
```python
|
||||
class Hello(msgspec.Struct, tag="hello"):
|
||||
langs: list[str] # as today: languages the model can produce
|
||||
model: str = "" # free-form model string (logging, debugging)
|
||||
modes: list[str] = ["segments"] # job granularities accepted
|
||||
```
|
||||
|
||||
Four job modes, in increasing granularity:
|
||||
|
||||
- **`segments`** — the current protocol, unchanged: `Job.texts` carries
|
||||
prose segments (markup never crosses the wire), `Result.texts` returns
|
||||
them, the server splices by offset (`segments.py`). For text-to-text
|
||||
models (Seed-X). Default when a client omits `modes`.
|
||||
- **`markdown`** (scoped instruct mode) — one fragment as full Markdown:
|
||||
a body chunk or a title. `Job.texts` carries a single element, the
|
||||
chunk's Markdown; `Job.contexts` carries up to two context strings
|
||||
(previous and next block of the **served hybrid** in the target
|
||||
language — current machine translation with user patches applied),
|
||||
"" where none. The client is instructed to output ONLY the translation
|
||||
of the target block; the context is terminology/tone reference.
|
||||
Using the *patched* hybrid as context propagates human corrections
|
||||
into fresh machine translations without the LLM ever touching patch
|
||||
storage. `Result.texts` carries one element, the translated block.
|
||||
The server validates: exactly one block after re-chunking, anchor
|
||||
constructs (URLs, image destinations, code fence content, `{...}`
|
||||
placeholders) preserved where the source block has them, then stores
|
||||
to `Data.trans` as usual.
|
||||
- **`article`** — a whole page. `Job.texts` carries one element, the full
|
||||
original Markdown (the chunk sequence is recoverable server-side via
|
||||
`node.chunks`); `Result.texts` carries one element, the full translated
|
||||
Markdown. The server decomposes (below) and stores per chunk.
|
||||
- **`nav`** — the whole navigation hierarchy. `Job.texts` carries one
|
||||
element, a nested Markdown list of every node title still pending for
|
||||
the language (`- Title`, indented by depth, in menu order);
|
||||
`Result.texts` carries one element, the translated list. The server
|
||||
decomposes by list structure (`align_nav`): item count and nesting
|
||||
depth must match the source item for item, then each item is stored as
|
||||
a per-title fragment under its title's chunk hash.
|
||||
|
||||
Titles are jobs like any other in all modes (`kind="title"` keeps its
|
||||
article-opening context rule; in `markdown` mode a title crosses as
|
||||
plain text, since it carries no markup by construction) — but for
|
||||
nav-capable connections a single `nav` job names the entire menu first:
|
||||
one round trip instead of one per page, with siblings, parents and
|
||||
children translating in sight of each other. A structurally mangled list
|
||||
is rejected wholesale and the titles fall back to scoped title jobs.
|
||||
Additionally, an
|
||||
`article` job carries the page title injected as a `# {title}` line at
|
||||
the top when the render would inject it (the body has no h1 of its own):
|
||||
the title translates in document context and the opening paragraphs see
|
||||
the heading. The menu title's and parent node's existing translations
|
||||
ride along as `Job.contexts` ("" where none), so the heading can match
|
||||
the menu while the model may still adapt the in-article title to the
|
||||
content. The heading's pair in the decomposed result becomes the
|
||||
title fragment (heading text only, never stored as a body chunk).
|
||||
|
||||
### Dispatch and validation
|
||||
|
||||
- Routing is per connection as today (wanted ∩ capable, one job in
|
||||
flight, requeue on disconnect), extended by mode: the smallest
|
||||
suitable unit goes to each free connection — `article` jobs only to
|
||||
article-capable connections, and only while a page is *mostly*
|
||||
pending (a whole new article or a full refresh); steady-state edit
|
||||
follow-up is `markdown`/`segments` jobs. Mixed translator fleets (a
|
||||
Seed-X instance, a local qwen, an API-backed client) run concurrently
|
||||
and share the work by capability.
|
||||
- The validation skip-list becomes **mode-scoped** (`(lang, key, mode)`):
|
||||
a fragment a Seed-X client rejects stays offerable to instruct clients
|
||||
(and vice versa) — near-deterministic re-failure applies per model,
|
||||
not across approaches.
|
||||
- `Result` matching is unchanged (lang, key); article results match on
|
||||
the key of the article's first chunk.
|
||||
|
||||
## Article result decomposition
|
||||
|
||||
1. Re-chunk the translated article with the same `chunk_markdown`.
|
||||
2. Align translated blocks to source blocks. A well-behaved model does
|
||||
not reorder paragraphs, so positional / `SequenceMatcher` alignment
|
||||
at block granularity suffices. Blocks that must not change — code
|
||||
fences, container fence lines, `{...}` placeholders, image
|
||||
destinations, raw HTML — are matched verbatim and serve as alignment
|
||||
anchors, like diff context lines.
|
||||
3. Store each translated block in `Data.trans[source_chunk_hash][lang]`.
|
||||
|
||||
Validation happens *before* anything is stored, same spirit as the
|
||||
`pure_prose` segment checks but structural:
|
||||
|
||||
- Anchor blocks must appear verbatim and in order (this is what rejects
|
||||
qwen3:30b-instruct's translated code comments automatically).
|
||||
- Per anchor-bounded region, source and translated block counts must
|
||||
match 1:1; regions that don't align store nothing and their chunks
|
||||
stay pending (they fall back to `markdown`-mode scoped jobs).
|
||||
|
||||
## Reference client
|
||||
|
||||
A second client script next to `scripts/translator.py` speaking the
|
||||
`markdown` and `article` modes. Internally it targets the **OpenAI
|
||||
Chat Completions API shape** (`POST /v1/chat/completions`): ollama
|
||||
serves it at `:11434/v1`, llama.cpp's server likewise, and hosted APIs
|
||||
(OpenAI and compatible providers) natively — `--base-url` + `--model`
|
||||
selects local GPU, local CPU or a remote model, the API key comes from
|
||||
the standard per-provider environment variable (`KIMI_API_KEY`,
|
||||
`MOONSHOT_API_KEY`, `OPENAI_API_KEY`, each sent only to its own
|
||||
provider's host; `LLM_API_KEY` for anything else) — deliberately never
|
||||
a CLI flag or a config file — and backend quirks (ollama's
|
||||
`think: false`, `num_predict` cap, per-model sampling) live in the
|
||||
script's `DEFAULT_CONFIG`.
|
||||
How the client drives its LLM is its internal matter; the wire protocol
|
||||
above is the contract.
|
||||
|
||||
Field-proven backends: the local qwen3.8:27b of the trials above, and
|
||||
the **Kimi Code API** (`--base-url https://api.kimi.com/coding` resp.
|
||||
`api.kimi.ai`, `--model k3-256k`): the `/coding` endpoint fixes sampling
|
||||
internally (the client drops `temperature`/`top_p` for it — they 400)
|
||||
and runs `reasoning_effort: low` from the config, which produces good
|
||||
translations at a fraction of the default (high) effort's latency and
|
||||
quota; thinking output is logged verbatim but stripped from the result.
|
||||
|
||||
The client announces in `Hello`:
|
||||
|
||||
- `model`: the model string it is actually serving (e.g. `qwen3.8:27b`)
|
||||
- `langs`: from its per-model language table — for the shipped qwen3.8
|
||||
configuration the site languages as configured server-side
|
||||
(de, es, fi, pt, zh; Finnish flagged as the weakest, patch-covered)
|
||||
- `modes`: `["markdown", "article", "nav"]` for a structure-proven model,
|
||||
`["markdown"]` for one that is only trusted in scoped mode
|
||||
|
||||
The Seed-X client is untouched and announces `["segments"]` (implicitly,
|
||||
by omitting `modes`).
|
||||
|
||||
## Importing human-made full translations
|
||||
|
||||
The decomposition function doubles as an import path for translations
|
||||
produced outside the pipeline — e.g. an article translated with ChatGPT
|
||||
and pasted back. Today such a paste lands in the translation editor and
|
||||
is stored as one giant user patch; feeding it through the same
|
||||
decomposition instead writes proper `Data.trans` fragments, so later
|
||||
source edits invalidate and re-translate per chunk rather than letting
|
||||
the monolithic patch silently go stale hunk by hunk. This import path is
|
||||
also the natural testbed for the decomposition and validation logic
|
||||
before any live LLM client uses it.
|
||||
+92
-13
@@ -337,15 +337,20 @@ transaction `user`.
|
||||
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
|
||||
`bytes` fields ride as base64):
|
||||
|
||||
- `{"type": "hello", "langs": [...]}` — client greeting announcing its
|
||||
**capabilities**: the language codes its model can produce (normalized
|
||||
to base subtags; `en`/empty dropped).
|
||||
- `{"type": "job", "lang", "key", "texts", "path", "kind", "contexts"}` —
|
||||
server push: ONE fragment to translate (an article title or a chunk), as
|
||||
a list of **prose segments** (see Segmentation below). `contexts` is
|
||||
parallel to `texts` ("" = none): the surround to translate the segment
|
||||
in — for clients that translate better with context (see below).
|
||||
Contexts are not part of the result.
|
||||
- `{"type": "hello", "langs": [...], "model", "modes"}` — client greeting
|
||||
announcing its **capabilities**: the language codes its model can produce
|
||||
(normalized to base subtags; `en`/empty dropped). `model` is a free-form
|
||||
model string (logging only); `modes` lists the job granularities the
|
||||
client accepts (default `["segments"]`, see Job modes below).
|
||||
- `{"type": "job", "lang", "key", "texts", "path", "kind", "mode",
|
||||
"contexts"}` — server push: ONE fragment to translate (an article title
|
||||
or a chunk; the bulk `article`/`nav` modes carry a whole page resp. the
|
||||
whole navigation tree, see Job modes). In the default `segments` mode
|
||||
`texts` is a list of **prose
|
||||
segments** (see Segmentation below) and `contexts` is parallel to `texts`
|
||||
("" = none): the surround to translate the segment in — for clients that
|
||||
translate better with context (see below). Contexts are not part of the
|
||||
result. See Job modes for the other modes.
|
||||
- `{"type": "result", "lang", "key", "texts"}` — client reply: the
|
||||
segments translated, same order and count, matching its job by (lang, key).
|
||||
|
||||
@@ -387,6 +392,78 @@ Results are stored into `trans` in one transaction and set
|
||||
pages gain a language from one fragment). Unknown keys are stored anyway
|
||||
and re-storing overwrites — results are idempotent.
|
||||
|
||||
#### Job modes: segments, markdown, article, nav
|
||||
|
||||
Instruct LLMs understand Markdown natively, so for them the segmentation
|
||||
round trip below is unnecessary scaffolding (docs/llm-translation.md for
|
||||
the design and the model trial evidence). `Hello.modes` announces which
|
||||
job granularities a connection accepts; routing is per connection and per
|
||||
mode, so a mixed fleet (a Seed-X instance, a local qwen, an API-backed
|
||||
client) shares the work by capability. The validation skip-list is
|
||||
mode-scoped — `(lang, key, mode)` — so a fragment one model rejects stays
|
||||
offerable to clients of another approach.
|
||||
|
||||
- **`segments`** (default when a client omits `modes`) — the protocol as
|
||||
described so far: `Job.texts` carries prose segments, `Result.texts`
|
||||
returns them, the server splices by offset.
|
||||
- **`markdown`** — one fragment as full Markdown: `Job.texts` carries a
|
||||
single element, the chunk (a title crosses as plain text, with the
|
||||
article's opening as its context as today). `Job.contexts` carries the
|
||||
previous and next block of the **served hybrid** in the target language
|
||||
(machine translation with user patches applied, "" where none), so human
|
||||
corrections propagate into fresh translations as terminology/tone
|
||||
reference; contexts are never part of the result. The result must
|
||||
re-chunk to exactly one block with the source's anchor constructs (link
|
||||
and image destinations, `{...}` placeholders) intact (`clean_block`),
|
||||
then stores to `Data.trans` as usual.
|
||||
- **`article`** — a whole page at once, offered only to article-capable
|
||||
connections and only while a page is *mostly* pending (a new article or
|
||||
a full refresh; steady-state edit follow-up stays scoped jobs). The
|
||||
job's key is the page's first chunk; `Job.texts` carries the full
|
||||
original Markdown — with the page title injected as a `# {title}` line
|
||||
at the top when the render would inject it (the body has no h1 of its
|
||||
own), so the title translates in document context and the opening
|
||||
paragraphs see the heading. The menu title's and parent node's existing
|
||||
translations (from a nav job or earlier work) ride along as
|
||||
`Job.contexts`, so the heading can match the menu while the model may
|
||||
still adapt the in-article title to the content. The result is
|
||||
decomposed per chunk
|
||||
(`align_article`): non-translatable blocks (code fences, container
|
||||
fences, raw HTML — everything `needs_translation` rejects) must appear
|
||||
verbatim and in order and anchor the alignment; regions between anchors
|
||||
pair positionally, a region whose block count changed stores nothing
|
||||
(its chunks stay pending and fall back to scoped jobs), and a paired
|
||||
block whose destinations/placeholders did not survive likewise. An
|
||||
injected title heading's pair becomes the title fragment (heading text
|
||||
only — never a body chunk; a demoted or merged heading simply skips it
|
||||
and the title stays pending for a scoped title job).
|
||||
- **`nav`** — the whole navigation hierarchy at once, offered only to
|
||||
nav-capable connections and ahead of any per-title jobs: `Job.texts`
|
||||
carries one element, a nested Markdown list of every node title still
|
||||
pending for the language (`- Title`, indented by depth, in menu order —
|
||||
pages and category labels alike); the job's key is the hash of that
|
||||
list. One round trip names the entire menu, and sibling titles
|
||||
translate in sight of each other. The result is decomposed back into
|
||||
per-title fragments (`align_nav`): it must be the same list item for
|
||||
item — same count, same nesting depth at every position — or it is
|
||||
rejected wholesale and the titles fall back to scoped title jobs; an
|
||||
item that comes back empty, marked-up or with its destinations/
|
||||
placeholders lost is skipped individually and likewise stays pending
|
||||
for a scoped title job.
|
||||
|
||||
`scripts/llm_translator.py` is the reference markdown+article+nav client
|
||||
(instruct LLMs via an OpenAI Chat Completions endpoint or ollama's native
|
||||
API); `scripts/translator.py` (Seed-X) is untouched and announces
|
||||
`["segments"]` implicitly.
|
||||
|
||||
**Importing human-made full translations:** `align_article` doubles as an
|
||||
import path — `scripts/import_translation.py PATH LANG FILE.md` (run with
|
||||
the server stopped) decomposes a pasted whole-article translation (e.g.
|
||||
from ChatGPT) into proper `Data.trans` fragments with the same validation,
|
||||
so later source edits invalidate and re-translate per chunk rather than
|
||||
letting the translation editor's one monolithic patch go stale hunk by
|
||||
hunk.
|
||||
|
||||
#### Segmentation
|
||||
|
||||
Fragments cross the wire as **prose segments** (`pagerite/segments.py`): the
|
||||
@@ -423,10 +500,12 @@ of the block it splices into, closing fence included — segments are
|
||||
inline prose, so `pure_prose` alone cannot see this) — each returned
|
||||
segment must parse as
|
||||
pure prose with no block-starting line or blank line, or the whole result
|
||||
is dropped and logged, and the (lang, key)
|
||||
pair is skipped for the rest of the server run (generation is
|
||||
near-deterministic, so an immediate retry would re-fail; the fragment stays
|
||||
pending and gets another chance on restart or `DELETE /_api/translations`).
|
||||
is dropped and logged, and the (lang, key, mode)
|
||||
combination is skipped for the rest of the server run (generation is
|
||||
near-deterministic per model, so an immediate retry in the same mode would
|
||||
re-fail; the fragment stays
|
||||
pending and gets another chance on restart, in another mode, or on
|
||||
`DELETE /_api/translations`).
|
||||
`Data.trans` therefore only ever holds clean translated Markdown.
|
||||
|
||||
Link- and formatting-carrying blocks are the one place a segment is not
|
||||
|
||||
Reference in New Issue
Block a user