pagerite/docs/llm-translation.md

# Whole-article and scoped LLM translation

Design for augmenting the fragment-based machine translation
(docs/localization.md) with general-purpose instruct LLMs that understand
Markdown natively — as opposed to pure text-to-text models like Seed-X.

## Motivation

The chunk + segment pipeline (`chunks.py` → `segments.py` → Seed-X) exists
because Seed-X mangles Markdown: links, formatting, fences and placeholders
must be stripped before dispatch and re-inserted into the result. The
re-insertion of link and formatting markup is the imprecise part: when
word-alignment by form similarity finds no anchor (always for CJK targets),
positions fall back to word-weight ratios, which land a word or so off.
All of `segments.py` — segmentation, offset splicing, `_find_mark`,
weight-ratio fallback, `_NEUTRAL` punctuation swaps, `<` encoding — is
defensive scaffolding around that one limitation.

An instruct LLM translates Markdown natively: `[text](url)` stays intact
and moves as a unit, fences, container markers, attrs and `{...}`
placeholders are preserved, and link texts translate in sentence context.
For such a translator the entire segments layer is unnecessary.

## Trial evidence (2026-09, RTX 4090 24 GB + 128 GB RAM, ollama 0.34)

Whole-article translation of two real articles (5.5 KB marketing, 18.3 KB
technical with code fences and `{dates}`) into fi/es/zh:

- **qwen3.8:27b** (dense, 17 GB Q4 — fits VRAM): structure-perfect on all
  runs — URLs, placeholders, heading/block counts preserved, fenced code
  byte-identical. es/zh excellent; fi fluent with occasional lexical slips
  (covered by the human patch layer). ~30 s per short article, ~2.5 min
  for 18 KB. **The reference model for article and markdown modes.**
- **qwen3:30b-instruct**: 3× faster, good prose, but rewrote comments and
  docstrings inside code fences despite explicit instructions — fails
  anchor validation (see below).
- **qwen3-next:80b** (MoE): best Finnish word choice on short documents,
  but degenerates on longer input in every configuration tried — runaway
  thinking loops (293k tokens), empty responses, 3× length output with
  hallucinated URLs, and ~10× blowup even in 2 KB scoped chunks. Unusable
  on current ollama builds.
- **CPU-only** (i7-14700, 14 threads): MoE 3B-active 10.6 t/s generation
  (viable for batch), dense 27B 2.5 t/s (not viable). Hybrid GPU+CPU
  splits bottleneck prompt evaluation (~52 t/s vs 62 t/s pure CPU) —
  dense GPU-resident or MoE CPU-resident are the sane configurations;
  mixing hurts.

Operational requirements established by the trials (all client-side):

- **Always disable thinking** for hybrid models (`think: false` on
  ollama): the reasoning phase adds minutes per article and can loop
  unbounded.
- **Always cap generation** (`num_predict` ≈ 2–3× input tokens): a
  runaway on a whole-article job burns hours, vs. seconds for a
  Seed-X segment.
- temperature 0.2 with the strict structure prompt works well for
  qwen3.8.

## What carries over unchanged

The valuable parts of the current design are the **storage and staleness
model**, not the segmentation — and none of them require the machine
translation to be produced chunk by chunk. The chunk store is a
storage/diffing format; LLM output at any granularity is *projected
into* it:

- Content-addressed source chunks (`Data.chunks`, `chunk_key`) — staleness
  still falls out of source-hash keys: editing the original invalidates
  exactly the edited chunks, all other translations keep applying.
- Per-chunk machine translations (`Data.trans[hash][lang]`) and the hybrid
  render with per-chunk fallback to the original.
- User patches (`Data.patches`) — search/replace hunks over the assembled
  hybrid, per-hunk independent and best-effort. Patches are orthogonal to
  how `Data.trans` entries were produced.
- `pending_items`: after a source edit, exactly the changed (lang, hash)
  pairs are pending — **focused retranslation of edits falls out of the
  existing bookkeeping**, no whole-article reruns.

## Protocol: capabilities and job modes

The `/_translate/{key}` WebSocket stays the single channel; Seed-X
clients work unchanged. The client→server `Hello` gains two optional
fields:

```python
class Hello(msgspec.Struct, tag="hello"):
    langs: list[str]           # as today: languages the model can produce
    model: str = ""            # free-form model string (logging, debugging)
    modes: list[str] = ["segments"]  # job granularities accepted
```

Three job modes, in increasing granularity:

- **`segments`** — the current protocol, unchanged: `Job.texts` carries
  prose segments (markup never crosses the wire), `Result.texts` returns
  them, the server splices by offset (`segments.py`). For text-to-text
  models (Seed-X). Default when a client omits `modes`.
- **`markdown`** (scoped instruct mode) — one fragment as full Markdown:
  a body chunk or a title. `Job.texts` carries a single element, the
  chunk's Markdown; `Job.contexts` carries up to two context strings
  (previous and next block of the **served hybrid** in the target
  language — current machine translation with user patches applied),
  "" where none. The client is instructed to output ONLY the translation
  of the target block; the context is terminology/tone reference.
  Using the *patched* hybrid as context propagates human corrections
  into fresh machine translations without the LLM ever touching patch
  storage. `Result.texts` carries one element, the translated block.
  The server validates: exactly one block after re-chunking, anchor
  constructs (URLs, image destinations, code fence content, `{...}`
  placeholders) preserved where the source block has them, then stores
  to `Data.trans` as usual.
- **`article`** — a whole page. `Job.texts` carries one element, the full
  original Markdown (the chunk sequence is recoverable server-side via
  `node.chunks`); `Result.texts` carries one element, the full translated
  Markdown. The server decomposes (below) and stores per chunk.

Titles are jobs like any other in all modes (`kind="title"` keeps its
article-opening context rule; in `markdown` mode a title crosses as
plain text, since it carries no markup by construction).

### Dispatch and validation

- Routing is per connection as today (wanted ∩ capable, one job in
  flight, requeue on disconnect), extended by mode: the smallest
  suitable unit goes to each free connection — `article` jobs only to
  article-capable connections, and only while a page is *mostly*
  pending (a whole new article or a full refresh); steady-state edit
  follow-up is `markdown`/`segments` jobs. Mixed translator fleets (a
  Seed-X instance, a local qwen, an API-backed client) run concurrently
  and share the work by capability.
- The validation skip-list becomes **mode-scoped** (`(lang, key, mode)`):
  a fragment a Seed-X client rejects stays offerable to instruct clients
  (and vice versa) — near-deterministic re-failure applies per model,
  not across approaches.
- `Result` matching is unchanged (lang, key); article results match on
  the key of the article's first chunk.

## Article result decomposition

1. Re-chunk the translated article with the same `chunk_markdown`.
2. Align translated blocks to source blocks. A well-behaved model does
   not reorder paragraphs, so positional / `SequenceMatcher` alignment
   at block granularity suffices. Blocks that must not change — code
   fences, container fence lines, `{...}` placeholders, image
   destinations, raw HTML — are matched verbatim and serve as alignment
   anchors, like diff context lines.
3. Store each translated block in `Data.trans[source_chunk_hash][lang]`.

Validation happens *before* anything is stored, same spirit as the
`pure_prose` segment checks but structural:

- Anchor blocks must appear verbatim and in order (this is what rejects
  qwen3:30b-instruct's translated code comments automatically).
- Per anchor-bounded region, source and translated block counts must
  match 1:1; regions that don't align store nothing and their chunks
  stay pending (they fall back to `markdown`-mode scoped jobs).

## Reference client

A second client script next to `scripts/translator.py` speaking the
`markdown` and `article` modes. Internally it targets the **OpenAI
Chat Completions API shape** (`POST /v1/chat/completions`): ollama
serves it at `:11434/v1`, llama.cpp's server likewise, and hosted APIs
(OpenAI and compatible providers) natively — `base_url` + `model` +
optional API key in the client's config selects local GPU, local CPU or
a remote model, with backend quirks (ollama's `think: false`,
`num_predict` cap, per-model sampling) in a per-model config section.
How the client drives its LLM is its internal matter; the wire protocol
above is the contract.

The client announces in `Hello`:

- `model`: the model string it is actually serving (e.g. `qwen3.8:27b`)
- `langs`: from its per-model language table — for the shipped qwen3.8
  configuration the site languages as configured server-side
  (de, es, fi, pt, zh; Finnish flagged as the weakest, patch-covered)
- `modes`: `["markdown", "article"]` for a structure-proven model,
  `["markdown"]` for one that is only trusted in scoped mode

The Seed-X client is untouched and announces `["segments"]` (implicitly,
by omitting `modes`).

## Importing human-made full translations

The decomposition function doubles as an import path for translations
produced outside the pipeline — e.g. an article translated with ChatGPT
and pasted back. Today such a paste lands in the translation editor and
is stored as one giant user patch; feeding it through the same
decomposition instead writes proper `Data.trans` fragments, so later
source edits invalidate and re-translate per chunk rather than letting
the monolithic patch silently go stale hunk by hunk. This import path is
also the natural testbed for the decomposition and validation logic
before any live LLM client uses it.