Links translated in sentence context, re-linked by text weight

- segments.py: a block of plain text with inline links now stays ONE wire
  segment (Mark/Span) — the link label crosses in sentence context, so
  translations come back grammatically coherent instead of incompatible
  snippets. join() re-inserts the link markdown into the translated block
  at weight-mapped positions (word units; CJK ideographs and kana runs
  count one unit each), degrading to the source label on empty slices and
  still rejecting markup injection. No sentinels on the wire: boundaries
  are found by text processing alone.
- translate.py: the Dispatcher tracks Span (offsets + link marks) for the
  in-flight job.
This commit is contained in:
2026-09-03 01:46:56 +00:00
parent b0866fc4f7
commit 43793ef9a4
4 changed files with 293 additions and 44 deletions
+24 -8
View File
@@ -339,8 +339,10 @@ fragment is parsed with the project's own markdown-it setup
(`markdown.make_md(verbatim=True)` — all extensions, but no typographer or
tasklist label wrapping, so token text stays byte-identical to the source)
and split into the runs a model may touch: paragraph/heading/table-cell text
(merged across soft line breaks), link text, image alt texts and captions,
footnote bodies. Everything else never leaves the server: code spans and
(merged across soft line breaks), image alt texts and captions, footnote
bodies. A block of plain text and inline **links stays whole** — link texts
cross inline, in sentence context (see below). Everything else never leaves
the server: code spans and
fences, URLs and autolinks, link/image *destinations*, `{...}` spans
(placeholders like `{dates}` as well as attrs), reference and footnote
labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all
@@ -366,11 +368,24 @@ near-deterministic, so an immediate retry would re-fail; the fragment stays
pending and gets another chance on restart or `DELETE /_api/translations`).
`Data.trans` therefore only ever holds clean translated Markdown.
The trade-off: segments splice back at fixed positions, so a translation
cannot move a link or image within a sentence — word order around inline
markup follows the original. That is the price for never feeding the model
markup (an earlier sentinel-masking design let the model see and mangle
exactly that punctuation: Seed-X turned `![` into `¡¡…!!`).
Link-carrying blocks are the one place a segment is not spliced verbatim:
a label translated apart from its sentence comes back grammatically
incompatible with it (case government, particles, word order), so the
block crosses whole and the server re-inserts the link markdown into the
translated block. The boundaries are found by **text processing alone**
markers on the wire are hopeless (an earlier sentinel-masking design let
the model see and mangle exactly that punctuation: Seed-X renumbered the
tokens and turned `![` into `¡¡…!!`). Each link's weight ratio in the
source block (word units before its text boundaries over the block total;
CJK ideographs count as one unit each, kana runs as one — no spaces to
count words by) is applied to the translation's units. Placement is
approximate and drift accumulates across several links in one block — the
accepted trade: better a coherent sentence with a slightly shifted link
than separately translated snippets that don't fit together. A boundary
that maps to an empty slice degrades to the source link text rather than
emitting a broken `[](url)`. Blocks mixing in any other inline markup
(emphasis, code spans, images) don't qualify and still split into runs at
those boundaries.
Punctuation is the translator's own job: Seed-X tends to "finish" short
labels (titles, nav items) with a comma or period the source never had.
@@ -395,7 +410,8 @@ framing token, which would trip a `<` stop immediately.)
Short fragments get more than a bare prompt: each segment may carry its
surround in `Job.contexts` — a title carries the article's opening prose
(its own block is just the title word), a segment carved out of a larger
block (a link text, a partial run) carries the block's plain text, and a
block (a partial run; a link text whose block didn't qualify for the
whole-block treatment) carries the block's plain text, and a
whole-block segment (a plain paragraph) is self-contextualizing and carries
"". The reference client translates segment and surround together, stops
generation at the blank line separating them, and keeps the segment's own