Links translated in sentence context, re-linked by text weight
- segments.py: a block of plain text with inline links now stays ONE wire segment (Mark/Span) — the link label crosses in sentence context, so translations come back grammatically coherent instead of incompatible snippets. join() re-inserts the link markdown into the translated block at weight-mapped positions (word units; CJK ideographs and kana runs count one unit each), degrading to the source label on empty slices and still rejecting markup injection. No sentinels on the wire: boundaries are found by text processing alone. - translate.py: the Dispatcher tracks Span (offsets + link marks) for the in-flight job.
This commit is contained in:
+24
-8
@@ -339,8 +339,10 @@ fragment is parsed with the project's own markdown-it setup
|
||||
(`markdown.make_md(verbatim=True)` — all extensions, but no typographer or
|
||||
tasklist label wrapping, so token text stays byte-identical to the source)
|
||||
and split into the runs a model may touch: paragraph/heading/table-cell text
|
||||
(merged across soft line breaks), link text, image alt texts and captions,
|
||||
footnote bodies. Everything else never leaves the server: code spans and
|
||||
(merged across soft line breaks), image alt texts and captions, footnote
|
||||
bodies. A block of plain text and inline **links stays whole** — link texts
|
||||
cross inline, in sentence context (see below). Everything else never leaves
|
||||
the server: code spans and
|
||||
fences, URLs and autolinks, link/image *destinations*, `{...}` spans
|
||||
(placeholders like `{dates}` as well as attrs), reference and footnote
|
||||
labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all
|
||||
@@ -366,11 +368,24 @@ near-deterministic, so an immediate retry would re-fail; the fragment stays
|
||||
pending and gets another chance on restart or `DELETE /_api/translations`).
|
||||
`Data.trans` therefore only ever holds clean translated Markdown.
|
||||
|
||||
The trade-off: segments splice back at fixed positions, so a translation
|
||||
cannot move a link or image within a sentence — word order around inline
|
||||
markup follows the original. That is the price for never feeding the model
|
||||
markup (an earlier sentinel-masking design let the model see and mangle
|
||||
exactly that punctuation: Seed-X turned ``. Blocks mixing in any other inline markup
|
||||
(emphasis, code spans, images) don't qualify and still split into runs at
|
||||
those boundaries.
|
||||
|
||||
Punctuation is the translator's own job: Seed-X tends to "finish" short
|
||||
labels (titles, nav items) with a comma or period the source never had.
|
||||
@@ -395,7 +410,8 @@ framing token, which would trip a `<` stop immediately.)
|
||||
Short fragments get more than a bare prompt: each segment may carry its
|
||||
surround in `Job.contexts` — a title carries the article's opening prose
|
||||
(its own block is just the title word), a segment carved out of a larger
|
||||
block (a link text, a partial run) carries the block's plain text, and a
|
||||
block (a partial run; a link text whose block didn't qualify for the
|
||||
whole-block treatment) carries the block's plain text, and a
|
||||
whole-block segment (a plain paragraph) is self-contextualizing and carries
|
||||
"". The reference client translates segment and surround together, stops
|
||||
generation at the blank line separating them, and keeps the segment's own
|
||||
|
||||
Reference in New Issue
Block a user