From a47a96ca3089fbfef98d2a5ff53014656b3a925d Mon Sep 17 00:00:00 2001 From: Leo Vasanko Date: Thu, 3 Sep 2026 03:46:26 +0000 Subject: [PATCH] Whole-block segments cover paired formatting (strong/em/s), not just links MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The model mangles Markdown it can see (Seed-X dropped the ** and the glued-on colon in '**Pagerite**: ...'), so emphasis now crosses stripped, inline in its sentence, and the syntax is weight-mapped back like links. Also fix _place_marks swallowing trailing punctuation into the inner text (end at the slice's last unit) — same-language round trips are now exact. --- AGENTS.md | 2 +- docs/localization.md | 40 +++++++------ pagerite/segments.py | 131 ++++++++++++++++++++++++------------------- 3 files changed, 98 insertions(+), 75 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df9c2b4..5bee8d8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,7 +15,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md). - `i18n.py` — language selection, translation assembly (chunks + patches) and translated-edit recording (user patches, per-language title overrides, refresh). - `translate.py` — translator service protocol (msgspec structs), the connected-client `Dispatcher` (job pipeline, result validation) and pending/store core for the `/_translate/{key}` WebSocket (docs/localization.md); app.py only registers the route. - - `segments.py` — the translation round trip: fragments split into pure-prose wire segments (via markdown.make_md's verbatim parser; link-carrying blocks stay whole, link texts inline) and translations spliced back by source offset, link markdown re-inserted at weight-mapped positions (docs/localization.md). + - `segments.py` — the translation round trip: fragments split into pure-prose wire segments (via markdown.make_md's verbatim parser; link- and formatting-carrying blocks stay whole, link/formatted texts inline, Markdown stripped) and translations spliced back by source offset, link/formatting markdown re-inserted at weight-mapped positions (docs/localization.md). - `migrations.py` — kanta migrations (`migrate_vN`); ALL schema/storage upgrades live here (raw state dict before struct decoding), never in the app lifespan: v1 moves legacy in-db file blobs to the on-disk store and rebuilds the legacy flat `pages` as the menu tree, v2 rewrites `/_f/{hash}.ext` image links to the extension-less form, backfills AVIF/WebP/JPEG derivatives on disk and drops the obsolete `version` field. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering; theme/user-font resolution across `THEME_DIRS` / `FONT_DIRS` (cwd, site, platform data roots, then built-in `pagerite/themes/`, see `docs/themes-and-assets.md`). diff --git a/docs/localization.md b/docs/localization.md index 6f2155e..e124e3d 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -376,13 +376,14 @@ fragment is parsed with the project's own markdown-it setup tasklist label wrapping, so token text stays byte-identical to the source) and split into the runs a model may touch: paragraph/heading/table-cell text (merged across soft line breaks), image alt texts and captions, footnote -bodies. A block of plain text and inline **links stays whole** — link texts -cross inline, in sentence context (see below). Everything else never leaves -the server: code spans and +bodies. A block of plain text, inline **links and paired text formatting** +(strong/em/s) **stays whole** — link and formatted texts cross inline, in +sentence context, with the Markdown stripped (see below). Everything else +never leaves the server: code spans and fences, URLs and autolinks, link/image *destinations*, `{...}` spans (placeholders like `{dates}` as well as attrs), reference and footnote -labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all -markup punctuation (`*`, `|`, `[]()`, `:::`), which is a run boundary. +labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and the +remaining markup punctuation (`|`, `:::`), which is a run boundary. Chunks with no segments (a lone `{dates}`, container fences, pure code/HTML) are never dispatched at all (`needs_translation`); every language renders them from the original chunk. Each segment is accompanied @@ -404,24 +405,29 @@ near-deterministic, so an immediate retry would re-fail; the fragment stays pending and gets another chance on restart or `DELETE /_api/translations`). `Data.trans` therefore only ever holds clean translated Markdown. -Link-carrying blocks are the one place a segment is not spliced verbatim: -a label translated apart from its sentence comes back grammatically -incompatible with it (case government, particles, word order), so the -block crosses whole and the server re-inserts the link markdown into the -translated block. The boundaries are found by **text processing alone** — +Link- and formatting-carrying blocks are the one place a segment is not +spliced verbatim: a label translated apart from its sentence comes back +grammatically incompatible with it (case government, particles, word +order), and shown the Markdown the model mangles it (Seed-X dropped the +`**` and the glued-on colon in `**Pagerite**: …`), so the block crosses +whole — all Markdown stripped — and the server re-inserts the link and +formatting syntax into the translated block. The boundaries are found by +**text processing alone** — markers on the wire are hopeless (an earlier sentinel-masking design let the model see and mangle exactly that punctuation: Seed-X renumbered the -tokens and turned `![` into `¡¡…!!`). Each link's weight ratio in the +tokens and turned `![` into `¡¡…!!`). Each mark's weight ratio in the source block (word units before its text boundaries over the block total; CJK ideographs count as one unit each, kana runs as one — no spaces to -count words by) is applied to the translation's units. Placement is -approximate and drift accumulates across several links in one block — the +count words by) is applied to the translation's units; the inner text ends +at its last unit, so punctuation and whitespace between the mark and the +next word stay outside it. Placement is +approximate and drift accumulates across several marks in one block — the accepted trade: better a coherent sentence with a slightly shifted link than separately translated snippets that don't fit together. A boundary -that maps to an empty slice degrades to the source link text rather than -emitting a broken `[](url)`. Blocks mixing in any other inline markup -(emphasis, code spans, images) don't qualify and still split into runs at -those boundaries. +that maps to an empty slice degrades to the source text rather than +emitting a broken `[](url)` or `**`. Blocks mixing in any other inline +markup (code spans, images, raw HTML) don't qualify and still split into +runs at those boundaries. Punctuation is the translator's own job: Seed-X tends to "finish" short labels (titles, nav items) with a comma or period the source never had. diff --git a/pagerite/segments.py b/pagerite/segments.py index 00dbfd7..7bfee88 100644 --- a/pagerite/segments.py +++ b/pagerite/segments.py @@ -23,18 +23,20 @@ never left the server. A returned segment must still be pure prose itself mismatch, empty segment, markup tokens — rejects the whole result and the fragment stays pending. -A block of plain text and prose links crosses as ONE segment — link texts -inline, in sentence context — because a label translated apart from its -sentence comes back grammatically incompatible with it (case government, -particles, word order). ``join`` re-inserts the link markdown into the -translated block at weight-mapped positions (``_place_marks``): no markers -on the wire (sentinels never survived the model — they got renumbered and -mangled), the boundaries are found by text processing alone — each link's -word/CJK-char weight ratio in the source applied to the translation's -units. Placement is approximate and CJK-safe: better a coherent sentence -with a slightly shifted link than separately translated snippets that -don't fit together. Blocks with any other inline markup (emphasis, code, -images) still split into runs at those boundaries. +A block of plain text, prose links and paired text formatting +(strong/em/s) crosses as ONE segment — link texts and formatted text +inline, in sentence context, with the Markdown stripped (the model +mangles it: sentinels get renumbered, ``**`` gets dropped or moved) — +because a label translated apart from its sentence comes back +grammatically incompatible with it (case government, particles, word +order). ``join`` re-inserts the link/formatting markdown into the +translated block at weight-mapped positions (``_place_marks``): no +markers on the wire, the boundaries are found by text processing alone — +each mark's word/CJK-char weight ratio in the source applied to the +translation's units. Placement is approximate and CJK-safe: better a +coherent sentence with a slightly shifted link than separately translated +snippets that don't fit together. Blocks with any other inline markup +(code, images, HTML) still split into runs at those boundaries. Locating is best effort: a run that is not a verbatim source substring (entity-decoded text, backslash escapes) is skipped — it simply stays in @@ -82,12 +84,13 @@ _UNIT = re.compile( class Mark(NamedTuple): - """One inline link inside a whole-block segment: the source weight - (unit count, see _UNIT) at the link text's start and end for mapping - the boundaries into the translation, the exact source syntax around - the text ("[" / "](url)" etc.) and the source text itself, used as the - fallback when the mapped slice comes out empty (better an untranslated - label than a broken "[](url)").""" + """One inline link or paired formatting (strong/em/s) inside a + whole-block segment: the source weight (unit count, see _UNIT) at the + inner text's start and end for mapping the boundaries into the + translation, the exact source syntax around the text ("[" / "](url)", + "**" / "**", ...) and the source text itself, used as the fallback when + the mapped slice comes out empty (better an untranslated label than a + broken "[](url)").""" w_start: int w_end: int @@ -206,54 +209,58 @@ def _locate(source: str, needle: str, cursor: int) -> int: def _linked_block( source: str, kids: list, cursor: int, strip_alert: bool ) -> tuple[Span, str] | None: - """A whole-block segment for an inline of plain text and prose links: - (Span, wire text) with the links as marks, or None when the block has - any other shape — the caller then falls back to per-run segments. + """A whole-block segment for an inline of plain text, prose links and + paired text formatting (strong/em/s): (Span, wire text) with the links + and formatting as marks, or None when the block has any other shape — + the caller then falls back to per-run segments. - The block crosses the wire as one prose piece, link texts inline, so a - translation that inflects or reorders around a link stays coherent; - join re-inserts the link markdown at weight-mapped positions. The - source span is located piece by piece and verified by reconstruction; - anything not byte-exact (entities, escapes, an odd link tail) bails to - the fallback. + The block crosses the wire as one prose piece, link texts and formatted + text inline (the model is never shown any Markdown — it mangles it), + so a translation that inflects or reorders around them stays coherent; + join re-inserts the link/formatting syntax at weight-mapped positions. + The source span is located piece by piece and verified by + reconstruction; anything not byte-exact (entities, escapes, an odd + link tail) bails to the fallback. """ - pieces: list[tuple[str, bool]] = [] # (text, is_link); plain pieces alternate with links + pieces: list[tuple[str, str]] = [] # (text, mark): "" plain, "link", else the delimiter buf: list[str] = [] # current plain piece - link: list[str] | None = None # current link's text parts + link: list[str] | None = None # current mark's text parts + mark_kind = "" # the current mark's opener ("link" or the delimiter) for tok in kids: - if tok.type == "link_open": + if tok.type in ("link_open", "strong_open", "em_open", "s_open"): if link is not None or tok.markup == "autolink": return None if buf: - pieces.append(("".join(buf), False)) + pieces.append(("".join(buf), "")) buf = [] link = [] - elif tok.type == "link_close": - if link is None: + mark_kind = "link" if tok.type == "link_open" else tok.markup + elif tok.type in ("link_close", "strong_close", "em_close", "s_close"): + if link is None or ("link" if tok.type == "link_close" else tok.markup) != mark_kind: return None inner = "".join(link) if not _LETTER.search(inner): return None - pieces.append((inner, True)) + pieces.append((inner, mark_kind)) link = None elif tok.type in ("text", "softbreak"): (link if link is not None else buf).append( "\n" if tok.type == "softbreak" else tok.content ) - else: # emphasis, code, images, HTML, footnote refs: run boundaries + else: # code, images, HTML, footnote refs: run boundaries return None if link is not None: return None # unbalanced (the parser should not do this) if buf: - pieces.append(("".join(buf), False)) - if not any(is_link for _, is_link in pieces): + pieces.append(("".join(buf), "")) + if not any(mark for _, mark in pieces): return None if strip_alert and pieces and not pieces[0][1]: # A GFM alert marker leading the blockquote's first paragraph is # syntax; strip it from the wire text (it stays out of the span). first = _ALERT.sub("", pieces[0][0], count=1) if first.strip(): - pieces[0] = (first, False) + pieces[0] = (first, "") else: pieces.pop(0) if not pieces: @@ -276,29 +283,35 @@ def _linked_block( span_start, span_end = located[0][0], located[-1][1] marks: list[Mark] = [] offset = 0 # raw (pre-strip) plain-text offset of the current piece - for i, ((text_, is_link), (s, e)) in enumerate(zip(pieces, located)): - if not is_link: + for i, ((text_, kind), (s, e)) in enumerate(zip(pieces, located)): + if not kind: offset += len(text_) continue # The syntax around the text: the gap between pieces goes to the - # link on its left as post (so between two links the whole "](u)[" - # is the first's post); a block-leading link takes the byte in - # front of its text ("["), a block-trailing one the scanned tail. + # mark on its left as post (so between two marks the whole "](u)[" + # or "**" is the first's post); a block-leading mark takes its + # opener in front of its text ("[" or the delimiter), a + # block-trailing one the scanned link tail or the close delimiter. if i == 0: - if s == 0: + opener = "[" if kind == "link" else kind + if s < len(opener) or source[s - len(opener):s] != opener: return None - pre, span_start = source[s - 1:s], s - 1 + pre, span_start = opener, s - len(opener) elif pieces[i - 1][1]: - pre = "" # the previous link's post covers the whole gap + pre = "" # the previous mark's post covers the whole gap else: pre = source[located[i - 1][1]:s] if i + 1 < len(pieces): post = source[e:located[i + 1][0]] - else: + elif kind == "link": m = _LINK_TAIL.match(source, e) if m is None: return None post, span_end = m.group(), m.end() + else: + if source[e:e + len(kind)] != kind: + return None + post, span_end = kind, e + len(kind) ps = min(max(offset - lead, 0), len(wire)) pe = min(max(offset + len(text_) - lead, 0), len(wire)) if pe <= ps: @@ -309,8 +322,8 @@ def _linked_block( # real risk is the guessed tail of a trailing link). rec: list[str] = [] mi = 0 - for text_, is_link in pieces: - if is_link: + for text_, kind in pieces: + if kind: mark = marks[mi] mi += 1 rec += [mark.pre, text_, mark.post] @@ -326,8 +339,9 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]: translate, their source spans in ``text`` for splicing the translations back, and per-segment translation context. - A block of plain text and prose links becomes ONE segment (link texts - inline, in context), the links recorded as marks on its Span for + A block of plain text, prose links and paired formatting (strong/em/s) + becomes ONE segment (link/formatted text inline, in context, Markdown + stripped), the links and formatting recorded as marks on its Span for weight-mapped re-insertion in join. Other blocks split into text runs at markup boundaries; runs containing {...} spans are carved further — the braces stay out of the wire text. A run that cannot be located @@ -445,14 +459,17 @@ def _place_marks(translation: str, weight: int, marks: list[Mark]) -> str | None x2 = bounds[min(round(mark.w_end / weight * total), total)] x1 = max(x1, cur) # monotonic: never before the previous mark's end x2 = max(x2, x1) - # The slice ends at the next unit's start, so the whitespace before - # that unit is inside it — but it belongs BETWEEN the link and the - # following word, not in the link text: strip it from the link and - # leave it for the following slice (cursor stays ahead of it). + # The slice ends at the next unit's start, so the whitespace and + # punctuation before that unit is inside it — but it belongs + # BETWEEN the mark and the following word, not in the inner text: + # end the inner text at its last unit and leave the rest for the + # following slice (the cursor stays ahead of it). raw = translation[x1:x2] - inner = raw.strip() or mark.inner + units = list(_UNIT.finditer(raw)) + inner_end = x1 + units[-1].end() if units else x1 + inner = translation[x1:inner_end].strip() or mark.inner out += [translation[cur:x1], mark.pre, inner, mark.post] - cur = x2 - (len(raw) - len(raw.rstrip())) + cur = inner_end out.append(translation[cur:]) return "".join(out)