AGENTS.md

@@ -15,7 +15,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke
   - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md).
   - `i18n.py` — language selection, translation assembly (chunks + patches) and translated-edit recording (user patches, per-language title overrides, refresh).
   - `translate.py` — translator service protocol (msgspec structs), the connected-client `Dispatcher` (job pipeline, result validation) and pending/store core for the `/_translate/{key}` WebSocket (docs/localization.md); app.py only registers the route.
-  - `segments.py` — the translation round trip: fragments split into pure-prose wire segments (via markdown.make_md's verbatim parser; link-carrying blocks stay whole, link texts inline) and translations spliced back by source offset, link markdown re-inserted at weight-mapped positions (docs/localization.md).
+  - `segments.py` — the translation round trip: fragments split into pure-prose wire segments (via markdown.make_md's verbatim parser; link- and formatting-carrying blocks stay whole, link/formatted texts inline, Markdown stripped) and translations spliced back by source offset, link/formatting markdown re-inserted at weight-mapped positions (docs/localization.md).
   - `migrations.py` — kanta migrations (`migrate_vN`); ALL schema/storage upgrades live here (raw state dict before struct decoding), never in the app lifespan: v1 moves legacy in-db file blobs to the on-disk store and rebuilds the legacy flat `pages` as the menu tree, v2 rewrites `/_f/{hash}.ext` image links to the extension-less form, backfills AVIF/WebP/JPEG derivatives on disk and drops the obsolete `version` field.
   - `markdown.py` — markdown-it-py renderer.
   - `views.py` — shared page layout and rendering; theme/user-font resolution across `THEME_DIRS` / `FONT_DIRS` (cwd, site, platform data roots, then built-in `pagerite/themes/`, see `docs/themes-and-assets.md`).

docs/localization.md

@@ -376,13 +376,14 @@ fragment is parsed with the project's own markdown-it setup
 tasklist label wrapping, so token text stays byte-identical to the source)
 and split into the runs a model may touch: paragraph/heading/table-cell text
 (merged across soft line breaks), image alt texts and captions, footnote
-bodies. A block of plain text and inline **links stays whole** — link texts
-cross inline, in sentence context (see below). Everything else never leaves
-the server: code spans and
+bodies. A block of plain text, inline **links and paired text formatting**
+(strong/em/s) **stays whole** — link and formatted texts cross inline, in
+sentence context, with the Markdown stripped (see below). Everything else
+never leaves the server: code spans and
 fences, URLs and autolinks, link/image *destinations*, `{...}` spans
 (placeholders like `{dates}` as well as attrs), reference and footnote
-labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and all
-markup punctuation (`*`, `|`, `[]()`, `:::`), which is a run boundary.
+labels, container fences, GFM alert markers (`[!NOTE]`), raw HTML — and the
+remaining markup punctuation (`|`, `:::`), which is a run boundary.
 Chunks with no segments (a lone `{dates}`, container fences, pure
 code/HTML) are never dispatched at all (`needs_translation`); every
 language renders them from the original chunk. Each segment is accompanied
@@ -404,24 +405,29 @@ near-deterministic, so an immediate retry would re-fail; the fragment stays
 pending and gets another chance on restart or `DELETE /_api/translations`).
 `Data.trans` therefore only ever holds clean translated Markdown.
 
-Link-carrying blocks are the one place a segment is not spliced verbatim:
-a label translated apart from its sentence comes back grammatically
-incompatible with it (case government, particles, word order), so the
-block crosses whole and the server re-inserts the link markdown into the
-translated block. The boundaries are found by **text processing alone** —
+Link- and formatting-carrying blocks are the one place a segment is not
+spliced verbatim: a label translated apart from its sentence comes back
+grammatically incompatible with it (case government, particles, word
+order), and shown the Markdown the model mangles it (Seed-X dropped the
+`**` and the glued-on colon in `**Pagerite**: …`), so the block crosses
+whole — all Markdown stripped — and the server re-inserts the link and
+formatting syntax into the translated block. The boundaries are found by
+**text processing alone** —
 markers on the wire are hopeless (an earlier sentinel-masking design let
 the model see and mangle exactly that punctuation: Seed-X renumbered the
-tokens and turned `![` into `¡¡…!!`). Each link's weight ratio in the
+tokens and turned `![` into `¡¡…!!`). Each mark's weight ratio in the
 source block (word units before its text boundaries over the block total;
 CJK ideographs count as one unit each, kana runs as one — no spaces to
-count words by) is applied to the translation's units. Placement is
-approximate and drift accumulates across several links in one block — the
+count words by) is applied to the translation's units; the inner text ends
+at its last unit, so punctuation and whitespace between the mark and the
+next word stay outside it. Placement is
+approximate and drift accumulates across several marks in one block — the
 accepted trade: better a coherent sentence with a slightly shifted link
 than separately translated snippets that don't fit together. A boundary
-that maps to an empty slice degrades to the source link text rather than
-emitting a broken `[](url)`. Blocks mixing in any other inline markup
-(emphasis, code spans, images) don't qualify and still split into runs at
-those boundaries.
+that maps to an empty slice degrades to the source text rather than
+emitting a broken `[](url)` or `**`. Blocks mixing in any other inline
+markup (code spans, images, raw HTML) don't qualify and still split into
+runs at those boundaries.
 
 Punctuation is the translator's own job: Seed-X tends to "finish" short
 labels (titles, nav items) with a comma or period the source never had.

pagerite/segments.py

@@ -23,18 +23,20 @@ never left the server. A returned segment must still be pure prose itself
 mismatch, empty segment, markup tokens — rejects the whole result and the
 fragment stays pending.
 
-A block of plain text and prose links crosses as ONE segment — link texts
-inline, in sentence context — because a label translated apart from its
-sentence comes back grammatically incompatible with it (case government,
-particles, word order). ``join`` re-inserts the link markdown into the
-translated block at weight-mapped positions (``_place_marks``): no markers
-on the wire (sentinels never survived the model — they got renumbered and
-mangled), the boundaries are found by text processing alone — each link's
-word/CJK-char weight ratio in the source applied to the translation's
-units. Placement is approximate and CJK-safe: better a coherent sentence
-with a slightly shifted link than separately translated snippets that
-don't fit together. Blocks with any other inline markup (emphasis, code,
-images) still split into runs at those boundaries.
+A block of plain text, prose links and paired text formatting
+(strong/em/s) crosses as ONE segment — link texts and formatted text
+inline, in sentence context, with the Markdown stripped (the model
+mangles it: sentinels get renumbered, ``**`` gets dropped or moved) —
+because a label translated apart from its sentence comes back
+grammatically incompatible with it (case government, particles, word
+order). ``join`` re-inserts the link/formatting markdown into the
+translated block at weight-mapped positions (``_place_marks``): no
+markers on the wire, the boundaries are found by text processing alone —
+each mark's word/CJK-char weight ratio in the source applied to the
+translation's units. Placement is approximate and CJK-safe: better a
+coherent sentence with a slightly shifted link than separately translated
+snippets that don't fit together. Blocks with any other inline markup
+(code, images, HTML) still split into runs at those boundaries.
 
 Locating is best effort: a run that is not a verbatim source substring
 (entity-decoded text, backslash escapes) is skipped — it simply stays in
@@ -82,12 +84,13 @@ _UNIT = re.compile(
 
 
 class Mark(NamedTuple):
-    """One inline link inside a whole-block segment: the source weight
-    (unit count, see _UNIT) at the link text's start and end for mapping
-    the boundaries into the translation, the exact source syntax around
-    the text ("[" / "](url)" etc.) and the source text itself, used as the
-    fallback when the mapped slice comes out empty (better an untranslated
-    label than a broken "[](url)")."""
+    """One inline link or paired formatting (strong/em/s) inside a
+    whole-block segment: the source weight (unit count, see _UNIT) at the
+    inner text's start and end for mapping the boundaries into the
+    translation, the exact source syntax around the text ("[" / "](url)",
+    "**" / "**", ...) and the source text itself, used as the fallback when
+    the mapped slice comes out empty (better an untranslated label than a
+    broken "[](url)")."""
 
     w_start: int
     w_end: int
@@ -206,54 +209,58 @@ def _locate(source: str, needle: str, cursor: int) -> int:
 def _linked_block(
     source: str, kids: list, cursor: int, strip_alert: bool
 ) -> tuple[Span, str] | None:
-    """A whole-block segment for an inline of plain text and prose links:
-    (Span, wire text) with the links as marks, or None when the block has
-    any other shape — the caller then falls back to per-run segments.
+    """A whole-block segment for an inline of plain text, prose links and
+    paired text formatting (strong/em/s): (Span, wire text) with the links
+    and formatting as marks, or None when the block has any other shape —
+    the caller then falls back to per-run segments.
 
-    The block crosses the wire as one prose piece, link texts inline, so a
-    translation that inflects or reorders around a link stays coherent;
-    join re-inserts the link markdown at weight-mapped positions. The
-    source span is located piece by piece and verified by reconstruction;
-    anything not byte-exact (entities, escapes, an odd link tail) bails to
-    the fallback.
+    The block crosses the wire as one prose piece, link texts and formatted
+    text inline (the model is never shown any Markdown — it mangles it),
+    so a translation that inflects or reorders around them stays coherent;
+    join re-inserts the link/formatting syntax at weight-mapped positions.
+    The source span is located piece by piece and verified by
+    reconstruction; anything not byte-exact (entities, escapes, an odd
+    link tail) bails to the fallback.
     """
-    pieces: list[tuple[str, bool]] = []  # (text, is_link); plain pieces alternate with links
+    pieces: list[tuple[str, str]] = []  # (text, mark): "" plain, "link", else the delimiter
     buf: list[str] = []  # current plain piece
-    link: list[str] | None = None  # current link's text parts
+    link: list[str] | None = None  # current mark's text parts
+    mark_kind = ""  # the current mark's opener ("link" or the delimiter)
     for tok in kids:
-        if tok.type == "link_open":
+        if tok.type in ("link_open", "strong_open", "em_open", "s_open"):
             if link is not None or tok.markup == "autolink":
                 return None
             if buf:
-                pieces.append(("".join(buf), False))
+                pieces.append(("".join(buf), ""))
                 buf = []
             link = []
-        elif tok.type == "link_close":
-            if link is None:
+            mark_kind = "link" if tok.type == "link_open" else tok.markup
+        elif tok.type in ("link_close", "strong_close", "em_close", "s_close"):
+            if link is None or ("link" if tok.type == "link_close" else tok.markup) != mark_kind:
                 return None
             inner = "".join(link)
             if not _LETTER.search(inner):
                 return None
-            pieces.append((inner, True))
+            pieces.append((inner, mark_kind))
             link = None
         elif tok.type in ("text", "softbreak"):
             (link if link is not None else buf).append(
                 "\n" if tok.type == "softbreak" else tok.content
             )
-        else:  # emphasis, code, images, HTML, footnote refs: run boundaries
+        else:  # code, images, HTML, footnote refs: run boundaries
             return None
     if link is not None:
         return None  # unbalanced (the parser should not do this)
     if buf:
-        pieces.append(("".join(buf), False))
-    if not any(is_link for _, is_link in pieces):
+        pieces.append(("".join(buf), ""))
+    if not any(mark for _, mark in pieces):
         return None
     if strip_alert and pieces and not pieces[0][1]:
         # A GFM alert marker leading the blockquote's first paragraph is
         # syntax; strip it from the wire text (it stays out of the span).
         first = _ALERT.sub("", pieces[0][0], count=1)
         if first.strip():
-            pieces[0] = (first, False)
+            pieces[0] = (first, "")
         else:
             pieces.pop(0)
             if not pieces:
@@ -276,29 +283,35 @@ def _linked_block(
     span_start, span_end = located[0][0], located[-1][1]
     marks: list[Mark] = []
     offset = 0  # raw (pre-strip) plain-text offset of the current piece
-    for i, ((text_, is_link), (s, e)) in enumerate(zip(pieces, located)):
-        if not is_link:
+    for i, ((text_, kind), (s, e)) in enumerate(zip(pieces, located)):
+        if not kind:
             offset += len(text_)
             continue
         # The syntax around the text: the gap between pieces goes to the
-        # link on its left as post (so between two links the whole "](u)["
-        # is the first's post); a block-leading link takes the byte in
-        # front of its text ("["), a block-trailing one the scanned tail.
+        # mark on its left as post (so between two marks the whole "](u)["
+        # or "**" is the first's post); a block-leading mark takes its
+        # opener in front of its text ("[" or the delimiter), a
+        # block-trailing one the scanned link tail or the close delimiter.
         if i == 0:
-            if s == 0:
+            opener = "[" if kind == "link" else kind
+            if s < len(opener) or source[s - len(opener):s] != opener:
                 return None
-            pre, span_start = source[s - 1:s], s - 1
+            pre, span_start = opener, s - len(opener)
         elif pieces[i - 1][1]:
-            pre = ""  # the previous link's post covers the whole gap
+            pre = ""  # the previous mark's post covers the whole gap
         else:
             pre = source[located[i - 1][1]:s]
         if i + 1 < len(pieces):
             post = source[e:located[i + 1][0]]
-        else:
+        elif kind == "link":
             m = _LINK_TAIL.match(source, e)
             if m is None:
                 return None
             post, span_end = m.group(), m.end()
+        else:
+            if source[e:e + len(kind)] != kind:
+                return None
+            post, span_end = kind, e + len(kind)
         ps = min(max(offset - lead, 0), len(wire))
         pe = min(max(offset + len(text_) - lead, 0), len(wire))
         if pe <= ps:
@@ -309,8 +322,8 @@ def _linked_block(
     # real risk is the guessed tail of a trailing link).
     rec: list[str] = []
     mi = 0
-    for text_, is_link in pieces:
-        if is_link:
+    for text_, kind in pieces:
+        if kind:
             mark = marks[mi]
             mi += 1
             rec += [mark.pre, text_, mark.post]
@@ -326,8 +339,9 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
     translate, their source spans in ``text`` for splicing the translations
     back, and per-segment translation context.
 
-    A block of plain text and prose links becomes ONE segment (link texts
-    inline, in context), the links recorded as marks on its Span for
+    A block of plain text, prose links and paired formatting (strong/em/s)
+    becomes ONE segment (link/formatted text inline, in context, Markdown
+    stripped), the links and formatting recorded as marks on its Span for
     weight-mapped re-insertion in join. Other blocks split into text runs
     at markup boundaries; runs containing {...} spans are carved further —
     the braces stay out of the wire text. A run that cannot be located
@@ -445,14 +459,17 @@ def _place_marks(translation: str, weight: int, marks: list[Mark]) -> str | None
         x2 = bounds[min(round(mark.w_end / weight * total), total)]
         x1 = max(x1, cur)  # monotonic: never before the previous mark's end
         x2 = max(x2, x1)
-        # The slice ends at the next unit's start, so the whitespace before
-        # that unit is inside it — but it belongs BETWEEN the link and the
-        # following word, not in the link text: strip it from the link and
-        # leave it for the following slice (cursor stays ahead of it).
+        # The slice ends at the next unit's start, so the whitespace and
+        # punctuation before that unit is inside it — but it belongs
+        # BETWEEN the mark and the following word, not in the inner text:
+        # end the inner text at its last unit and leave the rest for the
+        # following slice (the cursor stays ahead of it).
         raw = translation[x1:x2]
-        inner = raw.strip() or mark.inner
+        units = list(_UNIT.finditer(raw))
+        inner_end = x1 + units[-1].end() if units else x1
+        inner = translation[x1:inner_end].strip() or mark.inner
         out += [translation[cur:x1], mark.pre, inner, mark.post]
-        cur = x2 - (len(raw) - len(raw.rstrip()))
+        cur = inner_end
     out.append(translation[cur:])
     return "".join(out)