Skip npm's security audit on install (registry endpoint stalls for minutes)

docs/localization.md

@@ -88,6 +88,10 @@ Region tags normalize to their base subtag (`fi-FI` → `fi`).
 ### Rendering
 
 - The translated Markdown goes through the same `markdown.render` pipeline.
+- Section anchors (`#hash` ids on h1/h2 headings) stay in the original
+  language: render(anchors_from=...) pins the translated render's heading
+  ids to the original text's slugs, matched by heading position, so links
+  to sections don't break across languages.
 - Navigation/sidebar titles come from the translation's title map, with
   per-node fallback to the original title (a partially translated tree must
   still render).
@@ -398,9 +402,14 @@ verbatim source substring — entity-decoded text, backslash escapes — is
 skipped and stays in the original language), and the returned translations
 are swapped in by offset. Markup corruption is therefore impossible by
 construction; the failure modes that remain are a wrong segment count, an
-empty segment, or markup injected INTO a segment (a `<br>` in a title
-translation would splice live HTML) — each returned segment must parse as
-pure prose, or the whole result is dropped and logged, and the (lang, key)
+empty segment, markup injected INTO a segment (a `<br>` in a title
+translation would splice live HTML), or a line that would start a new
+block where the segment lands (a ``` or ::: fence line would eat the rest
+of the block it splices into, closing fence included — segments are
+inline prose, so `pure_prose` alone cannot see this) — each returned
+segment must parse as
+pure prose with no block-starting line or blank line, or the whole result
+is dropped and logged, and the (lang, key)
 pair is skipped for the rest of the server run (generation is
 near-deterministic, so an immediate retry would re-fail; the fragment stays
 pending and gets another chance on restart or `DELETE /_api/translations`).
@@ -451,14 +460,25 @@ stripped before the result goes back.
 
 The same client-side enforcement covers markup bleed as a CLASS, not per
 artifact: `<` is the prose/markup boundary on the wire and never appears in
-a segment in either direction. Source pieces containing `<` are never
-dispatched (they stay in the original language — segments.py), and the
+a segment in either direction. A literal `<` in the source text (`<1MB` is
+text, not markup — a tag needs a letter or `/!?`) crosses encoded as the
+fullwidth `＜` and is decoded on return, before the result is validated and
+spliced (segments.py) — the wire itself still never carries `<`, and the
 reference client cuts the model's output at the first `<`
 (scripts/translator.py) — echoed language tags, stray `<br>`s and any
 future variant are one handled case. (The cut is post-decode, not a
 generation stop string: Seed-X opens every generation with its `<s>`
 framing token, which would trip a `<` stop immediately.)
 
+Server-side, a second layer covers what the inline parser cannot: ASCII
+punctuation that is plain prose on the wire but Markdown syntax in the
+splice context — quotes (a translated `"` would close the quoted image
+title it lands in), brackets (alt texts, re-inserted link texts), `|` in
+table rows, `\` escapes. Rather than rejecting such results, `join` swaps
+them for Unicode look-alikes before splicing (`_NEUTRAL` in
+segments.py — curly quotes, fullwidth brackets; the renderer's
+typographer curls straight quotes anyway).
+
 Short fragments get more than a bare prompt: each segment may carry its
 surround in `Job.contexts` — a title carries the article's opening prose
 (its own block is just the title word), a segment carved out of a larger

frontend/.gitignore

@@ -37,3 +37,6 @@ __screenshots__/
 
 # Playwright browser downloads (if ever installed locally)
 .pw-browsers/
+
+# npm project config (audit/fund off: the audit endpoint stalls installs)
+!.npmrc

frontend/.npmrc

@@ -0,0 +1,2 @@
+audit=false
+fund=false

pagerite/api.py

@@ -503,6 +503,14 @@ async def editor_ws(ws: WebSocket) -> None:
                         # The title is injected as h1 when the markdown has
                         # none; the editor's title field edits live-preview.
                         title=msg.get("title") or (node.title if node else ""),
+                        # Pin section anchors to the original language so the
+                        # preview of a translation matches the served page
+                        # (no-op when the previewed markdown is the original).
+                        anchors_from=(
+                            (node_markdown(data, node) or "", node.title)
+                            if node
+                            else None
+                        ),
                     )
                     await ws.send_json(
                         {

pagerite/markdown.py

@@ -402,7 +402,9 @@ def _heading_ids(state) -> None:
     its self-link is ``href=""`` (back to the top of the page). An
     author-set `{#id}` always wins; auto ids slugify the heading text
     (python-slugify, mirroring the editor's slugify.js) and dedupe with
-    -2/-3 suffixes per render. Headings that already contain a link are
+    -2/-3 suffixes per render — unless env["anchor_ids"] presets them, as
+    render(anchors_from=...) does for translated pages so section URLs
+    stay in the original language. Headings that already contain a link are
     ``data-line`` records the heading's markdown source line (0-based, after
     undoing the render(title=...) injection offset via ``env``) — the page
     editor uses it for section pens and piecewise-linear scroll sync.
@@ -443,13 +445,23 @@ def _heading_ids(state) -> None:
     if len(heads) < ANCHOR_MIN_HEADINGS:
         return
     seen: set[str] = set()
-    for i, token in heads:
+    preset = state.env.get("anchor_ids")
+    for k, (i, token) in enumerate(heads):
         inline = tokens[i + 1]
         hid = token.attrGet("id")
         if not isinstance(hid, str) or not hid:
+            if preset is not None and k < len(preset):
+                # Translated render: the original language's slug, matched
+                # by heading position (a translation never adds, removes or
+                # reorders headings; a patched one that does falls back to
+                # slugging its own text past the end of the list).
+                base = preset[k]
+            else:
                 # Slug the visible text, not the raw markdown (`## [a](url)`).
                 text = "".join(
-                c.content for c in inline.children if c.type in ("text", "code_inline")
+                    c.content
+                    for c in inline.children
+                    if c.type in ("text", "code_inline")
                 )
                 base = slugify(text) or "section"
             hid, n = base, 2
@@ -463,6 +475,36 @@ def _heading_ids(state) -> None:
         wrap(i, token, f"#{hid}")
 
 
+def anchor_ids(text: str, title: str | None = None) -> list[str]:
+    """The section anchor ids of text, in heading order.
+
+    render(anchors_from=...) feeds these to _heading_ids via
+    env["anchor_ids"], pinning a translated render's anchors to the
+    original language's slugs. The selection mirrors _heading_ids exactly
+    (the same md instance assigns the ids during this parse, author-set
+    {#id} included as-is); the in-body title h1 is excluded.
+    """
+    if title and not has_h1(text):
+        text = f"# {title}\n\n{text}"
+    tokens = md.parse(text, {"page_path": ""})
+    first_h1 = next(
+        (
+            i
+            for i, t in enumerate(tokens)
+            if t.type == "heading_open" and t.tag == "h1" and t.level == 0
+        ),
+        None,
+    )
+    return [
+        t.attrGet("id")
+        for i, t in enumerate(tokens)
+        if t.type == "heading_open"
+        and t.tag in ("h1", "h2")
+        and t.level == 0
+        and i != first_h1
+    ]
+
+
 def make_md(*, verbatim: bool = False) -> MarkdownIt:
     """A fully configured parser. The module-level ``md`` (below) is the
     render instance; ``verbatim=True`` builds the segmentation instance for
@@ -618,12 +660,16 @@ def render(
     created: datetime | None = None,
     modified: datetime | None = None,
     title: str | None = None,
+    anchors_from: tuple[str, str] | None = None,
 ) -> Rendered:
     """Render Markdown text to the article body's HTML and layout flags.
 
     ``title`` injects a ``# {title}`` line at the top when the markdown has
     no h1 of its own, so the implicit page title goes through the exact
     same pipeline as an explicit one (first-h1 anchor treatment included).
+    ``anchors_from`` is the (markdown, title) of the ORIGINAL language when
+    rendering a translation: section anchors are pinned to its slugs so
+    localized pages keep the original #hash URLs.
 
     The top-level blocks are grouped into column segments: boundary blocks
     (h1/h2 headings, .wide — see _is_boundary) are rendered bare, the runs
@@ -641,6 +687,8 @@ def render(
     right after the article's h1.
     """
     env = {"page_path": page_path, "line_offset": 0}
+    if anchors_from is not None:
+        env["anchor_ids"] = anchor_ids(*anchors_from)
     if title and not has_h1(text):
         text = f"# {title}\n\n{text}"
         # The injected title shifts source lines by two; _heading_ids

pagerite/segments.py

@@ -20,8 +20,13 @@ each segment's source span was located at dispatch (``split``), and
 ``join`` swaps in the translations. Markup therefore cannot break — it
 never left the server. A returned segment must still be pure prose itself
 (the model could inject markup INTO a segment); anything else — count
-mismatch, empty segment, markup tokens — rejects the whole result and the
-fragment stays pending.
+mismatch, empty segment, markup tokens, a line that would start a new
+block (a ``` or ::: fence would eat the rest of the block it lands in) —
+rejects the whole result and the
+fragment stays pending. Punctuation that is prose on the wire but syntax
+in the splice context (quotes in a title attribute, brackets in an alt
+text, "|" in a table row) is not worth a rejection either: it is swapped
+for Unicode look-alikes (``_NEUTRAL``) before splicing.
 
 A block of plain text, prose links and paired text formatting
 (strong/em/s) crosses as ONE segment — link texts and formatted text
@@ -42,9 +47,11 @@ snippets that don't fit together. Blocks with any other inline markup
 
 Locating is best effort: a run that is not a verbatim source substring
 (entity-decoded text, backslash escapes) is skipped — it simply stays in
-the original language. So is any piece containing "<": "<" is the
-prose/markup boundary on the wire — translators cut their output there,
-so such pieces could not survive the round trip.
+the original language. A literal "<" in prose ("<1MB") is text, not
+markup, but cannot cross as-is — "<" is the prose/markup boundary on the
+wire, translators cut their output there — so it crosses encoded as the
+fullwidth "＜" (``_encode``) and ``join`` decodes it back before
+validating and splicing.
 """
 
 import bisect
@@ -69,6 +76,38 @@ _ALERT = re.compile(r"^\[![A-Za-z]+\][ \t]*")
 #: (inline attrs are consumed by the parser; a lone {dates} is not).
 _BRACES = re.compile(r"\{[^{}\n]*\}")
 
+
+def _encode(text: str) -> str:
+    """Wire form of a segment or context: a literal "<" as fullwidth "＜".
+
+    A "<" in prose is text, not markup ("<1MB" — a tag needs a letter or
+    /!?), but "<" is the prose/markup boundary on the wire (translators
+    cut output at the first "<", scripts/translator.py), so it cannot
+    cross as-is. join decodes it back before the pure_prose check and
+    splicing — anything tag-like the model may have formed around it is
+    still rejected there.
+    """
+    return text.replace("<", "＜")
+
+#: ASCII punctuation that is plain prose to the inline parser (so
+#: pure_prose cannot catch it) but Markdown SYNTAX in a splice context:
+#: quotes close a quoted image/link title, brackets the [...] of alt and
+#: re-inserted link texts, "|" splits a table row, and "\" escapes the
+#: character after it (a trailing one eats a title's closing quote).
+#: Neutralized to Unicode look-alikes (join), which Markdown treats as
+#: plain text everywhere — the quotes are curled the way typographer=True
+#: renders them anyway.
+_NEUTRAL = str.maketrans(
+    {
+        '"': "”",
+        "'": "’",
+        "[": "［",
+        "]": "］",
+        "\\": "＼",
+        "|": "│",
+    }
+)
+
 #: A link's tail after its text: "](dest)", "](dest \"title\")", "][ref]",
 #: "[]" or a bare "]" (shortcut reference); the destination may nest one
 #: level of parens. Best effort — a mis-scan fails the span-reconstruction
@@ -273,7 +312,7 @@ def _linked_block(
     raw = "".join(text for text, _ in pieces)
     lead = len(raw) - len(raw.lstrip())
     wire = raw.strip()
-    if not _LETTER.search(wire) or "<" in wire or _BRACES.search(wire):
+    if not _LETTER.search(wire) or _BRACES.search(wire):
         return None
     # Locate each piece verbatim, in order; the source slices between the
     # located pieces are then the link syntax, exact by construction.
@@ -338,7 +377,7 @@ def _linked_block(
             rec.append(text_)
     if source[span_start:span_end] != "".join(rec):
         return None
-    return Span(span_start, span_end, _weight(wire), marks), wire
+    return Span(span_start, span_end, _weight(wire), marks), _encode(wire)
 
 
 def split(text: str) -> tuple[list[Span], list[str], list[str]]:
@@ -367,10 +406,8 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
     def emit(run: str, at: int, ctx: str) -> None:
         """Carve {...} spans out of the located run; emit the prose pieces,
         stripped — padding whitespace stays in the template, off the wire.
-        Pieces containing "<" are never emitted: translators cut output at
-        the first "<" (the prose/markup boundary, scripts/translator.py),
-        so such a piece could not survive the round trip — it stays in the
-        original language instead."""
+        A literal "<" crosses encoded (``_encode``): it is text, not
+        markup, but the wire keeps "<" as the prose/markup boundary."""
         pieces = []
         pos = 0
         for m in _BRACES.finditer(run):
@@ -380,10 +417,10 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
         for p0, p1 in pieces:
             raw = run[p0:p1]
             piece = raw.strip()
-            if _LETTER.search(piece) and "<" not in piece:
+            if _LETTER.search(piece):
                 start = at + p0 + (len(raw) - len(raw.lstrip()))
                 spans.append(Span(start, start + len(piece), 0, []))
-                segments.append(piece)
+                segments.append(_encode(piece))
                 contexts.append(ctx)
 
     tokens = _MD.parse(text)
@@ -409,7 +446,7 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
                 cursor = span.end
                 continue
             runs = _runs(kids)
-            block = _block_text(kids).strip()
+            block = _encode(_block_text(kids).strip())
             if alert and runs:
                 run = _ALERT.sub("", runs[0], count=1)
                 if _LETTER.search(run):
@@ -417,7 +454,7 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
                 else:
                     runs.pop(0)
             for run in runs:
-                ctx = block if block and run.strip() != block else ""
+                ctx = block if block and _encode(run.strip()) != block else ""
                 pos = _locate(text, run, cursor)
                 if pos != -1:
                     emit(run, pos, ctx)
@@ -435,6 +472,22 @@ def split(text: str) -> tuple[list[Span], list[str], list[str]]:
     return spans, segments, contexts
 
 
+#: Block-level Markdown a translation must not introduce: a segment is
+#: spliced INSIDE a block of the fragment, so a line starting a heading,
+#: quote, list, code/container fence or a setext/thematic-break underline
+#: would break the fragment's block structure — a ``` or ::: line eats the
+#: rest of the fence it lands in, closing fence included. pure_prose only
+#: parses inline and lets such lines through as softbreak prose, so join
+#: rejects them here. Blank lines split the host block and are rejected
+#: too (a faithful translation of a single block has none).
+_BLOCK = re.compile(
+    r"^[ \t]*(?:#{1,6}(?:[ \t]|$)|>[ \t]?|(?:[-+*]|\d{1,9}[.)])[ \t]|`{3,}|~{3,}|:{3,}(?:[ \t]|$)"
+    r"|-(?:[ \t]*-){2,}[ \t]*$|=[ =]*$|_(?:[ \t]*_){2,}[ \t]*$)",
+    re.M,
+)
+_BLANK = re.compile(r"\n[ \t]*\n")
+
+
 def pure_prose(text: str) -> bool:
     """True when the text parses as nothing but prose (text and softbreak
     tokens) — the acceptance test for a translated segment: the model may
@@ -597,17 +650,37 @@ def _place_marks(translation: str, weight: int, marks: list[Mark]) -> str | None
 
 def join(original: str, spans: list[Span], texts: list[str]) -> str | None:
     """Splice translated segments back into the original fragment; None on
-    any validation failure (count mismatch, empty or non-prose segment) —
-    the caller drops the result and the fragment stays pending. Segments
-    with marks (a block that crossed as one piece) get their links
-    re-inserted at weight-mapped positions after the prose check."""
+    any validation failure (count mismatch, empty, non-prose or
+    block-structure segment) — the caller drops the result and the fragment
+    stays pending. Segments with marks (a block that crossed as one piece)
+    get their links re-inserted at weight-mapped positions after the prose
+    check.
+
+    Markdown-significant ASCII punctuation that pure_prose cannot see
+    (plain text inline, syntax in the splice context — quoted titles, alt
+    and link texts, table rows) is neutralized to Unicode look-alikes
+    (``_NEUTRAL``) before splicing and mark placement (the swap is
+    char-for-char, so unit alignment is unaffected); lines that would
+    start a new block (a heading, a ``` or ::: fence — they would eat the
+    rest of the block/fence they land in) reject the result outright
+    (``_BLOCK``, ``_BLANK``)."""
     if len(texts) != len(spans):
         return None
     out: list[str] = []
     cursor = 0
     for span, translation in zip(spans, texts):
-        if not translation.strip() or not pure_prose(translation):
+        # Decode the wire form ("＜" back to "<") first: pure_prose then
+        # validates exactly what gets spliced — a "＜" the model formed
+        # into anything tag-like is markup and rejects the result.
+        translation = translation.replace("＜", "<")
+        if (
+            not translation.strip()
+            or not pure_prose(translation)
+            or _BLOCK.search(translation)
+            or _BLANK.search(translation.strip())
+        ):
             return None
+        translation = translation.translate(_NEUTRAL)
         if span.marks:
             translation = _place_marks(translation, span.weight, span.marks)
             if translation is None:

pagerite/translate.py

@@ -277,16 +277,20 @@ class Dispatcher:
             job = None
             spans: list[Span] = []
             original = ""
+            # Titles before articles — across languages too, so every menu
+            # is named before any article body is worked on (a page's name
+            # is its most visible string). pending_items emits in menu
+            # order, a page's title before its chunks; filtering by kind
+            # keeps that stable order within each kind.
+            pending = {lang: pending_items(self.data, lang) for lang in sorted(langs)}
+            for kind in ("title", "chunk"):
                 for lang in sorted(langs):
-                # Titles first: a page's name in the menu is its most
-                # visible string (stable: menu order kept within each kind).
-                for item in sorted(
-                    pending_items(self.data, lang), key=lambda it: it.kind != "title"
+                    for item in pending[lang]:
+                        if (
+                            item.kind != kind
+                            or (lang, item.key) in inflight
+                            or (lang, item.key) in self.validation_failures
                         ):
-                    if (lang, item.key) in inflight or (
-                        lang,
-                        item.key,
-                    ) in self.validation_failures:
                             continue
                         spans, texts, contexts = split(item.text)
                         if not texts:
@@ -307,6 +311,8 @@ class Dispatcher:
                         break
                     if job is not None:
                         break
+                if job is not None:
+                    break
             if job is None:
                 continue
             state.inflight = (job.lang, job.key)  # before the await: no double-assign

pagerite/views.py

@@ -783,7 +783,11 @@ def page_content(
     node = resolve(menu, path)[-1]
     content = node_markdown(data, node) or ""
     title = node.title
+    # The original text pins the section anchors: on a translated page the
+    # heading slugs (and thus #hash URLs) stay in the original language.
+    anchors_from = None
     if translation:
+        anchors_from = (content, title)
         if translation.markdown is not None:
             content = translation.markdown
         title = (
@@ -793,7 +797,9 @@ def page_content(
         )
     # The title is injected into the markdown (as # title when it has no
     # h1 of its own), so title and content render as one article.
-    rendered = render(content, path, node.created, node.modified, title=title)
+    rendered = render(
+        content, path, node.created, node.modified, title=title, anchors_from=anchors_from
+    )
     # Long articles get .multicol: the article column cap lifts (see the
     # #content grid in pagerite.css) and the .cols segments lay out in at
     # most two columns. The html is already segmented by render() — the