Redesign translation overrides: keyed, structural, hash-anchored

Replace the old "patches" format (path:lang composite keys, ordered
hunk lists, text-anchored matching) with Data.overrides:
path -> lang -> LangEdits, keyed throughout so a save's database diff
touches only the edited chunks. The old "patches" key is ignored on
decode, discarding legacy data without a migration.

- Whole-paragraph additions/deletions are structural: a drop flag on
  the original chunk hash, and additions in their own dict anchored
  from the neighboring chunks' before/after (first live referrer wins),
  so they stay in place across retranslation and one-sided original
  edits.
- Within-paragraph edits (up to a full paragraph rewrite or split) are
  full-chunk replace patches applied by chunk hash alone: a
  retranslation is overridden wholesale, so user edits survive AI
  re-runs; editing the original changes the hash and orphans the patch.
  The old search-matching staleness gate is gone.
- Saving a translation on a page without original chunks is rejected
  (REST 400 / WS error); emptying the original afterwards renders the
  translation empty, with the orphaned overrides inert.
This commit is contained in:
2026-09-21 18:14:48 +00:00
parent cc2cc23b3b
commit 91a16f56c5
11 changed files with 500 additions and 236 deletions
+34 -30
View File
@@ -67,9 +67,12 @@ class Data(msgspec.Struct):
#: (nested, not tuple keys: msgspec's JSON serializer rejects them).
#: Also used for node titles (hash of the title text).
trans: dict[bytes, dict[str, str]] = {}
#: User override patches per article and language:
#: f"{path}:{lang}" -> ordered patches (see localization.md).
patches: dict[str, list[Patch]] = {}
#: User override edits per article and language:
#: path -> lang -> LangEdits (see localization.md) — keyed per original
#: chunk hash throughout, so a save's change diff touches only the
#: edited chunks. Replaced the old list-valued "patches" key (ignored
#: on decode, discarding that data — no migration).
overrides: dict[str, dict[str, LangEdits]] = {}
```
Notes:
@@ -88,13 +91,14 @@ Notes:
(keyed by chunk
hash, so a heavy edit silently drops the flag — acceptable and
self-healing).
- **Patch payloads stay inline** in `Patch.hunks` — patches are small by
construction (minimal server-computed diffs). If a pathological case shows
up, hunks can be hash-stored later without schema pain.
- **Override payloads stay inline** in the `LangEdits` struct — overrides
are small by construction (minimal server-computed diffs). If a
pathological case shows up, they can be hash-stored later without schema
pain.
## Language index maintenance (`node.langs`)
`node.langs` is a denormalized index over the `trans`/`patches` stores so
`node.langs` is a denormalized index over the `trans`/`overrides` stores so
that article rendering, `select_language`'s availability check, and hreflang
alternate links never enumerate chunks. It is written by whoever writes
translation data, in the same transaction:
@@ -107,11 +111,11 @@ translation data, in the same transaction:
writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on
every article that gained one and invalidates the page cache — all in
one transaction.
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
sets `node.langs[lang] = True` (patches alone make the version exist).
- **Removals:** deleting a patch or GC'ing translations re-derives the key:
- **Translated-view save:** recording the first override for a `(path, lang)`
sets `node.langs[lang] = True` (overrides alone make the version exist).
- **Removals:** deleting overrides or GC'ing translations re-derives the key:
keep `lang` if any `trans` entry for the article's current chunks/title or
any patch remains, otherwise drop it. Stale `langs` keys are benign (an
any override remains, otherwise drop it. Stale `langs` keys are benign (an
advertised language that renders as the original), so removal can lag.
## Render / save pipeline (summary)
@@ -119,8 +123,10 @@ translation data, in the same transaction:
- **Render:** `text = "\n\n".join(chunks[h] for h in node.chunks)` for the
original; for language `L` (only ever attempted when `L in node.langs`),
per chunk `trans.get(h, {}).get(L)` unless missing or `h in node.no_trans`,
falling back to `chunks[h]`; then apply `patches.get(f"{path}:{L}", [])`
in order (per-hunk, best effort); then `markdown.render` as today. All of
falling back to `chunks[h]`; then apply `overrides[path][L]` structurally
in the article's own chunk order (drops, search/replace pairs, anchored
additions — see docs/localization.md); then
`markdown.render` as today. All of
this assembles the `Translation` the phase-1 plumbing already consumes.
- **Availability:** `node.langs` is the availability index; `?lang=`
handling uses exactly this set. (hreflang alternates are site-wide from
@@ -128,9 +134,9 @@ translation data, in the same transaction:
- **Save (primary language):** server re-chunks the submitted Markdown,
inserts new hashes into `Data.chunks`, replaces `node.chunks`. Unchanged
chunks keep their hashes — only genuinely new text lands in the diff.
- **Save (translated view):** diff against the served hybrid, append a
`Patch` under `patches[f"{path}:{lang}"]`; `node.chunks` untouched.
- **Invalidate:** any write to `chunks` / `trans` / `patches` calls
- **Save (translated view):** diff against the served hybrid, record
per-chunk overrides under `overrides[path][lang]`; `node.chunks` untouched.
- **Invalidate:** any write to `chunks` / `trans` / `overrides` calls
`_invalidate_pages()`.
## migrate_v3 steps
@@ -138,10 +144,8 @@ translation data, in the same transaction:
1. Walk `menu`; for every node with a string `content`:
`chunks = chunk_markdown(content)`; write each into the new `chunks`
store; replace the field with the hash list (`None` stays `None`).
2. Initialize empty `chunks` / `trans` / `patches` stores.
3. Normalize stored paths: strip leading slashes anywhere paths are keys or
values.
4. `language`, `no_trans` and `langs` need nothing — struct defaults cover
2. Initialize empty `chunks` / `trans` stores.
3. `language`, `no_trans` and `langs` need nothing — struct defaults cover
them (`langs` starts empty; the translator job fills it as translations
land).
@@ -161,19 +165,19 @@ Chunking must be deterministic and shared with render/save, so
- `Translation.titles` stayed keyed by node path (phase-1 shape, views
untouched): `get_translation` builds it by walking the menu with the same
per-title `trans.get(chunk_key(node.title), {}).get(lang)` lookups.
- Insert hunks anchor on the whole preceding block (not just its tail) —
a stronger, simpler search context.
- `make_patch` diffs with `SequenceMatcher(autojunk=False)` so patches are
deterministic (popular lines like blank separators never become junk).
- Step 3's path normalization is a no-op in practice: the only path-keyed
store (`patches`) starts empty at v3; analytics paths live outside the
kantadb. The code still strips leading slashes defensively.
- User overrides (`record_override`) diff with `SequenceMatcher(autojunk=False)`
so overrides are deterministic (popular lines like blank separators never
become junk).
- The old list-valued `patches` store was later replaced by the keyed
`overrides` store above; the rename itself discarded the old data (msgspec
ignores the unknown key on decode), no migration.
## Garbage collection (later, manual or idle-time)
Orphaned entries accumulate: chunks no longer referenced by any
`node.chunks`/`node.title`, translations whose chunk hash is orphaned, patch
hunks that never match. All are harmless (never read). A GC pass is a single
`node.chunks`/`node.title`, translations whose chunk hash is orphaned,
overrides whose chunk hash is gone from the article (or whose `search`
never matches). All are harmless (never read). A GC pass is a single
tree walk collecting live hashes, then deleting the rest from `chunks` and
`trans`; patches whose every hunk is stale get pruned. Not part of
`trans`; override entries for dead hashes get pruned. Not part of
migrate_v3.