pagerite/docs/migrate.md

# migrate_v3: content-addressed chunk storage

Status: **implemented**. `migrate_v3` restructures how article text and
translations are stored, motivated by the localization model in
`docs/localization.md` (phase 2). Since it is a full migration, it is free to
break the current `Node.content: str | None` layout.

## Goals

- **Minimal change diffs.** kanta persists change diffs; editing one
  paragraph of a long article must not rewrite the whole article string, and
  a translation refresh must touch only the re-translated chunks.
- **Fast, simple lookup.** Everything heavy lives in flat
  `dict[hash, content]` stores; ordering lives in `list[hash]`. No large
  nested structures, no deep paths.
- **Path-independent text.** Chunks and their translations are keyed by
  content hash, not by article path — the same paragraph (or menu title)
  appearing in several articles is stored and translated once. Moving or
  renaming an article touches nothing.

## Design (chosen: global content-addressed stores)

Original articles are *also* stored as chunks; everything — originals and
translations — lives in flat hash-keyed dicts. Costs accepted: rendering does
one dict lookup per chunk (trivial), orphaned hashes need occasional garbage
collection, and the editor save path re-chunks server-side (it already
diffs). The rejected alternatives: per-article nested `LangVersion`
structures (churn, duplication, whole-string originals) and a hybrid with
whole originals plus global translations (keeps the worst change-diff
property).

## Target layout

```python
class Node(msgspec.Struct, omit_defaults=True):
    ...
    #: Replaces `content: str | None`. None = pure category label;
    #: a list (possibly empty) = a page, as ordered chunk hashes.
    chunks: list[bytes] | None = None
    #: Primary language of the article (BCP-47 base tag). "" = inherit
    #: (nearest ancestor, front page last, site default "en" final).
    language: str = ""
    #: Chunk hashes the editor marked "do not translate" (always served
    #: from the original). Presence-keys, value always True.
    no_trans: dict[bytes, True] = {}
    #: Languages this article is available in (besides its primary
    #: language). Presence-keys, value always True — rendering, language
    #: selection and hreflang alternates read this set instead of probing
    #: the trans store chunk by chunk. Maintained by the writers (see
    #: "Language index maintenance" below).
    langs: dict[str, True] = {}


class Data(msgspec.Struct):
    ...
    #: API keys gating the translator service WebSocket (/_translate/{key}):
    #: key -> display name; the first is generated at bootstrap (state.py).
    translate_keys: dict[str, str] = {}
    #: Wanted target languages for the translator service (presence-keys);
    #: jobs are offered only in these ∩ a connection's capabilities.
    translate_langs: dict[str, True] = {}
    #: All original-language text, content-addressed: blake3(normalized)
    #: digest[:9] -> Markdown chunk. Shared by every article. Keys are
    #: bytes; kanta/msgspec base64-encode them at the JSON level.
    chunks: dict[bytes, str] = {}
    #: Machine translations: chunk hash -> lang -> translated Markdown
    #: (nested, not tuple keys: msgspec's JSON serializer rejects them).
    #: Also used for node titles (hash of the title text).
    trans: dict[bytes, dict[str, str]] = {}
    #: User override edits per article and language:
    #: path -> lang -> LangEdits (see localization.md) — keyed per original
    #: chunk hash throughout, so a save's change diff touches only the
    #: edited chunks. Replaced the old list-valued "patches" key (ignored
    #: on decode, discarding that data — no migration).
    overrides: dict[str, dict[str, LangEdits]] = {}
```

Notes:

- **Article paths never carry a leading slash** in the DB or in lookup keys
  (`"docs/setup"`, front page `""`); the leading slash is added only when
  building hrefs. `migrate_v3` audits existing stored paths (translation
  keys, analytics references, any path-valued fields) and normalizes them.
- **Titles are chunks too**, by hash only: the nav renderer looks up
  `trans.get(hash(node.title), {}).get(lang)`. No separate title storage;
  editing a title invalidates its translations automatically.
- **Per-hunk options** live in two places: *inherent* options are derived at
  chunking time (code fences, HTML blocks and prose-free chunks are
  no-translate without storing anything — `needs_translation`, see
  docs/localization.md "Masking"); *editor-set* flags are `node.no_trans`
  (keyed by chunk
  hash, so a heavy edit silently drops the flag — acceptable and
  self-healing).
- **Override payloads stay inline** in the `LangEdits` struct — overrides
  are small by construction (minimal server-computed diffs). If a
  pathological case shows up, they can be hash-stored later without schema
  pain.

## Language index maintenance (`node.langs`)

`node.langs` is a denormalized index over the `trans`/`overrides` stores so
that article rendering, `select_language`'s availability check, and hreflang
alternate links never enumerate chunks. It is written by whoever writes
translation data, in the same transaction:

- **Translator service:** the WebSocket API at `/_translate/{key}` (see
  docs/localization.md) offers pending fragments (titles + translatable
  chunks lacking an entry for the language) as single-item jobs — one at
  a time per connection, in `Data.translate_langs` ∩ the connection's
  announced capabilities — and receives the matching result; storing it
  writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on
  every article that gained one and invalidates the page cache — all in
  one transaction.
- **Translated-view save:** recording the first override for a `(path, lang)`
  sets `node.langs[lang] = True` (overrides alone make the version exist).
- **Removals:** deleting overrides or GC'ing translations re-derives the key:
  keep `lang` if any `trans` entry for the article's current chunks/title or
  any override remains, otherwise drop it. Stale `langs` keys are benign (an
  advertised language that renders as the original), so removal can lag.

## Render / save pipeline (summary)

- **Render:** `text = "\n\n".join(chunks[h] for h in node.chunks)` for the
  original; for language `L` (only ever attempted when `L in node.langs`),
  per chunk `trans.get(h, {}).get(L)` unless missing or `h in node.no_trans`,
  falling back to `chunks[h]`; then apply `overrides[path][L]` structurally
  in the article's own chunk order (drops, search/replace pairs, anchored
  additions — see docs/localization.md); then
  `markdown.render` as today. All of
  this assembles the `Translation` the phase-1 plumbing already consumes.
- **Availability:** `node.langs` is the availability index; `?lang=`
  handling uses exactly this set. (hreflang alternates are site-wide from
  `translate_langs` instead — see docs/localization.md.)
- **Save (primary language):** server re-chunks the submitted Markdown,
  inserts new hashes into `Data.chunks`, replaces `node.chunks`. Unchanged
  chunks keep their hashes — only genuinely new text lands in the diff.
- **Save (translated view):** diff against the served hybrid, record
  per-chunk overrides under `overrides[path][lang]`; `node.chunks` untouched.
- **Invalidate:** any write to `chunks` / `trans` / `overrides` calls
  `_invalidate_pages()`.

## migrate_v3 steps

1. Walk `menu`; for every node with a string `content`:
   `chunks = chunk_markdown(content)`; write each into the new `chunks`
  store; replace the field with the hash list (`None` stays `None`).
2. Initialize empty `chunks` / `trans` stores.
3. `language`, `no_trans` and `langs` need nothing — struct defaults cover
   them (`langs` starts empty; the translator job fills it as translations
   land).

Chunking must be deterministic and shared with render/save, so
`chunk_markdown` + `chunk_key` live in `pagerite/i18n.py` (or a small
`pagerite/chunks.py`) and are imported by both `migrations.py` and
`views.py`/`state.py`.

## Implementation notes (deviations from the plan above)

- Chunking lives in `pagerite/chunks.py`; hashing uses the `blake3` package
  (already a dependency), truncated to a 9-byte `bytes` digest (kanta's
  JSON persistence base64-encodes bytes keys to 12-char strings).
- `trans` is keyed `hash -> lang -> text` (nested dict), not by
  `f"{hash}:{lang}"` tuples: msgspec's JSON serializer only supports
  str-like/number-like dict keys, and kanta persists as JSON lines.
- `Translation.titles` stayed keyed by node path (phase-1 shape, views
  untouched): `get_translation` builds it by walking the menu with the same
  per-title `trans.get(chunk_key(node.title), {}).get(lang)` lookups.
- User overrides (`record_override`) diff with `SequenceMatcher(autojunk=False)`
  so overrides are deterministic (popular lines like blank separators never
  become junk).
- The old list-valued `patches` store was later replaced by the keyed
  `overrides` store above; the rename itself discarded the old data (msgspec
  ignores the unknown key on decode), no migration.

## Garbage collection (later, manual or idle-time)

Orphaned entries accumulate: chunks no longer referenced by any
`node.chunks`/`node.title`, translations whose chunk hash is orphaned,
overrides whose chunk hash is gone from the article (or whose `search`
never matches). All are harmless (never read). A GC pass is a single
tree walk collecting live hashes, then deleting the rest from `chunks` and
`trans`; override entries for dead hashes get pruned. Not part of
migrate_v3.