Phase 1: content negotiation (?lang=, Accept-Language) + localization/migration design docs
- pagerite/i18n.py: parse_accept_language, select_language, Translation struct, get_translation/available_languages stubs - app.py: language threaded through show_page/_render_html/_cached_body, ETag and Vary: accept-language - views.py: <html lang>, translated nav titles with fallback, canonical with ?lang=, hreflang alternates - pagerite.js: strip ?lang= via replaceState, send it as Accept-Language on fetch-navigation/preloads, lang-aware page cache - docs/localization.md: negotiation + URL scheme + phase-2 fragment model - docs/migrate.md: migrate_v3 content-addressed chunk storage plan
This commit is contained in:
+142
@@ -0,0 +1,142 @@
|
||||
# migrate_v3: content-addressed chunk storage
|
||||
|
||||
Status: **final plan**. `migrate_v3` restructures how article text and
|
||||
translations are stored, motivated by the localization model in
|
||||
`docs/localization.md` (phase 2). Since it is a full migration, it is free to
|
||||
break the current `Node.content: str | None` layout.
|
||||
|
||||
## Goals
|
||||
|
||||
- **Minimal change diffs.** kanta persists change diffs; editing one
|
||||
paragraph of a long article must not rewrite the whole article string, and
|
||||
a translation refresh must touch only the re-translated chunks.
|
||||
- **Fast, simple lookup.** Everything heavy lives in flat
|
||||
`dict[hash, content]` stores; ordering lives in `list[hash]`. No large
|
||||
nested structures, no deep paths.
|
||||
- **Path-independent text.** Chunks and their translations are keyed by
|
||||
content hash, not by article path — the same paragraph (or menu title)
|
||||
appearing in several articles is stored and translated once. Moving or
|
||||
renaming an article touches nothing.
|
||||
|
||||
## Design (chosen: global content-addressed stores)
|
||||
|
||||
Original articles are *also* stored as chunks; everything — originals and
|
||||
translations — lives in flat hash-keyed dicts. Costs accepted: rendering does
|
||||
one dict lookup per chunk (trivial), orphaned hashes need occasional garbage
|
||||
collection, and the editor save path re-chunks server-side (it already
|
||||
diffs). The rejected alternatives: per-article nested `LangVersion`
|
||||
structures (churn, duplication, whole-string originals) and a hybrid with
|
||||
whole originals plus global translations (keeps the worst change-diff
|
||||
property).
|
||||
|
||||
## Target layout
|
||||
|
||||
```python
|
||||
class Node(msgspec.Struct, omit_defaults=True):
|
||||
...
|
||||
#: Replaces `content: str | None`. None = pure category label;
|
||||
#: a list (possibly empty) = a page, as ordered chunk hashes.
|
||||
chunks: list[str] | None = None
|
||||
#: Primary language of the article (BCP-47 base tag). "" = inherit
|
||||
#: (nearest ancestor, front page last, site default "en" final).
|
||||
language: str = ""
|
||||
#: Chunk hashes the editor marked "do not translate" (always served
|
||||
#: from the original). Presence-keys, value always True.
|
||||
no_trans: dict[str, True] = {}
|
||||
#: Languages this article is available in (besides its primary
|
||||
#: language). Presence-keys, value always True — rendering, language
|
||||
#: selection and hreflang alternates read this set instead of probing
|
||||
#: the trans store chunk by chunk. Maintained by the writers (see
|
||||
#: "Language index maintenance" below).
|
||||
langs: dict[str, True] = {}
|
||||
|
||||
class Data(msgspec.Struct):
|
||||
...
|
||||
#: All original-language text, content-addressed: blake3(normalized,
|
||||
#: digest 16) hex -> Markdown chunk. Shared by every article.
|
||||
chunks: dict[str, str] = {}
|
||||
#: Machine translations: f"{chunk_hash}:{lang}" -> translated Markdown.
|
||||
#: Also used for node titles (hash of the title text).
|
||||
trans: dict[str, str] = {}
|
||||
#: User override patches per article and language:
|
||||
#: f"{path}:{lang}" -> ordered patches (see localization.md).
|
||||
patches: dict[str, list[Patch]] = {}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- **Article paths never carry a leading slash** in the DB or in lookup keys
|
||||
(`"docs/setup"`, front page `""`); the leading slash is added only when
|
||||
building hrefs. `migrate_v3` audits existing stored paths (translation
|
||||
keys, analytics references, any path-valued fields) and normalizes them.
|
||||
- **Titles are chunks too**, by hash only: the nav renderer looks up
|
||||
`trans.get(f"{hash(node.title)}:{lang}")`. No separate title storage;
|
||||
editing a title invalidates its translations automatically.
|
||||
- **Per-hunk options** live in two places: *inherent* options are derived at
|
||||
chunking time (code fences and HTML blocks are marked no-translate without
|
||||
storing anything); *editor-set* flags are `node.no_trans` (keyed by chunk
|
||||
hash, so a heavy edit silently drops the flag — acceptable and
|
||||
self-healing).
|
||||
- **Patch payloads stay inline** in `Patch.hunks` — patches are small by
|
||||
construction (minimal server-computed diffs). If a pathological case shows
|
||||
up, hunks can be hash-stored later without schema pain.
|
||||
|
||||
## Language index maintenance (`node.langs`)
|
||||
|
||||
`node.langs` is a denormalized index over the `trans`/`patches` stores so
|
||||
that article rendering, `select_language`'s availability check, and hreflang
|
||||
alternate links never enumerate chunks. It is written by whoever writes
|
||||
translation data, in the same transaction:
|
||||
|
||||
- **Translator job:** after writing `trans[f"{h}:{lang}"]` entries for an
|
||||
article's chunks (or its title), set `node.langs[lang] = True`.
|
||||
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
|
||||
sets `node.langs[lang] = True` (patches alone make the version exist).
|
||||
- **Removals:** deleting a patch or GC'ing translations re-derives the key:
|
||||
keep `lang` if any `trans` entry for the article's current chunks/title or
|
||||
any patch remains, otherwise drop it. Stale `langs` keys are benign (an
|
||||
advertised language that renders as the original), so removal can lag.
|
||||
|
||||
## Render / save pipeline (summary)
|
||||
|
||||
- **Render:** `text = "\n\n".join(chunks[h] for h in node.chunks)` for the
|
||||
original; for language `L` (only ever attempted when `L in node.langs`),
|
||||
per chunk `trans[f"{h}:{L}"]` unless missing or `h in node.no_trans`,
|
||||
falling back to `chunks[h]`; then apply `patches.get(f"{path}:{L}", [])`
|
||||
in order (per-hunk, best effort); then `markdown.render` as today. All of
|
||||
this assembles the `Translation` the phase-1 plumbing already consumes.
|
||||
- **Availability:** `available_languages(path)` = `sorted(node.langs)`;
|
||||
hreflang alternates and `?lang=` handling use exactly this set.
|
||||
- **Save (primary language):** server re-chunks the submitted Markdown,
|
||||
inserts new hashes into `Data.chunks`, replaces `node.chunks`. Unchanged
|
||||
chunks keep their hashes — only genuinely new text lands in the diff.
|
||||
- **Save (translated view):** diff against the served hybrid, append a
|
||||
`Patch` under `patches[f"{path}:{lang}"]`; `node.chunks` untouched.
|
||||
- **Invalidate:** any write to `chunks` / `trans` / `patches` calls
|
||||
`_invalidate_pages()`.
|
||||
|
||||
## migrate_v3 steps
|
||||
|
||||
1. Walk `menu`; for every node with a string `content`:
|
||||
`chunks = chunk_markdown(content)`; write each into the new `chunks`
|
||||
store; replace the field with the hash list (`None` stays `None`).
|
||||
2. Initialize empty `chunks` / `trans` / `patches` stores.
|
||||
3. Normalize stored paths: strip leading slashes anywhere paths are keys or
|
||||
values.
|
||||
4. `language`, `no_trans` and `langs` need nothing — struct defaults cover
|
||||
them (`langs` starts empty; the translator job fills it as translations
|
||||
land).
|
||||
|
||||
Chunking must be deterministic and shared with render/save, so
|
||||
`chunk_markdown` + `chunk_key` live in `pagerite/i18n.py` (or a small
|
||||
`pagerite/chunks.py`) and are imported by both `migrations.py` and
|
||||
`views.py`/`app.py`.
|
||||
|
||||
## Garbage collection (later, manual or idle-time)
|
||||
|
||||
Orphaned entries accumulate: chunks no longer referenced by any
|
||||
`node.chunks`/`node.title`, translations whose chunk hash is orphaned, patch
|
||||
hunks that never match. All are harmless (never read). A GC pass is a single
|
||||
tree walk collecting live hashes, then deleting the rest from `chunks` and
|
||||
`trans`; patches whose every hunk is stale get pruned. Not part of
|
||||
migrate_v3.
|
||||
Reference in New Issue
Block a user