LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 1: content negotiation (?lang=, Accept-Language) + localization/migration design docs

Browse Source 
- pagerite/i18n.py: parse_accept_language, select_language, Translation struct,
  get_translation/available_languages stubs
- app.py: language threaded through show_page/_render_html/_cached_body,
  ETag and Vary: accept-language
- views.py: <html lang>, translated nav titles with fallback, canonical
  with ?lang=, hreflang alternates
- pagerite.js: strip ?lang= via replaceState, send it as Accept-Language
  on fetch-navigation/preloads, lang-aware page cache
- docs/localization.md: negotiation + URL scheme + phase-2 fragment model
- docs/migrate.md: migrate_v3 content-addressed chunk storage plan
This commit is contained in:
LeoVasanko
2026-09-02 01:14:10 +00:00
parent b6e6e46cfb
commit 218313ba7a
6 changed files with 584 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/localization.md
+219
View File
@@ -0,0 +1,219 @@
# Localization
Pages are served in the visitor's language based on a `?lang=` query
parameter or the `Accept-Language` header.
- **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering
plumbing. Translations are consumed through a stub interface; the database
still holds only the original language.
- **Phase 2 (final plan):** gettext-style fragment storage in the
database — machine-translated chunks plus user override patches, assembled
at render time. Storage details in `docs/migrate.md`.
## Phase 1: negotiation and URLs
### Language selection
Deliberately simple — **q-values are ignored**:
- All known `Accept-Language` implementations send the header **in order of
preference**, so we parse it as an ordered list and never reorder.
- Selection rule (`select_language` in `pagerite/i18n.py`):
1. If `?lang=<tag>` is present, use it (if a translation exists; otherwise
fall through to header logic).
2. If the article's original language appears anywhere in the header list,
use the **original**. Rationale: an AI translation is strictly worse
than the original for anyone who has that language configured at all
(e.g. `fi-FI, fi, en-US, en` gets English, not machine-translated
Finnish).
3. Otherwise walk the header list in order and use the first language for
which a translation exists.
4. Fall back to the original.
Region tags normalize to their base subtag (`fi-FI` → `fi`).
### URLs: pretty for users, indexable for search engines
- Canonical URLs stay pretty (`/some-page`). Each language version is
addressable as `/some-page?lang=fi` so search engines can index them.
- `<link rel="canonical">` points to the page **itself including the query**
(each language version is its own canonical).
- `<link rel="alternate" hreflang="…">` entries point to every other language
version (with `?lang=`), plus `x-default` for the plain URL.
- On page load, pagerite.js removes the `?lang=` query via
`history.replaceState` (restoring the pretty URL) but remembers the language
in a JS variable. All fetch-navigation and preloads it performs afterwards
send that language in the `Accept-Language` header, so the chosen language
sticks for the session of clicks.
- A full page refresh or a shared link resets to automatic selection (header
only). This gives a clean one-time override without cookies.
### Response correctness
- Content responses carry `Vary: accept-language` (added to the existing
`accept-encoding` vary).
- `_cached_body` and the page ETag include the **selected language** (not the
raw header, which would blow up the cache key space).
- `<html lang="…">` reflects the served language.
### Rendering
- The translated Markdown goes through the same `markdown.render` pipeline.
- Navigation/sidebar titles come from the translation's title map, with
per-node fallback to the original title (a partially translated tree must
still render).
- Fixed UI strings ("Not Found" etc.) and the editor UI stay English for now.
- The markdown typographer (SmartyPants) is English-centric; per-language
typographer options are a possible follow-up, not blocking.
## Phase 2: fragment-based translation storage (draft)
Phase 1 assumed whole-page translated Markdown delivered from outside. The
refined model is gettext-style: an article has **one primary version** (its
`content`, in its own language) plus, per target language, **machine
fragments** (translated chunks of Markdown) and **user patches** (minimal
editor overrides). Both are stored in the database and assembled into the
served Markdown at render time.
### The scenario this must handle
1. Article written in English.
2. Machine-translated into Spanish → fragments stored.
3. Editor fixes one Spanish paragraph and changes a link elsewhere to point
at a Spanish resource → user patch hunks stored.
4. English article edited → the edited chunk's key changes; its Spanish
fragment no longer matches.
5. Page requested before the machine translation refreshes → served as a
**hybrid**: old fragments for unchanged chunks, plain English for the
edited chunk. User patches are attempted against this hybrid, best effort,
each hunk independently: the text fix is stale (its search text no longer
exists) and silently skipped; the link change still applies even though
the link sits in the now-English paragraph.
6. Machine translation refreshes → full Spanish again, with both patch hunks
applying.
### Chunks
`chunk_markdown(markdown)` splits the source into block-level chunks —
blank-line-separated blocks: headings, paragraphs, code fences (kept whole),
list blocks, tables, HTML blocks. A chunk's identity is its **source text**,
gettext-msgid style:
```python
chunk_key = blake3(normalize(chunk_text), digest_size=16).hex()
```
(`normalize`: strip trailing whitespace per line, collapse surrounding blank
lines — so whitespace-only source edits don't invalidate translations.)
Consequences:
- Editing the English source invalidates exactly the edited chunks; all
other fragments keep applying. Stale fragments are simply never referenced
again and can be garbage-collected lazily (or left; they are tiny).
- No explicit "source version" bookkeeping is needed — staleness falls out
of the keys.
### User patches
Editors always edit **full Markdown** in the existing editor UX — never
fragments. When editing a translated view (`?lang=es`), the editor is loaded
with the *current hybrid Markdown*; on save, the server computes a minimal
diff against that hybrid and stores it as a patch:
```python
class Patch(msgspec.Struct, omit_defaults=True):
"""One editing session's overrides, applied independently per hunk."""
hunks: list[tuple[str, str]] = [] # (search, replace) on hybrid Markdown
```
Hunks are produced from `difflib.SequenceMatcher` on the hybrid vs. the
edited text at block granularity: each `replace`/`delete`/`insert` opcode
becomes one `(search, replace)` pair, with the preceding block's tail as
left context for `insert` (pure inserts have empty search context otherwise).
Application is dead simple:
```python
def apply_patch(hybrid: str, patch: Patch) -> str:
for search, replace in patch.hunks:
if search and search in hybrid:
hybrid = hybrid.replace(search, replace, 1)
# missing search text = stale hunk -> silently skipped
return hybrid
```
Per-hunk independence is the robustness property from the scenario: a stale
text fix does not block a still-valid link change. Patches are stored as an
ordered list and applied in order.
### Storage
Full storage design and the `migrate_v3` restructuring live in
`docs/migrate.md`. The short version, as it concerns this document:
- Originals **and** translations are content-addressed text chunks in flat
stores: `Data.chunks: dict[hash, str]` and
`Data.trans: dict[f"{chunk_hash}:{lang}", str]` — path-independent, so
repeated paragraphs and menu titles are translated once and article moves
touch nothing. `Node.chunks: list[hash]` gives each article its order.
- `Node` gains **`language: str = ""`**, inherited down the tree like
`banner` (empty = nearest ancestor, front page last, site default `en`
final). `select_language` and `<html lang>` use the resolved value instead
of the global `ORIGINAL_LANGUAGE` constant.
- **Known weakness:** changing a page's (or subtree's) `language` after
translations exist mis-keys everything — translations are keyed by
*source* chunks, so old entries silently stop matching and user patches
(searching for old-hybrid text) mostly go stale. That is acceptable:
the orphaned data is harmless and translations regenerate. We do not
migrate translations across a language change.
- Article paths are stored and keyed **without leading slashes**
(`"docs/setup"`, front page `""`); slashes are added only in hrefs.
### Render pipeline (replaces the phase-1 `get_translation` stub)
```python
def get_translation(path, lang, data) -> Translation | None:
if lang not in node.langs:
return None
hybrid = "\n\n".join(
chunks[h] if h in node.no_trans else trans.get(f"{h}:{lang}", chunks[h])
for h in node.chunks
)
for patch in data.patches.get(f"{path}:{lang}", []):
hybrid = apply_patch(hybrid, patch)
return Translation(markdown=hybrid, titles=title_map(data, lang))
```
- Availability is an article-level index: `node.langs: dict[lang, True]`,
maintained by the translation writers (translator job, patch saves) in the
same transaction as their data writes — rendering, language selection and
hreflang never probe the `trans` store chunk by chunk. A stale key is
benign (the "translation" just renders as the original).
- `titles` for nav/sidebar/cards: each node's translated title is
`trans.get(f"{hash(node.title)}:{lang}")` with per-node fallback — one dict
lookup per nav item at render time.
- Cache invalidation: writes to `chunks` / `trans` / `patches` (translator,
editor saves) call `_invalidate_pages()`, same as content writes.
### Editor flow
- `GET` of page Markdown for editing with a `lang` parameter returns the
hybrid (not the raw original) when the article has that language.
- `PUT`/WS save with `lang` does **not** touch `node.chunks`; it diffs
against the hybrid that was served and appends a `Patch`. (Serve a hybrid
generation token with the editor payload so a save based on a stale hybrid
can be rebased or rejected — simplest: recompute the diff against the
*current* hybrid and accept best-effort, matching the patch philosophy.)
- Saving the primary-language version re-chunks the submitted Markdown and
updates `Data.chunks` / `node.chunks` — only genuinely new text lands in
the kanta change diff (see docs/migrate.md).
### Explicitly out of scope for phase 2
- The machine translation itself: chunking output goes in, translated chunks
come back. A background job writes `trans` entries; this doc only defines
the storage key (`chunk_hash:lang`) and the merge semantics.
- Garbage collection of orphaned chunks/translations (see docs/migrate.md).
- sitemap.xml per-language entries; translated UI chrome; per-language
typographer options; multi-locale date/number formatting.
docs/migrate.md
+142
View File
@@ -0,0 +1,142 @@
# migrate_v3: content-addressed chunk storage
Status: **final plan**. `migrate_v3` restructures how article text and
translations are stored, motivated by the localization model in
`docs/localization.md` (phase 2). Since it is a full migration, it is free to
break the current `Node.content: str | None` layout.
## Goals
- **Minimal change diffs.** kanta persists change diffs; editing one
paragraph of a long article must not rewrite the whole article string, and
a translation refresh must touch only the re-translated chunks.
- **Fast, simple lookup.** Everything heavy lives in flat
`dict[hash, content]` stores; ordering lives in `list[hash]`. No large
nested structures, no deep paths.
- **Path-independent text.** Chunks and their translations are keyed by
content hash, not by article path — the same paragraph (or menu title)
appearing in several articles is stored and translated once. Moving or
renaming an article touches nothing.
## Design (chosen: global content-addressed stores)
Original articles are *also* stored as chunks; everything — originals and
translations — lives in flat hash-keyed dicts. Costs accepted: rendering does
one dict lookup per chunk (trivial), orphaned hashes need occasional garbage
collection, and the editor save path re-chunks server-side (it already
diffs). The rejected alternatives: per-article nested `LangVersion`
structures (churn, duplication, whole-string originals) and a hybrid with
whole originals plus global translations (keeps the worst change-diff
property).
## Target layout
```python
class Node(msgspec.Struct, omit_defaults=True):
...
#: Replaces `content: str | None`. None = pure category label;
#: a list (possibly empty) = a page, as ordered chunk hashes.
chunks: list[str] | None = None
#: Primary language of the article (BCP-47 base tag). "" = inherit
#: (nearest ancestor, front page last, site default "en" final).
language: str = ""
#: Chunk hashes the editor marked "do not translate" (always served
#: from the original). Presence-keys, value always True.
no_trans: dict[str, True] = {}
#: Languages this article is available in (besides its primary
#: language). Presence-keys, value always True — rendering, language
#: selection and hreflang alternates read this set instead of probing
#: the trans store chunk by chunk. Maintained by the writers (see
#: "Language index maintenance" below).
langs: dict[str, True] = {}
class Data(msgspec.Struct):
...
#: All original-language text, content-addressed: blake3(normalized,
#: digest 16) hex -> Markdown chunk. Shared by every article.
chunks: dict[str, str] = {}
#: Machine translations: f"{chunk_hash}:{lang}" -> translated Markdown.
#: Also used for node titles (hash of the title text).
trans: dict[str, str] = {}
#: User override patches per article and language:
#: f"{path}:{lang}" -> ordered patches (see localization.md).
patches: dict[str, list[Patch]] = {}
```
Notes:
- **Article paths never carry a leading slash** in the DB or in lookup keys
(`"docs/setup"`, front page `""`); the leading slash is added only when
building hrefs. `migrate_v3` audits existing stored paths (translation
keys, analytics references, any path-valued fields) and normalizes them.
- **Titles are chunks too**, by hash only: the nav renderer looks up
`trans.get(f"{hash(node.title)}:{lang}")`. No separate title storage;
editing a title invalidates its translations automatically.
- **Per-hunk options** live in two places: *inherent* options are derived at
chunking time (code fences and HTML blocks are marked no-translate without
storing anything); *editor-set* flags are `node.no_trans` (keyed by chunk
hash, so a heavy edit silently drops the flag — acceptable and
self-healing).
- **Patch payloads stay inline** in `Patch.hunks` — patches are small by
construction (minimal server-computed diffs). If a pathological case shows
up, hunks can be hash-stored later without schema pain.
## Language index maintenance (`node.langs`)
`node.langs` is a denormalized index over the `trans`/`patches` stores so
that article rendering, `select_language`'s availability check, and hreflang
alternate links never enumerate chunks. It is written by whoever writes
translation data, in the same transaction:
- **Translator job:** after writing `trans[f"{h}:{lang}"]` entries for an
article's chunks (or its title), set `node.langs[lang] = True`.
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
sets `node.langs[lang] = True` (patches alone make the version exist).
- **Removals:** deleting a patch or GC'ing translations re-derives the key:
keep `lang` if any `trans` entry for the article's current chunks/title or
any patch remains, otherwise drop it. Stale `langs` keys are benign (an
advertised language that renders as the original), so removal can lag.
## Render / save pipeline (summary)
- **Render:** `text = "\n\n".join(chunks[h] for h in node.chunks)` for the
original; for language `L` (only ever attempted when `L in node.langs`),
per chunk `trans[f"{h}:{L}"]` unless missing or `h in node.no_trans`,
falling back to `chunks[h]`; then apply `patches.get(f"{path}:{L}", [])`
in order (per-hunk, best effort); then `markdown.render` as today. All of
this assembles the `Translation` the phase-1 plumbing already consumes.
- **Availability:** `available_languages(path)` = `sorted(node.langs)`;
hreflang alternates and `?lang=` handling use exactly this set.
- **Save (primary language):** server re-chunks the submitted Markdown,
inserts new hashes into `Data.chunks`, replaces `node.chunks`. Unchanged
chunks keep their hashes — only genuinely new text lands in the diff.
- **Save (translated view):** diff against the served hybrid, append a
`Patch` under `patches[f"{path}:{lang}"]`; `node.chunks` untouched.
- **Invalidate:** any write to `chunks` / `trans` / `patches` calls
`_invalidate_pages()`.
## migrate_v3 steps
1. Walk `menu`; for every node with a string `content`:
`chunks = chunk_markdown(content)`; write each into the new `chunks`
store; replace the field with the hash list (`None` stays `None`).
2. Initialize empty `chunks` / `trans` / `patches` stores.
3. Normalize stored paths: strip leading slashes anywhere paths are keys or
values.
4. `language`, `no_trans` and `langs` need nothing — struct defaults cover
them (`langs` starts empty; the translator job fills it as translations
land).
Chunking must be deterministic and shared with render/save, so
`chunk_markdown` + `chunk_key` live in `pagerite/i18n.py` (or a small
`pagerite/chunks.py`) and are imported by both `migrations.py` and
`views.py`/`app.py`.
## Garbage collection (later, manual or idle-time)
Orphaned entries accumulate: chunks no longer referenced by any
`node.chunks`/`node.title`, translations whose chunk hash is orphaned, patch
hunks that never match. All are harmless (never read). A GC pass is a single
tree walk collecting live hashes, then deleting the rest from `chunks` and
`trans`; patches whose every hunk is stale get pruned. Not part of
migrate_v3.