diff --git a/AGENTS.md b/AGENTS.md index 47df64c..30bc268 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,8 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `pagerite/` — Python backend package (hatchling build target). - `app.py` — FastAPI app and route registration. - `data.py` — msgspec Structs for the kanta database. + - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md). + - `i18n.py` — language selection, translation assembly (chunks + patches). - `migrations.py` — kanta migrations (`migrate_vN`); ALL schema/storage upgrades live here (raw state dict before struct decoding), never in the app lifespan: v1 moves legacy in-db file blobs to the on-disk store and rebuilds the legacy flat `pages` as the menu tree, v2 rewrites `/_f/{hash}.ext` image links to the extension-less form, backfills AVIF/WebP/JPEG derivatives on disk and drops the obsolete `version` field. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering; theme/user-font resolution across `THEME_DIRS` / `FONT_DIRS` (cwd, site, platform data roots, then built-in `pagerite/themes/`, see `docs/themes-and-assets.md`). diff --git a/docs/localization.md b/docs/localization.md index 704336c..6da55c2 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -6,7 +6,7 @@ parameter or the `Accept-Language` header. - **Phase 1 (implemented):** negotiation, URL scheme, caching, rendering plumbing. Translations are consumed through a stub interface; the database still holds only the original language. -- **Phase 2 (final plan):** gettext-style fragment storage in the +- **Phase 2 (implemented):** gettext-style fragment storage in the database — machine-translated chunks plus user override patches, assembled at render time. Storage details in `docs/migrate.md`. @@ -66,7 +66,7 @@ Region tags normalize to their base subtag (`fi-FI` → `fi`). - The markdown typographer (SmartyPants) is English-centric; per-language typographer options are a possible follow-up, not blocking. -## Phase 2: fragment-based translation storage (draft) +## Phase 2: fragment-based translation storage (implemented) Phase 1 assumed whole-page translated Markdown delivered from outside. The refined model is gettext-style: an article has **one primary version** (its @@ -170,7 +170,7 @@ Full storage design and the `migrate_v3` restructuring live in - Article paths are stored and keyed **without leading slashes** (`"docs/setup"`, front page `""`); slashes are added only in hrefs. -### Render pipeline (replaces the phase-1 `get_translation` stub) +### Render pipeline (the phase-1 `get_translation` stub, now real) ```python def get_translation(path, lang, data) -> Translation | None: diff --git a/docs/migrate.md b/docs/migrate.md index 46d9493..1bf0380 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -1,6 +1,6 @@ # migrate_v3: content-addressed chunk storage -Status: **final plan**. `migrate_v3` restructures how article text and +Status: **implemented**. `migrate_v3` restructures how article text and translations are stored, motivated by the localization model in `docs/localization.md` (phase 2). Since it is a full migration, it is free to break the current `Node.content: str | None` layout. @@ -132,6 +132,21 @@ Chunking must be deterministic and shared with render/save, so `pagerite/chunks.py`) and are imported by both `migrations.py` and `views.py`/`app.py`. +## Implementation notes (deviations from the plan above) + +- Chunking lives in `pagerite/chunks.py`; hashing uses the `blake3` package + (already a dependency) with a 16-byte digest (`hexdigest(16)`). +- `Translation.titles` stayed keyed by node path (phase-1 shape, views + untouched): `get_translation` builds it by walking the menu with the same + per-title `trans[f"{chunk_key(node.title)}:{lang}"]` lookups. +- Insert hunks anchor on the whole preceding block (not just its tail) — + a stronger, simpler search context. +- `make_patch` diffs with `SequenceMatcher(autojunk=False)` so patches are + deterministic (popular lines like blank separators never become junk). +- Step 3's path normalization is a no-op in practice: the only path-keyed + store (`patches`) starts empty at v3; analytics paths live outside the + kantadb. The code still strips leading slashes defensively. + ## Garbage collection (later, manual or idle-time) Orphaned entries accumulate: chunks no longer referenced by any