diff --git a/AGENTS.md b/AGENTS.md index 30bc268..7a15ffb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,6 +14,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `data.py` — msgspec Structs for the kanta database. - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md). - `i18n.py` — language selection, translation assembly (chunks + patches). + - `translate.py` — translator service protocol (msgspec structs) and pending/store core for the `/_translate/{key}` WebSocket (docs/localization.md). - `migrations.py` — kanta migrations (`migrate_vN`); ALL schema/storage upgrades live here (raw state dict before struct decoding), never in the app lifespan: v1 moves legacy in-db file blobs to the on-disk store and rebuilds the legacy flat `pages` as the menu tree, v2 rewrites `/_f/{hash}.ext` image links to the extension-less form, backfills AVIF/WebP/JPEG derivatives on disk and drops the obsolete `version` field. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering; theme/user-font resolution across `THEME_DIRS` / `FONT_DIRS` (cwd, site, platform data roots, then built-in `pagerite/themes/`, see `docs/themes-and-assets.md`). @@ -25,6 +26,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `pagerite.js` — public page entry. - `assets/` — base CSS, Pygments styles, fonts. - `scripts/devserver.py` — dev server with auto reload (the user mostly uses this; avoid running the server yourself, ask the user to test). +- `scripts/translator.py` — Seed-X translator service client for the `/_translate/{key}` socket (reference client, runs in its own uv env via PEP 723). Server run by CLI entry point `uv run pagerite` (no auto reloads, build needed). Dev mode is `scripts/devserver.py` (auto reloads, no build needed). @@ -55,5 +57,5 @@ Server run by CLI entry point `uv run pagerite` (no auto reloads, build needed). - Keep dependencies minimal; add via `uv add` and mention it. - The public URL space belongs to content (pretty slugs at root). Reserve only `/_` for the machinery (`/_api/`, `/_f/`, `/_assets/`), plus `/favicon.ico` from the build. Slugs are lowercase ASCII letters, digits, hyphens and underscores `[a-z0-9_-]` (the site editor filters input live via `slugify.js`, built on the `transliteration` npm package — unicode folds to ASCII, spaces become hyphens; an empty slug on a new page is derived from its title), may not begin with `_` or `.`, and such URLs are never looked up as content. -- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. +- No auth in core code; the SSO/reverse proxy gates all of `/_api` (forward-auth) and owns `/auth/` (login/logout, session validation). Pages render identically for everyone; pagerite.js adds the editing UI only after the auth server validates the session. The one keyed exception is `/_translate/{key}` (translator service; `Data.translate_key`, see docs/localization.md). - Update the relevant MarkDown files when architecture, tooling, or conventions change. diff --git a/docs/localization.md b/docs/localization.md index 1964cd2..b00a16a 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -210,14 +210,38 @@ def get_translation(path, lang, data) -> Translation | None: updates `Data.chunks` / `node.chunks` — only genuinely new text lands in the kanta change diff (see docs/migrate.md). +### Translator service API + +An external machine-translation service connects over WebSocket at +`/_translate/{key}` — deliberately **not** under `/_api`: the SSO +forward-auth does not cover that route, and the key in the path is the +access control. The key is `Data.translate_key`, generated once at startup +and surfaced to the admin in `GET /_api/settings` as `translate_key`. A +wrong or empty key rejects the handshake (close-before-accept → HTTP 403). + +Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; +`bytes` fields ride as base64): + +- `{"type": "hello", "langs": [...]}` — client greeting: the target + languages it handles (normalized to base subtags; `en`/empty dropped). +- `{"type": "job", "lang", "items": [{key, text, path, kind}]}` — server + push: pending fragments (article titles and chunks), deduped by key. +- `{"type": "result", "lang", "items": [{key, text}]}` — client reply: + translated fragments, matched to content by chunk key alone. + +The model is **push**, not polling: on `hello` the server sends everything +pending per announced language; afterwards `_invalidate_pages()` (called by +every content/translation write) schedules a delta push of newly pending +items. Outstanding-item tracking is per connection, so a reconnecting client +simply re-receives everything still pending. Results are stored into `trans` +in one transaction and set `node.langs[lang]` on every article they touch +(shared chunks make several pages gain a language from one fragment). + ### Explicitly out of scope for phase 2 -- The machine translation itself: chunking output goes in, translated chunks - come back. The **service API exists** — `GET /_api/translate/{lang}` lists - pending items (`{"key", "text", "path", "kind"}`, key = base64 chunk hash), - `POST /_api/translate/{lang}` stores a batch (`{"items": [{key, text}]}`) - into `trans` and maintains `node.langs`; an external service does the - actual translating (gated by the /_api forward-auth like everything else). +- The machine translation itself: the API above moves fragments in and out; + the translating is external. `scripts/translator.py` is the reference + client (Seed-X-PPO-7B only — its 28 languages are the ceiling). - Garbage collection of orphaned chunks/translations (see docs/migrate.md). - sitemap.xml per-language entries; translated UI chrome; per-language typographer options; multi-locale date/number formatting. diff --git a/docs/migrate.md b/docs/migrate.md index 865551b..f55bc54 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -52,6 +52,9 @@ class Node(msgspec.Struct, omit_defaults=True): class Data(msgspec.Struct): ... + #: API key gating the translator service WebSocket (/_translate/{key}); + #: generated lazily at startup (see the lifespan in app.py). + translate_key: str = "" #: All original-language text, content-addressed: blake3(normalized) #: digest[:9] -> Markdown chunk. Shared by every article. Keys are #: bytes; kanta/msgspec base64-encode them at the JSON level. @@ -90,13 +93,12 @@ that article rendering, `select_language`'s availability check, and hreflang alternate links never enumerate chunks. It is written by whoever writes translation data, in the same transaction: -- **Translator job:** after writing `trans[h][lang]` entries for an - article's chunks (or its title), set `node.langs[lang] = True`. The - translation service API does both: `GET /_api/translate/{lang}` lists - pending items (titles + translatable chunks lacking an entry, deduped by - hash), `POST /_api/translate/{lang}` stores a batch into `trans`, sets - `langs` on every article that gained an entry and invalidates the page - cache — all in one transaction. +- **Translator service:** the WebSocket API at `/_translate/{key}` (see + docs/localization.md) pushes pending fragments (titles + translatable + chunks lacking an entry, deduped by hash) and receives result batches; + storing a batch writes `trans[h][lang]` entries, sets + `node.langs[lang] = True` on every article that gained one and + invalidates the page cache — all in one transaction. - **Translated-view save:** appending the first patch for `f"{path}:{lang}"` sets `node.langs[lang] = True` (patches alone make the version exist). - **Removals:** deleting a patch or GC'ing translations re-derives the key: