LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document the translation dispatcher and capability handshake

Browse Source
This commit is contained in:
LeoVasanko
2026-09-02 04:11:40 +00:00
parent 0610daf1af
commit dd0a6cc04b
2 changed files with 42 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/localization.md
+32 -13
View File
@@ -222,20 +222,39 @@ wrong or empty key rejects the handshake (close-before-accept → HTTP 403).
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
`bytes` fields ride as base64): `bytes` fields ride as base64):
- `{"type": "hello", "langs": [...]}` — client greeting: the target - `{"type": "hello", "langs": [...]}` — client greeting announcing its
languages it handles (normalized to base subtags; `en`/empty dropped). **capabilities**: the language codes its model can produce (normalized
- `{"type": "job", "lang", "items": [{key, text, path, kind}]}` — server to base subtags; `en`/empty dropped).
push: pending fragments (article titles and chunks), deduped by key. - `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
- `{"type": "result", "lang", "items": [{key, text}]}` — client reply: ONE fragment to translate (an article title or a chunk).
translated fragments, matched to content by chunk key alone. - `{"type": "result", "lang", "key", "text"}` — client reply: the
translation of the connection's current job, matching it by (lang, key).
The model is **push**, not polling: on `hello` the server sends everything Which languages get translated is **server-configured**:
pending per announced language; afterwards `_invalidate_pages()` (called by `Data.translate_langs` (presence-key dict, read/set via `/_api/settings`
every content/translation write) schedules a delta push of newly pending as `translate_langs`; no editing UI yet). The dispatcher offers a
items. Outstanding-item tracking is per connection, so a reconnecting client connection jobs only in `wanted ∩ capable`; a connection without overlap
simply re-receives everything still pending. Results are stored into `trans` simply stays idle.
in one transaction and set `node.langs[lang]` on every article they touch
(shared chunks make several pages gain a language from one fragment). Dispatch semantics (all in app.py):
- **One job at a time per connection** — the next job is sent only after
the current one's result. Clients wanting parallelism open multiple
connections (e.g. several `scripts/translator.py` instances).
- Pending work is derived from the `trans` store
(`translate.pending_items`) minus the items in flight on any connection,
so a **disconnect requeues** that connection's in-flight item and it is
offered to any free capable connection.
- Dispatch re-runs on every relevant event: Hello, result, disconnect and
content change (`_invalidate_pages()` schedules it, so the pass runs
after the writing transaction commits).
- A result with no job in flight, a mismatched (lang, key), a duplicate
hello, or any malformed frame closes the socket with a protocol error.
Results are stored into `trans` in one transaction and set
`node.langs[lang]` on every article they touch (shared chunks make several
pages gain a language from one fragment). Unknown keys are stored anyway
and re-storing overwrites — results are idempotent.
### Explicitly out of scope for phase 2 ### Explicitly out of scope for phase 2
docs/migrate.md
+10 -5
View File
@@ -55,6 +55,9 @@ class Data(msgspec.Struct):
#: API key gating the translator service WebSocket (/_translate/{key}); #: API key gating the translator service WebSocket (/_translate/{key});
#: generated lazily at startup (see the lifespan in app.py). #: generated lazily at startup (see the lifespan in app.py).
translate_key: str = "" translate_key: str = ""
#: Wanted target languages for the translator service (presence-keys);
#: jobs are offered only in these ∩ a connection's capabilities.
translate_langs: dict[str, True] = {}
#: All original-language text, content-addressed: blake3(normalized) #: All original-language text, content-addressed: blake3(normalized)
#: digest[:9] -> Markdown chunk. Shared by every article. Keys are #: digest[:9] -> Markdown chunk. Shared by every article. Keys are
#: bytes; kanta/msgspec base64-encode them at the JSON level. #: bytes; kanta/msgspec base64-encode them at the JSON level.
@@ -94,11 +97,13 @@ alternate links never enumerate chunks. It is written by whoever writes
translation data, in the same transaction: translation data, in the same transaction:
- **Translator service:** the WebSocket API at `/_translate/{key}` (see - **Translator service:** the WebSocket API at `/_translate/{key}` (see
docs/localization.md) pushes pending fragments (titles + translatable docs/localization.md) offers pending fragments (titles + translatable
chunks lacking an entry, deduped by hash) and receives result batches; chunks lacking an entry for the language) as single-item jobs — one at
storing a batch writes `trans[h][lang]` entries, sets a time per connection, in `Data.translate_langs` ∩ the connection's
`node.langs[lang] = True` on every article that gained one and announced capabilities — and receives the matching result; storing it
invalidates the page cache — all in one transaction. writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on
every article that gained one and invalidates the page cache — all in
one transaction.
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"` - **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
sets `node.langs[lang] = True` (patches alone make the version exist). sets `node.langs[lang] = True` (patches alone make the version exist).
- **Removals:** deleting a patch or GC'ing translations re-derives the key: - **Removals:** deleting a patch or GC'ing translations re-derives the key: