diff --git a/docs/localization.md b/docs/localization.md index b00a16a..07a9c0e 100644 --- a/docs/localization.md +++ b/docs/localization.md @@ -222,20 +222,39 @@ wrong or empty key rejects the handshake (close-before-accept → HTTP 403). Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`; `bytes` fields ride as base64): -- `{"type": "hello", "langs": [...]}` — client greeting: the target - languages it handles (normalized to base subtags; `en`/empty dropped). -- `{"type": "job", "lang", "items": [{key, text, path, kind}]}` — server - push: pending fragments (article titles and chunks), deduped by key. -- `{"type": "result", "lang", "items": [{key, text}]}` — client reply: - translated fragments, matched to content by chunk key alone. +- `{"type": "hello", "langs": [...]}` — client greeting announcing its + **capabilities**: the language codes its model can produce (normalized + to base subtags; `en`/empty dropped). +- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push: + ONE fragment to translate (an article title or a chunk). +- `{"type": "result", "lang", "key", "text"}` — client reply: the + translation of the connection's current job, matching it by (lang, key). -The model is **push**, not polling: on `hello` the server sends everything -pending per announced language; afterwards `_invalidate_pages()` (called by -every content/translation write) schedules a delta push of newly pending -items. Outstanding-item tracking is per connection, so a reconnecting client -simply re-receives everything still pending. Results are stored into `trans` -in one transaction and set `node.langs[lang]` on every article they touch -(shared chunks make several pages gain a language from one fragment). +Which languages get translated is **server-configured**: +`Data.translate_langs` (presence-key dict, read/set via `/_api/settings` +as `translate_langs`; no editing UI yet). The dispatcher offers a +connection jobs only in `wanted ∩ capable`; a connection without overlap +simply stays idle. + +Dispatch semantics (all in app.py): + +- **One job at a time per connection** — the next job is sent only after + the current one's result. Clients wanting parallelism open multiple + connections (e.g. several `scripts/translator.py` instances). +- Pending work is derived from the `trans` store + (`translate.pending_items`) minus the items in flight on any connection, + so a **disconnect requeues** that connection's in-flight item and it is + offered to any free capable connection. +- Dispatch re-runs on every relevant event: Hello, result, disconnect and + content change (`_invalidate_pages()` schedules it, so the pass runs + after the writing transaction commits). +- A result with no job in flight, a mismatched (lang, key), a duplicate + hello, or any malformed frame closes the socket with a protocol error. + +Results are stored into `trans` in one transaction and set +`node.langs[lang]` on every article they touch (shared chunks make several +pages gain a language from one fragment). Unknown keys are stored anyway +and re-storing overwrites — results are idempotent. ### Explicitly out of scope for phase 2 diff --git a/docs/migrate.md b/docs/migrate.md index f55bc54..721c121 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -55,6 +55,9 @@ class Data(msgspec.Struct): #: API key gating the translator service WebSocket (/_translate/{key}); #: generated lazily at startup (see the lifespan in app.py). translate_key: str = "" + #: Wanted target languages for the translator service (presence-keys); + #: jobs are offered only in these ∩ a connection's capabilities. + translate_langs: dict[str, True] = {} #: All original-language text, content-addressed: blake3(normalized) #: digest[:9] -> Markdown chunk. Shared by every article. Keys are #: bytes; kanta/msgspec base64-encode them at the JSON level. @@ -94,11 +97,13 @@ alternate links never enumerate chunks. It is written by whoever writes translation data, in the same transaction: - **Translator service:** the WebSocket API at `/_translate/{key}` (see - docs/localization.md) pushes pending fragments (titles + translatable - chunks lacking an entry, deduped by hash) and receives result batches; - storing a batch writes `trans[h][lang]` entries, sets - `node.langs[lang] = True` on every article that gained one and - invalidates the page cache — all in one transaction. + docs/localization.md) offers pending fragments (titles + translatable + chunks lacking an entry for the language) as single-item jobs — one at + a time per connection, in `Data.translate_langs` ∩ the connection's + announced capabilities — and receives the matching result; storing it + writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on + every article that gained one and invalidates the page cache — all in + one transaction. - **Translated-view save:** appending the first patch for `f"{path}:{lang}"` sets `node.langs[lang] = True` (patches alone make the version exist). - **Removals:** deleting a patch or GC'ing translations re-derives the key: