Localization #1

Merged
LeoVasanko merged 31 commits from localization into main 2026-09-03 03:54:28 +00:00
2 changed files with 42 additions and 18 deletions
Showing only changes of commit dd0a6cc04b - Show all commits
+32 -13
View File
@@ -222,20 +222,39 @@ wrong or empty key rejects the handshake (close-before-accept → HTTP 403).
Frames are JSON-encoded tagged msgspec structs (`pagerite/translate.py`;
`bytes` fields ride as base64):
- `{"type": "hello", "langs": [...]}` — client greeting: the target
languages it handles (normalized to base subtags; `en`/empty dropped).
- `{"type": "job", "lang", "items": [{key, text, path, kind}]}` — server
push: pending fragments (article titles and chunks), deduped by key.
- `{"type": "result", "lang", "items": [{key, text}]}` — client reply:
translated fragments, matched to content by chunk key alone.
- `{"type": "hello", "langs": [...]}` — client greeting announcing its
**capabilities**: the language codes its model can produce (normalized
to base subtags; `en`/empty dropped).
- `{"type": "job", "lang", "key", "text", "path", "kind"}` — server push:
ONE fragment to translate (an article title or a chunk).
- `{"type": "result", "lang", "key", "text"}` — client reply: the
translation of the connection's current job, matching it by (lang, key).
The model is **push**, not polling: on `hello` the server sends everything
pending per announced language; afterwards `_invalidate_pages()` (called by
every content/translation write) schedules a delta push of newly pending
items. Outstanding-item tracking is per connection, so a reconnecting client
simply re-receives everything still pending. Results are stored into `trans`
in one transaction and set `node.langs[lang]` on every article they touch
(shared chunks make several pages gain a language from one fragment).
Which languages get translated is **server-configured**:
`Data.translate_langs` (presence-key dict, read/set via `/_api/settings`
as `translate_langs`; no editing UI yet). The dispatcher offers a
connection jobs only in `wanted ∩ capable`; a connection without overlap
simply stays idle.
Dispatch semantics (all in app.py):
- **One job at a time per connection** — the next job is sent only after
the current one's result. Clients wanting parallelism open multiple
connections (e.g. several `scripts/translator.py` instances).
- Pending work is derived from the `trans` store
(`translate.pending_items`) minus the items in flight on any connection,
so a **disconnect requeues** that connection's in-flight item and it is
offered to any free capable connection.
- Dispatch re-runs on every relevant event: Hello, result, disconnect and
content change (`_invalidate_pages()` schedules it, so the pass runs
after the writing transaction commits).
- A result with no job in flight, a mismatched (lang, key), a duplicate
hello, or any malformed frame closes the socket with a protocol error.
Results are stored into `trans` in one transaction and set
`node.langs[lang]` on every article they touch (shared chunks make several
pages gain a language from one fragment). Unknown keys are stored anyway
and re-storing overwrites — results are idempotent.
### Explicitly out of scope for phase 2
+10 -5
View File
@@ -55,6 +55,9 @@ class Data(msgspec.Struct):
#: API key gating the translator service WebSocket (/_translate/{key});
#: generated lazily at startup (see the lifespan in app.py).
translate_key: str = ""
#: Wanted target languages for the translator service (presence-keys);
#: jobs are offered only in these ∩ a connection's capabilities.
translate_langs: dict[str, True] = {}
#: All original-language text, content-addressed: blake3(normalized)
#: digest[:9] -> Markdown chunk. Shared by every article. Keys are
#: bytes; kanta/msgspec base64-encode them at the JSON level.
@@ -94,11 +97,13 @@ alternate links never enumerate chunks. It is written by whoever writes
translation data, in the same transaction:
- **Translator service:** the WebSocket API at `/_translate/{key}` (see
docs/localization.md) pushes pending fragments (titles + translatable
chunks lacking an entry, deduped by hash) and receives result batches;
storing a batch writes `trans[h][lang]` entries, sets
`node.langs[lang] = True` on every article that gained one and
invalidates the page cache — all in one transaction.
docs/localization.md) offers pending fragments (titles + translatable
chunks lacking an entry for the language) as single-item jobs — one at
a time per connection, in `Data.translate_langs` ∩ the connection's
announced capabilities — and receives the matching result; storing it
writes the `trans[h][lang]` entry, sets `node.langs[lang] = True` on
every article that gained one and invalidates the page cache — all in
one transaction.
- **Translated-view save:** appending the first patch for `f"{path}:{lang}"`
sets `node.langs[lang] = True` (patches alone make the version exist).
- **Removals:** deleting a patch or GC'ing translations re-derives the key: