Localization #1

Merged
LeoVasanko merged 31 commits from localization into main 2026-09-03 03:54:28 +00:00
31 Commits
Author SHA1 Message Date
LeoVasanko a47a96ca30 Whole-block segments cover paired formatting (strong/em/s), not just links
The model mangles Markdown it can see (Seed-X dropped the ** and the
glued-on colon in '**Pagerite**: ...'), so emphasis now crosses stripped,
inline in its sentence, and the syntax is weight-mapped back like links.
Also fix _place_marks swallowing trailing punctuation into the inner text
(end at the slice's last unit) — same-language round trips are now exact.
2026-09-03 03:46:26 +00:00
LeoVasanko 081ab979f3 Docs: five editor tabs + localization tab, RTL/html-dir, build inputs 2026-09-03 03:30:46 +00:00
LeoVasanko 3046988a96 Translator: unload the model after 60s idle, reload on demand
The service stays connected full time; the GPU is held only while
translating. Loads at startup (backlog is likely after a downtime).
2026-09-03 03:24:46 +00:00
LeoVasanko 2635e8760c Sync <html> lang/dir on hot page swaps (fetch nav, editor re-renders) 2026-09-03 03:09:12 +00:00
LeoVasanko 3e72038ed2 dir=rtl on <html> for RTL page languages 2026-09-03 03:06:34 +00:00
LeoVasanko 87b7591bec LangSelect dropdown: fixed position, unclipped by the editor panel 2026-09-03 03:06:34 +00:00
LeoVasanko fb159f0cb0 Per-node primary language; home-country flags; grouped lang grid
- Node.language sets an article's primary language, inherited down the
  hierarchy, configured per row in the structure tab; English is now a
  regular language in the lang tab, still the root fallback
- Flags map a base tag to the language's home country (en → GB,
  pt → PT); explicit variant tags resolve their own region
- Lang tab flag grid: geographic group rows, shading-only selection
2026-09-03 03:01:13 +00:00
LeoVasanko 652f1f82ac Panels show their socket state; editors locked until the doc arrives
- ConnNote.vue: a status strip for the WebSocket-backed panels (page and
  banner editors, analytics view) — "connecting…" while the socket is
  pending (the staggered slot included), "reconnecting in ~N s…" during
  backoff, hidden once open. A pending/lost socket no longer reads as a
  silently empty or stale panel.
- PageEditor/BannerEditor: CodeMirror stays non-editable until the
  page/banner doc has been accepted — typing before it would be clobbered
  by the accept. A mere disconnect keeps the editor live: text stashes
  and pending saves flush on reconnect.
2026-09-03 01:59:35 +00:00
LeoVasanko 2a430247e0 Shared editor language selection; paced WebSocket reconnects
- LangSelect.vue: one language selector for the page and structure tabs
  (small flag, clean dropdown), v-modeled on the shell-wide editorLang —
  switching in either panel switches both AND the page preview. While the
  panel is open the selection overrides the normal language preferences
  (swapdoc.setLangOverride: loadPlain and pagerite.js fetches/prefetches
  pin ?lang=, the primary language by its own code); closing restores.
- reconnect.js: every socket (page/banner editors, analytics view, the
  activity channel) now connects through a staggered slot — simultaneous
  attempts at page load (Vite's HMR socket plus ours, repeated on every
  refresh) trip the browser's WebSocket throttling, leaving all sockets
  to the host "pending" for minutes, which was the recurring empty
  editor. A watchdog closes sockets stuck CONNECTING so they reschedule
  through the policy (jittered exponential backoff, reset only by a
  healthy connection) instead of hanging forever; a reconnected editor
  re-opens its document when the previous open died with its socket.
2026-09-03 01:47:24 +00:00
LeoVasanko 43793ef9a4 Links translated in sentence context, re-linked by text weight
- segments.py: a block of plain text with inline links now stays ONE wire
  segment (Mark/Span) — the link label crosses in sentence context, so
  translations come back grammatically coherent instead of incompatible
  snippets. join() re-inserts the link markdown into the translated block
  at weight-mapped positions (word units; CJK ideographs and kana runs
  count one unit each), degrading to the source label on empty slices and
  still rejecting markup injection. No sentinels on the wire: boundaries
  are found by text processing alone.
- translate.py: the Dispatcher tracks Span (offsets + link marks) for the
  in-flight job.
2026-09-03 01:46:56 +00:00
LeoVasanko b0866fc4f7 Segments, not sentinels: prose-only translation wire protocol
- pagerite/segments.py replaces masking.py: a fragment is parsed with the
  project's own markdown-it (markdown.make_md(verbatim=True), byte-identical
  tokens) and split into pure-prose segments with source spans; only the
  segments plus per-segment context surrounds cross the wire (Job.texts /
  contexts / Result.texts) and translations splice back by offset — markup
  can no longer break, it never leaves the server. Count/empty/non-prose
  results are rejected and skipped for the run.
- Localization machinery out of app.py: the translator dispatcher (clients,
  job pipeline, validation skip-list) moves into translate.Dispatcher;
  translated-edit recording moves into i18n (add_patch, set_title_translation,
  clear_translations). app.py keeps only the routes.
- Structure editor localized: flag strip switches the language titles are
  shown/edited in (GET /_api/pages?lang= flags translated rows, originals
  dimmed); retitling in a translation writes a per-language title fragment
  via StructureOp.lang — slugs, order and hierarchy stay language-independent.
- Localization tab: refresh-all button (DELETE /_api/translations) drops
  machine translations, keeps user patches and clears the skip-list so the
  dispatcher re-translates everything.
- PageEditor always opens in the primary language; editor socket gets
  reconnect/doc-mismatch logging. Reference translator: per-segment calls
  with context prompts, deterministic punctuation matching and the "<"
  markup-bleed cut.
2026-09-03 00:24:07 +00:00
LeoVasanko b1e8f0b454 Lang settings tab: flag-grid language picker, bootstrap es+zh targets
- New editor-shell tab (lang): translatable languages as clickable flag
  tiles (shared langs.js mapping of the 28 Seed-X languages to flags),
  saved via the settings round-trip; translator service URL(s) shown.
- Bootstrap seeds translate_langs with Spanish and Chinese (the original
  language is never a target).
- GET /_api/settings also returns primary_lang.
- PageEditor's flag/name helpers move to the shared langs.js.
2026-09-02 20:27:37 +00:00
LeoVasanko 4c886eda86 Masked translation round-trip; named translator keys
- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).
2026-09-02 19:56:54 +00:00
LeoVasanko af76277d68 Category pages now localized. 2026-09-02 17:48:54 +00:00
LeoVasanko 14a37f5ab3 Sticky ?lang= navigation + site-wide hreflang/canonical tags
- Server replicates a ?lang= override onto the navigation links it
  renders (nav, sidebar, cards, brand), so clicks and prefetches stay in
  the chosen language even without JS; link_lang is part of the ETag and
  body cache key (query and header renders of the same language differ
  in their links).
- pagerite.js drops the Accept-Language header hack: the remembered
  language rides internal fetches as ?lang= instead (added when a link
  lacks one), the page cache keys on path+query, and history/address bar
  keep the pretty query-less URL.
- Canonical names the actually served language (plain URL for the
  original, ?lang= for translations); hreflang alternates are site-wide
  from translate_langs, identical on every page: x-default (the plain
  autodetecting URL) first, then every language explicitly, default
  included, emitted right after canonical before the social tags.
2026-09-02 15:08:42 +00:00
LeoVasanko dd0a6cc04b Document the translation dispatcher and capability handshake 2026-09-02 04:11:40 +00:00
LeoVasanko 0610daf1af Translator client: full-URL arg, capability Hello, one-job-at-a-time 2026-09-02 04:10:06 +00:00
LeoVasanko 1ebd789220 Translation dispatcher: single-item jobs, capability Hello, wanted-langs setting 2026-09-02 04:08:19 +00:00
LeoVasanko f6eef1c75f Document the translator WebSocket API and reference client 2026-09-02 03:30:36 +00:00
LeoVasanko 2f78877f15 Seed-X translator service client for the /_translate socket 2026-09-02 03:24:57 +00:00
LeoVasanko 979504e526 Translator API as authed WebSocket /_translate/{key} with delta job push 2026-09-02 03:22:00 +00:00
LeoVasanko 69962df0c2 Translator WS protocol structs + pending/store core, Data.translate_key 2026-09-02 03:03:44 +00:00
LeoVasanko 10c4c9c4f9 Translator service API: GET/POST /_api/translate/{lang} 2026-09-02 02:16:28 +00:00
LeoVasanko cc750236bc Chunk keys as 9-byte bytes digests; trans nested by hash -> lang 2026-09-02 02:09:27 +00:00
LeoVasanko f63bb2c482 Mark phase-2 localization storage implemented; list chunks.py in AGENTS.md 2026-09-02 01:48:47 +00:00
LeoVasanko 98a1dc8d0f Rewire pages to chunk storage; translated-view open/save as patches 2026-09-02 01:46:27 +00:00
LeoVasanko 6e9f73f83a Real translation storage: chunk lookup, user patches, title map 2026-09-02 01:34:13 +00:00
LeoVasanko c4e731b3d0 migrate_v3: split node content into content-addressed chunks 2026-09-02 01:31:29 +00:00
LeoVasanko 9bc71b8717 Data model v3: content-addressed chunk stores, Patch, Node.chunks/langs 2026-09-02 01:30:04 +00:00
LeoVasanko 8558345eda Add chunks.py: deterministic block-level Markdown chunking + content keys 2026-09-02 01:29:10 +00:00
LeoVasanko 218313ba7a Phase 1: content negotiation (?lang=, Accept-Language) + localization/migration design docs
- pagerite/i18n.py: parse_accept_language, select_language, Translation struct,
  get_translation/available_languages stubs
- app.py: language threaded through show_page/_render_html/_cached_body,
  ETag and Vary: accept-language
- views.py: <html lang>, translated nav titles with fallback, canonical
  with ?lang=, hreflang alternates
- pagerite.js: strip ?lang= via replaceState, send it as Accept-Language
  on fetch-navigation/preloads, lang-aware page cache
- docs/localization.md: negotiation + URL scheme + phase-2 fragment model
- docs/migrate.md: migrate_v3 content-addressed chunk storage plan
2026-09-02 01:14:10 +00:00