Localization #1

2026-09-03T03:54:00Z

LeoVasanko commented

2026-09-03 03:54:00 +00:00

Implement comprehensive content localization, admin panels for editing each language, AI translation interface with automatic updates when base language version is changed.

SEO tags for all language URLs
Uses accept-language by default, ?lang=en overrides temporarily
User edits patched on top of translations
RTL language support

Implement comprehensive content localization, admin panels for editing each language, AI translation interface with automatic updates when base language version is changed. - SEO tags for all language URLs - Uses accept-language by default, ?lang=en overrides temporarily - User edits patched on top of translations - RTL language support

LeoVasanko added 31 commits 2026-09-03 03:54:00 +00:00

Phase 1: content negotiation (?lang=, Accept-Language) + localization/migration design docs 218313ba7a

- pagerite/i18n.py: parse_accept_language, select_language, Translation struct,
  get_translation/available_languages stubs
- app.py: language threaded through show_page/_render_html/_cached_body,
  ETag and Vary: accept-language
- views.py: <html lang>, translated nav titles with fallback, canonical
  with ?lang=, hreflang alternates
- pagerite.js: strip ?lang= via replaceState, send it as Accept-Language
  on fetch-navigation/preloads, lang-aware page cache
- docs/localization.md: negotiation + URL scheme + phase-2 fragment model
- docs/migrate.md: migrate_v3 content-addressed chunk storage plan

Add chunks.py: deterministic block-level Markdown chunking + content keys 8558345eda

Data model v3: content-addressed chunk stores, Patch, Node.chunks/langs 9bc71b8717

migrate_v3: split node content into content-addressed chunks c4e731b3d0

Real translation storage: chunk lookup, user patches, title map 6e9f73f83a

Rewire pages to chunk storage; translated-view open/save as patches 98a1dc8d0f

Mark phase-2 localization storage implemented; list chunks.py in AGENTS.md f63bb2c482

Chunk keys as 9-byte bytes digests; trans nested by hash -> lang cc750236bc

Translator service API: GET/POST /_api/translate/{lang} 10c4c9c4f9

Translator WS protocol structs + pending/store core, Data.translate_key 69962df0c2

Translator API as authed WebSocket /_translate/{key} with delta job push 979504e526

Seed-X translator service client for the /_translate socket 2f78877f15

Document the translator WebSocket API and reference client f6eef1c75f

Translation dispatcher: single-item jobs, capability Hello, wanted-langs setting 1ebd789220

Translator client: full-URL arg, capability Hello, one-job-at-a-time 0610daf1af

Document the translation dispatcher and capability handshake dd0a6cc04b

Sticky ?lang= navigation + site-wide hreflang/canonical tags 14a37f5ab3

- Server replicates a ?lang= override onto the navigation links it
  renders (nav, sidebar, cards, brand), so clicks and prefetches stay in
  the chosen language even without JS; link_lang is part of the ETag and
  body cache key (query and header renders of the same language differ
  in their links).
- pagerite.js drops the Accept-Language header hack: the remembered
  language rides internal fetches as ?lang= instead (added when a link
  lacks one), the page cache keys on path+query, and history/address bar
  keep the pretty query-less URL.
- Canonical names the actually served language (plain URL for the
  original, ?lang= for translations); hreflang alternates are site-wide
  from translate_langs, identical on every page: x-default (the plain
  autodetecting URL) first, then every language explicitly, default
  included, emitted right after canonical before the social tags.

Category pages now localized. af76277d68

Masked translation round-trip; named translator keys 4c886eda86

- pagerite/masking.py: technical spans (code, URLs, {placeholders}, attrs,
  footnote/link labels, container names, HTML tags) become numbered sentinels
  for the LLM round trip; results are restored by number and rejected when a
  sentinel is mangled (skipped for the rest of the run, stays pending).
  Chunks with no prose left after masking are never dispatched.
- Data.translate_key -> translate_keys dict (key -> name); the first key is
  generated at bootstrap, result transactions record the key as user=, and
  startup logs the service URL(s) via translate.log_service_urls.
- Fix /_translate proxying through the Vite dev server (missing slash).

Lang settings tab: flag-grid language picker, bootstrap es+zh targets b1e8f0b454

- New editor-shell tab (lang): translatable languages as clickable flag
  tiles (shared langs.js mapping of the 28 Seed-X languages to flags),
  saved via the settings round-trip; translator service URL(s) shown.
- Bootstrap seeds translate_langs with Spanish and Chinese (the original
  language is never a target).
- GET /_api/settings also returns primary_lang.
- PageEditor's flag/name helpers move to the shared langs.js.

Segments, not sentinels: prose-only translation wire protocol b0866fc4f7

- pagerite/segments.py replaces masking.py: a fragment is parsed with the
  project's own markdown-it (markdown.make_md(verbatim=True), byte-identical
  tokens) and split into pure-prose segments with source spans; only the
  segments plus per-segment context surrounds cross the wire (Job.texts /
  contexts / Result.texts) and translations splice back by offset — markup
  can no longer break, it never leaves the server. Count/empty/non-prose
  results are rejected and skipped for the run.
- Localization machinery out of app.py: the translator dispatcher (clients,
  job pipeline, validation skip-list) moves into translate.Dispatcher;
  translated-edit recording moves into i18n (add_patch, set_title_translation,
  clear_translations). app.py keeps only the routes.
- Structure editor localized: flag strip switches the language titles are
  shown/edited in (GET /_api/pages?lang= flags translated rows, originals
  dimmed); retitling in a translation writes a per-language title fragment
  via StructureOp.lang — slugs, order and hierarchy stay language-independent.
- Localization tab: refresh-all button (DELETE /_api/translations) drops
  machine translations, keeps user patches and clears the skip-list so the
  dispatcher re-translates everything.
- PageEditor always opens in the primary language; editor socket gets
  reconnect/doc-mismatch logging. Reference translator: per-segment calls
  with context prompts, deterministic punctuation matching and the "<"
  markup-bleed cut.

Links translated in sentence context, re-linked by text weight 43793ef9a4

- segments.py: a block of plain text with inline links now stays ONE wire
  segment (Mark/Span) — the link label crosses in sentence context, so
  translations come back grammatically coherent instead of incompatible
  snippets. join() re-inserts the link markdown into the translated block
  at weight-mapped positions (word units; CJK ideographs and kana runs
  count one unit each), degrading to the source label on empty slices and
  still rejecting markup injection. No sentinels on the wire: boundaries
  are found by text processing alone.
- translate.py: the Dispatcher tracks Span (offsets + link marks) for the
  in-flight job.

Shared editor language selection; paced WebSocket reconnects 2a430247e0

- LangSelect.vue: one language selector for the page and structure tabs
  (small flag, clean dropdown), v-modeled on the shell-wide editorLang —
  switching in either panel switches both AND the page preview. While the
  panel is open the selection overrides the normal language preferences
  (swapdoc.setLangOverride: loadPlain and pagerite.js fetches/prefetches
  pin ?lang=, the primary language by its own code); closing restores.
- reconnect.js: every socket (page/banner editors, analytics view, the
  activity channel) now connects through a staggered slot — simultaneous
  attempts at page load (Vite's HMR socket plus ours, repeated on every
  refresh) trip the browser's WebSocket throttling, leaving all sockets
  to the host "pending" for minutes, which was the recurring empty
  editor. A watchdog closes sockets stuck CONNECTING so they reschedule
  through the policy (jittered exponential backoff, reset only by a
  healthy connection) instead of hanging forever; a reconnected editor
  re-opens its document when the previous open died with its socket.

Panels show their socket state; editors locked until the doc arrives 652f1f82ac

- ConnNote.vue: a status strip for the WebSocket-backed panels (page and
  banner editors, analytics view) — "connecting…" while the socket is
  pending (the staggered slot included), "reconnecting in ~N s…" during
  backoff, hidden once open. A pending/lost socket no longer reads as a
  silently empty or stale panel.
- PageEditor/BannerEditor: CodeMirror stays non-editable until the
  page/banner doc has been accepted — typing before it would be clobbered
  by the accept. A mere disconnect keeps the editor live: text stashes
  and pending saves flush on reconnect.

Per-node primary language; home-country flags; grouped lang grid fb159f0cb0

- Node.language sets an article's primary language, inherited down the
  hierarchy, configured per row in the structure tab; English is now a
  regular language in the lang tab, still the root fallback
- Flags map a base tag to the language's home country (en → GB,
  pt → PT); explicit variant tags resolve their own region
- Lang tab flag grid: geographic group rows, shading-only selection

LangSelect dropdown: fixed position, unclipped by the editor panel 87b7591bec

dir=rtl on <html> for RTL page languages 3e72038ed2

Sync <html> lang/dir on hot page swaps (fetch nav, editor re-renders) 2635e8760c

Translator: unload the model after 60s idle, reload on demand 3046988a96

The service stays connected full time; the GPU is held only while
translating. Loads at startup (backlog is likely after a downtime).

Docs: five editor tabs + localization tab, RTL/html-dir, build inputs 081ab979f3

Whole-block segments cover paired formatting (strong/em/s), not just links a47a96ca30

The model mangles Markdown it can see (Seed-X dropped the ** and the
glued-on colon in '**Pagerite**: ...'), so emphasis now crosses stripped,
inline in its sentence, and the syntax is weight-mapped back like links.
Also fix _place_marks swallowing trailing punctuation into the inner text
(end at the slice's last unit) — same-language round trips are now exact.

LeoVasanko merged commit d3e2196c83 into main

2026-09-03 03:54:28 +00:00

LeoVasanko deleted branch localization

2026-09-03 03:54:28 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LeoVasanko/pagerite#1