Improved auto translation support #2

2026-09-21T14:20:18Z

LeoVasanko commented

2026-09-21 14:20:18 +00:00

Added full article / with markdown mode for advanced LLMs, with local ollama and API support, new llm_translator script.
Translate navigational titles as a whole (preserving site structure), and then retranslate as article headings with article context - potentially changing nav label
Improved human overrides to translations, in particular when structure is affected

- Added full article / with markdown mode for advanced LLMs, with local ollama and API support, new llm_translator script. - Translate navigational titles as a whole (preserving site structure), and then retranslate as article headings with article context - potentially changing nav label - Improved human overrides to translations, in particular when structure is affected

LeoVasanko added 10 commits 2026-09-21 14:20:18 +00:00

translate: job modes (segments/markdown/article), mode-scoped validation, article decomposition 2a144a7dd6

Hello gains model + modes; Job gains mode. Dispatch routes per connection
capability: article jobs only to article-capable connections and only
while a page is mostly pending; titles/chunks go as markdown (whole
fragment, hybrid-neighbor contexts) or segments jobs. Validation skip
list is now (lang, key, mode). align_article decomposes a whole-article
result into per-chunk stores: non-translatable blocks (fences, HTML,
containers) are verbatim anchors, regions between anchors pair
positionally, mismatched regions and blocks with altered link
destinations/placeholders stay pending.

Verified offline against /tmp/llmtrial outputs: qwen3.8:27b runs pair
105/105 and 43/43 blocks; qwen3:30b-instruct's translated code comments,
qwen3-next's degenerate runs and the structurally broken seedx trial
files (a dropped ::: fence) are all rejected.

scripts/llm_translator.py: instruct-LLM translator client (markdown + article modes) f1324e8387

OpenAI Chat Completions shape for llama.cpp/hosted APIs, ollama native
/api/chat via api="ollama" — ollama's /v1 endpoint silently ignores
think:false (verified on 0.34.2: reasoning ran despite the flag), which
hybrid models need off. Prompts and sampling from the /tmp/llmtrial
evidence (strict structure rules, temperature 0.2, num_predict capped at
~2.5x estimated source tokens); whole-output fence unwrapping and
single-line enforcement for titles are client-side. Config via JSON +
CLI overrides; pagerite itself carries no LLM specifics.

scripts/import_translation.py: import human-made whole-article translations 277a75ae23

Decomposes a pasted full translation with the same align_article
validation as article-mode LLM results and stores proper Data.trans
fragments, instead of the translation editor's one monolithic patch that
silently goes stale hunk by hunk. Runs against the kanta database
directly, with the server stopped. Verified on a scratch copy of the
localhost db: swe/app fi (qwen3.8 trial output) imports 43/43 blocks and
serves through the hybrid; the degenerate qwen3-next output is rejected
on anchor validation.

docs: job modes in localization.md, implementation status in llm-translation.md, new scripts in AGENTS.md bc1e8a8357

import_translation: read the input file outside the event loop (ruff ASYNC230/SIM115) 42609a54c1

llm_translator: one line per rule, extended-Markdown rules, <translate>/<context> markers 37e9ab2cd2

Prompt rules are single-line bullets now (no wrapped continuation lines).
Added the extended-Markdown rules the site syntax needs: all formatting
is syntax and is preserved exactly (only the text is translated), and
single newlines inside paragraphs render as actual line breaks, so line
structure must survive untranslated. The payload is no longer wrapped in
a ```markdown fence — the site's extended syntax (and this document
renderer) makes fences unreliable as delimiters; instead an explicit
sentence marks the instruction/text boundary and the payload rides in
faux <translate> tags (<context> for the hybrid-neighbor reference
blocks). unwrap_output strips echoed markers, still also a whole-output
fence. Re-verified live against ollama qwen3.8:27b: title + article jobs,
URL/code fence/{dates} preserved.

article mode: inject the page title as # heading, jargon loanword rule 859d11491d

When the render would inject the page title as an h1 (markdown.has_h1 is
False for the body), an article job's text carries the same '# {title}'
line: the title translates in document context and the opening paragraphs
see the heading (works both ways). The heading's pair in the decomposed
result becomes the title fragment — heading text only, pure prose, never
stored as a body chunk; a demoted or merged heading skips the title,
which stays pending for a scoped title job. The job covers the title key
so it is not double-dispatched afterwards.

Prompt: prefer established technical loanwords with English roots over
forced localizations (frontend -> frontti in Finnish, not etupääte).

Verified offline (title extraction, demoted-heading skip, no-injection
path) and live against ollama qwen3.8:27b: the standalone title job gave
'Alkuun pääseminen', the article job then overwrote it with the
in-context 'Aloitus' matching the body's translated heading.

llm_translator: self-documenting CLI — --api flag, per-flag expected values and defaults, devserver port in examples 2fef336943

scripts: uv-run shebang + executable; autodetect LLM api shape and language capabilities 911e28efbe

Both new scripts carry the '#!/usr/bin/env -S uv run' shebang like
devserver.py and run directly (scripts/llm_translator.py ws://...).

llm_translator figures out the LLM-side details itself: the endpoint
shape is probed at startup (an ollama server answers /api/version and
gets its native /api/chat; anything else gets /v1/chat/completions) and
the announced languages follow the model family — qwen models announce
the full 39-language table, unknown models a conservative 12-language
set — with --langs/config as user overrides. The --api flag is gone;
CLI options stay high-level (url, --model, --base-url, --langs, --modes,
--api-key, --config for the sampling/cap details).

Verified live: detection logged 'ollama api', all 39 languages
announced, title + article jobs completed as before.

nav job mode: whole-menu titles as one nested list; Kimi Code API backend b1ce15f3cc

- translate.py: new "nav" job mode (Hello.modes opt-in) — the whole
  navigation hierarchy crosses as one nested Markdown list of pending
  titles, decomposed back by align_nav: item count/depth must match or
  the job is rejected wholesale (titles fall back to scoped jobs);
  items failing title checks individually are skipped to scoped jobs.
  Dispatched ahead of per-title jobs; a lone pending title stays scoped.
- article jobs carry the already-translated menu title and parent title
  as contexts, so the injected heading can match the menu while the
  model may adapt the in-article title to the content.
- llm_translator.py: nav mode + nav_prompt; article prompt takes the
  title/location context; API keys from per-provider env vars only
  (KIMI/MOONSHOT/OPENAI_API_KEY, each sent only to its own host;
  LLM_API_KEY generic) — no CLI flag, no config file; Kimi Code /coding
  endpoint support (sampling fields dropped, reasoning_effort from
  config, field-proven with k3-256k at low effort); errors include the
  response body; verbose per-job logging with the raw response incl.
  thinking (stripped from results); Kimi models announce all languages.

LeoVasanko merged commit d0ce619db7 into main

2026-09-21 14:22:00 +00:00

LeoVasanko deleted branch llm-trans

2026-09-21 14:22:00 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LeoVasanko/pagerite#2