ruff

Cache feed/llms.txt bodies, serve with content-hash ETags
Rendering every article per request is expensive. Bodies are now RAM-cached keyed by the public base URL and cleared from _invalidate_pages (the same hook that drops the page render cache on any content/settings write). Responses carry a blake3 content-hash ETag and answer 304, so polling feed readers revalidate cheaply; matching page behavior, 304 revalidations are not recorded in analytics.
2026-09-23 07:32:57 +00:00 · 2026-09-23 07:30:32 +00:00 · 2026-09-23 07:25:08 +00:00 · 2026-09-23 07:12:15 +00:00 · 2026-09-23 07:07:42 +00:00 · 2026-09-23 07:02:17 +00:00
19 changed files with 567 additions and 87 deletions
@@ -16,6 +16,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke
  - `api.py` — editor REST + WS: `/_api/pages`, `/_api/structure`, `/_api/settings`, `/_api/toggle-task`, `/_api/translations`, `/_api/ws/editor`, `/_translate/{key}`.
  - `tracking.py` — visit analytics: GeoIP, client enrichment, favicon fetch, `/_ws`, `/_api/ws/analytics`, the `/_a` page (docs/analytics.md).
  - `pages.py` — public content pages: `/`, `/sitemap.xml`, `/robots.txt`, the `/{path:path}` catch-all.
+  - `feeds.py` — machine-readable exports: `/llms.txt`, `/feed.json` (JSON Feed 1.1), `/feed.xml` (RSS 2.0 + atom:link); all published articles, full content, linked from every page `<head>` and the sitemap, recorded in analytics.
  - `data.py` — msgspec Structs for the kanta database.
  - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md).
  - `i18n.py` — language selection, translation assembly (chunks + overrides) and translated-edit recording (per-chunk user overrides in `Data.overrides`, per-language title overrides, refresh).
@@ -9,7 +9,9 @@ from the kanta content database, path from `PAGERITE_ANALYTICS` (default:
  `Favicon`), the `Store` (raw log + atomic JSON persistence) and
  `Store.display()`, where **all** classification happens.
 - `pagerite/pages.py` — records every served document as one raw GET line
-  (`_record_get`, in `pagerite/tracking.py`) with its true HTTP status.
+  (`_record_get`, in `pagerite/tracking.py`) with its true HTTP status, plus
+  the `/robots.txt` and `/sitemap.xml` machinery GETs (never followed by an
+  activity message, they surface as crawler hits).
 - `pagerite/tracking.py` — the `/_ws` activity WebSocket, and
  `WebSocket /_api/ws/analytics` (admin-gated like every `/_api` endpoint).
 - `frontend/src/pagerite.js` — the client activity channel and the 📊 pen.
@@ -211,7 +213,10 @@ for misses.
  are caught by the abuse rules regardless. In the viewer, crawler hits are
  grouped by client hash and shown as a trail of pages, preceded by the
  referer when there is one (rendered with its favicon like visit
-  referers). The crawler table lists the most recent crawler first, with
+  referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml`,
+  `/llms.txt`, the feeds) appear as
+  emoji-marked steps (🤖 / 🗺️ / 🧠 / 📡) so they stand out from article steps.
+  The crawler table lists the most recent crawler first, with
  the most active as a tie-breaker.
 - **Abuse (scanner) hits**: a 404 on a telltale path — an empty URL segment
  (`//foo` — no real client generates those), any segment starting with a
@@ -438,8 +443,11 @@ map: each visit is attributed to `utm_campaign`, then `utm_source`, then the
 referer origin, then any other `utm_*` tag, so UTM-tagged visits are grouped
 under their campaign/source value rather than the referer domain. A UTM
 source node only links to its referer when every visit carrying that tag
-came from the same origin. External exits are full-size nodes in a matching
-row centered below the map, so the site itself stays in the middle), per-page view
+came from the same origin. Within the source and exit rows the pills are
+not sorted by count; each slides sideways toward the pages it connects to,
+minimizing the weighted horizontal connection distance while keeping a
+minimum pill spacing. External exits are full-size nodes in a matching
+row below the map, so the site itself stays in the middle), per-page view
 counts, the top transitions and the 50 most recent visit trails. Data is
 streamed live over `WebSocket /_api/ws/analytics`, which pushes the latest
 JSON snapshot on connect and again whenever the analytics file is updated
@@ -10,6 +10,7 @@ Thin FastAPI assembly: lifespan (open the kanta database, load the file store, t
 - `files.py` — the `FileStore` and image derivative helpers, and the file routes: `/_api/files`, `/_f/`, `/_themes/`, `/_fonts/`, the favicon settings endpoints.
 - `api.py` — the editor REST API and WebSockets: `/_api/pages`, `/_api/structure`, `/_api/settings`, `/_api/toggle-task`, `/_api/translations`, `/_api/ws/editor`, and the translator channel `/_translate/{clientkey}`.
 - `tracking.py` — visit analytics: GeoIP, client enrichment, favicon fetching, debounced broadcasts, the `/_ws` activity socket, the admin stream `/_api/ws/analytics`, and the `/_a` viewer page.
+- `feeds.py` — machine-readable site exports: `/llms.txt` (Markdown site map for LLM agents), `/feed.json` (JSON Feed 1.1) and `/feed.xml` (RSS 2.0 + atom:link), all carrying every published article with full content (relative URLs absolutized), linked from every page's `<head>` and from the sitemap, and recorded in analytics like page GETs. Bodies are RAM-cached (keyed by base URL, cleared by `_invalidate_pages` like the page render cache) and served with a content-hash ETag (304 revalidation).
 - `pages.py` — the public content pages: `/`, `/sitemap.xml`, `/robots.txt` and the `/{path:path}` catch-all.

 Route ordering is load-bearing and lives in `app.py`: the api/tracking/files routers are included BEFORE `frontend.route(app, "/")` is called — fastapi-vue inserts its file routes at the position where `route()` was called (during `load()` in the lifespan), so anything registered earlier wins. The content catch-all `/{path:path}` is included AFTER `frontend.route()` so that built frontend assets still take priority over content slugs. The `Frontend` is constructed with `spa=False` explicitly: it only serves the built files without a catch-all.
@@ -34,11 +35,11 @@ markdown-it-py renderer (html passthrough + attrs, footnote, deflist, tasklists,

 The shared page layout as an html5tagger `Template` with placeholders (`Title`, `Brand`, `Banner`, `Nav`, `Sidebar`, `Main`), nav rendering straight from the `Data.menu` tree (siblings sorted by `Node.order`; nav links to content-less labels point at their first child via `first_leaf`, the first published descendant with content), and page/404 rendering.

-Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text; the card image is the node's own `Node.image` when one resolves (nearest ancestor, front page last — see docs/content-model.md), otherwise mined from the article, preferring a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG; the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`SITE_URL` — `https://<hostname>` from the CLI hostname argument; on localhost the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. Additionally `twitter:image` pins extension-less `/_f/{hash}` card images to the `.webp` variant — X only honors WebP via twitter:image (not og:image) and its scraper cannot be trusted to negotiate via Accept. `twitter:card` is `summary_large_image` when the image's probed store dimensions suit a large card (>= 600px wide, aspect between 1.4 and 2.5; dimensions are read from the `<hash>.webp` derivative via pyvips, cached per hash) and `summary` for small or portrait images — external or unprobeable images keep the presence-based default (large when an image exists). The node's `Node.large` setting overrides that pick per article (False = small, True = large, the default None = automatic; NOT inherited like `Node.image`). The page title is injected as `# {title}` when the markdown has no h1 of its own, so it never appears twice (it always supplies `<title>` and nav labels).
+Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text; the card image is the node's own `Node.image` when set, otherwise mined from the article (preferring a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG), and only when the article yields none the inherited one (nearest ancestor, front page last — see docs/content-model.md); the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`SITE_URL` — `https://<hostname>` from the CLI hostname argument; on localhost the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. Additionally `twitter:image` pins extension-less `/_f/{hash}` card images to the `.webp` variant — X only honors WebP via twitter:image (not og:image) and its scraper cannot be trusted to negotiate via Accept. `twitter:card` is `summary_large_image` when the image's probed store dimensions suit a large card (wider than 600px, taller than 400px and clearly wider than tall — aspect > 1.05, so square and portrait images keep the compact layout; dimensions are read from the `<hash>.webp` derivative via pyvips, cached per hash) and `summary` otherwise — external or unprobeable images keep the presence-based default (large when an image exists). The node's `Node.large` setting overrides that pick per article (False = small, True = large, the default None = automatic; NOT inherited like `Node.image`). The page title is injected as `# {title}` when the markdown has no h1 of its own, so it never appears twice (it always supplies `<title>` and nav labels).

 The navbar holds top-level items only; the current section's subitems go to a left `#sidebar` as a nested list (the section's direct children plain, deeper levels indented with article-list-style markers), rendered only from the second level down — main-level pages list their children as cards after the content instead. Below that, the sidebar renders when the section offers at least two published items, or exactly one while viewing anything other than that only page — the section index, a 404, a grandchild (so those pages can reach the child), and also on that only page itself when it has published children of its own; no aside element at all on the front page, main-level pages, leaf pages and the sole childless page of a one-page section. Also, category labels are nodes without content — None *or* empty markdown — and their nav links point at their first child page. Dynamic regions have stable ids (`#page-banner`, `#nav`, `#sidebar`, `#main`) for fetch-navigation swaps (`#sidebar` may be absent on either side of a swap).

-Any page with published children — a category page — lists them as a card grid (`nav.cards`) after the markdown content, as does the content-less category 404. Each card links to the child page (a content-less child to its first leaf) and shows the child's card image (its resolved `Node.image` when set, else the same hero → first raster → first SVG heuristics as `og:image`). The layout follows the same selection as `twitter:card` (_card_large — the child's per-article `Node.large` override, else the image's probed dimensions): large cards show the image as a full-card cover with the title overlaid, small cards (`.card.compact`) split horizontally at the golden ratio (two sub-grids, top φ : bottom 1): the square image fills the top part with the title beside it at its bottom, the article description (which only the small format carries) tops the bottom part — title and description carry the translucent band (the same band color as the large cards' title) as their own background; imageless cards keep the image space as a blank gradient. Each card carries the target article's language as its `lang` (the page language when the target is translated into it, else the target's primary language — matching the per-card text fallback) so the clamped title/description hyphenate correctly (`hyphens: auto`).
+Any page with published children — a category page — lists them as a card grid (`nav.cards`) after the markdown content, as does the content-less category 404. Each card links to the child page (a content-less child to its first leaf) and shows the child's card image (its own `Node.image` when set, else the same hero → first raster → first SVG heuristics as `og:image`, else the inherited image). The layout follows the same selection as `twitter:card` (_card_large — the child's per-article `Node.large` override, else the image's probed dimensions): large cards show the image as a full-card cover with the title overlaid, small cards (`.card.compact`) split horizontally at the golden ratio (two sub-grids, top φ : bottom 1): the square image fills the top part with the title beside it at its bottom, the article description (which only the small format carries) tops the bottom part — title and description carry the translucent band (the same band color as the large cards' title) as their own background; imageless cards keep the image space as a blank gradient. Each card carries the target article's language as its `lang` (the page language when the target is translated into it, else the target's primary language — matching the per-card text fallback) so the clamped title/description hyphenate correctly (`hyphens: auto`).

 ## `seed.py`

@@ -24,7 +24,7 @@ Files are content-addressed (blake3[:12] + extension) and stored **on disk** und

 ## Card images

-`Node.image` names a content-addressed store file (12-hex hash, served at `/_f/{name}`) used as the page's card image: `og:image`/`twitter:image` meta and the card cover in listings. Empty inherits the nearest ancestor's image, the front page last; unset everywhere, the meta tags fall back to mining the rendered article (hero → first raster → first SVG). Set in the editor's banner panel (upload → `PUT /_api/files/{name}`, then a `save` with `image` over the editor WebSocket), stored at `IMAGE_MAXSIZE` like other uploads. `twitter:card` picks `summary_large_image` vs `summary` from the image's probed dimensions (views.py `_image_dims`).
+`Node.image` names a content-addressed store file (12-hex hash, served at `/_f/{name}`) used as the page's card image: `og:image`/`twitter:image` meta and the card cover in listings. The effective image follows the priority: the node's own `image`, then one mined from the rendered article (hero → first raster → first SVG), then the inherited image (the nearest ancestor's, the front page last). The special value `@favicon` resolves to the site icon (`Data.favicon`) at render time — it follows favicon changes rather than copying the current icon. Set in the editor's banner panel (upload → `PUT /_api/files/{name}` or paste from the pasteboard — an image uploads, an image URL is sent as the `image` setting itself and fetched/stored server-side by the save handler, since cross-origin URLs are CORS-blocked for the browser — then a `save` with `image` over the editor WebSocket; the site-icon button saves `@favicon`), stored at `IMAGE_MAXSIZE` like other uploads. `twitter:card` picks `summary_large_image` vs `summary` from the image's probed dimensions (views.py `_image_dims`).

 `Node.large: bool | None` overrides the automatic card-mode pick per article: None = automatic, False forces a small card, True a large one. Unlike `image`, it is NOT inherited down the tree. Set from the banner panel's card previews (a `save` with `large` over the editor WebSocket).

@@ -27,7 +27,7 @@ Pagerite is a single-user CMS/blog. This document records the initial high-level
 - All pages share one static layout, defined once as an **html5tagger Template** with capitalized placeholders (`Title`, `Banner`, `Nav`, `Sidebar`, `Main`) filled per request. The dynamic regions carry stable ids (`#page-banner`, `#nav`, `#sidebar`, `#main`).
 - The page top is a **full-width banner header** with the site name and the navigation bar overlaid on it — no separate chrome header. The banner combines two layers, stacked in `#page-banner` (a grid, so they overlay): first the **banner design** — a named design living in a theme folder (`pagerite/themes/{name}/banner.css` plus artwork as `banner.html` — arbitrary markup like canvas + style + script — or `banner.svg`), chosen per page via `Node.banner_design` (a design name, "" for none, None to inherit from the nearest ancestor, then the front page, then the active theme's own design). The artwork is inlined into a `div[data-design]` wrapper: SVG artwork can be recolored from the theme stylesheet (corporate's single SVG serves both light and dark mode via `var()`-driven stops). Second, **per-page author code**: `Node.banner` holds an arbitrary trusted HTML snippet (an image, a styled div, canvas + script — anything), resolved by walking up the node's ancestors to the front page and rendered **after** the design artwork, so author styles always win over the design's own. The base stylesheet falls back to a plain gradient. There is deliberately no scrim fading the banner into the page background — any such fade would ruin user-supplied designs; themes that want one bake it into their SVG (purple does).
 - **Fetch-navigation.** Links are plain `<a href>`; a small script (`frontend/src/pagerite.js`) intercepts same-origin clicks, fetches the page, and swaps the `#page-banner`, `#nav`, `#sidebar` and `#main` regions, the document title, and the site-wide custom CSS (`<style id="pagerite-user">` in `<head>`), keeping the rest of `<head>` and the layout chrome. Without JS everything works as normal page loads. Scripts inside fetched banner and content regions are re-created so they execute. Swaps run inside `document.startViewTransition` for the page transition selected in the site settings (`Data.transition`; the `cube` design — CSS adapted from termotohtori.fi, fragile, do not tweak — rotates, mirrored on browser back; `crossfade` fades; both skipped under `prefers-reduced-motion`). With `cube`, navigation within the same top-level section crossfades instead of rotating.
- **The site structure is a tree of labels.** `Data.menu` holds the top-level items by slug, each with `children` keyed by slug — the URL path is the slug chain. The front page is a top-level node with slug "" (an item *parallel* to the other main level pages, not their parent) and cannot have children. The header navbar holds only the top level; a top-level item is highlighted when viewing any of its subpages. A page with published children lists them as **cards** after its content (the child page's card image as the cover — its resolved `Node.image` when set, else mined like the og tags — laid out by the child's card-mode selection: full-card cover with the title overlaid, or a golden-ratio split with a square image and the title in the top part, the description below it on a translucent band); a **left sidebar** (`#sidebar`) with the section's sub-navigation appears only from the second level down, when there is something to navigate — main-level pages, sections with fewer than two published items, leaf pages and the front page render no aside element at all. Other sections' subitems are never shown without navigating into them first.
+- **The site structure is a tree of labels.** `Data.menu` holds the top-level items by slug, each with `children` keyed by slug — the URL path is the slug chain. The front page is a top-level node with slug "" (an item *parallel* to the other main level pages, not their parent) and cannot have children. The header navbar holds only the top level; a top-level item is highlighted when viewing any of its subpages. A page with published children lists them as **cards** after its content (the child page's card image as the cover — its own `Node.image` when set, else mined like the og tags, else the inherited one — laid out by the child's card-mode selection: full-card cover with the title overlaid, or a golden-ratio split with a square image and the title in the top part, the description below it on a translucent band); a **left sidebar** (`#sidebar`) with the section's sub-navigation appears only from the second level down, when there is something to navigate — main-level pages, sections with fewer than two published items, leaf pages and the front page render no aside element at all. Other sections' subitems are never shown without navigating into them first.
 - **Landing pages are optional.** Every label can either have content (`Node.content`, a Markdown page) or none — a content-less label renders a 404 page listing its children as cards (with a pen to create the landing page) instead of redirecting, while nav links to it point straight at its first child, so categories need no filler content and normal navigation never sees the 404. Title and slug of every label are editable; renaming a slug moves the whole subtree. The sidebar never lists the section itself, avoiding title duplication with the navbar.
 - **Menu order is manual.** Each node has a fractional `order` key among its siblings; reordering/moving writes only the moved node (it takes a fresh value halfway between its new siblings; all other items keep theirs). New pages append at the end of their menu. Structure edits (reorder, move/rename with the whole subtree, retitle) go through `POST /_api/structure` and the editor's structure panel.
 - Unpublished pages are hidden from both nav and URL access (404).
@@ -7,7 +7,7 @@ The Vue editor is a single tabbed `EditorShell.vue` mounted in a host div create
 The shell hosts five kept-alive tabs (ordered site-wide first — site, structure, localization — then, after a visual break, the per-page tabs — article, banner):

 - `PageEditor.vue` — CodeMirror + server-rendered preview over WebSocket `/_api/ws/editor`, previewing into the visible article; editor and article scrolls are linked piecewise-linearly, keyed on the section anchors' `data-line` (markdown source line the backend stamps on top-level anchored h1/h2s): the page follows the cursor (fractional, wrap-aware, scrolling only when the cursor's page position leaves the viewport, with an edge margin), the editor follows page scroll with a progress-based viewport anchor, applied instantly (the window keeps scrolling normally while any editor is open — the panel is fixed to the viewport's left edge, its top tracking the banner's bottom edge until the banner scrolls away — and the panel scrolls internally); anchored h2s carry their own edit pens that open the editor scrolled to that section; a format bar offers Markdown helpers — bold/italic/code/link/table/image upload (always block-level on a fresh blank-separated line of its own — a cursor on a non-empty line, e.g. inside an existing image tag, inserts after that line, never into it; always with an empty `""` caption, cursor inside the quotes), toggling fences (` ``` ` code blocks and `::: aside` containers share the same machinery: clicked inside one they remove it and select the content, otherwise they wrap the selection or the cursor's line, keeping it selected), and `.left`/`.right`/`.wide`/`.margin` placement toggles plus `.small`/`.large`/`.huge` text-size toggles (brace attributes on the block at the cursor, mutually exclusive within each group; on `:::` containers a placement class replaces the container name instead — `::: aside` → `::: margin`), with Ctrl/Cmd-B/I/S bindings — for the hard-to-remember syntax. Edits content and title only, never the path.
- `BannerEditor.vue` — per-page banner HTML + banner design selector, previewed into `#page-banner`, plus the page's card image (`Node.image`, inherited by the subtree): just an upload button and a ✕ clearing the node's own (back to inherit) — the label states which image is in use (none / inherited from … / set for this article, “used in /<path>/*” when it has children / mined from the article) and the card previews below show it. Below it, the site's own cards preview in both modes (small and large) with the real `.card` markup and styles from pagerite.css — theme variables included, they are the site's look — scaled down via font-size (the card internals are all em, so the layout proportions match real cards exactly); both render the effective card image (the resolved node image, else the image the server mines from the article), and the description appears only in the small format, like the backend's `_card`. The previews double as the card-mode selector for the per-article `Node.large` override: clicking one forces that mode (thin solid outline), clicking the selected one returns to automatic; under automatic the mode auto currently resolves to gets a dashed marker (both outlines — selection never shifts the layout), approximated from image presence only (the server's dimension probe is not available in the panel).
+- `BannerEditor.vue` — per-page banner HTML + banner design selector, previewed into `#page-banner`, plus the page's card image (`Node.image`, inherited by the subtree): upload and paste (pasteboard: an image uploads, an image URL is fetched and stored server-side via the `save` socket message) buttons, a site-icon button (saves `@favicon`, which resolves to the current site icon at render time, following favicon changes) and a ✕ clearing the node's own (back to inherit) — the label states which image is in use (none / set for this article, “used in /<path>/*” when it has children / the site icon / mined from the article / inherited from …) and the card previews below show it. Below it, the site's own cards preview in both modes (small and large) with the real `.card` markup and styles from pagerite.css — theme variables included, they are the site's look — scaled down via font-size (the card internals are all em, so the layout proportions match real cards exactly); both render the effective card image (the node's own, else the image the server mines from the article, else the inherited one), and the description appears only in the small format, like the backend's `_card`. The previews double as the card-mode selector for the per-article `Node.large` override: clicking one forces that mode (thin solid outline), clicking the selected one returns to automatic; under automatic the mode auto currently resolves to gets a dashed marker (both outlines — selection never shifts the layout), approximated from image presence only (the server's dimension probe is not available in the panel).
 - `SiteEditor.vue` — site brand + optional custom brand HTML with image/video upload + theme selector + page-transition selector + font picker + favicon upload — clicking the preview tile picks a new one — + site-wide custom CSS, CSS injected into `<head id="pagerite-user">`.
 - `StructureEditor.vue` — the vue-draggable structure tree with always-editable title/slug inputs per row and a per-row flag dropdown setting the page's primary language (`Node.language`, inherited by the subtree).
 - `LocalizationEditor.vue` — the site-wide translation settings: target languages as a flag grid (toggles, grouped in geographic rows; see docs/localization.md), the refresh-all-translations button, and the translator service WebSocket URL(s) to connect `scripts/translator.py` to.
@@ -67,42 +67,57 @@ const bannerDesignInherited = ref('')
 let refreshOnSave = null

 // --- Card image (Node.image, '' = inherit, like the banner design) ------
-// The node's own setting, the effective image after inheritance ("" =
-// none) and which node supplied an inherited one ("" = the front page,
-// "" also when own/none — mirrors bannerFrom).
+// The node's own setting ('@favicon' = the site icon, resolved live
+// against the favicon ref), the effective image after inheritance ("" =
+// none, already resolved server-side) and which node supplied an
+// inherited one ("" = the front page, "" also when own/none — mirrors
+// bannerFrom).
 const image = ref('')
 const imageResolved = ref('')
 const imageSource = ref('')
+// The site icon (bare store hash, "" = none): the "@favicon" own setting
+// resolves against it, live, in the previews.
+const favicon = ref('')
 // The image the server would mine from the article itself — the card
-// previews fall back to it when no node image resolves (mirrors og:image).
+// previews fall back to it when the node has no image of its own, and it
+// beats an inherited one (mirrors og:image).
 const imageMined = ref('')
 // Whether the page has children (from the doc message): an own share
 // image is inherited by the whole section.
 const hasChildren = ref(false)
 // The block label states which image is currently in use.
 const imageLabel = computed(() => {
+  if (image.value === '@favicon') {
+    return 'card image: the site icon (follows favicon changes)'
+  }
  if (image.value) {
    return hasChildren.value
      ? `card image: set for this article — used in /${path.value}/*`
      : 'card image: set for this article'
  }
+  if (imageMined.value) return 'card image: from the article'
  if (imageResolved.value) {
    const where = imageSource.value === '' ? 'the front page' : `/${imageSource.value}`
    return `card image: inherited from ${where}`
  }
-  if (imageMined.value) return 'card image: from the article'
  return 'card image: none'
 })
 // The page title and description (from the doc message) feed the mock card
 // previews; empty shows placeholder bars / text instead.
 const pageTitle = ref('')
 const pageDesc = ref('')
-// The image the Twitter cards preview with: the resolved node card image,
-// else the mined article image (what og:image would use). image_resolved is
-// a bare store hash; image_mined is already a src path.
-const cardImage = computed(() =>
-  imageResolved.value ? `/_f/${imageResolved.value}` : imageMined.value,
-)
+// The image the Twitter cards preview with: the node's own card image
+// ("@favicon" resolves to the site icon), else the mined article image,
+// else the inherited node image (what og:image would use).
+// image/image_resolved are bare store hashes; image_mined is already a
+// src path.
+const cardImage = computed(() => {
+  // "@favicon" with no site icon configured falls through like no own
+  // image at all (mirrors the backend's resolution).
+  const own = image.value === '@favicon' ? favicon.value : image.value
+  if (own) return `/_f/${own}`
+  return imageMined.value || (imageResolved.value ? `/_f/${imageResolved.value}` : '')
+})
 // Card-mode override (Node.large, per-article, NOT inherited):
 // null = automatic, false = small, true = large.
 const large = ref(null)
@@ -144,13 +159,39 @@ async function uploadCardImage(ev) {
  const file = ev.target.files[0]
  ev.target.value = '' // allow re-picking the same file
  if (!file || !file.type.startsWith('image/')) return
-  const name = file.name.replace(/[^\w.-]/g, '-')
-  const res = await apiFetch(`/_api/files/${encodeURIComponent(name)}`, { method: 'PUT', body: file })
+  storeCardImage(file, file.name)
+}
+
+async function storeCardImage(blob, filename) {
+  const name = filename.replace(/[^\w.-]/g, '-')
+  const res = await apiFetch(`/_api/files/${encodeURIComponent(name)}`, { method: 'PUT', body: blob })
  if (!res.ok) return
  const { path: stored } = await res.json() // "/_f/<hash>[.ext]"
  saveImage(stored.split('/').pop().split('.')[0])
 }

+async function pasteCardImage() {
+  // The pasteboard button (unlike pasting into a text editor, where the
+  // paste event carries files) must read the clipboard explicitly.
+  try {
+    for (const item of await navigator.clipboard.read()) {
+      const type = item.types.find((t) => t.startsWith('image/'))
+      if (type) {
+        const blob = await item.getType(type)
+        await storeCardImage(blob, `paste.${type.split('/')[1].replace('+xml', '')}`)
+        return
+      }
+    }
+    // No image on the pasteboard: a pasted URL goes as the setting
+    // itself — the server fetches and stores it (cross-origin URLs are
+    // CORS-blocked for the browser).
+    const text = (await navigator.clipboard.readText()).trim()
+    if (/^https?:\/\/\S+$/.test(text)) saveImage(text)
+  } catch {
+    // Clipboard read denied or empty: nothing to do.
+  }
+}
+
 // The inherit option names the design actually in effect and its source.
 const inheritLabel = computed(() => {
  if (bannerDesignFrom.value === null) {
@@ -326,6 +367,7 @@ function onMessage(ev) {
    imageResolved.value = msg.image_resolved ?? ''
    imageMined.value = msg.image_mined ?? ''
    imageSource.value = msg.image_source ?? ''
+    favicon.value = msg.favicon ?? ''
    hasChildren.value = msg.has_children ?? false
    large.value = msg.large ?? null
    pageTitle.value = msg.title ?? ''
@@ -451,6 +493,19 @@ onUnmounted(() => {
          title="upload card image (og:image / card covers) — the subtree inherits it"
          @click="imageInput.click()"
        >🖼︎</button>
+        <button
+          type="button"
+          class="icon-btn"
+          title="paste a card image from the pasteboard (image or image URL)"
+          @click="pasteCardImage"
+        >📋</button>
+        <button
+          v-if="favicon"
+          type="button"
+          class="icon-btn favicon-btn"
+          title="use the site icon as the card image — follows favicon changes"
+          @click="saveImage('@favicon')"
+        ><img :src="`/_f/${favicon}`" alt="site icon" /></button>
        <input
          ref="imageInput"
          type="file"
@@ -611,6 +666,14 @@ onUnmounted(() => {
  font-size: 1rem;
 }

+/* The "use site icon" button shows the icon itself. */
+.favicon-btn img {
+  display: block;
+  width: 1rem;
+  height: 1rem;
+  object-fit: contain;
+}
+
 /* The first icon button pushes itself (and any siblings after it, like
   the card-image clear button) to the end of the row. */
 .block-head .icon-btn:first-of-type {
@@ -160,9 +160,32 @@ function externalOrigin(url) {
  }
 }

+// Non-article machinery paths shown in trails with an emoji marker:
+// recorded like page GETs but fetched by crawlers/feed readers, so they
+// surface in the crawler rows. Feed paths are pre-registered for the
+// future RSS/Atom routes.
+const MACHINE_STEPS = {
+  '/robots.txt': ['🤖', 'robots.txt'],
+  '/sitemap.xml': ['🗺️', 'sitemap.xml'],
+  '/llms.txt': ['🧠', 'llms.txt'],
+  '/feed.json': ['📡', 'feed.json'],
+  '/feed.xml': ['📡', 'feed.xml'],
+  '/rss.xml': ['📡', 'rss.xml'],
+  '/atom.xml': ['📡', 'atom.xml'],
+  '/feed': ['📡', 'feed'],
+}
+
 /** Format one trail step: an internal page or an external https origin. */
 function stepOf(path, titles) {
  if (path?.startsWith('/')) {
+    // Known non-article machinery GETs (fetched by crawlers and feed
+    // readers, recorded like page GETs): emoji-marked so they stand out
+    // from article steps in the trails.
+    const machine = MACHINE_STEPS[path]
+    if (machine) {
+      const [emoji, name] = machine
+      return { path, slug: `${emoji} ${name}`, title: name, external: false, machine: true }
+    }
    return { path, slug: slugOf(path), title: titles.get(path) || '', external: false, home: path === '/' }
  }
  if (path?.startsWith('https://')) {
@@ -18,7 +18,9 @@
 * connections. Animated beads flow along every edge in each direction,
 * emitted at time intervals inversely proportional (linear) to the
 * directional count.
- * External sources appear as nodes in a row above the map. Sources are
+ * External sources appear as nodes in a row above the map, each pill
+ * slid sideways toward the pages it connects to (weighted by count,
+ * minimum spacing kept). Sources are
 * identified from visit records in this order: utm_campaign, utm_source,
 * referer, then other utm_* tags. Visits with a UTM tag are grouped under
 * that tag's value, not under the referer domain. A UTM source node only
@@ -700,6 +702,51 @@ function buildInternalEdges(pairs, byPath, dayScale = 1) {
  return { edges, flows }
 }

+/**
+ * Weighted median of {x, w} targets: the x minimizing Σ w|x − t| —
+ * the spot where a pill's total horizontal connection pull balances.
+ */
+function weightedMedian(targets) {
+  const ts = [...targets].sort((a, b) => a.x - b.x)
+  let total = 0
+  for (const t of ts) total += t.w
+  let acc = 0
+  for (const t of ts) {
+    acc += t.w
+    if (acc >= total / 2) return t.x
+  }
+  return ts[ts.length - 1].x
+}
+
+/**
+ * Slide the pills of an external row sideways so each sits as close as
+ * possible to the pages it connects to: minimize Σ w|pill.x − target.x|
+ * over all drawn connections (weight = count), subject to a minimum
+ * center `spacing` (no overlap). Pills are ordered by their weighted
+ * median target (swapping any inverted adjacent pair can only add
+ * crossing distance), then positioned by coordinate descent on this
+ * convex objective: each pass snaps a pill to its weighted median,
+ * clamped to the spacing window between its current neighbors.
+ */
+function placeRow(items, spacing) {
+  items.sort((a, b) => a.anchor - b.anchor)
+  const n = items.length
+  for (const it of items) it.x = it.anchor
+  for (let pass = 0; pass < 40; pass++) {
+    let moved = 0
+    const sweep = (i) => {
+      const lo = i > 0 ? items[i - 1].x + spacing : -Infinity
+      const hi = i < n - 1 ? items[i + 1].x - spacing : Infinity
+      const x = Math.min(Math.max(items[i].anchor, lo), hi)
+      moved = Math.max(moved, Math.abs(x - items[i].x))
+      items[i].x = x
+    }
+    for (let i = 0; i < n; i++) sweep(i)
+    for (let i = n - 1; i >= 0; i--) sweep(i)
+    if (moved < 0.01) break
+  }
+}
+
 const UTM_PRIORITY = ['utm_campaign', 'utm_source']
 const UTM_FALLBACK = ['utm_medium', 'utm_content', 'utm_term', 'utm_id']

@@ -764,10 +811,12 @@ function collectSourcePairs(visits) {
 * Place external source and exit nodes and build their edges and bead
 * flows.
 * Sources (incoming links) are derived from visit UTM/referer data and form
- * a row centered above the map, hottest first; exits come from the
- * transition matrix and form a matching row centered below the map, so
+ * a row above the map; exits come from the
+ * transition matrix and form a matching row below the map, so
 * the site itself stays in the middle. Both rows sit EXT_GAP beyond the
- * map's bounds.
+ * map's bounds. Within a row the pills slide sideways toward the pages
+ * they connect to (placeRow), minimizing the weighted horizontal
+ * connection distance while keeping a minimum center spacing.
 * Widths and pruning use the same log scale and traffic-share rule as
 * internal connections.
 */
@@ -785,9 +834,10 @@ function buildExternal({ sources, exits }, byPath, innerBounds, dayScale = 1) {

  const width = (count) => scaledWidth(count * dayScale)

-  // Incoming: one source node per identified source, in a row centered
-  // above the map, with an edge to each page that source led to. A source
-  // whose connectors are all culled (below MIN_WMID) is dropped itself.
+  // Incoming: one source node per identified source, in a row above the
+  // map, with an edge to each page that source led to. A source whose
+  // connectors are all culled (below MIN_WMID) is dropped itself. Pills
+  // slide sideways toward their connection targets (see placeRow).
  const bySource = new Map() // source -> pairs, sorted by total incoming count
  for (const p of liveSources.filter((p) => p.in >= minCount)) {
    const g = bySource.get(p.source) || []
@@ -804,20 +854,25 @@ function buildExternal({ sources, exits }, byPath, innerBounds, dayScale = 1) {
    }))
    .sort((a, b) => b.total - a.total)
    .slice(0, MAX_EXT_IN)
-    .filter(({ ps }) =>
-      ps.some((p) => !byPath.get(p.page).hidden && width(p.in) >= MIN_WMID))
+    .map((o) => ({
+      ...o,
+      // Anchoring targets: the drawn (non-culled) connections only.
+      targets: o.ps
+        .filter((p) => !byPath.get(p.page).hidden && width(p.in) >= MIN_WMID)
+        .map((p) => ({ x: byPath.get(p.page).x, w: p.in })),
+    }))
+    .filter((o) => o.targets.length)
  if (origins.length) {
-    const cx = (innerBounds.x0 + innerBounds.x1) / 2
    const y = innerBounds.y0 - TNODE_BOUND - EXT_GAP
-    const spacing = TNODE_W + 44
-    const x0 = cx - ((origins.length - 1) * spacing) / 2
-    origins.forEach(({ source, ps, total, href, isUtm }, i) => {
+    for (const o of origins) o.anchor = weightedMedian(o.targets)
+    placeRow(origins, TNODE_W + 44)
+    origins.forEach(({ source, ps, total, href, isUtm, x }) => {
      const label = isUtm ? source : extLabel(source)
      const xn = {
        path: source,
        href,
        label, // clipped at the pill border on render
-        x: x0 + i * spacing,
+        x,
        y,
        count: total,
        kind: 'source',
@@ -836,10 +891,11 @@ function buildExternal({ sources, exits }, byPath, innerBounds, dayScale = 1) {

  // Outgoing: one exit node per distinct full URL (so several links to
  // the same domain stay distinct), showing the total count across all
-  // pages linking to it, in a row centered below the map (hottest
-  // first), mirroring the source row above. Each (URL, page) pair
-  // contributes an edge from that page. An exit whose connectors are all
-  // culled (below MIN_WMID) is dropped itself.
+  // pages linking to it, in a row below the map mirroring the source row
+  // above. Each (URL, page) pair contributes an edge from that page. An
+  // exit whose connectors are all culled (below MIN_WMID) is dropped
+  // itself. Pills slide sideways toward their connection targets (see
+  // placeRow).
  const byExt = new Map() // full URL -> { ext, out, pairs }
  for (const p of liveExits.filter((p) => p.out >= minCount)) {
    const g = byExt.get(p.ext) || { ext: p.ext, out: 0, pairs: [] }
@@ -850,19 +906,23 @@ function buildExternal({ sources, exits }, byPath, innerBounds, dayScale = 1) {
  const targets = [...byExt.values()]
    .sort((a, b) => b.out - a.out)
    .slice(0, MAX_EXT_OUT)
-    .filter(({ pairs }) =>
-      pairs.some((p) => !byPath.get(p.page).hidden && width(p.out) >= MIN_WMID))
+    .map((t) => ({
+      ...t,
+      targets: t.pairs
+        .filter((p) => !byPath.get(p.page).hidden && width(p.out) >= MIN_WMID)
+        .map((p) => ({ x: byPath.get(p.page).x, w: p.out })),
+    }))
+    .filter((t) => t.targets.length)
  if (targets.length) {
-    const cx = (innerBounds.x0 + innerBounds.x1) / 2
    const y = innerBounds.y1 + TNODE_BOUND + EXT_GAP
-    const spacing = TNODE_W + 44
-    const x0 = cx - ((targets.length - 1) * spacing) / 2
-    targets.forEach(({ ext, out, pairs }, i) => {
+    for (const t of targets) t.anchor = weightedMedian(t.targets)
+    placeRow(targets, TNODE_W + 44)
+    targets.forEach(({ ext, out, pairs, x }) => {
      const xn = {
        path: ext,
        href: ext,
        label: extLabel(ext),
-        x: x0 + i * spacing,
+        x,
        y,
        count: out,
        kind: 'exit',
@@ -5,7 +5,10 @@ to ``Analytics.gets`` as a raw access-log line (path with query string, true
 HTTP status, external referer origin, preload flag, rendered content
 language) and every pagerite.js activity message from the /_ws WebSocket is
 appended to ``Analytics.msgs``
-(navigations ``fr`` -> ``to`` and active reading-time updates).  Nothing is
+(navigations ``fr`` -> ``to`` and active reading-time updates).  The
+non-document machinery GETs ``/robots.txt`` and ``/sitemap.xml`` are
+recorded the same way: never followed by an activity message, they surface
+as crawler hits at display time.  Nothing is
 classified when it is recorded: whether a client turns out to be a reader,
 a crawler or a scanner is decided by ``Store.display()`` from the raw
 events, so the stored data survives any future change to the classification
@@ -407,8 +407,8 @@ async def editor_ws(ws: WebSocket) -> None:
      -> {"type": "doc", "path", "exists", "title", "markdown", "published",
          "banner", "banner_design", "banner_from", "banner_design_from",
          "banner_design_inherited", "description", "image", "image_resolved",
-          "image_mined", "image_source", "has_children", "large", "lang",
-          "primary_lang", "langs",
+          "image_mined", "image_source", "favicon", "has_children", "large",
+          "lang", "primary_lang", "langs",
          "translate_langs"}
      <- {"type": "render", "path", "markdown"}
      -> {"type": "html", "path", "html"}
@@ -416,7 +416,8 @@ async def editor_ws(ws: WebSocket) -> None:
          "banner"?, "banner_design"?, "image"?, "large"?, "move_from"?,
          "lang"?, "base"?}
          (absent fields keep their old values; move_from: rename/move a
-          page, subtree included)
+          page, subtree included; image: a store hash, "@favicon", "" to
+          inherit, or an http(s) URL the server fetches and stores)
      -> {"type": "saved", "path"} | {"type": "error", "detail"}

    With "lang" (a translation, not the primary language), open returns the
@@ -458,11 +459,13 @@ async def editor_ws(ws: WebSocket) -> None:
                            # original (docs/localization.md editor flow).
                            markdown = i18n.hybrid_markdown(data, node, path, lang)
                            title = i18n.title_map(data, lang).get(path) or title
-                    # The node's card image: its own setting ("" = inherit),
-                    # the effective one after inheritance ("" = none) and
+                    # The node's card image: its own setting ("" = inherit,
+                    # "@favicon" = the site icon), the effective one after
+                    # inheritance ("" = none, resolved to a store name) and
                    # which node supplied an inherited one ("" = front page;
                    # "" also when own/none — mirrors banner_from).
                    img, img_source = views.card_image(data.menu, path)
+                    img = views._resolve_image_name(data, img)
                    # The card preview's description and mined image,
                    # from the same rendered-article heuristics as the
                    # og:/twitter: meta (_description, _media).
@@ -503,9 +506,13 @@ async def editor_ws(ws: WebSocket) -> None:
                            # /<path>/*"): the subtree inherits it.
                            "has_children": bool(node.children) if node else False,
                            "image_resolved": img,
+                            # The site icon ("" = none): the "@favicon" own
+                            # setting previews/resolves against it.
+                            "favicon": data.favicon,
                            # The image the og:/twitter: heuristics would mine
                            # from the article itself ("" = none): the previews
-                            # show it when no node image resolves.
+                            # show it when the node has no image of its own
+                            # (it beats an inherited one).
                            "image_mined": img_mined,
                            "image_source": (
                                "" if node is None or node.image else img_source
@@ -655,9 +662,28 @@ async def editor_ws(ws: WebSocket) -> None:
                    image = msg.get("image")
                    if image is not None:
                        # Card-image setting (inherited by the subtree): a
-                        # 12-hex content-addressed store name, "" = inherit.
+                        # 12-hex content-addressed store name, "@favicon" =
+                        # the site icon, "" = inherit, or an http(s) URL —
+                        # pasted image links are fetched and stored
+                        # server-side (cross-origin is CORS-blocked for the
+                        # browser), the stored name becomes the setting.
                        image = str(image).strip()
-                        if image and not re.fullmatch(r"[0-9a-f]{12}", image):
+                        if image.startswith(("http://", "https://")):
+                            from pagerite.files import fetch_image
+
+                            try:
+                                # Bare hash name, like an upload's setting.
+                                image = (await fetch_image(image)).split(".")[0]
+                            except HTTPException as e:
+                                await ws.send_json(
+                                    {"type": "error", "detail": str(e.detail)}
+                                )
+                                continue
+                        if (
+                            image
+                            and image != "@favicon"
+                            and not re.fullmatch(r"[0-9a-f]{12}", image)
+                        ):
                            await ws.send_json(
                                {
                                    "type": "error",
@@ -11,6 +11,8 @@ The routes live in specialized modules, included below as APIRouters:
  (``/_api/*``, ``/_translate/{clientkey}``).
 - ``pagerite.tracking`` — visit analytics (``/_ws``, ``/_api/ws/analytics``,
  the ``/_a`` viewer page).
+- ``pagerite.feeds`` — machine-readable site exports: ``/llms.txt``,
+  ``/feed.json`` (JSON Feed) and ``/feed.xml`` (RSS).
 - ``pagerite.pages`` — the public content pages: ``/``, ``/sitemap.xml``,
  ``/robots.txt`` and the ``/{path:path}`` catch-all.

@@ -39,7 +41,7 @@ from fastapi.responses import Response
 from fastapi_vue import Frontend, env
 from starlette.types import ASGIApp, Receive, Scope, Send

-from pagerite import api, files, pages, tracking
+from pagerite import api, feeds, files, pages, tracking
 from pagerite.files import file_store
 from pagerite.state import analytics_store, config, kanta

@@ -128,4 +130,5 @@ frontend.route(app, "/")

 # The content catch-all goes last: built assets win over content slugs,
 # anything unmatched falls through to content (and 404).
+app.include_router(feeds.router)
 app.include_router(pages.router)
@@ -93,7 +93,9 @@ class Node(msgspec.Struct, omit_defaults=True):
    #: Content-addressed card image name (served at "/_f/{name}") for
    #: og:image/twitter:image and card covers. "" inherits the nearest
    #: ancestor's image, the front page last; unset everywhere falls back
-    #: to mining the rendered article.
+    #: to mining the rendered article (which beats an inherited image).
+    #: "@favicon" resolves to the site icon (Data.favicon) at render time,
+    #: following favicon changes.
    image: str = ""
    #: Card-mode override (site cards + twitter:card): None = pick
    #: automatically from the card image's dimensions, False forces a
@@ -0,0 +1,182 @@
+"""Machine-readable site exports: /llms.txt, /feed.json and /feed.xml.
+
+- ``/llms.txt`` (llmstxt.org convention): a Markdown map of the site for
+  LLM agents — the brand as title, then every published article as a link
+  with a short excerpt.
+- ``/feed.json``: JSON Feed 1.1 of all published articles, full content.
+- ``/feed.xml``: the same as RSS 2.0 (with an atom:link self reference)
+  for older feed readers.
+
+All three are linked from every page's <head> (see views._layout) and from
+the sitemap, recorded in analytics like page GETs (they surface as crawler
+hits — no activity message ever follows them), and rendered in the site's
+primary language only (feeds have no per-language negotiation here).
+
+Rendering every article is expensive, so bodies are cached in RAM keyed by
+the public base URL and cleared by ``state._invalidate_pages`` on any
+content/settings write — the same hook that drops the page render cache.
+Each response also carries a content-hash ETag and answers 304, so polling
+feed readers revalidate cheaply.
+"""
+
+import blake3
+import json
+from datetime import UTC
+from email.utils import format_datetime
+from functools import lru_cache
+from xml.sax.saxutils import escape as xml_escape
+
+from fastapi import APIRouter, Request
+from fastapi.responses import Response
+
+from pagerite.data import Node, node_markdown, sorted_nodes
+from pagerite.markdown import make_md
+from pagerite.state import SITE_URL, data
+from pagerite.tracking import _record_get
+from pagerite.views import _description
+
+router = APIRouter()
+
+_md = make_md()
+
+
+def _articles() -> list[tuple[str, Node]]:
+    """All published content pages in menu order: (path, node)."""
+    out: list[tuple[str, Node]] = []
+
+    def walk(nodes: dict[str, Node], prefix: str) -> None:
+        for slug, node in sorted_nodes(nodes):
+            path = f"{prefix}/{slug}" if prefix else slug
+            if node.published and node.chunks is not None:
+                out.append((path, node))
+            if node.children:
+                walk(node.children, path)
+
+    walk(data.menu, "")
+    return out
+
+
+def _body_html(node: Node, base: str) -> str:
+    """Full article HTML for feed content, with relative URLs absolutized.
+
+    Rendered without the layout segmentation of page rendering (colseg
+    wrappers are meaningless in a feed reader); the item title carries the
+    page title, so no implicit h1 is injected either.
+    """
+    html = _md.render(node_markdown(data, node) or "")
+    return html.replace('src="/', f'src="{base}/').replace('href="/', f'href="{base}/')
+
+
+def _rfc822(node: Node) -> str:
+    return format_datetime(node.modified.astimezone(UTC), usegmt=True)
+
+
+def _iso(node: Node) -> str:
+    return node.modified.astimezone(UTC).replace(microsecond=0).isoformat()
+
+
+def _render_llms(base: str) -> str:
+    lines = [f"# {data.brand}", "", "## Pages", ""]
+    for path, node in _articles():
+        url = f"{base}/{path}" if path else base
+        excerpt = _description(_body_html(node, base), 120)
+        suffix = f": {excerpt}" if excerpt else ""
+        lines.append(f"- [{node.title or path}]({url}){suffix}")
+    return "\n".join(lines) + "\n"
+
+
+def _render_feed_json(base: str) -> str:
+    items = [
+        {
+            "id": (url := f"{base}/{path}" if path else base),
+            "url": url,
+            "title": node.title or path,
+            "content_html": _body_html(node, base),
+            "date_published": node.created.astimezone(UTC)
+            .replace(microsecond=0)
+            .isoformat(),
+            "date_modified": _iso(node),
+        }
+        for path, node in _articles()
+    ]
+    feed = {
+        "version": "https://jsonfeed.org/version/1.1",
+        "title": data.brand,
+        "home_page_url": base,
+        "feed_url": f"{base}/feed.json",
+        "items": items,
+    }
+    return json.dumps(feed, ensure_ascii=False, indent=1)
+
+
+def _render_feed_xml(base: str) -> str:
+    items = []
+    for path, node in _articles():
+        url = f"{base}/{path}" if path else base
+        items.append(
+            f"<item><title>{xml_escape(node.title or path)}</title>"
+            f"<link>{xml_escape(url)}</link>"
+            f'<guid isPermaLink="true">{xml_escape(url)}</guid>'
+            f"<pubDate>{_rfc822(node)}</pubDate>"
+            f"<description><![CDATA[{_body_html(node, base)}]]></description>"
+            f"</item>"
+        )
+    return (
+        '<?xml version="1.0" encoding="UTF-8"?>'
+        '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">'
+        f"<channel><title>{xml_escape(data.brand)}</title>"
+        f"<link>{xml_escape(base)}</link>"
+        f"<description>{xml_escape(data.brand)}</description>"
+        f'<atom:link href="{xml_escape(base)}/feed.xml" rel="self" type="application/rss+xml" />'
+        + "".join(items)
+        + "</channel></rss>"
+    )
+
+
+#: Export body builders by route path.
+_RENDERERS = {
+    "/llms.txt": (_render_llms, "text/plain"),
+    "/feed.json": (_render_feed_json, "application/feed+json"),
+    "/feed.xml": (_render_feed_xml, "application/rss+xml"),
+}
+
+
+@lru_cache(maxsize=12)
+def _cached_feed(path: str, base_url: str) -> str:
+    """Rendered export body; cleared by ``state._invalidate_pages`` on any
+    content/settings write. base_url is part of the key because the bodies
+    bake absolute URLs into every link, image and guid."""
+    return _RENDERERS[path][0](base_url)
+
+
+def _feed_response(request: Request, path: str) -> Response:
+    """Cached export response with a content-hash ETag (304 on match), so
+    polling feed readers revalidate without a rerender or a download."""
+    base = SITE_URL or str(request.base_url).rstrip("/")
+    body = _cached_feed(path, base)
+    headers = {"cache-control": "no-cache"}
+    tag = f'"{blake3.blake3(body.encode()).hexdigest()[:32]}"'
+    headers["etag"] = tag
+    if request.headers.get("if-none-match") == tag:
+        return Response(status_code=304, headers=headers)
+    _record_get(request)
+    return Response(body, media_type=_RENDERERS[path][1], headers=headers)
+
+
+@router.get("/llms.txt")
+async def llms_txt(request: Request) -> Response:
+    """Markdown map of the site for LLM agents (llmstxt.org)."""
+    return _feed_response(request, "/llms.txt")
+
+
+@router.get("/feed.json")
+async def feed_json(request: Request) -> Response:
+    """JSON Feed 1.1 of all published articles, full content."""
+    return _feed_response(request, "/feed.json")
+
+
+@router.get("/feed.xml")
+async def feed_xml(request: Request) -> Response:
+    """RSS 2.0 of all published articles (full content in CDATA), with an
+    atom:link self reference."""
+    return _feed_response(request, "/feed.xml")
@@ -17,8 +17,10 @@ import mimetypes
 import tempfile
 from contextlib import suppress
 from pathlib import Path
+from urllib.parse import urlsplit

 import blake3
+import httpx
 from fastapi import APIRouter, HTTPException, Request
 from fastapi.responses import RedirectResponse, Response
 from mediapreview import dispatch
@@ -213,6 +215,34 @@ def store_image(
    return digest


+#: Pasted-URL image fetches (fetch_image) refuse bodies over this size.
+FETCH_MAXSIZE = 20 * 1024 * 1024
+
+
+async def fetch_image(url: str) -> str:
+    """Fetch an image URL and store it like an upload (store_image's
+    derivative pipeline), returning the stored file name.
+
+    Server-side because arbitrary cross-origin URLs are CORS-blocked for
+    the browser; used by the editor socket when a card-image setting
+    arrives as a URL (paste). 415 for non-images, 413 over FETCH_MAXSIZE,
+    502 when the fetch itself fails.
+    """
+    try:
+        async with httpx.AsyncClient(follow_redirects=True, timeout=15) as client:
+            r = await client.get(url)
+        r.raise_for_status()
+    except httpx.HTTPError as e:
+        raise HTTPException(502, f"image fetch failed: {e}") from e
+    ctype = r.headers.get("content-type", "").split(";")[0].strip().lower()
+    if not ctype.startswith("image/"):
+        raise HTTPException(415, "the URL is not an image")
+    if len(r.content) > FETCH_MAXSIZE:
+        raise HTTPException(413, "image too large")
+    ext = _ext(Path(urlsplit(url).path).name) or mimetypes.guess_extension(ctype) or ""
+    return await asyncio.to_thread(store_image, r.content, ext, derive=ext != ".gif")
+
+
@router.put("/_api/files/{name}")
 async def upload_file(name: str, request: Request) -> dict[str, str]:
    """Store an upload (image, video...) in the content-addressed store.
@@ -102,7 +102,9 @@ def select_language(
    return original


-def hybrid_items(data: Data, node: Node, path: str, lang: str) -> list[tuple[bytes | None, str]]:
+def hybrid_items(
+    data: Data, node: Node, path: str, lang: str
+) -> list[tuple[bytes | None, str]]:
    """The served hybrid as (anchor, block text) pairs: the anchor is the
    ORIGINAL chunk hash behind the block (None for translation-only
    addition blocks), in article order.
@@ -5,7 +5,9 @@ the page (or a category placeholder, or 404); it must be registered AFTER
 the fastapi-vue asset routes so built frontend files win over content slugs
 (see app.py). Every served document is recorded raw in analytics (one
 access-log line with its true HTTP status; classification happens at
-display time — see pagerite/analytics.py).
+display time — see pagerite/analytics.py), as are robots.txt and sitemap.xml
+fetches (they surface as crawler hits, since no activity message ever
+follows them).
 """

 import logging
@@ -53,7 +55,8 @@ async def front_page(request: Request) -> Response:

@router.get("/sitemap.xml")
 async def sitemap(request: Request) -> Response:
-    """Dynamically generate a sitemap of all published article pages."""
+    """Dynamically generate a sitemap of all published article pages, plus
+    the machine-readable exports (feeds, llms.txt)."""
    base = SITE_URL or str(request.base_url).rstrip("/")
    entries: list[tuple[str, datetime, int]] = []

@@ -110,6 +113,26 @@ async def sitemap(request: Request) -> Response:
        )
    lines.append("</urlset>")

+    # The machine-readable exports (feeds, llms.txt) are linked too, with
+    # the latest article's modification time as their lastmod.
+    if entries:
+        latest = max(m for _, m, _ in entries)
+        lastmod = (
+            latest.astimezone(UTC)
+            .replace(microsecond=0)
+            .isoformat()
+            .replace("+00:00", "Z")
+        )
+        for special in ("llms.txt", "feed.json", "feed.xml"):
+            lines.insert(
+                -1,
+                f"  <url><loc>{xml_escape(f'{base}/{special}')}</loc>"
+                f"<lastmod>{lastmod}</lastmod></url>",
+            )
+
+    # Recorded like a page GET: never followed by an activity message, so
+    # it lands in the crawler list at display time (docs/analytics.md).
+    _record_get(request)
    return Response(
        "\n".join(lines),
        media_type="application/xml",
@@ -124,6 +147,7 @@ async def robots_txt(request: Request) -> Response:
    the sitemap."""
    base = SITE_URL or str(request.base_url).rstrip("/")
    body = f"User-agent: *\nAllow: /\nDisallow: /auth/\nDisallow: /_api\nSitemap: {base}/sitemap.xml\n"
+    _record_get(request)
    return Response(
        body,
        media_type="text/plain",
@@ -171,11 +171,16 @@ _render_gen = 0


 def _invalidate_pages() -> None:
-    """Drop cached page bodies and bump the render generation (ETags);
-    any content change also re-runs translation dispatch."""
+    """Drop cached page bodies (and the feed/llms.txt export bodies) and
+    bump the render generation (ETags); any content change also re-runs
+    translation dispatch."""
    global _render_gen
    _render_gen += 1
    _cached_body.cache_clear()
+    # Local import: pagerite.feeds imports this module.
+    from pagerite import feeds
+
+    feeds._cached_feed.cache_clear()
    dispatcher.schedule()


@@ -379,6 +379,18 @@ def _layout(
        doc.link(rel="canonical", href=canonical)
    for hreflang, href in alternates:
        doc.link(rel="alternate", hreflang=hreflang, href=href)
+    # Feed/LLM discovery links (see pagerite/feeds.py): identical on every
+    # page, so the positional <head> sync (swapdoc.js) is unaffected.
+    doc.link(
+        rel="alternate",
+        type="application/feed+json",
+        title="JSON Feed",
+        href="/feed.json",
+    )
+    doc.link(
+        rel="alternate", type="application/rss+xml", title="RSS feed", href="/feed.xml"
+    )
+    doc.link(rel="llms-txt", type="text/markdown", title="llms.txt", href="/llms.txt")
    for key, value in (social or {}).items():
        if value:
            if key.startswith(("og:", "article:")):
@@ -786,14 +798,23 @@ def banner_source(menu: dict[str, Node], path: str) -> str | None:
    return None


+def _resolve_image_name(data: Data, name: str) -> str:
+    """The store name a card-image setting resolves to: the "@favicon"
+    sentinel follows the site icon (set in the site editor — "" when
+    unset); anything else is already a store name."""
+    return data.favicon if name == "@favicon" else name
+
+
 def card_image(menu: dict[str, Node], path: str) -> tuple[str, str]:
    """The effective card image at ``path`` and which node supplied it.

    Nearest ancestor with ``image`` set wins (the node itself first), the
    front page — a top-level sibling of the chain — last. ("", "") when no
-    node sets one: rendering falls back to mining the article HTML. The
-    source path ("" = front page) feeds the editor banner panel's inherit
-    label.
+    node sets one: rendering falls back to mining the article HTML (which
+    beats an inherited image — see _card/_social_meta). The returned name
+    is the raw setting (may be "@favicon"; resolve with
+    _resolve_image_name). The source path ("" = front page) feeds the
+    editor banner panel's inherit label.
    """
    chain = resolve(menu, path) or []
    segs = path.split("/")
@@ -1036,20 +1057,22 @@ def _walk(node: Node, path: str):
 def _card_large(node: Node, image: str) -> bool:
    """Whether the card renders large (True) or small (False).

-    Automatic: large when the image's probed store dimensions suit a large
-    card (>= 600px wide, landscape-ish aspect 1.4–2.5), small for
-    small/portrait images — and, when dimensions are unknown or the image
-    is external, for any present image. The node's ``large`` setting
-    (per-article, not inherited) overrides the automatic pick; None
-    means automatic. Shared by twitter:card (_social_meta, which maps it
-    to "summary_large_image"/"summary") and the site's own cards (_card).
+    Automatic: large when the image is big and wide enough for the
+    full-card cover — wider than 600px, taller than 400px and clearly
+    wider than tall (aspect > 1.05, so square and portrait images keep
+    the compact layout whose box they fit). When dimensions are unknown
+    or the image is external, any present image defaults to large. The
+    node's ``large`` setting (per-article, not inherited) overrides the
+    automatic pick; None means automatic. Shared by twitter:card
+    (_social_meta, which maps it to "summary_large_image"/"summary") and
+    the site's own cards (_card).
    """
    large = bool(image)
    if (m := re.search(r"/_f/([0-9a-f]{12})$", image)) and (
        dims := _image_dims(m.group(1))
    ):
        w, h = dims
-        large = w >= 600 and h > 0 and 1.4 <= w / h <= 2.5
+        large = w > 600 and h > 400 and w / h > 1.05
    if node.large is not None:
        large = node.large
    return large
@@ -1071,9 +1094,10 @@ def _card(
    the small format). Imageless cards keep the image space blank (a
    gradient cover).

-    The cover is the page's resolved card image (Node.image, inheriting
-    down the tree) when set, else mined from the rendered article like
-    og:image; the mode follows the same selection as twitter:card
+    The cover is the node's own ``image`` when set, else mined from the
+    rendered article like og:image, else the inherited image (the nearest
+    ancestor's or the front page's); the mode follows the same selection
+    as twitter:card
    (_card_large: the node's override, else the image's dimensions). The
    card text localizes per target article where that page is available in
    the language: the title comes from the translation's title map and the
@@ -1081,8 +1105,9 @@ def _card(
    with per-card fallback to the original otherwise.
    """
    image = html = ""
-    if name := card_image(menu, path)[0]:
-        image = f"/_f/{name}"
+    if node.image:
+        if name := _resolve_image_name(data, node.image):
+            image = f"/_f/{name}"
    if node.chunks and not image:
        md = node_markdown(data, node) or ""
        if lang and lang in node.langs:
@@ -1097,6 +1122,12 @@ def _card(
            directives={"cards": lambda _args, _env: ""},
        ).html
        image, _ = _media(html)
+    if not image:
+        # The node's own setting was empty: an inherited image applies
+        # only when the article itself yielded none (card_image with no
+        # own image set resolves to the nearest ancestor's/front page's).
+        if name := _resolve_image_name(data, card_image(menu, path)[0]):
+            image = f"/_f/{name}"
    large = _card_large(node, image)
    description = ""
    if not large and node.chunks and not html:
@@ -1227,16 +1258,18 @@ def _social_meta(
    brand: str,
    base_url: str,
    card: str = "",
+    card_fallback: str = "",
 ) -> dict[str, str]:
    """Open Graph/Twitter/SEO meta tags for a content page.

-    The card image is the node's own ``image`` setting when one resolves
-    (``card``, see card_image — the nearest ancestor's or the front
-    page's otherwise); with none set, heuristics over the rendered article
-    pick the first representative <img> (a {.hero} first, then raster,
-    then SVG). The description is the first paragraph's text; the first
-    <video> yields og:video. Absolute URLs are built from the request's
-    base (social scrapers cannot use relative ones).
+    The card image is the node's own ``image`` setting (``card``) when
+    set; with none set, heuristics over the rendered article pick the
+    first representative <img> (a {.hero} first, then raster, then SVG);
+    only when the article yields none does the inherited image
+    (``card_fallback`` — the nearest ancestor's or the front page's, see
+    card_image) apply. The description is the first paragraph's text; the
+    first <video> yields og:video. Absolute URLs are built from the
+    request's base (social scrapers cannot use relative ones).

    ``twitter:image`` pins extension-less store links to the ``.webp``
    variant: X only honors WebP via twitter:image (not og:image) and its
@@ -1252,6 +1285,8 @@ def _social_meta(
        _, video = _card_media(html, base_url)
    else:
        image, video = _card_media(html, base_url)
+        if not image and card_fallback and base_url:
+            image = f"{base_url}/_f/{card_fallback}"
    twitter_image = re.sub(r"(/_f/[0-9a-f]{12})$", r"\1.webp", image) if image else ""
    large = _card_large(node, image)
    return {
@@ -1334,8 +1369,20 @@ def render_page(
        lang = original
    title = _title(path.rpartition("/")[2], node, translation, path)
    main = page_content(menu, data, path, translation, link_lang, lang)
+    # The card image priority: the node's own setting, then mined from the
+    # article (inside _social_meta), then the inherited one. card_image
+    # resolves both: its source equals the path iff the node itself set it.
+    img, img_from = card_image(menu, path)
+    img = _resolve_image_name(data, img)
    social = _social_meta(
-        node, path, title, str(main), brand, base_url, card_image(menu, path)[0]
+        node,
+        path,
+        title,
+        str(main),
+        brand,
+        base_url,
+        card=img if img_from == path else "",
+        card_fallback="" if img_from == path else img,
    )
    canonical, alternates = _language_urls(data, path, node, lang, original, base_url)
    return str(
Author	SHA1	Message	Date
LeoVasanko	0ff5c6b526	ruff	2026-09-23 07:32:57 +00:00
LeoVasanko	2aed176c9e	Cache feed/llms.txt bodies, serve with content-hash ETags Rendering every article per request is expensive. Bodies are now RAM-cached keyed by the public base URL and cleared from _invalidate_pages (the same hook that drops the page render cache on any content/settings write). Responses carry a blake3 content-hash ETag and answer 304, so polling feed readers revalidate cheaply; matching page behavior, 304 revalidations are not recorded in analytics.	2026-09-23 07:30:32 +00:00
LeoVasanko	9ff22016d1	Add llms.txt, JSON Feed and RSS feed exports New pagerite/feeds.py: /llms.txt (Markdown site map with excerpts for LLM agents), /feed.json (JSON Feed 1.1) and /feed.xml (RSS 2.0 + atom:link), all carrying every published article with full content and absolutized URLs. Linked from every page <head> (feed alternates + llms-txt) and from the sitemap, recorded in analytics like page GETs and emoji-marked in the trails (🧠 llms.txt, 📡 feeds).	2026-09-23 07:25:08 +00:00
LeoVasanko	97bde49296	Record robots.txt/sitemap.xml GETs, emoji-mark machinery steps in trails The robots and sitemap routes never called _record_get, so those accesses were invisible. Recorded like page GETs, they surface as crawler hits at display time (no activity message ever follows). In the trail rendering, known non-article machinery paths (robots, sitemap, future feed paths) get an emoji marker instead of a bare slug.	2026-09-23 07:12:15 +00:00
LeoVasanko	4f97a6592c	Relax automatic large-card pick: w > 600, h > 400, aspect > 1.05 The old 1.4-2.5 aspect window sent big near-square images (e.g. 1350x1200) to the compact layout despite ample resolution. The new rule keeps square and portrait images compact (the compact box fits them) while letting anything big and clearly wider than tall render as a full-card cover.	2026-09-23 07:07:42 +00:00
LeoVasanko	7e86efeb9f	Place external source/exit pills near their connection targets Replace count-sorted, group-centered rows with a spring-like placement: pills order by their weighted median target x and settle by clamped coordinate descent, minimizing weighted horizontal connection distance while keeping the minimum pill spacing.	2026-09-23 07:02:17 +00:00
LeoVasanko	580ebac06c	Card image: paste button and site-icon (@favicon) option The banner panel's card-image block gains a pasteboard button (an image uploads; an image URL goes as the save message's image setting itself and is fetched/stored server-side, cross-origin being CORS-blocked for the browser) and a site-icon button saving the @favicon sentinel, which resolves to Data.favicon at render time and follows favicon changes.	2026-09-23 06:31:06 +00:00
LeoVasanko	4d6735f609	Fix card image priority: own > mined from article > inherited The resolved (inheriting) Node.image was checked before mining the article, so a front-page banner image shadowed the articles' own extracted figures everywhere: card covers, og:image and the editor's card previews.	2026-09-23 06:10:01 +00:00