Per-hostname data directory <hostname>/{content.kantadb, analytics.json, files} from CLI arg; drop learned site_url.

The first positional CLI argument (default localhost) names the site's
public hostname and its data directory under the cwd, replacing the
CWD-relative pagerite.* files. The public origin (https://<hostname>)
is now authoritative configuration instead of a value learned from
admin browsers: POST /_api/site-url, Data.site_url and the pagerite.js
reporter are removed, and page rendering, sitemap, robots.txt and
analytics own_origin all use SITE_URL consistently (localhost falls
back to the request's base URL).

.gitignore

@@ -1,8 +1,7 @@
 .*
 !.gitignore
 *.lock
-*.kantadb
-pagerite.analytics.json
+/localhost
 dbip-*.mmdb*
 /pagerite/frontend-build
 package-lock.json

AGENTS.md

@@ -12,6 +12,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke
 - `pagerite/` — Python backend package (hatchling build target).
   - `app.py` — FastAPI app and route registration.
   - `data.py` — msgspec Structs for the kanta database.
+  - `migrations.py` — kanta schema migrations (`migrate_vN`), e.g. v1 moves legacy in-db file blobs to the on-disk store.
   - `markdown.py` — markdown-it-py renderer.
   - `views.py` — shared page layout and rendering.
   - `seed.py` — demo content, written only on first database creation.

docs/analytics.md

@@ -2,11 +2,11 @@
 
 Server-side visit analytics. Data lives in a plain JSON file — a msgspec
 Struct dumped to disk — separate from the kanta content database, path from
-`PAGERITE_ANALYTICS` (default: the database path with `.kantadb` replaced by
-`.analytics.json`, e.g. `pagerite.analytics.json`).
+`PAGERITE_ANALYTICS` (default: `analytics.json` in the per-site data
+directory, e.g. `localhost/analytics.json`).
 
 - `pagerite/analytics.py` — data model (`Analytics`, `Client`, `Visit`,
-  `CrawlerHit`, `AbuseHit`) and the `Store` (in-memory data + session map,
+  `CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map,
   atomic JSON persistence).
 - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
   the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
@@ -73,6 +73,20 @@ falsy values are omitted):
 - The server validates `to`: internal paths must be valid slug paths
   ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
   https origin and accepted only when the client sent exactly that.
+- **External-site favicons**: for every external https origin seen as a visit
+  referer or an exit link, the server fetches `{origin}/favicon.ico` in a
+  background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only —
+  SVG is sniffed from the body when served without an image type) and stores
+  the icon content-hashed on disk in the FileStore (served at `/_f/{name}`,
+  extension matching the actual MIME).  The origin → file name mapping is
+  recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are
+  recorded too and retried only after 7 days.  Fetches are scheduled after
+  each ping and once at startup, which backfills icons for already-recorded
+  data.  The viewer payload carries `favicons` (origin → `/_f/...` path),
+  and the viewer shows the icon wherever an external site is mentioned:
+  referer/exit trail links in the visit table and the source/exit pills of
+  the transition map (UTM-attributed source nodes without an https origin
+  stay text-only).
 - **Client records**: the visitor's IP (IPv4 or IPv6 /64 network), raw
   `User-Agent` and extracted `Accept-Language` tag are hashed with blake3;
   the first 6 bytes identify a shared `Client` record.  The `Client` stores

docs/backend.md

@@ -10,6 +10,8 @@ The build mirrors the URL space — hashed immutable assets under `/_assets/`, `
 
 Generated HTML pages (content pages, category/404 placeholders, `/_a`) go through `_html_response`: zstd-compressed per request at level 9 when the client sends `accept-encoding: zstd` (no gzip fallback; static assets are pre-compressed by the `Frontend`), with `vary: accept-encoding` set and the ETag kept identical across encodings so `if-none-match` revalidation still works. In production the rendered bodies are cached in an LRU keyed by everything the output depends on — page kind, path, the site origin (social meta), encoding, and `data.version`, which bumps on every content/settings change and so transparently invalidates the whole cache. The cache is bypassed in dev, where theme/design CSS is re-read from disk per request. Content pages carry an ETag built from the node's modified timestamp and `data.version`; `/_a` instead gets a blake3 hash of the rendered body (it has no Node), with matching `if-none-match` revalidations answered by a 304.
 
+Uploaded files, seed assets and fetched external-site favicons live in the `FileStore`: content-addressed files on disk under `<hostname>/files/` (`PAGERITE_FILES`), fully cached in RAM at startup — both the raw body and a zstd-compressed copy (kept only when smaller). `GET /_f/{name}` serves from the RAM cache with immutable caching, answering the zstd variant when the client accepts it; the name is the ETag. Legacy databases that still carry blobs in a `files` kanta field are migrated to disk by `pagerite/migrations.py::migrate_v1` (kanta's `migrate_vN` mechanism, wired via `Kanta(..., migrations="pagerite.migrations")`), which pops the field from the raw state before struct decoding.
+
 ## `data.py`
 
 msgspec Structs for the kanta database. See `docs/content-model.md` for the full data model.
@@ -24,7 +26,7 @@ markdown-it-py renderer (html passthrough + attrs, footnote, deflist, tasklists,
 
 The shared page layout as an html5tagger `Template` with placeholders (`Title`, `Brand`, `Banner`, `Nav`, `Sidebar`, `Main`), nav rendering straight from the `Data.menu` tree (siblings sorted by `Node.order`; nav links to content-less labels point at their first child via `first_leaf`, the first published descendant with content), and page/404 rendering.
 
-Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG; the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`Data.site_url` — learned from admin browsers reporting their `location.origin` via `POST /_api/site-url`, correct even behind reverse proxies; until learned, the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. If the markdown contains its own h1, the page title is NOT rendered as an additional h1 (it still supplies `<title>` and nav labels).
+Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG; the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`SITE_URL` — `https://<hostname>` from the CLI hostname argument; on localhost the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. If the markdown contains its own h1, the page title is NOT rendered as an additional h1 (it still supplies `<title>` and nav labels).
 
 The navbar holds top-level items only; the current section's subitems go to a left `#sidebar` as a nested list (the section's direct children plain, deeper levels indented with article-list-style markers), rendered only from the second level down — main-level pages list their children as cards after the content instead. Below that, the sidebar renders when the section offers at least two published items, or exactly one while viewing anything other than that only page — the section index, a 404, a grandchild (so those pages can reach the child), and also on that only page itself when it has published children of its own; no aside element at all on the front page, main-level pages, leaf pages and the sole childless page of a one-page section. Also, category labels are nodes without content — None *or* empty markdown — and their nav links point at their first child page. Dynamic regions have stable ids (`#page-banner`, `#nav`, `#sidebar`, `#main`) for fetch-navigation swaps (`#sidebar` may be absent on either side of a swap).
 

docs/content-model.md

@@ -14,7 +14,7 @@ Siblings order by the fractional `Node.order` key: a moved item gets a fresh key
 
 ## Files
 
-`Data.files` is a content-addressed store (blake3[:12] + extension) mapping file names to bytes, served at `/_f/{name}` with immutable caching; pages reference files by absolute `/_f/` URLs so hierarchy moves never break them.
+Files are content-addressed (blake3[:12] + extension) and stored **on disk** under `<hostname>/files/` (path from `PAGERITE_FILES`), served at `/_f/{name}` with immutable caching; the `FileStore` in app.py caches every file in RAM, both uncompressed and zstd-compressed (the compressed copy only when smaller), so `/_f` answers both encodings without disk reads. Pages reference files by absolute `/_f/` URLs so hierarchy moves never break them. Pre-refactor databases kept the blobs in a `Data.files` kanta field; the kanta migration `pagerite/migrations.py::migrate_v1` writes them to disk on open and drops the field (removed from `Data`). Fetched favicons of external analytics sites live in the same store (see `docs/analytics.md`).
 
 ## Banners
 
@@ -26,10 +26,10 @@ Siblings order by the fractional `Node.order` key: a moved item gets a fresh key
 
 `Data.brand` is the site name (header link + `<title>` suffix), editable in the site editor via `/_api/settings`; empty = no header link and no `<title>` suffix.
 
-`Data.brand_html` is raw trusted HTML replacing the brand link entirely (rendered in a `#brand` div on top of the banner, next to the nav) — site-wide, not per-page like banners; edited in the site editor with image/video upload into `Data.files`.
+`Data.brand_html` is raw trusted HTML replacing the brand link entirely (rendered in a `#brand` div on top of the banner, next to the nav) — site-wide, not per-page like banners; edited in the site editor with image/video upload into the content-addressed file store.
 
 `Data.theme` is the active theme name (empty = none/base only); themes are folders in `pagerite/themes/{name}` containing `theme.css` and/or `banner.css` (+ `banner.svg` artwork and any extra assets the CSS references, like summer's `grass.svg`), served by the backend at `/_themes/{name}/...` — read from disk per request (etag by mtime), never built, so on-disk edits show on the next page load even in prod. The theme selector and banner-design selector enumerate these folders via `GET /_api/settings`.
 
 `Data.custom_css` is raw trusted CSS injected inline in every page `<head>` (id `pagerite-user`) and swapped during fetch-navigation; editable in the site editor. Font picks (heading/body/brand) in the site editor are stored as plain `:root` rows in `custom_css` (`--font-body: var(--font-source-sans);` format — parsed out and rewritten on change, the `:root` block added/removed as needed), referencing the per-family variables (`--font-source-sans` etc.) from `pagerite.css`; the base stylesheet's `--font-brand` defaults to `var(--font-heading)`.
 
-`Data.favicon` names a file in the content-addressed `files` store, uploaded/cleared in the site editor via `PUT`/`DELETE /_api/settings/favicon`; when set it is linked as `<link rel="icon">` on every page, otherwise browsers fall back to the build's `/favicon.ico` by convention.
+`Data.favicon` names a file in the content-addressed store (on disk under `<hostname>/files/`), uploaded/cleared in the site editor via `PUT`/`DELETE /_api/settings/favicon`; when set it is linked as `<link rel="icon">` on every page, otherwise browsers fall back to the build's `/favicon.ico` by convention.

docs/design-principles.md

@@ -20,7 +20,7 @@ Pagerite is a single-user CMS/blog. This document records the initial high-level
 - Content is written in **Markdown** with powerful extensions (tables, footnotes, code highlighting, etc.).
 - **Embedded HTML is passed through unfiltered**, including inline scripts and other dynamic content the author wants to post. This is safe by the single-trusted-author assumption above.
 - Renderer: **markdown-it-py** with mdit-py-plugins (footnotes, definition lists, task lists, brace-attributes, admonitions and `::: name` containers — generic `<div class="name">` wrappers (the name may be followed by brace attributes: `::: aside {.right}`), of which `::: aside` floats as a muted side box and `{.margin}` / `::: margin` marks any block a margin note — on all but phone widths they float in the side zone at the article's left (the region the nav sidebar overlays, or the sidebar's own track when the layout reserves one) and the text never moves — and `::: nocols` opts its section out of column layout; tables and strikethrough from the default preset), GitHub-style alerts (`> [!NOTE]` / TIP / IMPORTANT / WARNING / CAUTION, rendered in the admonition callout styling), with `html=True` for raw passthrough, `typographer=True` for SmartyPants-style replacements in body text (curly quotes, `--` / `---` → en / em dashes, `...` → ellipsis, `(c)` → ©, etc.), and `breaks=True` so single line breaks inside paragraphs become `<br>` — including inside blockquotes, where every newline is kept and a blank `>` line starts a new paragraph. Code spans/blocks and raw HTML are left untouched. Fenced code blocks are highlighted server-side with **Pygments** (`nowrap` spans styled by `/_assets/pygments-*.css`, which maps every token class onto the `--code-*` variables; the base stylesheet defines light and dark palette sets resolved via `light-dark()`, so each theme gets the set matching its `color-scheme` and may only retint `--code-bg` to keep the well in the page's color family); a JS copy button appears on hover. Should this prove limiting, we implement our own renderer on top of html5tagger, which we already use for all HTML generation.
-- **Files are content-addressed.** Uploads (`PUT /_api/files/{filename}`) are stored by content hash — blake3, first 6 bytes hex + original extension — and served immutable from `/_f/{hash}.ext`. Absolute URLs that survive page renames and dedupe identical content; pages no longer own files. An image standing alone in its paragraph becomes a block `<figure>` — with `<figcaption>` when it has a title; images inline with text and raw `<img>` HTML stay plain inline images. Positioning is by attribute classes: `![alt](/_f/….avif "Caption"){.right}` — `{.right}`, `{.left}` float at 30% of the text column (the caption wraps within it; an explicit `width=300` makes the figure shrink-wrap the image instead), `{.margin}` makes it a margin note, floating in the side zone left of the text on all but phone widths, `{.wide}` goes full bleed (viewport edge to edge, or up to the docked editor; the sidebar stacks on top of it); plain attributes like `width=300` work too. The same brace syntax on a block's last line (no blank line between) applies to the whole block: a paragraph ending with `{.wide}` becomes a full-width element that breaks out of the column layout; written on the line after a block it applies to that preceding block — this is how headings, `::: containers` and code fences take classes (a wide code fence goes full bleed like a wide figure). Headings (h1/h2) clear floats, so images never overflow into the next section.
+- **Files are content-addressed.** Uploads (`PUT /_api/files/{filename}`) are stored on disk (`<hostname>/files/`, RAM-cached uncompressed + zstd) by content hash — blake3, first 6 bytes hex + original extension — and served immutable from `/_f/{hash}.ext`. Absolute URLs that survive page renames and dedupe identical content; pages no longer own files. An image standing alone in its paragraph becomes a block `<figure>` — with `<figcaption>` when it has a title; images inline with text and raw `<img>` HTML stay plain inline images. Positioning is by attribute classes: `![alt](/_f/….avif "Caption"){.right}` — `{.right}`, `{.left}` float at 30% of the text column (the caption wraps within it; an explicit `width=300` makes the figure shrink-wrap the image instead), `{.margin}` makes it a margin note, floating in the side zone left of the text on all but phone widths, `{.wide}` goes full bleed (viewport edge to edge, or up to the docked editor; the sidebar stacks on top of it); plain attributes like `width=300` work too. The same brace syntax on a block's last line (no blank line between) applies to the whole block: a paragraph ending with `{.wide}` becomes a full-width element that breaks out of the column layout; written on the line after a block it applies to that preceding block — this is how headings, `::: containers` and code fences take classes (a wide code fence goes full bleed like a wide figure). Headings (h1/h2) clear floats, so images never overflow into the next section.
 
 ## Page structure and navigation
 

docs/frontend-runtime.md

@@ -22,6 +22,10 @@ The `::view-transition*` block at the end of `pagerite.css` (from termotohtori.f
 
 Vite builds ES-module `.js` outputs; in dev the backend links them as `<script type="module">` (module scripts defer by default), in production it inlines them at the end of the body.
 
-## Database file
+## Data directory
 
-The database file is `pagerite.kantadb` in the cwd (`PAGERITE_DB` overrides); gitignored. Do not delete it without asking.
+All site data lives under `<hostname>/` in the cwd — `content.kantadb`,
+`analytics.json` and `files/` — where `<hostname>` is the CLI's first
+positional argument (default `localhost`, exported as `PAGERITE_HOSTNAME`;
+`PAGERITE_DB`/`PAGERITE_ANALYTICS`/`PAGERITE_FILES` override individual
+paths). gitignored. Do not delete it without asking.

frontend/src/AnalyticsView.vue

@@ -130,6 +130,7 @@ watch(range, (r) => {
 })
 
 const clients = computed(() => data.value?.clients || {})
+const favicons = computed(() => data.value?.favicons || {})
 const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
 const crawlers = computed(() => rangeData.value?.crawlers || [])
 const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
@@ -161,7 +162,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
         </section>
 
         <VisitorCharts :data="data" :range="range" />
-        <TransitionGraph :data="rangeData" :window="window" :page-tree="pageTree" />
+        <TransitionGraph :data="rangeData" :window="window" :page-tree="pageTree" :favicons="favicons" />
 
         <section>
           <h2>Recent visits</h2>
@@ -177,9 +178,9 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
               <tbody>
                 <tr v-for="(v, i) in visitRows" :key="i">
                   <td class="trail">
-                    <TrailLink v-if="v.refererStep" :step="v.refererStep" @close="$emit('close')" />
+                    <TrailLink v-if="v.refererStep" :step="v.refererStep" :favicons="favicons" @close="$emit('close')" />
                     <span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span>
-                    <TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" @close="$emit('close')" />
+                    <TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" :favicons="favicons" @close="$emit('close')" />
                   </td>
                   <VisitorCell
                     :ip="v.ip"

frontend/src/TrailLink.vue

@@ -5,11 +5,15 @@ import { formatCount, formatReadTime } from './analytics/format.js'
 const props = defineProps({
   step: { type: Object, required: true },
   count: { type: Number, default: 0 },
+  favicons: { type: Object, default: null },
 })
 
 defineEmits(['close'])
 
 const hasError = computed(() => props.step.status >= 400)
+const favicon = computed(() =>
+  props.step.external && props.step.origin ? props.favicons?.[props.step.origin] : null,
+)
 
 const title = computed(() => {
   const parts = [props.step.title]
@@ -32,6 +36,16 @@ const title = computed(() => {
      :rel="step.external ? 'noopener' : undefined"
      @click="(e) => { if (!step.external) $emit('close') }">
     <small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small>
+    <img v-if="favicon" class="favicon" :src="favicon" alt="" />
     <span>{{ step.slug }}</span>
   </a>
 </template>
+
+<style scoped>
+.favicon {
+  width: 1em;
+  height: 1em;
+  margin-right: 0.25em;
+  vertical-align: -0.1em;
+}
+</style>

frontend/src/TransitionGraph.vue

@@ -20,8 +20,21 @@ const props = defineProps({
   data: { type: Object, default: null },
   window: { type: Object, required: true },
   pageTree: { type: Array, default: null },
+  favicons: { type: Object, default: null },
 })
 
+// origin -> /_f/... icon URL, keyed by the node's origin (source/exit
+// pills only; UTM-tagged source nodes without an https origin stay
+// text-only).
+const extFavicon = (x) => {
+  if (!x.path?.startsWith('https://')) return null
+  try {
+    return props.favicons?.[new URL(x.path).origin] || null
+  } catch {
+    return null
+  }
+}
+
 const dayScale = computed(() => {
   const { t0, t1 } = props.window
   // Convert raw counts to a daily hit rate (hits/day).
@@ -150,6 +163,12 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
 // middle) survives the clip. Width estimate: ~0.52 em per glyph.
 const fitsPill = (label, fontPx = 19) => label.length * 0.52 * fontPx <= TNODE_W - 16
 
+// With a favicon the label leaves room for the icon at the pill's left
+// and is always left-anchored past it.
+const labelX = (x) =>
+  extFavicon(x) ? x.x - TNODE_W / 2 + 36 : fitsPill(x.label) ? x.x : x.x - TNODE_W / 2 + 8
+const labelAnchor = (x) => (!extFavicon(x) && fitsPill(x.label) ? 'middle' : 'start')
+
 const countLabel = (n) =>
   n.readSec ? `${formatCount(n.views)}×${formatReadTime(n.readSec)}` : formatCount(n.views)
 </script>
@@ -180,7 +199,8 @@ const countLabel = (n) =>
           <rect :x="x.x - TNODE_W/2" :y="x.y - TNODE_H/2" :width="TNODE_W" :height="TNODE_H" :rx="TNODE_H/2"
                 :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
           <g :clip-path="`url(#xclip${i})`">
-            <text :x="fitsPill(x.label) ? x.x : x.x - TNODE_W/2 + 8" :y="x.y - TNODE_H*0.16" class="tnodeslug" dominant-baseline="middle" :style="{ textAnchor: fitsPill(x.label) ? 'middle' : 'start' }">{{ x.label }}</text>
+            <image v-if="extFavicon(x)" :href="extFavicon(x)" :x="x.x - TNODE_W/2 + 12" :y="x.y - TNODE_H*0.16 - 11" width="22" height="22" />
+            <text :x="labelX(x)" :y="x.y - TNODE_H*0.16" class="tnodeslug" dominant-baseline="middle" :style="{ textAnchor: labelAnchor(x) }">{{ x.label }}</text>
             <text :x="x.x" :y="x.y + TNODE_H*0.24" class="tnodecount" dominant-baseline="middle">{{ formatCount(x.count) }}</text>
           </g>
         </a>
@@ -189,7 +209,8 @@ const countLabel = (n) =>
           <rect :x="x.x - TNODE_W/2" :y="x.y - TNODE_H/2" :width="TNODE_W" :height="TNODE_H" :rx="TNODE_H/2"
                 :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
           <g :clip-path="`url(#xclip${i})`">
-            <text :x="fitsPill(x.label) ? x.x : x.x - TNODE_W/2 + 8" :y="x.y - TNODE_H*0.16" class="tnodeslug" dominant-baseline="middle" :style="{ textAnchor: fitsPill(x.label) ? 'middle' : 'start' }">{{ x.label }}</text>
+            <image v-if="extFavicon(x)" :href="extFavicon(x)" :x="x.x - TNODE_W/2 + 12" :y="x.y - TNODE_H*0.16 - 11" width="22" height="22" />
+            <text :x="labelX(x)" :y="x.y - TNODE_H*0.16" class="tnodeslug" dominant-baseline="middle" :style="{ textAnchor: labelAnchor(x) }">{{ x.label }}</text>
             <text :x="x.x" :y="x.y + TNODE_H*0.24" class="tnodecount" dominant-baseline="middle">{{ formatCount(x.count) }}</text>
           </g>
         </g>

frontend/src/analytics/format.js

@@ -152,6 +152,15 @@ function externalSlug(origin) {
   }
 }
 
+/** Origin (scheme://host) of an external https URL, for favicon lookup. */
+function externalOrigin(url) {
+  try {
+    return new URL(url).origin
+  } catch {
+    return ''
+  }
+}
+
 /** Format one trail step: an internal page or an external https origin. */
 function stepOf(path, titles) {
   if (path?.startsWith('/')) {
@@ -163,6 +172,7 @@ function stepOf(path, titles) {
       slug: externalSlug(path),
       title: 'External site',
       external: true,
+      origin: externalOrigin(path),
     }
   }
   return null

frontend/src/pagerite.js

@@ -175,14 +175,6 @@ import "overlayscrollbars/overlayscrollbars.css";
     }
 
     if (isAdmin) {
-      // Teach the backend the site's public origin (used for absolute
-      // social/canonical URLs): unlike request headers, location.origin
-      // reflects the real scheme and host even behind reverse proxies.
-      fetch("/_api/site-url", {
-        method: "POST",
-        headers: { "content-type": "application/json" },
-        body: JSON.stringify({ url: location.origin }),
-      }).catch(() => {});
       // Warm the cache with the editor bundle: the hashed asset is
       // immutable, so preloading costs nothing and the pens then open
       // instantly. The analytics page has no editor.

pagerite/__main__.py

@@ -73,6 +73,14 @@ def _download_dbip() -> None:
 def main() -> None:
     """Run the backend server with optional arguments."""
     parser = argparse.ArgumentParser(description="Run the pagerite server.")
+    parser.add_argument(
+        "hostname",
+        nargs="?",
+        default="localhost",
+        help=("Public hostname of the site; names the data directory "
+              "<hostname>/{content.kantadb, analytics.json, files} under the "
+              "cwd (default: localhost)."),
+    )
     parser.add_argument(
         "-l",
         "--listen",
@@ -85,6 +93,9 @@ def main() -> None:
         help="Download/update the DB-IP city lite database before starting.",
     )
     args = parser.parse_args()
+    # Export the hostname before pagerite.app is imported: it derives the
+    # data directory and public origin from it at import time.
+    os.environ["PAGERITE_HOSTNAME"] = args.hostname
     if args.dbip:
         _download_dbip()
     dev = {"reload": True, "reload_dirs": ["pagerite"]} if DEVMODE else {}

pagerite/analytics.py

@@ -190,11 +190,27 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
     is_404: bool = False
 
 
+class Favicon(msgspec.Struct, omit_defaults=True):
+    """Favicon fetch record for one external https origin.
+
+    The icon itself is stored on disk under a content-hashed name (like
+    uploads, but outside the kanta db), referenced here by ``file``; an
+    empty ``file`` is a known miss, retried after ``_FAVICON_RETRY``.
+    """
+
+    #: Content-hashed file name of the stored icon, "" when the fetch failed.
+    file: str = ""
+    #: When the fetch was last attempted.
+    fetched: datetime | None = None
+
+
 class Analytics(msgspec.Struct, omit_defaults=True):
     """Root of the analytics JSON file. Append-only by design: old data is
     dropped by deleting list entries / bucket keys."""
 
     visits: list[Visit] = []
+    #: Favicon fetch records keyed by external https origin.
+    favicons: dict[str, Favicon] = {}
     #: Document GETs that never produced a ping, treated as crawler/bot hits.
     crawlers: list[CrawlerHit] = []
     #: Requests from abusive IPs (see AbuseHit), grouped by IP in the viewer.
@@ -219,6 +235,9 @@ class Display(msgspec.Struct, omit_defaults=True):
     crawlers: list[CrawlerHit] = []
     abuse: list[AbuseHit] = []
     clients: dict[bytes, Client] = {}
+    #: origin -> URL path of the stored favicon ("/_favicons/<file>"),
+    #: only for origins whose icon was fetched successfully.
+    favicons: dict[str, str] = {}
     #: Page transitions per 5-minute bucket (sparse):
     #: from -> to -> bucket ISO -> count. ``from`` is the referer origin or
     #: "(direct)" for initial loads, a page path for pings.
@@ -302,6 +321,9 @@ def _utm_tags(query: str) -> dict[str, str]:
 
 _CRAWLER_TIMEOUT = timedelta(seconds=10)
 
+#: How long a failed favicon fetch suppresses retries for the same origin.
+_FAVICON_RETRY = timedelta(days=7)
+
 #: UAs of JS-running crawlers, which would register as visitors on their
 #: ping.  Anything calling itself a "bot" or "spider" matches; known crawlers
 #: without those tokens (GoogleOther) are listed as extra alternates.  No
@@ -470,6 +492,11 @@ class Store:
             crawlers=[h for h in self.data.crawlers if not self._hidden(h.client)],
             abuse=[h for h in self.data.abuse if not self._hidden(h.client)],
             clients={h: c for h, c in self.data.clients.items() if not c.hide},
+            favicons={
+                origin: f"/_f/{f.file}"
+                for origin, f in self.data.favicons.items()
+                if f.file
+            },
         )
         for visit in visits:
             bucket = _bucket(visit.start)
@@ -544,6 +571,34 @@ class Store:
         if changed:
             self._save()
 
+    def favicon_origins_needed(self) -> list[str]:
+        """External https origins seen in visits whose favicon needs fetching.
+
+        Covers visit referers and external exit targets (trail and navs).
+        Origins with a stored icon, or a miss younger than
+        ``_FAVICON_RETRY``, are skipped.
+        """
+        origins: set[str] = set()
+        for visit in self.data.visits:
+            if visit.referer:
+                origins.add(visit.referer)
+            for target in list(visit.trail.values()) + list(visit.navs.values()):
+                origin = _origin(target.to)
+                if origin is not None:
+                    origins.add(origin)
+        now = datetime.now(UTC)
+        return [
+            origin
+            for origin in origins
+            if (f := self.data.favicons.get(origin)) is None
+            or (not f.file and (f.fetched is None or now - f.fetched > _FAVICON_RETRY))
+        ]
+
+    def record_favicon(self, origin: str, file: str = "") -> None:
+        """Store the favicon fetch result for ``origin`` ("" = miss)."""
+        self.data.favicons[origin] = Favicon(file=file, fetched=datetime.now(UTC))
+        self._save()
+
     def _abuse_hit(
         self,
         client_hash: bytes,

pagerite/app.py

@@ -21,7 +21,7 @@ import re
 import shutil
 import socket
 from collections.abc import AsyncIterator
-from contextlib import asynccontextmanager
+from contextlib import asynccontextmanager, suppress
 from datetime import UTC, datetime
 from email.utils import format_datetime
 from functools import lru_cache
@@ -30,7 +30,7 @@ from urllib.parse import urlparse
 from xml.sax.saxutils import escape as xml_escape
 
 import blake3
-import msgspec
+import httpx
 from fastapi import (
     FastAPI,
     HTTPException,
@@ -58,14 +58,26 @@ from pagerite.data import (
 )
 from pagerite.markdown import has_h1, render, toggle_task
 
-DB_PATH = os.getenv("PAGERITE_DB", "pagerite.kantadb")
+# Site identity: the hostname comes from the CLI (first positional argument,
+# exported as PAGERITE_HOSTNAME) and names the per-site data directory
+# ``<hostname>/{content.kantadb, analytics.json, files}`` under the cwd.
+HOSTNAME = os.getenv("PAGERITE_HOSTNAME", "localhost")
+SITE_DIR = Path(HOSTNAME)
+#: Public origin of the site, used for absolute social/canonical/sitemap
+#: URLs. Localhost serves varying ports, so it falls back to the request's
+#: own base URL instead.
+SITE_URL = f"https://{HOSTNAME}" if HOSTNAME != "localhost" else ""
+
+DB_PATH = os.getenv("PAGERITE_DB", str(SITE_DIR / "content.kantadb"))
 
 # Visit analytics go to their own JSON file, not the kanta database.
-ANALYTICS_PATH = Path(
-    os.getenv("PAGERITE_ANALYTICS", DB_PATH.replace(".kantadb", "") + ".analytics.json")
-)
+ANALYTICS_PATH = Path(os.getenv("PAGERITE_ANALYTICS", str(SITE_DIR / "analytics.json")))
 analytics_store = analytics.Store(ANALYTICS_PATH)
 
+# Content-addressed file store (uploads, seed assets, fetched favicons):
+# files on disk under hash-prefixed names, cached in RAM, served at /_f/.
+FILES_DIR = Path(os.getenv("PAGERITE_FILES", str(SITE_DIR / "files")))
+
 # Live WebSocket clients for the analytics stream.
 _analytics_ws_clients: set[WebSocket] = set()
 _analytics_broadcast_task: asyncio.Task | None = None
@@ -160,7 +172,7 @@ _geoip = GeoIP()
 
 # Our own data root; kanta edits it in place, reads are plain attribute access.
 data = Data()
-kanta = Kanta(DB_PATH, data)
+kanta = Kanta(DB_PATH, data, migrations="pagerite.migrations")
 
 # Vue build served at the site root, no SPA catch-all (assets only). The
 # build mirrors the URL space: hashed, immutable files live under
@@ -178,7 +190,7 @@ def _hash_name(body: bytes, orig: str) -> str:
 def _store_seed_file(markdown: str, banner: str, orig: str, body: bytes) -> tuple[str, str]:
     """Store a seed file content-addressed and point references at /_f/."""
     name = _hash_name(body, orig)
-    data.files.setdefault(name, body)
+    file_store.put(name, body)
     markdown = markdown.replace(f"]({orig}", f"](/_f/{name}")
     banner = banner.replace(f'src="/{orig}"', f'src="/_f/{name}"')
     banner = banner.replace(f'src="{orig}"', f'src="/_f/{name}"')
@@ -259,12 +271,15 @@ def _seed(data: Data) -> None:
 async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
     """Open the database, migrate legacy content, load assets, load GeoIP."""
     await kanta.open()
+    await asyncio.to_thread(file_store.load)
     _migrate_legacy()
     await frontend.load()
     # Decompress/open the DB-IP MMDB once at startup.  Lookups are then
     # read-only and safe to run in background ``to_thread`` workers.
     await asyncio.to_thread(_geoip._load)
     analytics_store.subscribe(_schedule_analytics_broadcast)
+    # Backfill favicons for external sites already in the recorded data.
+    _schedule_favicon_fetch()
     yield
     analytics_store.unsubscribe(_schedule_analytics_broadcast)
     await kanta.close()
@@ -295,6 +310,57 @@ async def _headers(request: Request, call_next) -> Response:
 _zstd = ZstdCompressor(9)
 
 
+class FileStore:
+    """Content-addressed files on disk, fully cached in RAM.
+
+    Every file is kept in RAM uncompressed and zstd-compressed (the
+    compressed copy only when it actually shrinks the body), so ``/_f``
+    serves both encodings without touching disk or re-compressing.
+    """
+
+    def __init__(self, path: Path) -> None:
+        self.path = path
+        #: name -> (uncompressed body, zstd body or None)
+        self._cache: dict[str, tuple[bytes, bytes | None]] = {}
+
+    @staticmethod
+    def _entry(body: bytes) -> tuple[bytes, bytes | None]:
+        compressed = _zstd.compress(body)
+        return body, compressed if len(compressed) < len(body) else None
+
+    def load(self) -> None:
+        """Read every stored file into the RAM cache (startup)."""
+        try:
+            entries = sorted(self.path.iterdir())
+        except FileNotFoundError:
+            return
+        for f in entries:
+            if f.is_file() and not f.name.startswith("."):
+                self._cache.setdefault(f.name, self._entry(f.read_bytes()))
+
+    def get(self, name: str) -> tuple[bytes, bytes | None] | None:
+        return self._cache.get(name)
+
+    def put(self, name: str, body: bytes) -> None:
+        """Store ``body`` under ``name`` on disk and in the RAM cache."""
+        if name in self._cache:
+            return
+        self.path.mkdir(parents=True, exist_ok=True)
+        (self.path / name).write_bytes(body)
+        self._cache[name] = self._entry(body)
+
+    def delete(self, name: str) -> None:
+        self._cache.pop(name, None)
+        with suppress(FileNotFoundError):
+            (self.path / name).unlink()
+
+    def __contains__(self, name: str) -> bool:
+        return name in self._cache
+
+
+file_store = FileStore(FILES_DIR)
+
+
 def _render_html(kind: str, path: str, base_url: str) -> str:
     """Render one of the generated pages (see _html_response)."""
     if kind == "page":
@@ -341,9 +407,9 @@ def _html_response(
     if-none-match revalidations with a 304.
     """
     zstd = "zstd" in request.headers.get("accept-encoding", "")
-    # Absolute social/canonical URLs use the learned public origin; until
-    # an admin visit teaches it, fall back to the request's own base URL.
-    base_url = data.site_url or str(request.base_url).rstrip("/")
+    # Absolute social/canonical URLs use the site's public origin; on
+    # localhost (varying ports) fall back to the request's own base URL.
+    base_url = SITE_URL or str(request.base_url).rstrip("/")
     if DEVMODE:
         identity = _render_html(kind, path, base_url).encode()
         body = _zstd.compress(identity) if zstd else identity
@@ -516,33 +582,6 @@ async def put_settings(settings: SettingsIn) -> None:
         data.version += 1
 
 
-class SiteUrlIn(BaseModel):
-    """Payload for learning the site's public origin."""
-
-    url: str
-
-
-@app.post("/_api/site-url", status_code=204)
-async def learn_site_url(payload: SiteUrlIn) -> None:
-    """Learn the site's public origin (scheme + host) from an admin browser.
-
-    pagerite.js reports location.origin once an admin session is detected:
-    unlike request Host headers it reflects the real public scheme and host
-    even behind reverse proxies, with zero manual configuration. Stored in
-    the database with a version bump so cached pages re-render with correct
-    absolute social/canonical URLs.
-    """
-    url = payload.url.rstrip("/")
-    parsed = urlparse(url)
-    if parsed.scheme not in ("http", "https") or not parsed.netloc or parsed.path:
-        raise HTTPException(400, "not an origin")
-    if url == data.site_url:
-        return
-    with kanta.transaction("learn site url"):
-        data.site_url = url
-        data.version += 1
-
-
 @app.put("/_api/settings/favicon")
 async def put_favicon(request: Request) -> dict[str, str]:
     """Upload a favicon into the content-addressed store and activate it.
@@ -555,8 +594,8 @@ async def put_favicon(request: Request) -> dict[str, str]:
     if not body:
         raise HTTPException(400, "empty file")
     stored = _hash_name(body, request.headers.get("x-filename", "favicon.ico"))
+    file_store.put(stored, body)
     with kanta.transaction("upload favicon"):
-        data.files[stored] = body
         data.favicon = stored
         data.version += 1
     return {"path": f"/_f/{stored}"}
@@ -622,9 +661,7 @@ async def upload_file(name: str, request: Request) -> dict[str, str]:
         raise HTTPException(400, "bad file name")
     body = await request.body()
     stored = _hash_name(body, name)
-    with kanta.transaction("upload file", extra=name):
-        data.files[stored] = body
-        data.version += 1
+    file_store.put(stored, body)
     return {"path": f"/_f/{stored}"}
 
 
@@ -632,11 +669,9 @@ async def upload_file(name: str, request: Request) -> dict[str, str]:
 async def delete_file(name: str) -> None:
     """Remove a file from the content-addressed store (no refcounting:
     other pages referencing the same content will 404)."""
-    if name not in data.files:
+    if name not in file_store:
         raise HTTPException(404, "no such file")
-    with kanta.transaction("delete file", extra=name):
-        del data.files[name]
-        data.version += 1
+    file_store.delete(name)
 
 
 @app.get("/_themes/{name}/{filename}")
@@ -676,18 +711,22 @@ async def theme_file(name: str, filename: str, request: Request) -> Response:
 @app.get("/_f/{name}")
 async def stored_file(name: str, request: Request) -> Response:
     """Serve a file from the content-addressed store (immutable: the name
-    is its own hash, so cache forever)."""
-    body = data.files.get(name)
-    if body is None:
+    is its own hash, so cache forever).  Bodies are served from the RAM
+    cache, zstd-compressed when the client accepts it and compression
+    actually shrank the file."""
+    entry = file_store.get(name)
+    if entry is None:
         raise HTTPException(404)
     if request.headers.get("if-none-match") == name:
         return Response(status_code=304)
+    body, compressed = entry
+    headers = {"etag": name, "cache-control": "public, max-age=31536000, immutable"}
+    if compressed is not None and "zstd" in request.headers.get("accept-encoding", ""):
+        headers["content-encoding"] = "zstd"
+        headers["vary"] = "accept-encoding"
+        body = compressed
     mime = mimetypes.guess_type(name)[0] or "application/octet-stream"
-    return Response(
-        body,
-        media_type=mime,
-        headers={"etag": name, "cache-control": "public, max-age=31536000, immutable"},
-    )
+    return Response(body, media_type=mime, headers=headers)
 
 
 @app.delete("/_api/pages/{path:path}", status_code=204)
@@ -778,6 +817,68 @@ def _schedule_client_enrichment(client_hashes: list[bytes]) -> None:
         asyncio.create_task(_enrich_client(client_hash))
 
 
+#: Icon MIME -> file extension for the stored favicon name.  The extension
+#: reflects the actual content, not the /favicon.ico request path.
+_FAVICON_EXT = {
+    "image/x-icon": ".ico",
+    "image/vnd.microsoft.icon": ".ico",
+    "image/png": ".png",
+    "image/gif": ".gif",
+    "image/jpeg": ".jpg",
+    "image/webp": ".webp",
+    "image/avif": ".avif",
+    "image/svg+xml": ".svg",
+}
+
+_FAVICON_MAX_BYTES = 65536
+
+#: Origins with a fetch task currently in flight.
+_favicon_in_flight: set[str] = set()
+
+
+async def _fetch_favicon(origin: str) -> None:
+    """Fetch ``{origin}/favicon.ico`` and store it content-hashed on disk.
+
+    The result (icon file name, or "" for a miss) is recorded in the
+    analytics store; misses are retried after analytics._FAVICON_RETRY.
+    Never raises: analytics must not break page serving.
+    """
+    try:
+        async with httpx.AsyncClient(follow_redirects=True, timeout=8) as client:
+            r = await client.get(f"{origin}/favicon.ico")
+        body = r.content
+        if not (200 <= r.status_code < 300) or not body or len(body) > _FAVICON_MAX_BYTES:
+            analytics_store.record_favicon(origin)
+            return
+        mime = r.headers.get("content-type", "").split(";")[0].strip().lower()
+        if not mime.startswith("image/"):
+            # Served without an image type: sniff SVG, else assume ICO.
+            if b"<svg" in body[:1024]:
+                mime = "image/svg+xml"
+            elif mime in ("", "application/octet-stream", "text/plain"):
+                mime = "image/x-icon"
+            else:
+                analytics_store.record_favicon(origin)
+                return
+        ext = _FAVICON_EXT.get(mime, ".ico")
+        name = _hash_name(body, f"favicon{ext}")
+        file_store.put(name, body)
+        analytics_store.record_favicon(origin, name)
+    except (httpx.HTTPError, OSError):
+        analytics_store.record_favicon(origin)
+    finally:
+        _favicon_in_flight.discard(origin)
+
+
+def _schedule_favicon_fetch() -> None:
+    """Start background favicon fetches for origins that need one."""
+    for origin in analytics_store.favicon_origins_needed():
+        if origin in _favicon_in_flight:
+            continue
+        _favicon_in_flight.add(origin)
+        asyncio.create_task(_fetch_favicon(origin))
+
+
 async def _broadcast_analytics() -> None:
     """Send the current analytics snapshot to every connected WS client."""
     if not _analytics_ws_clients:
@@ -857,6 +958,7 @@ async def analytics_ping(
         visit = analytics_store.data.visits[visit_index]
         asyncio.create_task(_enrich_client(visit.client))
     _schedule_client_enrichment(flushed_clients)
+    _schedule_favicon_fetch()
 
 
 def _track_entry(path: str, request: Request, *, status: int = 200) -> list[bytes]:
@@ -888,7 +990,7 @@ def _track_entry(path: str, request: Request, *, status: int = 200) -> list[byte
         and _client_ip(request) == "127.0.0.1"
     ):
         return []
-    own_origin = f"https://{urlparse(str(request.base_url)).netloc}"
+    own_origin = SITE_URL or f"https://{urlparse(str(request.base_url)).netloc}"
     full_path = f"{request.url.path}{_query_suffix(request)}"
     return analytics_store.track_entry(
         request.headers.get("referer", ""),
@@ -1112,7 +1214,7 @@ async def front_page(request: Request) -> Response:
 @app.get("/sitemap.xml")
 async def sitemap(request: Request) -> Response:
     """Dynamically generate a sitemap of all published article pages."""
-    base = str(request.base_url).rstrip("/")
+    base = SITE_URL or str(request.base_url).rstrip("/")
     entries: list[tuple[str, datetime, int]] = []
 
     def walk(
@@ -1175,7 +1277,7 @@ async def sitemap(request: Request) -> Response:
 @app.get("/robots.txt")
 async def robots_txt(request: Request) -> Response:
     """Allow all crawling and point crawlers at the sitemap."""
-    base = str(request.base_url).rstrip("/")
+    base = SITE_URL or str(request.base_url).rstrip("/")
     body = f"User-agent: *\nAllow: /\nSitemap: {base}/sitemap.xml\n"
     return Response(
         body,

pagerite/data.py

@@ -75,10 +75,6 @@ class Data(msgspec.Struct):
 
     #: Top-level menu items by slug; "" is the front page.
     menu: dict[str, Node] = {}
-    #: Content-addressed file store: name (blake3 hash prefix + extension)
-    #: -> bytes, served immutable at "/_f/{name}". Absolute URLs that stay
-    #: valid when pages move.
-    files: dict[str, bytes] = {}
     #: Bumped on every structure/content write, so page ETags (which embed
     #: it) invalidate cached copies when navigation-affecting changes happen.
     version: int = 0
@@ -97,15 +93,10 @@ class Data(msgspec.Struct):
     #: Raw site-wide custom CSS, injected inline in every page <head>.
     #: Trusted author content; not sanitized.
     custom_css: str = ""
-    #: Favicon: name of a file in `files` (content-addressed), linked as
-    #: <link rel="icon"> on every page. Empty = the build's /favicon.ico.
+    #: Favicon: content-addressed file name (served at "/_f/{name}"),
+    #: linked as <link rel="icon"> on every page. Empty = the build's
+    #: /favicon.ico.
     favicon: str = ""
-    #: Public origin (scheme + host) of the site, learned from admin
-    #: browsers (POST /_api/site-url — location.origin is correct even
-    #: behind reverse proxies, unlike request Host headers). Used for
-    #: absolute social/canonical URLs; empty = fall back to the request's
-    #: own base URL.
-    site_url: str = ""
     #: Legacy flat page store (pre-tree databases); migrated into `menu`
     #: on startup, then cleared. Never written otherwise.
     pages: dict[str, Page] = {}

pagerite/migrations.py

@@ -0,0 +1,23 @@
+"""Kanta schema migrations, discovered by name (``migrate_vN``).
+
+Each function receives the raw state dict (JSON-level: bytes are base64
+strings) before it is decoded into ``Data`` structs, and runs exactly once
+per database based on its recorded version.
+"""
+
+import base64
+
+
+def migrate_v1(d: dict) -> None:
+    """Move in-database file blobs to the on-disk content-addressed store."""
+    files = d.pop("files", None)
+    if not files:
+        return
+    # Deferred import: app.py owns the file store and passes this module to
+    # Kanta; at migration time (lifespan open) the module is fully loaded.
+    from pagerite.app import file_store
+
+    for name, body in files.items():
+        if isinstance(body, str):  # JSON-level bytes are base64 strings
+            body = base64.b64decode(body)
+        file_store.put(name, body)